JUnit version 4.13.2
.02:11:36.029 [DEBUG - main] (RandomUtils.java:49) Using random seed: 1783476696021
02:11:36.088 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
02:11:36.121 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
02:11:36.134 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskf2jM_v/test-tmp
02:11:36.161 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
02:11:36.164 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskf2jM_v/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
02:11:36.395 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

02:11:36.570 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
02:11:36.571 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

02:11:36.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:36.576552  7578 mini_oidc.cc:150] Starting JWKS server
02:11:36.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:36.580785  7578 webserver.cc:534] Webserver started at http://127.0.0.1:34115/ using document root <none> and password file <none>
02:11:36.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:36.581341  7578 mini_oidc.cc:186] Starting OIDC Discovery server
02:11:36.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:36.583580  7578 webserver.cc:534] Webserver started at http://127.0.0.1:43665/ using document root <none> and password file <none>
02:11:36.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T02:11:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
02:11:36.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T02:11:36Z Disabled control of system clock
02:11:36.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:36.646940  7578 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:11:36.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:11:36.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-0/wal
02:11:36.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-0/data
02:11:36.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:11:36.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:11:36.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:11:36.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:11:36.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:11:36.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:11:36.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-0/logs
02:11:36.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-0/data/info.pb
02:11:36.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:11:36.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:11:36.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:11:36.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:11:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:11:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:11:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:11:36.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:11:36.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
02:11:36.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:11:36.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:11:36.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:11:36.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.190:37199
02:11:36.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.190
02:11:36.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:11:36.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:11:36.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:11:36.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:11:36.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:11:36.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:11:36.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:11:36.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:11:36.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34115/jwks.json with env {}
02:11:36.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:36.955924  7590 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:11:36.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:36.956490  7590 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:11:36.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:36.956971  7590 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:11:36.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:36.989634  7590 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
02:11:36.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:36.989951  7590 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:11:36.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:36.990159  7590 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
02:11:36.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:36.990378  7590 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
02:11:36.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:36.990572  7590 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
02:11:36.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:36.990717  7590 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:34115/jwks.json
02:11:37.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:37.027344  7590 master_runner.cc:387] Master server non-default flags:
02:11:37.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:11:37.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:11:37.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:11:37.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:11:37.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-0/data
02:11:37.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-0/wal
02:11:37.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:11:37.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:11:37.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:11:37.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:11:37.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:11:37.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.190:37199
02:11:37.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:11:37.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:11:37.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:11:37.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34115/jwks.json
02:11:37.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:11:37.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:11:37.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-0/data/info.pb
02:11:37.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.190
02:11:37.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:11:37.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:11:37.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:11:37.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:11:37.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:11:37.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:11:37.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-0/logs
02:11:37.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:11:37.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:11:37.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:11:37.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
02:11:37.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:11:37.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:11:37.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:11:37.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on e7f111948823
02:11:37.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16850
02:11:37.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:11:37.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:37.028734  7590 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:11:37.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:37.030464  7590 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:11:37.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:37.041798  7596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:11:38.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:38.445856  7595 debug-util.cc:398] Leaking SignalData structure 0x7b0800028020 after lost signal to thread 7590
02:11:38.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:38.916481  7595 kernel_stack_watchdog.cc:198] Thread 7590 stuck at /home/jenkins-slave/workspace/build_and_test@2/src/kudu/util/thread.cc:642 for 401ms:
02:11:38.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
02:11:38.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
02:11:38.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
02:11:38.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
02:11:38.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:37.043149  7597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:11:38.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:38.918167  7598 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1874 milliseconds
02:11:38.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:38.918972  7590 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.877s	user 0.662s	sys 1.212s
02:11:38.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:38.919324  7590 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.877s	user 0.662s	sys 1.212s
02:11:38.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:38.920936  7599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:11:38.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:38.920861  7590 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:11:38.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:38.923422  7590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:11:38.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:38.926054  7590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:11:38.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:38.927451  7590 hybrid_clock.cc:648] HybridClock initialized: now 1783476698927415 us; error 37 us; skew 500 ppm
02:11:38.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:38.928416  7590 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:11:38.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:38.934851  7590 webserver.cc:534] Webserver started at http://127.7.102.190:34679/ using document root <none> and password file <none>
02:11:38.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:38.936043  7590 fs_manager.cc:362] Metadata directory not provided
02:11:38.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:38.936307  7590 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:11:38.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:38.936916  7590 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:11:38.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:38.941403  7590 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-0/data/instance:
02:11:38.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "11e6b48067d646c2a58d9d85239ca222"
02:11:38.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:38 on dist-test-slave-5s6b"
02:11:38.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:38.942858  7590 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-0/wal/instance:
02:11:38.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "11e6b48067d646c2a58d9d85239ca222"
02:11:38.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:38 on dist-test-slave-5s6b"
02:11:38.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:38.950624  7590 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.007s	sys 0.001s
02:11:38.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:38.957011  7606 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:11:38.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:38.958592  7590 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
02:11:38.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:38.959049  7590 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-0/data,/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-0/wal
02:11:38.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "11e6b48067d646c2a58d9d85239ca222"
02:11:38.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:38 on dist-test-slave-5s6b"
02:11:38.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:38.959529  7590 fs_report.cc:389] FS layout report
02:11:38.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:11:38.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-0/wal
02:11:38.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-0/wal
02:11:38.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-0/data/data
02:11:38.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:11:38.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:11:38.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:11:38.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:11:38.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:11:38.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:11:38.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:11:38.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:11:38.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:11:38.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:11:38.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:11:39.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:39.021356  7590 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:11:39.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:39.022902  7590 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:11:39.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:39.023368  7590 kserver.cc:163] Server-wide thread pool size limit: 3276
02:11:39.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:39.093782  7590 rpc_server.cc:307] RPC server started. Bound to: 127.7.102.190:37199
02:11:39.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:39.093875  7657 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.102.190:37199 every 8 connection(s)
02:11:39.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:39.096663  7590 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-0/data/info.pb
02:11:39.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:39.099522  7578 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu as pid 7590
02:11:39.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:39.100085  7578 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-0/wal/instance
02:11:39.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:39.104835  7658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
02:11:39.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:39.106032  7578 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:11:39.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:11:39.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-1/wal
02:11:39.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-1/data
02:11:39.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:11:39.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:11:39.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:11:39.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:11:39.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:11:39.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:11:39.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-1/logs
02:11:39.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-1/data/info.pb
02:11:39.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:11:39.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:11:39.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:11:39.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:11:39.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:11:39.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:11:39.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:11:39.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:11:39.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
02:11:39.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:11:39.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:11:39.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:11:39.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.189:33589
02:11:39.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.189
02:11:39.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:11:39.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:11:39.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:11:39.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:11:39.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:11:39.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:11:39.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:11:39.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:11:39.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34115/jwks.json with env {}
02:11:39.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:39.125030  7658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } has no permanent_uuid. Determining permanent_uuid...
02:11:39.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:39.157404  7658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } has no permanent_uuid. Determining permanent_uuid...
02:11:39.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:39.159984  7608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.102.189:33589: connect: Connection refused (error 111)
02:11:39.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:39.164512  7658 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.102.189:33589: Network error: Client connection negotiation failed: client connection to 127.7.102.189:33589: connect: Connection refused (error 111)
02:11:39.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:39.214035  7658 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } attempt: 1
02:11:39.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:39.218868  7658 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.102.189:33589: Network error: Client connection negotiation failed: client connection to 127.7.102.189:33589: connect: Connection refused (error 111)
02:11:39.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:39.287297  7658 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } attempt: 2
02:11:39.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:39.291854  7658 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.102.189:33589: Network error: Client connection negotiation failed: client connection to 127.7.102.189:33589: connect: Connection refused (error 111)
02:11:39.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:39.383284  7658 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } attempt: 3
02:11:39.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:39.387804  7658 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.102.189:33589: Network error: Client connection negotiation failed: client connection to 127.7.102.189:33589: connect: Connection refused (error 111)
02:11:39.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:39.433171  7660 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:11:39.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:39.433736  7660 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:11:39.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:39.434391  7660 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:11:39.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:39.466900  7660 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
02:11:39.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:39.467257  7660 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:11:39.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:39.467466  7660 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
02:11:39.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:39.467655  7660 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
02:11:39.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:39.467844  7660 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
02:11:39.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:39.467984  7660 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:34115/jwks.json
02:11:39.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:39.505084  7660 master_runner.cc:387] Master server non-default flags:
02:11:39.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:11:39.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:11:39.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:11:39.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:11:39.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-1/data
02:11:39.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-1/wal
02:11:39.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:11:39.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:11:39.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:11:39.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:11:39.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:11:39.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.189:33589
02:11:39.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:11:39.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:11:39.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:11:39.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34115/jwks.json
02:11:39.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:11:39.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:11:39.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-1/data/info.pb
02:11:39.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.189
02:11:39.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:11:39.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:11:39.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:11:39.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:11:39.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:11:39.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:11:39.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-1/logs
02:11:39.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:11:39.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:11:39.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:11:39.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
02:11:39.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:11:39.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:11:39.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:11:39.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on e7f111948823
02:11:39.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16850
02:11:39.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:11:39.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:39.506552  7660 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:11:39.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:39.508371  7660 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:11:39.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:39.520678  7669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:11:39.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:39.531234  7658 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } attempt: 4
02:11:39.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:39.537407  7658 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.102.189:33589: Network error: Client connection negotiation failed: client connection to 127.7.102.189:33589: connect: Connection refused (error 111)
02:11:39.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:39.836920  7658 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } attempt: 5
02:11:39.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:39.845520  7658 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.102.189:33589: Network error: Client connection negotiation failed: client connection to 127.7.102.189:33589: connect: Connection refused (error 111)
02:11:40.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.393272  7658 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } attempt: 6
02:11:40.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:40.406888  7658 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.102.189:33589: Network error: Client connection negotiation failed: client connection to 127.7.102.189:33589: connect: Connection refused (error 111)
02:11:40.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:39.521701  7670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:11:40.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:40.819518  7672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:11:40.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:40.822129  7671 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1296 milliseconds
02:11:40.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.822280  7660 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:11:40.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.823724  7660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:11:40.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.826460  7660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:11:40.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.827849  7660 hybrid_clock.cc:648] HybridClock initialized: now 1783476700827818 us; error 40 us; skew 500 ppm
02:11:40.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.828750  7660 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:11:40.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.834733  7660 webserver.cc:534] Webserver started at http://127.7.102.189:34283/ using document root <none> and password file <none>
02:11:40.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.835717  7660 fs_manager.cc:362] Metadata directory not provided
02:11:40.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.835994  7660 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:11:40.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.836572  7660 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:11:40.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.840941  7660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-1/data/instance:
02:11:40.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "226ef9aaf0bb4c959b799ae41508d53c"
02:11:40.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:40 on dist-test-slave-5s6b"
02:11:40.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.842365  7660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-1/wal/instance:
02:11:40.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "226ef9aaf0bb4c959b799ae41508d53c"
02:11:40.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:40 on dist-test-slave-5s6b"
02:11:40.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.849828  7660 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.000s
02:11:40.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.855333  7681 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:11:40.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.856441  7660 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.002s
02:11:40.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.856825  7660 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-1/data,/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-1/wal
02:11:40.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "226ef9aaf0bb4c959b799ae41508d53c"
02:11:40.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:40 on dist-test-slave-5s6b"
02:11:40.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.857230  7660 fs_report.cc:389] FS layout report
02:11:40.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:11:40.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-1/wal
02:11:40.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-1/wal
02:11:40.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-1/data/data
02:11:40.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:11:40.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:11:40.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:11:40.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:11:40.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:11:40.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:11:40.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:11:40.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:11:40.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:11:40.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:11:40.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:11:40.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.910656  7660 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:11:40.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.912328  7660 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:11:40.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.912858  7660 kserver.cc:163] Server-wide thread pool size limit: 3276
02:11:40.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.982712  7660 rpc_server.cc:307] RPC server started. Bound to: 127.7.102.189:33589
02:11:40.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.982792  7732 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.102.189:33589 every 8 connection(s)
02:11:40.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.985667  7660 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-1/data/info.pb
02:11:40.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.991914  7733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
02:11:40.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.991958  7578 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu as pid 7660
02:11:40.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.992396  7578 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-1/wal/instance
02:11:40.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:40.994740  7578 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:11:40.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:11:40.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-2/wal
02:11:40.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-2/data
02:11:40.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:11:40.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:11:40.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:11:40.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:11:40.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:11:40.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:11:40.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-2/logs
02:11:40.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-2/data/info.pb
02:11:40.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:11:40.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:11:40.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:11:40.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:11:40.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:11:40.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:11:40.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:11:40.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:11:40.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
02:11:40.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:11:40.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:11:40.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:11:40.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.188:43935
02:11:40.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.188
02:11:40.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:11:40.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:11:40.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:11:40.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:11:40.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:11:40.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:11:40.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:11:40.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:11:40.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34115/jwks.json with env {}
02:11:41.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:41.005517  7733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } has no permanent_uuid. Determining permanent_uuid...
02:11:41.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:41.029256  7733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } has no permanent_uuid. Determining permanent_uuid...
02:11:41.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:41.043177  7733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } has no permanent_uuid. Determining permanent_uuid...
02:11:41.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.045652  7683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.102.188:43935: connect: Connection refused (error 111)
02:11:41.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.050451  7733 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.102.188:43935: Network error: Client connection negotiation failed: client connection to 127.7.102.188:43935: connect: Connection refused (error 111)
02:11:41.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:41.100006  7733 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } attempt: 1
02:11:41.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.104249  7733 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.102.188:43935: Network error: Client connection negotiation failed: client connection to 127.7.102.188:43935: connect: Connection refused (error 111)
02:11:41.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:41.172669  7733 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } attempt: 2
02:11:41.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.177229  7733 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.102.188:43935: Network error: Client connection negotiation failed: client connection to 127.7.102.188:43935: connect: Connection refused (error 111)
02:11:41.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:41.268741  7733 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } attempt: 3
02:11:41.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.273368  7733 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.102.188:43935: Network error: Client connection negotiation failed: client connection to 127.7.102.188:43935: connect: Connection refused (error 111)
02:11:41.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.329663  7735 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:11:41.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.330217  7735 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:11:41.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.330731  7735 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:11:41.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.363068  7735 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
02:11:41.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.363354  7735 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:11:41.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.363623  7735 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
02:11:41.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.363826  7735 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
02:11:41.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.364010  7735 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
02:11:41.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.364148  7735 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:34115/jwks.json
02:11:41.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:41.400424  7735 master_runner.cc:387] Master server non-default flags:
02:11:41.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:11:41.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:11:41.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:11:41.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:11:41.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-2/data
02:11:41.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-2/wal
02:11:41.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
02:11:41.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:11:41.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:11:41.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:11:41.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
02:11:41.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.188:43935
02:11:41.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
02:11:41.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:11:41.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
02:11:41.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34115/jwks.json
02:11:41.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:11:41.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:11:41.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-2/data/info.pb
02:11:41.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.188
02:11:41.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:11:41.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:11:41.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:11:41.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:11:41.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:11:41.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:11:41.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-2/logs
02:11:41.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:11:41.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:11:41.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:11:41.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
02:11:41.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:11:41.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:11:41.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:11:41.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on e7f111948823
02:11:41.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16850
02:11:41.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:11:41.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:41.401726  7735 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:11:41.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:41.403940  7735 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:11:41.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:41.416814  7733 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } attempt: 4
02:11:41.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.417141  7745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:11:41.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.427338  7733 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.102.188:43935: Network error: Client connection negotiation failed: client connection to 127.7.102.188:43935: connect: Connection refused (error 111)
02:11:41.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:41.467583  7658 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } attempt: 7
02:11:41.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:41.501500  7658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } has no permanent_uuid. Determining permanent_uuid...
02:11:41.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.522823  7658 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.102.188:43935: Network error: Client connection negotiation failed: client connection to 127.7.102.188:43935: connect: Connection refused (error 111)
02:11:41.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:41.581405  7658 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } attempt: 1
02:11:41.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.602515  7658 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.102.188:43935: Network error: Client connection negotiation failed: client connection to 127.7.102.188:43935: connect: Connection refused (error 111)
02:11:41.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:41.684087  7658 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } attempt: 2
02:11:41.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.692570  7658 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.102.188:43935: Network error: Client connection negotiation failed: client connection to 127.7.102.188:43935: connect: Connection refused (error 111)
02:11:41.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:41.726894  7733 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } attempt: 5
02:11:41.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.739970  7733 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.102.188:43935: Network error: Client connection negotiation failed: client connection to 127.7.102.188:43935: connect: Connection refused (error 111)
02:11:41.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:41.778103  7658 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } attempt: 3
02:11:41.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.785811  7658 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.102.188:43935: Network error: Client connection negotiation failed: client connection to 127.7.102.188:43935: connect: Connection refused (error 111)
02:11:41.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:41.926406  7658 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } attempt: 4
02:11:41.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.935899  7658 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.102.188:43935: Network error: Client connection negotiation failed: client connection to 127.7.102.188:43935: connect: Connection refused (error 111)
02:11:42.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:42.219465  7658 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } attempt: 5
02:11:42.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:42.228554  7658 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.102.188:43935: Network error: Client connection negotiation failed: client connection to 127.7.102.188:43935: connect: Connection refused (error 111)
02:11:42.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:42.287510  7733 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } attempt: 6
02:11:42.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:42.312101  7733 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.102.188:43935: Network error: Client connection negotiation failed: client connection to 127.7.102.188:43935: connect: Connection refused (error 111)
02:11:42.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:42.781138  7658 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } attempt: 6
02:11:42.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:42.813512  7658 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.7.102.188:43935: Network error: Client connection negotiation failed: client connection to 127.7.102.188:43935: connect: Connection refused (error 111)
02:11:43.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:42.818710  7744 debug-util.cc:398] Leaking SignalData structure 0x7b0800028020 after lost signal to thread 7735
02:11:43.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:43.152545  7735 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.735s	user 0.595s	sys 1.126s
02:11:43.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:43.152881  7735 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.735s	user 0.595s	sys 1.126s
02:11:43.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:41.420157  7746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:11:43.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:43.155810  7748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:11:43.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:43.158890  7747 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1739 milliseconds
02:11:43.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.158984  7735 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:11:43.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.160394  7735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:11:43.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.163303  7735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:11:43.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.164724  7735 hybrid_clock.cc:648] HybridClock initialized: now 1783476703164661 us; error 77 us; skew 500 ppm
02:11:43.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.165737  7735 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:11:43.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.172353  7735 webserver.cc:534] Webserver started at http://127.7.102.188:46159/ using document root <none> and password file <none>
02:11:43.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.173405  7735 fs_manager.cc:362] Metadata directory not provided
02:11:43.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.173663  7735 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:11:43.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.174225  7735 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:11:43.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.178714  7735 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-2/data/instance:
02:11:43.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d1e8969ccd524f099e9ed8cf83cfd83d"
02:11:43.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:43 on dist-test-slave-5s6b"
02:11:43.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.180016  7735 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-2/wal/instance:
02:11:43.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d1e8969ccd524f099e9ed8cf83cfd83d"
02:11:43.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:43 on dist-test-slave-5s6b"
02:11:43.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.187273  7735 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.002s	sys 0.006s
02:11:43.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.193514  7761 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:11:43.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.194782  7735 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.001s
02:11:43.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.195152  7735 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-2/data,/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-2/wal
02:11:43.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d1e8969ccd524f099e9ed8cf83cfd83d"
02:11:43.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:43 on dist-test-slave-5s6b"
02:11:43.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.195555  7735 fs_report.cc:389] FS layout report
02:11:43.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:11:43.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-2/wal
02:11:43.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-2/wal
02:11:43.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-2/data/data
02:11:43.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:11:43.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:11:43.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:11:43.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:11:43.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:11:43.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:11:43.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:11:43.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:11:43.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:11:43.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:11:43.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:11:43.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.248095  7735 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:11:43.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.249908  7735 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:11:43.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.250525  7735 kserver.cc:163] Server-wide thread pool size limit: 3276
02:11:43.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.356989  7735 rpc_server.cc:307] RPC server started. Bound to: 127.7.102.188:43935
02:11:43.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.357054  7812 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.102.188:43935 every 8 connection(s)
02:11:43.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.359997  7735 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-2/data/info.pb
02:11:43.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.365839  7813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
02:11:43.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.367429  7578 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu as pid 7735
02:11:43.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.367753  7578 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/master-2/wal/instance
02:11:43.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.372651  7733 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } attempt: 7
02:11:43.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.381392  7813 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } has no permanent_uuid. Determining permanent_uuid...
02:11:43.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.416791  7813 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } has no permanent_uuid. Determining permanent_uuid...
02:11:43.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.425494  7733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c: Bootstrap starting.
02:11:43.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.433501  7813 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } has no permanent_uuid. Determining permanent_uuid...
02:11:43.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.434525  7733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c: Neither blocks nor log segments found. Creating new log.
02:11:43.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.437536  7733 log.cc:826] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c: Log is configured to *not* fsync() on all Append() calls
02:11:43.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.443650  7733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c: No bootstrap required, opened a new log
02:11:43.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.458271  7813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d: Bootstrap starting.
02:11:43.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.465322  7813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d: Neither blocks nor log segments found. Creating new log.
02:11:43.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.468488  7813 log.cc:826] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d: Log is configured to *not* fsync() on all Append() calls
02:11:43.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.468401  7733 raft_consensus.cc:359] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } }
02:11:43.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.469089  7733 raft_consensus.cc:385] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:11:43.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.469326  7733 raft_consensus.cc:740] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226ef9aaf0bb4c959b799ae41508d53c, State: Initialized, Role: FOLLOWER
02:11:43.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.470173  7733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } }
02:11:43.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.472987  7825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } } }
02:11:43.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.473886  7825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [sys.catalog]: This master's current role is: FOLLOWER
02:11:43.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.474637  7733 sys_catalog.cc:565] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [sys.catalog]: configured and running, proceeding with master startup.
02:11:43.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.475160  7813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d: No bootstrap required, opened a new log
02:11:43.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.500855  7813 raft_consensus.cc:359] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } }
02:11:43.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.501943  7813 raft_consensus.cc:385] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:11:43.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.502290  7813 raft_consensus.cc:740] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e8969ccd524f099e9ed8cf83cfd83d, State: Initialized, Role: FOLLOWER
02:11:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.503334  7813 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } }
02:11:43.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.512331  7837 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } } }
02:11:43.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.513269  7837 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [sys.catalog]: This master's current role is: FOLLOWER
02:11:43.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.516444  7813 sys_catalog.cc:565] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [sys.catalog]: configured and running, proceeding with master startup.
02:11:43.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:43.522368  7836 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
02:11:43.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:43.523180  7836 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
02:11:43.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.509526  7660 master_runner.cc:187] Error getting master registration for 127.7.102.190:37199: OK, instance_id { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" instance_seqno: 1783476699074762 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
02:11:43.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.547677  7660 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
02:11:43.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:43.572518  7849 catalog_manager.cc:1593] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
02:11:43.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:43.573635  7849 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
02:11:43.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.558732  7735 master_runner.cc:187] Error getting master registration for 127.7.102.190:37199: OK, instance_id { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" instance_seqno: 1783476699074762 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
02:11:43.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.585602  7735 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
02:11:43.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:43.671154  7687 tablet.cc:2406] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:11:43.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:43.757157  7767 tablet.cc:2406] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:11:43.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.762511  7825 raft_consensus.cc:493] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:11:43.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.763067  7825 raft_consensus.cc:515] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } }
02:11:43.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.766520  7837 raft_consensus.cc:493] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:11:43.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.767019  7837 raft_consensus.cc:515] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } }
02:11:43.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.769474  7837 leader_election.cc:290] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11e6b48067d646c2a58d9d85239ca222 (127.7.102.190:37199), 226ef9aaf0bb4c959b799ae41508d53c (127.7.102.189:33589)
02:11:43.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.770886  7788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "226ef9aaf0bb4c959b799ae41508d53c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" is_pre_election: true
02:11:43.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.771612  7788 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 226ef9aaf0bb4c959b799ae41508d53c in term 0.
02:11:43.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.772990  7683 leader_election.cc:304] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226ef9aaf0bb4c959b799ae41508d53c, d1e8969ccd524f099e9ed8cf83cfd83d; no voters: 
02:11:43.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.774112  7708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226ef9aaf0bb4c959b799ae41508d53c" is_pre_election: true
02:11:43.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.774966  7708 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e8969ccd524f099e9ed8cf83cfd83d in term 0.
02:11:43.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.765779  7825 leader_election.cc:290] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11e6b48067d646c2a58d9d85239ca222 (127.7.102.190:37199), d1e8969ccd524f099e9ed8cf83cfd83d (127.7.102.188:43935)
02:11:43.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.777354  7825 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [term 0 FOLLOWER]: Leader pre-election won for term 1
02:11:43.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.779341  7825 raft_consensus.cc:493] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
02:11:43.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.779400  7763 leader_election.cc:304] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226ef9aaf0bb4c959b799ae41508d53c, d1e8969ccd524f099e9ed8cf83cfd83d; no voters: 
02:11:43.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.781286  7825 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [term 0 FOLLOWER]: Advancing to term 1
02:11:43.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.783361  7837 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [term 0 FOLLOWER]: Leader pre-election won for term 1
02:11:43.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.783715  7837 raft_consensus.cc:493] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
02:11:43.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.784021  7837 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [term 0 FOLLOWER]: Advancing to term 1
02:11:43.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.791812  7837 raft_consensus.cc:515] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } }
02:11:43.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.793968  7837 leader_election.cc:290] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [CANDIDATE]: Term 1 election: Requested vote from peers 11e6b48067d646c2a58d9d85239ca222 (127.7.102.190:37199), 226ef9aaf0bb4c959b799ae41508d53c (127.7.102.189:33589)
02:11:43.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.797969  7708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226ef9aaf0bb4c959b799ae41508d53c"
02:11:43.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.807035  7825 raft_consensus.cc:515] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } }
02:11:43.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.808545  7708 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d1e8969ccd524f099e9ed8cf83cfd83d in current term 1: Already voted for candidate 226ef9aaf0bb4c959b799ae41508d53c in this term.
02:11:43.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.810773  7788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "226ef9aaf0bb4c959b799ae41508d53c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d"
02:11:43.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.811554  7788 raft_consensus.cc:2393] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 226ef9aaf0bb4c959b799ae41508d53c in current term 1: Already voted for candidate d1e8969ccd524f099e9ed8cf83cfd83d in this term.
02:11:43.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.812997  7825 leader_election.cc:290] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [CANDIDATE]: Term 1 election: Requested vote from peers 11e6b48067d646c2a58d9d85239ca222 (127.7.102.190:37199), d1e8969ccd524f099e9ed8cf83cfd83d (127.7.102.188:43935)
02:11:43.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.847158  7658 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } attempt: 7
02:11:43.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.886917  7658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222: Bootstrap starting.
02:11:43.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.898852  7658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222: Neither blocks nor log segments found. Creating new log.
02:11:43.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.907742  7658 log.cc:826] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222: Log is configured to *not* fsync() on all Append() calls
02:11:43.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.921627  7658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222: No bootstrap required, opened a new log
02:11:43.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.958899  7658 raft_consensus.cc:359] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } }
02:11:43.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.960224  7658 raft_consensus.cc:385] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:11:43.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.960839  7658 raft_consensus.cc:740] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11e6b48067d646c2a58d9d85239ca222, State: Initialized, Role: FOLLOWER
02:11:43.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.962013  7658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } }
02:11:43.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.966526  7658 sys_catalog.cc:565] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222 [sys.catalog]: configured and running, proceeding with master startup.
02:11:43.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.967721  7633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "226ef9aaf0bb4c959b799ae41508d53c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e6b48067d646c2a58d9d85239ca222" is_pre_election: true
02:11:43.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.968600  7632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e6b48067d646c2a58d9d85239ca222" is_pre_election: true
02:11:43.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.971192  7631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e6b48067d646c2a58d9d85239ca222"
02:11:43.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.967092  7630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "226ef9aaf0bb4c959b799ae41508d53c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e6b48067d646c2a58d9d85239ca222"
02:11:43.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.973977  7683 leader_election.cc:304] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 226ef9aaf0bb4c959b799ae41508d53c; no voters: 11e6b48067d646c2a58d9d85239ca222, d1e8969ccd524f099e9ed8cf83cfd83d
02:11:43.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.975129  7825 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
02:11:43.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.977398  7763 leader_election.cc:304] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e8969ccd524f099e9ed8cf83cfd83d; no voters: 11e6b48067d646c2a58d9d85239ca222, 226ef9aaf0bb4c959b799ae41508d53c
02:11:43.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.978596  7837 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
02:11:44.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:43.997967  7633 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 226ef9aaf0bb4c959b799ae41508d53c in term 0.
02:11:44.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:44.005651  7854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } } }
02:11:44.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:44.015053  7854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222 [sys.catalog]: This master's current role is: FOLLOWER
02:11:44.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:44.034721  7612 tablet.cc:2406] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:11:44.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:44.059376  7865 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
02:11:44.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:44.077534  7590 master_runner.cc:264] No leader master found from master 11e6b48067d646c2a58d9d85239ca222
02:11:44.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:44.077858  7590 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
02:11:44.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:44.528124  7836 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
02:11:44.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:44.528515  7836 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
02:11:44.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:44.578223  7849 catalog_manager.cc:1593] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
02:11:44.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:44.578603  7849 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
02:11:45.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:45.068073  7865 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
02:11:45.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:45.069119  7865 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
02:11:45.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:45.533903  7836 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
02:11:45.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:45.534835  7836 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
02:11:45.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:45.583537  7849 catalog_manager.cc:1593] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
02:11:45.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:45.584376  7849 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
02:11:45.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.775704  7867 raft_consensus.cc:493] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:11:45.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.777033  7867 raft_consensus.cc:515] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } }
02:11:45.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.779955  7633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "226ef9aaf0bb4c959b799ae41508d53c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e6b48067d646c2a58d9d85239ca222" is_pre_election: true
02:11:45.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.780560  7633 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 226ef9aaf0bb4c959b799ae41508d53c in term 0.
02:11:45.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.781222  7788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "226ef9aaf0bb4c959b799ae41508d53c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" is_pre_election: true
02:11:45.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.781832  7788 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 226ef9aaf0bb4c959b799ae41508d53c in term 1.
02:11:45.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.783785  7683 leader_election.cc:304] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226ef9aaf0bb4c959b799ae41508d53c, d1e8969ccd524f099e9ed8cf83cfd83d; no voters: 
02:11:45.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.786120  7867 leader_election.cc:290] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 11e6b48067d646c2a58d9d85239ca222 (127.7.102.190:37199), d1e8969ccd524f099e9ed8cf83cfd83d (127.7.102.188:43935)
02:11:45.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.787144  7867 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [term 1 FOLLOWER]: Leader pre-election won for term 2
02:11:45.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.788062  7867 raft_consensus.cc:493] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
02:11:45.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.788868  7867 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [term 1 FOLLOWER]: Advancing to term 2
02:11:45.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.802755  7867 raft_consensus.cc:515] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } }
02:11:45.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.806850  7867 leader_election.cc:290] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [CANDIDATE]: Term 2 election: Requested vote from peers 11e6b48067d646c2a58d9d85239ca222 (127.7.102.190:37199), d1e8969ccd524f099e9ed8cf83cfd83d (127.7.102.188:43935)
02:11:45.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.806811  7633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "226ef9aaf0bb4c959b799ae41508d53c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11e6b48067d646c2a58d9d85239ca222"
02:11:45.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.807385  7633 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222 [term 0 FOLLOWER]: Advancing to term 2
02:11:45.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.809242  7788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "226ef9aaf0bb4c959b799ae41508d53c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d"
02:11:45.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.809800  7788 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [term 1 FOLLOWER]: Advancing to term 2
02:11:45.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.816645  7788 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 226ef9aaf0bb4c959b799ae41508d53c in term 2.
02:11:45.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.819056  7683 leader_election.cc:304] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226ef9aaf0bb4c959b799ae41508d53c, d1e8969ccd524f099e9ed8cf83cfd83d; no voters: 
02:11:45.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.821790  7867 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [term 2 FOLLOWER]: Leader election won for term 2
02:11:45.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.822527  7867 raft_consensus.cc:697] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [term 2 LEADER]: Becoming Leader. State: Replica: 226ef9aaf0bb4c959b799ae41508d53c, State: Running, Role: LEADER
02:11:45.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.823872  7867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } }
02:11:45.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.824437  7633 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 226ef9aaf0bb4c959b799ae41508d53c in term 2.
02:11:45.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.833292  7868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 226ef9aaf0bb4c959b799ae41508d53c. Latest consensus state: current_term: 2 leader_uuid: "226ef9aaf0bb4c959b799ae41508d53c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } } }
02:11:45.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.834100  7868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [sys.catalog]: This master's current role is: LEADER
02:11:45.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.839332  7870 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
02:11:45.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.843607  7870 catalog_manager.cc:1519] Initializing Kudu cluster ID...
02:11:45.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.857263  7788 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [term 2 FOLLOWER]: Refusing update from remote peer 226ef9aaf0bb4c959b799ae41508d53c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
02:11:45.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.857802  7633 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222 [term 2 FOLLOWER]: Refusing update from remote peer 226ef9aaf0bb4c959b799ae41508d53c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
02:11:45.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.859546  7868 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [LEADER]: Connected to new peer: Peer: permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:11:45.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.860387  7867 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:11:45.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.911855  7872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 226ef9aaf0bb4c959b799ae41508d53c. Latest consensus state: current_term: 2 leader_uuid: "226ef9aaf0bb4c959b799ae41508d53c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } } }
02:11:45.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.913399  7872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222 [sys.catalog]: This master's current role is: FOLLOWER
02:11:45.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.915529  7873 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 226ef9aaf0bb4c959b799ae41508d53c. Latest consensus state: current_term: 2 leader_uuid: "226ef9aaf0bb4c959b799ae41508d53c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } } }
02:11:45.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.918857  7873 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [sys.catalog]: This master's current role is: FOLLOWER
02:11:45.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.920010  7871 mvcc.cc:204] Tried to move back new op lower bound from 7305120587174035456 to 7305120587068035072. Current Snapshot: MvccSnapshot[applied={T|T < 7305120587174035456}]
02:11:45.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.926697  7870 catalog_manager.cc:1382] Generated new cluster ID: 1b912629fab8432da72a8c1734ce9fac
02:11:45.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.927019  7870 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
02:11:45.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.928161  7875 mvcc.cc:204] Tried to move back new op lower bound from 7305120587174035456 to 7305120587068035072. Current Snapshot: MvccSnapshot[applied={T|T < 7305120587174035456}]
02:11:45.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.930631  7874 mvcc.cc:204] Tried to move back new op lower bound from 7305120587174035456 to 7305120587068035072. Current Snapshot: MvccSnapshot[applied={T|T < 7305120587174035456}]
02:11:45.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.932993  7867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "226ef9aaf0bb4c959b799ae41508d53c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } } }
02:11:45.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.934026  7867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [sys.catalog]: This master's current role is: LEADER
02:11:45.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.935905  7868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "226ef9aaf0bb4c959b799ae41508d53c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } } }
02:11:45.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.936651  7868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c [sys.catalog]: This master's current role is: LEADER
02:11:45.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.940779  7872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "226ef9aaf0bb4c959b799ae41508d53c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } } }
02:11:45.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.941823  7872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222 [sys.catalog]: This master's current role is: FOLLOWER
02:11:45.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.947300  7873 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "226ef9aaf0bb4c959b799ae41508d53c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11e6b48067d646c2a58d9d85239ca222" member_type: VOTER last_known_addr { host: "127.7.102.190" port: 37199 } } peers { permanent_uuid: "226ef9aaf0bb4c959b799ae41508d53c" member_type: VOTER last_known_addr { host: "127.7.102.189" port: 33589 } } peers { permanent_uuid: "d1e8969ccd524f099e9ed8cf83cfd83d" member_type: VOTER last_known_addr { host: "127.7.102.188" port: 43935 } } }
02:11:45.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.948076  7873 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d [sys.catalog]: This master's current role is: FOLLOWER
02:11:45.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.974133  7870 catalog_manager.cc:1405] Generated new certificate authority record
02:11:45.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:45.976671  7870 catalog_manager.cc:1539] Loading token signing keys...
02:11:46.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:46.012158  7870 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 226ef9aaf0bb4c959b799ae41508d53c: Generated new TSK 0
02:11:46.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:46.014153  7870 catalog_manager.cc:1549] Initializing in-progress tserver states...
02:11:46.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:46.062999  7578 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:11:46.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:11:46.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal
02:11:46.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data
02:11:46.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:11:46.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:11:46.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:11:46.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:11:46.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:11:46.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:11:46.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/logs
02:11:46.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data/info.pb
02:11:46.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:11:46.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:11:46.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:11:46.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:11:46.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:11:46.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:11:46.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:11:46.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:11:46.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:11:46.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:11:46.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.129:0
02:11:46.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.129
02:11:46.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.129
02:11:46.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:11:46.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:11:46.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:11:46.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:11:46.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:11:46.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:11:46.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:46.076123  7865 catalog_manager.cc:1294] Loaded cluster ID: 1b912629fab8432da72a8c1734ce9fac
02:11:46.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:46.076438  7865 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222: loading cluster ID for follower catalog manager: success
02:11:46.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:46.084996  7865 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222: acquiring CA information for follower catalog manager: success
02:11:46.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:46.090557  7865 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 11e6b48067d646c2a58d9d85239ca222: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
02:11:46.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:46.469537  7881 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:11:46.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:46.470012  7881 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:11:46.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:46.470511  7881 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:11:46.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:46.507891  7881 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:11:46.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:46.508683  7881 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.102.129
02:11:46.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:46.547107  7881 tablet_server_runner.cc:78] Tablet server non-default flags:
02:11:46.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:11:46.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:11:46.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:11:46.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:11:46.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data
02:11:46.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal
02:11:46.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:11:46.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:11:46.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.129:0
02:11:46.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:11:46.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:11:46.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:11:46.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data/info.pb
02:11:46.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.129
02:11:46.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:11:46.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:11:46.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:11:46.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:11:46.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:11:46.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:11:46.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:11:46.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.129
02:11:46.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/logs
02:11:46.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:11:46.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:11:46.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:11:46.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:11:46.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:11:46.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:11:46.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:11:46.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on e7f111948823
02:11:46.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16850
02:11:46.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:11:46.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:46.548372  7881 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:11:46.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:46.549999  7881 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:11:46.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:46.563880  7888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:11:46.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:46.591619  7849 catalog_manager.cc:1294] Loaded cluster ID: 1b912629fab8432da72a8c1734ce9fac
02:11:46.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:46.592535  7849 catalog_manager.cc:1587] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d: loading cluster ID for follower catalog manager: success
02:11:46.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:46.601925  7849 catalog_manager.cc:1609] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d: acquiring CA information for follower catalog manager: success
02:11:46.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:46.613670  7849 catalog_manager.cc:1637] T 00000000000000000000000000000000 P d1e8969ccd524f099e9ed8cf83cfd83d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
02:11:48.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:46.565755  7889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:11:48.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:47.970003  7887 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 7881
02:11:48.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:48.191303  7881 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.627s	user 0.593s	sys 0.945s
02:11:48.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:48.191625  7881 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.627s	user 0.593s	sys 0.945s
02:11:48.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:48.195292  7890 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1624 milliseconds
02:11:48.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:48.196794  7891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:11:48.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.196918  7881 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:11:48.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.198294  7881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:11:48.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.200657  7881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:11:48.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.202040  7881 hybrid_clock.cc:648] HybridClock initialized: now 1783476708201989 us; error 47 us; skew 500 ppm
02:11:48.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.202958  7881 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:11:48.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.210739  7881 webserver.cc:534] Webserver started at http://127.7.102.129:37707/ using document root <none> and password file <none>
02:11:48.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.211684  7881 fs_manager.cc:362] Metadata directory not provided
02:11:48.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.211942  7881 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:11:48.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.212390  7881 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:11:48.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.217177  7881 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data/instance:
02:11:48.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cc9ab047ac31487896edb4faf2fb74b5"
02:11:48.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:48 on dist-test-slave-5s6b"
02:11:48.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.218484  7881 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal/instance:
02:11:48.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cc9ab047ac31487896edb4faf2fb74b5"
02:11:48.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:48 on dist-test-slave-5s6b"
02:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.226634  7881 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.002s
02:11:48.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.234191  7899 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:11:48.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.235553  7881 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.002s
02:11:48.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.235846  7881 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data,/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal
02:11:48.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cc9ab047ac31487896edb4faf2fb74b5"
02:11:48.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:48 on dist-test-slave-5s6b"
02:11:48.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.236147  7881 fs_report.cc:389] FS layout report
02:11:48.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:11:48.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal
02:11:48.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal
02:11:48.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data/data
02:11:48.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:11:48.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:11:48.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:11:48.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:11:48.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:11:48.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:11:48.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:11:48.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:11:48.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:11:48.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:11:48.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:11:48.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.293143  7881 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:11:48.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.294662  7881 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:11:48.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.295326  7881 kserver.cc:163] Server-wide thread pool size limit: 3276
02:11:48.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.298717  7881 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:11:48.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.303076  7881 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
02:11:48.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.303268  7881 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
02:11:48.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.303454  7881 ts_tablet_manager.cc:616] Registered 0 tablets
02:11:48.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.303646  7881 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
02:11:48.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.468539  7881 rpc_server.cc:307] RPC server started. Bound to: 127.7.102.129:42373
02:11:48.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.468663  8011 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.102.129:42373 every 8 connection(s)
02:11:48.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.471402  7881 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data/info.pb
02:11:48.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.475322  7578 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu as pid 7881
02:11:48.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.475966  7578 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal/instance
02:11:48.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.493214  7578 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:11:48.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:11:48.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal
02:11:48.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data
02:11:48.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:11:48.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:11:48.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:11:48.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:11:48.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:11:48.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:11:48.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/logs
02:11:48.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data/info.pb
02:11:48.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:11:48.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:11:48.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:11:48.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:11:48.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:11:48.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:11:48.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:11:48.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:11:48.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:11:48.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:11:48.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.130:0
02:11:48.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.130
02:11:48.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.130
02:11:48.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:11:48.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:11:48.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:11:48.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:11:48.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:11:48.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:11:48.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.517809  8012 heartbeater.cc:344] Connected to a master server at 127.7.102.188:43935
02:11:48.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.518363  8012 heartbeater.cc:461] Registering TS with master...
02:11:48.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.519932  8012 heartbeater.cc:507] Master 127.7.102.188:43935 requested a full tablet report, sending...
02:11:48.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.522258  8013 heartbeater.cc:344] Connected to a master server at 127.7.102.189:33589
02:11:48.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.522696  8013 heartbeater.cc:461] Registering TS with master...
02:11:48.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.523801  8013 heartbeater.cc:507] Master 127.7.102.189:33589 requested a full tablet report, sending...
02:11:48.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.523957  7778 ts_manager.cc:194] Registered new tserver with Master: cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373)
02:11:48.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.526865  8014 heartbeater.cc:344] Connected to a master server at 127.7.102.190:37199
02:11:48.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.527244  8014 heartbeater.cc:461] Registering TS with master...
02:11:48.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.527281  7698 ts_manager.cc:194] Registered new tserver with Master: cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373)
02:11:48.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.528348  8014 heartbeater.cc:507] Master 127.7.102.190:37199 requested a full tablet report, sending...
02:11:48.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.529448  7698 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.102.129:58553
02:11:48.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.532512  7622 ts_manager.cc:194] Registered new tserver with Master: cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373)
02:11:48.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:48.841001  8022 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:11:48.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:48.841554  8022 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:11:48.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:48.842123  8022 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:11:48.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:48.874354  8022 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:11:48.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:48.875375  8022 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.102.130
02:11:48.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.912384  8022 tablet_server_runner.cc:78] Tablet server non-default flags:
02:11:48.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:11:48.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:11:48.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:11:48.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:11:48.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data
02:11:48.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal
02:11:48.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:11:48.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:11:48.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.130:0
02:11:48.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:11:48.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:11:48.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:11:48.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data/info.pb
02:11:48.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.130
02:11:48.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:11:48.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:11:48.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:11:48.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:11:48.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:11:48.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:11:48.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:11:48.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.130
02:11:48.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/logs
02:11:48.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:11:48.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:11:48.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:11:48.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:11:48.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:11:48.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:11:48.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:11:48.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on e7f111948823
02:11:48.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16850
02:11:48.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:11:48.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.914198  8022 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:11:48.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:48.916121  8022 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:11:48.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:48.928800  8028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:11:49.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:49.533818  8013 heartbeater.cc:499] Master 127.7.102.189:33589 was elected leader, sending a full tablet report...
02:11:50.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:50.332602  8027 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 8022
02:11:50.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:48.931474  8029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:11:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:50.341574  8027 kernel_stack_watchdog.cc:198] Thread 8022 stuck at /home/jenkins-slave/workspace/build_and_test@2/src/kudu/util/thread.cc:642 for 399ms:
02:11:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
02:11:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
02:11:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
02:11:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
02:11:50.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:50.342070  8022 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.411s	user 0.513s	sys 0.836s
02:11:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:50.342473  8022 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.412s	user 0.514s	sys 0.836s
02:11:50.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:50.344254  8030 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1411 milliseconds
02:11:50.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.344918  8022 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:11:50.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:50.345006  8031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:11:50.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.348789  8022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:11:50.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.351161  8022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:11:50.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.352553  8022 hybrid_clock.cc:648] HybridClock initialized: now 1783476710352514 us; error 35 us; skew 500 ppm
02:11:50.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.353350  8022 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:11:50.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.359956  8022 webserver.cc:534] Webserver started at http://127.7.102.130:41213/ using document root <none> and password file <none>
02:11:50.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.360977  8022 fs_manager.cc:362] Metadata directory not provided
02:11:50.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.361233  8022 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:11:50.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.361786  8022 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:11:50.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.366245  8022 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data/instance:
02:11:50.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "372cdb4006a84d17800acd046e7b5a41"
02:11:50.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:50 on dist-test-slave-5s6b"
02:11:50.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.367637  8022 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal/instance:
02:11:50.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "372cdb4006a84d17800acd046e7b5a41"
02:11:50.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:50 on dist-test-slave-5s6b"
02:11:50.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.375840  8022 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.002s	sys 0.006s
02:11:50.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.381772  8039 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:11:50.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.382961  8022 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
02:11:50.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.383334  8022 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data,/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal
02:11:50.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "372cdb4006a84d17800acd046e7b5a41"
02:11:50.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:50 on dist-test-slave-5s6b"
02:11:50.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.383802  8022 fs_report.cc:389] FS layout report
02:11:50.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:11:50.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal
02:11:50.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal
02:11:50.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data/data
02:11:50.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:11:50.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:11:50.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:11:50.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:11:50.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:11:50.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:11:50.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:11:50.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:11:50.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:11:50.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:11:50.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:11:50.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.441581  8022 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:11:50.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.443549  8022 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:11:50.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.444159  8022 kserver.cc:163] Server-wide thread pool size limit: 3276
02:11:50.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.447781  8022 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:11:50.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.452533  8022 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
02:11:50.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.452745  8022 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
02:11:50.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.453011  8022 ts_tablet_manager.cc:616] Registered 0 tablets
02:11:50.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.453235  8022 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
02:11:50.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.628456  8022 rpc_server.cc:307] RPC server started. Bound to: 127.7.102.130:41431
02:11:50.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.628571  8151 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.102.130:41431 every 8 connection(s)
02:11:50.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.631165  8022 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data/info.pb
02:11:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.637485  7578 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu as pid 8022
02:11:50.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.637959  7578 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal/instance
02:11:50.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.675732  7578 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:11:50.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:11:50.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:11:50.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data
02:11:50.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:11:50.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:11:50.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:11:50.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:11:50.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:11:50.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:11:50.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/logs
02:11:50.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data/info.pb
02:11:50.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:11:50.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:11:50.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:11:50.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:11:50.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:11:50.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:11:50.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:11:50.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:11:50.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:11:50.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:11:50.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.131:0
02:11:50.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.131
02:11:50.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.131
02:11:50.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:11:50.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:11:50.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:11:50.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:11:50.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:11:50.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:11:50.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.689932  8153 heartbeater.cc:344] Connected to a master server at 127.7.102.189:33589
02:11:50.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.690615  8153 heartbeater.cc:461] Registering TS with master...
02:11:50.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.692325  8153 heartbeater.cc:507] Master 127.7.102.189:33589 requested a full tablet report, sending...
02:11:50.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.695909  7698 ts_manager.cc:194] Registered new tserver with Master: 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:11:50.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.698706  7698 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.102.130:40807
02:11:50.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.700819  8154 heartbeater.cc:344] Connected to a master server at 127.7.102.190:37199
02:11:50.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.701030  8152 heartbeater.cc:344] Connected to a master server at 127.7.102.188:43935
02:11:50.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.701249  8154 heartbeater.cc:461] Registering TS with master...
02:11:50.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.701413  8152 heartbeater.cc:461] Registering TS with master...
02:11:50.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.702440  8152 heartbeater.cc:507] Master 127.7.102.188:43935 requested a full tablet report, sending...
02:11:50.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.702471  8154 heartbeater.cc:507] Master 127.7.102.190:37199 requested a full tablet report, sending...
02:11:50.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.705680  7778 ts_manager.cc:194] Registered new tserver with Master: 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:11:50.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:50.706398  7622 ts_manager.cc:194] Registered new tserver with Master: 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:11:51.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:51.031195  8162 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:11:51.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:51.031683  8162 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:11:51.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:51.032091  8162 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:11:51.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:51.065821  8162 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:11:51.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:51.066728  8162 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.102.131
02:11:51.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:51.103977  8162 tablet_server_runner.cc:78] Tablet server non-default flags:
02:11:51.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:11:51.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:11:51.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:11:51.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:11:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data
02:11:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:11:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:11:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:11:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.131:0
02:11:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:11:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:11:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:11:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data/info.pb
02:11:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.131
02:11:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
02:11:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:11:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:11:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:11:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:11:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:11:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:11:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.131
02:11:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/logs
02:11:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:11:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:11:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:11:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:11:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:11:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:11:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:11:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on e7f111948823
02:11:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16850
02:11:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:11:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:51.105262  8162 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:11:51.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:51.107163  8162 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:11:51.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:51.122605  8168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:11:51.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:51.704635  8153 heartbeater.cc:499] Master 127.7.102.189:33589 was elected leader, sending a full tablet report...
02:11:52.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:52.524080  8167 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 8162
02:11:52.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:52.938117  8162 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.815s	user 0.626s	sys 1.157s
02:11:52.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:51.125751  8169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:11:52.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:52.939168  8170 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1814 milliseconds
02:11:52.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:52.938583  8162 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.815s	user 0.626s	sys 1.157s
02:11:52.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:52.940858  8171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:11:52.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:52.941125  8162 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:11:52.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:52.943810  8162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:11:52.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:52.945936  8162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:11:52.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:52.947368  8162 hybrid_clock.cc:648] HybridClock initialized: now 1783476712947315 us; error 35 us; skew 500 ppm
02:11:52.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:52.948477  8162 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:11:52.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:52.955570  8162 webserver.cc:534] Webserver started at http://127.7.102.131:38903/ using document root <none> and password file <none>
02:11:52.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:52.956621  8162 fs_manager.cc:362] Metadata directory not provided
02:11:52.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:52.956899  8162 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:11:52.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:52.957472  8162 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
02:11:52.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:52.962759  8162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data/instance:
02:11:52.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c"
02:11:52.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:52 on dist-test-slave-5s6b"
02:11:52.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:52.964116  8162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal/instance:
02:11:52.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c"
02:11:52.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:52 on dist-test-slave-5s6b"
02:11:52.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:52.972077  8162 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.004s	sys 0.004s
02:11:52.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:52.978394  8179 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:11:52.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:52.979862  8162 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.001s
02:11:52.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:52.980222  8162 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data,/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:11:52.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c"
02:11:52.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:52 on dist-test-slave-5s6b"
02:11:52.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:52.980531  8162 fs_report.cc:389] FS layout report
02:11:52.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:11:52.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:11:52.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:11:52.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data/data
02:11:52.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:11:52.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:11:52.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:11:52.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:11:52.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:11:52.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:11:52.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:11:52.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:11:52.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:11:52.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:11:52.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:11:53.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.033553  8162 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:11:53.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.035172  8162 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:11:53.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.035600  8162 kserver.cc:163] Server-wide thread pool size limit: 3276
02:11:53.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.038242  8162 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:11:53.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.042479  8162 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
02:11:53.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.042673  8162 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
02:11:53.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.042858  8162 ts_tablet_manager.cc:616] Registered 0 tablets
02:11:53.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.042990  8162 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
02:11:53.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.228765  8162 rpc_server.cc:307] RPC server started. Bound to: 127.7.102.131:37689
02:11:53.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.228884  8291 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.102.131:37689 every 8 connection(s)
02:11:53.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.231490  8162 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data/info.pb
02:11:53.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.240307  7578 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu as pid 8162
02:11:53.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.240839  7578 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal/instance
02:11:53.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.271529  8293 heartbeater.cc:344] Connected to a master server at 127.7.102.189:33589
02:11:53.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.272073  8293 heartbeater.cc:461] Registering TS with master...
02:11:53.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.272590  8294 heartbeater.cc:344] Connected to a master server at 127.7.102.190:37199
02:11:53.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.272974  8294 heartbeater.cc:461] Registering TS with master...
02:11:53.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.273617  8293 heartbeater.cc:507] Master 127.7.102.189:33589 requested a full tablet report, sending...
02:11:53.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.273653  8292 heartbeater.cc:344] Connected to a master server at 127.7.102.188:43935
02:11:53.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.274101  8292 heartbeater.cc:461] Registering TS with master...
02:11:53.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.274039  8294 heartbeater.cc:507] Master 127.7.102.190:37199 requested a full tablet report, sending...
02:11:53.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.275043  8292 heartbeater.cc:507] Master 127.7.102.188:43935 requested a full tablet report, sending...
02:11:53.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.276320  7698 ts_manager.cc:194] Registered new tserver with Master: fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689)
02:11:53.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.277196  7622 ts_manager.cc:194] Registered new tserver with Master: fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689)
02:11:53.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.277384  7778 ts_manager.cc:194] Registered new tserver with Master: fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689)
02:11:53.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.278175  7698 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.102.131:54795
02:11:53.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:53.294552  7578 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
02:11:53.307 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
02:11:53.309 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

02:11:53.318 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.7.102.190"
      port: 37199
    }
    bound_http_address {
      host: "127.7.102.190"
      port: 34679
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.7.102.189"
      port: 33589
    }
    bound_http_address {
      host: "127.7.102.189"
      port: 34283
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.7.102.188"
      port: 43935
    }
    bound_http_address {
      host: "127.7.102.188"
      port: 46159
    }
  }
}

02:11:53.322 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

02:11:53.324 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.7.102.129"
      port: 42373
    }
    bound_http_address {
      host: "127.7.102.129"
      port: 37707
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.7.102.130"
      port: 41431
    }
    bound_http_address {
      host: "127.7.102.130"
      port: 41213
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.7.102.131"
      port: 37689
    }
    bound_http_address {
      host: "127.7.102.131"
      port: 38903
    }
  }
}

02:11:53.324 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
02:11:53.582 [DEBUG - main] (CompositeStrategy.java:59) Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2677)
	at org.apache.kudu.client.internals.ModernStrategy.<init>(ModernStrategy.java:51)
	at org.apache.kudu.client.internals.CompositeStrategy.<init>(CompositeStrategy.java:52)
	at org.apache.kudu.client.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:41)
	at org.apache.kudu.client.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:43)
	at org.apache.kudu.client.SecurityContext.setupSubject(SecurityContext.java:155)
	at org.apache.kudu.client.SecurityContext.<init>(SecurityContext.java:138)
	at org.apache.kudu.client.AsyncKuduClient.<init>(AsyncKuduClient.java:431)
	at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.build(AsyncKuduClient.java:3109)
	at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:161)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttempt(RetryRule.java:225)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:246)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
02:11:53.631 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
02:11:53.782 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.7.102.188:43935(127.7.102.188:43935)] connecting to peer
02:11:53.842 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.7.102.190:37199(127.7.102.190:37199)] connecting to peer
02:11:53.843 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.7.102.189:33589(127.7.102.189:33589)] connecting to peer
02:11:53.861 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.102.188:43935(127.7.102.188:43935)] Successfully connected to peer
02:11:53.864 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.102.190:37199(127.7.102.190:37199)] Successfully connected to peer
02:11:53.874 [DEBUG - kudu-nio-2] (Connection.java:757) [peer master-127.7.102.189:33589(127.7.102.189:33589)] Successfully connected to peer
02:11:53.975 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.7.102.190
02:11:53.975 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.7.102.188
02:11:53.975 [DEBUG - kudu-nio-2] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.7.102.189
02:11:54.203 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xbec96364, L:/127.0.0.1:47364 - R:/127.7.102.188:43935] using SASL/PLAIN
02:11:54.203 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x5c0ba39f, L:/127.0.0.1:59360 - R:/127.7.102.190:37199] using SASL/PLAIN
02:11:54.203 [DEBUG - kudu-nio-2] (Negotiator.java:919) Authenticated connection [id: 0x62fc239a, L:/127.0.0.1:46612 - R:/127.7.102.189:33589] using SASL/PLAIN
02:11:54.240 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
02:11:54.241 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=502), Traces: [0ms] refreshing cache from master, [28ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.102.188:43935, [100ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.102.190:37199, [101ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.102.189:33589, [476ms] Sub RPC ConnectToMaster: received response from server master-127.7.102.190:37199: OK, [476ms] Sub RPC ConnectToMaster: received response from server master-127.7.102.188:43935: OK, [477ms] Sub RPC ConnectToMaster: received response from server master-127.7.102.189:33589: OK, deferred=null) after lookup
02:11:54.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.282621  8293 heartbeater.cc:499] Master 127.7.102.189:33589 was elected leader, sending a full tablet report...
02:11:54.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.290156  7698 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46612:
02:11:54.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726"
02:11:54.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
02:11:54.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:11:54.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
02:11:54.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
02:11:54.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
02:11:54.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:11:54.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:11:54.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:11:54.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:11:54.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:11:54.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:11:54.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
02:11:54.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
02:11:54.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:11:54.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:11:54.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:11:54.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:11:54.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:11:54.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:11:54.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:11:54.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
02:11:54.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
02:11:54.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
02:11:54.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:11:54.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
02:11:54.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
02:11:54.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
02:11:54.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
02:11:54.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
02:11:54.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
02:11:54.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
02:11:54.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
02:11:54.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:54.294872  7698 catalog_manager.cc:7076] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
02:11:54.348 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:894) Opened table 7a441225a07d422b80ad85a1f3f0953d
02:11:54.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.376061  8227 tablet_service.cc:1511] Processing CreateTablet for tablet 064f4af2277f4b52af05ed12eaafba04 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726 [id=7a441225a07d422b80ad85a1f3f0953d]), partition=RANGE (key) PARTITION UNBOUNDED
02:11:54.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.377835  8227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 064f4af2277f4b52af05ed12eaafba04. 1 dirs total, 0 dirs full, 0 dirs failed
02:11:54.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.410784  7947 tablet_service.cc:1511] Processing CreateTablet for tablet 064f4af2277f4b52af05ed12eaafba04 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726 [id=7a441225a07d422b80ad85a1f3f0953d]), partition=RANGE (key) PARTITION UNBOUNDED
02:11:54.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.414719  7947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 064f4af2277f4b52af05ed12eaafba04. 1 dirs total, 0 dirs full, 0 dirs failed
02:11:54.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.417310  8087 tablet_service.cc:1511] Processing CreateTablet for tablet 064f4af2277f4b52af05ed12eaafba04 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726 [id=7a441225a07d422b80ad85a1f3f0953d]), partition=RANGE (key) PARTITION UNBOUNDED
02:11:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.419152  8087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 064f4af2277f4b52af05ed12eaafba04. 1 dirs total, 0 dirs full, 0 dirs failed
02:11:54.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.426100  8326 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Bootstrap starting.
02:11:54.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.433996  8326 tablet_bootstrap.cc:654] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Neither blocks nor log segments found. Creating new log.
02:11:54.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.436995  8326 log.cc:826] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Log is configured to *not* fsync() on all Append() calls
02:11:54.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.463299  8326 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: No bootstrap required, opened a new log
02:11:54.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.464179  8326 ts_tablet_manager.cc:1403] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Time spent bootstrapping tablet: real 0.040s	user 0.003s	sys 0.030s
02:11:54.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.483815  8328 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Bootstrap starting.
02:11:54.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.492862  8328 tablet_bootstrap.cc:654] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Neither blocks nor log segments found. Creating new log.
02:11:54.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.500741  8329 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Bootstrap starting.
02:11:54.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.507822  8328 log.cc:826] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Log is configured to *not* fsync() on all Append() calls
02:11:54.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.510912  8326 raft_consensus.cc:359] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:11:54.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.511828  8326 raft_consensus.cc:385] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:11:54.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.512133  8326 raft_consensus.cc:740] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe0a2e8ea6da4fa8b844e7d99e3cae0c, State: Initialized, Role: FOLLOWER
02:11:54.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.513175  8326 consensus_queue.cc:260] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:11:54.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.515340  8329 tablet_bootstrap.cc:654] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Neither blocks nor log segments found. Creating new log.
02:11:54.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.519488  8329 log.cc:826] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Log is configured to *not* fsync() on all Append() calls
02:11:54.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.522755  8326 ts_tablet_manager.cc:1434] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Time spent starting tablet: real 0.058s	user 0.026s	sys 0.013s
02:11:54.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.534071  8328 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: No bootstrap required, opened a new log
02:11:54.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.535135  8328 ts_tablet_manager.cc:1403] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Time spent bootstrapping tablet: real 0.052s	user 0.016s	sys 0.012s
02:11:54.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:54.538290  8148 debug-util.cc:398] Leaking SignalData structure 0x7b08000ad060 after lost signal to thread 8023
02:11:54.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.541182  8329 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: No bootstrap required, opened a new log
02:11:54.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.541996  8329 ts_tablet_manager.cc:1403] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Time spent bootstrapping tablet: real 0.042s	user 0.007s	sys 0.016s
02:11:54.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.568683  8328 raft_consensus.cc:359] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:11:54.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.570039  8328 raft_consensus.cc:385] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:11:54.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.570664  8328 raft_consensus.cc:740] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9ab047ac31487896edb4faf2fb74b5, State: Initialized, Role: FOLLOWER
02:11:54.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.572371  8328 consensus_queue.cc:260] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:11:54.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.574501  8329 raft_consensus.cc:359] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:11:54.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.575882  8329 raft_consensus.cc:385] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
02:11:54.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.576426  8329 raft_consensus.cc:740] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372cdb4006a84d17800acd046e7b5a41, State: Initialized, Role: FOLLOWER
02:11:54.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.577783  8329 consensus_queue.cc:260] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:11:54.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.588151  8328 ts_tablet_manager.cc:1434] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Time spent starting tablet: real 0.052s	user 0.039s	sys 0.009s
02:11:54.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.588646  8329 ts_tablet_manager.cc:1434] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Time spent starting tablet: real 0.046s	user 0.035s	sys 0.003s
02:11:54.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.667207  8334 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:11:54.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.667797  8334 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:11:54.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.671770  8334 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689), cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373)
02:11:54.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.695560  7967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "372cdb4006a84d17800acd046e7b5a41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9ab047ac31487896edb4faf2fb74b5" is_pre_election: true
02:11:54.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.696589  7967 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372cdb4006a84d17800acd046e7b5a41 in term 0.
02:11:54.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.697095  8247 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "372cdb4006a84d17800acd046e7b5a41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" is_pre_election: true
02:11:54.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.698108  8247 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372cdb4006a84d17800acd046e7b5a41 in term 0.
02:11:54.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.698268  8040 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372cdb4006a84d17800acd046e7b5a41, cc9ab047ac31487896edb4faf2fb74b5; no voters: 
02:11:54.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.699352  8334 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 0 FOLLOWER]: Leader pre-election won for term 1
02:11:54.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.699716  8334 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
02:11:54.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.700062  8334 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 0 FOLLOWER]: Advancing to term 1
02:11:54.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:54.702723  8156 tablet.cc:2406] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:11:54.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.709455  8334 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:11:54.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.713120  8247 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "372cdb4006a84d17800acd046e7b5a41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c"
02:11:54.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.713227  7967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "372cdb4006a84d17800acd046e7b5a41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9ab047ac31487896edb4faf2fb74b5"
02:11:54.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.713796  7967 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 0 FOLLOWER]: Advancing to term 1
02:11:54.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.714001  8247 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 0 FOLLOWER]: Advancing to term 1
02:11:54.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.720983  8247 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372cdb4006a84d17800acd046e7b5a41 in term 1.
02:11:54.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.722153  7967 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372cdb4006a84d17800acd046e7b5a41 in term 1.
02:11:54.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.722733  8041 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372cdb4006a84d17800acd046e7b5a41, fe0a2e8ea6da4fa8b844e7d99e3cae0c; no voters: 
02:11:54.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.722828  8334 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 1 election: Requested vote from peers fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689), cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373)
02:11:54.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.724963  8334 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 1 FOLLOWER]: Leader election won for term 1
02:11:54.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.725626  8334 raft_consensus.cc:697] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 1 LEADER]: Becoming Leader. State: Replica: 372cdb4006a84d17800acd046e7b5a41, State: Running, Role: LEADER
02:11:54.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.727276  8334 consensus_queue.cc:237] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:11:54.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:54.748054  8017 tablet.cc:2406] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:11:54.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:54.748751  8295 tablet.cc:2406] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
02:11:54.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:54.748180  7697 catalog_manager.cc:5696] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130). New cstate: current_term: 1 leader_uuid: "372cdb4006a84d17800acd046e7b5a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } health_report { overall_health: HEALTHY } } }
02:11:54.911 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:894) Opened table 7a441225a07d422b80ad85a1f3f0953d
02:11:54.940 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:11:54.949 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299991, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)[L],cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:11:54.951 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=28), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [15ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=null) after lookup
02:11:54.963 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:11:54.967 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] Successfully connected to peer
02:11:55.098 [DEBUG - kudu-nio-3] (Negotiator.java:919) Authenticated connection [id: 0xc4257e6f, L:/127.0.0.1:43534 - R:/127.7.102.130:41431] using TOKEN/null
02:11:55.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:55.122407  7967 raft_consensus.cc:1275] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 1 FOLLOWER]: Refusing update from remote peer 372cdb4006a84d17800acd046e7b5a41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:11:55.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:55.124964  8337 consensus_queue.cc:1048] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:11:55.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:55.134851  8247 raft_consensus.cc:1275] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 1 FOLLOWER]: Refusing update from remote peer 372cdb4006a84d17800acd046e7b5a41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
02:11:55.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:55.136799  8334 consensus_queue.cc:1048] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
02:11:55.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:55.198449  8341 mvcc.cc:204] Tried to move back new op lower bound from 7305120625120382976 to 7305120623537569792. Current Snapshot: MvccSnapshot[applied={T|T < 7305120625120382976}]
02:11:55.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:55.242564  8344 mvcc.cc:204] Tried to move back new op lower bound from 7305120625120382976 to 7305120623537569792. Current Snapshot: MvccSnapshot[applied={T|T < 7305120625120382976}]
02:11:55.414 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726, tablet=064f4af2277f4b52af05ed12eaafba04, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:11:55.417 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.7.102.130:41431
02:11:55.419 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:11:55.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:55.419580  7578 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu with pid 8022
02:11:55.454 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:11:55.456 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] handling channelInactive
02:11:55.457 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state READY due to: connection closed
02:11:55.457 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431) for tablet 064f4af2277f4b52af05ed12eaafba04: connection closed
02:11:55.464 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:11:55.464 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)[L],cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:11:55.464 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=9), Traces: [0ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [2ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [9ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1325513527(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@736dcc4b -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b569afa -> wakeup thread Time-limited test)) after lookup
02:11:55.465 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:11:55.468 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:11:55.468 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state CONNECTING due to: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:11:55.468 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:11:55.484 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:11:55.485 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)[L],cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:11:55.485 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=30), Traces: [0ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [2ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [9ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [10ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [13ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [13ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [29ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1325513527(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@736dcc4b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2119317241 after retry RPC@1707330066), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b569afa -> wakeup thread Time-limited test -> (continuation of Deferred@2119317241 after retry RPC@1707330066))) after lookup
02:11:55.486 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:11:55.488 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:11:55.488 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state CONNECTING due to: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:11:55.488 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:11:55.524 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:11:55.525 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)[L],cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:11:55.525 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=70), Traces: [0ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [2ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [9ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [10ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [13ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [13ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [29ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [30ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [33ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [33ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [69ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1325513527(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@736dcc4b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2119317241 after retry RPC@1707330066) -> (continuation of Deferred@2023260211 after retry RPC@1227926231), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b569afa -> wakeup thread Time-limited test -> (continuation of Deferred@2119317241 after retry RPC@1707330066) -> (continuation of Deferred@2023260211 after retry RPC@1227926231))) after lookup
02:11:55.526 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:11:55.528 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:11:55.528 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state CONNECTING due to: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:11:55.528 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:11:55.545 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:11:55.545 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)[L],cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:11:55.545 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=90), Traces: [0ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [2ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [9ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [10ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [13ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [13ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [29ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [30ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [33ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [33ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [69ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [71ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [73ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [74ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [86ms] refreshing cache from master, [86ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [89ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1325513527(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@736dcc4b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2119317241 after retry RPC@1707330066) -> (continuation of Deferred@2023260211 after retry RPC@1227926231) -> (continuation of Deferred@104736564 after retry RPC@899151714), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b569afa -> wakeup thread Time-limited test -> (continuation of Deferred@2119317241 after retry RPC@1707330066) -> (continuation of Deferred@2023260211 after retry RPC@1227926231) -> (continuation of Deferred@104736564 after retry RPC@899151714))) after lookup
02:11:55.546 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:11:55.548 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:11:55.548 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state CONNECTING due to: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:11:55.548 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:11:55.645 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:11:55.645 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)[L],cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:11:55.646 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=190), Traces: [0ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [2ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [9ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [10ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [13ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [13ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [29ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [30ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [33ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [33ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [69ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [71ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [73ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [74ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [86ms] refreshing cache from master, [86ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [89ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [91ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [93ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [93ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [185ms] refreshing cache from master, [185ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [189ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1325513527(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@736dcc4b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2119317241 after retry RPC@1707330066) -> (continuation of Deferred@2023260211 after retry RPC@1227926231) -> (continuation of Deferred@104736564 after retry RPC@899151714) -> (continuation of Deferred@1477152828 after retry RPC@929665425), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b569afa -> wakeup thread Time-limited test -> (continuation of Deferred@2119317241 after retry RPC@1707330066) -> (continuation of Deferred@2023260211 after retry RPC@1227926231) -> (continuation of Deferred@104736564 after retry RPC@899151714) -> (continuation of Deferred@1477152828 after retry RPC@929665425))) after lookup
02:11:55.647 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:11:55.648 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:11:55.648 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state CONNECTING due to: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:11:55.648 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:11:56.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.830741  8361 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372cdb4006a84d17800acd046e7b5a41)
02:11:56.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.831338  8361 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:11:56.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.831456  8362 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372cdb4006a84d17800acd046e7b5a41)
02:11:56.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.832029  8362 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:11:56.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.834306  8361 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373), 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:11:56.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:56.850241  8180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.102.130:41431: connect: Connection refused (error 111)
02:11:56.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.850697  8362 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689), 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:11:56.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:56.853940  7900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.102.130:41431: connect: Connection refused (error 111)
02:11:56.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:56.862933  7900 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431): Network error: Client connection negotiation failed: client connection to 127.7.102.130:41431: connect: Connection refused (error 111)
02:11:56.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.866782  7967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cc9ab047ac31487896edb4faf2fb74b5" is_pre_election: true
02:11:56.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.867560  7967 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe0a2e8ea6da4fa8b844e7d99e3cae0c in term 1.
02:11:56.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:56.868438  8180 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431): Network error: Client connection negotiation failed: client connection to 127.7.102.130:41431: connect: Connection refused (error 111)
02:11:56.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.869479  8180 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cc9ab047ac31487896edb4faf2fb74b5, fe0a2e8ea6da4fa8b844e7d99e3cae0c; no voters: 372cdb4006a84d17800acd046e7b5a41
02:11:56.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.870529  8361 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 1 FOLLOWER]: Leader pre-election won for term 2
02:11:56.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.870774  8361 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 372cdb4006a84d17800acd046e7b5a41)
02:11:56.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.871090  8361 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 1 FOLLOWER]: Advancing to term 2
02:11:56.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.871981  8247 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "cc9ab047ac31487896edb4faf2fb74b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" is_pre_election: true
02:11:56.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.878150  8361 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:11:56.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.879384  8247 raft_consensus.cc:2393] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cc9ab047ac31487896edb4faf2fb74b5 in current term 2: Already voted for candidate fe0a2e8ea6da4fa8b844e7d99e3cae0c in this term.
02:11:56.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.880257  8361 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 2 election: Requested vote from peers cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373), 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:11:56.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.881094  7967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cc9ab047ac31487896edb4faf2fb74b5"
02:11:56.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.881668  7967 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 1 FOLLOWER]: Advancing to term 2
02:11:56.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.884001  7901 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc9ab047ac31487896edb4faf2fb74b5; no voters: 372cdb4006a84d17800acd046e7b5a41, fe0a2e8ea6da4fa8b844e7d99e3cae0c
02:11:56.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:56.886224  8180 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431): Network error: Client connection negotiation failed: client connection to 127.7.102.130:41431: connect: Connection refused (error 111)
02:11:56.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.888355  7967 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe0a2e8ea6da4fa8b844e7d99e3cae0c in term 2.
02:11:56.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.888747  8362 raft_consensus.cc:2749] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
02:11:56.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.889484  8180 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cc9ab047ac31487896edb4faf2fb74b5, fe0a2e8ea6da4fa8b844e7d99e3cae0c; no voters: 372cdb4006a84d17800acd046e7b5a41
02:11:56.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.890103  8361 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 2 FOLLOWER]: Leader election won for term 2
02:11:56.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.892359  8361 raft_consensus.cc:697] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 2 LEADER]: Becoming Leader. State: Replica: fe0a2e8ea6da4fa8b844e7d99e3cae0c, State: Running, Role: LEADER
02:11:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.893325  8361 consensus_queue.cc:237] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:11:56.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:56.902295  7697 catalog_manager.cc:5696] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c reported cstate change: term changed from 1 to 2, leader changed from 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130) to fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131). New cstate: current_term: 2 leader_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } health_report { overall_health: UNKNOWN } } }
02:11:57.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:57.327544  7967 raft_consensus.cc:1275] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 2 FOLLOWER]: Refusing update from remote peer fe0a2e8ea6da4fa8b844e7d99e3cae0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
02:11:57.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:57.328930  8361 consensus_queue.cc:1048] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
02:11:57.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:57.349287  8180 consensus_peers.cc:597] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c -> Peer 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431): Couldn't send request to peer 372cdb4006a84d17800acd046e7b5a41. Status: Network error: Client connection negotiation failed: client connection to 127.7.102.130:41431: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:11:57.425 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:11:57.426 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:11:57.426 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=1971), Traces: [0ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [2ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: connection closed, [2ms] delaying RPC due to: Network error: connection closed, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [9ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [10ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [13ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [13ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [29ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [30ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [33ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [33ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [69ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [71ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [73ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [74ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [86ms] refreshing cache from master, [86ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [89ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [91ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [93ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [93ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [185ms] refreshing cache from master, [185ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [189ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [191ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [193ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [193ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [1965ms] refreshing cache from master, [1965ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [1970ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1325513527(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@736dcc4b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2119317241 after retry RPC@1707330066) -> (continuation of Deferred@2023260211 after retry RPC@1227926231) -> (continuation of Deferred@104736564 after retry RPC@899151714) -> (continuation of Deferred@1477152828 after retry RPC@929665425) -> (continuation of Deferred@1086652159 after retry RPC@995756655), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2b569afa -> wakeup thread Time-limited test -> (continuation of Deferred@2119317241 after retry RPC@1707330066) -> (continuation of Deferred@2023260211 after retry RPC@1227926231) -> (continuation of Deferred@104736564 after retry RPC@899151714) -> (continuation of Deferred@1477152828 after retry RPC@929665425) -> (continuation of Deferred@1086652159 after retry RPC@995756655))) after lookup
02:11:57.427 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:11:57.429 [DEBUG - kudu-nio-1] (Connection.java:757) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] Successfully connected to peer
02:11:57.456 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x35c3f83e, L:/127.0.0.1:47058 - R:/127.7.102.131:37689] using TOKEN/null
02:11:57.504 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.7.102.130:41431
02:11:57.506 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:57.508637  7578 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/logs
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data/info.pb
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.130:41431
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.130
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41213
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.130
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:11:57.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:11:57.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:11:57.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:57.850407  8393 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:11:57.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:57.850880  8393 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:11:57.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:57.851522  8393 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:11:57.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:57.884601  8393 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:11:57.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:57.885447  8393 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.102.130
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:57.921595  8393 tablet_server_runner.cc:78] Tablet server non-default flags:
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.130:41431
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data/info.pb
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.130
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41213
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.130
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/logs
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:11:57.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:11:57.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:11:57.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:11:57.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on e7f111948823
02:11:57.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16850
02:11:57.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:11:57.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:57.923003  8393 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:11:57.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:57.925204  8393 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:11:57.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:57.940757  8399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:11:59.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:57.942094  8400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:11:59.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:57.946143  8402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:11:59.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:11:59.168938  8401 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
02:11:59.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.169222  8393 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:11:59.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.173311  8393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:11:59.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.175724  8393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:11:59.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.177138  8393 hybrid_clock.cc:648] HybridClock initialized: now 1783476719177099 us; error 46 us; skew 500 ppm
02:11:59.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.178041  8393 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:11:59.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.185268  8393 webserver.cc:534] Webserver started at http://127.7.102.130:41213/ using document root <none> and password file <none>
02:11:59.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.186319  8393 fs_manager.cc:362] Metadata directory not provided
02:11:59.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.186650  8393 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:11:59.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.195248  8393 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.007s	sys 0.000s
02:11:59.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.200078  8411 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:11:59.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.201359  8393 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
02:11:59.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.201658  8393 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data,/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal
02:11:59.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "372cdb4006a84d17800acd046e7b5a41"
02:11:59.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:50 on dist-test-slave-5s6b"
02:11:59.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.203650  8393 fs_report.cc:389] FS layout report
02:11:59.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:11:59.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal
02:11:59.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal
02:11:59.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data/data
02:11:59.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:11:59.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:11:59.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:11:59.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:11:59.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:11:59.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:11:59.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:11:59.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:11:59.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:11:59.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:11:59.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:11:59.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.252666  8393 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:11:59.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.254721  8393 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:11:59.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.255141  8393 kserver.cc:163] Server-wide thread pool size limit: 3276
02:11:59.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.257652  8393 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:11:59.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.263933  8418 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:11:59.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.272377  8393 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:11:59.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.272640  8393 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.000s	sys 0.002s
02:11:59.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.272969  8393 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:11:59.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.278776  8393 ts_tablet_manager.cc:616] Registered 1 tablets
02:11:59.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.278998  8393 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.005s	sys 0.001s
02:11:59.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.279444  8418 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Bootstrap starting.
02:11:59.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.339102  8418 log.cc:826] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Log is configured to *not* fsync() on all Append() calls
02:11:59.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.373457  8418 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:11:59.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.375758  8418 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Bootstrap complete.
02:11:59.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.377856  8418 ts_tablet_manager.cc:1403] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Time spent bootstrapping tablet: real 0.099s	user 0.064s	sys 0.029s
02:11:59.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.405634  8418 raft_consensus.cc:359] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:11:59.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.409294  8418 raft_consensus.cc:740] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372cdb4006a84d17800acd046e7b5a41, State: Initialized, Role: FOLLOWER
02:11:59.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.410774  8418 consensus_queue.cc:260] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:11:59.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.416271  8418 ts_tablet_manager.cc:1434] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Time spent starting tablet: real 0.038s	user 0.032s	sys 0.006s
02:11:59.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.475687  8393 rpc_server.cc:307] RPC server started. Bound to: 127.7.102.130:41431
02:11:59.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.476082  8528 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.102.130:41431 every 8 connection(s)
02:11:59.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.479192  8393 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data/info.pb
02:11:59.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.487571  7578 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu as pid 8393
02:11:59.488 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:11:59.521 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726, tablet=064f4af2277f4b52af05ed12eaafba04, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:11:59.522 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.7.102.131:37689
02:11:59.522 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:11:59.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.523017  7578 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu with pid 8162
02:11:59.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.558965  8529 heartbeater.cc:344] Connected to a master server at 127.7.102.188:43935
02:11:59.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.559450  8529 heartbeater.cc:461] Registering TS with master...
02:11:59.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.560954  8529 heartbeater.cc:507] Master 127.7.102.188:43935 requested a full tablet report, sending...
02:11:59.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.566362  7778 ts_manager.cc:194] Re-registered known tserver with Master: 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:11:59.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.567770  8530 heartbeater.cc:344] Connected to a master server at 127.7.102.189:33589
02:11:59.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.568173  8530 heartbeater.cc:461] Registering TS with master...
02:11:59.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.569242  8530 heartbeater.cc:507] Master 127.7.102.189:33589 requested a full tablet report, sending...
02:11:59.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.572146  8531 heartbeater.cc:344] Connected to a master server at 127.7.102.190:37199
02:11:59.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.572453  8531 heartbeater.cc:461] Registering TS with master...
02:11:59.573 [DEBUG - kudu-nio-1] (Connection.java:239) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] handling channelInactive
02:11:59.573 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state READY due to: connection closed
02:11:59.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.573235  7696 ts_manager.cc:194] Re-registered known tserver with Master: 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:11:59.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.573370  8531 heartbeater.cc:507] Master 127.7.102.190:37199 requested a full tablet report, sending...
02:11:59.575 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:11:59.576 [DEBUG - Time-limited test] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:11:59.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.576977  7696 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.102.130:34707
02:11:59.577 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:11:59.577 [DEBUG - kudu-nio-2] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:11:59.577 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:11:59.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:11:59.580456  7623 ts_manager.cc:194] Re-registered known tserver with Master: 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:11:59.587 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:11:59.587 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:11:59.587 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=12), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [3ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [11ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@836234183(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@30d0fae5 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7739fc48 -> wakeup thread Time-limited test)) after lookup
02:11:59.588 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:11:59.589 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:11:59.589 [DEBUG - kudu-nio-3] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:11:59.589 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:11:59.604 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:11:59.605 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:11:59.605 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [3ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [11ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [12ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [14ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [14ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [29ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@836234183(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@30d0fae5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1905389432 after retry RPC@1240630974), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7739fc48 -> wakeup thread Time-limited test -> (continuation of Deferred@1905389432 after retry RPC@1240630974))) after lookup
02:11:59.606 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:11:59.607 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:11:59.607 [DEBUG - kudu-nio-4] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:11:59.607 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:11:59.624 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:11:59.624 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:11:59.625 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=49), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [3ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [11ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [12ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [14ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [14ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [29ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [30ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [32ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [32ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [49ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@836234183(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@30d0fae5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1905389432 after retry RPC@1240630974) -> (continuation of Deferred@55418731 after retry RPC@1591160522), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7739fc48 -> wakeup thread Time-limited test -> (continuation of Deferred@1905389432 after retry RPC@1240630974) -> (continuation of Deferred@55418731 after retry RPC@1591160522))) after lookup
02:11:59.625 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:11:59.626 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:11:59.627 [DEBUG - kudu-nio-5] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:11:59.627 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:11:59.684 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:11:59.685 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:11:59.685 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=109), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [3ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [11ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [12ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [14ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [14ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [29ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [30ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [32ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [32ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [49ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [50ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [52ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [52ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [105ms] refreshing cache from master, [105ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [109ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@836234183(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@30d0fae5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1905389432 after retry RPC@1240630974) -> (continuation of Deferred@55418731 after retry RPC@1591160522) -> (continuation of Deferred@748979344 after retry RPC@1273126576), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7739fc48 -> wakeup thread Time-limited test -> (continuation of Deferred@1905389432 after retry RPC@1240630974) -> (continuation of Deferred@55418731 after retry RPC@1591160522) -> (continuation of Deferred@748979344 after retry RPC@1273126576))) after lookup
02:11:59.686 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:11:59.687 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:11:59.687 [DEBUG - kudu-nio-6] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:11:59.687 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:00.024 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:00.025 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:12:00.026 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=450), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [3ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [11ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [12ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [14ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [14ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [29ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [30ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [32ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [32ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [49ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [50ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [52ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [52ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [105ms] refreshing cache from master, [105ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [109ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [110ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [112ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [112ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [445ms] refreshing cache from master, [445ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [449ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@836234183(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@30d0fae5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1905389432 after retry RPC@1240630974) -> (continuation of Deferred@55418731 after retry RPC@1591160522) -> (continuation of Deferred@748979344 after retry RPC@1273126576) -> (continuation of Deferred@70746516 after retry RPC@1248834408), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7739fc48 -> wakeup thread Time-limited test -> (continuation of Deferred@1905389432 after retry RPC@1240630974) -> (continuation of Deferred@55418731 after retry RPC@1591160522) -> (continuation of Deferred@748979344 after retry RPC@1273126576) -> (continuation of Deferred@70746516 after retry RPC@1248834408))) after lookup
02:12:00.027 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:00.029 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:00.029 [DEBUG - kudu-nio-7] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:00.029 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:00.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:00.581624  8530 heartbeater.cc:499] Master 127.7.102.189:33589 was elected leader, sending a full tablet report...
02:12:00.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:00.606189  8541 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe0a2e8ea6da4fa8b844e7d99e3cae0c)
02:12:00.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:00.606596  8541 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:00.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:00.608093  8541 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689), 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:12:00.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:00.614070  7901 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689): Network error: Client connection negotiation failed: client connection to 127.7.102.131:37689: connect: Connection refused (error 111)
02:12:00.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:00.620558  8484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "cc9ab047ac31487896edb4faf2fb74b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "372cdb4006a84d17800acd046e7b5a41" is_pre_election: true
02:12:00.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:00.621284  8484 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc9ab047ac31487896edb4faf2fb74b5 in term 1.
02:12:00.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:00.622510  7900 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 372cdb4006a84d17800acd046e7b5a41, cc9ab047ac31487896edb4faf2fb74b5; no voters: fe0a2e8ea6da4fa8b844e7d99e3cae0c
02:12:00.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:00.623231  8541 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
02:12:00.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:00.623548  8541 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader fe0a2e8ea6da4fa8b844e7d99e3cae0c)
02:12:00.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:00.623775  8541 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 2 FOLLOWER]: Advancing to term 3
02:12:00.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:00.627960  8541 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:00.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:00.629562  8541 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 3 election: Requested vote from peers fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689), 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:12:00.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:00.630626  8484 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "cc9ab047ac31487896edb4faf2fb74b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "372cdb4006a84d17800acd046e7b5a41"
02:12:00.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:00.631168  8484 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 1 FOLLOWER]: Advancing to term 3
02:12:00.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:00.633144  7901 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689): Network error: Client connection negotiation failed: client connection to 127.7.102.131:37689: connect: Connection refused (error 111)
02:12:00.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:00.638439  8484 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc9ab047ac31487896edb4faf2fb74b5 in term 3.
02:12:00.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:00.639390  7900 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 372cdb4006a84d17800acd046e7b5a41, cc9ab047ac31487896edb4faf2fb74b5; no voters: fe0a2e8ea6da4fa8b844e7d99e3cae0c
02:12:00.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:00.640069  8541 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 3 FOLLOWER]: Leader election won for term 3
02:12:00.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:00.641155  8541 raft_consensus.cc:697] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 3 LEADER]: Becoming Leader. State: Replica: cc9ab047ac31487896edb4faf2fb74b5, State: Running, Role: LEADER
02:12:00.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:00.642441  8541 consensus_queue.cc:237] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:00.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:00.651015  7696 catalog_manager.cc:5696] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 reported cstate change: term changed from 2 to 3, leader changed from fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131) to cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129). New cstate: current_term: 3 leader_uuid: "cc9ab047ac31487896edb4faf2fb74b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } health_report { overall_health: UNKNOWN } } }
02:12:01.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:01.053890  7901 consensus_peers.cc:597] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 -> Peer fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689): Couldn't send request to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c. Status: Network error: Client connection negotiation failed: client connection to 127.7.102.131:37689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:12:01.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:01.222391  8484 raft_consensus.cc:1275] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 3 FOLLOWER]: Refusing update from remote peer cc9ab047ac31487896edb4faf2fb74b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
02:12:01.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:01.224175  8541 consensus_queue.cc:1048] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
02:12:01.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:01.239447  8561 mvcc.cc:204] Tried to move back new op lower bound from 7305120634719391744 to 7305120632402870272. Current Snapshot: MvccSnapshot[applied={T|T < 7305120634719391744}]
02:12:01.846 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:01.847 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)[L],fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:01.848 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=2272), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [3ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [5ms] refreshing cache from master, [5ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [11ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [12ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [14ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [14ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [29ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [30ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [32ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [32ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [49ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [50ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [52ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [52ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [105ms] refreshing cache from master, [105ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [109ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [110ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [112ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [112ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [445ms] refreshing cache from master, [445ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [449ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [452ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [454ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [454ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [2265ms] refreshing cache from master, [2265ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [2271ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@836234183(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@30d0fae5 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1905389432 after retry RPC@1240630974) -> (continuation of Deferred@55418731 after retry RPC@1591160522) -> (continuation of Deferred@748979344 after retry RPC@1273126576) -> (continuation of Deferred@70746516 after retry RPC@1248834408) -> (continuation of Deferred@1480768517 after retry RPC@393652000), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7739fc48 -> wakeup thread Time-limited test -> (continuation of Deferred@1905389432 after retry RPC@1240630974) -> (continuation of Deferred@55418731 after retry RPC@1591160522) -> (continuation of Deferred@748979344 after retry RPC@1273126576) -> (continuation of Deferred@70746516 after retry RPC@1248834408) -> (continuation of Deferred@1480768517 after retry RPC@393652000))) after lookup
02:12:01.849 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] connecting to peer
02:12:01.850 [DEBUG - kudu-nio-0] (Connection.java:757) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] Successfully connected to peer
02:12:01.875 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xda8ea9a3, L:/127.0.0.1:35978 - R:/127.7.102.129:42373] using TOKEN/null
02:12:01.930 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.7.102.131:37689
02:12:01.931 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:12:01.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:01.932862  7578 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:12:01.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:12:01.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:12:01.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data
02:12:01.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:12:01.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:12:01.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:12:01.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:01.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:01.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:01.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/logs
02:12:01.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data/info.pb
02:12:01.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:01.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:12:01.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:12:01.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:12:01.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:12:01.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:01.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:01.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:01.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:12:01.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:12:01.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.131:37689
02:12:01.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.131
02:12:01.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:12:01.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38903
02:12:01.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.131
02:12:01.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:12:01.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:01.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:01.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:12:02.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:02.162611  7901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.102.131:37689: connect: Connection refused (error 111) [suppressed 4 similar messages]
02:12:02.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:02.270655  8579 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:12:02.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:02.271242  8579 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:12:02.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:02.271884  8579 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:12:02.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:02.305008  8579 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:12:02.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:02.306056  8579 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.102.131
02:12:02.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:02.342777  8579 tablet_server_runner.cc:78] Tablet server non-default flags:
02:12:02.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:02.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:12:02.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:02.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:02.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data
02:12:02.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:12:02.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:02.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:02.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.131:37689
02:12:02.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:12:02.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:02.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:02.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data/info.pb
02:12:02.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.131
02:12:02.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38903
02:12:02.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:12:02.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:12:02.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:02.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:12:02.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:12:02.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:02.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.131
02:12:02.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/logs
02:12:02.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:02.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:12:02.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:12:02.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:12:02.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:12:02.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:12:02.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:12:02.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on e7f111948823
02:12:02.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16850
02:12:02.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:12:02.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:02.344560  8579 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:12:02.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:02.346282  8579 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:12:02.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:02.359951  8587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:03.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:03.768539  8586 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 8579
02:12:03.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:03.772500  8586 kernel_stack_watchdog.cc:198] Thread 8579 stuck at /home/jenkins-slave/workspace/build_and_test@2/src/kudu/util/thread.cc:642 for 403ms:
02:12:03.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
02:12:03.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
02:12:03.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
02:12:03.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
02:12:03.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:03.773785  8579 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.412s	user 0.447s	sys 0.856s
02:12:03.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:03.774206  8579 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.412s	user 0.447s	sys 0.856s
02:12:03.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:02.362423  8588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:03.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:03.774957  8589 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1411 milliseconds
02:12:03.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:03.776233  8590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:03.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.776229  8579 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:12:03.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.779345  8579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:12:03.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.781388  8579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:12:03.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.782730  8579 hybrid_clock.cc:648] HybridClock initialized: now 1783476723782680 us; error 108 us; skew 500 ppm
02:12:03.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.783483  8579 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:12:03.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.789517  8579 webserver.cc:534] Webserver started at http://127.7.102.131:38903/ using document root <none> and password file <none>
02:12:03.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.790588  8579 fs_manager.cc:362] Metadata directory not provided
02:12:03.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.790864  8579 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:12:03.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.798647  8579 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.006s	sys 0.002s
02:12:03.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.803236  8602 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:12:03.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.804482  8579 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.002s
02:12:03.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.804909  8579 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data,/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:12:03.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c"
02:12:03.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:52 on dist-test-slave-5s6b"
02:12:03.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.806970  8579 fs_report.cc:389] FS layout report
02:12:03.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:12:03.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:12:03.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:12:03.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data/data
02:12:03.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:12:03.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:12:03.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:12:03.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:12:03.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:12:03.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:12:03.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:12:03.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:12:03.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:12:03.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:12:03.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:12:03.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.864269  8579 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:12:03.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.865988  8579 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:12:03.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.866580  8579 kserver.cc:163] Server-wide thread pool size limit: 3276
02:12:03.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.869467  8579 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:12:03.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.875540  8609 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:12:03.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.884155  8579 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:12:03.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.884511  8579 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.000s	sys 0.002s
02:12:03.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.884918  8579 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:12:03.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.890244  8579 ts_tablet_manager.cc:616] Registered 1 tablets
02:12:03.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.890523  8579 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.002s	sys 0.004s
02:12:03.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.890913  8609 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Bootstrap starting.
02:12:03.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.948261  8609 log.cc:826] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Log is configured to *not* fsync() on all Append() calls
02:12:03.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.997941  8609 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:12:04.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:03.999178  8609 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Bootstrap complete.
02:12:04.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.001230  8609 ts_tablet_manager.cc:1403] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Time spent bootstrapping tablet: real 0.111s	user 0.076s	sys 0.027s
02:12:04.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:04.009537  7901 consensus_peers.cc:597] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 -> Peer fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689): Couldn't send request to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c. Status: Network error: Client connection negotiation failed: client connection to 127.7.102.131:37689: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
02:12:04.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.029605  8609 raft_consensus.cc:359] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:04.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.033397  8609 raft_consensus.cc:740] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe0a2e8ea6da4fa8b844e7d99e3cae0c, State: Initialized, Role: FOLLOWER
02:12:04.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.035044  8609 consensus_queue.cc:260] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:04.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.042502  8609 ts_tablet_manager.cc:1434] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Time spent starting tablet: real 0.041s	user 0.036s	sys 0.004s
02:12:04.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.088855  8579 rpc_server.cc:307] RPC server started. Bound to: 127.7.102.131:37689
02:12:04.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.089170  8719 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.102.131:37689 every 8 connection(s)
02:12:04.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.091791  8579 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data/info.pb
02:12:04.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.095916  7578 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu as pid 8579
02:12:04.096 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:04.131 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726, tablet=064f4af2277f4b52af05ed12eaafba04, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:12:04.131 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.7.102.129:42373
02:12:04.132 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:12:04.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.132668  7578 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu with pid 7881
02:12:04.171 [DEBUG - kudu-nio-0] (Connection.java:239) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] handling channelInactive
02:12:04.172 [DEBUG - kudu-nio-0] (Connection.java:709) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] cleaning up while in state READY due to: connection closed
02:12:04.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.171936  8721 heartbeater.cc:344] Connected to a master server at 127.7.102.189:33589
02:12:04.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.172403  8721 heartbeater.cc:461] Registering TS with master...
02:12:04.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.173797  8721 heartbeater.cc:507] Master 127.7.102.189:33589 requested a full tablet report, sending...
02:12:04.175 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:04.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.178409  8720 heartbeater.cc:344] Connected to a master server at 127.7.102.188:43935
02:12:04.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.178750  8720 heartbeater.cc:461] Registering TS with master...
02:12:04.179 [DEBUG - Time-limited test] (Connection.java:748) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] connecting to peer
02:12:04.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.179708  8720 heartbeater.cc:507] Master 127.7.102.188:43935 requested a full tablet report, sending...
02:12:04.181 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:04.182 [DEBUG - kudu-nio-1] (Connection.java:709) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] cleaning up while in state CONNECTING due to: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:04.182 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:04.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.183198  7697 ts_manager.cc:194] Re-registered known tserver with Master: fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689)
02:12:04.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.184578  8722 heartbeater.cc:344] Connected to a master server at 127.7.102.190:37199
02:12:04.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.184917  8722 heartbeater.cc:461] Registering TS with master...
02:12:04.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.184846  7778 ts_manager.cc:194] Re-registered known tserver with Master: fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689)
02:12:04.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.185863  8722 heartbeater.cc:507] Master 127.7.102.190:37199 requested a full tablet report, sending...
02:12:04.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.186513  7697 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.102.131:38021
02:12:04.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:04.189107  7622 ts_manager.cc:194] Re-registered known tserver with Master: fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689)
02:12:04.205 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:04.206 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)[L],fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:04.206 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [5ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [5ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [28ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1972686209(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@57793bb4 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@271fdfcf -> wakeup thread Time-limited test)) after lookup
02:12:04.207 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] connecting to peer
02:12:04.208 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:04.208 [DEBUG - kudu-nio-2] (Connection.java:709) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] cleaning up while in state CONNECTING due to: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:04.208 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:04.225 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:04.225 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)[L],fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:04.226 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=48), Traces: [0ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [5ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [5ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [28ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [29ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [31ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [31ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [43ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [47ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1972686209(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@57793bb4 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@248920877 after retry RPC@1589769361), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@271fdfcf -> wakeup thread Time-limited test -> (continuation of Deferred@248920877 after retry RPC@1589769361))) after lookup
02:12:04.226 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] connecting to peer
02:12:04.227 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:04.227 [DEBUG - kudu-nio-3] (Connection.java:709) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] cleaning up while in state CONNECTING due to: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:04.228 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:04.244 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:04.245 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)[L],fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:04.245 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=68), Traces: [0ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [5ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [5ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [28ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [29ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [31ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [31ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [43ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [47ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [49ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [51ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [51ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [67ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1972686209(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@57793bb4 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@248920877 after retry RPC@1589769361) -> (continuation of Deferred@2053947924 after retry RPC@735630516), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@271fdfcf -> wakeup thread Time-limited test -> (continuation of Deferred@248920877 after retry RPC@1589769361) -> (continuation of Deferred@2053947924 after retry RPC@735630516))) after lookup
02:12:04.246 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] connecting to peer
02:12:04.247 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:04.247 [DEBUG - kudu-nio-4] (Connection.java:709) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] cleaning up while in state CONNECTING due to: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:04.247 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:04.284 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:04.285 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)[L],fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:04.285 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=107), Traces: [0ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [5ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [5ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [28ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [29ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [31ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [31ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [43ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [47ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [49ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [51ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [51ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [67ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [68ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [70ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [70ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [103ms] refreshing cache from master, [103ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [107ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1972686209(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@57793bb4 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@248920877 after retry RPC@1589769361) -> (continuation of Deferred@2053947924 after retry RPC@735630516) -> (continuation of Deferred@408377703 after retry RPC@1967604653), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@271fdfcf -> wakeup thread Time-limited test -> (continuation of Deferred@248920877 after retry RPC@1589769361) -> (continuation of Deferred@2053947924 after retry RPC@735630516) -> (continuation of Deferred@408377703 after retry RPC@1967604653))) after lookup
02:12:04.285 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] connecting to peer
02:12:04.286 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:04.286 [DEBUG - kudu-nio-5] (Connection.java:709) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] cleaning up while in state CONNECTING due to: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:04.287 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:04.765 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:04.765 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)[L],fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:04.766 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=588), Traces: [0ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [5ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [5ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [28ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [29ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [31ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [31ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [43ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [47ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [49ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [51ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [51ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [67ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [68ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [70ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [70ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [103ms] refreshing cache from master, [103ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [107ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [108ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [110ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [110ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [584ms] refreshing cache from master, [584ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [587ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1972686209(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@57793bb4 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@248920877 after retry RPC@1589769361) -> (continuation of Deferred@2053947924 after retry RPC@735630516) -> (continuation of Deferred@408377703 after retry RPC@1967604653) -> (continuation of Deferred@768580565 after retry RPC@1650126826), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@271fdfcf -> wakeup thread Time-limited test -> (continuation of Deferred@248920877 after retry RPC@1589769361) -> (continuation of Deferred@2053947924 after retry RPC@735630516) -> (continuation of Deferred@408377703 after retry RPC@1967604653) -> (continuation of Deferred@768580565 after retry RPC@1650126826))) after lookup
02:12:04.766 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] connecting to peer
02:12:04.767 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:04.767 [DEBUG - kudu-nio-6] (Connection.java:709) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] cleaning up while in state CONNECTING due to: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:04.767 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:05.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.143337  8731 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cc9ab047ac31487896edb4faf2fb74b5)
02:12:05.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.143824  8731 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:05.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.146217  8731 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689), cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373)
02:12:05.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:05.150906  8412 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.102.129:42373: connect: Connection refused (error 111)
02:12:05.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:05.154613  8412 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373): Network error: Client connection negotiation failed: client connection to 127.7.102.129:42373: connect: Connection refused (error 111)
02:12:05.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.157931  8674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "372cdb4006a84d17800acd046e7b5a41" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" is_pre_election: true
02:12:05.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.158800  8674 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372cdb4006a84d17800acd046e7b5a41 in term 2.
02:12:05.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.160256  8413 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 372cdb4006a84d17800acd046e7b5a41, fe0a2e8ea6da4fa8b844e7d99e3cae0c; no voters: cc9ab047ac31487896edb4faf2fb74b5
02:12:05.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.161052  8731 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 3 FOLLOWER]: Leader pre-election won for term 4
02:12:05.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.161304  8731 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 3 FOLLOWER]: Starting leader election (detected failure of leader cc9ab047ac31487896edb4faf2fb74b5)
02:12:05.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.161551  8731 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 3 FOLLOWER]: Advancing to term 4
02:12:05.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.166061  8731 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:05.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.167779  8731 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 4 election: Requested vote from peers fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689), cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373)
02:12:05.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.168675  8674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "372cdb4006a84d17800acd046e7b5a41" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c"
02:12:05.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.169256  8674 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 2 FOLLOWER]: Advancing to term 4
02:12:05.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:05.171772  8412 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373): Network error: Client connection negotiation failed: client connection to 127.7.102.129:42373: connect: Connection refused (error 111)
02:12:05.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.176366  8674 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372cdb4006a84d17800acd046e7b5a41 in term 4.
02:12:05.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.177472  8413 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 372cdb4006a84d17800acd046e7b5a41, fe0a2e8ea6da4fa8b844e7d99e3cae0c; no voters: cc9ab047ac31487896edb4faf2fb74b5
02:12:05.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.178112  8731 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 4 FOLLOWER]: Leader election won for term 4
02:12:05.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.179661  8731 raft_consensus.cc:697] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 4 LEADER]: Becoming Leader. State: Replica: 372cdb4006a84d17800acd046e7b5a41, State: Running, Role: LEADER
02:12:05.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.180609  8731 consensus_queue.cc:237] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:05.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.190485  8721 heartbeater.cc:499] Master 127.7.102.189:33589 was elected leader, sending a full tablet report...
02:12:05.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.192181  7697 catalog_manager.cc:5696] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 reported cstate change: term changed from 3 to 4, leader changed from cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129) to 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130). New cstate: current_term: 4 leader_uuid: "372cdb4006a84d17800acd046e7b5a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } health_report { overall_health: HEALTHY } } }
02:12:05.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:05.599861  8412 consensus_peers.cc:597] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 -> Peer cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373): Couldn't send request to peer cc9ab047ac31487896edb4faf2fb74b5. Status: Network error: Client connection negotiation failed: client connection to 127.7.102.129:42373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:12:05.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.729991  8674 raft_consensus.cc:1275] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 4 FOLLOWER]: Refusing update from remote peer 372cdb4006a84d17800acd046e7b5a41: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
02:12:05.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.731428  8731 consensus_queue.cc:1048] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.000s
02:12:05.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:05.746078  8749 mvcc.cc:204] Tried to move back new op lower bound from 7305120652823416832 to 7305120647759032320. Current Snapshot: MvccSnapshot[applied={T|T < 7305120652823416832}]
02:12:06.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:06.223858  7654 debug-util.cc:398] Leaking SignalData structure 0x7b0800094640 after lost signal to thread 7591
02:12:06.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:06.224400  7654 debug-util.cc:398] Leaking SignalData structure 0x7b080009df60 after lost signal to thread 7657
02:12:06.825 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:06.826 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)[L],cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:06.826 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=2648), Traces: [0ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [5ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [5ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [24ms] refreshing cache from master, [24ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [28ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [29ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [31ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [31ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [43ms] refreshing cache from master, [44ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [47ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [49ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [51ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [51ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [63ms] refreshing cache from master, [63ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [67ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [68ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [70ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [70ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [103ms] refreshing cache from master, [103ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [107ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [108ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [110ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [110ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [584ms] refreshing cache from master, [584ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [587ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [589ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [590ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [590ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [2643ms] refreshing cache from master, [2643ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [2648ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1972686209(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@57793bb4 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@248920877 after retry RPC@1589769361) -> (continuation of Deferred@2053947924 after retry RPC@735630516) -> (continuation of Deferred@408377703 after retry RPC@1967604653) -> (continuation of Deferred@768580565 after retry RPC@1650126826) -> (continuation of Deferred@640156924 after retry RPC@322476652), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@271fdfcf -> wakeup thread Time-limited test -> (continuation of Deferred@248920877 after retry RPC@1589769361) -> (continuation of Deferred@2053947924 after retry RPC@735630516) -> (continuation of Deferred@408377703 after retry RPC@1967604653) -> (continuation of Deferred@768580565 after retry RPC@1650126826) -> (continuation of Deferred@640156924 after retry RPC@322476652))) after lookup
02:12:06.827 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:12:06.829 [DEBUG - kudu-nio-7] (Connection.java:757) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] Successfully connected to peer
02:12:06.868 [DEBUG - kudu-nio-7] (Negotiator.java:919) Authenticated connection [id: 0x5927a1ad, L:/127.0.0.1:56198 - R:/127.7.102.130:41431] using TOKEN/null
02:12:06.929 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.7.102.129:42373
02:12:06.929 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:12:06.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:06.931134  7578 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:12:06.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:12:06.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal
02:12:06.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data
02:12:06.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:12:06.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:12:06.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:12:06.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:06.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:06.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/logs
02:12:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data/info.pb
02:12:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:12:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:12:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:12:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:12:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:12:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:12:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.129:42373
02:12:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.129
02:12:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:12:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37707
02:12:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.129
02:12:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:12:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:12:07.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:07.279340  8770 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:12:07.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:07.279919  8770 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:12:07.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:07.280773  8770 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:12:07.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:07.316751  8770 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:12:07.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:07.317766  8770 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.102.129
02:12:07.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:07.356392  8770 tablet_server_runner.cc:78] Tablet server non-default flags:
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.129:42373
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data/info.pb
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.129
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37707
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:07.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.129
02:12:07.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/logs
02:12:07.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:07.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:12:07.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:12:07.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:12:07.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:12:07.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:12:07.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:12:07.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on e7f111948823
02:12:07.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16850
02:12:07.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:12:07.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:07.358254  8770 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:12:07.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:07.360318  8770 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:12:07.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:07.378188  8779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:08.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:08.347658  8412 consensus_peers.cc:597] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 -> Peer cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373): Couldn't send request to peer cc9ab047ac31487896edb4faf2fb74b5. Status: Network error: Client connection negotiation failed: client connection to 127.7.102.129:42373: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
02:12:08.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:07.381961  8780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:08.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:07.384361  8782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:08.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:08.820061  8781 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1435 milliseconds
02:12:08.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.820227  8770 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:12:08.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.821660  8770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:12:08.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.824059  8770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:12:08.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.825502  8770 hybrid_clock.cc:648] HybridClock initialized: now 1783476728825469 us; error 46 us; skew 500 ppm
02:12:08.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.826411  8770 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:12:08.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.832563  8770 webserver.cc:534] Webserver started at http://127.7.102.129:37707/ using document root <none> and password file <none>
02:12:08.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.833693  8770 fs_manager.cc:362] Metadata directory not provided
02:12:08.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.833957  8770 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:12:08.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.841948  8770 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.006s	sys 0.001s
02:12:08.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.846549  8793 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:12:08.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.847590  8770 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.003s
02:12:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.847950  8770 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data,/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal
02:12:08.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cc9ab047ac31487896edb4faf2fb74b5"
02:12:08.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:48 on dist-test-slave-5s6b"
02:12:08.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.849896  8770 fs_report.cc:389] FS layout report
02:12:08.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:12:08.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal
02:12:08.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal
02:12:08.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data/data
02:12:08.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:12:08.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:12:08.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:12:08.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:12:08.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:12:08.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:12:08.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:12:08.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:12:08.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:12:08.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:12:08.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:12:08.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.933071  8770 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:12:08.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.934798  8770 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:12:08.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.935415  8770 kserver.cc:163] Server-wide thread pool size limit: 3276
02:12:08.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.938213  8770 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:12:08.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.944552  8800 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:12:08.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.953068  8770 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:12:08.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.953379  8770 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.002s	sys 0.000s
02:12:08.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.953703  8770 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:12:08.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.958638  8770 ts_tablet_manager.cc:616] Registered 1 tablets
02:12:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.958855  8770 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.005s	sys 0.000s
02:12:08.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:08.959211  8800 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Bootstrap starting.
02:12:09.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.033043  8800 log.cc:826] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Log is configured to *not* fsync() on all Append() calls
02:12:09.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.104176  8800 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:12:09.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.105226  8800 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Bootstrap complete.
02:12:09.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.107079  8800 ts_tablet_manager.cc:1403] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Time spent bootstrapping tablet: real 0.148s	user 0.095s	sys 0.046s
02:12:09.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.134519  8800 raft_consensus.cc:359] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:09.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.138202  8800 raft_consensus.cc:740] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9ab047ac31487896edb4faf2fb74b5, State: Initialized, Role: FOLLOWER
02:12:09.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.140249  8800 consensus_queue.cc:260] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:09.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.145423  8800 ts_tablet_manager.cc:1434] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Time spent starting tablet: real 0.038s	user 0.023s	sys 0.013s
02:12:09.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.150794  8770 rpc_server.cc:307] RPC server started. Bound to: 127.7.102.129:42373
02:12:09.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.151088  8908 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.102.129:42373 every 8 connection(s)
02:12:09.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.153436  8770 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data/info.pb
02:12:09.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.155645  7578 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu as pid 8770
02:12:09.156 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:09.200 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726, tablet=064f4af2277f4b52af05ed12eaafba04, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:12:09.221 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.7.102.130:41431
02:12:09.221 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:12:09.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.222265  7578 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu with pid 8393
02:12:09.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.255640  8910 heartbeater.cc:344] Connected to a master server at 127.7.102.189:33589
02:12:09.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.256150  8910 heartbeater.cc:461] Registering TS with master...
02:12:09.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.257570  8910 heartbeater.cc:507] Master 127.7.102.189:33589 requested a full tablet report, sending...
02:12:09.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.258158  8909 heartbeater.cc:344] Connected to a master server at 127.7.102.188:43935
02:12:09.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.258694  8909 heartbeater.cc:461] Registering TS with master...
02:12:09.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.259855  8909 heartbeater.cc:507] Master 127.7.102.188:43935 requested a full tablet report, sending...
02:12:09.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.263540  7696 ts_manager.cc:194] Re-registered known tserver with Master: cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373)
02:12:09.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.265048  7778 ts_manager.cc:194] Re-registered known tserver with Master: cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373)
02:12:09.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.266660  7696 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.102.129:48563
02:12:09.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.268375  8911 heartbeater.cc:344] Connected to a master server at 127.7.102.190:37199
02:12:09.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.268766  8911 heartbeater.cc:461] Registering TS with master...
02:12:09.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.269495  8911 heartbeater.cc:507] Master 127.7.102.190:37199 requested a full tablet report, sending...
02:12:09.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:09.272886  7623 ts_manager.cc:194] Re-registered known tserver with Master: cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373)
02:12:09.276 [DEBUG - kudu-nio-7] (Connection.java:239) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] handling channelInactive
02:12:09.277 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state READY due to: connection closed
02:12:09.277 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:09.278 [DEBUG - Time-limited test] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:12:09.279 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:09.279 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state CONNECTING due to: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:09.279 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:09.283 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:09.284 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)[L],cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:09.284 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=6), Traces: [0ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [1ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [1ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [2ms] refreshing cache from master, [2ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [5ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@397516543(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@408d9d9f -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@9e7e219 -> wakeup thread Time-limited test)) after lookup
02:12:09.284 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:12:09.285 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:09.285 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state CONNECTING due to: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:09.286 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:09.304 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:09.305 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)[L],cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:09.305 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=27), Traces: [0ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [1ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [1ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [2ms] refreshing cache from master, [2ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [5ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [6ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [8ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [8ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [22ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [26ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@397516543(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@408d9d9f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@205773178 after retry RPC@973215169), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@9e7e219 -> wakeup thread Time-limited test -> (continuation of Deferred@205773178 after retry RPC@973215169))) after lookup
02:12:09.305 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:12:09.307 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:09.307 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state CONNECTING due to: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:09.307 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:09.324 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:09.324 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)[L],cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:09.325 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=47), Traces: [0ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [1ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [1ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [2ms] refreshing cache from master, [2ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [5ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [6ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [8ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [8ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [22ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [26ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [27ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [29ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [29ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [46ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@397516543(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@408d9d9f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@205773178 after retry RPC@973215169) -> (continuation of Deferred@1489249881 after retry RPC@40692600), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@9e7e219 -> wakeup thread Time-limited test -> (continuation of Deferred@205773178 after retry RPC@973215169) -> (continuation of Deferred@1489249881 after retry RPC@40692600))) after lookup
02:12:09.325 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:12:09.327 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:09.327 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state CONNECTING due to: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:09.327 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:09.424 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:09.425 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)[L],cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:09.425 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=147), Traces: [0ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [1ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [1ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [2ms] refreshing cache from master, [2ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [5ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [6ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [8ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [8ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [22ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [26ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [27ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [29ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [29ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [46ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [47ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [49ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [49ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [142ms] refreshing cache from master, [143ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [146ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@397516543(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@408d9d9f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@205773178 after retry RPC@973215169) -> (continuation of Deferred@1489249881 after retry RPC@40692600) -> (continuation of Deferred@1689187106 after retry RPC@33350628), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@9e7e219 -> wakeup thread Time-limited test -> (continuation of Deferred@205773178 after retry RPC@973215169) -> (continuation of Deferred@1489249881 after retry RPC@40692600) -> (continuation of Deferred@1689187106 after retry RPC@33350628))) after lookup
02:12:09.426 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:12:09.427 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:09.427 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state CONNECTING due to: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:09.427 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:09.568 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:09.568 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)[L],cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:09.569 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=291), Traces: [0ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [1ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [1ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [2ms] refreshing cache from master, [2ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [5ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [6ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [8ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [8ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [22ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [26ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [27ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [29ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [29ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [46ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [47ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [49ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [49ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [142ms] refreshing cache from master, [143ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [146ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [147ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [149ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [149ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [282ms] refreshing cache from master, [282ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [289ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@397516543(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@408d9d9f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@205773178 after retry RPC@973215169) -> (continuation of Deferred@1489249881 after retry RPC@40692600) -> (continuation of Deferred@1689187106 after retry RPC@33350628) -> (continuation of Deferred@1091813860 after retry RPC@655817185), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@9e7e219 -> wakeup thread Time-limited test -> (continuation of Deferred@205773178 after retry RPC@973215169) -> (continuation of Deferred@1489249881 after retry RPC@40692600) -> (continuation of Deferred@1689187106 after retry RPC@33350628) -> (continuation of Deferred@1091813860 after retry RPC@655817185))) after lookup
02:12:09.569 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:12:09.571 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:09.571 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state CONNECTING due to: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:09.571 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:10.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.270129  8910 heartbeater.cc:499] Master 127.7.102.189:33589 was elected leader, sending a full tablet report...
02:12:10.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.384449  8921 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 372cdb4006a84d17800acd046e7b5a41)
02:12:10.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.386364  8921 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:10.388 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:10.388 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)[L],cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:10.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.389710  8921 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373), 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:12:10.390 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=1111), Traces: [0ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [1ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [1ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [2ms] refreshing cache from master, [2ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [5ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [6ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [8ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [8ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [22ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [26ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [27ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [29ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [29ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [46ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [47ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [49ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [49ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [142ms] refreshing cache from master, [143ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [146ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [147ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [149ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [149ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [282ms] refreshing cache from master, [282ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [289ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [291ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [293ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [293ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [1102ms] refreshing cache from master, [1102ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [1107ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@397516543(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@408d9d9f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@205773178 after retry RPC@973215169) -> (continuation of Deferred@1489249881 after retry RPC@40692600) -> (continuation of Deferred@1689187106 after retry RPC@33350628) -> (continuation of Deferred@1091813860 after retry RPC@655817185) -> (continuation of Deferred@2053598667 after retry RPC@1781422743), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@9e7e219 -> wakeup thread Time-limited test -> (continuation of Deferred@205773178 after retry RPC@973215169) -> (continuation of Deferred@1489249881 after retry RPC@40692600) -> (continuation of Deferred@1689187106 after retry RPC@33350628) -> (continuation of Deferred@1091813860 after retry RPC@655817185) -> (continuation of Deferred@2053598667 after retry RPC@1781422743))) after lookup
02:12:10.391 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:12:10.394 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:10.394 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state CONNECTING due to: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:10.395 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:10.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:10.403367  8603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.102.130:41431: connect: Connection refused (error 111)
02:12:10.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.406457  8926 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
02:12:10.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.407028  8926 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:10.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:10.409164  8603 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431): Network error: Client connection negotiation failed: client connection to 127.7.102.130:41431: connect: Connection refused (error 111)
02:12:10.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.409838  8863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "cc9ab047ac31487896edb4faf2fb74b5" is_pre_election: true
02:12:10.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.410727  8863 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe0a2e8ea6da4fa8b844e7d99e3cae0c in term 3.
02:12:10.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.411230  8926 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689), 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:12:10.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:10.415740  8794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.102.130:41431: connect: Connection refused (error 111)
02:12:10.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.416563  8603 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cc9ab047ac31487896edb4faf2fb74b5, fe0a2e8ea6da4fa8b844e7d99e3cae0c; no voters: 372cdb4006a84d17800acd046e7b5a41
02:12:10.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.417583  8921 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 4 FOLLOWER]: Leader pre-election won for term 5
02:12:10.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.418030  8921 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 372cdb4006a84d17800acd046e7b5a41)
02:12:10.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.418486  8921 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 4 FOLLOWER]: Advancing to term 5
02:12:10.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:10.425724  8794 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431): Network error: Client connection negotiation failed: client connection to 127.7.102.130:41431: connect: Connection refused (error 111)
02:12:10.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.426219  8921 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:10.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.426831  8674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "cc9ab047ac31487896edb4faf2fb74b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" is_pre_election: true
02:12:10.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.427531  8674 raft_consensus.cc:2368] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cc9ab047ac31487896edb4faf2fb74b5 for earlier term 4. Current term is 5.
02:12:10.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.428171  8921 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 5 election: Requested vote from peers cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373), 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:12:10.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.428833  8863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 16 } } ignore_live_leader: false dest_uuid: "cc9ab047ac31487896edb4faf2fb74b5"
02:12:10.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.429323  8863 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 3 FOLLOWER]: Advancing to term 5
02:12:10.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.429734  8795 leader_election.cc:400] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 4 pre-election: Vote denied by peer fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689) with higher term. Message: Invalid argument: T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cc9ab047ac31487896edb4faf2fb74b5 for earlier term 4. Current term is 5.
02:12:10.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.430009  8795 leader_election.cc:403] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
02:12:10.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:10.432335  8603 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431): Network error: Client connection negotiation failed: client connection to 127.7.102.130:41431: connect: Connection refused (error 111)
02:12:10.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.437041  8863 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe0a2e8ea6da4fa8b844e7d99e3cae0c in term 5.
02:12:10.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.437430  8926 raft_consensus.cc:2749] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689) with higher term. Message: Invalid argument: T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cc9ab047ac31487896edb4faf2fb74b5 for earlier term 4. Current term is 5.
02:12:10.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.438136  8603 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cc9ab047ac31487896edb4faf2fb74b5, fe0a2e8ea6da4fa8b844e7d99e3cae0c; no voters: 372cdb4006a84d17800acd046e7b5a41
02:12:10.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.438951  8921 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 5 FOLLOWER]: Leader election won for term 5
02:12:10.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.440650  8921 raft_consensus.cc:697] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 5 LEADER]: Becoming Leader. State: Replica: fe0a2e8ea6da4fa8b844e7d99e3cae0c, State: Running, Role: LEADER
02:12:10.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.441419  8921 consensus_queue.cc:237] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:10.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.449757  7696 catalog_manager.cc:5696] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c reported cstate change: term changed from 4 to 5, leader changed from 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130) to fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131). New cstate: current_term: 5 leader_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } health_report { overall_health: UNKNOWN } } }
02:12:10.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:10.830097  8603 consensus_peers.cc:597] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c -> Peer 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431): Couldn't send request to peer 372cdb4006a84d17800acd046e7b5a41. Status: Network error: Client connection negotiation failed: client connection to 127.7.102.130:41431: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:12:10.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.849707  8863 raft_consensus.cc:1275] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 5 FOLLOWER]: Refusing update from remote peer fe0a2e8ea6da4fa8b844e7d99e3cae0c: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 5 index: 17. (index mismatch)
02:12:10.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:10.851226  8921 consensus_queue.cc:1048] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
02:12:10.945 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:10.946 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:12:10.946 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=14, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=1668), Traces: [0ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [1ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [1ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [2ms] refreshing cache from master, [2ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [5ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [6ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [8ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [8ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [22ms] refreshing cache from master, [23ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [26ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [27ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [29ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [29ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [42ms] refreshing cache from master, [42ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [46ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [47ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [49ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [49ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [142ms] refreshing cache from master, [143ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [146ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [147ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [149ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [149ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [282ms] refreshing cache from master, [282ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [289ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [291ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [293ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [293ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [1102ms] refreshing cache from master, [1102ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [1107ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [1113ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [1116ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [1117ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [1662ms] refreshing cache from master, [1663ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [1667ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@397516543(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@408d9d9f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@205773178 after retry RPC@973215169) -> (continuation of Deferred@1489249881 after retry RPC@40692600) -> (continuation of Deferred@1689187106 after retry RPC@33350628) -> (continuation of Deferred@1091813860 after retry RPC@655817185) -> (continuation of Deferred@2053598667 after retry RPC@1781422743) -> (continuation of Deferred@1938598762 after retry RPC@1802415470), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@9e7e219 -> wakeup thread Time-limited test -> (continuation of Deferred@205773178 after retry RPC@973215169) -> (continuation of Deferred@1489249881 after retry RPC@40692600) -> (continuation of Deferred@1689187106 after retry RPC@33350628) -> (continuation of Deferred@1091813860 after retry RPC@655817185) -> (continuation of Deferred@2053598667 after retry RPC@1781422743) -> (continuation of Deferred@1938598762 after retry RPC@1802415470))) after lookup
02:12:10.947 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:10.948 [DEBUG - kudu-nio-7] (Connection.java:757) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] Successfully connected to peer
02:12:10.968 [DEBUG - kudu-nio-7] (Negotiator.java:919) Authenticated connection [id: 0x70dffd00, L:/127.0.0.1:44028 - R:/127.7.102.131:37689] using TOKEN/null
02:12:11.024 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.7.102.130:41431
02:12:11.024 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:12:11.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:11.025552  7578 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/logs
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data/info.pb
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.130:41431
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.130
02:12:11.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:12:11.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41213
02:12:11.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.130
02:12:11.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:12:11.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:11.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:11.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:12:11.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:11.363200  8953 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:12:11.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:11.363783  8953 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:12:11.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:11.364488  8953 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:12:11.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:11.422255  8953 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:12:11.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:11.423923  8953 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.102.130
02:12:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:11.475574  8953 tablet_server_runner.cc:78] Tablet server non-default flags:
02:12:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:12:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data
02:12:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal
02:12:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.130:41431
02:12:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:12:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data/info.pb
02:12:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.130
02:12:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41213
02:12:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:12:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:12:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:12:11.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:12:11.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:11.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.130
02:12:11.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/logs
02:12:11.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:11.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:12:11.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:12:11.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:12:11.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:12:11.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:12:11.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:12:11.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on e7f111948823
02:12:11.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16850
02:12:11.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:12:11.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:11.477322  8953 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:12:11.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:11.479113  8953 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:12:11.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:11.491369  8962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:13.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:12.896847  8961 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 8953
02:12:13.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:11.493897  8963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:13.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:13.063488  8953 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.571s	user 0.525s	sys 0.951s
02:12:13.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:13.063848  8953 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.571s	user 0.525s	sys 0.951s
02:12:13.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:13.068288  8964 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1570 milliseconds
02:12:13.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:13.069756  8965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:13.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.069819  8953 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:12:13.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.071136  8953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:12:13.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.073163  8953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:12:13.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.074499  8953 hybrid_clock.cc:648] HybridClock initialized: now 1783476733074474 us; error 46 us; skew 500 ppm
02:12:13.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.075253  8953 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:12:13.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.081270  8953 webserver.cc:534] Webserver started at http://127.7.102.130:41213/ using document root <none> and password file <none>
02:12:13.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.082118  8953 fs_manager.cc:362] Metadata directory not provided
02:12:13.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.082306  8953 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:12:13.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.090878  8953 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.005s	sys 0.002s
02:12:13.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.096377  8976 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:12:13.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.097679  8953 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.001s
02:12:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.098052  8953 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data,/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal
02:12:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "372cdb4006a84d17800acd046e7b5a41"
02:12:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:50 on dist-test-slave-5s6b"
02:12:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.100092  8953 fs_report.cc:389] FS layout report
02:12:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:12:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal
02:12:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal
02:12:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data/data
02:12:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:12:13.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:12:13.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:12:13.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:12:13.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:12:13.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:12:13.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:12:13.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:12:13.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:12:13.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:12:13.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:12:13.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.154527  8953 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:12:13.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.156510  8953 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:12:13.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.157070  8953 kserver.cc:163] Server-wide thread pool size limit: 3276
02:12:13.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.160774  8953 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:12:13.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.170286  8983 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:12:13.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.178963  8953 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:12:13.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.179194  8953 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.001s	sys 0.001s
02:12:13.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.179405  8953 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:12:13.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.184185  8953 ts_tablet_manager.cc:616] Registered 1 tablets
02:12:13.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.184459  8953 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.003s	sys 0.000s
02:12:13.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.184831  8983 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Bootstrap starting.
02:12:13.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.265659  8983 log.cc:826] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Log is configured to *not* fsync() on all Append() calls
02:12:13.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.325330  8983 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:12:13.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.326359  8983 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Bootstrap complete.
02:12:13.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.328116  8983 ts_tablet_manager.cc:1403] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Time spent bootstrapping tablet: real 0.144s	user 0.096s	sys 0.042s
02:12:13.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.349049  8983 raft_consensus.cc:359] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:13.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.352417  8983 raft_consensus.cc:740] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372cdb4006a84d17800acd046e7b5a41, State: Initialized, Role: FOLLOWER
02:12:13.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.353726  8983 consensus_queue.cc:260] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:13.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.357410  8983 ts_tablet_manager.cc:1434] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Time spent starting tablet: real 0.029s	user 0.026s	sys 0.003s
02:12:13.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.373467  8953 rpc_server.cc:307] RPC server started. Bound to: 127.7.102.130:41431
02:12:13.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.374234  9091 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.102.130:41431 every 8 connection(s)
02:12:13.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.376611  8953 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data/info.pb
02:12:13.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.386345  7578 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu as pid 8953
02:12:13.387 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:13.421 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726, tablet=064f4af2277f4b52af05ed12eaafba04, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:12:13.421 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.7.102.131:37689
02:12:13.422 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:12:13.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.422739  7578 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu with pid 8579
02:12:13.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.453454  9092 heartbeater.cc:344] Connected to a master server at 127.7.102.188:43935
02:12:13.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.453908  9092 heartbeater.cc:461] Registering TS with master...
02:12:13.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.454535  9094 heartbeater.cc:344] Connected to a master server at 127.7.102.190:37199
02:12:13.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.454945  9094 heartbeater.cc:461] Registering TS with master...
02:12:13.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.455456  9092 heartbeater.cc:507] Master 127.7.102.188:43935 requested a full tablet report, sending...
02:12:13.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.456126  9094 heartbeater.cc:507] Master 127.7.102.190:37199 requested a full tablet report, sending...
02:12:13.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.456384  9093 heartbeater.cc:344] Connected to a master server at 127.7.102.189:33589
02:12:13.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.456686  9093 heartbeater.cc:461] Registering TS with master...
02:12:13.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.457567  9093 heartbeater.cc:507] Master 127.7.102.189:33589 requested a full tablet report, sending...
02:12:13.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.460352  7696 ts_manager.cc:194] Re-registered known tserver with Master: 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:12:13.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.460469  7623 ts_manager.cc:194] Re-registered known tserver with Master: 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:12:13.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.462952  7777 ts_manager.cc:194] Re-registered known tserver with Master: 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:12:13.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:13.463554  7696 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.102.130:35419
02:12:13.466 [DEBUG - kudu-nio-7] (Connection.java:239) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] handling channelInactive
02:12:13.467 [DEBUG - kudu-nio-7] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state READY due to: connection closed
02:12:13.467 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:13.468 [DEBUG - Time-limited test] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:13.470 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:13.470 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:13.470 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:13.484 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:13.485 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:12:13.485 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=17), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [2ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [12ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [16ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1244709754(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@54ab028f -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7492af39 -> wakeup thread Time-limited test)) after lookup
02:12:13.486 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:13.487 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:13.487 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:13.487 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:13.504 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:13.504 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:12:13.505 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=36), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [2ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [12ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [16ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [17ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [19ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [19ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [36ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1244709754(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@54ab028f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1741146898 after retry RPC@114121652), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7492af39 -> wakeup thread Time-limited test -> (continuation of Deferred@1741146898 after retry RPC@114121652))) after lookup
02:12:13.505 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:13.506 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:13.506 [DEBUG - kudu-nio-2] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:13.506 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:13.545 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:13.545 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:12:13.546 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=78), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [2ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [12ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [16ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [17ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [19ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [19ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [36ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [37ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [38ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [38ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [77ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1244709754(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@54ab028f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1741146898 after retry RPC@114121652) -> (continuation of Deferred@983530160 after retry RPC@282212817), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7492af39 -> wakeup thread Time-limited test -> (continuation of Deferred@1741146898 after retry RPC@114121652) -> (continuation of Deferred@983530160 after retry RPC@282212817))) after lookup
02:12:13.547 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:13.548 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:13.548 [DEBUG - kudu-nio-3] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:13.548 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:13.684 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:13.685 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:12:13.685 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=217), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [2ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [12ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [16ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [17ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [19ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [19ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [36ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [37ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [38ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [38ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [77ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [78ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [80ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [80ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [212ms] refreshing cache from master, [213ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [216ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1244709754(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@54ab028f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1741146898 after retry RPC@114121652) -> (continuation of Deferred@983530160 after retry RPC@282212817) -> (continuation of Deferred@1516138553 after retry RPC@1531168274), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7492af39 -> wakeup thread Time-limited test -> (continuation of Deferred@1741146898 after retry RPC@114121652) -> (continuation of Deferred@983530160 after retry RPC@282212817) -> (continuation of Deferred@1516138553 after retry RPC@1531168274))) after lookup
02:12:13.686 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:13.687 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:13.687 [DEBUG - kudu-nio-4] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:13.687 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:14.085 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:14.085 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:12:14.086 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=617), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [2ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [12ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [16ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [17ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [19ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [19ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [36ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [37ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [38ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [38ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [77ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [78ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [80ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [80ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [212ms] refreshing cache from master, [213ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [216ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [217ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [219ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [219ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [613ms] refreshing cache from master, [613ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [617ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1244709754(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@54ab028f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1741146898 after retry RPC@114121652) -> (continuation of Deferred@983530160 after retry RPC@282212817) -> (continuation of Deferred@1516138553 after retry RPC@1531168274) -> (continuation of Deferred@676475009 after retry RPC@1843658959), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7492af39 -> wakeup thread Time-limited test -> (continuation of Deferred@1741146898 after retry RPC@114121652) -> (continuation of Deferred@983530160 after retry RPC@282212817) -> (continuation of Deferred@1516138553 after retry RPC@1531168274) -> (continuation of Deferred@676475009 after retry RPC@1843658959))) after lookup
02:12:14.086 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:14.087 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:14.087 [DEBUG - kudu-nio-5] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:14.087 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:14.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.448628  9103 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe0a2e8ea6da4fa8b844e7d99e3cae0c)
02:12:14.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.449016  9103 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:14.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.450558  9103 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689), 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:12:14.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:14.455659  8795 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689): Network error: Client connection negotiation failed: client connection to 127.7.102.131:37689: connect: Connection refused (error 111)
02:12:14.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.462304  9046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "cc9ab047ac31487896edb4faf2fb74b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "372cdb4006a84d17800acd046e7b5a41" is_pre_election: true
02:12:14.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.463109  9046 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc9ab047ac31487896edb4faf2fb74b5 in term 4.
02:12:14.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.464394  8794 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 372cdb4006a84d17800acd046e7b5a41, cc9ab047ac31487896edb4faf2fb74b5; no voters: fe0a2e8ea6da4fa8b844e7d99e3cae0c
02:12:14.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.465197  9103 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 5 FOLLOWER]: Leader pre-election won for term 6
02:12:14.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.465508  9103 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader fe0a2e8ea6da4fa8b844e7d99e3cae0c)
02:12:14.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.465734  9103 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 5 FOLLOWER]: Advancing to term 6
02:12:14.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.469364  9093 heartbeater.cc:499] Master 127.7.102.189:33589 was elected leader, sending a full tablet report...
02:12:14.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.470897  9103 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:14.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.472837  9103 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 6 election: Requested vote from peers fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689), 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:12:14.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.473866  9046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "cc9ab047ac31487896edb4faf2fb74b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 20 } } ignore_live_leader: false dest_uuid: "372cdb4006a84d17800acd046e7b5a41"
02:12:14.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.474455  9046 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 4 FOLLOWER]: Advancing to term 6
02:12:14.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:14.476671  8795 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689): Network error: Client connection negotiation failed: client connection to 127.7.102.131:37689: connect: Connection refused (error 111)
02:12:14.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.481592  9046 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc9ab047ac31487896edb4faf2fb74b5 in term 6.
02:12:14.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.482589  8794 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 372cdb4006a84d17800acd046e7b5a41, cc9ab047ac31487896edb4faf2fb74b5; no voters: fe0a2e8ea6da4fa8b844e7d99e3cae0c
02:12:14.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.483484  9103 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 6 FOLLOWER]: Leader election won for term 6
02:12:14.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.484586  9103 raft_consensus.cc:697] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 6 LEADER]: Becoming Leader. State: Replica: cc9ab047ac31487896edb4faf2fb74b5, State: Running, Role: LEADER
02:12:14.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.485357  9103 consensus_queue.cc:237] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:14.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.494479  7696 catalog_manager.cc:5696] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 reported cstate change: term changed from 5 to 6, leader changed from fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131) to cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129). New cstate: current_term: 6 leader_uuid: "cc9ab047ac31487896edb4faf2fb74b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } health_report { overall_health: UNKNOWN } } }
02:12:14.585 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:14.585 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)[L],fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:14.586 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=1117), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [2ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [12ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [16ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [17ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [19ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [19ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [36ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [37ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [38ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [38ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [77ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [78ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [80ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [80ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [212ms] refreshing cache from master, [213ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [216ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [217ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [219ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [219ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [613ms] refreshing cache from master, [613ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [617ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [618ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [619ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [619ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [1112ms] refreshing cache from master, [1113ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [1117ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1244709754(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@54ab028f -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1741146898 after retry RPC@114121652) -> (continuation of Deferred@983530160 after retry RPC@282212817) -> (continuation of Deferred@1516138553 after retry RPC@1531168274) -> (continuation of Deferred@676475009 after retry RPC@1843658959) -> (continuation of Deferred@587466556 after retry RPC@1856739819), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7492af39 -> wakeup thread Time-limited test -> (continuation of Deferred@1741146898 after retry RPC@114121652) -> (continuation of Deferred@983530160 after retry RPC@282212817) -> (continuation of Deferred@1516138553 after retry RPC@1531168274) -> (continuation of Deferred@676475009 after retry RPC@1843658959) -> (continuation of Deferred@587466556 after retry RPC@1856739819))) after lookup
02:12:14.586 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] connecting to peer
02:12:14.587 [DEBUG - kudu-nio-6] (Connection.java:757) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] Successfully connected to peer
02:12:14.615 [DEBUG - kudu-nio-6] (Negotiator.java:919) Authenticated connection [id: 0x55e3838a, L:/127.0.0.1:46996 - R:/127.7.102.129:42373] using TOKEN/null
02:12:14.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.626150  9046 raft_consensus.cc:1275] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 6 FOLLOWER]: Refusing update from remote peer cc9ab047ac31487896edb4faf2fb74b5: Log matching property violated. Preceding OpId in replica: term: 4 index: 16. Preceding OpId from leader: term: 6 index: 22. (index mismatch)
02:12:14.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.627910  9108 consensus_queue.cc:1048] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.001s
02:12:14.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:14.628695  8795 consensus_peers.cc:597] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 -> Peer fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689): Couldn't send request to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c. Status: Network error: Client connection negotiation failed: client connection to 127.7.102.131:37689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:12:14.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.646271  9124 mvcc.cc:204] Tried to move back new op lower bound from 7305120690062790656 to 7305120687896854528. Current Snapshot: MvccSnapshot[applied={T|T < 7305120690062790656}]
02:12:14.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.655946  9122 mvcc.cc:204] Tried to move back new op lower bound from 7305120705008971776 to 7305120704460263424. Current Snapshot: MvccSnapshot[applied={T|T < 7305120705008971776}]
02:12:14.709 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.7.102.131:37689
02:12:14.711 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:14.712690  7578 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/logs
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data/info.pb
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.131:37689
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.131
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38903
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.131
02:12:14.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:12:14.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:14.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:14.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:12:15.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.024098  9131 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:12:15.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.024601  9131 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:12:15.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.025063  9131 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:12:15.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.058624  9131 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:12:15.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.059470  9131 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.102.131
02:12:15.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.098448  9131 tablet_server_runner.cc:78] Tablet server non-default flags:
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.131:37689
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data/info.pb
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.131
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38903
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.131
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/logs
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on e7f111948823
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16850
02:12:15.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:12:15.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.099740  9131 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:12:15.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:15.101372  9131 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:12:15.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.113657  9137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:15.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.155771  7809 debug-util.cc:398] Leaking SignalData structure 0x7b0800070c20 after lost signal to thread 7740
02:12:15.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.156816  7809 debug-util.cc:398] Leaking SignalData structure 0x7b0800070100 after lost signal to thread 7812
02:12:15.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.566577  8795 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.102.131:37689: connect: Connection refused (error 111) [suppressed 4 similar messages]
02:12:16.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.119306  9140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:16.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:15.115753  9138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:16.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:16.695041  9139 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1575 milliseconds
02:12:16.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.695211  9131 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:12:16.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.696640  9131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:12:16.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.699640  9131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:12:16.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.701136  9131 hybrid_clock.cc:648] HybridClock initialized: now 1783476736701031 us; error 116 us; skew 500 ppm
02:12:16.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.702108  9131 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:12:16.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.713199  9131 webserver.cc:534] Webserver started at http://127.7.102.131:38903/ using document root <none> and password file <none>
02:12:16.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.714445  9131 fs_manager.cc:362] Metadata directory not provided
02:12:16.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.714722  9131 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:12:16.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.724494  9131 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.001s	sys 0.003s
02:12:16.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.730628  9149 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:12:16.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.732214  9131 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.001s
02:12:16.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.732684  9131 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data,/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:12:16.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c"
02:12:16.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:52 on dist-test-slave-5s6b"
02:12:16.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.735951  9131 fs_report.cc:389] FS layout report
02:12:16.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:12:16.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:12:16.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:12:16.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data/data
02:12:16.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:12:16.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:12:16.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:12:16.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:12:16.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:12:16.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:12:16.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:12:16.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:12:16.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:12:16.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:12:16.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:12:16.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.832079  9131 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:12:16.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.833683  9131 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:12:16.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.834172  9131 kserver.cc:163] Server-wide thread pool size limit: 3276
02:12:16.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.837451  9131 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:12:16.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.844683  9156 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:12:16.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.853327  9131 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:12:16.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.853561  9131 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.002s	sys 0.000s
02:12:16.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.853775  9131 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:12:16.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.858520  9131 ts_tablet_manager.cc:616] Registered 1 tablets
02:12:16.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.858781  9131 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.005s	sys 0.000s
02:12:16.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.859197  9156 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Bootstrap starting.
02:12:16.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:16.945730  9156 log.cc:826] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Log is configured to *not* fsync() on all Append() calls
02:12:17.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.053426  9156 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:12:17.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.054746  9156 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Bootstrap complete.
02:12:17.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.057266  9156 ts_tablet_manager.cc:1403] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Time spent bootstrapping tablet: real 0.199s	user 0.160s	sys 0.033s
02:12:17.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.089046  9156 raft_consensus.cc:359] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:17.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.093171  9156 raft_consensus.cc:740] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe0a2e8ea6da4fa8b844e7d99e3cae0c, State: Initialized, Role: FOLLOWER
02:12:17.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.094962  9156 consensus_queue.cc:260] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 5.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:17.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.100180  9156 ts_tablet_manager.cc:1434] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Time spent starting tablet: real 0.042s	user 0.044s	sys 0.000s
02:12:17.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.166241  9131 rpc_server.cc:307] RPC server started. Bound to: 127.7.102.131:37689
02:12:17.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.167351  9265 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.102.131:37689 every 8 connection(s)
02:12:17.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.169564  9131 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data/info.pb
02:12:17.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.177695  7578 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu as pid 9131
02:12:17.180 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:17.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.218966  9219 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 5 FOLLOWER]: Advancing to term 6
02:12:17.224 [DEBUG - kudu-nio-6] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726, tablet=064f4af2277f4b52af05ed12eaafba04, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:12:17.225 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.7.102.129:42373
02:12:17.225 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:12:17.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.226909  7578 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu with pid 8770
02:12:17.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.243037  9266 heartbeater.cc:344] Connected to a master server at 127.7.102.188:43935
02:12:17.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.243559  9266 heartbeater.cc:461] Registering TS with master...
02:12:17.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.244942  9266 heartbeater.cc:507] Master 127.7.102.188:43935 requested a full tablet report, sending...
02:12:17.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.254901  9269 heartbeater.cc:344] Connected to a master server at 127.7.102.189:33589
02:12:17.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.255306  9269 heartbeater.cc:461] Registering TS with master...
02:12:17.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.256400  9269 heartbeater.cc:507] Master 127.7.102.189:33589 requested a full tablet report, sending...
02:12:17.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.289170  9275 heartbeater.cc:344] Connected to a master server at 127.7.102.190:37199
02:12:17.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.291224  9275 heartbeater.cc:461] Registering TS with master...
02:12:17.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.292313  9275 heartbeater.cc:507] Master 127.7.102.190:37199 requested a full tablet report, sending...
02:12:17.297 [DEBUG - kudu-nio-6] (Connection.java:239) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] handling channelInactive
02:12:17.297 [DEBUG - kudu-nio-6] (Connection.java:709) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] cleaning up while in state READY due to: connection closed
02:12:17.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:17.306530  9150 connection.cc:466] server connection from 127.7.102.129:41915 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.7.102.129:41915 (request call id 0) could send its response
02:12:17.308 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:17.309 [DEBUG - Time-limited test] (Connection.java:748) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] connecting to peer
02:12:17.310 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:17.310 [DEBUG - kudu-nio-7] (Connection.java:709) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] cleaning up while in state CONNECTING due to: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:17.310 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:17.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.315798  7777 ts_manager.cc:194] Re-registered known tserver with Master: fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689)
02:12:17.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.315929  7623 ts_manager.cc:194] Re-registered known tserver with Master: fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689)
02:12:17.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.316697  7697 ts_manager.cc:194] Re-registered known tserver with Master: fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689)
02:12:17.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.320276  7697 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.102.131:52783
02:12:17.324 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:17.325 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)[L],fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:17.325 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=16), Traces: [0ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [2ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [3ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [16ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1033122454(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@fc2900e -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@67c00c81 -> wakeup thread Time-limited test)) after lookup
02:12:17.327 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] connecting to peer
02:12:17.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:17.327605  9269 heartbeater.cc:499] Master 127.7.102.189:33589 was elected leader, sending a full tablet report...
02:12:17.329 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:17.329 [DEBUG - kudu-nio-0] (Connection.java:709) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] cleaning up while in state CONNECTING due to: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:17.329 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:17.346 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:17.346 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)[L],fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:17.346 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=38), Traces: [0ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [2ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [3ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [16ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [17ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [21ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [21ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [37ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1033122454(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@fc2900e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@163240615 after retry RPC@1861843733), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@67c00c81 -> wakeup thread Time-limited test -> (continuation of Deferred@163240615 after retry RPC@1861843733))) after lookup
02:12:17.347 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] connecting to peer
02:12:17.348 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:17.348 [DEBUG - kudu-nio-1] (Connection.java:709) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] cleaning up while in state CONNECTING due to: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:17.349 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:17.388 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:17.389 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)[L],fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:17.389 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=81), Traces: [0ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [2ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [3ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [16ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [17ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [21ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [21ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [37ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [39ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [40ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [41ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [72ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [80ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1033122454(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@fc2900e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@163240615 after retry RPC@1861843733) -> (continuation of Deferred@1228120888 after retry RPC@1815566552), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@67c00c81 -> wakeup thread Time-limited test -> (continuation of Deferred@163240615 after retry RPC@1861843733) -> (continuation of Deferred@1228120888 after retry RPC@1815566552))) after lookup
02:12:17.391 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] connecting to peer
02:12:17.394 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:17.394 [DEBUG - kudu-nio-2] (Connection.java:709) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] cleaning up while in state CONNECTING due to: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:17.396 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:17.506 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:17.507 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)[L],fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:17.507 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=198), Traces: [0ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [2ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [3ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [16ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [17ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [21ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [21ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [37ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [39ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [40ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [41ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [72ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [80ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [83ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [88ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [88ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [193ms] refreshing cache from master, [193ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [198ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1033122454(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@fc2900e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@163240615 after retry RPC@1861843733) -> (continuation of Deferred@1228120888 after retry RPC@1815566552) -> (continuation of Deferred@1353149773 after retry RPC@1981721395), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@67c00c81 -> wakeup thread Time-limited test -> (continuation of Deferred@163240615 after retry RPC@1861843733) -> (continuation of Deferred@1228120888 after retry RPC@1815566552) -> (continuation of Deferred@1353149773 after retry RPC@1981721395))) after lookup
02:12:17.508 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] connecting to peer
02:12:17.509 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:17.509 [DEBUG - kudu-nio-3] (Connection.java:709) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] cleaning up while in state CONNECTING due to: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:17.509 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:17.985 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:17.985 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)[L],fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:17.985 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=677), Traces: [0ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [2ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [3ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [16ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [17ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [21ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [21ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [37ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [39ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [40ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [41ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [72ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [80ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [83ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [88ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [88ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [193ms] refreshing cache from master, [193ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [198ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [199ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [201ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [201ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [673ms] refreshing cache from master, [673ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [676ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1033122454(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@fc2900e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@163240615 after retry RPC@1861843733) -> (continuation of Deferred@1228120888 after retry RPC@1815566552) -> (continuation of Deferred@1353149773 after retry RPC@1981721395) -> (continuation of Deferred@1893643869 after retry RPC@1931136070), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@67c00c81 -> wakeup thread Time-limited test -> (continuation of Deferred@163240615 after retry RPC@1861843733) -> (continuation of Deferred@1228120888 after retry RPC@1815566552) -> (continuation of Deferred@1353149773 after retry RPC@1981721395) -> (continuation of Deferred@1893643869 after retry RPC@1931136070))) after lookup
02:12:17.986 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] connecting to peer
02:12:17.987 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:17.987 [DEBUG - kudu-nio-4] (Connection.java:709) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] cleaning up while in state CONNECTING due to: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:17.988 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:18.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.495332  9283 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cc9ab047ac31487896edb4faf2fb74b5)
02:12:18.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.495772  9283 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:18.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.498072  9283 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689), cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373)
02:12:18.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:18.501718  8977 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.102.129:42373: connect: Connection refused (error 111)
02:12:18.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:18.505191  8977 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373): Network error: Client connection negotiation failed: client connection to 127.7.102.129:42373: connect: Connection refused (error 111)
02:12:18.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.508802  9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "372cdb4006a84d17800acd046e7b5a41" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" is_pre_election: true
02:12:18.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.510457  8978 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 372cdb4006a84d17800acd046e7b5a41; no voters: cc9ab047ac31487896edb4faf2fb74b5, fe0a2e8ea6da4fa8b844e7d99e3cae0c
02:12:18.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.511242  9283 raft_consensus.cc:2749] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
02:12:18.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.744035  9289 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cc9ab047ac31487896edb4faf2fb74b5)
02:12:18.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.744517  9289 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:18.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.746982  9289 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373), 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:12:18.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:18.751689  9150 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.102.129:42373: connect: Connection refused (error 111)
02:12:18.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:18.756883  9150 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373): Network error: Client connection negotiation failed: client connection to 127.7.102.129:42373: connect: Connection refused (error 111)
02:12:18.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.761188  9046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "372cdb4006a84d17800acd046e7b5a41" is_pre_election: true
02:12:18.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.761816  9046 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe0a2e8ea6da4fa8b844e7d99e3cae0c in term 6.
02:12:18.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.762979  9150 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 372cdb4006a84d17800acd046e7b5a41, fe0a2e8ea6da4fa8b844e7d99e3cae0c; no voters: cc9ab047ac31487896edb4faf2fb74b5
02:12:18.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.763768  9289 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 6 FOLLOWER]: Leader pre-election won for term 7
02:12:18.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.764077  9289 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 6 FOLLOWER]: Starting leader election (detected failure of leader cc9ab047ac31487896edb4faf2fb74b5)
02:12:18.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.764305  9289 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 6 FOLLOWER]: Advancing to term 7
02:12:18.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.768895  9289 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:18.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.770501  9289 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 7 election: Requested vote from peers cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373), 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:12:18.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.772327  9046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "372cdb4006a84d17800acd046e7b5a41"
02:12:18.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.772816  9046 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 6 FOLLOWER]: Advancing to term 7
02:12:18.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:18.774425  9150 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373): Network error: Client connection negotiation failed: client connection to 127.7.102.129:42373: connect: Connection refused (error 111)
02:12:18.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.777822  9046 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe0a2e8ea6da4fa8b844e7d99e3cae0c in term 7.
02:12:18.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.778873  9150 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 372cdb4006a84d17800acd046e7b5a41, fe0a2e8ea6da4fa8b844e7d99e3cae0c; no voters: cc9ab047ac31487896edb4faf2fb74b5
02:12:18.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.779563  9289 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 7 FOLLOWER]: Leader election won for term 7
02:12:18.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.781163  9289 raft_consensus.cc:697] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 7 LEADER]: Becoming Leader. State: Replica: fe0a2e8ea6da4fa8b844e7d99e3cae0c, State: Running, Role: LEADER
02:12:18.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.782105  9289 consensus_queue.cc:237] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:18.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:18.791528  7697 catalog_manager.cc:5696] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c reported cstate change: term changed from 6 to 7, leader changed from cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129) to fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131). New cstate: current_term: 7 leader_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } health_report { overall_health: UNKNOWN } } }
02:12:19.185 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:19.185 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:12:19.186 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=1877), Traces: [0ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [2ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [3ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [16ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [17ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [21ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [21ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [37ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [39ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [40ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [41ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [72ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [80ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [83ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [88ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [88ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [193ms] refreshing cache from master, [193ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [198ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [199ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [201ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [201ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [673ms] refreshing cache from master, [673ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [676ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [678ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [680ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [680ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [1872ms] refreshing cache from master, [1872ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [1876ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@1033122454(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@fc2900e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@163240615 after retry RPC@1861843733) -> (continuation of Deferred@1228120888 after retry RPC@1815566552) -> (continuation of Deferred@1353149773 after retry RPC@1981721395) -> (continuation of Deferred@1893643869 after retry RPC@1931136070) -> (continuation of Deferred@1706477806 after retry RPC@1089163490), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@67c00c81 -> wakeup thread Time-limited test -> (continuation of Deferred@163240615 after retry RPC@1861843733) -> (continuation of Deferred@1228120888 after retry RPC@1815566552) -> (continuation of Deferred@1353149773 after retry RPC@1981721395) -> (continuation of Deferred@1893643869 after retry RPC@1931136070) -> (continuation of Deferred@1706477806 after retry RPC@1089163490))) after lookup
02:12:19.187 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:19.188 [DEBUG - kudu-nio-5] (Connection.java:757) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] Successfully connected to peer
02:12:19.216 [DEBUG - kudu-nio-5] (Negotiator.java:919) Authenticated connection [id: 0xe83d16bd, L:/127.0.0.1:41720 - R:/127.7.102.131:37689] using TOKEN/null
02:12:19.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.225564  9046 raft_consensus.cc:1275] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 7 FOLLOWER]: Refusing update from remote peer fe0a2e8ea6da4fa8b844e7d99e3cae0c: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 7 index: 26. (index mismatch)
02:12:19.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.227109  9289 consensus_queue.cc:1048] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
02:12:19.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:19.228266  9150 consensus_peers.cc:597] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c -> Peer cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373): Couldn't send request to peer cc9ab047ac31487896edb4faf2fb74b5. Status: Network error: Client connection negotiation failed: client connection to 127.7.102.129:42373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:12:19.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.237500  9308 mvcc.cc:204] Tried to move back new op lower bound from 7305120723852959744 to 7305120722060107776. Current Snapshot: MvccSnapshot[applied={T|T < 7305120723852959744}]
02:12:19.275 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.7.102.129:42373
02:12:19.276 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:12:19.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.277343  7578 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:12:19.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:12:19.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal
02:12:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data
02:12:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:12:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:12:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:12:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/logs
02:12:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data/info.pb
02:12:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:12:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:12:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:12:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:12:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:12:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:12:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.129:42373
02:12:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.129
02:12:19.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:12:19.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37707
02:12:19.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.129
02:12:19.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:12:19.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:19.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:19.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:12:19.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:19.607555  9315 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:12:19.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:19.608075  9315 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:12:19.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:19.608587  9315 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:12:19.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:19.643978  9315 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:12:19.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:19.644924  9315 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.102.129
02:12:19.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.681118  9315 tablet_server_runner.cc:78] Tablet server non-default flags:
02:12:19.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:19.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:12:19.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:19.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:19.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data
02:12:19.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal
02:12:19.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.129:42373
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data/info.pb
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.129
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37707
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.129
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/logs
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on e7f111948823
02:12:19.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16850
02:12:19.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:12:19.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.682487  9315 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:12:19.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:19.684051  9315 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:12:19.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:19.696264  9322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:21.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:21.101019  9321 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 9315
02:12:21.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:21.127410  9315 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.430s	user 0.469s	sys 0.953s
02:12:21.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:21.127867  9315 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.430s	user 0.469s	sys 0.953s
02:12:21.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:19.697907  9323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:21.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:21.129777  9325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:21.132838  9324 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1429 milliseconds
02:12:21.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.133165  9315 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:12:21.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.134510  9315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:12:21.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.137080  9315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:12:21.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.138470  9315 hybrid_clock.cc:648] HybridClock initialized: now 1783476741138424 us; error 44 us; skew 500 ppm
02:12:21.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.139267  9315 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:12:21.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.145556  9315 webserver.cc:534] Webserver started at http://127.7.102.129:37707/ using document root <none> and password file <none>
02:12:21.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.146637  9315 fs_manager.cc:362] Metadata directory not provided
02:12:21.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.146898  9315 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:12:21.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.154929  9315 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.000s	sys 0.007s
02:12:21.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.159883  9339 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:12:21.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.161136  9315 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.002s
02:12:21.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.161540  9315 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data,/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal
02:12:21.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cc9ab047ac31487896edb4faf2fb74b5"
02:12:21.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:48 on dist-test-slave-5s6b"
02:12:21.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.163587  9315 fs_report.cc:389] FS layout report
02:12:21.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:12:21.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal
02:12:21.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal
02:12:21.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data/data
02:12:21.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:12:21.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:12:21.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:12:21.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:12:21.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:12:21.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:12:21.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:12:21.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:12:21.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:12:21.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:12:21.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:12:21.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.225006  9315 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:12:21.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.226751  9315 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:12:21.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.227291  9315 kserver.cc:163] Server-wide thread pool size limit: 3276
02:12:21.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.229997  9315 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:12:21.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.236156  9346 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:12:21.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.244453  9315 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:12:21.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.244742  9315 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.002s	sys 0.000s
02:12:21.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.245077  9315 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:12:21.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.249861  9315 ts_tablet_manager.cc:616] Registered 1 tablets
02:12:21.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.250095  9315 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.000s	sys 0.002s
02:12:21.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.250499  9346 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Bootstrap starting.
02:12:21.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.323606  9346 log.cc:826] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Log is configured to *not* fsync() on all Append() calls
02:12:21.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.439451  9346 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:12:21.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.441381  9346 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Bootstrap complete.
02:12:21.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.444509  9346 ts_tablet_manager.cc:1403] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Time spent bootstrapping tablet: real 0.194s	user 0.149s	sys 0.036s
02:12:21.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.481326  9346 raft_consensus.cc:359] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:21.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.488562  9346 raft_consensus.cc:740] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9ab047ac31487896edb4faf2fb74b5, State: Initialized, Role: FOLLOWER
02:12:21.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.490259  9346 consensus_queue.cc:260] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:21.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.496338  9346 ts_tablet_manager.cc:1434] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Time spent starting tablet: real 0.049s	user 0.038s	sys 0.004s
02:12:21.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.510941  9315 rpc_server.cc:307] RPC server started. Bound to: 127.7.102.129:42373
02:12:21.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.511214  9458 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.102.129:42373 every 8 connection(s)
02:12:21.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.513513  9315 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data/info.pb
02:12:21.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.521883  7578 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu as pid 9315
02:12:21.523 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:21.541 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726, tablet=064f4af2277f4b52af05ed12eaafba04, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:12:21.542 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.7.102.131:37689
02:12:21.542 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:12:21.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.543177  7578 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu with pid 9131
02:12:21.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.549266  9459 heartbeater.cc:344] Connected to a master server at 127.7.102.188:43935
02:12:21.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.549767  9459 heartbeater.cc:461] Registering TS with master...
02:12:21.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.551192  9459 heartbeater.cc:507] Master 127.7.102.188:43935 requested a full tablet report, sending...
02:12:21.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.556730  7777 ts_manager.cc:194] Re-registered known tserver with Master: cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373)
02:12:21.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.560302  9461 heartbeater.cc:344] Connected to a master server at 127.7.102.190:37199
02:12:21.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.560678  9461 heartbeater.cc:461] Registering TS with master...
02:12:21.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.561668  9461 heartbeater.cc:507] Master 127.7.102.190:37199 requested a full tablet report, sending...
02:12:21.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.565903  7622 ts_manager.cc:194] Re-registered known tserver with Master: cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373)
02:12:21.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.569068  9460 heartbeater.cc:344] Connected to a master server at 127.7.102.189:33589
02:12:21.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.569324  9460 heartbeater.cc:461] Registering TS with master...
02:12:21.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.570082  9460 heartbeater.cc:507] Master 127.7.102.189:33589 requested a full tablet report, sending...
02:12:21.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.572780  7697 ts_manager.cc:194] Re-registered known tserver with Master: cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373)
02:12:21.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:21.575052  7697 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.102.129:33367
02:12:21.586 [DEBUG - kudu-nio-5] (Connection.java:239) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] handling channelInactive
02:12:21.587 [DEBUG - kudu-nio-5] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state READY due to: connection closed
02:12:21.587 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:21.588 [DEBUG - Time-limited test] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:21.589 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:21.589 [DEBUG - kudu-nio-6] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:21.589 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:21.605 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:21.605 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:12:21.605 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=18), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [2ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [17ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@382538371(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@c3b2cf8 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@158abbda -> wakeup thread Time-limited test)) after lookup
02:12:21.606 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:21.607 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:21.607 [DEBUG - kudu-nio-7] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:21.607 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:21.624 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:21.624 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:12:21.625 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=37), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [2ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [17ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [18ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [20ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [20ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [37ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@382538371(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@c3b2cf8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@24093401 after retry RPC@1107008837), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@158abbda -> wakeup thread Time-limited test -> (continuation of Deferred@24093401 after retry RPC@1107008837))) after lookup
02:12:21.625 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:21.626 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:21.626 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:21.626 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:21.665 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:21.665 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:12:21.666 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=78), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [2ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [17ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [18ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [20ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [20ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [37ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [38ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [39ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [39ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [78ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@382538371(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@c3b2cf8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@24093401 after retry RPC@1107008837) -> (continuation of Deferred@2032911838 after retry RPC@1680305675), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@158abbda -> wakeup thread Time-limited test -> (continuation of Deferred@24093401 after retry RPC@1107008837) -> (continuation of Deferred@2032911838 after retry RPC@1680305675))) after lookup
02:12:21.666 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:21.667 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:21.667 [DEBUG - kudu-nio-1] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:21.667 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:21.804 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:21.804 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:12:21.804 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=217), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [2ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [17ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [18ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [20ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [20ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [37ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [38ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [39ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [39ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [78ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [79ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [80ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [80ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [213ms] refreshing cache from master, [213ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [216ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@382538371(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@c3b2cf8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@24093401 after retry RPC@1107008837) -> (continuation of Deferred@2032911838 after retry RPC@1680305675) -> (continuation of Deferred@644285245 after retry RPC@2037348044), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@158abbda -> wakeup thread Time-limited test -> (continuation of Deferred@24093401 after retry RPC@1107008837) -> (continuation of Deferred@2032911838 after retry RPC@1680305675) -> (continuation of Deferred@644285245 after retry RPC@2037348044))) after lookup
02:12:21.805 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:21.806 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:21.806 [DEBUG - kudu-nio-2] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:21.806 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:22.144 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:22.144 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:12:22.145 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=557), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [2ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [17ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [18ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [20ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [20ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [37ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [38ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [39ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [39ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [78ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [79ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [80ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [80ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [213ms] refreshing cache from master, [213ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [216ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [218ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [219ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [219ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [554ms] refreshing cache from master, [554ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [557ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@382538371(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@c3b2cf8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@24093401 after retry RPC@1107008837) -> (continuation of Deferred@2032911838 after retry RPC@1680305675) -> (continuation of Deferred@644285245 after retry RPC@2037348044) -> (continuation of Deferred@1522626511 after retry RPC@1919543779), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@158abbda -> wakeup thread Time-limited test -> (continuation of Deferred@24093401 after retry RPC@1107008837) -> (continuation of Deferred@2032911838 after retry RPC@1680305675) -> (continuation of Deferred@644285245 after retry RPC@2037348044) -> (continuation of Deferred@1522626511 after retry RPC@1919543779))) after lookup
02:12:22.145 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:22.146 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:22.146 [DEBUG - kudu-nio-3] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:22.147 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:22.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.471105  9469 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe0a2e8ea6da4fa8b844e7d99e3cae0c)
02:12:22.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.471881  9469 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:22.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.474325  9469 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689), cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373)
02:12:22.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:22.484930  8978 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689): Network error: Client connection negotiation failed: client connection to 127.7.102.131:37689: connect: Connection refused (error 111)
02:12:22.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.490820  9410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "372cdb4006a84d17800acd046e7b5a41" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "cc9ab047ac31487896edb4faf2fb74b5" is_pre_election: true
02:12:22.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.491500  9410 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372cdb4006a84d17800acd046e7b5a41 in term 6.
02:12:22.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.492903  8977 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 372cdb4006a84d17800acd046e7b5a41, cc9ab047ac31487896edb4faf2fb74b5; no voters: fe0a2e8ea6da4fa8b844e7d99e3cae0c
02:12:22.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.493575  9469 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 7 FOLLOWER]: Leader pre-election won for term 8
02:12:22.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.493947  9469 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 7 FOLLOWER]: Starting leader election (detected failure of leader fe0a2e8ea6da4fa8b844e7d99e3cae0c)
02:12:22.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.494261  9469 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 7 FOLLOWER]: Advancing to term 8
02:12:22.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.499356  9469 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:22.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.501296  9469 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 8 election: Requested vote from peers fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689), cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373)
02:12:22.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.502629  9410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "372cdb4006a84d17800acd046e7b5a41" candidate_term: 8 candidate_status { last_received { term: 7 index: 28 } } ignore_live_leader: false dest_uuid: "cc9ab047ac31487896edb4faf2fb74b5"
02:12:22.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.503244  9410 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 6 FOLLOWER]: Advancing to term 8
02:12:22.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:22.505631  8978 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689): Network error: Client connection negotiation failed: client connection to 127.7.102.131:37689: connect: Connection refused (error 111)
02:12:22.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.510491  9410 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372cdb4006a84d17800acd046e7b5a41 in term 8.
02:12:22.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.511724  8977 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 372cdb4006a84d17800acd046e7b5a41, cc9ab047ac31487896edb4faf2fb74b5; no voters: fe0a2e8ea6da4fa8b844e7d99e3cae0c
02:12:22.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.512499  9469 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 8 FOLLOWER]: Leader election won for term 8
02:12:22.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.513914  9469 raft_consensus.cc:697] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 8 LEADER]: Becoming Leader. State: Replica: 372cdb4006a84d17800acd046e7b5a41, State: Running, Role: LEADER
02:12:22.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.515254  9469 consensus_queue.cc:237] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:22.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.526577  7697 catalog_manager.cc:5696] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 reported cstate change: term changed from 7 to 8, leader changed from fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131) to 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130). New cstate: current_term: 8 leader_uuid: "372cdb4006a84d17800acd046e7b5a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } health_report { overall_health: HEALTHY } } }
02:12:22.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.578209  9460 heartbeater.cc:499] Master 127.7.102.189:33589 was elected leader, sending a full tablet report...
02:12:22.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.971230  9410 raft_consensus.cc:1275] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 8 FOLLOWER]: Refusing update from remote peer 372cdb4006a84d17800acd046e7b5a41: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 8 index: 29. (index mismatch)
02:12:22.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:22.973013  9469 consensus_queue.cc:1048] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
02:12:23.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:23.014995  8978 consensus_peers.cc:597] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 -> Peer fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689): Couldn't send request to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c. Status: Network error: Client connection negotiation failed: client connection to 127.7.102.131:37689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:12:23.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:23.518847  8978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.102.131:37689: connect: Connection refused (error 111) [suppressed 3 similar messages]
02:12:23.825 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:23.825 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)[L],cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:23.826 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=2238), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [2ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [13ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [17ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [18ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [20ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [20ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [37ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [38ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [39ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [39ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [74ms] refreshing cache from master, [74ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [78ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [79ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [80ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [80ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [213ms] refreshing cache from master, [213ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [216ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [218ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [219ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [219ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [554ms] refreshing cache from master, [554ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [557ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [558ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [559ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [560ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [2233ms] refreshing cache from master, [2234ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [2237ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@382538371(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@c3b2cf8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@24093401 after retry RPC@1107008837) -> (continuation of Deferred@2032911838 after retry RPC@1680305675) -> (continuation of Deferred@644285245 after retry RPC@2037348044) -> (continuation of Deferred@1522626511 after retry RPC@1919543779) -> (continuation of Deferred@2142430520 after retry RPC@1109994972), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@158abbda -> wakeup thread Time-limited test -> (continuation of Deferred@24093401 after retry RPC@1107008837) -> (continuation of Deferred@2032911838 after retry RPC@1680305675) -> (continuation of Deferred@644285245 after retry RPC@2037348044) -> (continuation of Deferred@1522626511 after retry RPC@1919543779) -> (continuation of Deferred@2142430520 after retry RPC@1109994972))) after lookup
02:12:23.826 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:12:23.831 [DEBUG - kudu-nio-4] (Connection.java:757) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] Successfully connected to peer
02:12:23.858 [DEBUG - kudu-nio-4] (Negotiator.java:919) Authenticated connection [id: 0xc7ee8552, L:/127.0.0.1:42468 - R:/127.7.102.130:41431] using TOKEN/null
02:12:23.904 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.7.102.131:37689
02:12:23.904 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:12:23.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:23.905565  7578 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:12:23.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:12:23.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/logs
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data/info.pb
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.131:37689
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.131
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38903
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.131
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:23.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:12:24.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:24.263628  9506 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:12:24.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:24.264149  9506 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:12:24.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:24.264765  9506 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:12:24.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:24.301765  9506 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:12:24.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:24.302711  9506 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.102.131
02:12:24.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:24.341928  9506 tablet_server_runner.cc:78] Tablet server non-default flags:
02:12:24.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:24.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:12:24.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:24.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:24.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data
02:12:24.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:12:24.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:24.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:24.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.131:37689
02:12:24.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:12:24.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:24.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:24.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data/info.pb
02:12:24.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.131
02:12:24.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38903
02:12:24.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:12:24.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:12:24.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:24.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:12:24.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:12:24.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:24.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.131
02:12:24.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/logs
02:12:24.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:24.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:12:24.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:12:24.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:12:24.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:12:24.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:12:24.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:12:24.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on e7f111948823
02:12:24.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16850
02:12:24.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:12:24.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:24.343482  9506 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:12:24.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:24.345217  9506 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:12:24.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:24.361581  9512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:24.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:24.975185  9455 debug-util.cc:398] Leaking SignalData structure 0x7b08000bf060 after lost signal to thread 9316
02:12:24.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:24.976644  9455 debug-util.cc:398] Leaking SignalData structure 0x7b08000bf7e0 after lost signal to thread 9458
02:12:25.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:25.282219  8978 consensus_peers.cc:597] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 -> Peer fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689): Couldn't send request to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c. Status: Network error: Client connection negotiation failed: client connection to 127.7.102.131:37689: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
02:12:26.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:25.761058  9511 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 9506
02:12:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:26.400650  9511 kernel_stack_watchdog.cc:198] Thread 9506 stuck at /home/jenkins-slave/workspace/build_and_test@2/src/kudu/util/thread.cc:642 for 396ms:
02:12:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
02:12:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
02:12:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
02:12:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
02:12:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:26.400820  9506 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 2.038s	user 0.671s	sys 1.109s
02:12:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:24.361864  9513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:26.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:26.401129  9506 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 2.039s	user 0.671s	sys 1.109s
02:12:26.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.401780  9506 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:12:26.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:26.401933  9516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:26.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:26.401907  9514 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2037 milliseconds
02:12:26.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.405699  9506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:12:26.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.418188  9506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:12:26.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.419687  9506 hybrid_clock.cc:648] HybridClock initialized: now 1783476746419649 us; error 58 us; skew 500 ppm
02:12:26.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.420779  9506 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:12:26.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.428552  9506 webserver.cc:534] Webserver started at http://127.7.102.131:38903/ using document root <none> and password file <none>
02:12:26.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.429706  9506 fs_manager.cc:362] Metadata directory not provided
02:12:26.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.429973  9506 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:12:26.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.441995  9506 fs_manager.cc:714] Time spent opening directory manager: real 0.008s	user 0.008s	sys 0.002s
02:12:26.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.448313  9524 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:12:26.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.449766  9506 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.006s	sys 0.000s
02:12:26.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.450234  9506 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data,/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:12:26.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c"
02:12:26.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:52 on dist-test-slave-5s6b"
02:12:26.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.453385  9506 fs_report.cc:389] FS layout report
02:12:26.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:12:26.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:12:26.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:12:26.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data/data
02:12:26.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:12:26.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:12:26.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:12:26.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:12:26.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:12:26.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:12:26.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:12:26.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:12:26.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:12:26.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:12:26.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:12:26.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.512508  9506 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:12:26.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.514192  9506 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:12:26.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.514791  9506 kserver.cc:163] Server-wide thread pool size limit: 3276
02:12:26.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.518435  9506 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:12:26.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.526247  9531 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:12:26.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.536340  9506 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:12:26.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.536713  9506 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.013s	user 0.002s	sys 0.002s
02:12:26.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.537184  9506 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:12:26.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.543394  9506 ts_tablet_manager.cc:616] Registered 1 tablets
02:12:26.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.543684  9506 ts_tablet_manager.cc:595] Time spent register tablets: real 0.007s	user 0.006s	sys 0.000s
02:12:26.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.544149  9531 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Bootstrap starting.
02:12:26.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.640750  9531 log.cc:826] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Log is configured to *not* fsync() on all Append() calls
02:12:26.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.791273  9531 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:12:26.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.792919  9531 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Bootstrap complete.
02:12:26.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.795470  9531 ts_tablet_manager.cc:1403] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Time spent bootstrapping tablet: real 0.252s	user 0.183s	sys 0.057s
02:12:26.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.828256  9531 raft_consensus.cc:359] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:26.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.832224  9531 raft_consensus.cc:740] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe0a2e8ea6da4fa8b844e7d99e3cae0c, State: Initialized, Role: FOLLOWER
02:12:26.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.833729  9531 consensus_queue.cc:260] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 7.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:26.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.841711  9531 ts_tablet_manager.cc:1434] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Time spent starting tablet: real 0.046s	user 0.037s	sys 0.005s
02:12:26.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.846596  9506 rpc_server.cc:307] RPC server started. Bound to: 127.7.102.131:37689
02:12:26.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.847049  9640 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.102.131:37689 every 8 connection(s)
02:12:26.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.849594  9506 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data/info.pb
02:12:26.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.858431  7578 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu as pid 9506
02:12:26.859 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:26.947 [DEBUG - kudu-nio-4] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726, tablet=064f4af2277f4b52af05ed12eaafba04, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:12:26.952 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.7.102.130:41431
02:12:26.953 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:12:26.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.954435  7578 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu with pid 8953
02:12:26.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.965867  9642 heartbeater.cc:344] Connected to a master server at 127.7.102.189:33589
02:12:26.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.966471  9642 heartbeater.cc:461] Registering TS with master...
02:12:26.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.967861  9642 heartbeater.cc:507] Master 127.7.102.189:33589 requested a full tablet report, sending...
02:12:26.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.969532  9643 heartbeater.cc:344] Connected to a master server at 127.7.102.190:37199
02:12:26.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.969903  9643 heartbeater.cc:461] Registering TS with master...
02:12:26.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.971007  9643 heartbeater.cc:507] Master 127.7.102.190:37199 requested a full tablet report, sending...
02:12:26.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.972539  9641 heartbeater.cc:344] Connected to a master server at 127.7.102.188:43935
02:12:26.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.972918  9641 heartbeater.cc:461] Registering TS with master...
02:12:26.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.973935  9641 heartbeater.cc:507] Master 127.7.102.188:43935 requested a full tablet report, sending...
02:12:26.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.977387  7622 ts_manager.cc:194] Re-registered known tserver with Master: fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689)
02:12:26.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.977833  7777 ts_manager.cc:194] Re-registered known tserver with Master: fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689)
02:12:26.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.979499  7697 ts_manager.cc:194] Re-registered known tserver with Master: fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689)
02:12:26.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:26.982667  7697 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.102.131:51035
02:12:27.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:27.010982  9653 negotiation.cc:336] Failed RPC negotiation. Trace:
02:12:27.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:26.892229 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.7.102.130:40573 (local address 127.7.102.131:37689)
02:12:27.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:26.922184 (+ 29955us) server_negotiation.cc:207] Beginning negotiation
02:12:27.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:26.922238 (+    54us) server_negotiation.cc:400] Waiting for connection header
02:12:27.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:26.930378 (+  8140us) server_negotiation.cc:408] Connection header received
02:12:27.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:26.930694 (+   316us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
02:12:27.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:26.930756 (+    62us) server_negotiation.cc:462] Received NEGOTIATE request from client
02:12:27.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:26.931338 (+   582us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
02:12:27.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:26.931664 (+   326us) server_negotiation.cc:227] Negotiated authn=SASL
02:12:27.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0708 02:12:27.008945 (+ 77281us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.7.102.130:40573: BlockingRecv error: recv got EOF from 127.7.102.130:40573 (error 108)
02:12:27.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":28872,"thread_start_us":28394,"threads_started":1}
02:12:27.013 [DEBUG - kudu-nio-4] (Connection.java:239) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] handling channelInactive
02:12:27.014 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state READY due to: connection closed
02:12:27.014 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:27.015 [DEBUG - Time-limited test] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:12:27.016 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:27.017 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state CONNECTING due to: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:27.017 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:27.026 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:27.027 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)[L],cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:27.027 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=12), Traces: [0ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [2ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [2ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [11ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@431571818(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@12f5da4c -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7446448d -> wakeup thread Time-limited test)) after lookup
02:12:27.028 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:12:27.031 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:27.031 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state CONNECTING due to: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:27.031 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:27.045 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:27.045 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)[L],cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:27.046 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=31), Traces: [0ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [2ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [2ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [11ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [13ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [16ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [16ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [30ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@431571818(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@12f5da4c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@621315850 after retry RPC@1229181925), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7446448d -> wakeup thread Time-limited test -> (continuation of Deferred@621315850 after retry RPC@1229181925))) after lookup
02:12:27.047 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:12:27.048 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:27.048 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state CONNECTING due to: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:27.048 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:27.065 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:27.065 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)[L],cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:27.066 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=50), Traces: [0ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [2ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [2ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [11ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [13ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [16ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [16ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [30ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [31ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [33ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [33ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [50ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@431571818(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@12f5da4c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@621315850 after retry RPC@1229181925) -> (continuation of Deferred@1003108673 after retry RPC@1824594214), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7446448d -> wakeup thread Time-limited test -> (continuation of Deferred@621315850 after retry RPC@1229181925) -> (continuation of Deferred@1003108673 after retry RPC@1824594214))) after lookup
02:12:27.067 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:12:27.068 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:27.068 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state CONNECTING due to: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:27.068 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:27.165 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:27.165 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)[L],cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:27.166 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=151), Traces: [0ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [2ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [2ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [11ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [13ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [16ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [16ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [30ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [31ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [33ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [33ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [50ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [51ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [53ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [53ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [145ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [150ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@431571818(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@12f5da4c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@621315850 after retry RPC@1229181925) -> (continuation of Deferred@1003108673 after retry RPC@1824594214) -> (continuation of Deferred@1084501573 after retry RPC@608704350), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7446448d -> wakeup thread Time-limited test -> (continuation of Deferred@621315850 after retry RPC@1229181925) -> (continuation of Deferred@1003108673 after retry RPC@1824594214) -> (continuation of Deferred@1084501573 after retry RPC@608704350))) after lookup
02:12:27.167 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:12:27.168 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:27.168 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state CONNECTING due to: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:27.168 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:27.386 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:27.386 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)[L],cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:27.387 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=371), Traces: [0ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [2ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [2ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [11ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [13ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [16ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [16ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [30ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [31ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [33ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [33ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [50ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [51ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [53ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [53ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [145ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [150ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [151ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [153ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [153ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [365ms] refreshing cache from master, [365ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [370ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@431571818(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@12f5da4c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@621315850 after retry RPC@1229181925) -> (continuation of Deferred@1003108673 after retry RPC@1824594214) -> (continuation of Deferred@1084501573 after retry RPC@608704350) -> (continuation of Deferred@1294692740 after retry RPC@700284312), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7446448d -> wakeup thread Time-limited test -> (continuation of Deferred@621315850 after retry RPC@1229181925) -> (continuation of Deferred@1003108673 after retry RPC@1824594214) -> (continuation of Deferred@1084501573 after retry RPC@608704350) -> (continuation of Deferred@1294692740 after retry RPC@700284312))) after lookup
02:12:27.388 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:12:27.390 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:27.390 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state CONNECTING due to: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:27.390 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431
02:12:27.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.986831  9642 heartbeater.cc:499] Master 127.7.102.189:33589 was elected leader, sending a full tablet report...
02:12:27.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.989859  9655 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 372cdb4006a84d17800acd046e7b5a41)
02:12:27.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:27.992707  9655 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:28.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.013286  9655 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689), 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:12:28.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:28.020953  9340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.102.130:41431: connect: Connection refused (error 111)
02:12:28.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:28.026453  9340 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431): Network error: Client connection negotiation failed: client connection to 127.7.102.130:41431: connect: Connection refused (error 111)
02:12:28.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.027943  9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "cc9ab047ac31487896edb4faf2fb74b5" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" is_pre_election: true
02:12:28.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.028870  9595 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc9ab047ac31487896edb4faf2fb74b5 in term 7.
02:12:28.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.030643  9341 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cc9ab047ac31487896edb4faf2fb74b5, fe0a2e8ea6da4fa8b844e7d99e3cae0c; no voters: 372cdb4006a84d17800acd046e7b5a41
02:12:28.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.031577  9655 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 8 FOLLOWER]: Leader pre-election won for term 9
02:12:28.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.031950  9655 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 372cdb4006a84d17800acd046e7b5a41)
02:12:28.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.032317  9655 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 8 FOLLOWER]: Advancing to term 9
02:12:28.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.039786  9655 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:28.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.043313  9595 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "cc9ab047ac31487896edb4faf2fb74b5" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c"
02:12:28.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.044034  9595 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 7 FOLLOWER]: Advancing to term 9
02:12:28.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:28.048014  9340 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431): Network error: Client connection negotiation failed: client connection to 127.7.102.130:41431: connect: Connection refused (error 111)
02:12:28.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.048919  9655 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 9 election: Requested vote from peers fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689), 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:12:28.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.055261  9595 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc9ab047ac31487896edb4faf2fb74b5 in term 9.
02:12:28.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.056463  9341 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cc9ab047ac31487896edb4faf2fb74b5, fe0a2e8ea6da4fa8b844e7d99e3cae0c; no voters: 372cdb4006a84d17800acd046e7b5a41
02:12:28.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.057308  9655 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 9 FOLLOWER]: Leader election won for term 9
02:12:28.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.059754  9655 raft_consensus.cc:697] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 9 LEADER]: Becoming Leader. State: Replica: cc9ab047ac31487896edb4faf2fb74b5, State: Running, Role: LEADER
02:12:28.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.061040  9655 consensus_queue.cc:237] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:28.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.072122  7697 catalog_manager.cc:5696] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 reported cstate change: term changed from 8 to 9, leader changed from 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130) to cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129). New cstate: current_term: 9 leader_uuid: "cc9ab047ac31487896edb4faf2fb74b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } health_report { overall_health: UNKNOWN } } }
02:12:28.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:28.510052  9340 consensus_peers.cc:597] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 -> Peer 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431): Couldn't send request to peer 372cdb4006a84d17800acd046e7b5a41. Status: Network error: Client connection negotiation failed: client connection to 127.7.102.130:41431: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:12:28.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.535235  9595 raft_consensus.cc:1275] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 9 FOLLOWER]: Refusing update from remote peer cc9ab047ac31487896edb4faf2fb74b5: Log matching property violated. Preceding OpId in replica: term: 7 index: 28. Preceding OpId from leader: term: 9 index: 33. (index mismatch)
02:12:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:28.536625  9655 consensus_queue.cc:1048] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
02:12:29.324 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:29.324 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)[L],fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:29.325 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=2310), Traces: [0ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [2ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [2ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [6ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [11ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [13ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [16ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [16ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [25ms] refreshing cache from master, [26ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [30ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [31ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [33ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [33ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [45ms] refreshing cache from master, [45ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [50ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [51ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [53ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [53ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [145ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [150ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [151ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [153ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [153ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [365ms] refreshing cache from master, [365ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [370ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [372ms] sending RPC to server 372cdb4006a84d17800acd046e7b5a41, [375ms] received response from server 372cdb4006a84d17800acd046e7b5a41: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [375ms] delaying RPC due to: Network error: Failed to connect to peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431): Connection refused: /127.7.102.130:41431, [2306ms] refreshing cache from master, [2306ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [2309ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@431571818(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@12f5da4c -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@621315850 after retry RPC@1229181925) -> (continuation of Deferred@1003108673 after retry RPC@1824594214) -> (continuation of Deferred@1084501573 after retry RPC@608704350) -> (continuation of Deferred@1294692740 after retry RPC@700284312) -> (continuation of Deferred@800316942 after retry RPC@1822638311), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@7446448d -> wakeup thread Time-limited test -> (continuation of Deferred@621315850 after retry RPC@1229181925) -> (continuation of Deferred@1003108673 after retry RPC@1824594214) -> (continuation of Deferred@1084501573 after retry RPC@608704350) -> (continuation of Deferred@1294692740 after retry RPC@700284312) -> (continuation of Deferred@800316942 after retry RPC@1822638311))) after lookup
02:12:29.325 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] connecting to peer
02:12:29.326 [DEBUG - kudu-nio-3] (Connection.java:757) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] Successfully connected to peer
02:12:29.343 [DEBUG - kudu-nio-3] (Negotiator.java:919) Authenticated connection [id: 0x76efefbc, L:/127.0.0.1:37250 - R:/127.7.102.129:42373] using TOKEN/null
02:12:29.392 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.7.102.130:41431
02:12:29.392 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

02:12:29.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.393828  7578 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:12:29.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:12:29.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal
02:12:29.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data
02:12:29.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:12:29.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:12:29.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:12:29.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:29.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:29.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:29.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/logs
02:12:29.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data/info.pb
02:12:29.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:29.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:12:29.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:12:29.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:12:29.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:12:29.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:29.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:29.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:29.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:12:29.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:12:29.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.130:41431
02:12:29.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.130
02:12:29.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:12:29.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41213
02:12:29.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.130
02:12:29.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:12:29.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:29.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:29.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:12:29.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:29.749408  9690 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:12:29.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:29.749908  9690 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:12:29.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:29.750454  9690 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:12:29.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:29.788748  9690 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:12:29.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:29.789590  9690 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.102.130
02:12:29.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.827028  9690 tablet_server_runner.cc:78] Tablet server non-default flags:
02:12:29.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:29.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:12:29.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:29.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:29.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data
02:12:29.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal
02:12:29.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:29.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:29.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.130:41431
02:12:29.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:12:29.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:29.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:29.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data/info.pb
02:12:29.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.130
02:12:29.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41213
02:12:29.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:12:29.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:12:29.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:29.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:12:29.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:12:29.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:29.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.130
02:12:29.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/logs
02:12:29.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:29.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:12:29.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:12:29.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:12:29.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:12:29.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:12:29.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:12:29.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on e7f111948823
02:12:29.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16850
02:12:29.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:12:29.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.828363  9690 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:12:29.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:29.829922  9690 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:12:29.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:29.843870  9698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:30.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:30.972556  9340 consensus_peers.cc:597] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 -> Peer 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431): Couldn't send request to peer 372cdb4006a84d17800acd046e7b5a41. Status: Network error: Client connection negotiation failed: client connection to 127.7.102.130:41431: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
02:12:31.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:29.844589  9699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:31.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:31.400449  9690 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.557s	user 0.471s	sys 1.044s
02:12:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:31.400908  9690 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.558s	user 0.471s	sys 1.044s
02:12:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:31.246174  9697 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 9690
02:12:31.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:31.403049  9701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:31.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:31.406692  9700 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1562 milliseconds
02:12:31.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.406704  9690 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:12:31.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.408022  9690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:12:31.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.410281  9690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:12:31.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.411684  9690 hybrid_clock.cc:648] HybridClock initialized: now 1783476751411635 us; error 59 us; skew 500 ppm
02:12:31.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.412611  9690 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:12:31.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.419180  9690 webserver.cc:534] Webserver started at http://127.7.102.130:41213/ using document root <none> and password file <none>
02:12:31.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.420130  9690 fs_manager.cc:362] Metadata directory not provided
02:12:31.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.420379  9690 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:12:31.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.428387  9690 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.000s	sys 0.004s
02:12:31.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.433249  9710 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:12:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.434507  9690 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.002s
02:12:31.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.434885  9690 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data,/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal
02:12:31.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "372cdb4006a84d17800acd046e7b5a41"
02:12:31.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:50 on dist-test-slave-5s6b"
02:12:31.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.437094  9690 fs_report.cc:389] FS layout report
02:12:31.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:12:31.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal
02:12:31.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/wal
02:12:31.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data/data
02:12:31.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:12:31.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:12:31.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:12:31.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:12:31.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:12:31.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:12:31.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:12:31.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:12:31.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:12:31.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:12:31.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:12:31.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.488881  9690 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:12:31.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.490888  9690 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:12:31.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.491482  9690 kserver.cc:163] Server-wide thread pool size limit: 3276
02:12:31.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.494671  9690 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:12:31.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.501461  9718 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:12:31.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.511256  9690 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:12:31.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.511555  9690 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.012s	user 0.001s	sys 0.001s
02:12:31.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.511793  9690 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:12:31.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.516839  9690 ts_tablet_manager.cc:616] Registered 1 tablets
02:12:31.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.517063  9690 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.003s	sys 0.000s
02:12:31.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.517402  9718 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Bootstrap starting.
02:12:31.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.587402  9718 log.cc:826] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Log is configured to *not* fsync() on all Append() calls
02:12:31.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.697449  9718 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:12:31.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.698416  9718 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Bootstrap complete.
02:12:31.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.699900  9718 ts_tablet_manager.cc:1403] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Time spent bootstrapping tablet: real 0.183s	user 0.151s	sys 0.024s
02:12:31.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.722312  9718 raft_consensus.cc:359] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:31.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.725888  9718 raft_consensus.cc:740] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372cdb4006a84d17800acd046e7b5a41, State: Initialized, Role: FOLLOWER
02:12:31.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.727308  9827 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.102.130:41431 every 8 connection(s)
02:12:31.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.727676  9690 rpc_server.cc:307] RPC server started. Bound to: 127.7.102.130:41431
02:12:31.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.727530  9718 consensus_queue.cc:260] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 8.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:31.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.731655  9718 ts_tablet_manager.cc:1434] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41: Time spent starting tablet: real 0.031s	user 0.027s	sys 0.004s
02:12:31.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.731873  9690 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-1/data/info.pb
02:12:31.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.735328  7578 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu as pid 9690
02:12:31.736 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:31.786 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726, tablet=064f4af2277f4b52af05ed12eaafba04, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:12:31.787 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.7.102.129:42373
02:12:31.788 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:12:31.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.788333  7578 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu with pid 9315
02:12:31.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.789629  9831 heartbeater.cc:344] Connected to a master server at 127.7.102.190:37199
02:12:31.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.789762  9829 heartbeater.cc:344] Connected to a master server at 127.7.102.188:43935
02:12:31.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.790143  9831 heartbeater.cc:461] Registering TS with master...
02:12:31.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.791512  9831 heartbeater.cc:507] Master 127.7.102.190:37199 requested a full tablet report, sending...
02:12:31.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.790161  9829 heartbeater.cc:461] Registering TS with master...
02:12:31.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.793015  9829 heartbeater.cc:507] Master 127.7.102.188:43935 requested a full tablet report, sending...
02:12:31.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.796077  7622 ts_manager.cc:194] Re-registered known tserver with Master: 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:12:31.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.799897  7777 ts_manager.cc:194] Re-registered known tserver with Master: 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:12:31.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.804320  9830 heartbeater.cc:344] Connected to a master server at 127.7.102.189:33589
02:12:31.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.804711  9830 heartbeater.cc:461] Registering TS with master...
02:12:31.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.805712  9830 heartbeater.cc:507] Master 127.7.102.189:33589 requested a full tablet report, sending...
02:12:31.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.809057  7696 ts_manager.cc:194] Re-registered known tserver with Master: 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:12:31.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:31.812354  7696 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.102.130:57773
02:12:31.825 [DEBUG - kudu-nio-3] (Connection.java:239) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] handling channelInactive
02:12:31.826 [DEBUG - kudu-nio-3] (Connection.java:709) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] cleaning up while in state READY due to: connection closed
02:12:31.826 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:31.826 [DEBUG - Time-limited test] (Connection.java:748) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] connecting to peer
02:12:31.827 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:31.827 [DEBUG - kudu-nio-4] (Connection.java:709) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] cleaning up while in state CONNECTING due to: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:31.828 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:31.845 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:31.845 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)[L],fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:31.845 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=19), Traces: [0ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [2ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [2ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [18ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@2402291(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@76766b68 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4878ff38 -> wakeup thread Time-limited test)) after lookup
02:12:31.846 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] connecting to peer
02:12:31.847 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:31.847 [DEBUG - kudu-nio-5] (Connection.java:709) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] cleaning up while in state CONNECTING due to: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:31.847 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:31.864 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:31.864 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)[L],fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:31.865 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=38), Traces: [0ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [2ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [2ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [18ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [19ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [21ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [21ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [38ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@2402291(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@76766b68 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@723022178 after retry RPC@605477382), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4878ff38 -> wakeup thread Time-limited test -> (continuation of Deferred@723022178 after retry RPC@605477382))) after lookup
02:12:31.865 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] connecting to peer
02:12:31.866 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:31.866 [DEBUG - kudu-nio-6] (Connection.java:709) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] cleaning up while in state CONNECTING due to: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:31.866 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:31.883 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:31.883 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)[L],fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:31.884 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=57), Traces: [0ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [2ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [2ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [18ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [19ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [21ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [21ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [38ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [39ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [40ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [40ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [57ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@2402291(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@76766b68 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@723022178 after retry RPC@605477382) -> (continuation of Deferred@418019214 after retry RPC@180246655), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4878ff38 -> wakeup thread Time-limited test -> (continuation of Deferred@723022178 after retry RPC@605477382) -> (continuation of Deferred@418019214 after retry RPC@180246655))) after lookup
02:12:31.884 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] connecting to peer
02:12:31.885 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:31.885 [DEBUG - kudu-nio-7] (Connection.java:709) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] cleaning up while in state CONNECTING due to: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:31.885 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:31.924 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:31.925 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)[L],fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:31.925 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=98), Traces: [0ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [2ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [2ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [18ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [19ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [21ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [21ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [38ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [39ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [40ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [40ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [57ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [58ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [59ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [59ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [95ms] refreshing cache from master, [95ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [98ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@2402291(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@76766b68 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@723022178 after retry RPC@605477382) -> (continuation of Deferred@418019214 after retry RPC@180246655) -> (continuation of Deferred@2036858231 after retry RPC@23211908), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4878ff38 -> wakeup thread Time-limited test -> (continuation of Deferred@723022178 after retry RPC@605477382) -> (continuation of Deferred@418019214 after retry RPC@180246655) -> (continuation of Deferred@2036858231 after retry RPC@23211908))) after lookup
02:12:31.926 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] connecting to peer
02:12:31.926 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:31.927 [DEBUG - kudu-nio-0] (Connection.java:709) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] cleaning up while in state CONNECTING due to: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:31.927 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:32.406 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:32.406 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)[L],fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:32.407 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=580), Traces: [0ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [2ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [2ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [18ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [19ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [21ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [21ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [38ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [39ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [40ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [40ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [57ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [58ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [59ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [59ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [95ms] refreshing cache from master, [95ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [98ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [99ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [101ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [101ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [574ms] refreshing cache from master, [575ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [580ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@2402291(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@76766b68 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@723022178 after retry RPC@605477382) -> (continuation of Deferred@418019214 after retry RPC@180246655) -> (continuation of Deferred@2036858231 after retry RPC@23211908) -> (continuation of Deferred@911604041 after retry RPC@643558920), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4878ff38 -> wakeup thread Time-limited test -> (continuation of Deferred@723022178 after retry RPC@605477382) -> (continuation of Deferred@418019214 after retry RPC@180246655) -> (continuation of Deferred@2036858231 after retry RPC@23211908) -> (continuation of Deferred@911604041 after retry RPC@643558920))) after lookup
02:12:32.408 [DEBUG - kudu-nio-2] (Connection.java:748) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] connecting to peer
02:12:32.409 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:32.409 [DEBUG - kudu-nio-1] (Connection.java:709) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] cleaning up while in state CONNECTING due to: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:32.409 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373
02:12:32.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.816309  9830 heartbeater.cc:499] Master 127.7.102.189:33589 was elected leader, sending a full tablet report...
02:12:32.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.863073  9840 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cc9ab047ac31487896edb4faf2fb74b5)
02:12:32.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.863606  9840 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:32.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.866959  9840 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373), 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:12:32.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:32.871003  9525 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.102.129:42373: connect: Connection refused (error 111)
02:12:32.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:32.876503  9525 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373): Network error: Client connection negotiation failed: client connection to 127.7.102.129:42373: connect: Connection refused (error 111)
02:12:32.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.882500  9783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "372cdb4006a84d17800acd046e7b5a41" is_pre_election: true
02:12:32.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.883297  9783 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe0a2e8ea6da4fa8b844e7d99e3cae0c in term 8.
02:12:32.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.884727  9525 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 372cdb4006a84d17800acd046e7b5a41, fe0a2e8ea6da4fa8b844e7d99e3cae0c; no voters: cc9ab047ac31487896edb4faf2fb74b5
02:12:32.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.885666  9840 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 9 FOLLOWER]: Leader pre-election won for term 10
02:12:32.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.885946  9840 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 9 FOLLOWER]: Starting leader election (detected failure of leader cc9ab047ac31487896edb4faf2fb74b5)
02:12:32.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.886276  9840 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 9 FOLLOWER]: Advancing to term 10
02:12:32.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.892848  9840 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:32.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.894450  9840 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 10 election: Requested vote from peers cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373), 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130:41431)
02:12:32.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.896402  9783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: false dest_uuid: "372cdb4006a84d17800acd046e7b5a41"
02:12:32.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.896981  9783 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 8 FOLLOWER]: Advancing to term 10
02:12:32.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:32.898458  9525 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373): Network error: Client connection negotiation failed: client connection to 127.7.102.129:42373: connect: Connection refused (error 111)
02:12:32.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.903718  9783 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe0a2e8ea6da4fa8b844e7d99e3cae0c in term 10.
02:12:32.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.904712  9525 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 372cdb4006a84d17800acd046e7b5a41, fe0a2e8ea6da4fa8b844e7d99e3cae0c; no voters: cc9ab047ac31487896edb4faf2fb74b5
02:12:32.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.905334  9840 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 10 FOLLOWER]: Leader election won for term 10
02:12:32.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.906564  9840 raft_consensus.cc:697] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 10 LEADER]: Becoming Leader. State: Replica: fe0a2e8ea6da4fa8b844e7d99e3cae0c, State: Running, Role: LEADER
02:12:32.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.907657  9840 consensus_queue.cc:237] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:32.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:32.916733  7696 catalog_manager.cc:5696] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c reported cstate change: term changed from 9 to 10, leader changed from cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129) to fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131). New cstate: current_term: 10 leader_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } health_report { overall_health: UNKNOWN } } }
02:12:33.005 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:33.005 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:12:33.005 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=1179), Traces: [0ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [2ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [2ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [15ms] refreshing cache from master, [15ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [18ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [19ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [21ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [21ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [34ms] refreshing cache from master, [34ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [38ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [39ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [40ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [40ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [54ms] refreshing cache from master, [54ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [57ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [58ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [59ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [59ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [95ms] refreshing cache from master, [95ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [98ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [99ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [101ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [101ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [574ms] refreshing cache from master, [575ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [580ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [581ms] sending RPC to server cc9ab047ac31487896edb4faf2fb74b5, [583ms] received response from server cc9ab047ac31487896edb4faf2fb74b5: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [583ms] delaying RPC due to: Network error: Failed to connect to peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373): Connection refused: /127.7.102.129:42373, [1174ms] refreshing cache from master, [1174ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [1178ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@2402291(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@76766b68 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@723022178 after retry RPC@605477382) -> (continuation of Deferred@418019214 after retry RPC@180246655) -> (continuation of Deferred@2036858231 after retry RPC@23211908) -> (continuation of Deferred@911604041 after retry RPC@643558920) -> (continuation of Deferred@415838633 after retry RPC@1629206294), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@4878ff38 -> wakeup thread Time-limited test -> (continuation of Deferred@723022178 after retry RPC@605477382) -> (continuation of Deferred@418019214 after retry RPC@180246655) -> (continuation of Deferred@2036858231 after retry RPC@23211908) -> (continuation of Deferred@911604041 after retry RPC@643558920) -> (continuation of Deferred@415838633 after retry RPC@1629206294))) after lookup
02:12:33.006 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:33.007 [DEBUG - kudu-nio-2] (Connection.java:757) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] Successfully connected to peer
02:12:33.026 [DEBUG - kudu-nio-2] (Negotiator.java:919) Authenticated connection [id: 0x7802ccf5, L:/127.0.0.1:45640 - R:/127.7.102.131:37689] using TOKEN/null
02:12:33.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:33.036803  9783 raft_consensus.cc:1275] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 10 FOLLOWER]: Refusing update from remote peer fe0a2e8ea6da4fa8b844e7d99e3cae0c: Log matching property violated. Preceding OpId in replica: term: 8 index: 32. Preceding OpId from leader: term: 10 index: 38. (index mismatch)
02:12:33.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:33.038410  9840 consensus_queue.cc:1048] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
02:12:33.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:33.040346  9525 consensus_peers.cc:597] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c -> Peer cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373): Couldn't send request to peer cc9ab047ac31487896edb4faf2fb74b5. Status: Network error: Client connection negotiation failed: client connection to 127.7.102.129:42373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:12:33.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:33.066141  9858 mvcc.cc:204] Tried to move back new op lower bound from 7305120780423217152 to 7305120779917643776. Current Snapshot: MvccSnapshot[applied={T|T < 7305120780423217152}]
02:12:33.109 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.7.102.129:42373
02:12:33.110 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

02:12:33.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:33.111156  7578 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:12:33.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:12:33.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal
02:12:33.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data
02:12:33.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:12:33.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:12:33.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:12:33.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:33.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:33.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:33.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/logs
02:12:33.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data/info.pb
02:12:33.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:33.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:12:33.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:12:33.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:12:33.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:12:33.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:33.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:33.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:33.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:12:33.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:12:33.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.129:42373
02:12:33.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.129
02:12:33.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:12:33.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37707
02:12:33.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.129
02:12:33.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:12:33.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:33.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:33.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:12:33.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:33.464327  9867 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:12:33.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:33.464941  9867 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:12:33.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:33.465590  9867 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:12:33.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:33.498267  9867 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:12:33.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:33.499303  9867 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.102.129
02:12:33.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:33.540295  9867 tablet_server_runner.cc:78] Tablet server non-default flags:
02:12:33.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:33.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.129:42373
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data/info.pb
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.129
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=37707
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.129
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/logs
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on e7f111948823
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16850
02:12:33.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:12:33.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:33.542094  9867 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:12:33.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:33.543828  9867 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:12:33.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:33.557406  9873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:35.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:33.564304  9876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:35.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:33.559674  9874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:35.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:35.020411  9875 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1456 milliseconds
02:12:35.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.020579  9867 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:12:35.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.022063  9867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:12:35.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.025256  9867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:12:35.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.026710  9867 hybrid_clock.cc:648] HybridClock initialized: now 1783476755026665 us; error 67 us; skew 500 ppm
02:12:35.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.027633  9867 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:12:35.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.035246  9867 webserver.cc:534] Webserver started at http://127.7.102.129:37707/ using document root <none> and password file <none>
02:12:35.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.036280  9867 fs_manager.cc:362] Metadata directory not provided
02:12:35.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.036542  9867 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:12:35.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.044565  9867 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.001s	sys 0.004s
02:12:35.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.049646  9888 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:12:35.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.051124  9867 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
02:12:35.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.051463  9867 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data,/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal
02:12:35.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "cc9ab047ac31487896edb4faf2fb74b5"
02:12:35.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:48 on dist-test-slave-5s6b"
02:12:35.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.053431  9867 fs_report.cc:389] FS layout report
02:12:35.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:12:35.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal
02:12:35.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/wal
02:12:35.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data/data
02:12:35.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:12:35.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:12:35.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:12:35.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:12:35.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:12:35.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:12:35.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:12:35.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:12:35.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:12:35.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:12:35.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:12:35.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.116787  9867 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:12:35.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.118408  9867 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:12:35.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.118861  9867 kserver.cc:163] Server-wide thread pool size limit: 3276
02:12:35.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.121580  9867 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:12:35.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.127537  9895 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:12:35.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.136193  9867 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:12:35.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.136451  9867 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s	user 0.000s	sys 0.002s
02:12:35.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.136669  9867 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:12:35.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.141570  9867 ts_tablet_manager.cc:616] Registered 1 tablets
02:12:35.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.141774  9867 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.002s	sys 0.001s
02:12:35.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.142320  9895 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Bootstrap starting.
02:12:35.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.213348  9895 log.cc:826] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Log is configured to *not* fsync() on all Append() calls
02:12:35.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.325937  9867 rpc_server.cc:307] RPC server started. Bound to: 127.7.102.129:42373
02:12:35.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.326188 10002 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.102.129:42373 every 8 connection(s)
02:12:35.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.328125  9895 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:12:35.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.328804  9867 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-0/data/info.pb
02:12:35.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.328830  9895 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Bootstrap complete.
02:12:35.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.330248  9895 ts_tablet_manager.cc:1403] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Time spent bootstrapping tablet: real 0.188s	user 0.150s	sys 0.035s
02:12:35.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.332386  7578 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu as pid 9867
02:12:35.333 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:35.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.362032  9895 raft_consensus.cc:359] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:35.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.367830  9895 raft_consensus.cc:740] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9ab047ac31487896edb4faf2fb74b5, State: Initialized, Role: FOLLOWER
02:12:35.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.369544  9895 consensus_queue.cc:260] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:35.395 [DEBUG - kudu-nio-2] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726, tablet=064f4af2277f4b52af05ed12eaafba04, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:12:35.396 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.7.102.131:37689
02:12:35.396 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:12:35.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.397181  7578 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu with pid 9506
02:12:35.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.430960 10003 heartbeater.cc:344] Connected to a master server at 127.7.102.188:43935
02:12:35.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.431440 10003 heartbeater.cc:461] Registering TS with master...
02:12:35.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.432766 10003 heartbeater.cc:507] Master 127.7.102.188:43935 requested a full tablet report, sending...
02:12:35.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.439620  7778 ts_manager.cc:194] Re-registered known tserver with Master: cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373)
02:12:35.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.440514  9895 ts_tablet_manager.cc:1434] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5: Time spent starting tablet: real 0.110s	user 0.037s	sys 0.013s
02:12:35.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.445586 10007 heartbeater.cc:344] Connected to a master server at 127.7.102.190:37199
02:12:35.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.446012 10007 heartbeater.cc:461] Registering TS with master...
02:12:35.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.447090 10007 heartbeater.cc:507] Master 127.7.102.190:37199 requested a full tablet report, sending...
02:12:35.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.448776 10004 heartbeater.cc:344] Connected to a master server at 127.7.102.189:33589
02:12:35.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.449044 10004 heartbeater.cc:461] Registering TS with master...
02:12:35.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.450023 10004 heartbeater.cc:507] Master 127.7.102.189:33589 requested a full tablet report, sending...
02:12:35.453 [DEBUG - kudu-nio-2] (Connection.java:239) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] handling channelInactive
02:12:35.453 [DEBUG - kudu-nio-2] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state READY due to: connection closed
02:12:35.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.451839  7623 ts_manager.cc:194] Re-registered known tserver with Master: cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373)
02:12:35.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.453568  7697 ts_manager.cc:194] Re-registered known tserver with Master: cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373)
02:12:35.455 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:35.456 [DEBUG - Time-limited test] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:35.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.456549  7697 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.102.129:40853
02:12:35.457 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:35.457 [DEBUG - kudu-nio-3] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:35.457 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:35.460609 10004 heartbeater.cc:499] Master 127.7.102.189:33589 was elected leader, sending a full tablet report...
02:12:35.464 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:35.465 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:12:35.465 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=9), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [2ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [5ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [9ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@2056910143(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@471e8402 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@494b9a13 -> wakeup thread Time-limited test)) after lookup
02:12:35.466 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:35.467 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:35.467 [DEBUG - kudu-nio-4] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:35.467 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:35.484 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:35.484 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:12:35.484 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [2ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [5ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [9ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [10ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [12ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [12ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [28ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@2056910143(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@471e8402 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@342810977 after retry RPC@31484669), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@494b9a13 -> wakeup thread Time-limited test -> (continuation of Deferred@342810977 after retry RPC@31484669))) after lookup
02:12:35.485 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:35.486 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:35.486 [DEBUG - kudu-nio-5] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:35.486 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:35.525 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:35.525 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:12:35.525 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=70), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [2ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [5ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [9ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [10ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [12ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [12ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [28ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [29ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [31ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [31ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [69ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@2056910143(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@471e8402 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@342810977 after retry RPC@31484669) -> (continuation of Deferred@2114244700 after retry RPC@2083650275), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@494b9a13 -> wakeup thread Time-limited test -> (continuation of Deferred@342810977 after retry RPC@31484669) -> (continuation of Deferred@2114244700 after retry RPC@2083650275))) after lookup
02:12:35.526 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:35.527 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:35.527 [DEBUG - kudu-nio-6] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:35.528 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:35.624 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:35.624 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:12:35.624 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=169), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [2ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [5ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [9ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [10ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [12ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [12ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [28ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [29ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [31ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [31ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [69ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [71ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [73ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [73ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [165ms] refreshing cache from master, [165ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [168ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@2056910143(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@471e8402 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@342810977 after retry RPC@31484669) -> (continuation of Deferred@2114244700 after retry RPC@2083650275) -> (continuation of Deferred@763423927 after retry RPC@1586887942), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@494b9a13 -> wakeup thread Time-limited test -> (continuation of Deferred@342810977 after retry RPC@31484669) -> (continuation of Deferred@2114244700 after retry RPC@2083650275) -> (continuation of Deferred@763423927 after retry RPC@1586887942))) after lookup
02:12:35.625 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:35.626 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:35.626 [DEBUG - kudu-nio-7] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:35.627 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:36.105 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:36.106 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431),cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)[L]]}]
02:12:36.106 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=650), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [2ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [5ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [9ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [10ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [12ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [12ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [28ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [29ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [31ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [31ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [69ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [71ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [73ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [73ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [165ms] refreshing cache from master, [165ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [168ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [170ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [171ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [172ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [646ms] refreshing cache from master, [646ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [650ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@2056910143(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@471e8402 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@342810977 after retry RPC@31484669) -> (continuation of Deferred@2114244700 after retry RPC@2083650275) -> (continuation of Deferred@763423927 after retry RPC@1586887942) -> (continuation of Deferred@1264128767 after retry RPC@987081567), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@494b9a13 -> wakeup thread Time-limited test -> (continuation of Deferred@342810977 after retry RPC@31484669) -> (continuation of Deferred@2114244700 after retry RPC@2083650275) -> (continuation of Deferred@763423927 after retry RPC@1586887942) -> (continuation of Deferred@1264128767 after retry RPC@987081567))) after lookup
02:12:36.107 [DEBUG - kudu-nio-2] (Connection.java:748) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] connecting to peer
02:12:36.108 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:36.108 [DEBUG - kudu-nio-0] (Connection.java:709) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] cleaning up while in state CONNECTING due to: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:36.108 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689) for tablet 064f4af2277f4b52af05ed12eaafba04: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689
02:12:36.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:36.548673 10017 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader fe0a2e8ea6da4fa8b844e7d99e3cae0c)
02:12:36.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:36.549089 10017 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:36.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:36.551771 10017 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689), cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373)
02:12:36.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:36.554572  9713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.102.131:37689: connect: Connection refused (error 111)
02:12:36.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:36.558982  9713 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689): Network error: Client connection negotiation failed: client connection to 127.7.102.131:37689: connect: Connection refused (error 111)
02:12:36.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:36.566941  9958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "372cdb4006a84d17800acd046e7b5a41" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "cc9ab047ac31487896edb4faf2fb74b5" is_pre_election: true
02:12:36.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:36.567706  9958 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372cdb4006a84d17800acd046e7b5a41 in term 9.
02:12:36.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:36.569209  9711 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 372cdb4006a84d17800acd046e7b5a41, cc9ab047ac31487896edb4faf2fb74b5; no voters: fe0a2e8ea6da4fa8b844e7d99e3cae0c
02:12:36.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:36.569983 10017 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 10 FOLLOWER]: Leader pre-election won for term 11
02:12:36.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:36.570297 10017 raft_consensus.cc:493] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 10 FOLLOWER]: Starting leader election (detected failure of leader fe0a2e8ea6da4fa8b844e7d99e3cae0c)
02:12:36.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:36.570618 10017 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 10 FOLLOWER]: Advancing to term 11
02:12:36.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:36.575043 10017 raft_consensus.cc:515] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:36.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:36.576541 10017 leader_election.cc:290] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 11 election: Requested vote from peers fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689), cc9ab047ac31487896edb4faf2fb74b5 (127.7.102.129:42373)
02:12:36.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:36.577654  9958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "064f4af2277f4b52af05ed12eaafba04" candidate_uuid: "372cdb4006a84d17800acd046e7b5a41" candidate_term: 11 candidate_status { last_received { term: 10 index: 40 } } ignore_live_leader: false dest_uuid: "cc9ab047ac31487896edb4faf2fb74b5"
02:12:36.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:36.578205  9958 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 9 FOLLOWER]: Advancing to term 11
02:12:36.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:36.580039  9713 leader_election.cc:336] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689): Network error: Client connection negotiation failed: client connection to 127.7.102.131:37689: connect: Connection refused (error 111)
02:12:36.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:36.584733  9958 raft_consensus.cc:2468] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372cdb4006a84d17800acd046e7b5a41 in term 11.
02:12:36.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:36.585757  9711 leader_election.cc:304] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 372cdb4006a84d17800acd046e7b5a41, cc9ab047ac31487896edb4faf2fb74b5; no voters: fe0a2e8ea6da4fa8b844e7d99e3cae0c
02:12:36.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:36.586406 10017 raft_consensus.cc:2804] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 11 FOLLOWER]: Leader election won for term 11
02:12:36.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:36.587877 10017 raft_consensus.cc:697] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [term 11 LEADER]: Becoming Leader. State: Replica: 372cdb4006a84d17800acd046e7b5a41, State: Running, Role: LEADER
02:12:36.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:36.588557 10017 consensus_queue.cc:237] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:36.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:36.597339  7697 catalog_manager.cc:5696] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 reported cstate change: term changed from 10 to 11, leader changed from fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131) to 372cdb4006a84d17800acd046e7b5a41 (127.7.102.130). New cstate: current_term: 11 leader_uuid: "372cdb4006a84d17800acd046e7b5a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } health_report { overall_health: HEALTHY } } }
02:12:37.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:37.004801  9713 consensus_peers.cc:597] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 -> Peer fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689): Couldn't send request to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c. Status: Network error: Client connection negotiation failed: client connection to 127.7.102.131:37689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
02:12:37.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:37.188431  9958 raft_consensus.cc:1275] T 064f4af2277f4b52af05ed12eaafba04 P cc9ab047ac31487896edb4faf2fb74b5 [term 11 FOLLOWER]: Refusing update from remote peer 372cdb4006a84d17800acd046e7b5a41: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 11 index: 41. (index mismatch)
02:12:37.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:37.190083 10017 consensus_queue.cc:1048] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
02:12:37.385 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:2538) Learned about tablet 064f4af2277f4b52af05ed12eaafba04 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726' with partition [<start>, <end>)
02:12:37.385 [DEBUG - kudu-nio-2] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=064f4af2277f4b52af05ed12eaafba04@[372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)[L],cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373),fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)]}]
02:12:37.386 [DEBUG - kudu-nio-2] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=064f4af2277f4b52af05ed12eaafba04, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=1930), Traces: [0ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [2ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [2ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [5ms] refreshing cache from master, [6ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [9ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [10ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [12ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [12ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [28ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [29ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [31ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [31ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [66ms] refreshing cache from master, [66ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [69ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [71ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [73ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [73ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [165ms] refreshing cache from master, [165ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [168ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [170ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [171ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [172ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [646ms] refreshing cache from master, [646ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [650ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, [651ms] sending RPC to server fe0a2e8ea6da4fa8b844e7d99e3cae0c, [653ms] received response from server fe0a2e8ea6da4fa8b844e7d99e3cae0c: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [653ms] delaying RPC due to: Network error: Failed to connect to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689): Connection refused: /127.7.102.131:37689, [1926ms] refreshing cache from master, [1926ms] Sub RPC GetTableLocations: sending RPC to server master-127.7.102.189:33589, [1929ms] Sub RPC GetTableLocations: received response from server master-127.7.102.189:33589: OK, deferred=Deferred@2056910143(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$235/0x00007fa4e836b9a0@471e8402 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@342810977 after retry RPC@31484669) -> (continuation of Deferred@2114244700 after retry RPC@2083650275) -> (continuation of Deferred@763423927 after retry RPC@1586887942) -> (continuation of Deferred@1264128767 after retry RPC@987081567) -> (continuation of Deferred@1864051203 after retry RPC@2021915628), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@494b9a13 -> wakeup thread Time-limited test -> (continuation of Deferred@342810977 after retry RPC@31484669) -> (continuation of Deferred@2114244700 after retry RPC@2083650275) -> (continuation of Deferred@763423927 after retry RPC@1586887942) -> (continuation of Deferred@1264128767 after retry RPC@987081567) -> (continuation of Deferred@1864051203 after retry RPC@2021915628))) after lookup
02:12:37.386 [DEBUG - kudu-nio-2] (Connection.java:748) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] connecting to peer
02:12:37.387 [DEBUG - kudu-nio-1] (Connection.java:757) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] Successfully connected to peer
02:12:37.412 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0x78fabd43, L:/127.0.0.1:45646 - R:/127.7.102.130:41431] using TOKEN/null
02:12:37.459 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.7.102.131:37689
02:12:37.459 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

02:12:37.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:37.461331  7578 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:12:37.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu
02:12:37.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:12:37.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data
02:12:37.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
02:12:37.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
02:12:37.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
02:12:37.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:37.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:37.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/logs
02:12:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data/info.pb
02:12:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
02:12:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
02:12:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
02:12:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
02:12:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
02:12:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
02:12:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.131:37689
02:12:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.131
02:12:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:12:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38903
02:12:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.131
02:12:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:12:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:37.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
02:12:37.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:37.810039 10047 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
02:12:37.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:37.810650 10047 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
02:12:37.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:37.811260 10047 flags.cc:432] Enabled unsafe flag: --never_fsync=true
02:12:37.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:37.844235 10047 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
02:12:37.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:37.845255 10047 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.102.131
02:12:37.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:37.881960 10047 tablet_server_runner.cc:78] Tablet server non-default flags:
02:12:37.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
02:12:37.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.102.148:41461
02:12:37.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
02:12:37.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
02:12:37.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data
02:12:37.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:12:37.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
02:12:37.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
02:12:37.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.102.131:37689
02:12:37.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
02:12:37.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
02:12:37.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
02:12:37.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data/info.pb
02:12:37.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.102.131
02:12:37.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=38903
02:12:37.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.7.102.190:37199,127.7.102.189:33589,127.7.102.188:43935
02:12:37.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
02:12:37.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
02:12:37.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
02:12:37.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
02:12:37.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
02:12:37.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.7.102.131
02:12:37.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/logs
02:12:37.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
02:12:37.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
02:12:37.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
02:12:37.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
02:12:37.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
02:12:37.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision a3f35afc2b6dad6aa403ec58378221544a26edd0
02:12:37.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
02:12:37.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 08 Jul 2026 01:40:41 UTC on e7f111948823
02:12:37.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 16850
02:12:37.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
02:12:37.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:37.883778 10047 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
02:12:37.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:37.885493 10047 file_cache.cc:492] Constructed file cache file cache with capacity 419430
02:12:37.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:37.897645 10053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:37.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:37.953892  7729 debug-util.cc:398] Leaking SignalData structure 0x7b0800042200 after lost signal to thread 7664
02:12:37.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:37.955036  7729 debug-util.cc:398] Leaking SignalData structure 0x7b0800065f00 after lost signal to thread 7732
02:12:39.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:39.506443  9713 consensus_peers.cc:597] T 064f4af2277f4b52af05ed12eaafba04 P 372cdb4006a84d17800acd046e7b5a41 -> Peer fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689): Couldn't send request to peer fe0a2e8ea6da4fa8b844e7d99e3cae0c. Status: Network error: Client connection negotiation failed: client connection to 127.7.102.131:37689: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
02:12:39.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:39.300598 10052 debug-util.cc:398] Leaking SignalData structure 0x7b0800028000 after lost signal to thread 10047
02:12:39.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:39.610350 10047 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.712s	user 0.000s	sys 0.002s
02:12:39.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:37.899595 10054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:39.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:39.610771 10047 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.712s	user 0.000s	sys 0.002s
02:12:39.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:39.612933 10056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
02:12:39.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:39.615868 10055 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1711 milliseconds
02:12:39.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.615984 10047 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
02:12:39.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.617650 10047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
02:12:39.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.620571 10047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
02:12:39.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.622169 10047 hybrid_clock.cc:648] HybridClock initialized: now 1783476759622108 us; error 97 us; skew 500 ppm
02:12:39.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.623304 10047 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
02:12:39.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.634927 10047 webserver.cc:534] Webserver started at http://127.7.102.131:38903/ using document root <none> and password file <none>
02:12:39.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.636866 10047 fs_manager.cc:362] Metadata directory not provided
02:12:39.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.637238 10047 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
02:12:39.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.648255 10047 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
02:12:39.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.653631 10068 log_block_manager.cc:4031] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
02:12:39.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.654917 10047 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
02:12:39.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.655211 10047 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data,/tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:12:39.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c"
02:12:39.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-07-08 02:11:52 on dist-test-slave-5s6b"
02:12:39.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.657132 10047 fs_report.cc:389] FS layout report
02:12:39.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
02:12:39.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:12:39.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/wal
02:12:39.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data/data
02:12:39.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
02:12:39.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
02:12:39.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
02:12:39.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
02:12:39.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
02:12:39.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
02:12:39.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
02:12:39.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
02:12:39.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
02:12:39.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
02:12:39.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
02:12:39.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.710543 10047 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
02:12:39.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.713043 10047 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
02:12:39.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.713927 10047 kserver.cc:163] Server-wide thread pool size limit: 3276
02:12:39.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.718096 10047 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
02:12:39.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.725510 10075 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
02:12:39.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.738708 10047 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
02:12:39.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.739001 10047 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.016s	user 0.002s	sys 0.001s
02:12:39.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.739338 10047 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
02:12:39.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.744418 10047 ts_tablet_manager.cc:616] Registered 1 tablets
02:12:39.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.744688 10047 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.005s	sys 0.000s
02:12:39.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.745404 10075 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Bootstrap starting.
02:12:39.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:39.873402 10075 log.cc:826] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Log is configured to *not* fsync() on all Append() calls
02:12:40.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.010464 10047 rpc_server.cc:307] RPC server started. Bound to: 127.7.102.131:37689
02:12:40.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.010658 10183 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.102.131:37689 every 8 connection(s)
02:12:40.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.014590 10047 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskf2jM_v/test-tmp/mini-kudu-cluster17930050994777445089/ts-2/data/info.pb
02:12:40.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.022974  7578 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu as pid 10047
02:12:40.024 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
02:12:40.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.027983 10075 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
02:12:40.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.029270 10075 tablet_bootstrap.cc:492] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Bootstrap complete.
02:12:40.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.031430 10075 ts_tablet_manager.cc:1403] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Time spent bootstrapping tablet: real 0.287s	user 0.190s	sys 0.062s
02:12:40.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.060878 10075 raft_consensus.cc:359] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:40.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.064604 10075 raft_consensus.cc:740] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe0a2e8ea6da4fa8b844e7d99e3cae0c, State: Initialized, Role: FOLLOWER
02:12:40.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.065582 10075 consensus_queue.cc:260] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 10.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0a2e8ea6da4fa8b844e7d99e3cae0c" member_type: VOTER last_known_addr { host: "127.7.102.131" port: 37689 } } peers { permanent_uuid: "cc9ab047ac31487896edb4faf2fb74b5" member_type: VOTER last_known_addr { host: "127.7.102.129" port: 42373 } } peers { permanent_uuid: "372cdb4006a84d17800acd046e7b5a41" member_type: VOTER last_known_addr { host: "127.7.102.130" port: 41431 } }
02:12:40.078 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726, tablet=064f4af2277f4b52af05ed12eaafba04, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:12:40.096 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1783476713726, tablet=064f4af2277f4b52af05ed12eaafba04, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
02:12:40.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.097126 10075 ts_tablet_manager.cc:1434] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c: Time spent starting tablet: real 0.065s	user 0.036s	sys 0.028s
02:12:40.103 [DEBUG - main] (Connection.java:575) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] disconnecting while in state READY
02:12:40.110 [DEBUG - kudu-nio-1] (Connection.java:239) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] handling channelInactive
02:12:40.111 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 372cdb4006a84d17800acd046e7b5a41(127.7.102.130:41431)] cleaning up while in state READY due to: connection closed
02:12:40.114 [DEBUG - main] (Connection.java:575) [peer master-127.7.102.188:43935(127.7.102.188:43935)] disconnecting while in state READY
02:12:40.115 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.102.188:43935(127.7.102.188:43935)] handling channelInactive
02:12:40.117 [DEBUG - main] (Connection.java:575) [peer master-127.7.102.189:33589(127.7.102.189:33589)] disconnecting while in state READY
02:12:40.117 [DEBUG - main] (Connection.java:575) [peer master-127.7.102.190:37199(127.7.102.190:37199)] disconnecting while in state READY
02:12:40.117 [DEBUG - main] (Connection.java:575) [peer fe0a2e8ea6da4fa8b844e7d99e3cae0c(127.7.102.131:37689)] disconnecting while in state TERMINATED
02:12:40.117 [DEBUG - main] (Connection.java:575) [peer cc9ab047ac31487896edb4faf2fb74b5(127.7.102.129:42373)] disconnecting while in state TERMINATED
02:12:40.118 [DEBUG - kudu-nio-2] (Connection.java:239) [peer master-127.7.102.189:33589(127.7.102.189:33589)] handling channelInactive
02:12:40.118 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.102.188:43935(127.7.102.188:43935)] cleaning up while in state READY due to: connection closed
02:12:40.118 [DEBUG - kudu-nio-2] (Connection.java:709) [peer master-127.7.102.189:33589(127.7.102.189:33589)] cleaning up while in state READY due to: connection closed
02:12:40.119 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2704) Releasing all remaining resources
02:12:40.135 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.102.190:37199(127.7.102.190:37199)] handling channelInactive
02:12:40.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.135414  7578 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu with pid 9867
02:12:40.136 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.102.190:37199(127.7.102.190:37199)] cleaning up while in state READY due to: connection closed
02:12:40.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.157878 10185 heartbeater.cc:344] Connected to a master server at 127.7.102.189:33589
02:12:40.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.158510 10185 heartbeater.cc:461] Registering TS with master...
02:12:40.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.158573 10184 heartbeater.cc:344] Connected to a master server at 127.7.102.188:43935
02:12:40.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.158995 10184 heartbeater.cc:461] Registering TS with master...
02:12:40.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.160085 10185 heartbeater.cc:507] Master 127.7.102.189:33589 requested a full tablet report, sending...
02:12:40.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.160269 10184 heartbeater.cc:507] Master 127.7.102.188:43935 requested a full tablet report, sending...
02:12:40.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.166924  7697 ts_manager.cc:194] Re-registered known tserver with Master: fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689)
02:12:40.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.171240  7697 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.7.102.131:39001
02:12:40.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.175741 10185 heartbeater.cc:499] Master 127.7.102.189:33589 was elected leader, sending a full tablet report...
02:12:40.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.175632 10139 raft_consensus.cc:3060] T 064f4af2277f4b52af05ed12eaafba04 P fe0a2e8ea6da4fa8b844e7d99e3cae0c [term 10 FOLLOWER]: Advancing to term 11
02:12:40.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.187788  7777 ts_manager.cc:194] Re-registered known tserver with Master: fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689)
02:12:40.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.189175 10189 heartbeater.cc:344] Connected to a master server at 127.7.102.190:37199
02:12:40.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.189572 10189 heartbeater.cc:461] Registering TS with master...
02:12:40.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.190249 10189 heartbeater.cc:507] Master 127.7.102.190:37199 requested a full tablet report, sending...
02:12:40.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.225425 10199 mvcc.cc:204] Tried to move back new op lower bound from 7305120798377009152 to 7305120794995023872. Current Snapshot: MvccSnapshot[applied={T|T < 7305120798377009152}]
02:12:40.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.228233  7578 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu with pid 9690
02:12:40.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.247012  7622 ts_manager.cc:194] Re-registered known tserver with Master: fe0a2e8ea6da4fa8b844e7d99e3cae0c (127.7.102.131:37689)
02:12:40.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260708 02:12:40.272825 10072 connection.cc:573] server connection from 127.7.102.130:46957 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
02:12:40.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.274197  7578 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu with pid 10047
02:12:40.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.305776  7578 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu with pid 7590
02:12:40.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.339154  7578 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu with pid 7660
02:12:40.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260708 02:12:40.375161  7578 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskf2jM_v/build/tsan/bin/kudu with pid 7735
02:12:41.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-07-08T02:12:41Z chronyd exiting

Time: 67.178

OK (1 test)

