JUnit version 4.13.2
.14:04:03.943 [DEBUG - main] (RandomUtils.java:49) Using random seed: 1779113043933
14:04:04.024 [INFO - main] (RetryRule.java:138) Creating RetryStatement with result reporter and retry count of 0 (no retries)
14:04:04.102 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
14:04:04.115 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskSAAnLi/test-tmp
14:04:04.125 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
14:04:04.128 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-taskSAAnLi/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
14:04:04.365 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
  num_masters: 3
  num_tservers: 3
  enable_kerberos: false
  cluster_root: "/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152"
  hms_mode: NONE
  mini_kdc_options {
  }
  principal: "kudu"
  mini_oidc_options {
  }
}

14:04:04.591 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
14:04:04.594 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}

14:04:04.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:04.599272 27264 mini_oidc.cc:150] Starting JWKS server
14:04:04.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:04.603673 27264 webserver.cc:533] Webserver started at http://127.0.0.1:41487/ using document root <none> and password file <none>
14:04:04.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:04.604382 27264 mini_oidc.cc:186] Starting OIDC Discovery server
14:04:04.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:04.607041 27264 webserver.cc:533] Webserver started at http://127.0.0.1:33495/ using document root <none> and password file <none>
14:04:04.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-05-18T14:04:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
14:04:04.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-05-18T14:04:04Z Disabled control of system clock
14:04:04.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:04.678293 27264 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:04.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:04.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-0/wal
14:04:04.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-0/data
14:04:04.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:04:04.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:04:04.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:04:04.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:04.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:04.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:04.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-0/logs
14:04:04.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-0/data/info.pb
14:04:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:04:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:04:04.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:04:04.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:04:04.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:04.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:04.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:04.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
14:04:04.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:04:04.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:04:04.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:04:04.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.62:43077
14:04:04.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.62
14:04:04.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:04:04.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:04.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:04.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:04.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:04:04.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:04:04.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:04.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:04:04.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41487/jwks.json with env {}
14:04:05.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:05.022833 27277 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:04:05.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:05.023402 27277 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:04:05.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:05.024223 27277 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:04:05.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:05.060427 27277 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
14:04:05.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:05.060890 27277 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:04:05.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:05.061250 27277 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
14:04:05.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:05.061622 27277 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
14:04:05.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:05.062011 27277 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
14:04:05.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:05.062296 27277 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:41487/jwks.json
14:04:05.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:05.102634 27277 master_runner.cc:387] Master server non-default flags:
14:04:05.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:05.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:05.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:05.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:04:05.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-0/data
14:04:05.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-0/wal
14:04:05.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:04:05.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:05.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:05.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:05.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:04:05.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.62:43077
14:04:05.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:04:05.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:04:05.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:04:05.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41487/jwks.json
14:04:05.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:05.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:05.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-0/data/info.pb
14:04:05.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.62
14:04:05.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:04:05.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:04:05.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:05.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:04:05.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:04:05.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:05.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-0/logs
14:04:05.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:05.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:04:05.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:04:05.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
14:04:05.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:04:05.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:04:05.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:04:05.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:22 UTC on e7f111948823
14:04:05.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11959
14:04:05.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:04:05.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:05.104512 27277 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:04:05.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:05.106921 27277 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:04:05.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:05.123956 27283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:06.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:06.525672 27282 debug-util.cc:398] Leaking SignalData structure 0x7b08000271e0 after lost signal to thread 27277
14:04:06.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:06.569279 27282 kernel_stack_watchdog.cc:198] Thread 27277 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
14:04:06.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
14:04:06.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
14:04:06.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
14:04:06.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
14:04:06.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:05.125504 27284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:06.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:06.570657 27285 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1445 milliseconds
14:04:06.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:06.571054 27277 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.448s	user 0.440s	sys 1.003s
14:04:06.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:06.571843 27277 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.449s	user 0.440s	sys 1.003s
14:04:06.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:06.572531 27286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:06.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.572458 27277 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:04:06.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.577888 27277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:04:06.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.580909 27277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:04:06.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.582319 27277 hybrid_clock.cc:648] HybridClock initialized: now 1779113046582268 us; error 70 us; skew 500 ppm
14:04:06.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.583225 27277 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:04:06.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.590540 27277 webserver.cc:533] Webserver started at http://127.26.160.62:44935/ using document root <none> and password file <none>
14:04:06.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.591594 27277 fs_manager.cc:362] Metadata directory not provided
14:04:06.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.591805 27277 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:04:06.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.592262 27277 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:04:06.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.599231 27277 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-0/data/instance:
14:04:06.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66"
14:04:06.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:06 on dist-test-slave-8gxc"
14:04:06.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.600904 27277 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-0/wal/instance:
14:04:06.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66"
14:04:06.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:06 on dist-test-slave-8gxc"
14:04:06.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.608904 27277 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.009s	sys 0.001s
14:04:06.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.615118 27293 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:04:06.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.616715 27277 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.002s
14:04:06.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.617153 27277 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-0/data,/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-0/wal
14:04:06.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66"
14:04:06.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:06 on dist-test-slave-8gxc"
14:04:06.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.617645 27277 fs_report.cc:389] FS layout report
14:04:06.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:04:06.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-0/wal
14:04:06.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-0/wal
14:04:06.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-0/data/data
14:04:06.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:04:06.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:04:06.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:04:06.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:04:06.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:04:06.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:04:06.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:04:06.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:04:06.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:04:06.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:04:06.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:04:06.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.690071 27277 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:04:06.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.691776 27277 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:04:06.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.692199 27277 kserver.cc:163] Server-wide thread pool size limit: 3276
14:04:06.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.765893 27277 rpc_server.cc:307] RPC server started. Bound to: 127.26.160.62:43077
14:04:06.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.766044 27344 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.160.62:43077 every 8 connection(s)
14:04:06.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.769181 27277 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-0/data/info.pb
14:04:06.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.774070 27264 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu as pid 27277
14:04:06.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.774537 27264 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-0/wal/instance
14:04:06.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.776458 27345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
14:04:06.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.779577 27264 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:06.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:06.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-1/wal
14:04:06.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-1/data
14:04:06.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:04:06.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:04:06.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:04:06.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:06.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:06.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:06.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-1/logs
14:04:06.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-1/data/info.pb
14:04:06.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:06.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:04:06.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:04:06.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:04:06.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:04:06.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:06.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:06.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:06.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
14:04:06.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:04:06.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:04:06.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:04:06.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.61:45315
14:04:06.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.61
14:04:06.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:04:06.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:06.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:06.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:06.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:04:06.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:04:06.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:06.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:04:06.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41487/jwks.json with env {}
14:04:06.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.793244 27345 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } has no permanent_uuid. Determining permanent_uuid...
14:04:06.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.824734 27345 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } has no permanent_uuid. Determining permanent_uuid...
14:04:06.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:06.827320 27297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.160.61:45315: connect: Connection refused (error 111)
14:04:06.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:06.832043 27345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.61:45315: Network error: Client connection negotiation failed: client connection to 127.26.160.61:45315: connect: Connection refused (error 111)
14:04:06.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.881613 27345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } attempt: 1
14:04:06.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:06.887323 27345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.61:45315: Network error: Client connection negotiation failed: client connection to 127.26.160.61:45315: connect: Connection refused (error 111)
14:04:06.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:06.955837 27345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } attempt: 2
14:04:06.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:06.960561 27345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.61:45315: Network error: Client connection negotiation failed: client connection to 127.26.160.61:45315: connect: Connection refused (error 111)
14:04:07.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:07.052017 27345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } attempt: 3
14:04:07.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:07.056732 27345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.61:45315: Network error: Client connection negotiation failed: client connection to 127.26.160.61:45315: connect: Connection refused (error 111)
14:04:07.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:07.125491 27347 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:04:07.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:07.126183 27347 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:04:07.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:07.126732 27347 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:04:07.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:07.161293 27347 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
14:04:07.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:07.161610 27347 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:04:07.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:07.161814 27347 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
14:04:07.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:07.162082 27347 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
14:04:07.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:07.162289 27347 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
14:04:07.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:07.162436 27347 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:41487/jwks.json
14:04:07.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:07.200410 27345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } attempt: 4
14:04:07.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:07.204591 27345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.61:45315: Network error: Client connection negotiation failed: client connection to 127.26.160.61:45315: connect: Connection refused (error 111)
14:04:07.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:07.204895 27347 master_runner.cc:387] Master server non-default flags:
14:04:07.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:07.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:07.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:07.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:04:07.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-1/data
14:04:07.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-1/wal
14:04:07.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:04:07.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:07.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:07.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:07.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:04:07.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.61:45315
14:04:07.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:04:07.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:04:07.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:04:07.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41487/jwks.json
14:04:07.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:07.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:07.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-1/data/info.pb
14:04:07.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.61
14:04:07.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:04:07.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:04:07.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:07.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:04:07.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:04:07.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:07.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-1/logs
14:04:07.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:07.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:04:07.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:04:07.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
14:04:07.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:04:07.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:04:07.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:04:07.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:22 UTC on e7f111948823
14:04:07.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11959
14:04:07.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:04:07.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:07.206274 27347 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:04:07.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:07.208093 27347 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:04:07.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:07.223436 27356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:07.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:07.504091 27345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } attempt: 5
14:04:07.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:07.513185 27345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.61:45315: Network error: Client connection negotiation failed: client connection to 127.26.160.61:45315: connect: Connection refused (error 111)
14:04:08.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.060786 27345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } attempt: 6
14:04:08.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:08.106956 27345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.61:45315: Network error: Client connection negotiation failed: client connection to 127.26.160.61:45315: connect: Connection refused (error 111)
14:04:08.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:08.627477 27355 debug-util.cc:398] Leaking SignalData structure 0x7b08000271e0 after lost signal to thread 27347
14:04:08.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:08.636096 27355 kernel_stack_watchdog.cc:198] Thread 27347 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
14:04:08.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
14:04:08.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
14:04:08.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
14:04:08.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
14:04:08.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:07.226266 27357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:08.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:08.636390 27347 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.413s	user 0.412s	sys 0.993s
14:04:08.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:08.636947 27347 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.414s	user 0.412s	sys 0.993s
14:04:08.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:08.638090 27358 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1413 milliseconds
14:04:08.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:08.640292 27359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:08.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.640344 27347 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:04:08.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.645619 27347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:04:08.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.648494 27347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:04:08.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.650108 27347 hybrid_clock.cc:648] HybridClock initialized: now 1779113048650105 us; error 85 us; skew 500 ppm
14:04:08.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.651139 27347 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:04:08.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.657855 27347 webserver.cc:533] Webserver started at http://127.26.160.61:44309/ using document root <none> and password file <none>
14:04:08.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.659085 27347 fs_manager.cc:362] Metadata directory not provided
14:04:08.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.659356 27347 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:04:08.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.659919 27347 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:04:08.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.664541 27347 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-1/data/instance:
14:04:08.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "52b0f69192334e17be4a1acf1702fbf3"
14:04:08.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:08 on dist-test-slave-8gxc"
14:04:08.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.665971 27347 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-1/wal/instance:
14:04:08.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "52b0f69192334e17be4a1acf1702fbf3"
14:04:08.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:08 on dist-test-slave-8gxc"
14:04:08.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.673506 27347 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.001s	sys 0.005s
14:04:08.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.679695 27368 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:04:08.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.681268 27347 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.002s
14:04:08.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.681674 27347 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-1/data,/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-1/wal
14:04:08.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "52b0f69192334e17be4a1acf1702fbf3"
14:04:08.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:08 on dist-test-slave-8gxc"
14:04:08.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.682149 27347 fs_report.cc:389] FS layout report
14:04:08.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:04:08.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-1/wal
14:04:08.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-1/wal
14:04:08.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-1/data/data
14:04:08.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:04:08.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:04:08.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:04:08.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:04:08.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:04:08.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:04:08.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:04:08.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:04:08.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:04:08.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:04:08.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:04:08.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.745530 27347 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:04:08.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.747473 27347 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:04:08.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.748034 27347 kserver.cc:163] Server-wide thread pool size limit: 3276
14:04:08.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.827562 27347 rpc_server.cc:307] RPC server started. Bound to: 127.26.160.61:45315
14:04:08.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.827644 27419 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.160.61:45315 every 8 connection(s)
14:04:08.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.830749 27347 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-1/data/info.pb
14:04:08.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.837038 27420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
14:04:08.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.840921 27264 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu as pid 27347
14:04:08.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.841363 27264 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-1/wal/instance
14:04:08.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.844004 27264 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:08.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:08.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-2/wal
14:04:08.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-2/data
14:04:08.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:04:08.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:04:08.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:04:08.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:08.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:08.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:08.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-2/logs
14:04:08.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-2/data/info.pb
14:04:08.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:08.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:04:08.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:04:08.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:04:08.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:04:08.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:08.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:08.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:08.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
14:04:08.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:04:08.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:04:08.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:04:08.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.60:34099
14:04:08.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.60
14:04:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:04:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:04:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:04:08.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:08.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:04:08.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41487/jwks.json with env {}
14:04:08.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.850003 27420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } has no permanent_uuid. Determining permanent_uuid...
14:04:08.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.879969 27420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } has no permanent_uuid. Determining permanent_uuid...
14:04:08.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.895288 27420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } has no permanent_uuid. Determining permanent_uuid...
14:04:08.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:08.897858 27372 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.160.60:34099: connect: Connection refused (error 111)
14:04:08.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:08.902459 27420 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.60:34099: Network error: Client connection negotiation failed: client connection to 127.26.160.60:34099: connect: Connection refused (error 111)
14:04:08.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:08.952028 27420 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } attempt: 1
14:04:08.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:08.957254 27420 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.60:34099: Network error: Client connection negotiation failed: client connection to 127.26.160.60:34099: connect: Connection refused (error 111)
14:04:09.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:09.025913 27420 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } attempt: 2
14:04:09.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:09.030323 27420 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.60:34099: Network error: Client connection negotiation failed: client connection to 127.26.160.60:34099: connect: Connection refused (error 111)
14:04:09.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:09.121892 27420 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } attempt: 3
14:04:09.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:09.126266 27420 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.60:34099: Network error: Client connection negotiation failed: client connection to 127.26.160.60:34099: connect: Connection refused (error 111)
14:04:09.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:09.167605 27345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } attempt: 7
14:04:09.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:09.180374 27345 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } has no permanent_uuid. Determining permanent_uuid...
14:04:09.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:09.186782 27345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.60:34099: Network error: Client connection negotiation failed: client connection to 127.26.160.60:34099: connect: Connection refused (error 111)
14:04:09.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:09.192358 27422 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:04:09.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:09.193059 27422 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:04:09.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:09.193758 27422 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:04:09.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:09.230515 27422 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
14:04:09.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:09.230934 27422 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:04:09.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:09.231315 27422 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
14:04:09.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:09.231695 27422 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
14:04:09.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:09.232086 27422 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
14:04:09.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:09.232364 27422 flags.cc:432] Enabled experimental flag: --jwks_url=http://localhost:41487/jwks.json
14:04:09.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:09.245311 27345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } attempt: 1
14:04:09.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:09.250022 27345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.60:34099: Network error: Client connection negotiation failed: client connection to 127.26.160.60:34099: connect: Connection refused (error 111)
14:04:09.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:09.269776 27420 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } attempt: 4
14:04:09.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:09.274060 27420 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.60:34099: Network error: Client connection negotiation failed: client connection to 127.26.160.60:34099: connect: Connection refused (error 111)
14:04:09.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:09.274554 27422 master_runner.cc:387] Master server non-default flags:
14:04:09.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:09.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:09.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:09.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:04:09.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-2/data
14:04:09.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-2/wal
14:04:09.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
14:04:09.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:09.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:09.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:09.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
14:04:09.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.60:34099
14:04:09.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
14:04:09.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:04:09.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
14:04:09.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:41487/jwks.json
14:04:09.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:09.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:09.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-2/data/info.pb
14:04:09.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.60
14:04:09.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:04:09.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:04:09.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:09.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:04:09.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:04:09.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:09.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-2/logs
14:04:09.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:09.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:04:09.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:04:09.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
14:04:09.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:04:09.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:04:09.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:04:09.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:22 UTC on e7f111948823
14:04:09.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11959
14:04:09.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:04:09.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:09.276285 27422 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:04:09.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:09.278251 27422 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:04:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:09.289682 27434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:09.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:09.331517 27345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } attempt: 2
14:04:09.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:09.339934 27345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.60:34099: Network error: Client connection negotiation failed: client connection to 127.26.160.60:34099: connect: Connection refused (error 111)
14:04:09.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:09.425570 27345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } attempt: 3
14:04:09.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:09.433662 27345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.60:34099: Network error: Client connection negotiation failed: client connection to 127.26.160.60:34099: connect: Connection refused (error 111)
14:04:09.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:09.573541 27420 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } attempt: 5
14:04:09.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:09.582033 27420 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.60:34099: Network error: Client connection negotiation failed: client connection to 127.26.160.60:34099: connect: Connection refused (error 111)
14:04:09.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:09.582638 27345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } attempt: 4
14:04:09.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:09.593345 27345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.60:34099: Network error: Client connection negotiation failed: client connection to 127.26.160.60:34099: connect: Connection refused (error 111)
14:04:09.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:09.877152 27345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } attempt: 5
14:04:09.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:09.890103 27345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.60:34099: Network error: Client connection negotiation failed: client connection to 127.26.160.60:34099: connect: Connection refused (error 111)
14:04:10.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:10.129676 27420 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } attempt: 6
14:04:10.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:10.170409 27420 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.60:34099: Network error: Client connection negotiation failed: client connection to 127.26.160.60:34099: connect: Connection refused (error 111)
14:04:10.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:10.442764 27345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } attempt: 6
14:04:10.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:10.469039 27345 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.60:34099: Network error: Client connection negotiation failed: client connection to 127.26.160.60:34099: connect: Connection refused (error 111)
14:04:11.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:10.694211 27433 debug-util.cc:398] Leaking SignalData structure 0x7b08000271e0 after lost signal to thread 27422
14:04:11.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:11.126579 27422 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.837s	user 0.682s	sys 1.143s
14:04:11.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:11.127046 27422 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.838s	user 0.682s	sys 1.143s
14:04:11.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:09.291527 27435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:11.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:11.133149 27437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:11.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:11.146247 27436 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1841 milliseconds
14:04:11.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.146375 27422 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:04:11.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.148962 27422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:04:11.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.152719 27422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:04:11.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.154626 27422 hybrid_clock.cc:648] HybridClock initialized: now 1779113051154533 us; error 122 us; skew 500 ppm
14:04:11.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.156597 27422 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:04:11.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.165907 27422 webserver.cc:533] Webserver started at http://127.26.160.60:34971/ using document root <none> and password file <none>
14:04:11.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.167148 27422 fs_manager.cc:362] Metadata directory not provided
14:04:11.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.167423 27422 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:04:11.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.168192 27422 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:04:11.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.173027 27422 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-2/data/instance:
14:04:11.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "992a93ff1d1e4fb183da745f8d696101"
14:04:11.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:11 on dist-test-slave-8gxc"
14:04:11.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.174750 27422 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-2/wal/instance:
14:04:11.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "992a93ff1d1e4fb183da745f8d696101"
14:04:11.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:11 on dist-test-slave-8gxc"
14:04:11.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.185045 27422 fs_manager.cc:696] Time spent creating directory manager: real 0.009s	user 0.005s	sys 0.004s
14:04:11.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.192579 27448 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:04:11.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.194161 27422 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.002s
14:04:11.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.194576 27422 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-2/data,/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-2/wal
14:04:11.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "992a93ff1d1e4fb183da745f8d696101"
14:04:11.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:11 on dist-test-slave-8gxc"
14:04:11.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.195081 27422 fs_report.cc:389] FS layout report
14:04:11.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:04:11.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-2/wal
14:04:11.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-2/wal
14:04:11.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-2/data/data
14:04:11.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:04:11.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:04:11.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:04:11.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:04:11.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:04:11.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:04:11.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:04:11.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:04:11.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:04:11.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:04:11.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:04:11.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.231035 27420 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } attempt: 7
14:04:11.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:11.239351 27420 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.26.160.60:34099: Network error: Client connection negotiation failed: client connection to 127.26.160.60:34099: connect: Connection refused (error 111)
14:04:11.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.254962 27422 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:04:11.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.256723 27422 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:04:11.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.257390 27422 kserver.cc:163] Server-wide thread pool size limit: 3276
14:04:11.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.333889 27422 rpc_server.cc:307] RPC server started. Bound to: 127.26.160.60:34099
14:04:11.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.333976 27501 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.160.60:34099 every 8 connection(s)
14:04:11.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.337035 27422 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-2/data/info.pb
14:04:11.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.343509 27502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
14:04:11.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.346316 27264 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu as pid 27422
14:04:11.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.346655 27264 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/master-2/wal/instance
14:04:11.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.358150 27502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } has no permanent_uuid. Determining permanent_uuid...
14:04:11.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.389822 27502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } has no permanent_uuid. Determining permanent_uuid...
14:04:11.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.403409 27502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } has no permanent_uuid. Determining permanent_uuid...
14:04:11.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.421423 27502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101: Bootstrap starting.
14:04:11.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.428025 27502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101: Neither blocks nor log segments found. Creating new log.
14:04:11.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.430881 27502 log.cc:826] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101: Log is configured to *not* fsync() on all Append() calls
14:04:11.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.436071 27502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101: No bootstrap required, opened a new log
14:04:11.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.456526 27502 raft_consensus.cc:359] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } } peers { permanent_uuid: "52b0f69192334e17be4a1acf1702fbf3" member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } } peers { permanent_uuid: "992a93ff1d1e4fb183da745f8d696101" member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } }
14:04:11.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.457482 27502 raft_consensus.cc:385] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:04:11.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.457803 27502 raft_consensus.cc:740] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 992a93ff1d1e4fb183da745f8d696101, State: Initialized, Role: FOLLOWER
14:04:11.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.458730 27502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } } peers { permanent_uuid: "52b0f69192334e17be4a1acf1702fbf3" member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } } peers { permanent_uuid: "992a93ff1d1e4fb183da745f8d696101" member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } }
14:04:11.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.461802 27512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } } peers { permanent_uuid: "52b0f69192334e17be4a1acf1702fbf3" member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } } peers { permanent_uuid: "992a93ff1d1e4fb183da745f8d696101" member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } } }
14:04:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.462550 27512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [sys.catalog]: This master's current role is: FOLLOWER
14:04:11.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.463094 27502 sys_catalog.cc:565] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [sys.catalog]: configured and running, proceeding with master startup.
14:04:11.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:11.488768 27523 catalog_manager.cc:1593] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
14:04:11.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:11.489166 27523 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
14:04:11.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.502672 27345 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } attempt: 7
14:04:11.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.483489 27422 master_runner.cc:187] Error getting master registration for 127.26.160.62:43077: OK, instance_id { permanent_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" instance_seqno: 1779113046745972 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
14:04:11.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.509696 27422 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
14:04:11.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:11.512969 27456 tablet.cc:2404] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:04:11.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.526598 27345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66: Bootstrap starting.
14:04:11.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.532675 27345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66: Neither blocks nor log segments found. Creating new log.
14:04:11.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.534762 27345 log.cc:826] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66: Log is configured to *not* fsync() on all Append() calls
14:04:11.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.540906 27345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66: No bootstrap required, opened a new log
14:04:11.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.562513 27345 raft_consensus.cc:359] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } } peers { permanent_uuid: "52b0f69192334e17be4a1acf1702fbf3" member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } } peers { permanent_uuid: "992a93ff1d1e4fb183da745f8d696101" member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } }
14:04:11.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.563160 27345 raft_consensus.cc:385] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:04:11.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.563360 27345 raft_consensus.cc:740] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fd0d13ae5c494cbf4b47f87b7b2a66, State: Initialized, Role: FOLLOWER
14:04:11.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.565414 27345 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } } peers { permanent_uuid: "52b0f69192334e17be4a1acf1702fbf3" member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } } peers { permanent_uuid: "992a93ff1d1e4fb183da745f8d696101" member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } }
14:04:11.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.568658 27527 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } } peers { permanent_uuid: "52b0f69192334e17be4a1acf1702fbf3" member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } } peers { permanent_uuid: "992a93ff1d1e4fb183da745f8d696101" member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } } }
14:04:11.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.569336 27527 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66 [sys.catalog]: This master's current role is: FOLLOWER
14:04:11.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.570859 27345 sys_catalog.cc:565] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66 [sys.catalog]: configured and running, proceeding with master startup.
14:04:11.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:11.601894 27538 catalog_manager.cc:1593] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
14:04:11.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:11.602828 27538 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
14:04:11.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.603665 27277 master_runner.cc:187] Error getting master registration for 127.26.160.61:45315: OK, instance_id { permanent_uuid: "52b0f69192334e17be4a1acf1702fbf3" instance_seqno: 1779113048808161 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
14:04:11.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.642940 27277 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
14:04:11.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.643980 27512 raft_consensus.cc:493] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:04:11.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.644546 27512 raft_consensus.cc:515] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } } peers { permanent_uuid: "52b0f69192334e17be4a1acf1702fbf3" member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } } peers { permanent_uuid: "992a93ff1d1e4fb183da745f8d696101" member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } }
14:04:11.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.647780 27512 leader_election.cc:290] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9fd0d13ae5c494cbf4b47f87b7b2a66 (127.26.160.62:43077), 52b0f69192334e17be4a1acf1702fbf3 (127.26.160.61:45315)
14:04:11.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.649021 27320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "992a93ff1d1e4fb183da745f8d696101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" is_pre_election: true
14:04:11.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.650017 27320 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 992a93ff1d1e4fb183da745f8d696101 in term 0.
14:04:11.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.651427 27454 leader_election.cc:304] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 992a93ff1d1e4fb183da745f8d696101, e9fd0d13ae5c494cbf4b47f87b7b2a66; no voters: 
14:04:11.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.652400 27512 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [term 0 FOLLOWER]: Leader pre-election won for term 1
14:04:11.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.652746 27512 raft_consensus.cc:493] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
14:04:11.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.653082 27512 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [term 0 FOLLOWER]: Advancing to term 1
14:04:11.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.660470 27512 raft_consensus.cc:515] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } } peers { permanent_uuid: "52b0f69192334e17be4a1acf1702fbf3" member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } } peers { permanent_uuid: "992a93ff1d1e4fb183da745f8d696101" member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } }
14:04:11.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.663590 27320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "992a93ff1d1e4fb183da745f8d696101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66"
14:04:11.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.664186 27320 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66 [term 0 FOLLOWER]: Advancing to term 1
14:04:11.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.667147 27512 leader_election.cc:290] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [CANDIDATE]: Term 1 election: Requested vote from peers e9fd0d13ae5c494cbf4b47f87b7b2a66 (127.26.160.62:43077), 52b0f69192334e17be4a1acf1702fbf3 (127.26.160.61:45315)
14:04:11.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.671484 27320 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 992a93ff1d1e4fb183da745f8d696101 in term 1.
14:04:11.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.672724 27454 leader_election.cc:304] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 992a93ff1d1e4fb183da745f8d696101, e9fd0d13ae5c494cbf4b47f87b7b2a66; no voters: 
14:04:11.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.673715 27512 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [term 1 FOLLOWER]: Leader election won for term 1
14:04:11.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.677506 27512 raft_consensus.cc:697] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [term 1 LEADER]: Becoming Leader. State: Replica: 992a93ff1d1e4fb183da745f8d696101, State: Running, Role: LEADER
14:04:11.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.679113 27512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } } peers { permanent_uuid: "52b0f69192334e17be4a1acf1702fbf3" member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } } peers { permanent_uuid: "992a93ff1d1e4fb183da745f8d696101" member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } }
14:04:11.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:11.703796 27299 tablet.cc:2404] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:04:11.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.704941 27540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 992a93ff1d1e4fb183da745f8d696101. Latest consensus state: current_term: 1 leader_uuid: "992a93ff1d1e4fb183da745f8d696101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } } peers { permanent_uuid: "52b0f69192334e17be4a1acf1702fbf3" member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } } peers { permanent_uuid: "992a93ff1d1e4fb183da745f8d696101" member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } } }
14:04:11.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.706132 27540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [sys.catalog]: This master's current role is: LEADER
14:04:11.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.710009 27542 catalog_manager.cc:1510] Loading table and tablet metadata into memory...
14:04:11.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.717877 27542 catalog_manager.cc:1519] Initializing Kudu cluster ID...
14:04:11.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.747891 27320 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66 [term 1 FOLLOWER]: Refusing update from remote peer 992a93ff1d1e4fb183da745f8d696101: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:04:11.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.750597 27540 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
14:04:11.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.791193 27527 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 992a93ff1d1e4fb183da745f8d696101. Latest consensus state: current_term: 1 leader_uuid: "992a93ff1d1e4fb183da745f8d696101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } } peers { permanent_uuid: "52b0f69192334e17be4a1acf1702fbf3" member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } } peers { permanent_uuid: "992a93ff1d1e4fb183da745f8d696101" member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } } }
14:04:11.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.792562 27527 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66 [sys.catalog]: This master's current role is: FOLLOWER
14:04:11.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.802327 27543 mvcc.cc:204] Tried to move back new op lower bound from 7287247059939926016 to 7287247059689152512. Current Snapshot: MvccSnapshot[applied={T|T < 7287247059939926016}]
14:04:11.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.816424 27512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "992a93ff1d1e4fb183da745f8d696101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } } peers { permanent_uuid: "52b0f69192334e17be4a1acf1702fbf3" member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } } peers { permanent_uuid: "992a93ff1d1e4fb183da745f8d696101" member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } } }
14:04:11.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.817363 27512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [sys.catalog]: This master's current role is: LEADER
14:04:11.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.818908 27542 catalog_manager.cc:1382] Generated new cluster ID: 36e43bafab8942d0983efd0f09f73b94
14:04:11.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.819213 27542 catalog_manager.cc:1530] Initializing Kudu internal certificate authority...
14:04:11.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.854991 27527 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "992a93ff1d1e4fb183da745f8d696101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } } peers { permanent_uuid: "52b0f69192334e17be4a1acf1702fbf3" member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } } peers { permanent_uuid: "992a93ff1d1e4fb183da745f8d696101" member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } } }
14:04:11.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.855999 27527 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66 [sys.catalog]: This master's current role is: FOLLOWER
14:04:11.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.882093 27542 catalog_manager.cc:1405] Generated new certificate authority record
14:04:11.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.884367 27542 catalog_manager.cc:1539] Loading token signing keys...
14:04:11.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.923427 27542 catalog_manager.cc:6069] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101: Generated new TSK 0
14:04:11.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.925053 27542 catalog_manager.cc:1549] Initializing in-progress tserver states...
14:04:11.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:11.964440 27264 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:11.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:11.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal
14:04:11.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data
14:04:11.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:04:11.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:04:11.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:04:11.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:11.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:11.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:11.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/logs
14:04:11.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data/info.pb
14:04:11.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:11.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:04:11.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:04:11.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:04:11.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:04:11.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:11.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:11.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:11.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:04:11.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:04:11.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.1:0
14:04:11.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.1
14:04:11.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.1
14:04:11.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:04:11.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:11.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:11.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:11.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:11.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:04:12.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:12.610014 27538 catalog_manager.cc:1294] Loaded cluster ID: 36e43bafab8942d0983efd0f09f73b94
14:04:12.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:12.610316 27538 catalog_manager.cc:1587] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66: loading cluster ID for follower catalog manager: success
14:04:12.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:12.623838 27538 catalog_manager.cc:1609] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66: acquiring CA information for follower catalog manager: success
14:04:12.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:12.629810 27538 catalog_manager.cc:1637] T 00000000000000000000000000000000 P e9fd0d13ae5c494cbf4b47f87b7b2a66: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
14:04:12.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:12.747488 27548 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:04:12.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:12.748178 27548 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:04:12.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:12.748966 27548 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:04:12.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:12.814117 27548 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:04:12.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:12.815503 27548 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.160.1
14:04:12.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:12.887626 27548 tablet_server_runner.cc:78] Tablet server non-default flags:
14:04:12.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:12.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:12.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:12.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:04:12.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data
14:04:12.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal
14:04:12.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:12.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:12.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.1:0
14:04:12.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:04:12.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:12.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:12.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data/info.pb
14:04:12.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.1
14:04:12.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:04:12.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:12.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:04:12.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:12.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:04:12.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:04:12.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:12.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.1
14:04:12.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/logs
14:04:12.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:12.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:04:12.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:04:12.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:04:12.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:04:12.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:04:12.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:04:12.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:22 UTC on e7f111948823
14:04:12.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11959
14:04:12.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:04:12.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:12.889693 27548 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:04:12.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:12.892680 27548 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:04:12.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:12.987951 27554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:13.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:13.314586 27454 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.26.160.61:45315 timed out after 1.646s (SENT)
14:04:13.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.329807 27420 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } attempt: 8
14:04:13.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.596685 27420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52b0f69192334e17be4a1acf1702fbf3: Bootstrap starting.
14:04:13.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.607052 27420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52b0f69192334e17be4a1acf1702fbf3: Neither blocks nor log segments found. Creating new log.
14:04:13.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.628077 27420 log.cc:826] T 00000000000000000000000000000000 P 52b0f69192334e17be4a1acf1702fbf3: Log is configured to *not* fsync() on all Append() calls
14:04:13.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.639387 27420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52b0f69192334e17be4a1acf1702fbf3: No bootstrap required, opened a new log
14:04:13.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.721053 27420 raft_consensus.cc:359] T 00000000000000000000000000000000 P 52b0f69192334e17be4a1acf1702fbf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } } peers { permanent_uuid: "52b0f69192334e17be4a1acf1702fbf3" member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } } peers { permanent_uuid: "992a93ff1d1e4fb183da745f8d696101" member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } }
14:04:13.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.723798 27420 raft_consensus.cc:385] T 00000000000000000000000000000000 P 52b0f69192334e17be4a1acf1702fbf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:04:13.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.724891 27420 raft_consensus.cc:740] T 00000000000000000000000000000000 P 52b0f69192334e17be4a1acf1702fbf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b0f69192334e17be4a1acf1702fbf3, State: Initialized, Role: FOLLOWER
14:04:13.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.727039 27420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52b0f69192334e17be4a1acf1702fbf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } } peers { permanent_uuid: "52b0f69192334e17be4a1acf1702fbf3" member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } } peers { permanent_uuid: "992a93ff1d1e4fb183da745f8d696101" member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } }
14:04:13.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.739285 27420 sys_catalog.cc:565] T 00000000000000000000000000000000 P 52b0f69192334e17be4a1acf1702fbf3 [sys.catalog]: configured and running, proceeding with master startup.
14:04:13.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.744266 27395 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "992a93ff1d1e4fb183da745f8d696101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b0f69192334e17be4a1acf1702fbf3" is_pre_election: true
14:04:13.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.736372 27564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52b0f69192334e17be4a1acf1702fbf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } } peers { permanent_uuid: "52b0f69192334e17be4a1acf1702fbf3" member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } } peers { permanent_uuid: "992a93ff1d1e4fb183da745f8d696101" member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } } }
14:04:13.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.746567 27391 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "992a93ff1d1e4fb183da745f8d696101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b0f69192334e17be4a1acf1702fbf3" is_pre_election: true
14:04:13.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.747442 27564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52b0f69192334e17be4a1acf1702fbf3 [sys.catalog]: This master's current role is: FOLLOWER
14:04:13.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.749152 27394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "992a93ff1d1e4fb183da745f8d696101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b0f69192334e17be4a1acf1702fbf3"
14:04:13.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:13.752838 27394 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40292 (request call id 2) took 2085 ms (client timeout 1646 ms). Trace:
14:04:13.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.755712 27393 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 52b0f69192334e17be4a1acf1702fbf3 [term 0 FOLLOWER]: Advancing to term 1
14:04:13.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:13.757455 27395 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40292 (request call id 1) took 2108 ms (client timeout 2059 ms). Trace:
14:04:13.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:13.758028 27395 rpcz_store.cc:269] 0518 14:04:11.648727 (+     0us) service_pool.cc:167] Inserting onto call queue
14:04:13.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:04:11.648853 (+   126us) service_pool.cc:224] Handling call
14:04:13.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:04:13.757344 (+2108491us) inbound_call.cc:177] Queueing success response
14:04:13.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {}
14:04:13.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:13.760532 27374 tablet.cc:2404] T 00000000000000000000000000000000 P 52b0f69192334e17be4a1acf1702fbf3: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:04:13.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.741016 27392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "992a93ff1d1e4fb183da745f8d696101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b0f69192334e17be4a1acf1702fbf3"
14:04:13.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:13.760761 27394 rpcz_store.cc:269] 0518 14:04:11.667915 (+     0us) service_pool.cc:167] Inserting onto call queue
14:04:13.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:04:11.668079 (+   164us) service_pool.cc:224] Handling call
14:04:13.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:04:13.752704 (+2084625us) inbound_call.cc:177] Queueing success response
14:04:13.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {}
14:04:13.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.803308 27393 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 52b0f69192334e17be4a1acf1702fbf3 [term 1 FOLLOWER]: Refusing update from remote peer 992a93ff1d1e4fb183da745f8d696101: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:04:13.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.805069 27393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40292 (request call id 3) took 2055 ms. Trace:
14:04:13.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.806219 27393 rpcz_store.cc:276] 0518 14:04:11.749301 (+     0us) service_pool.cc:167] Inserting onto call queue
14:04:13.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:04:11.749475 (+   174us) service_pool.cc:224] Handling call
14:04:13.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:04:13.754228 (+2004753us) raft_consensus.cc:1471] Updating replica for 0 ops
14:04:13.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:04:13.804486 (+ 50258us) raft_consensus.cc:1697] Filling consensus response to leader.
14:04:13.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0518 14:04:13.804965 (+   479us) inbound_call.cc:177] Queueing success response
14:04:13.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {}
14:04:13.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.811614 27559 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52b0f69192334e17be4a1acf1702fbf3" member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
14:04:13.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.983189 27564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52b0f69192334e17be4a1acf1702fbf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 992a93ff1d1e4fb183da745f8d696101. Latest consensus state: current_term: 1 leader_uuid: "992a93ff1d1e4fb183da745f8d696101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } } peers { permanent_uuid: "52b0f69192334e17be4a1acf1702fbf3" member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } } peers { permanent_uuid: "992a93ff1d1e4fb183da745f8d696101" member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } } }
14:04:13.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.984140 27564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52b0f69192334e17be4a1acf1702fbf3 [sys.catalog]: This master's current role is: FOLLOWER
14:04:13.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.990403 27577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52b0f69192334e17be4a1acf1702fbf3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "992a93ff1d1e4fb183da745f8d696101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } } peers { permanent_uuid: "52b0f69192334e17be4a1acf1702fbf3" member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } } peers { permanent_uuid: "992a93ff1d1e4fb183da745f8d696101" member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } } }
14:04:13.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.991726 27577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52b0f69192334e17be4a1acf1702fbf3 [sys.catalog]: This master's current role is: FOLLOWER
14:04:13.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:13.997311 27581 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
14:04:14.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:14.373123 27559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "992a93ff1d1e4fb183da745f8d696101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fd0d13ae5c494cbf4b47f87b7b2a66" member_type: VOTER last_known_addr { host: "127.26.160.62" port: 43077 } } peers { permanent_uuid: "52b0f69192334e17be4a1acf1702fbf3" member_type: VOTER last_known_addr { host: "127.26.160.61" port: 45315 } } peers { permanent_uuid: "992a93ff1d1e4fb183da745f8d696101" member_type: VOTER last_known_addr { host: "127.26.160.60" port: 34099 } } }
14:04:14.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:14.373879 27559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 992a93ff1d1e4fb183da745f8d696101 [sys.catalog]: This master's current role is: LEADER
14:04:14.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:12.992134 27555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:14.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:14.329946 27553 debug-util.cc:398] Leaking SignalData structure 0x7b08000271c0 after lost signal to thread 27548
14:04:14.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:14.989468 27548 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 2.007s	user 0.551s	sys 0.962s
14:04:14.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:14.991039 27548 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 2.008s	user 0.551s	sys 0.962s
14:04:15.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:15.011978 27586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:15.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.016525 27581 catalog_manager.cc:1294] Loaded cluster ID: 36e43bafab8942d0983efd0f09f73b94
14:04:15.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.017419 27581 catalog_manager.cc:1587] T 00000000000000000000000000000000 P 52b0f69192334e17be4a1acf1702fbf3: loading cluster ID for follower catalog manager: success
14:04:15.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.019512 27548 server_base.cc:1061] running on GCE node
14:04:15.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.020928 27548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:04:15.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.031956 27548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:04:15.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.035609 27548 hybrid_clock.cc:648] HybridClock initialized: now 1779113055035523 us; error 91 us; skew 500 ppm
14:04:15.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.036762 27548 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:04:15.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.038187 27581 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 52b0f69192334e17be4a1acf1702fbf3: acquiring CA information for follower catalog manager: success
14:04:15.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.046432 27581 catalog_manager.cc:1637] T 00000000000000000000000000000000 P 52b0f69192334e17be4a1acf1702fbf3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
14:04:15.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.079030 27548 webserver.cc:533] Webserver started at http://127.26.160.1:34423/ using document root <none> and password file <none>
14:04:15.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.080374 27548 fs_manager.cc:362] Metadata directory not provided
14:04:15.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.080713 27548 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:04:15.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.081326 27548 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:04:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.088308 27548 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data/instance:
14:04:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "9f9c16da37ad440eb0d7a7a590be07a0"
14:04:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:15 on dist-test-slave-8gxc"
14:04:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.089951 27548 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal/instance:
14:04:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "9f9c16da37ad440eb0d7a7a590be07a0"
14:04:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:15 on dist-test-slave-8gxc"
14:04:15.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.106174 27548 fs_manager.cc:696] Time spent creating directory manager: real 0.015s	user 0.005s	sys 0.009s
14:04:15.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.119896 27593 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:04:15.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.121363 27548 fs_manager.cc:730] Time spent opening block manager: real 0.011s	user 0.006s	sys 0.003s
14:04:15.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.122179 27548 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data,/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal
14:04:15.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "9f9c16da37ad440eb0d7a7a590be07a0"
14:04:15.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:15 on dist-test-slave-8gxc"
14:04:15.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.122711 27548 fs_report.cc:389] FS layout report
14:04:15.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:04:15.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal
14:04:15.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal
14:04:15.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data/data
14:04:15.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:04:15.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:04:15.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:04:15.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:04:15.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:04:15.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:04:15.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:04:15.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:04:15.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:04:15.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:04:15.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:04:15.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.245338 27548 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:04:15.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.247613 27548 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:04:15.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.248215 27548 kserver.cc:163] Server-wide thread pool size limit: 3276
14:04:15.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.262707 27548 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:04:15.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.276714 27548 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
14:04:15.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.277071 27548 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
14:04:15.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.277354 27548 ts_tablet_manager.cc:616] Registered 0 tablets
14:04:15.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.277603 27548 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
14:04:15.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.562366 27548 rpc_server.cc:307] RPC server started. Bound to: 127.26.160.1:40635
14:04:15.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.562451 27706 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.160.1:40635 every 8 connection(s)
14:04:15.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.566223 27548 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data/info.pb
14:04:15.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.571364 27264 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu as pid 27548
14:04:15.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.572172 27264 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal/instance
14:04:15.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.601411 27264 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:15.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:15.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal
14:04:15.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data
14:04:15.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:04:15.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:04:15.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:04:15.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:15.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:15.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:15.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/logs
14:04:15.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data/info.pb
14:04:15.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:15.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:04:15.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:04:15.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:04:15.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:04:15.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:15.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:15.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:15.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:04:15.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:04:15.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.2:0
14:04:15.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.2
14:04:15.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.2
14:04:15.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:04:15.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:15.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:15.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:15.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:15.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:04:15.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.639873 27708 heartbeater.cc:344] Connected to a master server at 127.26.160.61:45315
14:04:15.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.640265 27707 heartbeater.cc:344] Connected to a master server at 127.26.160.60:34099
14:04:15.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.640434 27708 heartbeater.cc:461] Registering TS with master...
14:04:15.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.640793 27707 heartbeater.cc:461] Registering TS with master...
14:04:15.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.641898 27708 heartbeater.cc:507] Master 127.26.160.61:45315 requested a full tablet report, sending...
14:04:15.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.642041 27707 heartbeater.cc:507] Master 127.26.160.60:34099 requested a full tablet report, sending...
14:04:15.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.644917 27709 heartbeater.cc:344] Connected to a master server at 127.26.160.62:43077
14:04:15.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.645293 27709 heartbeater.cc:461] Registering TS with master...
14:04:15.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.645740 27467 ts_manager.cc:194] Registered new tserver with Master: 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635)
14:04:15.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.646250 27709 heartbeater.cc:507] Master 127.26.160.62:43077 requested a full tablet report, sending...
14:04:15.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.646386 27384 ts_manager.cc:194] Registered new tserver with Master: 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635)
14:04:15.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.649329 27467 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.26.160.1:51593
14:04:15.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:15.649755 27310 ts_manager.cc:194] Registered new tserver with Master: 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635)
14:04:16.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:16.012116 27717 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:04:16.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:16.012605 27717 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:04:16.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:16.013018 27717 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:04:16.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:16.051003 27717 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:04:16.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:16.052142 27717 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.160.2
14:04:16.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:16.095460 27717 tablet_server_runner.cc:78] Tablet server non-default flags:
14:04:16.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:16.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:16.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:16.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:04:16.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data
14:04:16.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal
14:04:16.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:16.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:16.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.2:0
14:04:16.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:04:16.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:16.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:16.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data/info.pb
14:04:16.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.2
14:04:16.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:04:16.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:16.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:04:16.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:16.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:04:16.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:04:16.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:16.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.2
14:04:16.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/logs
14:04:16.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:16.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:04:16.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:04:16.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:04:16.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:04:16.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:04:16.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:04:16.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:22 UTC on e7f111948823
14:04:16.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11959
14:04:16.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:04:16.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:16.096801 27717 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:04:16.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:16.098677 27717 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:04:16.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:16.115918 27723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:16.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:16.654234 27707 heartbeater.cc:499] Master 127.26.160.60:34099 was elected leader, sending a full tablet report...
14:04:18.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:17.518414 27722 debug-util.cc:398] Leaking SignalData structure 0x7b08000271c0 after lost signal to thread 27717
14:04:18.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:18.075320 27722 kernel_stack_watchdog.cc:198] Thread 27717 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
14:04:18.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
14:04:18.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
14:04:18.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
14:04:18.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
14:04:18.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:16.119446 27724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:18.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:18.076496 27717 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.961s	user 0.640s	sys 1.165s
14:04:18.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:18.076845 27717 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.962s	user 0.640s	sys 1.165s
14:04:18.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:18.081265 27725 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1960 milliseconds
14:04:18.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:18.083099 27726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:18.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.083165 27717 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:04:18.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.084513 27717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:04:18.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.087306 27717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:04:18.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.088827 27717 hybrid_clock.cc:648] HybridClock initialized: now 1779113058088716 us; error 119 us; skew 500 ppm
14:04:18.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.089759 27717 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:04:18.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.096691 27717 webserver.cc:533] Webserver started at http://127.26.160.2:34355/ using document root <none> and password file <none>
14:04:18.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.097738 27717 fs_manager.cc:362] Metadata directory not provided
14:04:18.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.097991 27717 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:04:18.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.098559 27717 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:04:18.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.103068 27717 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data/instance:
14:04:18.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d1c13f941b874a778af09e1ef1b4a7bb"
14:04:18.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:18 on dist-test-slave-8gxc"
14:04:18.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.104357 27717 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal/instance:
14:04:18.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d1c13f941b874a778af09e1ef1b4a7bb"
14:04:18.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:18 on dist-test-slave-8gxc"
14:04:18.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.112921 27717 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.006s	sys 0.003s
14:04:18.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.120774 27734 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:04:18.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.122288 27717 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.004s
14:04:18.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.122870 27717 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data,/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal
14:04:18.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d1c13f941b874a778af09e1ef1b4a7bb"
14:04:18.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:18 on dist-test-slave-8gxc"
14:04:18.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.123263 27717 fs_report.cc:389] FS layout report
14:04:18.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:04:18.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal
14:04:18.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal
14:04:18.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data/data
14:04:18.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:04:18.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:04:18.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:04:18.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:04:18.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:04:18.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:04:18.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:04:18.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:04:18.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:04:18.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:04:18.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:04:18.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.201879 27717 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:04:18.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.204077 27717 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:04:18.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.204658 27717 kserver.cc:163] Server-wide thread pool size limit: 3276
14:04:18.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.208318 27717 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:04:18.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.214468 27717 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
14:04:18.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.214778 27717 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
14:04:18.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.215088 27717 ts_tablet_manager.cc:616] Registered 0 tablets
14:04:18.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.215319 27717 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
14:04:18.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.400849 27717 rpc_server.cc:307] RPC server started. Bound to: 127.26.160.2:37023
14:04:18.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.401050 27846 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.160.2:37023 every 8 connection(s)
14:04:18.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.403744 27717 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data/info.pb
14:04:18.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.413013 27264 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu as pid 27717
14:04:18.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.413605 27264 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal/instance
14:04:18.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.428118 27264 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:18.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:18.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:04:18.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data
14:04:18.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:04:18.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:04:18.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:04:18.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:18.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:18.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:18.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/logs
14:04:18.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data/info.pb
14:04:18.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:18.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:04:18.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:04:18.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:04:18.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:04:18.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:18.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:18.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:18.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:04:18.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:04:18.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.3:0
14:04:18.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.3
14:04:18.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.3
14:04:18.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:04:18.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:18.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:18.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:18.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:18.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:04:18.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.478106 27848 heartbeater.cc:344] Connected to a master server at 127.26.160.61:45315
14:04:18.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.478675 27848 heartbeater.cc:461] Registering TS with master...
14:04:18.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.480158 27848 heartbeater.cc:507] Master 127.26.160.61:45315 requested a full tablet report, sending...
14:04:18.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.478695 27847 heartbeater.cc:344] Connected to a master server at 127.26.160.60:34099
14:04:18.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.481858 27847 heartbeater.cc:461] Registering TS with master...
14:04:18.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.482862 27849 heartbeater.cc:344] Connected to a master server at 127.26.160.62:43077
14:04:18.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.483237 27849 heartbeater.cc:461] Registering TS with master...
14:04:18.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.484169 27849 heartbeater.cc:507] Master 127.26.160.62:43077 requested a full tablet report, sending...
14:04:18.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.484967 27385 ts_manager.cc:194] Registered new tserver with Master: d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:18.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.482874 27847 heartbeater.cc:507] Master 127.26.160.60:34099 requested a full tablet report, sending...
14:04:18.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.487044 27310 ts_manager.cc:194] Registered new tserver with Master: d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:18.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.489037 27467 ts_manager.cc:194] Registered new tserver with Master: d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:18.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.490588 27467 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.26.160.2:52959
14:04:18.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:18.819401 27853 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:04:18.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:18.819998 27853 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:04:18.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:18.820426 27853 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:04:18.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:18.859839 27853 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:04:18.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:18.860715 27853 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.160.3
14:04:18.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.900406 27853 tablet_server_runner.cc:78] Tablet server non-default flags:
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.3:0
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data/info.pb
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.3
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.3
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/logs
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:22 UTC on e7f111948823
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11959
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:04:18.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.901775 27853 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:04:18.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:18.903443 27853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:04:18.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:18.917330 27864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:19.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:19.494313 27847 heartbeater.cc:499] Master 127.26.160.60:34099 was elected leader, sending a full tablet report...
14:04:20.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:18.919582 27865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:20.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:20.228717 27867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:20.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:20.232050 27866 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1307 milliseconds
14:04:20.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.232219 27853 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:04:20.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.233704 27853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:04:20.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.236475 27853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:04:20.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.238108 27853 hybrid_clock.cc:648] HybridClock initialized: now 1779113060238063 us; error 44 us; skew 500 ppm
14:04:20.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.239167 27853 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:04:20.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.246178 27853 webserver.cc:533] Webserver started at http://127.26.160.3:42189/ using document root <none> and password file <none>
14:04:20.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.247162 27853 fs_manager.cc:362] Metadata directory not provided
14:04:20.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.247351 27853 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:04:20.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.247749 27853 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
14:04:20.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.252308 27853 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data/instance:
14:04:20.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "353498eabc2240a29da45c31f7cc3c4e"
14:04:20.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:20 on dist-test-slave-8gxc"
14:04:20.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.253520 27853 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal/instance:
14:04:20.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "353498eabc2240a29da45c31f7cc3c4e"
14:04:20.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:20 on dist-test-slave-8gxc"
14:04:20.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.262212 27853 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.007s	sys 0.000s
14:04:20.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.268261 27875 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:04:20.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.269490 27853 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.001s
14:04:20.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.269955 27853 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data,/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:04:20.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "353498eabc2240a29da45c31f7cc3c4e"
14:04:20.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:20 on dist-test-slave-8gxc"
14:04:20.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.270373 27853 fs_report.cc:389] FS layout report
14:04:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:04:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:04:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:04:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data/data
14:04:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:04:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:04:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:04:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:04:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:04:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:04:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:04:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:04:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:04:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:04:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:04:20.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.327939 27853 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:04:20.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.329627 27853 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:04:20.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.330194 27853 kserver.cc:163] Server-wide thread pool size limit: 3276
14:04:20.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.332962 27853 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:04:20.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.338646 27853 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
14:04:20.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.338938 27853 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
14:04:20.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.339224 27853 ts_tablet_manager.cc:616] Registered 0 tablets
14:04:20.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.339437 27853 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
14:04:20.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.488415 27853 rpc_server.cc:307] RPC server started. Bound to: 127.26.160.3:39969
14:04:20.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.488498 27987 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.160.3:39969 every 8 connection(s)
14:04:20.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.491201 27853 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data/info.pb
14:04:20.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.493060 27264 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu as pid 27853
14:04:20.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.493486 27264 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal/instance
14:04:20.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.538614 27989 heartbeater.cc:344] Connected to a master server at 127.26.160.61:45315
14:04:20.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.539163 27989 heartbeater.cc:461] Registering TS with master...
14:04:20.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.539502 27988 heartbeater.cc:344] Connected to a master server at 127.26.160.60:34099
14:04:20.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.539844 27988 heartbeater.cc:461] Registering TS with master...
14:04:20.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.540629 27989 heartbeater.cc:507] Master 127.26.160.61:45315 requested a full tablet report, sending...
14:04:20.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.540920 27988 heartbeater.cc:507] Master 127.26.160.60:34099 requested a full tablet report, sending...
14:04:20.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.543797 27467 ts_manager.cc:194] Registered new tserver with Master: 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:04:20.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.545529 27385 ts_manager.cc:194] Registered new tserver with Master: 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:04:20.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.545686 27467 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.26.160.3:43397
14:04:20.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.549258 27990 heartbeater.cc:344] Connected to a master server at 127.26.160.62:43077
14:04:20.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.549633 27990 heartbeater.cc:461] Registering TS with master...
14:04:20.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.550431 27990 heartbeater.cc:507] Master 127.26.160.62:43077 requested a full tablet report, sending...
14:04:20.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.552712 27310 ts_manager.cc:194] Registered new tserver with Master: 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:04:20.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:20.555024 27264 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
14:04:20.566 [DEBUG - main] (MiniKuduCluster.java:195) Response: 
14:04:20.568 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}

14:04:20.577 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
  masters {
    id {
      type: MASTER
      index: 0
    }
    bound_rpc_address {
      host: "127.26.160.62"
      port: 43077
    }
    bound_http_address {
      host: "127.26.160.62"
      port: 44935
    }
  }
  masters {
    id {
      type: MASTER
      index: 1
    }
    bound_rpc_address {
      host: "127.26.160.61"
      port: 45315
    }
    bound_http_address {
      host: "127.26.160.61"
      port: 44309
    }
  }
  masters {
    id {
      type: MASTER
      index: 2
    }
    bound_rpc_address {
      host: "127.26.160.60"
      port: 34099
    }
    bound_http_address {
      host: "127.26.160.60"
      port: 34971
    }
  }
}

14:04:20.581 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}

14:04:20.583 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
  tservers {
    id {
      type: TSERVER
      index: 0
    }
    bound_rpc_address {
      host: "127.26.160.1"
      port: 40635
    }
    bound_http_address {
      host: "127.26.160.1"
      port: 34423
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 1
    }
    bound_rpc_address {
      host: "127.26.160.2"
      port: 37023
    }
    bound_http_address {
      host: "127.26.160.2"
      port: 34355
    }
  }
  tservers {
    id {
      type: TSERVER
      index: 2
    }
    bound_rpc_address {
      host: "127.26.160.3"
      port: 39969
    }
    bound_http_address {
      host: "127.26.160.3"
      port: 42189
    }
  }
}

14:04:20.584 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
14:04:20.790 [DEBUG - main] (CompositeStrategy.java:59) Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2677)
	at org.apache.kudu.client.internals.ModernStrategy.<init>(ModernStrategy.java:51)
	at org.apache.kudu.client.internals.CompositeStrategy.<init>(CompositeStrategy.java:52)
	at org.apache.kudu.client.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:41)
	at org.apache.kudu.client.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:43)
	at org.apache.kudu.client.SecurityContext.setupSubject(SecurityContext.java:155)
	at org.apache.kudu.client.SecurityContext.<init>(SecurityContext.java:138)
	at org.apache.kudu.client.AsyncKuduClient.<init>(AsyncKuduClient.java:431)
	at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.build(AsyncKuduClient.java:3109)
	at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:161)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttemptAndReport(RetryRule.java:184)
	at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:244)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
	at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
14:04:20.861 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 
14:04:21.044 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.26.160.62:43077(127.26.160.62:43077)] connecting to peer
14:04:21.109 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.26.160.60:34099(127.26.160.60:34099)] connecting to peer
14:04:21.111 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.26.160.61:45315(127.26.160.61:45315)] connecting to peer
14:04:21.133 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.26.160.62:43077(127.26.160.62:43077)] Successfully connected to peer
14:04:21.138 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.26.160.60:34099(127.26.160.60:34099)] Successfully connected to peer
14:04:21.144 [DEBUG - kudu-nio-2] (Connection.java:757) [peer master-127.26.160.61:45315(127.26.160.61:45315)] Successfully connected to peer
14:04:21.243 [DEBUG - kudu-nio-2] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.26.160.61
14:04:21.243 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.26.160.62
14:04:21.243 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism PLAIN chosen for peer 127.26.160.60
14:04:21.529 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xf0a470df, L:/127.0.0.1:37676 - R:/127.26.160.62:43077] using SASL/PLAIN
14:04:21.529 [DEBUG - kudu-nio-2] (Negotiator.java:919) Authenticated connection [id: 0xfde4568d, L:/127.0.0.1:53320 - R:/127.26.160.61:45315] using SASL/PLAIN
14:04:21.530 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xcf153406, L:/127.0.0.1:58136 - R:/127.26.160.60:34099] using SASL/PLAIN
14:04:21.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.549876 27988 heartbeater.cc:499] Master 127.26.160.60:34099 was elected leader, sending a full tablet report...
14:04:21.578 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet Kudu Master for table 'Kudu Master' with partition [<start>, <end>)
14:04:21.579 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=577), Traces: [0ms] refreshing cache from master, [25ms] Sub RPC ConnectToMaster: sending RPC to server master-127.26.160.62:43077, [103ms] Sub RPC ConnectToMaster: sending RPC to server master-127.26.160.60:34099, [105ms] Sub RPC ConnectToMaster: sending RPC to server master-127.26.160.61:45315, [544ms] Sub RPC ConnectToMaster: received response from server master-127.26.160.61:45315: OK, [544ms] Sub RPC ConnectToMaster: received response from server master-127.26.160.62:43077: OK, [544ms] Sub RPC ConnectToMaster: received response from server master-127.26.160.60:34099: OK, deferred=null) after lookup
14:04:21.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.638370 27467 catalog_manager.cc:2282] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58136:
14:04:21.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) name: "org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990"
14:04:21.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) schema {
14:04:21.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:04:21.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "key"
14:04:21.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
14:04:21.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: true
14:04:21.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:04:21.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:04:21.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:04:21.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:04:21.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:04:21.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:04:21.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column1_i"
14:04:21.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
14:04:21.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:04:21.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:04:21.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:04:21.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:04:21.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:04:21.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:04:21.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:04:21.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column2_i"
14:04:21.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: INT32
14:04:21.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:04:21.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:04:21.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:04:21.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:04:21.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:04:21.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:04:21.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:04:21.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column3_s"
14:04:21.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: STRING
14:04:21.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:04:21.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: true
14:04:21.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     encoding: DICT_ENCODING
14:04:21.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     compression: LZ4
14:04:21.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 4096
14:04:21.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:04:21.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:04:21.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:04:21.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   columns {
14:04:21.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     name: "column4_b"
14:04:21.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     type: BOOL
14:04:21.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_key: false
14:04:21.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_nullable: false
14:04:21.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     cfile_block_size: 0
14:04:21.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     immutable: false
14:04:21.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     is_auto_incrementing: false
14:04:21.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:04:21.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
14:04:21.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) partition_schema {
14:04:21.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   range_schema {
14:04:21.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     columns {
14:04:21.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)       name: "key"
14:04:21.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)     }
14:04:21.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)   }
14:04:21.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) }
14:04:21.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:21.643304 27467 catalog_manager.cc:7058] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
14:04:21.720 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table 86076ece6ce749ba921d8e1cd99c523c
14:04:21.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.733162 27642 tablet_service.cc:1511] Processing CreateTablet for tablet 266c976e4879434f8b95ee412dfafcc5 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990 [id=86076ece6ce749ba921d8e1cd99c523c]), partition=RANGE (key) PARTITION UNBOUNDED
14:04:21.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.735311 27642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266c976e4879434f8b95ee412dfafcc5. 1 dirs total, 0 dirs full, 0 dirs failed
14:04:21.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.750262 27923 tablet_service.cc:1511] Processing CreateTablet for tablet 266c976e4879434f8b95ee412dfafcc5 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990 [id=86076ece6ce749ba921d8e1cd99c523c]), partition=RANGE (key) PARTITION UNBOUNDED
14:04:21.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.752545 27923 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266c976e4879434f8b95ee412dfafcc5. 1 dirs total, 0 dirs full, 0 dirs failed
14:04:21.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.767524 27782 tablet_service.cc:1511] Processing CreateTablet for tablet 266c976e4879434f8b95ee412dfafcc5 (DEFAULT_TABLE table=org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990 [id=86076ece6ce749ba921d8e1cd99c523c]), partition=RANGE (key) PARTITION UNBOUNDED
14:04:21.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.769753 27782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266c976e4879434f8b95ee412dfafcc5. 1 dirs total, 0 dirs full, 0 dirs failed
14:04:21.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.766387 28023 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Bootstrap starting.
14:04:21.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.779152 28023 tablet_bootstrap.cc:654] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Neither blocks nor log segments found. Creating new log.
14:04:21.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.781558 28023 log.cc:826] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Log is configured to *not* fsync() on all Append() calls
14:04:21.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.794909 28024 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Bootstrap starting.
14:04:21.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.798043 28023 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: No bootstrap required, opened a new log
14:04:21.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.798588 28023 ts_tablet_manager.cc:1403] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Time spent bootstrapping tablet: real 0.035s	user 0.018s	sys 0.004s
14:04:21.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.802681 28024 tablet_bootstrap.cc:654] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Neither blocks nor log segments found. Creating new log.
14:04:21.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.812439 28026 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Bootstrap starting.
14:04:21.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.812533 28024 log.cc:826] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Log is configured to *not* fsync() on all Append() calls
14:04:21.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.828480 28026 tablet_bootstrap.cc:654] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Neither blocks nor log segments found. Creating new log.
14:04:21.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.831455 28026 log.cc:826] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Log is configured to *not* fsync() on all Append() calls
14:04:21.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.837666 28024 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: No bootstrap required, opened a new log
14:04:21.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.838680 28024 ts_tablet_manager.cc:1403] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Time spent bootstrapping tablet: real 0.045s	user 0.009s	sys 0.015s
14:04:21.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.843299 28023 raft_consensus.cc:359] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:21.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.846272 28023 raft_consensus.cc:385] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:04:21.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.846738 28023 raft_consensus.cc:740] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9c16da37ad440eb0d7a7a590be07a0, State: Initialized, Role: FOLLOWER
14:04:21.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.847838 28023 consensus_queue.cc:260] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:21.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.851545 28026 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: No bootstrap required, opened a new log
14:04:21.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.852574 28026 ts_tablet_manager.cc:1403] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Time spent bootstrapping tablet: real 0.041s	user 0.014s	sys 0.003s
14:04:21.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.860939 28023 ts_tablet_manager.cc:1434] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Time spent starting tablet: real 0.062s	user 0.034s	sys 0.006s
14:04:21.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.877794 28024 raft_consensus.cc:359] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:21.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.879315 28024 raft_consensus.cc:385] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:04:21.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.880021 28024 raft_consensus.cc:740] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353498eabc2240a29da45c31f7cc3c4e, State: Initialized, Role: FOLLOWER
14:04:21.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.881325 28024 consensus_queue.cc:260] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:21.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:21.889012 27712 tablet.cc:2404] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:04:21.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.892179 28024 ts_tablet_manager.cc:1434] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Time spent starting tablet: real 0.053s	user 0.038s	sys 0.002s
14:04:21.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.899612 28030 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:04:21.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.900575 28030 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:21.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.904489 28030 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635), d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:21.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.910921 28026 raft_consensus.cc:359] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:21.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.911794 28026 raft_consensus.cc:385] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
14:04:21.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.912072 28026 raft_consensus.cc:740] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c13f941b874a778af09e1ef1b4a7bb, State: Initialized, Role: FOLLOWER
14:04:21.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.912935 28026 consensus_queue.cc:260] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:21.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.923117 28026 ts_tablet_manager.cc:1434] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Time spent starting tablet: real 0.070s	user 0.036s	sys 0.008s
14:04:21.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.937033 27802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "353498eabc2240a29da45c31f7cc3c4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" is_pre_election: true
14:04:21.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.938270 27802 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 353498eabc2240a29da45c31f7cc3c4e in term 0.
14:04:21.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.940259 27878 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 353498eabc2240a29da45c31f7cc3c4e, d1c13f941b874a778af09e1ef1b4a7bb; no voters: 
14:04:21.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.941375 28030 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 0 FOLLOWER]: Leader pre-election won for term 1
14:04:21.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.941742 28030 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
14:04:21.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.942090 28030 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 0 FOLLOWER]: Advancing to term 1
14:04:21.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.943318 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "353498eabc2240a29da45c31f7cc3c4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" is_pre_election: true
14:04:21.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:21.943699 27852 tablet.cc:2404] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:04:21.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.944272 27662 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 353498eabc2240a29da45c31f7cc3c4e in term 0.
14:04:21.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.948047 28030 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:21.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.949702 28030 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 1 election: Requested vote from peers 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635), d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:21.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.950585 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "353498eabc2240a29da45c31f7cc3c4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9c16da37ad440eb0d7a7a590be07a0"
14:04:21.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.950821 27802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "353498eabc2240a29da45c31f7cc3c4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1c13f941b874a778af09e1ef1b4a7bb"
14:04:21.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.951231 27662 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 0 FOLLOWER]: Advancing to term 1
14:04:21.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.951323 27802 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 0 FOLLOWER]: Advancing to term 1
14:04:21.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.958328 27662 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 353498eabc2240a29da45c31f7cc3c4e in term 1.
14:04:21.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.958328 27802 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 353498eabc2240a29da45c31f7cc3c4e in term 1.
14:04:21.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.959486 27878 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 353498eabc2240a29da45c31f7cc3c4e, d1c13f941b874a778af09e1ef1b4a7bb; no voters: 
14:04:21.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.960206 28030 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 1 FOLLOWER]: Leader election won for term 1
14:04:21.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.961936 28030 raft_consensus.cc:697] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 1 LEADER]: Becoming Leader. State: Replica: 353498eabc2240a29da45c31f7cc3c4e, State: Running, Role: LEADER
14:04:21.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.962950 28030 consensus_queue.cc:237] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:21.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:21.978657 27467 catalog_manager.cc:5696] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3). New cstate: current_term: 1 leader_uuid: "353498eabc2240a29da45c31f7cc3c4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } health_report { overall_health: UNKNOWN } } }
14:04:22.013 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:894) Opened table 86076ece6ce749ba921d8e1cd99c523c
14:04:22.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:22.018055 27993 tablet.cc:2404] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
14:04:22.035 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:22.041 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299994, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:22.043 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, sequence_id=-1, TimeoutTracker(timeout=30000, elapsed=20), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [11ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=null) after lookup
14:04:22.052 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:22.055 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] Successfully connected to peer
14:04:22.135 [DEBUG - kudu-nio-3] (Negotiator.java:919) Authenticated connection [id: 0x1b5557c5, L:/127.0.0.1:54388 - R:/127.26.160.3:39969] using TOKEN/null
14:04:22.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:22.152808 27662 raft_consensus.cc:1275] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 1 FOLLOWER]: Refusing update from remote peer 353498eabc2240a29da45c31f7cc3c4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:04:22.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:22.152796 27802 raft_consensus.cc:1275] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 1 FOLLOWER]: Refusing update from remote peer 353498eabc2240a29da45c31f7cc3c4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
14:04:22.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:22.155035 28034 consensus_queue.cc:1048] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
14:04:22.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:22.155778 28030 consensus_queue.cc:1048] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
14:04:22.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:22.187042 28038 mvcc.cc:204] Tried to move back new op lower bound from 7287247102558908416 to 7287247101809258496. Current Snapshot: MvccSnapshot[applied={T|T < 7287247102558908416}]
14:04:22.361 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 3 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990, tablet=266c976e4879434f8b95ee412dfafcc5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:04:22.363 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.26.160.3:39969
14:04:22.364 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:04:22.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:22.365469 27264 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu with pid 27853
14:04:22.395 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:04:22.398 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] handling channelInactive
14:04:22.399 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state READY due to: connection closed
14:04:22.399 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: connection closed
14:04:22.409 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:22.410 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:22.410 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=2, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=14), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [3ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [13ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1232983287(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@30e2e698 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@17017f00 -> wakeup thread Time-limited test)) after lookup
14:04:22.411 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:22.433 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:22.435 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:22.435 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:22.452 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:22.452 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:22.453 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=4, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=56), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [3ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [13ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [14ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [39ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [39ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [55ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1232983287(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@30e2e698 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1236123072 after retry RPC@1065479610), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@17017f00 -> wakeup thread Time-limited test -> (continuation of Deferred@1236123072 after retry RPC@1065479610))) after lookup
14:04:22.453 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:22.456 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:22.456 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:22.456 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:22.488 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:22.489 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:22.490 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=6, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=93), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [3ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [13ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [14ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [39ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [39ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [55ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [57ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [60ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [60ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [87ms] refreshing cache from master, [87ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [92ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1232983287(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@30e2e698 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1236123072 after retry RPC@1065479610) -> (continuation of Deferred@60059932 after retry RPC@1491917175), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@17017f00 -> wakeup thread Time-limited test -> (continuation of Deferred@1236123072 after retry RPC@1065479610) -> (continuation of Deferred@60059932 after retry RPC@1491917175))) after lookup
14:04:22.491 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:22.493 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:22.493 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:22.493 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:22.589 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:22.589 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:22.590 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=8, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=194), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [3ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [13ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [14ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [39ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [39ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [55ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [57ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [60ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [60ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [87ms] refreshing cache from master, [87ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [92ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [95ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [97ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [97ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [188ms] refreshing cache from master, [188ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [192ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1232983287(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@30e2e698 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1236123072 after retry RPC@1065479610) -> (continuation of Deferred@60059932 after retry RPC@1491917175) -> (continuation of Deferred@1565631597 after retry RPC@89162409), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@17017f00 -> wakeup thread Time-limited test -> (continuation of Deferred@1236123072 after retry RPC@1065479610) -> (continuation of Deferred@60059932 after retry RPC@1491917175) -> (continuation of Deferred@1565631597 after retry RPC@89162409))) after lookup
14:04:22.591 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:22.593 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:22.593 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:22.594 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:22.849 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:22.850 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:22.850 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=10, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=454), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [3ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [13ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [14ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [39ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [39ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [55ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [57ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [60ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [60ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [87ms] refreshing cache from master, [87ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [92ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [95ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [97ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [97ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [188ms] refreshing cache from master, [188ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [192ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [195ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [198ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [198ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [448ms] refreshing cache from master, [448ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [453ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1232983287(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@30e2e698 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1236123072 after retry RPC@1065479610) -> (continuation of Deferred@60059932 after retry RPC@1491917175) -> (continuation of Deferred@1565631597 after retry RPC@89162409) -> (continuation of Deferred@283224840 after retry RPC@789913799), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@17017f00 -> wakeup thread Time-limited test -> (continuation of Deferred@1236123072 after retry RPC@1065479610) -> (continuation of Deferred@60059932 after retry RPC@1491917175) -> (continuation of Deferred@1565631597 after retry RPC@89162409) -> (continuation of Deferred@283224840 after retry RPC@789913799))) after lookup
14:04:22.852 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:22.853 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:22.853 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:22.853 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:23.469 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:23.469 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:23.470 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=12, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=1073), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [3ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [13ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [14ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [39ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [39ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [55ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [57ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [60ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [60ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [87ms] refreshing cache from master, [87ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [92ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [95ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [97ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [97ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [188ms] refreshing cache from master, [188ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [192ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [195ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [198ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [198ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [448ms] refreshing cache from master, [448ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [453ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [455ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [457ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [457ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [1067ms] refreshing cache from master, [1068ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [1072ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1232983287(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@30e2e698 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1236123072 after retry RPC@1065479610) -> (continuation of Deferred@60059932 after retry RPC@1491917175) -> (continuation of Deferred@1565631597 after retry RPC@89162409) -> (continuation of Deferred@283224840 after retry RPC@789913799) -> (continuation of Deferred@2011351276 after retry RPC@1912069583), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@17017f00 -> wakeup thread Time-limited test -> (continuation of Deferred@1236123072 after retry RPC@1065479610) -> (continuation of Deferred@60059932 after retry RPC@1491917175) -> (continuation of Deferred@1565631597 after retry RPC@89162409) -> (continuation of Deferred@283224840 after retry RPC@789913799) -> (continuation of Deferred@2011351276 after retry RPC@1912069583))) after lookup
14:04:23.471 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:23.472 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:23.472 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:23.472 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:23.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.790042 28055 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 353498eabc2240a29da45c31f7cc3c4e)
14:04:23.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.790501 28055 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:23.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.790730 28056 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 353498eabc2240a29da45c31f7cc3c4e)
14:04:23.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.791323 28056 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:23.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.792949 28055 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635), 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:04:23.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.794400 28056 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969), d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:23.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:23.798619 27598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111)
14:04:23.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:23.813566 27738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111)
14:04:23.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:23.822430 27598 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969): Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111)
14:04:23.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.822618 27802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" is_pre_election: true
14:04:23.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.822731 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" is_pre_election: true
14:04:23.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.823251 27802 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f9c16da37ad440eb0d7a7a590be07a0 in term 1.
14:04:23.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.823448 27662 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c13f941b874a778af09e1ef1b4a7bb in term 1.
14:04:23.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.824738 27596 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9f9c16da37ad440eb0d7a7a590be07a0, d1c13f941b874a778af09e1ef1b4a7bb; no voters: 353498eabc2240a29da45c31f7cc3c4e
14:04:23.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:23.825040 27738 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969): Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111)
14:04:23.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.825673 28056 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 1 FOLLOWER]: Leader pre-election won for term 2
14:04:23.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.825850 27738 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9f9c16da37ad440eb0d7a7a590be07a0, d1c13f941b874a778af09e1ef1b4a7bb; no voters: 353498eabc2240a29da45c31f7cc3c4e
14:04:23.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.826192 28056 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 353498eabc2240a29da45c31f7cc3c4e)
14:04:23.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.826795 28056 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 1 FOLLOWER]: Advancing to term 2
14:04:23.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.827198 28055 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 1 FOLLOWER]: Leader pre-election won for term 2
14:04:23.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.827533 28055 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 353498eabc2240a29da45c31f7cc3c4e)
14:04:23.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.828024 28055 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 1 FOLLOWER]: Advancing to term 2
14:04:23.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.834277 28055 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:23.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.834790 28056 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:23.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.836092 28055 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 2 election: Requested vote from peers 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635), 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:04:23.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.837328 28056 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 2 election: Requested vote from peers 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969), d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:23.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.838519 27662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9f9c16da37ad440eb0d7a7a590be07a0"
14:04:23.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.839283 27662 raft_consensus.cc:2393] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d1c13f941b874a778af09e1ef1b4a7bb in current term 2: Already voted for candidate 9f9c16da37ad440eb0d7a7a590be07a0 in this term.
14:04:23.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.838927 27802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1c13f941b874a778af09e1ef1b4a7bb"
14:04:23.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.840137 27802 raft_consensus.cc:2393] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f9c16da37ad440eb0d7a7a590be07a0 in current term 2: Already voted for candidate d1c13f941b874a778af09e1ef1b4a7bb in this term.
14:04:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:23.843397 27598 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969): Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111)
14:04:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:23.843647 27738 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969): Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111)
14:04:23.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.843911 27598 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f9c16da37ad440eb0d7a7a590be07a0; no voters: 353498eabc2240a29da45c31f7cc3c4e, d1c13f941b874a778af09e1ef1b4a7bb
14:04:23.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.844044 27738 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1c13f941b874a778af09e1ef1b4a7bb; no voters: 353498eabc2240a29da45c31f7cc3c4e, 9f9c16da37ad440eb0d7a7a590be07a0
14:04:23.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.845139 28055 raft_consensus.cc:2749] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
14:04:23.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:23.845139 28056 raft_consensus.cc:2749] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
14:04:24.888 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:24.889 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:24.889 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=14, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=2493), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [3ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [13ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [14ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [39ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [39ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [55ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [57ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [60ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [60ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [87ms] refreshing cache from master, [87ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [92ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [95ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [97ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [97ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [188ms] refreshing cache from master, [188ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [192ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [195ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [198ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [198ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [448ms] refreshing cache from master, [448ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [453ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [455ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [457ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [457ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [1067ms] refreshing cache from master, [1068ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [1072ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [1074ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [1076ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [1076ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [2488ms] refreshing cache from master, [2488ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [2492ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1232983287(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@30e2e698 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1236123072 after retry RPC@1065479610) -> (continuation of Deferred@60059932 after retry RPC@1491917175) -> (continuation of Deferred@1565631597 after retry RPC@89162409) -> (continuation of Deferred@283224840 after retry RPC@789913799) -> (continuation of Deferred@2011351276 after retry RPC@1912069583) -> (continuation of Deferred@1079951710 after retry RPC@1359611127), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@17017f00 -> wakeup thread Time-limited test -> (continuation of Deferred@1236123072 after retry RPC@1065479610) -> (continuation of Deferred@60059932 after retry RPC@1491917175) -> (continuation of Deferred@1565631597 after retry RPC@89162409) -> (continuation of Deferred@283224840 after retry RPC@789913799) -> (continuation of Deferred@2011351276 after retry RPC@1912069583) -> (continuation of Deferred@1079951710 after retry RPC@1359611127))) after lookup
14:04:24.890 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:24.892 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:24.892 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:24.893 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:26.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.024295 28069 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:04:26.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.024703 28069 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:26.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.026252 28069 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969), d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:26.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.028007 27802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" is_pre_election: true
14:04:26.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.029000 27802 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f9c16da37ad440eb0d7a7a590be07a0 in term 2.
14:04:26.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.030488 27596 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f9c16da37ad440eb0d7a7a590be07a0, d1c13f941b874a778af09e1ef1b4a7bb; no voters: 
14:04:26.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.031224 28069 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 2 FOLLOWER]: Leader pre-election won for term 3
14:04:26.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:26.031891 27598 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969): Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111)
14:04:26.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.031641 28069 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
14:04:26.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.032703 28069 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 2 FOLLOWER]: Advancing to term 3
14:04:26.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.038938 28069 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:26.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.040592 28069 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 3 election: Requested vote from peers 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969), d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:26.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.041857 27802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1c13f941b874a778af09e1ef1b4a7bb"
14:04:26.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.042464 27802 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 2 FOLLOWER]: Advancing to term 3
14:04:26.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:26.043977 27598 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969): Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111)
14:04:26.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.047755 27802 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f9c16da37ad440eb0d7a7a590be07a0 in term 3.
14:04:26.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.048868 27596 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9f9c16da37ad440eb0d7a7a590be07a0, d1c13f941b874a778af09e1ef1b4a7bb; no voters: 353498eabc2240a29da45c31f7cc3c4e
14:04:26.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.049561 28069 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 3 FOLLOWER]: Leader election won for term 3
14:04:26.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.050505 28069 raft_consensus.cc:697] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 3 LEADER]: Becoming Leader. State: Replica: 9f9c16da37ad440eb0d7a7a590be07a0, State: Running, Role: LEADER
14:04:26.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.051499 28069 consensus_queue.cc:237] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:26.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.062034 27467 catalog_manager.cc:5696] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 reported cstate change: term changed from 1 to 3, leader changed from 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3) to 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1). New cstate: current_term: 3 leader_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } health_report { overall_health: UNKNOWN } } }
14:04:26.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.488173 27802 raft_consensus.cc:1275] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 3 FOLLOWER]: Refusing update from remote peer 9f9c16da37ad440eb0d7a7a590be07a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
14:04:26.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.489636 28069 consensus_queue.cc:1048] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
14:04:26.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:26.512374 27598 consensus_peers.cc:597] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 -> Peer 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969): Couldn't send request to peer 353498eabc2240a29da45c31f7cc3c4e. Status: Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:04:26.648 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:26.648 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)[L],d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:26.649 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=16, sequence_id=4, TimeoutTracker(timeout=30000, elapsed=4253), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [3ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: connection closed, [3ms] delaying RPC due to: Network error: connection closed, [7ms] refreshing cache from master, [8ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [13ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [14ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [39ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [39ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [48ms] refreshing cache from master, [48ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [55ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [57ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [60ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [60ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [87ms] refreshing cache from master, [87ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [92ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [95ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [97ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [97ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [188ms] refreshing cache from master, [188ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [192ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [195ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [198ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [198ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [448ms] refreshing cache from master, [448ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [453ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [455ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [457ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [457ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [1067ms] refreshing cache from master, [1068ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [1072ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [1074ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [1076ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [1076ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [2488ms] refreshing cache from master, [2488ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [2492ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [2494ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [2496ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [2497ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [4247ms] refreshing cache from master, [4247ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [4252ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1232983287(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@30e2e698 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1236123072 after retry RPC@1065479610) -> (continuation of Deferred@60059932 after retry RPC@1491917175) -> (continuation of Deferred@1565631597 after retry RPC@89162409) -> (continuation of Deferred@283224840 after retry RPC@789913799) -> (continuation of Deferred@2011351276 after retry RPC@1912069583) -> (continuation of Deferred@1079951710 after retry RPC@1359611127) -> (continuation of Deferred@493960063 after retry RPC@475900314), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@17017f00 -> wakeup thread Time-limited test -> (continuation of Deferred@1236123072 after retry RPC@1065479610) -> (continuation of Deferred@60059932 after retry RPC@1491917175) -> (continuation of Deferred@1565631597 after retry RPC@89162409) -> (continuation of Deferred@283224840 after retry RPC@789913799) -> (continuation of Deferred@2011351276 after retry RPC@1912069583) -> (continuation of Deferred@1079951710 after retry RPC@1359611127) -> (continuation of Deferred@493960063 after retry RPC@475900314))) after lookup
14:04:26.651 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:26.652 [DEBUG - kudu-nio-3] (Connection.java:757) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] Successfully connected to peer
14:04:26.682 [DEBUG - kudu-nio-3] (Negotiator.java:919) Authenticated connection [id: 0xa9d39e1e, L:/127.0.0.1:52728 - R:/127.26.160.1:40635] using TOKEN/null
14:04:26.734 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.26.160.3:39969
14:04:26.736 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:04:26.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:26.738725 27264 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:26.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:26.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:04:26.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data
14:04:26.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:04:26.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:04:26.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:04:26.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:26.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:26.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/logs
14:04:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data/info.pb
14:04:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:04:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:04:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:04:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:04:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:04:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:04:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.3:39969
14:04:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.3
14:04:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42189
14:04:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.3
14:04:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:26.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:04:27.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:27.086501 28094 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:04:27.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:27.087014 28094 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:04:27.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:27.087555 28094 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:04:27.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:27.123556 28094 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:04:27.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:27.124752 28094 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.160.3
14:04:27.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:27.167368 28094 tablet_server_runner.cc:78] Tablet server non-default flags:
14:04:27.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:27.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:27.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:27.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:04:27.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data
14:04:27.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:04:27.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:27.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:27.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.3:39969
14:04:27.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:04:27.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:27.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:27.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data/info.pb
14:04:27.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.3
14:04:27.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42189
14:04:27.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:27.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:04:27.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:27.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:04:27.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:04:27.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:27.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.3
14:04:27.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/logs
14:04:27.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:27.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:04:27.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:04:27.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:04:27.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:04:27.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:04:27.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:04:27.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:22 UTC on e7f111948823
14:04:27.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11959
14:04:27.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:04:27.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:27.169121 28094 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:04:27.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:27.171139 28094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:04:27.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:27.187059 28103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:28.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:27.187909 28104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:28.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:27.197926 28106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:28.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:28.502292 28105 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1310 milliseconds
14:04:28.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.502521 28094 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:04:28.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.504112 28094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:04:28.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.506961 28094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:04:28.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.508420 28094 hybrid_clock.cc:648] HybridClock initialized: now 1779113068508368 us; error 70 us; skew 500 ppm
14:04:28.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.509388 28094 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:04:28.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.516242 28094 webserver.cc:533] Webserver started at http://127.26.160.3:42189/ using document root <none> and password file <none>
14:04:28.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.517371 28094 fs_manager.cc:362] Metadata directory not provided
14:04:28.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.517663 28094 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:04:28.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.526389 28094 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.007s	sys 0.000s
14:04:28.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.531544 28115 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:04:28.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.533347 28094 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.002s
14:04:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.534066 28094 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data,/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:04:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "353498eabc2240a29da45c31f7cc3c4e"
14:04:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:20 on dist-test-slave-8gxc"
14:04:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.536962 28094 fs_report.cc:389] FS layout report
14:04:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:04:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:04:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:04:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data/data
14:04:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:04:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:04:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:04:28.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:04:28.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:04:28.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:04:28.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:04:28.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:04:28.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:04:28.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:04:28.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:04:28.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.608191 28094 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:04:28.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.609680 28094 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:04:28.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.610198 28094 kserver.cc:163] Server-wide thread pool size limit: 3276
14:04:28.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.612828 28094 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:04:28.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.619014 28122 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:04:28.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.627946 28094 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:04:28.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.628206 28094 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.000s	sys 0.002s
14:04:28.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.628425 28094 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:04:28.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.634765 28094 ts_tablet_manager.cc:616] Registered 1 tablets
14:04:28.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.635001 28094 ts_tablet_manager.cc:595] Time spent register tablets: real 0.007s	user 0.001s	sys 0.005s
14:04:28.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.635797 28122 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Bootstrap starting.
14:04:28.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.697767 28122 log.cc:826] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Log is configured to *not* fsync() on all Append() calls
14:04:28.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.731977 28122 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:04:28.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.732973 28122 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Bootstrap complete.
14:04:28.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.735698 28122 ts_tablet_manager.cc:1403] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Time spent bootstrapping tablet: real 0.100s	user 0.059s	sys 0.036s
14:04:28.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.758796 28122 raft_consensus.cc:359] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:28.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.761734 28122 raft_consensus.cc:740] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353498eabc2240a29da45c31f7cc3c4e, State: Initialized, Role: FOLLOWER
14:04:28.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.762854 28122 consensus_queue.cc:260] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:28.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.767911 28122 ts_tablet_manager.cc:1434] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Time spent starting tablet: real 0.032s	user 0.024s	sys 0.009s
14:04:28.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.844170 28094 rpc_server.cc:307] RPC server started. Bound to: 127.26.160.3:39969
14:04:28.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.844497 28230 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.160.3:39969 every 8 connection(s)
14:04:28.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.847105 28094 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data/info.pb
14:04:28.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.857062 27264 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu as pid 28094
14:04:28.858 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:04:28.882 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 6 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990, tablet=266c976e4879434f8b95ee412dfafcc5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:04:28.883 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.26.160.1:40635
14:04:28.883 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:04:28.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.884135 27264 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu with pid 27548
14:04:28.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.895778 28231 heartbeater.cc:344] Connected to a master server at 127.26.160.60:34099
14:04:28.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.896334 28231 heartbeater.cc:461] Registering TS with master...
14:04:28.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.897368 28233 heartbeater.cc:344] Connected to a master server at 127.26.160.62:43077
14:04:28.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.897730 28233 heartbeater.cc:461] Registering TS with master...
14:04:28.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.897778 28231 heartbeater.cc:507] Master 127.26.160.60:34099 requested a full tablet report, sending...
14:04:28.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.899232 28233 heartbeater.cc:507] Master 127.26.160.62:43077 requested a full tablet report, sending...
14:04:28.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.903036 27467 ts_manager.cc:194] Re-registered known tserver with Master: 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:04:28.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.904032 27310 ts_manager.cc:194] Re-registered known tserver with Master: 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:04:28.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.906905 27467 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.26.160.3:43443
14:04:28.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.910487 28232 heartbeater.cc:344] Connected to a master server at 127.26.160.61:45315
14:04:28.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.910796 28232 heartbeater.cc:461] Registering TS with master...
14:04:28.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.911299 28232 heartbeater.cc:507] Master 127.26.160.61:45315 requested a full tablet report, sending...
14:04:28.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:28.914450 27385 ts_manager.cc:194] Re-registered known tserver with Master: 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:04:28.923 [DEBUG - kudu-nio-3] (Connection.java:239) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] handling channelInactive
14:04:28.923 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state READY due to: connection closed
14:04:28.924 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:04:28.925 [DEBUG - Time-limited test] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:28.926 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:28.926 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state CONNECTING due to: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:28.927 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:28.948 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:28.948 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)[L],d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:28.948 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=2, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=24), Traces: [0ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [2ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [3ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [23ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@187003625(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@7ac51b6e -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@40ef8273 -> wakeup thread Time-limited test)) after lookup
14:04:28.949 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:28.950 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:28.950 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state CONNECTING due to: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:28.950 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:28.968 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:28.968 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)[L],d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:28.969 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=4, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=44), Traces: [0ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [2ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [3ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [23ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [25ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [26ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [27ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [44ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@187003625(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@7ac51b6e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1973982628 after retry RPC@1231580536), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@40ef8273 -> wakeup thread Time-limited test -> (continuation of Deferred@1973982628 after retry RPC@1231580536))) after lookup
14:04:28.970 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:28.971 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:28.971 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state CONNECTING due to: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:28.971 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:29.009 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:29.009 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)[L],d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:29.010 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=6, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=85), Traces: [0ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [2ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [3ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [23ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [25ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [26ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [27ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [44ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [45ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [47ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [47ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [84ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@187003625(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@7ac51b6e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1973982628 after retry RPC@1231580536) -> (continuation of Deferred@1287387009 after retry RPC@1177698499), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@40ef8273 -> wakeup thread Time-limited test -> (continuation of Deferred@1973982628 after retry RPC@1231580536) -> (continuation of Deferred@1287387009 after retry RPC@1177698499))) after lookup
14:04:29.011 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:29.012 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:29.013 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state CONNECTING due to: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:29.013 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:29.048 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:29.048 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)[L],d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:29.049 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=8, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=124), Traces: [0ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [2ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [3ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [23ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [25ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [26ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [27ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [44ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [45ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [47ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [47ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [84ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [86ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [89ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [89ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [119ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [123ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@187003625(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@7ac51b6e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1973982628 after retry RPC@1231580536) -> (continuation of Deferred@1287387009 after retry RPC@1177698499) -> (continuation of Deferred@1102813319 after retry RPC@2090817394), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@40ef8273 -> wakeup thread Time-limited test -> (continuation of Deferred@1973982628 after retry RPC@1231580536) -> (continuation of Deferred@1287387009 after retry RPC@1177698499) -> (continuation of Deferred@1102813319 after retry RPC@2090817394))) after lookup
14:04:29.050 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:29.051 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:29.052 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state CONNECTING due to: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:29.052 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:29.108 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:29.108 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)[L],d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:29.108 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=10, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=184), Traces: [0ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [2ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [3ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [23ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [25ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [26ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [27ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [44ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [45ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [47ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [47ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [84ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [86ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [89ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [89ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [119ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [123ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [126ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [128ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [128ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [179ms] refreshing cache from master, [179ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [183ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@187003625(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@7ac51b6e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1973982628 after retry RPC@1231580536) -> (continuation of Deferred@1287387009 after retry RPC@1177698499) -> (continuation of Deferred@1102813319 after retry RPC@2090817394) -> (continuation of Deferred@919452466 after retry RPC@820993630), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@40ef8273 -> wakeup thread Time-limited test -> (continuation of Deferred@1973982628 after retry RPC@1231580536) -> (continuation of Deferred@1287387009 after retry RPC@1177698499) -> (continuation of Deferred@1102813319 after retry RPC@2090817394) -> (continuation of Deferred@919452466 after retry RPC@820993630))) after lookup
14:04:29.109 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:29.110 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:29.110 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state CONNECTING due to: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:29.110 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:29.888 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:29.889 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)[L],d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:29.889 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=12, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=965), Traces: [0ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [2ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [3ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [23ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [25ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [26ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [27ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [44ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [45ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [47ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [47ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [84ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [86ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [89ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [89ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [119ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [123ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [126ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [128ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [128ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [179ms] refreshing cache from master, [179ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [183ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [185ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [186ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [186ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [959ms] refreshing cache from master, [960ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [964ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@187003625(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@7ac51b6e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1973982628 after retry RPC@1231580536) -> (continuation of Deferred@1287387009 after retry RPC@1177698499) -> (continuation of Deferred@1102813319 after retry RPC@2090817394) -> (continuation of Deferred@919452466 after retry RPC@820993630) -> (continuation of Deferred@1322937731 after retry RPC@805335624), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@40ef8273 -> wakeup thread Time-limited test -> (continuation of Deferred@1973982628 after retry RPC@1231580536) -> (continuation of Deferred@1287387009 after retry RPC@1177698499) -> (continuation of Deferred@1102813319 after retry RPC@2090817394) -> (continuation of Deferred@919452466 after retry RPC@820993630) -> (continuation of Deferred@1322937731 after retry RPC@805335624))) after lookup
14:04:29.890 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:29.891 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:29.892 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state CONNECTING due to: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:29.892 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:29.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:29.910895 28231 heartbeater.cc:499] Master 127.26.160.60:34099 was elected leader, sending a full tablet report...
14:04:30.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:30.216135 28242 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9f9c16da37ad440eb0d7a7a590be07a0)
14:04:30.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:30.216578 28242 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:30.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:30.218323 28242 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635), 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:04:30.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:30.223232 27738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.160.1:40635: connect: Connection refused (error 111) [suppressed 1 similar messages]
14:04:30.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:30.227133 27738 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635): Network error: Client connection negotiation failed: client connection to 127.26.160.1:40635: connect: Connection refused (error 111)
14:04:30.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:30.232551 28185 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "353498eabc2240a29da45c31f7cc3c4e" is_pre_election: true
14:04:30.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:30.233292 28185 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c13f941b874a778af09e1ef1b4a7bb in term 1.
14:04:30.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:30.234643 27738 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 353498eabc2240a29da45c31f7cc3c4e, d1c13f941b874a778af09e1ef1b4a7bb; no voters: 9f9c16da37ad440eb0d7a7a590be07a0
14:04:30.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:30.235412 28242 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 3 FOLLOWER]: Leader pre-election won for term 4
14:04:30.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:30.235682 28242 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9f9c16da37ad440eb0d7a7a590be07a0)
14:04:30.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:30.235919 28242 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 3 FOLLOWER]: Advancing to term 4
14:04:30.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:30.241062 28242 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:30.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:30.242672 28242 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 4 election: Requested vote from peers 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635), 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:04:30.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:30.244274 28185 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "353498eabc2240a29da45c31f7cc3c4e"
14:04:30.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:30.244880 28185 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 1 FOLLOWER]: Advancing to term 4
14:04:30.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:30.246537 27738 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635): Network error: Client connection negotiation failed: client connection to 127.26.160.1:40635: connect: Connection refused (error 111)
14:04:30.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:30.251994 28185 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c13f941b874a778af09e1ef1b4a7bb in term 4.
14:04:30.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:30.253002 27738 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 353498eabc2240a29da45c31f7cc3c4e, d1c13f941b874a778af09e1ef1b4a7bb; no voters: 9f9c16da37ad440eb0d7a7a590be07a0
14:04:30.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:30.253840 28242 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 4 FOLLOWER]: Leader election won for term 4
14:04:30.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:30.255105 28242 raft_consensus.cc:697] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 4 LEADER]: Becoming Leader. State: Replica: d1c13f941b874a778af09e1ef1b4a7bb, State: Running, Role: LEADER
14:04:30.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:30.256305 28242 consensus_queue.cc:237] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:30.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:30.265271 27467 catalog_manager.cc:5696] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb reported cstate change: term changed from 3 to 4, leader changed from 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1) to d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2). New cstate: current_term: 4 leader_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } health_report { overall_health: HEALTHY } } }
14:04:30.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:30.789937 27738 consensus_peers.cc:597] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb -> Peer 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635): Couldn't send request to peer 9f9c16da37ad440eb0d7a7a590be07a0. Status: Network error: Client connection negotiation failed: client connection to 127.26.160.1:40635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:04:30.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:30.843216 28185 raft_consensus.cc:1275] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 4 FOLLOWER]: Refusing update from remote peer d1c13f941b874a778af09e1ef1b4a7bb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 9. (index mismatch)
14:04:30.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:30.845284 28260 consensus_queue.cc:1048] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
14:04:32.849 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:32.849 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)[L]]}]
14:04:32.849 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=14, sequence_id=7, TimeoutTracker(timeout=30000, elapsed=3925), Traces: [0ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [2ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [3ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [19ms] refreshing cache from master, [19ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [23ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [25ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [26ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [27ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [39ms] refreshing cache from master, [39ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [44ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [45ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [47ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [47ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [79ms] refreshing cache from master, [79ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [84ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [86ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [89ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [89ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [119ms] refreshing cache from master, [119ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [123ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [126ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [128ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [128ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [179ms] refreshing cache from master, [179ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [183ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [185ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [186ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [186ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [959ms] refreshing cache from master, [960ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [964ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [966ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [968ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [968ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [3920ms] refreshing cache from master, [3920ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [3924ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@187003625(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@7ac51b6e -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1973982628 after retry RPC@1231580536) -> (continuation of Deferred@1287387009 after retry RPC@1177698499) -> (continuation of Deferred@1102813319 after retry RPC@2090817394) -> (continuation of Deferred@919452466 after retry RPC@820993630) -> (continuation of Deferred@1322937731 after retry RPC@805335624) -> (continuation of Deferred@1511533806 after retry RPC@1765382103), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@40ef8273 -> wakeup thread Time-limited test -> (continuation of Deferred@1973982628 after retry RPC@1231580536) -> (continuation of Deferred@1287387009 after retry RPC@1177698499) -> (continuation of Deferred@1102813319 after retry RPC@2090817394) -> (continuation of Deferred@919452466 after retry RPC@820993630) -> (continuation of Deferred@1322937731 after retry RPC@805335624) -> (continuation of Deferred@1511533806 after retry RPC@1765382103))) after lookup
14:04:32.850 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:04:32.855 [DEBUG - kudu-nio-3] (Connection.java:757) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] Successfully connected to peer
14:04:32.882 [DEBUG - kudu-nio-3] (Negotiator.java:919) Authenticated connection [id: 0xa308a05f, L:/127.0.0.1:54982 - R:/127.26.160.2:37023] using TOKEN/null
14:04:32.954 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.26.160.1:40635
14:04:32.955 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:04:32.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:32.956617 27264 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/logs
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data/info.pb
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.1:40635
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.1
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34423
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.1
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:32.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:32.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:32.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:04:33.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:33.288367 28287 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:04:33.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:33.289265 28287 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:04:33.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:33.290323 28287 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:04:33.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:33.329728 28287 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:04:33.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:33.330832 28287 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.160.1
14:04:33.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:33.341307 27738 consensus_peers.cc:597] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb -> Peer 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635): Couldn't send request to peer 9f9c16da37ad440eb0d7a7a590be07a0. Status: Network error: Client connection negotiation failed: client connection to 127.26.160.1:40635: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:04:33.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:33.374476 28287 tablet_server_runner.cc:78] Tablet server non-default flags:
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.1:40635
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data/info.pb
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.1
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34423
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.1
14:04:33.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/logs
14:04:33.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:33.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:04:33.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:04:33.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:04:33.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:04:33.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:04:33.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:04:33.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:22 UTC on e7f111948823
14:04:33.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11959
14:04:33.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:04:33.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:33.376317 28287 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:04:33.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:33.378069 28287 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:04:33.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:33.392175 28293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:35.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:34.796707 28292 debug-util.cc:398] Leaking SignalData structure 0x7b08000271c0 after lost signal to thread 28287
14:04:35.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:35.019045 28287 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.627s	user 0.545s	sys 1.068s
14:04:35.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:35.019441 28287 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.628s	user 0.545s	sys 1.068s
14:04:35.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:33.392374 28294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:35.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:35.023392 28296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:35.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:35.034514 28295 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1628 milliseconds
14:04:35.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.034530 28287 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:04:35.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.036988 28287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:04:35.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.039336 28287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:04:35.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.040783 28287 hybrid_clock.cc:648] HybridClock initialized: now 1779113075040753 us; error 104 us; skew 500 ppm
14:04:35.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.041749 28287 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:04:35.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.049139 28287 webserver.cc:533] Webserver started at http://127.26.160.1:34423/ using document root <none> and password file <none>
14:04:35.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.050278 28287 fs_manager.cc:362] Metadata directory not provided
14:04:35.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.050508 28287 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:04:35.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.060217 28287 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.000s	sys 0.004s
14:04:35.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.068221 28307 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:04:35.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.069646 28287 fs_manager.cc:730] Time spent opening block manager: real 0.006s	user 0.005s	sys 0.001s
14:04:35.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.070129 28287 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data,/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal
14:04:35.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "9f9c16da37ad440eb0d7a7a590be07a0"
14:04:35.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:15 on dist-test-slave-8gxc"
14:04:35.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.072294 28287 fs_report.cc:389] FS layout report
14:04:35.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:04:35.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal
14:04:35.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal
14:04:35.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data/data
14:04:35.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:04:35.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:04:35.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:04:35.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:04:35.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:04:35.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:04:35.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:04:35.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:04:35.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:04:35.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:04:35.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:04:35.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.133390 28287 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:04:35.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.135109 28287 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:04:35.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.135588 28287 kserver.cc:163] Server-wide thread pool size limit: 3276
14:04:35.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.139627 28287 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:04:35.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.146914 28315 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:04:35.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.157395 28287 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:04:35.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.157660 28287 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.013s	user 0.000s	sys 0.003s
14:04:35.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.158125 28287 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:04:35.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.164527 28287 ts_tablet_manager.cc:616] Registered 1 tablets
14:04:35.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.164752 28287 ts_tablet_manager.cc:595] Time spent register tablets: real 0.007s	user 0.004s	sys 0.000s
14:04:35.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.166043 28315 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Bootstrap starting.
14:04:35.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.238682 28315 log.cc:826] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Log is configured to *not* fsync() on all Append() calls
14:04:35.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.285482 28315 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=6 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:04:35.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.286384 28315 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Bootstrap complete.
14:04:35.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.288213 28315 ts_tablet_manager.cc:1403] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Time spent bootstrapping tablet: real 0.123s	user 0.074s	sys 0.043s
14:04:35.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.310633 28315 raft_consensus.cc:359] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:35.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.315143 28315 raft_consensus.cc:740] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9c16da37ad440eb0d7a7a590be07a0, State: Initialized, Role: FOLLOWER
14:04:35.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.316259 28315 consensus_queue.cc:260] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 3.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:35.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.321730 28315 ts_tablet_manager.cc:1434] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Time spent starting tablet: real 0.033s	user 0.029s	sys 0.004s
14:04:35.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.370275 28287 rpc_server.cc:307] RPC server started. Bound to: 127.26.160.1:40635
14:04:35.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.370570 28423 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.160.1:40635 every 8 connection(s)
14:04:35.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.373144 28287 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data/info.pb
14:04:35.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.375075 27264 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu as pid 28287
14:04:35.376 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:04:35.406 [DEBUG - kudu-nio-3] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 9 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990, tablet=266c976e4879434f8b95ee412dfafcc5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:04:35.407 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.26.160.2:37023
14:04:35.407 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:04:35.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.407917 27264 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu with pid 27717
14:04:35.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.412130 28424 heartbeater.cc:344] Connected to a master server at 127.26.160.60:34099
14:04:35.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.412722 28424 heartbeater.cc:461] Registering TS with master...
14:04:35.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.414248 28424 heartbeater.cc:507] Master 127.26.160.60:34099 requested a full tablet report, sending...
14:04:35.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.418485 27467 ts_manager.cc:194] Re-registered known tserver with Master: 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635)
14:04:35.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.421463 27467 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.26.160.1:37165
14:04:35.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.424461 28425 heartbeater.cc:344] Connected to a master server at 127.26.160.61:45315
14:04:35.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.424822 28425 heartbeater.cc:461] Registering TS with master...
14:04:35.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.425874 28425 heartbeater.cc:507] Master 127.26.160.61:45315 requested a full tablet report, sending...
14:04:35.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.428184 28426 heartbeater.cc:344] Connected to a master server at 127.26.160.62:43077
14:04:35.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.428526 28426 heartbeater.cc:461] Registering TS with master...
14:04:35.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.429194 28426 heartbeater.cc:507] Master 127.26.160.62:43077 requested a full tablet report, sending...
14:04:35.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.429731 27385 ts_manager.cc:194] Re-registered known tserver with Master: 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635)
14:04:35.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:35.433038 27310 ts_manager.cc:194] Re-registered known tserver with Master: 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635)
14:04:35.444 [DEBUG - kudu-nio-3] (Connection.java:239) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] handling channelInactive
14:04:35.445 [DEBUG - kudu-nio-3] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state READY due to: connection closed
14:04:35.445 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:04:35.446 [DEBUG - Time-limited test] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:04:35.448 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:35.448 [DEBUG - kudu-nio-4] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state CONNECTING due to: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:35.448 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:35.468 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:35.469 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)[L]]}]
14:04:35.469 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=2, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=23), Traces: [0ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [2ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [2ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [21ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1498547740(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@7f1e8217 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6983cb1d -> wakeup thread Time-limited test)) after lookup
14:04:35.470 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:04:35.471 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:35.471 [DEBUG - kudu-nio-5] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state CONNECTING due to: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:35.471 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:35.488 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:35.488 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)[L]]}]
14:04:35.489 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=4, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=43), Traces: [0ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [2ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [2ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [21ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [23ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [25ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [25ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [42ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1498547740(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@7f1e8217 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2038656243 after retry RPC@237997053), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6983cb1d -> wakeup thread Time-limited test -> (continuation of Deferred@2038656243 after retry RPC@237997053))) after lookup
14:04:35.489 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:04:35.490 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:35.491 [DEBUG - kudu-nio-6] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state CONNECTING due to: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:35.491 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:35.508 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:35.509 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)[L]]}]
14:04:35.509 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=6, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=63), Traces: [0ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [2ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [2ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [21ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [23ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [25ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [25ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [42ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [43ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [45ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [45ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [57ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [62ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1498547740(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@7f1e8217 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2038656243 after retry RPC@237997053) -> (continuation of Deferred@2039205461 after retry RPC@185193210), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6983cb1d -> wakeup thread Time-limited test -> (continuation of Deferred@2038656243 after retry RPC@237997053) -> (continuation of Deferred@2039205461 after retry RPC@185193210))) after lookup
14:04:35.510 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:04:35.512 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:35.512 [DEBUG - kudu-nio-7] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state CONNECTING due to: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:35.512 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:35.648 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:35.649 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)[L]]}]
14:04:35.649 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=8, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=203), Traces: [0ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [2ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [2ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [21ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [23ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [25ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [25ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [42ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [43ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [45ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [45ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [57ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [62ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [64ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [66ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [66ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [197ms] refreshing cache from master, [197ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [202ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1498547740(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@7f1e8217 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2038656243 after retry RPC@237997053) -> (continuation of Deferred@2039205461 after retry RPC@185193210) -> (continuation of Deferred@2022701434 after retry RPC@1037335187), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6983cb1d -> wakeup thread Time-limited test -> (continuation of Deferred@2038656243 after retry RPC@237997053) -> (continuation of Deferred@2039205461 after retry RPC@185193210) -> (continuation of Deferred@2022701434 after retry RPC@1037335187))) after lookup
14:04:35.650 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:04:35.651 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:35.651 [DEBUG - kudu-nio-0] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state CONNECTING due to: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:35.651 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:36.149 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:36.150 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)[L]]}]
14:04:36.151 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=10, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=705), Traces: [0ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [2ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [2ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [21ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [23ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [25ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [25ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [42ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [43ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [45ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [45ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [57ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [62ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [64ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [66ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [66ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [197ms] refreshing cache from master, [197ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [202ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [203ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [205ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [205ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [698ms] refreshing cache from master, [698ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [703ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1498547740(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@7f1e8217 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2038656243 after retry RPC@237997053) -> (continuation of Deferred@2039205461 after retry RPC@185193210) -> (continuation of Deferred@2022701434 after retry RPC@1037335187) -> (continuation of Deferred@1189046941 after retry RPC@1480691767), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6983cb1d -> wakeup thread Time-limited test -> (continuation of Deferred@2038656243 after retry RPC@237997053) -> (continuation of Deferred@2039205461 after retry RPC@185193210) -> (continuation of Deferred@2022701434 after retry RPC@1037335187) -> (continuation of Deferred@1189046941 after retry RPC@1480691767))) after lookup
14:04:36.152 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:04:36.154 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:36.154 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state CONNECTING due to: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:36.154 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:36.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.430517 28424 heartbeater.cc:499] Master 127.26.160.60:34099 was elected leader, sending a full tablet report...
14:04:36.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.679306 28434 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:04:36.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.679893 28434 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:36.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.682121 28434 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969), d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:36.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:36.686642 28311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.160.2:37023: connect: Connection refused (error 111)
14:04:36.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:36.690927 28311 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023): Network error: Client connection negotiation failed: client connection to 127.26.160.2:37023: connect: Connection refused (error 111)
14:04:36.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.694926 28185 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "353498eabc2240a29da45c31f7cc3c4e" is_pre_election: true
14:04:36.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.696372 28312 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f9c16da37ad440eb0d7a7a590be07a0; no voters: 353498eabc2240a29da45c31f7cc3c4e, d1c13f941b874a778af09e1ef1b4a7bb
14:04:36.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.697168 28434 raft_consensus.cc:2749] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
14:04:36.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.816845 28439 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d1c13f941b874a778af09e1ef1b4a7bb)
14:04:36.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.817286 28439 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:36.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.819831 28439 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635), d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:36.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:36.824980 28118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.160.2:37023: connect: Connection refused (error 111)
14:04:36.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:36.829658 28118 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023): Network error: Client connection negotiation failed: client connection to 127.26.160.2:37023: connect: Connection refused (error 111)
14:04:36.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.833882 28378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "353498eabc2240a29da45c31f7cc3c4e" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" is_pre_election: true
14:04:36.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.834690 28378 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 353498eabc2240a29da45c31f7cc3c4e in term 3.
14:04:36.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.836287 28119 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 353498eabc2240a29da45c31f7cc3c4e, 9f9c16da37ad440eb0d7a7a590be07a0; no voters: d1c13f941b874a778af09e1ef1b4a7bb
14:04:36.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.837149 28439 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 4 FOLLOWER]: Leader pre-election won for term 5
14:04:36.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.837476 28439 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 4 FOLLOWER]: Starting leader election (detected failure of leader d1c13f941b874a778af09e1ef1b4a7bb)
14:04:36.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.837841 28439 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 4 FOLLOWER]: Advancing to term 5
14:04:36.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.844262 28439 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:36.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.846226 28439 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 5 election: Requested vote from peers 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635), d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:36.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.847179 28378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "353498eabc2240a29da45c31f7cc3c4e" candidate_term: 5 candidate_status { last_received { term: 4 index: 12 } } ignore_live_leader: false dest_uuid: "9f9c16da37ad440eb0d7a7a590be07a0"
14:04:36.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.847726 28378 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 3 FOLLOWER]: Advancing to term 5
14:04:36.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.858242 28378 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 353498eabc2240a29da45c31f7cc3c4e in term 5.
14:04:36.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.859584 28119 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 353498eabc2240a29da45c31f7cc3c4e, 9f9c16da37ad440eb0d7a7a590be07a0; no voters: 
14:04:36.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.860713 28439 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 5 FOLLOWER]: Leader election won for term 5
14:04:36.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:36.861306 28118 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023): Network error: Client connection negotiation failed: client connection to 127.26.160.2:37023: connect: Connection refused (error 111)
14:04:36.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.863376 28439 raft_consensus.cc:697] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 5 LEADER]: Becoming Leader. State: Replica: 353498eabc2240a29da45c31f7cc3c4e, State: Running, Role: LEADER
14:04:36.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.865306 28439 consensus_queue.cc:237] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:36.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:36.877602 27467 catalog_manager.cc:5696] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e reported cstate change: term changed from 4 to 5, leader changed from d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2) to 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3). New cstate: current_term: 5 leader_uuid: "353498eabc2240a29da45c31f7cc3c4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } health_report { overall_health: UNKNOWN } } }
14:04:37.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:37.295143 28378 raft_consensus.cc:1275] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 5 FOLLOWER]: Refusing update from remote peer 353498eabc2240a29da45c31f7cc3c4e: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 5 index: 13. (index mismatch)
14:04:37.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:37.297114 28439 consensus_queue.cc:1048] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.001s
14:04:37.309 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:37.309 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:37.310 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=12, sequence_id=10, TimeoutTracker(timeout=30000, elapsed=1864), Traces: [0ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [2ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [2ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [17ms] refreshing cache from master, [17ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [21ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [23ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [25ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [25ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [38ms] refreshing cache from master, [38ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [42ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [43ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [45ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [45ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [57ms] refreshing cache from master, [58ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [62ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [64ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [66ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [66ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [197ms] refreshing cache from master, [197ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [202ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [203ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [205ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [205ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [698ms] refreshing cache from master, [698ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [703ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [705ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [708ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [708ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [1857ms] refreshing cache from master, [1858ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [1863ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1498547740(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@7f1e8217 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@2038656243 after retry RPC@237997053) -> (continuation of Deferred@2039205461 after retry RPC@185193210) -> (continuation of Deferred@2022701434 after retry RPC@1037335187) -> (continuation of Deferred@1189046941 after retry RPC@1480691767) -> (continuation of Deferred@1570789159 after retry RPC@318681941), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6983cb1d -> wakeup thread Time-limited test -> (continuation of Deferred@2038656243 after retry RPC@237997053) -> (continuation of Deferred@2039205461 after retry RPC@185193210) -> (continuation of Deferred@2022701434 after retry RPC@1037335187) -> (continuation of Deferred@1189046941 after retry RPC@1480691767) -> (continuation of Deferred@1570789159 after retry RPC@318681941))) after lookup
14:04:37.312 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:37.313 [DEBUG - kudu-nio-2] (Connection.java:757) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] Successfully connected to peer
14:04:37.365 [DEBUG - kudu-nio-2] (Negotiator.java:919) Authenticated connection [id: 0x94c76493, L:/127.0.0.1:39536 - R:/127.26.160.3:39969] using TOKEN/null
14:04:37.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:37.421772 28118 consensus_peers.cc:597] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e -> Peer d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023): Couldn't send request to peer d1c13f941b874a778af09e1ef1b4a7bb. Status: Network error: Client connection negotiation failed: client connection to 127.26.160.2:37023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:04:37.467 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.26.160.2:37023
14:04:37.467 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:37.468629 27264 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal
14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data
14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/logs
14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data/info.pb
14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:04:37.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:04:37.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.2:37023
14:04:37.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.2
14:04:37.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:37.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34355
14:04:37.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.2
14:04:37.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:37.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:37.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:37.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:04:37.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:37.815403 28470 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:04:37.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:37.815940 28470 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:04:37.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:37.816589 28470 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:04:37.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:37.858290 28470 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:04:37.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:37.859439 28470 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.160.2
14:04:37.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:37.910233 28470 tablet_server_runner.cc:78] Tablet server non-default flags:
14:04:37.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:37.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.2:37023
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data/info.pb
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.2
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34355
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.2
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/logs
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:04:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:22 UTC on e7f111948823
14:04:37.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11959
14:04:37.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:04:37.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:37.912060 28470 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:04:37.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:37.914705 28470 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:04:37.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:37.932569 28476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:39.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:37.933254 28477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:39.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:39.270717 28479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:39.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:39.273810 28478 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1340 milliseconds
14:04:39.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.273891 28470 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:04:39.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.275225 28470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:04:39.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.277408 28470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:04:39.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.278813 28470 hybrid_clock.cc:648] HybridClock initialized: now 1779113079278799 us; error 61 us; skew 500 ppm
14:04:39.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.279676 28470 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:04:39.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.286484 28470 webserver.cc:533] Webserver started at http://127.26.160.2:34355/ using document root <none> and password file <none>
14:04:39.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.287485 28470 fs_manager.cc:362] Metadata directory not provided
14:04:39.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.287695 28470 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:04:39.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:39.294085 27416 debug-util.cc:398] Leaking SignalData structure 0x7b0800076100 after lost signal to thread 27351
14:04:39.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:39.294940 27416 debug-util.cc:398] Leaking SignalData structure 0x7b0800079de0 after lost signal to thread 27419
14:04:39.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.296252 28470 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.001s	sys 0.004s
14:04:39.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.301116 28489 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:04:39.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.302392 28470 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
14:04:39.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.302695 28470 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data,/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal
14:04:39.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d1c13f941b874a778af09e1ef1b4a7bb"
14:04:39.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:18 on dist-test-slave-8gxc"
14:04:39.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.304909 28470 fs_report.cc:389] FS layout report
14:04:39.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:04:39.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal
14:04:39.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal
14:04:39.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data/data
14:04:39.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:04:39.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:04:39.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:04:39.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:04:39.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:04:39.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:04:39.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:04:39.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:04:39.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:04:39.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:04:39.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:04:39.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.378093 28470 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:04:39.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.380673 28470 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:04:39.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.381263 28470 kserver.cc:163] Server-wide thread pool size limit: 3276
14:04:39.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.407344 28470 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:04:39.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.416292 28496 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:04:39.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.426589 28470 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:04:39.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.426908 28470 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.013s	user 0.002s	sys 0.000s
14:04:39.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.427232 28470 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:04:39.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.435019 28470 ts_tablet_manager.cc:616] Registered 1 tablets
14:04:39.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.435304 28470 ts_tablet_manager.cc:595] Time spent register tablets: real 0.008s	user 0.007s	sys 0.000s
14:04:39.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.435742 28496 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Bootstrap starting.
14:04:39.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.514346 28496 log.cc:826] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Log is configured to *not* fsync() on all Append() calls
14:04:39.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.587512 28496 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:04:39.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.588815 28496 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Bootstrap complete.
14:04:39.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.591023 28496 ts_tablet_manager.cc:1403] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Time spent bootstrapping tablet: real 0.156s	user 0.110s	sys 0.039s
14:04:39.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.620918 28496 raft_consensus.cc:359] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:39.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.625883 28496 raft_consensus.cc:740] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c13f941b874a778af09e1ef1b4a7bb, State: Initialized, Role: FOLLOWER
14:04:39.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.627513 28496 consensus_queue.cc:260] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:39.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.633280 28496 ts_tablet_manager.cc:1434] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Time spent starting tablet: real 0.042s	user 0.024s	sys 0.016s
14:04:39.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.643942 28470 rpc_server.cc:307] RPC server started. Bound to: 127.26.160.2:37023
14:04:39.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.644297 28604 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.160.2:37023 every 8 connection(s)
14:04:39.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.647056 28470 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data/info.pb
14:04:39.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.648865 27264 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu as pid 28470
14:04:39.649 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:04:39.687 [DEBUG - kudu-nio-2] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 12 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990, tablet=266c976e4879434f8b95ee412dfafcc5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:04:39.699 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.26.160.3:39969
14:04:39.700 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:04:39.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.700457 27264 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu with pid 28094
14:04:39.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.733186 28605 heartbeater.cc:344] Connected to a master server at 127.26.160.60:34099
14:04:39.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.733721 28605 heartbeater.cc:461] Registering TS with master...
14:04:39.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.735057 28605 heartbeater.cc:507] Master 127.26.160.60:34099 requested a full tablet report, sending...
14:04:39.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.740137 27467 ts_manager.cc:194] Re-registered known tserver with Master: d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:39.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.744130 27467 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.26.160.2:59877
14:04:39.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.755785 28610 heartbeater.cc:344] Connected to a master server at 127.26.160.62:43077
14:04:39.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.756182 28610 heartbeater.cc:461] Registering TS with master...
14:04:39.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.757019 28610 heartbeater.cc:507] Master 127.26.160.62:43077 requested a full tablet report, sending...
14:04:39.763 [DEBUG - kudu-nio-2] (Connection.java:239) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] handling channelInactive
14:04:39.764 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state READY due to: connection closed
14:04:39.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.763893 27310 ts_manager.cc:194] Re-registered known tserver with Master: d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:39.767 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:04:39.768 [DEBUG - Time-limited test] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:39.769 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:39.769 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:39.770 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:39.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.770962 28606 heartbeater.cc:344] Connected to a master server at 127.26.160.61:45315
14:04:39.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.771360 28606 heartbeater.cc:461] Registering TS with master...
14:04:39.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.772068 28606 heartbeater.cc:507] Master 127.26.160.61:45315 requested a full tablet report, sending...
14:04:39.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:39.776319 27385 ts_manager.cc:194] Re-registered known tserver with Master: d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:39.789 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:39.789 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:39.789 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=2, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=22), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [3ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [3ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [21ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1047051343(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@596bd4cc -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@42f98cac -> wakeup thread Time-limited test)) after lookup
14:04:39.790 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:39.792 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:39.792 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:39.792 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:39.808 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:39.808 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:39.809 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=4, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=41), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [3ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [3ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [21ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [23ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [25ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [25ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [40ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1047051343(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@596bd4cc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@692741386 after retry RPC@660082787), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@42f98cac -> wakeup thread Time-limited test -> (continuation of Deferred@692741386 after retry RPC@660082787))) after lookup
14:04:39.810 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:39.811 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:39.811 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:39.812 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:39.849 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:39.850 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:39.850 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=6, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=82), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [3ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [3ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [21ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [23ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [25ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [25ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [40ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [42ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [45ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [45ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [82ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1047051343(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@596bd4cc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@692741386 after retry RPC@660082787) -> (continuation of Deferred@744585762 after retry RPC@1272822981), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@42f98cac -> wakeup thread Time-limited test -> (continuation of Deferred@692741386 after retry RPC@660082787) -> (continuation of Deferred@744585762 after retry RPC@1272822981))) after lookup
14:04:39.851 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:39.853 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:39.853 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:39.853 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:39.968 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:39.969 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:39.969 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=8, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=201), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [3ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [3ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [21ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [23ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [25ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [25ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [40ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [42ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [45ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [45ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [82ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [84ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [86ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [86ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [197ms] refreshing cache from master, [197ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [201ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1047051343(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@596bd4cc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@692741386 after retry RPC@660082787) -> (continuation of Deferred@744585762 after retry RPC@1272822981) -> (continuation of Deferred@1547722030 after retry RPC@2052975095), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@42f98cac -> wakeup thread Time-limited test -> (continuation of Deferred@692741386 after retry RPC@660082787) -> (continuation of Deferred@744585762 after retry RPC@1272822981) -> (continuation of Deferred@1547722030 after retry RPC@2052975095))) after lookup
14:04:39.971 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:39.972 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:39.972 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:39.973 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:40.229 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:40.229 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:40.230 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=10, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=462), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [3ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [3ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [21ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [23ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [25ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [25ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [40ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [42ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [45ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [45ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [82ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [84ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [86ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [86ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [197ms] refreshing cache from master, [197ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [201ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [203ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [206ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [206ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [456ms] refreshing cache from master, [457ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [462ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1047051343(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@596bd4cc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@692741386 after retry RPC@660082787) -> (continuation of Deferred@744585762 after retry RPC@1272822981) -> (continuation of Deferred@1547722030 after retry RPC@2052975095) -> (continuation of Deferred@2018089453 after retry RPC@1833878850), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@42f98cac -> wakeup thread Time-limited test -> (continuation of Deferred@692741386 after retry RPC@660082787) -> (continuation of Deferred@744585762 after retry RPC@1272822981) -> (continuation of Deferred@1547722030 after retry RPC@2052975095) -> (continuation of Deferred@2018089453 after retry RPC@1833878850))) after lookup
14:04:40.231 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:40.232 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:40.232 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:40.233 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:40.608 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:40.608 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:40.608 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=12, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=841), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [3ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [3ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [21ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [23ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [25ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [25ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [40ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [42ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [45ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [45ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [82ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [84ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [86ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [86ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [197ms] refreshing cache from master, [197ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [201ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [203ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [206ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [206ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [456ms] refreshing cache from master, [457ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [462ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [463ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [465ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [466ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [837ms] refreshing cache from master, [837ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [840ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1047051343(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@596bd4cc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@692741386 after retry RPC@660082787) -> (continuation of Deferred@744585762 after retry RPC@1272822981) -> (continuation of Deferred@1547722030 after retry RPC@2052975095) -> (continuation of Deferred@2018089453 after retry RPC@1833878850) -> (continuation of Deferred@435310785 after retry RPC@34464360), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@42f98cac -> wakeup thread Time-limited test -> (continuation of Deferred@692741386 after retry RPC@660082787) -> (continuation of Deferred@744585762 after retry RPC@1272822981) -> (continuation of Deferred@1547722030 after retry RPC@2052975095) -> (continuation of Deferred@2018089453 after retry RPC@1833878850) -> (continuation of Deferred@435310785 after retry RPC@34464360))) after lookup
14:04:40.609 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:40.610 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:40.610 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:40.611 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:40.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:40.757180 28605 heartbeater.cc:499] Master 127.26.160.60:34099 was elected leader, sending a full tablet report...
14:04:40.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:40.901301 28616 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 353498eabc2240a29da45c31f7cc3c4e)
14:04:40.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:40.901736 28616 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:40.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:40.903589 28616 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969), d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:40.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:40.923413 28312 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969): Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111)
14:04:40.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:40.929647 28559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" is_pre_election: true
14:04:40.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:40.930457 28559 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f9c16da37ad440eb0d7a7a590be07a0 in term 4.
14:04:40.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:40.931667 28311 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9f9c16da37ad440eb0d7a7a590be07a0, d1c13f941b874a778af09e1ef1b4a7bb; no voters: 353498eabc2240a29da45c31f7cc3c4e
14:04:40.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:40.932273 28616 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 5 FOLLOWER]: Leader pre-election won for term 6
14:04:40.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:40.932600 28616 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 353498eabc2240a29da45c31f7cc3c4e)
14:04:40.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:40.932835 28616 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 5 FOLLOWER]: Advancing to term 6
14:04:40.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:40.937323 28616 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:40.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:40.938983 28616 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 6 election: Requested vote from peers 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969), d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:40.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:40.940223 28559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 16 } } ignore_live_leader: false dest_uuid: "d1c13f941b874a778af09e1ef1b4a7bb"
14:04:40.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:40.940887 28559 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 4 FOLLOWER]: Advancing to term 6
14:04:40.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:40.942514 28312 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969): Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111)
14:04:40.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:40.952194 28559 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f9c16da37ad440eb0d7a7a590be07a0 in term 6.
14:04:40.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:40.953433 28311 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9f9c16da37ad440eb0d7a7a590be07a0, d1c13f941b874a778af09e1ef1b4a7bb; no voters: 353498eabc2240a29da45c31f7cc3c4e
14:04:40.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:40.954171 28616 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 6 FOLLOWER]: Leader election won for term 6
14:04:40.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:40.955642 28616 raft_consensus.cc:697] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 6 LEADER]: Becoming Leader. State: Replica: 9f9c16da37ad440eb0d7a7a590be07a0, State: Running, Role: LEADER
14:04:40.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:40.956857 28616 consensus_queue.cc:237] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 5.16, Last appended by leader: 16, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:40.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:40.966181 27467 catalog_manager.cc:5696] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 reported cstate change: term changed from 5 to 6, leader changed from 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3) to 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1). New cstate: current_term: 6 leader_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } health_report { overall_health: UNKNOWN } } }
14:04:41.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:41.426265 28559 raft_consensus.cc:1275] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 6 FOLLOWER]: Refusing update from remote peer 9f9c16da37ad440eb0d7a7a590be07a0: Log matching property violated. Preceding OpId in replica: term: 4 index: 12. Preceding OpId from leader: term: 6 index: 17. (index mismatch)
14:04:41.429 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:41.430 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)[L],d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:41.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:41.430631 28616 consensus_queue.cc:1048] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.000s
14:04:41.436 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=14, sequence_id=13, TimeoutTracker(timeout=30000, elapsed=1662), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [3ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [3ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [16ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [21ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [23ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [25ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [25ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [37ms] refreshing cache from master, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [40ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [42ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [45ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [45ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [77ms] refreshing cache from master, [77ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [82ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [84ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [86ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [86ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [197ms] refreshing cache from master, [197ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [201ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [203ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [206ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [206ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [456ms] refreshing cache from master, [457ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [462ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [463ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [465ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [466ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [837ms] refreshing cache from master, [837ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [840ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [842ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [844ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [844ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [1657ms] refreshing cache from master, [1657ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [1662ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1047051343(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@596bd4cc -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@692741386 after retry RPC@660082787) -> (continuation of Deferred@744585762 after retry RPC@1272822981) -> (continuation of Deferred@1547722030 after retry RPC@2052975095) -> (continuation of Deferred@2018089453 after retry RPC@1833878850) -> (continuation of Deferred@435310785 after retry RPC@34464360) -> (continuation of Deferred@1222285198 after retry RPC@1772371445), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@42f98cac -> wakeup thread Time-limited test -> (continuation of Deferred@692741386 after retry RPC@660082787) -> (continuation of Deferred@744585762 after retry RPC@1272822981) -> (continuation of Deferred@1547722030 after retry RPC@2052975095) -> (continuation of Deferred@2018089453 after retry RPC@1833878850) -> (continuation of Deferred@435310785 after retry RPC@34464360) -> (continuation of Deferred@1222285198 after retry RPC@1772371445))) after lookup
14:04:41.437 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:41.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:41.444352 28312 consensus_peers.cc:597] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 -> Peer 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969): Couldn't send request to peer 353498eabc2240a29da45c31f7cc3c4e. Status: Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:04:41.446 [DEBUG - kudu-nio-2] (Connection.java:757) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] Successfully connected to peer
14:04:41.502 [DEBUG - kudu-nio-2] (Negotiator.java:919) Authenticated connection [id: 0x6670d378, L:/127.0.0.1:45822 - R:/127.26.160.1:40635] using TOKEN/null
14:04:41.550 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.26.160.3:39969
14:04:41.550 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:04:41.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:41.551532 27264 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/logs
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data/info.pb
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.3:39969
14:04:41.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.3
14:04:41.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:41.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42189
14:04:41.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.3
14:04:41.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:41.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:41.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:41.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:04:41.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:41.874459 28312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111) [suppressed 3 similar messages]
14:04:41.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:41.882490 28644 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:04:41.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:41.883082 28644 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:04:41.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:41.883738 28644 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:04:41.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:41.918107 28644 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:04:41.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:41.919166 28644 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.160.3
14:04:41.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:41.956539 28644 tablet_server_runner.cc:78] Tablet server non-default flags:
14:04:41.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:41.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:41.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:41.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:04:41.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data
14:04:41.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:04:41.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:41.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:41.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.3:39969
14:04:41.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:04:41.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:41.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data/info.pb
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.3
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42189
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.3
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/logs
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:22 UTC on e7f111948823
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11959
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:04:41.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:41.958284 28644 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:04:41.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:41.960067 28644 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:04:41.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:41.974761 28650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:43.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:41.974769 28651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:43.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:43.310063 28653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:43.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:43.313450 28652 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1333 milliseconds
14:04:43.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.313589 28644 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:04:43.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.315006 28644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:04:43.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.317538 28644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:04:43.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.318949 28644 hybrid_clock.cc:648] HybridClock initialized: now 1779113083318855 us; error 110 us; skew 500 ppm
14:04:43.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.319905 28644 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:04:43.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.326922 28644 webserver.cc:533] Webserver started at http://127.26.160.3:42189/ using document root <none> and password file <none>
14:04:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.327970 28644 fs_manager.cc:362] Metadata directory not provided
14:04:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.328236 28644 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:04:43.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.336512 28644 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
14:04:43.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.342435 28663 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:04:43.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.344017 28644 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.001s	sys 0.003s
14:04:43.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.344338 28644 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data,/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:04:43.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "353498eabc2240a29da45c31f7cc3c4e"
14:04:43.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:20 on dist-test-slave-8gxc"
14:04:43.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.346410 28644 fs_report.cc:389] FS layout report
14:04:43.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:04:43.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:04:43.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:04:43.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data/data
14:04:43.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:04:43.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:04:43.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:04:43.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:04:43.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:04:43.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:04:43.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:04:43.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:04:43.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:04:43.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:04:43.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:04:43.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.400862 28644 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:04:43.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.402491 28644 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:04:43.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.402961 28644 kserver.cc:163] Server-wide thread pool size limit: 3276
14:04:43.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.406306 28644 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:04:43.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.412497 28670 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:04:43.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.422179 28644 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:04:43.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.422456 28644 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.012s	user 0.002s	sys 0.001s
14:04:43.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.422768 28644 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:04:43.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.428993 28644 ts_tablet_manager.cc:616] Registered 1 tablets
14:04:43.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.429236 28644 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.004s	sys 0.000s
14:04:43.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.429963 28670 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Bootstrap starting.
14:04:43.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.501662 28670 log.cc:826] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Log is configured to *not* fsync() on all Append() calls
14:04:43.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.584811 28670 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=12 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:04:43.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.585925 28670 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Bootstrap complete.
14:04:43.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.587330 28670 ts_tablet_manager.cc:1403] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Time spent bootstrapping tablet: real 0.158s	user 0.122s	sys 0.032s
14:04:43.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.612602 28670 raft_consensus.cc:359] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:43.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.615849 28670 raft_consensus.cc:740] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353498eabc2240a29da45c31f7cc3c4e, State: Initialized, Role: FOLLOWER
14:04:43.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.617054 28670 consensus_queue.cc:260] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 5.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:43.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.621884 28670 ts_tablet_manager.cc:1434] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Time spent starting tablet: real 0.034s	user 0.026s	sys 0.008s
14:04:43.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.625505 28644 rpc_server.cc:307] RPC server started. Bound to: 127.26.160.3:39969
14:04:43.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.625886 28780 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.160.3:39969 every 8 connection(s)
14:04:43.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.628113 28644 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data/info.pb
14:04:43.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.632380 27264 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu as pid 28644
14:04:43.633 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:04:43.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.666028 28783 heartbeater.cc:344] Connected to a master server at 127.26.160.62:43077
14:04:43.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.666575 28783 heartbeater.cc:461] Registering TS with master...
14:04:43.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.668170 28783 heartbeater.cc:507] Master 127.26.160.62:43077 requested a full tablet report, sending...
14:04:43.671 [DEBUG - kudu-nio-2] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 15 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990, tablet=266c976e4879434f8b95ee412dfafcc5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:04:43.672 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.26.160.1:40635
14:04:43.673 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:04:43.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.672842 27310 ts_manager.cc:194] Re-registered known tserver with Master: 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:04:43.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.673760 27264 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu with pid 28287
14:04:43.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.673851 28781 heartbeater.cc:344] Connected to a master server at 127.26.160.60:34099
14:04:43.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.674226 28781 heartbeater.cc:461] Registering TS with master...
14:04:43.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.675153 28781 heartbeater.cc:507] Master 127.26.160.60:34099 requested a full tablet report, sending...
14:04:43.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.678594 27466 ts_manager.cc:194] Re-registered known tserver with Master: 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:04:43.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.678750 28782 heartbeater.cc:344] Connected to a master server at 127.26.160.61:45315
14:04:43.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.679076 28782 heartbeater.cc:461] Registering TS with master...
14:04:43.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.680004 28782 heartbeater.cc:507] Master 127.26.160.61:45315 requested a full tablet report, sending...
14:04:43.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.682137 27466 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.26.160.3:35343
14:04:43.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:43.683964 27385 ts_manager.cc:194] Re-registered known tserver with Master: 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:04:43.707 [DEBUG - kudu-nio-2] (Connection.java:239) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] handling channelInactive
14:04:43.707 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state READY due to: connection closed
14:04:43.708 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:04:43.708 [DEBUG - Time-limited test] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:43.709 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:43.709 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state CONNECTING due to: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:43.710 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:43.728 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:43.728 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)[L],d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:43.729 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=2, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=20), Traces: [0ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [2ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [2ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [19ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@660081949(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@247d29d8 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6ff8afe8 -> wakeup thread Time-limited test)) after lookup
14:04:43.729 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:43.730 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:43.730 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state CONNECTING due to: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:43.731 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:43.747 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:43.748 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)[L],d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:43.748 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=4, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=40), Traces: [0ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [2ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [2ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [19ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [21ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [22ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [23ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [39ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@660081949(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@247d29d8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1484218608 after retry RPC@1918505977), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6ff8afe8 -> wakeup thread Time-limited test -> (continuation of Deferred@1484218608 after retry RPC@1918505977))) after lookup
14:04:43.749 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:43.751 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:43.751 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state CONNECTING due to: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:43.751 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:43.767 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:43.767 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)[L],d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:43.768 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=6, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=59), Traces: [0ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [2ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [2ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [19ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [21ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [22ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [23ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [39ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [41ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [43ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [43ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [59ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@660081949(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@247d29d8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1484218608 after retry RPC@1918505977) -> (continuation of Deferred@1209716672 after retry RPC@1786628321), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6ff8afe8 -> wakeup thread Time-limited test -> (continuation of Deferred@1484218608 after retry RPC@1918505977) -> (continuation of Deferred@1209716672 after retry RPC@1786628321))) after lookup
14:04:43.769 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:43.770 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:43.770 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state CONNECTING due to: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:43.770 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:43.807 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:43.808 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)[L],d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:43.808 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=8, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=99), Traces: [0ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [2ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [2ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [19ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [21ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [22ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [23ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [39ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [41ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [43ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [43ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [59ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [60ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [62ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [62ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [95ms] refreshing cache from master, [95ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [99ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@660081949(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@247d29d8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1484218608 after retry RPC@1918505977) -> (continuation of Deferred@1209716672 after retry RPC@1786628321) -> (continuation of Deferred@1069495689 after retry RPC@989976671), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6ff8afe8 -> wakeup thread Time-limited test -> (continuation of Deferred@1484218608 after retry RPC@1918505977) -> (continuation of Deferred@1209716672 after retry RPC@1786628321) -> (continuation of Deferred@1069495689 after retry RPC@989976671))) after lookup
14:04:43.809 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:43.810 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:43.810 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state CONNECTING due to: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:43.810 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:44.208 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:44.209 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)[L],d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:44.209 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=10, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=501), Traces: [0ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [2ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [2ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [19ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [21ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [22ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [23ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [39ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [41ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [43ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [43ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [59ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [60ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [62ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [62ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [95ms] refreshing cache from master, [95ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [99ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [101ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [102ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [102ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [495ms] refreshing cache from master, [495ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [500ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@660081949(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@247d29d8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1484218608 after retry RPC@1918505977) -> (continuation of Deferred@1209716672 after retry RPC@1786628321) -> (continuation of Deferred@1069495689 after retry RPC@989976671) -> (continuation of Deferred@1884805218 after retry RPC@1168192791), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6ff8afe8 -> wakeup thread Time-limited test -> (continuation of Deferred@1484218608 after retry RPC@1918505977) -> (continuation of Deferred@1209716672 after retry RPC@1786628321) -> (continuation of Deferred@1069495689 after retry RPC@989976671) -> (continuation of Deferred@1884805218 after retry RPC@1168192791))) after lookup
14:04:44.210 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:44.212 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:44.212 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state CONNECTING due to: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:44.213 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:44.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:44.686743 28781 heartbeater.cc:499] Master 127.26.160.60:34099 was elected leader, sending a full tablet report...
14:04:44.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:44.978030 28793 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9f9c16da37ad440eb0d7a7a590be07a0)
14:04:44.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:44.978422 28793 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:44.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:44.980559 28793 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635), 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:04:44.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:44.984521 28493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.160.1:40635: connect: Connection refused (error 111)
14:04:44.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:44.990381 28493 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635): Network error: Client connection negotiation failed: client connection to 127.26.160.1:40635: connect: Connection refused (error 111)
14:04:44.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:44.994529 28735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "353498eabc2240a29da45c31f7cc3c4e" is_pre_election: true
14:04:44.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:44.995235 28735 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c13f941b874a778af09e1ef1b4a7bb in term 5.
14:04:44.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:44.996450 28493 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 353498eabc2240a29da45c31f7cc3c4e, d1c13f941b874a778af09e1ef1b4a7bb; no voters: 9f9c16da37ad440eb0d7a7a590be07a0
14:04:44.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:44.997134 28793 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 6 FOLLOWER]: Leader pre-election won for term 7
14:04:44.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:44.997457 28793 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 6 FOLLOWER]: Starting leader election (detected failure of leader 9f9c16da37ad440eb0d7a7a590be07a0)
14:04:44.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:44.997694 28793 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 6 FOLLOWER]: Advancing to term 7
14:04:45.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:45.002699 28793 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:45.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:45.004662 28793 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 7 election: Requested vote from peers 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635), 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:04:45.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:45.006721 28735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" candidate_term: 7 candidate_status { last_received { term: 6 index: 20 } } ignore_live_leader: false dest_uuid: "353498eabc2240a29da45c31f7cc3c4e"
14:04:45.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:45.007251 28735 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 5 FOLLOWER]: Advancing to term 7
14:04:45.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:45.009274 28493 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635): Network error: Client connection negotiation failed: client connection to 127.26.160.1:40635: connect: Connection refused (error 111)
14:04:45.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:45.015991 28735 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c13f941b874a778af09e1ef1b4a7bb in term 7.
14:04:45.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:45.017942 28493 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 353498eabc2240a29da45c31f7cc3c4e, d1c13f941b874a778af09e1ef1b4a7bb; no voters: 9f9c16da37ad440eb0d7a7a590be07a0
14:04:45.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:45.019119 28793 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 7 FOLLOWER]: Leader election won for term 7
14:04:45.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:45.021117 28793 raft_consensus.cc:697] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 7 LEADER]: Becoming Leader. State: Replica: d1c13f941b874a778af09e1ef1b4a7bb, State: Running, Role: LEADER
14:04:45.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:45.022408 28793 consensus_queue.cc:237] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:45.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:45.032147 27466 catalog_manager.cc:5696] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb reported cstate change: term changed from 6 to 7, leader changed from 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1) to d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2). New cstate: current_term: 7 leader_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } health_report { overall_health: HEALTHY } } }
14:04:45.249 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:45.249 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)[L]]}]
14:04:45.249 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=12, sequence_id=16, TimeoutTracker(timeout=30000, elapsed=1541), Traces: [0ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [2ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [2ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [19ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [21ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [22ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [23ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [39ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [41ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [43ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [43ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [55ms] refreshing cache from master, [55ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [59ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [60ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [62ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [62ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [95ms] refreshing cache from master, [95ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [99ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [101ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [102ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [102ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [495ms] refreshing cache from master, [495ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [500ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [502ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [504ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [506ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [1536ms] refreshing cache from master, [1536ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [1540ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@660081949(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@247d29d8 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@1484218608 after retry RPC@1918505977) -> (continuation of Deferred@1209716672 after retry RPC@1786628321) -> (continuation of Deferred@1069495689 after retry RPC@989976671) -> (continuation of Deferred@1884805218 after retry RPC@1168192791) -> (continuation of Deferred@1768509471 after retry RPC@623692200), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@6ff8afe8 -> wakeup thread Time-limited test -> (continuation of Deferred@1484218608 after retry RPC@1918505977) -> (continuation of Deferred@1209716672 after retry RPC@1786628321) -> (continuation of Deferred@1069495689 after retry RPC@989976671) -> (continuation of Deferred@1884805218 after retry RPC@1168192791) -> (continuation of Deferred@1768509471 after retry RPC@623692200))) after lookup
14:04:45.250 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:04:45.251 [DEBUG - kudu-nio-1] (Connection.java:757) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] Successfully connected to peer
14:04:45.273 [DEBUG - kudu-nio-1] (Negotiator.java:919) Authenticated connection [id: 0xba82fcc6, L:/127.0.0.1:48292 - R:/127.26.160.2:37023] using TOKEN/null
14:04:45.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:45.284483 28735 raft_consensus.cc:1275] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 7 FOLLOWER]: Refusing update from remote peer d1c13f941b874a778af09e1ef1b4a7bb: Log matching property violated. Preceding OpId in replica: term: 5 index: 16. Preceding OpId from leader: term: 7 index: 22. (index mismatch)
14:04:45.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:45.286144 28793 consensus_queue.cc:1048] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 16, Time since last communication: 0.000s
14:04:45.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:45.286825 28493 consensus_peers.cc:597] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb -> Peer 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635): Couldn't send request to peer 9f9c16da37ad440eb0d7a7a590be07a0. Status: Network error: Client connection negotiation failed: client connection to 127.26.160.1:40635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:04:45.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:45.307590 28815 mvcc.cc:204] Tried to move back new op lower bound from 7287247181853458432 to 7287247179609092096. Current Snapshot: MvccSnapshot[applied={T|T < 7287247181853458432}]
14:04:45.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:45.317687 28813 mvcc.cc:204] Tried to move back new op lower bound from 7287247197308043264 to 7287247196259078144. Current Snapshot: MvccSnapshot[applied={T|T < 7287247197308043264}]
14:04:45.378 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.26.160.1:40635
14:04:45.380 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:04:45.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:45.382143 27264 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:45.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:45.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal
14:04:45.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data
14:04:45.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:04:45.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:04:45.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/logs
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data/info.pb
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.1:40635
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.1
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34423
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.1
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:45.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:04:45.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:45.745055 28822 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:04:45.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:45.745572 28822 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:04:45.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:45.746073 28822 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:04:45.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:45.782902 28822 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:04:45.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:45.784071 28822 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.160.1
14:04:45.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:45.823913 28822 tablet_server_runner.cc:78] Tablet server non-default flags:
14:04:45.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:45.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:45.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:45.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:04:45.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data
14:04:45.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal
14:04:45.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:45.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:45.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.1:40635
14:04:45.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:04:45.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:45.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:45.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data/info.pb
14:04:45.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.1
14:04:45.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34423
14:04:45.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:45.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:04:45.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:45.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:04:45.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:04:45.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:45.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.1
14:04:45.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/logs
14:04:45.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:45.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:04:45.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:04:45.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:04:45.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:04:45.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:04:45.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:04:45.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:22 UTC on e7f111948823
14:04:45.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11959
14:04:45.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:04:45.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:45.825702 28822 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:04:45.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:45.827880 28822 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:04:45.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:45.842047 28828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:47.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:45.845508 28829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:45.848822 28831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:47.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:47.208891 28830 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1362 milliseconds
14:04:47.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.209049 28822 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:04:47.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.210628 28822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:04:47.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.214959 28822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:04:47.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.216668 28822 hybrid_clock.cc:648] HybridClock initialized: now 1779113087216595 us; error 95 us; skew 500 ppm
14:04:47.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.217669 28822 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:04:47.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.227262 28822 webserver.cc:533] Webserver started at http://127.26.160.1:34423/ using document root <none> and password file <none>
14:04:47.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.228616 28822 fs_manager.cc:362] Metadata directory not provided
14:04:47.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.228933 28822 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:04:47.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.241178 28822 fs_manager.cc:714] Time spent opening directory manager: real 0.007s	user 0.003s	sys 0.004s
14:04:47.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.248137 28842 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:04:47.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.250442 28822 fs_manager.cc:730] Time spent opening block manager: real 0.006s	user 0.003s	sys 0.003s
14:04:47.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.250918 28822 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data,/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal
14:04:47.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "9f9c16da37ad440eb0d7a7a590be07a0"
14:04:47.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:15 on dist-test-slave-8gxc"
14:04:47.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.253870 28822 fs_report.cc:389] FS layout report
14:04:47.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:04:47.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal
14:04:47.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal
14:04:47.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data/data
14:04:47.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:04:47.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:04:47.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:04:47.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:04:47.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:04:47.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:04:47.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:04:47.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:04:47.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:04:47.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:04:47.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:04:47.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.315006 28822 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:04:47.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.316521 28822 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:04:47.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.316937 28822 kserver.cc:163] Server-wide thread pool size limit: 3276
14:04:47.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.319753 28822 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:04:47.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.325852 28849 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:04:47.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.334635 28822 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:04:47.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.334952 28822 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.001s	sys 0.001s
14:04:47.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.335269 28822 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:04:47.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.340970 28822 ts_tablet_manager.cc:616] Registered 1 tablets
14:04:47.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.341224 28822 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.006s	sys 0.001s
14:04:47.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.343266 28849 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Bootstrap starting.
14:04:47.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.427408 28849 log.cc:826] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Log is configured to *not* fsync() on all Append() calls
14:04:47.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.525183 28849 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=15 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:04:47.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.526343 28849 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Bootstrap complete.
14:04:47.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.528155 28849 ts_tablet_manager.cc:1403] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Time spent bootstrapping tablet: real 0.185s	user 0.107s	sys 0.070s
14:04:47.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.549623 28822 rpc_server.cc:307] RPC server started. Bound to: 127.26.160.1:40635
14:04:47.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.549718 28958 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.160.1:40635 every 8 connection(s)
14:04:47.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.552457 28822 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data/info.pb
14:04:47.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.551208 28849 raft_consensus.cc:359] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:47.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.553300 28849 raft_consensus.cc:740] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9c16da37ad440eb0d7a7a590be07a0, State: Initialized, Role: FOLLOWER
14:04:47.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.554059 28849 consensus_queue.cc:260] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 6.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:47.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.557895 27264 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu as pid 28822
14:04:47.559 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:04:47.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.576700 28849 ts_tablet_manager.cc:1434] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Time spent starting tablet: real 0.048s	user 0.028s	sys 0.020s
14:04:47.587 [DEBUG - kudu-nio-1] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 18 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990, tablet=266c976e4879434f8b95ee412dfafcc5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:04:47.589 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.26.160.2:37023
14:04:47.589 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:04:47.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.590557 27264 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu with pid 28470
14:04:47.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.618973 28961 heartbeater.cc:344] Connected to a master server at 127.26.160.61:45315
14:04:47.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.619963 28961 heartbeater.cc:461] Registering TS with master...
14:04:47.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.621348 28961 heartbeater.cc:507] Master 127.26.160.61:45315 requested a full tablet report, sending...
14:04:47.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.626580 27385 ts_manager.cc:194] Re-registered known tserver with Master: 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635)
14:04:47.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.633337 28959 heartbeater.cc:344] Connected to a master server at 127.26.160.60:34099
14:04:47.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.633754 28959 heartbeater.cc:461] Registering TS with master...
14:04:47.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.634809 28959 heartbeater.cc:507] Master 127.26.160.60:34099 requested a full tablet report, sending...
14:04:47.638 [DEBUG - kudu-nio-1] (Connection.java:239) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] handling channelInactive
14:04:47.639 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state READY due to: connection closed
14:04:47.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.638978 27466 ts_manager.cc:194] Re-registered known tserver with Master: 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635)
14:04:47.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.642107 27466 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.26.160.1:50131
14:04:47.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.644307 28964 heartbeater.cc:344] Connected to a master server at 127.26.160.62:43077
14:04:47.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.644697 28964 heartbeater.cc:461] Registering TS with master...
14:04:47.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.645733 28964 heartbeater.cc:507] Master 127.26.160.62:43077 requested a full tablet report, sending...
14:04:47.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.647035 28959 heartbeater.cc:499] Master 127.26.160.60:34099 was elected leader, sending a full tablet report...
14:04:47.648 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:04:47.648 [DEBUG - Time-limited test] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:04:47.650 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:47.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:47.649479 27310 ts_manager.cc:194] Re-registered known tserver with Master: 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635)
14:04:47.650 [DEBUG - kudu-nio-2] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state CONNECTING due to: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:47.650 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:47.668 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:47.669 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)[L]]}]
14:04:47.669 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=2, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=21), Traces: [0ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [2ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [2ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [20ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1810165841(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@15144ccf -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3dda1b07 -> wakeup thread Time-limited test)) after lookup
14:04:47.670 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:04:47.671 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:47.671 [DEBUG - kudu-nio-3] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state CONNECTING due to: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:47.672 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:47.688 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:47.689 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)[L]]}]
14:04:47.689 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=4, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=41), Traces: [0ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [2ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [2ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [20ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [22ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [24ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [24ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [40ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1810165841(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@15144ccf -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@645707210 after retry RPC@1267957793), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3dda1b07 -> wakeup thread Time-limited test -> (continuation of Deferred@645707210 after retry RPC@1267957793))) after lookup
14:04:47.690 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:04:47.691 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:47.691 [DEBUG - kudu-nio-4] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state CONNECTING due to: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:47.691 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:47.728 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:47.728 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)[L]]}]
14:04:47.728 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=6, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=80), Traces: [0ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [2ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [2ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [20ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [22ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [24ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [24ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [40ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [41ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [43ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [43ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [79ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1810165841(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@15144ccf -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@645707210 after retry RPC@1267957793) -> (continuation of Deferred@337178215 after retry RPC@1130231373), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3dda1b07 -> wakeup thread Time-limited test -> (continuation of Deferred@645707210 after retry RPC@1267957793) -> (continuation of Deferred@337178215 after retry RPC@1130231373))) after lookup
14:04:47.729 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:04:47.730 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:47.730 [DEBUG - kudu-nio-5] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state CONNECTING due to: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:47.730 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:47.868 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:47.868 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)[L]]}]
14:04:47.869 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=8, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=220), Traces: [0ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [2ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [2ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [20ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [22ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [24ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [24ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [40ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [41ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [43ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [43ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [79ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [81ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [82ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [82ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [215ms] refreshing cache from master, [215ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [219ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1810165841(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@15144ccf -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@645707210 after retry RPC@1267957793) -> (continuation of Deferred@337178215 after retry RPC@1130231373) -> (continuation of Deferred@899976564 after retry RPC@426244627), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3dda1b07 -> wakeup thread Time-limited test -> (continuation of Deferred@645707210 after retry RPC@1267957793) -> (continuation of Deferred@337178215 after retry RPC@1130231373) -> (continuation of Deferred@899976564 after retry RPC@426244627))) after lookup
14:04:47.869 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:04:47.871 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:47.871 [DEBUG - kudu-nio-6] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state CONNECTING due to: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:47.871 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:48.048 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:48.048 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)[L]]}]
14:04:48.048 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=10, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=400), Traces: [0ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [2ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [2ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [20ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [22ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [24ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [24ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [40ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [41ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [43ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [43ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [79ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [81ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [82ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [82ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [215ms] refreshing cache from master, [215ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [219ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [221ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [223ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [223ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [395ms] refreshing cache from master, [395ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [399ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1810165841(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@15144ccf -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@645707210 after retry RPC@1267957793) -> (continuation of Deferred@337178215 after retry RPC@1130231373) -> (continuation of Deferred@899976564 after retry RPC@426244627) -> (continuation of Deferred@2034730201 after retry RPC@1607884416), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3dda1b07 -> wakeup thread Time-limited test -> (continuation of Deferred@645707210 after retry RPC@1267957793) -> (continuation of Deferred@337178215 after retry RPC@1130231373) -> (continuation of Deferred@899976564 after retry RPC@426244627) -> (continuation of Deferred@2034730201 after retry RPC@1607884416))) after lookup
14:04:48.049 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:04:48.050 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:48.050 [DEBUG - kudu-nio-7] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state CONNECTING due to: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:48.050 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:04:48.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:48.997043 28971 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d1c13f941b874a778af09e1ef1b4a7bb)
14:04:48.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:48.997470 28971 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:49.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.000331 28971 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635), d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:49.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:49.005427 28666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.160.2:37023: connect: Connection refused (error 111)
14:04:49.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:49.011212 28666 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023): Network error: Client connection negotiation failed: client connection to 127.26.160.2:37023: connect: Connection refused (error 111)
14:04:49.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.015771 28914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "353498eabc2240a29da45c31f7cc3c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" is_pre_election: true
14:04:49.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.016669 28914 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 353498eabc2240a29da45c31f7cc3c4e in term 6.
14:04:49.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.018281 28667 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 353498eabc2240a29da45c31f7cc3c4e, 9f9c16da37ad440eb0d7a7a590be07a0; no voters: d1c13f941b874a778af09e1ef1b4a7bb
14:04:49.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.019158 28971 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 7 FOLLOWER]: Leader pre-election won for term 8
14:04:49.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.019685 28971 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 7 FOLLOWER]: Starting leader election (detected failure of leader d1c13f941b874a778af09e1ef1b4a7bb)
14:04:49.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.019971 28971 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 7 FOLLOWER]: Advancing to term 8
14:04:49.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.025194 28971 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:49.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.027007 28971 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 8 election: Requested vote from peers 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635), d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:49.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.028240 28914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "353498eabc2240a29da45c31f7cc3c4e" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "9f9c16da37ad440eb0d7a7a590be07a0"
14:04:49.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.028754 28914 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 6 FOLLOWER]: Advancing to term 8
14:04:49.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:49.031986 28666 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023): Network error: Client connection negotiation failed: client connection to 127.26.160.2:37023: connect: Connection refused (error 111)
14:04:49.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.037045 28914 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 353498eabc2240a29da45c31f7cc3c4e in term 8.
14:04:49.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.038584 28667 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 353498eabc2240a29da45c31f7cc3c4e, 9f9c16da37ad440eb0d7a7a590be07a0; no voters: d1c13f941b874a778af09e1ef1b4a7bb
14:04:49.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.039686 28971 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 8 FOLLOWER]: Leader election won for term 8
14:04:49.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.042176 28971 raft_consensus.cc:697] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 8 LEADER]: Becoming Leader. State: Replica: 353498eabc2240a29da45c31f7cc3c4e, State: Running, Role: LEADER
14:04:49.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.043166 28971 consensus_queue.cc:237] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 7.24, Last appended by leader: 24, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:49.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.054971 27466 catalog_manager.cc:5696] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e reported cstate change: term changed from 7 to 8, leader changed from d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2) to 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3). New cstate: current_term: 8 leader_uuid: "353498eabc2240a29da45c31f7cc3c4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } health_report { overall_health: UNKNOWN } } }
14:04:49.269 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:49.269 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:49.269 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=12, sequence_id=19, TimeoutTracker(timeout=30000, elapsed=1621), Traces: [0ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [2ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [2ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [15ms] refreshing cache from master, [16ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [20ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [22ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [24ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [24ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [35ms] refreshing cache from master, [35ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [40ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [41ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [43ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [43ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [75ms] refreshing cache from master, [75ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [79ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [81ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [82ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [82ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [215ms] refreshing cache from master, [215ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [219ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [221ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [223ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [223ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [395ms] refreshing cache from master, [395ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [399ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [401ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [402ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [402ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [1615ms] refreshing cache from master, [1616ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [1621ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1810165841(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@15144ccf -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@645707210 after retry RPC@1267957793) -> (continuation of Deferred@337178215 after retry RPC@1130231373) -> (continuation of Deferred@899976564 after retry RPC@426244627) -> (continuation of Deferred@2034730201 after retry RPC@1607884416) -> (continuation of Deferred@1178215736 after retry RPC@737493281), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@3dda1b07 -> wakeup thread Time-limited test -> (continuation of Deferred@645707210 after retry RPC@1267957793) -> (continuation of Deferred@337178215 after retry RPC@1130231373) -> (continuation of Deferred@899976564 after retry RPC@426244627) -> (continuation of Deferred@2034730201 after retry RPC@1607884416) -> (continuation of Deferred@1178215736 after retry RPC@737493281))) after lookup
14:04:49.270 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:49.271 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] Successfully connected to peer
14:04:49.299 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xf72c21fd, L:/127.0.0.1:42622 - R:/127.26.160.3:39969] using TOKEN/null
14:04:49.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.311302 28914 raft_consensus.cc:1275] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 8 FOLLOWER]: Refusing update from remote peer 353498eabc2240a29da45c31f7cc3c4e: Log matching property violated. Preceding OpId in replica: term: 6 index: 20. Preceding OpId from leader: term: 8 index: 26. (index mismatch)
14:04:49.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.313649 28976 consensus_queue.cc:1048] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 20, Time since last communication: 0.000s
14:04:49.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:49.314386 28666 consensus_peers.cc:597] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e -> Peer d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023): Couldn't send request to peer d1c13f941b874a778af09e1ef1b4a7bb. Status: Network error: Client connection negotiation failed: client connection to 127.26.160.2:37023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:04:49.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.340030 28992 mvcc.cc:204] Tried to move back new op lower bound from 7287247197571035136 to 7287247196259078144. Current Snapshot: MvccSnapshot[applied={T|T < 7287247197308043264}]
14:04:49.406 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.26.160.2:37023
14:04:49.407 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:04:49.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.408219 27264 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:49.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:49.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal
14:04:49.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data
14:04:49.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:04:49.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:04:49.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:04:49.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:49.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:49.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/logs
14:04:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data/info.pb
14:04:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:04:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:04:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:04:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:04:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:04:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:04:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.2:37023
14:04:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.2
14:04:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34355
14:04:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.2
14:04:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:49.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:04:49.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:49.776360 28999 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:04:49.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:49.776868 28999 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:04:49.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:49.777314 28999 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:04:49.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:49.815233 28999 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:04:49.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:49.816149 28999 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.160.2
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.856839 28999 tablet_server_runner.cc:78] Tablet server non-default flags:
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.2:37023
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data/info.pb
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.2
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34355
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:04:49.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:49.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.2
14:04:49.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/logs
14:04:49.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:49.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:04:49.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:04:49.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:04:49.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:04:49.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:04:49.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:04:49.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:22 UTC on e7f111948823
14:04:49.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11959
14:04:49.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:04:49.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.858240 28999 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:04:49.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:49.859841 28999 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:04:49.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:49.874511 29006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:51.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:49.880867 29009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:51.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:49.877036 29007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:51.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:51.209828 29008 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1328 milliseconds
14:04:51.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.210037 28999 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:04:51.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.211540 28999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:04:51.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.214324 28999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:04:51.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.215934 28999 hybrid_clock.cc:648] HybridClock initialized: now 1779113091215860 us; error 77 us; skew 500 ppm
14:04:51.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.216971 28999 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:04:51.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.224893 28999 webserver.cc:533] Webserver started at http://127.26.160.2:34355/ using document root <none> and password file <none>
14:04:51.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.226294 28999 fs_manager.cc:362] Metadata directory not provided
14:04:51.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.226577 28999 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:04:51.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.237847 28999 fs_manager.cc:714] Time spent opening directory manager: real 0.007s	user 0.006s	sys 0.001s
14:04:51.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.243805 29018 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:04:51.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.245250 28999 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
14:04:51.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.245661 28999 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data,/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal
14:04:51.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d1c13f941b874a778af09e1ef1b4a7bb"
14:04:51.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:18 on dist-test-slave-8gxc"
14:04:51.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.248180 28999 fs_report.cc:389] FS layout report
14:04:51.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:04:51.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal
14:04:51.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal
14:04:51.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data/data
14:04:51.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:04:51.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:04:51.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:04:51.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:04:51.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:04:51.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:04:51.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:04:51.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:04:51.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:04:51.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:04:51.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:04:51.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.303020 28999 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:04:51.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.304775 28999 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:04:51.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.305400 28999 kserver.cc:163] Server-wide thread pool size limit: 3276
14:04:51.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.308288 28999 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:04:51.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.314772 29025 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:04:51.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.324160 28999 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:04:51.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.324549 28999 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.012s	user 0.002s	sys 0.000s
14:04:51.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.324882 28999 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:04:51.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.330636 28999 ts_tablet_manager.cc:616] Registered 1 tablets
14:04:51.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.330935 28999 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.006s	sys 0.000s
14:04:51.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.331375 29025 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Bootstrap starting.
14:04:51.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.407824 29025 log.cc:826] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Log is configured to *not* fsync() on all Append() calls
14:04:51.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.524713 29025 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=0} inserts{seen=18 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:04:51.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.525720 29025 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Bootstrap complete.
14:04:51.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.527992 29025 ts_tablet_manager.cc:1403] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Time spent bootstrapping tablet: real 0.197s	user 0.130s	sys 0.059s
14:04:51.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.544142 28999 rpc_server.cc:307] RPC server started. Bound to: 127.26.160.2:37023
14:04:51.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.544307 29132 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.160.2:37023 every 8 connection(s)
14:04:51.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.547919 28999 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data/info.pb
14:04:51.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.548262 29025 raft_consensus.cc:359] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 7 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:51.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.551558 29025 raft_consensus.cc:740] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c13f941b874a778af09e1ef1b4a7bb, State: Initialized, Role: FOLLOWER
14:04:51.553 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:04:51.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.552273 27264 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu as pid 28999
14:04:51.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.552897 29025 consensus_queue.cc:260] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 7.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:51.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.599946 29025 ts_tablet_manager.cc:1434] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Time spent starting tablet: real 0.072s	user 0.035s	sys 0.003s
14:04:51.609 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 21 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990, tablet=266c976e4879434f8b95ee412dfafcc5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:04:51.610 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.26.160.3:39969
14:04:51.610 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:04:51.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.611495 27264 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu with pid 28644
14:04:51.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.636057 29133 heartbeater.cc:344] Connected to a master server at 127.26.160.60:34099
14:04:51.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.636528 29133 heartbeater.cc:461] Registering TS with master...
14:04:51.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.637957 29133 heartbeater.cc:507] Master 127.26.160.60:34099 requested a full tablet report, sending...
14:04:51.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.639786 29134 heartbeater.cc:344] Connected to a master server at 127.26.160.61:45315
14:04:51.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.640156 29134 heartbeater.cc:461] Registering TS with master...
14:04:51.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.641016 29134 heartbeater.cc:507] Master 127.26.160.61:45315 requested a full tablet report, sending...
14:04:51.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.641117 29139 heartbeater.cc:344] Connected to a master server at 127.26.160.62:43077
14:04:51.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.641479 29139 heartbeater.cc:461] Registering TS with master...
14:04:51.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.642413 29139 heartbeater.cc:507] Master 127.26.160.62:43077 requested a full tablet report, sending...
14:04:51.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.646773 27310 ts_manager.cc:194] Re-registered known tserver with Master: d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:51.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.646647 27466 ts_manager.cc:194] Re-registered known tserver with Master: d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:51.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.647048 27384 ts_manager.cc:194] Re-registered known tserver with Master: d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:51.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.650028 27466 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.26.160.2:54557
14:04:51.655 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] handling channelInactive
14:04:51.655 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state READY due to: connection closed
14:04:51.658 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:04:51.659 [DEBUG - Time-limited test] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:51.660 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:51.660 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:51.661 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:51.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:51.661808 29133 heartbeater.cc:499] Master 127.26.160.60:34099 was elected leader, sending a full tablet report...
14:04:51.668 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:51.668 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:51.668 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=2, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=9), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [2ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [2ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [8ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@797731554(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@121e624 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1fb7fd41 -> wakeup thread Time-limited test)) after lookup
14:04:51.669 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:51.670 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:51.670 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:51.670 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:51.688 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:51.688 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:51.688 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=4, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=29), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [2ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [2ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [8ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [9ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [11ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [11ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [29ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@797731554(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@121e624 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@800887764 after retry RPC@255934103), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1fb7fd41 -> wakeup thread Time-limited test -> (continuation of Deferred@800887764 after retry RPC@255934103))) after lookup
14:04:51.689 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:51.690 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:51.690 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:51.691 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:51.728 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:51.728 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:51.728 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=6, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=69), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [2ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [2ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [8ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [9ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [11ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [11ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [29ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [30ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [31ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [32ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [68ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@797731554(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@121e624 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@800887764 after retry RPC@255934103) -> (continuation of Deferred@900698761 after retry RPC@865908525), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1fb7fd41 -> wakeup thread Time-limited test -> (continuation of Deferred@800887764 after retry RPC@255934103) -> (continuation of Deferred@900698761 after retry RPC@865908525))) after lookup
14:04:51.729 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:51.730 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:51.730 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:51.730 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:51.808 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:51.808 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:51.809 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=8, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=150), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [2ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [2ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [8ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [9ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [11ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [11ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [29ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [30ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [31ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [32ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [68ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [70ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [71ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [71ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [145ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [149ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@797731554(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@121e624 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@800887764 after retry RPC@255934103) -> (continuation of Deferred@900698761 after retry RPC@865908525) -> (continuation of Deferred@110945562 after retry RPC@1349459771), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1fb7fd41 -> wakeup thread Time-limited test -> (continuation of Deferred@800887764 after retry RPC@255934103) -> (continuation of Deferred@900698761 after retry RPC@865908525) -> (continuation of Deferred@110945562 after retry RPC@1349459771))) after lookup
14:04:51.810 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:51.811 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:51.811 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:51.811 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:51.908 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:51.908 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:51.909 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=10, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=250), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [2ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [2ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [8ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [9ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [11ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [11ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [29ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [30ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [31ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [32ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [68ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [70ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [71ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [71ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [145ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [149ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [151ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [152ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [153ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [244ms] refreshing cache from master, [244ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [249ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@797731554(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@121e624 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@800887764 after retry RPC@255934103) -> (continuation of Deferred@900698761 after retry RPC@865908525) -> (continuation of Deferred@110945562 after retry RPC@1349459771) -> (continuation of Deferred@925423643 after retry RPC@521572098), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1fb7fd41 -> wakeup thread Time-limited test -> (continuation of Deferred@800887764 after retry RPC@255934103) -> (continuation of Deferred@900698761 after retry RPC@865908525) -> (continuation of Deferred@110945562 after retry RPC@1349459771) -> (continuation of Deferred@925423643 after retry RPC@521572098))) after lookup
14:04:51.910 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:51.911 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:51.911 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:51.911 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:52.408 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:52.408 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:52.408 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=12, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=749), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [2ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [2ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [8ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [9ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [11ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [11ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [29ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [30ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [31ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [32ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [68ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [70ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [71ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [71ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [145ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [149ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [151ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [152ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [153ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [244ms] refreshing cache from master, [244ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [249ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [250ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [252ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [252ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [744ms] refreshing cache from master, [744ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [748ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@797731554(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@121e624 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@800887764 after retry RPC@255934103) -> (continuation of Deferred@900698761 after retry RPC@865908525) -> (continuation of Deferred@110945562 after retry RPC@1349459771) -> (continuation of Deferred@925423643 after retry RPC@521572098) -> (continuation of Deferred@880077306 after retry RPC@523308777), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1fb7fd41 -> wakeup thread Time-limited test -> (continuation of Deferred@800887764 after retry RPC@255934103) -> (continuation of Deferred@900698761 after retry RPC@865908525) -> (continuation of Deferred@110945562 after retry RPC@1349459771) -> (continuation of Deferred@925423643 after retry RPC@521572098) -> (continuation of Deferred@880077306 after retry RPC@523308777))) after lookup
14:04:52.409 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:04:52.410 [INFO - kudu-nio-7] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:52.411 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:52.411 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:04:52.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.742111 29145 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:04:52.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.742568 29145 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:52.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.744769 29145 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635), 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:04:52.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:52.750224 29022 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111)
14:04:52.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:52.754522 29022 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969): Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111)
14:04:52.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.755579 28914 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" candidate_term: 8 candidate_status { last_received { term: 7 index: 24 } } ignore_live_leader: false dest_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" is_pre_election: true
14:04:52.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.757107 29022 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1c13f941b874a778af09e1ef1b4a7bb; no voters: 353498eabc2240a29da45c31f7cc3c4e, 9f9c16da37ad440eb0d7a7a590be07a0
14:04:52.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.757843 29145 raft_consensus.cc:2749] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
14:04:52.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.958321 29150 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 353498eabc2240a29da45c31f7cc3c4e)
14:04:52.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.958743 29150 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:52.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.961160 29150 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969), d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:52.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:52.969367 28846 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111)
14:04:52.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:52.973953 28846 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969): Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111)
14:04:52.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.978888 29088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" is_pre_election: true
14:04:52.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.979648 29088 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f9c16da37ad440eb0d7a7a590be07a0 in term 7.
14:04:52.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.981186 28845 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9f9c16da37ad440eb0d7a7a590be07a0, d1c13f941b874a778af09e1ef1b4a7bb; no voters: 353498eabc2240a29da45c31f7cc3c4e
14:04:52.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.981952 29150 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 8 FOLLOWER]: Leader pre-election won for term 9
14:04:52.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.982316 29150 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 353498eabc2240a29da45c31f7cc3c4e)
14:04:52.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.982645 29150 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 8 FOLLOWER]: Advancing to term 9
14:04:52.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.987848 29150 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:52.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.989346 29150 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 9 election: Requested vote from peers 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969), d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:04:52.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.990408 29088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "d1c13f941b874a778af09e1ef1b4a7bb"
14:04:52.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.990891 29088 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 7 FOLLOWER]: Advancing to term 9
14:04:52.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:52.992467 28846 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969): Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111)
14:04:52.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.997314 29088 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f9c16da37ad440eb0d7a7a590be07a0 in term 9.
14:04:52.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.998293 28845 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9f9c16da37ad440eb0d7a7a590be07a0, d1c13f941b874a778af09e1ef1b4a7bb; no voters: 353498eabc2240a29da45c31f7cc3c4e
14:04:52.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:52.998929 29150 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 9 FOLLOWER]: Leader election won for term 9
14:04:53.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:53.000447 29150 raft_consensus.cc:697] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 9 LEADER]: Becoming Leader. State: Replica: 9f9c16da37ad440eb0d7a7a590be07a0, State: Running, Role: LEADER
14:04:53.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:53.001315 29150 consensus_queue.cc:237] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 8.28, Last appended by leader: 28, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:53.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:53.010885 27466 catalog_manager.cc:5696] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 reported cstate change: term changed from 8 to 9, leader changed from 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3) to 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1). New cstate: current_term: 9 leader_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } health_report { overall_health: UNKNOWN } } }
14:04:53.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:53.408695 29088 raft_consensus.cc:1275] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 9 FOLLOWER]: Refusing update from remote peer 9f9c16da37ad440eb0d7a7a590be07a0: Log matching property violated. Preceding OpId in replica: term: 7 index: 24. Preceding OpId from leader: term: 9 index: 29. (index mismatch)
14:04:53.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:53.410347 29150 consensus_queue.cc:1048] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 24, Time since last communication: 0.000s
14:04:53.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:53.425675 28846 consensus_peers.cc:597] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 -> Peer 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969): Couldn't send request to peer 353498eabc2240a29da45c31f7cc3c4e. Status: Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:04:54.868 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:54.868 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)[L],d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:54.869 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=14, sequence_id=22, TimeoutTracker(timeout=30000, elapsed=3209), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [2ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [2ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [4ms] refreshing cache from master, [4ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [8ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [9ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [11ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [11ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [25ms] refreshing cache from master, [25ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [29ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [30ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [31ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [32ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [65ms] refreshing cache from master, [65ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [68ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [70ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [71ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [71ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [145ms] refreshing cache from master, [145ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [149ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [151ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [152ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [153ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [244ms] refreshing cache from master, [244ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [249ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [250ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [252ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [252ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [744ms] refreshing cache from master, [744ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [748ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [750ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [752ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [752ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [3205ms] refreshing cache from master, [3205ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [3209ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@797731554(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@121e624 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@800887764 after retry RPC@255934103) -> (continuation of Deferred@900698761 after retry RPC@865908525) -> (continuation of Deferred@110945562 after retry RPC@1349459771) -> (continuation of Deferred@925423643 after retry RPC@521572098) -> (continuation of Deferred@880077306 after retry RPC@523308777) -> (continuation of Deferred@1203646589 after retry RPC@1357854757), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@1fb7fd41 -> wakeup thread Time-limited test -> (continuation of Deferred@800887764 after retry RPC@255934103) -> (continuation of Deferred@900698761 after retry RPC@865908525) -> (continuation of Deferred@110945562 after retry RPC@1349459771) -> (continuation of Deferred@925423643 after retry RPC@521572098) -> (continuation of Deferred@880077306 after retry RPC@523308777) -> (continuation of Deferred@1203646589 after retry RPC@1357854757))) after lookup
14:04:54.870 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:54.871 [DEBUG - kudu-nio-0] (Connection.java:757) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] Successfully connected to peer
14:04:54.893 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0xf508c35e, L:/127.0.0.1:43874 - R:/127.26.160.1:40635] using TOKEN/null
14:04:54.941 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.26.160.3:39969
14:04:54.941 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:54.943593 27264 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/logs
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data/info.pb
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.3:39969
14:04:54.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.3
14:04:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42189
14:04:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.3
14:04:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:54.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:04:55.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:55.303854 29185 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:04:55.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:55.304476 29185 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:04:55.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:55.304960 29185 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:04:55.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:55.342173 29185 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:04:55.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:55.343528 29185 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.160.3
14:04:55.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:55.400520 29185 tablet_server_runner.cc:78] Tablet server non-default flags:
14:04:55.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:04:55.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:04:55.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:04:55.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:04:55.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data
14:04:55.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:04:55.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:04:55.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:04:55.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.3:39969
14:04:55.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:04:55.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data/info.pb
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.3
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42189
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.3
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/logs
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:22 UTC on e7f111948823
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11959
14:04:55.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:04:55.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:55.402209 29185 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:04:55.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:55.404137 29185 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:04:55.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:55.420987 29194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:55.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:55.551474 28846 consensus_peers.cc:597] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 -> Peer 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969): Couldn't send request to peer 353498eabc2240a29da45c31f7cc3c4e. Status: Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:04:57.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:55.422029 29195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:57.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:56.821143 29193 debug-util.cc:398] Leaking SignalData structure 0x7b08000271c0 after lost signal to thread 29185
14:04:57.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:57.550604 29193 kernel_stack_watchdog.cc:198] Thread 29185 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 398ms:
14:04:57.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Kernel stack:
14:04:57.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) (could not read kernel stack)
14:04:57.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) User stack:
14:04:57.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) <Timed out: thread did not respond: maybe it is blocking signals>
14:04:57.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:57.551335 29185 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 2.131s	user 0.739s	sys 1.259s
14:04:57.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:57.551692 29185 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 2.131s	user 0.739s	sys 1.259s
14:04:57.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:57.556288 29196 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 2134 milliseconds
14:04:57.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:57.557474 29197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:04:57.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.557542 29185 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:04:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.559021 29185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:04:57.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.561645 29185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:04:57.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.563093 29185 hybrid_clock.cc:648] HybridClock initialized: now 1779113097562956 us; error 129 us; skew 500 ppm
14:04:57.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.564280 29185 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:04:57.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.572394 29185 webserver.cc:533] Webserver started at http://127.26.160.3:42189/ using document root <none> and password file <none>
14:04:57.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.573764 29185 fs_manager.cc:362] Metadata directory not provided
14:04:57.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.574069 29185 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:04:57.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.582290 29185 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.007s	sys 0.000s
14:04:57.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.587355 29208 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:04:57.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.588755 29185 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.002s
14:04:57.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.589083 29185 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data,/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:04:57.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "353498eabc2240a29da45c31f7cc3c4e"
14:04:57.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:20 on dist-test-slave-8gxc"
14:04:57.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.591126 29185 fs_report.cc:389] FS layout report
14:04:57.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:04:57.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:04:57.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:04:57.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data/data
14:04:57.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:04:57.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:04:57.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:04:57.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:04:57.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:04:57.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:04:57.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:04:57.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:04:57.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:04:57.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:04:57.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:04:57.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.641091 29185 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:04:57.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.642764 29185 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:04:57.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.643239 29185 kserver.cc:163] Server-wide thread pool size limit: 3276
14:04:57.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.646333 29185 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:04:57.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.653012 29215 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:04:57.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.665261 29185 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:04:57.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.665568 29185 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.014s	user 0.002s	sys 0.000s
14:04:57.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.665911 29185 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:04:57.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.671140 29185 ts_tablet_manager.cc:616] Registered 1 tablets
14:04:57.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.671393 29185 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s	user 0.005s	sys 0.000s
14:04:57.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.671907 29215 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Bootstrap starting.
14:04:57.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.737613 29215 log.cc:826] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Log is configured to *not* fsync() on all Append() calls
14:04:57.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:57.853264 28846 consensus_peers.cc:597] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 -> Peer 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969): Couldn't send request to peer 353498eabc2240a29da45c31f7cc3c4e. Status: Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
14:04:57.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.856145 29185 rpc_server.cc:307] RPC server started. Bound to: 127.26.160.3:39969
14:04:57.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.856446 29322 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.160.3:39969 every 8 connection(s)
14:04:57.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.858879 29185 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data/info.pb
14:04:57.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.860546 29215 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=21 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:04:57.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.861577 29215 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Bootstrap complete.
14:04:57.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.863479 29215 ts_tablet_manager.cc:1403] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Time spent bootstrapping tablet: real 0.192s	user 0.149s	sys 0.039s
14:04:57.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.869604 27264 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu as pid 29185
14:04:57.870 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:04:57.902 [DEBUG - kudu-nio-0] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 24 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990, tablet=266c976e4879434f8b95ee412dfafcc5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:04:57.904 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.26.160.1:40635
14:04:57.905 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:04:57.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.905894 27264 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu with pid 28822
14:04:57.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.906018 29215 raft_consensus.cc:359] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:57.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.913851 29215 raft_consensus.cc:740] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353498eabc2240a29da45c31f7cc3c4e, State: Initialized, Role: FOLLOWER
14:04:57.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.914844 29215 consensus_queue.cc:260] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 8.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:57.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.925719 29215 ts_tablet_manager.cc:1434] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Time spent starting tablet: real 0.062s	user 0.028s	sys 0.008s
14:04:57.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.931293 29323 heartbeater.cc:344] Connected to a master server at 127.26.160.60:34099
14:04:57.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.931847 29323 heartbeater.cc:461] Registering TS with master...
14:04:57.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.933238 29323 heartbeater.cc:507] Master 127.26.160.60:34099 requested a full tablet report, sending...
14:04:57.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.934374 29324 heartbeater.cc:344] Connected to a master server at 127.26.160.61:45315
14:04:57.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.934777 29324 heartbeater.cc:461] Registering TS with master...
14:04:57.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.935818 29324 heartbeater.cc:507] Master 127.26.160.61:45315 requested a full tablet report, sending...
14:04:57.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.937386 27467 ts_manager.cc:194] Re-registered known tserver with Master: 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:04:57.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.940022 29325 heartbeater.cc:344] Connected to a master server at 127.26.160.62:43077
14:04:57.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.940378 29325 heartbeater.cc:461] Registering TS with master...
14:04:57.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.940462 27384 ts_manager.cc:194] Re-registered known tserver with Master: 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:04:57.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.940821 27467 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.26.160.3:44475
14:04:57.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.941306 29325 heartbeater.cc:507] Master 127.26.160.62:43077 requested a full tablet report, sending...
14:04:57.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.945047 29323 heartbeater.cc:499] Master 127.26.160.60:34099 was elected leader, sending a full tablet report...
14:04:57.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:57.945238 27310 ts_manager.cc:194] Re-registered known tserver with Master: 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:04:57.955 [DEBUG - kudu-nio-0] (Connection.java:239) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] handling channelInactive
14:04:57.955 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:04:57.955 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state READY due to: connection closed
14:04:57.956 [DEBUG - Time-limited test] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:57.957 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:57.957 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state CONNECTING due to: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:57.958 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:57.967 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:57.968 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)[L],d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:57.968 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=2, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=12), Traces: [0ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [1ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [2ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [11ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@2015585504(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@41ec897b -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@11c72646 -> wakeup thread Time-limited test)) after lookup
14:04:57.968 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:57.969 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:57.969 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state CONNECTING due to: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:57.969 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:57.987 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:57.988 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)[L],d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:57.988 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=4, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=32), Traces: [0ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [1ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [2ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [11ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [12ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [13ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [14ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [31ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@2015585504(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@41ec897b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@401764263 after retry RPC@1068469769), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@11c72646 -> wakeup thread Time-limited test -> (continuation of Deferred@401764263 after retry RPC@1068469769))) after lookup
14:04:57.988 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:57.989 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:57.990 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state CONNECTING due to: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:57.990 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:58.027 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:58.027 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)[L],d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:58.027 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=6, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=71), Traces: [0ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [1ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [2ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [11ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [12ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [13ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [14ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [31ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [32ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [34ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [34ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [71ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@2015585504(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@41ec897b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@401764263 after retry RPC@1068469769) -> (continuation of Deferred@1674460983 after retry RPC@550644353), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@11c72646 -> wakeup thread Time-limited test -> (continuation of Deferred@401764263 after retry RPC@1068469769) -> (continuation of Deferred@1674460983 after retry RPC@550644353))) after lookup
14:04:58.028 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:58.029 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:58.029 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state CONNECTING due to: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:58.029 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:58.087 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:58.088 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)[L],d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:58.088 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=8, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=132), Traces: [0ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [1ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [2ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [11ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [12ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [13ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [14ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [31ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [32ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [34ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [34ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [71ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [72ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [73ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [73ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [127ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [131ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@2015585504(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@41ec897b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@401764263 after retry RPC@1068469769) -> (continuation of Deferred@1674460983 after retry RPC@550644353) -> (continuation of Deferred@1151496081 after retry RPC@1964364710), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@11c72646 -> wakeup thread Time-limited test -> (continuation of Deferred@401764263 after retry RPC@1068469769) -> (continuation of Deferred@1674460983 after retry RPC@550644353) -> (continuation of Deferred@1151496081 after retry RPC@1964364710))) after lookup
14:04:58.089 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:58.090 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:58.090 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state CONNECTING due to: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:58.090 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:58.470 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:58.470 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)[L],d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:04:58.470 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=10, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=514), Traces: [0ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [1ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [2ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [11ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [12ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [13ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [14ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [31ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [32ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [34ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [34ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [71ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [72ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [73ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [73ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [127ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [131ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [132ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [134ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [134ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [508ms] refreshing cache from master, [508ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [513ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@2015585504(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@41ec897b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@401764263 after retry RPC@1068469769) -> (continuation of Deferred@1674460983 after retry RPC@550644353) -> (continuation of Deferred@1151496081 after retry RPC@1964364710) -> (continuation of Deferred@1246197280 after retry RPC@294917165), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@11c72646 -> wakeup thread Time-limited test -> (continuation of Deferred@401764263 after retry RPC@1068469769) -> (continuation of Deferred@1674460983 after retry RPC@550644353) -> (continuation of Deferred@1151496081 after retry RPC@1964364710) -> (continuation of Deferred@1246197280 after retry RPC@294917165))) after lookup
14:04:58.471 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:04:58.472 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:58.472 [DEBUG - kudu-nio-6] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state CONNECTING due to: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:58.473 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635
14:04:59.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:59.156378 29335 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 9f9c16da37ad440eb0d7a7a590be07a0)
14:04:59.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:59.156984 29335 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:59.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:59.158723 29335 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635), 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:04:59.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:59.162274 29022 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.160.1:40635: connect: Connection refused (error 111)
14:04:59.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:59.165879 29022 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635): Network error: Client connection negotiation failed: client connection to 127.26.160.1:40635: connect: Connection refused (error 111)
14:04:59.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:59.172137 29278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "353498eabc2240a29da45c31f7cc3c4e" is_pre_election: true
14:04:59.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:59.173039 29278 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1c13f941b874a778af09e1ef1b4a7bb in term 8.
14:04:59.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:59.174415 29022 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 353498eabc2240a29da45c31f7cc3c4e, d1c13f941b874a778af09e1ef1b4a7bb; no voters: 9f9c16da37ad440eb0d7a7a590be07a0
14:04:59.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:59.175200 29335 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 9 FOLLOWER]: Leader pre-election won for term 10
14:04:59.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:59.175536 29335 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 9 FOLLOWER]: Starting leader election (detected failure of leader 9f9c16da37ad440eb0d7a7a590be07a0)
14:04:59.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:59.175762 29335 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 9 FOLLOWER]: Advancing to term 10
14:04:59.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:59.180968 29335 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:59.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:59.182724 29335 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 10 election: Requested vote from peers 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635), 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:04:59.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:59.184340 29278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" candidate_term: 10 candidate_status { last_received { term: 9 index: 32 } } ignore_live_leader: false dest_uuid: "353498eabc2240a29da45c31f7cc3c4e"
14:04:59.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:59.184926 29278 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 8 FOLLOWER]: Advancing to term 10
14:04:59.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:59.186272 29022 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635): Network error: Client connection negotiation failed: client connection to 127.26.160.1:40635: connect: Connection refused (error 111)
14:04:59.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:59.191649 29278 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1c13f941b874a778af09e1ef1b4a7bb in term 10.
14:04:59.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:59.192637 29022 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 353498eabc2240a29da45c31f7cc3c4e, d1c13f941b874a778af09e1ef1b4a7bb; no voters: 9f9c16da37ad440eb0d7a7a590be07a0
14:04:59.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:59.193506 29335 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 10 FOLLOWER]: Leader election won for term 10
14:04:59.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:59.194674 29335 raft_consensus.cc:697] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 10 LEADER]: Becoming Leader. State: Replica: d1c13f941b874a778af09e1ef1b4a7bb, State: Running, Role: LEADER
14:04:59.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:59.195560 29335 consensus_queue.cc:237] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 9.32, Last appended by leader: 32, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:04:59.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:59.203832 27467 catalog_manager.cc:5696] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb reported cstate change: term changed from 9 to 10, leader changed from 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1) to d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2). New cstate: current_term: 10 leader_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } health_report { overall_health: HEALTHY } } }
14:04:59.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:04:59.646819 29022 consensus_peers.cc:597] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb -> Peer 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635): Couldn't send request to peer 9f9c16da37ad440eb0d7a7a590be07a0. Status: Network error: Client connection negotiation failed: client connection to 127.26.160.1:40635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:04:59.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:59.751346 29278 raft_consensus.cc:1275] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 10 FOLLOWER]: Refusing update from remote peer d1c13f941b874a778af09e1ef1b4a7bb: Log matching property violated. Preceding OpId in replica: term: 8 index: 28. Preceding OpId from leader: term: 10 index: 33. (index mismatch)
14:04:59.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:04:59.752962 29335 consensus_queue.cc:1048] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 28, Time since last communication: 0.000s
14:04:59.988 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:04:59.988 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)[L]]}]
14:04:59.989 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=12, sequence_id=25, TimeoutTracker(timeout=30000, elapsed=2033), Traces: [0ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [1ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [2ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [7ms] refreshing cache from master, [7ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [11ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [12ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [13ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [14ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [27ms] refreshing cache from master, [27ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [31ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [32ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [34ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [34ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [67ms] refreshing cache from master, [67ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [71ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [72ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [73ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [73ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [127ms] refreshing cache from master, [127ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [131ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [132ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [134ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [134ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [508ms] refreshing cache from master, [508ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [513ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [515ms] sending RPC to server 9f9c16da37ad440eb0d7a7a590be07a0, [516ms] received response from server 9f9c16da37ad440eb0d7a7a590be07a0: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [517ms] delaying RPC due to: Network error: Failed to connect to peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635): Connection refused: /127.26.160.1:40635, [2028ms] refreshing cache from master, [2028ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [2032ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@2015585504(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@41ec897b -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@401764263 after retry RPC@1068469769) -> (continuation of Deferred@1674460983 after retry RPC@550644353) -> (continuation of Deferred@1151496081 after retry RPC@1964364710) -> (continuation of Deferred@1246197280 after retry RPC@294917165) -> (continuation of Deferred@266637503 after retry RPC@1046530066), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@11c72646 -> wakeup thread Time-limited test -> (continuation of Deferred@401764263 after retry RPC@1068469769) -> (continuation of Deferred@1674460983 after retry RPC@550644353) -> (continuation of Deferred@1151496081 after retry RPC@1964364710) -> (continuation of Deferred@1246197280 after retry RPC@294917165) -> (continuation of Deferred@266637503 after retry RPC@1046530066))) after lookup
14:04:59.990 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:04:59.991 [DEBUG - kudu-nio-7] (Connection.java:757) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] Successfully connected to peer
14:05:00.024 [DEBUG - kudu-nio-7] (Negotiator.java:919) Authenticated connection [id: 0x68e541b3, L:/127.0.0.1:55270 - R:/127.26.160.2:37023] using TOKEN/null
14:05:00.080 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.26.160.1:40635
14:05:00.081 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 0
  }
}

14:05:00.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:00.082512 27264 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:05:00.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:05:00.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal
14:05:00.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data
14:05:00.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:05:00.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:05:00.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:05:00.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:05:00.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:05:00.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:05:00.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/logs
14:05:00.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data/info.pb
14:05:00.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:05:00.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:05:00.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:05:00.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:05:00.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:05:00.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:05:00.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:05:00.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:05:00.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:05:00.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:05:00.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.1:40635
14:05:00.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.1
14:05:00.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:05:00.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34423
14:05:00.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.1
14:05:00.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:05:00.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:05:00.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:05:00.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:05:00.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:00.423207 29364 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:05:00.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:00.424001 29364 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:05:00.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:00.424733 29364 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:05:00.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:00.462847 29364 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:05:00.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:00.463935 29364 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.160.1
14:05:00.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:00.505926 29364 tablet_server_runner.cc:78] Tablet server non-default flags:
14:05:00.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:05:00.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:05:00.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.1:40635
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data/info.pb
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.1
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34423
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.1
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/logs
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:22 UTC on e7f111948823
14:05:00.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11959
14:05:00.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:05:00.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:00.507351 29364 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:05:00.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:00.509228 29364 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:05:00.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:00.524012 29373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:01.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:00.524682 29372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:01.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:00.528148 29375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:01.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:01.761132 29374 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
14:05:01.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.761350 29364 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:05:01.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.773424 29364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:05:01.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.776649 29364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:05:01.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.780426 29364 hybrid_clock.cc:648] HybridClock initialized: now 1779113101780186 us; error 213 us; skew 500 ppm
14:05:01.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.781832 29364 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:05:01.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.792132 29364 webserver.cc:533] Webserver started at http://127.26.160.1:34423/ using document root <none> and password file <none>
14:05:01.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.793713 29364 fs_manager.cc:362] Metadata directory not provided
14:05:01.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.794049 29364 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:05:01.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.804792 29364 fs_manager.cc:714] Time spent opening directory manager: real 0.007s	user 0.005s	sys 0.000s
14:05:01.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.810596 29387 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:05:01.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.812268 29364 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.002s	sys 0.002s
14:05:01.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.812682 29364 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data,/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal
14:05:01.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "9f9c16da37ad440eb0d7a7a590be07a0"
14:05:01.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:15 on dist-test-slave-8gxc"
14:05:01.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.815356 29364 fs_report.cc:389] FS layout report
14:05:01.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:05:01.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal
14:05:01.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/wal
14:05:01.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data/data
14:05:01.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:05:01.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:05:01.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:05:01.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:05:01.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:05:01.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:05:01.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:05:01.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:05:01.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:05:01.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:05:01.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:05:01.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.911093 29364 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:05:01.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.912858 29364 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:05:01.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.913272 29364 kserver.cc:163] Server-wide thread pool size limit: 3276
14:05:01.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.915935 29364 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:05:01.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.921969 29394 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:05:01.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.931226 29364 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:05:01.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.931457 29364 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s	user 0.001s	sys 0.001s
14:05:01.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.931728 29364 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:05:01.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.936573 29364 ts_tablet_manager.cc:616] Registered 1 tablets
14:05:01.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.936805 29364 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.003s	sys 0.000s
14:05:01.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:01.937259 29394 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Bootstrap starting.
14:05:02.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.004848 29394 log.cc:826] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Log is configured to *not* fsync() on all Append() calls
14:05:02.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.113468 29394 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=24 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:05:02.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.114373 29394 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Bootstrap complete.
14:05:02.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.116015 29394 ts_tablet_manager.cc:1403] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Time spent bootstrapping tablet: real 0.179s	user 0.123s	sys 0.052s
14:05:02.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.126920 29364 rpc_server.cc:307] RPC server started. Bound to: 127.26.160.1:40635
14:05:02.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.127092 29501 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.160.1:40635 every 8 connection(s)
14:05:02.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.130667 29364 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-0/data/info.pb
14:05:02.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.131196 27264 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu as pid 29364
14:05:02.131 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:05:02.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.136560 29394 raft_consensus.cc:359] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:05:02.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.146872 29394 raft_consensus.cc:740] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9c16da37ad440eb0d7a7a590be07a0, State: Initialized, Role: FOLLOWER
14:05:02.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.150978 29394 consensus_queue.cc:260] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 9.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:05:02.171 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 27 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990, tablet=266c976e4879434f8b95ee412dfafcc5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:05:02.173 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.26.160.2:37023
14:05:02.173 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:05:02.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.174127 27264 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu with pid 28999
14:05:02.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.183861 29502 heartbeater.cc:344] Connected to a master server at 127.26.160.60:34099
14:05:02.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.184368 29502 heartbeater.cc:461] Registering TS with master...
14:05:02.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.185894 29502 heartbeater.cc:507] Master 127.26.160.60:34099 requested a full tablet report, sending...
14:05:02.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.187479 29394 ts_tablet_manager.cc:1434] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0: Time spent starting tablet: real 0.071s	user 0.035s	sys 0.023s
14:05:02.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.190305 27467 ts_manager.cc:194] Re-registered known tserver with Master: 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635)
14:05:02.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.190724 29503 heartbeater.cc:344] Connected to a master server at 127.26.160.61:45315
14:05:02.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.191090 29503 heartbeater.cc:461] Registering TS with master...
14:05:02.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.192036 29503 heartbeater.cc:507] Master 127.26.160.61:45315 requested a full tablet report, sending...
14:05:02.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.193509 27467 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.26.160.1:35251
14:05:02.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.196100 27384 ts_manager.cc:194] Re-registered known tserver with Master: 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635)
14:05:02.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.197448 29506 heartbeater.cc:344] Connected to a master server at 127.26.160.62:43077
14:05:02.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.197769 29506 heartbeater.cc:461] Registering TS with master...
14:05:02.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.198400 29502 heartbeater.cc:499] Master 127.26.160.60:34099 was elected leader, sending a full tablet report...
14:05:02.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.198529 29506 heartbeater.cc:507] Master 127.26.160.62:43077 requested a full tablet report, sending...
14:05:02.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:02.202314 27310 ts_manager.cc:194] Re-registered known tserver with Master: 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635)
14:05:02.211 [DEBUG - kudu-nio-7] (Connection.java:239) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] handling channelInactive
14:05:02.211 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:05:02.211 [DEBUG - kudu-nio-7] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state READY due to: connection closed
14:05:02.211 [DEBUG - Time-limited test] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:05:02.212 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:05:02.213 [DEBUG - kudu-nio-0] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state CONNECTING due to: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:05:02.213 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:05:02.228 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:05:02.228 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)[L]]}]
14:05:02.228 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=2, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=17), Traces: [0ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [2ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [2ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [12ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [16ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1246236766(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@7250615d -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@41a4bdb -> wakeup thread Time-limited test)) after lookup
14:05:02.229 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:05:02.230 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:05:02.230 [DEBUG - kudu-nio-1] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state CONNECTING due to: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:05:02.230 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:05:02.247 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:05:02.247 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)[L]]}]
14:05:02.248 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=4, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=36), Traces: [0ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [2ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [2ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [12ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [16ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [17ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [19ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [19ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [36ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1246236766(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@7250615d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@216320225 after retry RPC@1406564746), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@41a4bdb -> wakeup thread Time-limited test -> (continuation of Deferred@216320225 after retry RPC@1406564746))) after lookup
14:05:02.249 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:05:02.250 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:05:02.250 [DEBUG - kudu-nio-2] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state CONNECTING due to: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:05:02.250 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:05:02.287 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:05:02.287 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)[L]]}]
14:05:02.288 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=6, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=76), Traces: [0ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [2ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [2ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [12ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [16ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [17ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [19ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [19ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [36ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [37ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [39ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [39ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [76ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1246236766(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@7250615d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@216320225 after retry RPC@1406564746) -> (continuation of Deferred@960073460 after retry RPC@1497606768), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@41a4bdb -> wakeup thread Time-limited test -> (continuation of Deferred@216320225 after retry RPC@1406564746) -> (continuation of Deferred@960073460 after retry RPC@1497606768))) after lookup
14:05:02.288 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:05:02.289 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:05:02.290 [DEBUG - kudu-nio-3] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state CONNECTING due to: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:05:02.290 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:05:02.347 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:05:02.347 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)[L]]}]
14:05:02.348 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=8, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=136), Traces: [0ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [2ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [2ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [12ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [16ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [17ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [19ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [19ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [36ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [37ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [39ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [39ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [76ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [77ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [79ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [79ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [132ms] refreshing cache from master, [132ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [136ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1246236766(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@7250615d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@216320225 after retry RPC@1406564746) -> (continuation of Deferred@960073460 after retry RPC@1497606768) -> (continuation of Deferred@270982031 after retry RPC@1425204177), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@41a4bdb -> wakeup thread Time-limited test -> (continuation of Deferred@216320225 after retry RPC@1406564746) -> (continuation of Deferred@960073460 after retry RPC@1497606768) -> (continuation of Deferred@270982031 after retry RPC@1425204177))) after lookup
14:05:02.349 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:05:02.350 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:05:02.350 [DEBUG - kudu-nio-4] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state CONNECTING due to: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:05:02.350 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:05:02.747 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:05:02.747 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)[L]]}]
14:05:02.747 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=10, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=536), Traces: [0ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [2ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [2ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [12ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [16ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [17ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [19ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [19ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [36ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [37ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [39ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [39ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [76ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [77ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [79ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [79ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [132ms] refreshing cache from master, [132ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [136ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [138ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [139ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [139ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [532ms] refreshing cache from master, [532ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [535ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1246236766(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@7250615d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@216320225 after retry RPC@1406564746) -> (continuation of Deferred@960073460 after retry RPC@1497606768) -> (continuation of Deferred@270982031 after retry RPC@1425204177) -> (continuation of Deferred@797805406 after retry RPC@1814173973), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@41a4bdb -> wakeup thread Time-limited test -> (continuation of Deferred@216320225 after retry RPC@1406564746) -> (continuation of Deferred@960073460 after retry RPC@1497606768) -> (continuation of Deferred@270982031 after retry RPC@1425204177) -> (continuation of Deferred@797805406 after retry RPC@1814173973))) after lookup
14:05:02.748 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:05:02.749 [INFO - kudu-nio-5] (Connection.java:429) Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:05:02.749 [DEBUG - kudu-nio-5] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state CONNECTING due to: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:05:02.749 [INFO - kudu-nio-5] (AsyncKuduClient.java:2355) Invalidating location d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:05:03.168 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:05:03.168 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)[L]]}]
14:05:03.168 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=12, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=957), Traces: [0ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [2ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [2ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [12ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [16ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [17ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [19ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [19ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [36ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [37ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [39ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [39ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [76ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [77ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [79ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [79ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [132ms] refreshing cache from master, [132ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [136ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [138ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [139ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [139ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [532ms] refreshing cache from master, [532ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [535ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [537ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [538ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [538ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [952ms] refreshing cache from master, [953ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [956ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1246236766(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@7250615d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@216320225 after retry RPC@1406564746) -> (continuation of Deferred@960073460 after retry RPC@1497606768) -> (continuation of Deferred@270982031 after retry RPC@1425204177) -> (continuation of Deferred@797805406 after retry RPC@1814173973) -> (continuation of Deferred@1928329686 after retry RPC@251247663), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@41a4bdb -> wakeup thread Time-limited test -> (continuation of Deferred@216320225 after retry RPC@1406564746) -> (continuation of Deferred@960073460 after retry RPC@1497606768) -> (continuation of Deferred@270982031 after retry RPC@1425204177) -> (continuation of Deferred@797805406 after retry RPC@1814173973) -> (continuation of Deferred@1928329686 after retry RPC@251247663))) after lookup
14:05:03.169 [DEBUG - kudu-nio-1] (Connection.java:748) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] connecting to peer
14:05:03.170 [INFO - kudu-nio-6] (Connection.java:429) Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:05:03.170 [DEBUG - kudu-nio-6] (Connection.java:709) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] cleaning up while in state CONNECTING due to: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:05:03.170 [INFO - kudu-nio-6] (AsyncKuduClient.java:2355) Invalidating location d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023
14:05:03.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.284729 29513 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d1c13f941b874a778af09e1ef1b4a7bb)
14:05:03.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.285144 29513 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:05:03.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.287292 29513 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635), d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:05:03.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:03.290402 29211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.160.2:37023: connect: Connection refused (error 111)
14:05:03.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:03.295802 29211 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023): Network error: Client connection negotiation failed: client connection to 127.26.160.2:37023: connect: Connection refused (error 111)
14:05:03.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.299810 29457 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "353498eabc2240a29da45c31f7cc3c4e" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" is_pre_election: true
14:05:03.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.300573 29457 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 353498eabc2240a29da45c31f7cc3c4e in term 9.
14:05:03.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.302162 29212 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 353498eabc2240a29da45c31f7cc3c4e, 9f9c16da37ad440eb0d7a7a590be07a0; no voters: d1c13f941b874a778af09e1ef1b4a7bb
14:05:03.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.303004 29513 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 10 FOLLOWER]: Leader pre-election won for term 11
14:05:03.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.303347 29513 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 10 FOLLOWER]: Starting leader election (detected failure of leader d1c13f941b874a778af09e1ef1b4a7bb)
14:05:03.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.303871 29513 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 10 FOLLOWER]: Advancing to term 11
14:05:03.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.308575 29513 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:05:03.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.310320 29513 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 11 election: Requested vote from peers 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635), d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:05:03.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.311233 29457 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "353498eabc2240a29da45c31f7cc3c4e" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "9f9c16da37ad440eb0d7a7a590be07a0"
14:05:03.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.311741 29457 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 9 FOLLOWER]: Advancing to term 11
14:05:03.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:03.314070 29211 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023): Network error: Client connection negotiation failed: client connection to 127.26.160.2:37023: connect: Connection refused (error 111)
14:05:03.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.318894 29457 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 353498eabc2240a29da45c31f7cc3c4e in term 11.
14:05:03.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.320046 29212 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 353498eabc2240a29da45c31f7cc3c4e, 9f9c16da37ad440eb0d7a7a590be07a0; no voters: d1c13f941b874a778af09e1ef1b4a7bb
14:05:03.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.320791 29513 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 11 FOLLOWER]: Leader election won for term 11
14:05:03.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.322363 29513 raft_consensus.cc:697] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 11 LEADER]: Becoming Leader. State: Replica: 353498eabc2240a29da45c31f7cc3c4e, State: Running, Role: LEADER
14:05:03.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.323618 29513 consensus_queue.cc:237] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 10.36, Last appended by leader: 36, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:05:03.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.334379 27467 catalog_manager.cc:5696] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e reported cstate change: term changed from 10 to 11, leader changed from d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2) to 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3). New cstate: current_term: 11 leader_uuid: "353498eabc2240a29da45c31f7cc3c4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } health_report { overall_health: UNKNOWN } } }
14:05:03.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.708003 29457 raft_consensus.cc:1275] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 11 FOLLOWER]: Refusing update from remote peer 353498eabc2240a29da45c31f7cc3c4e: Log matching property violated. Preceding OpId in replica: term: 9 index: 32. Preceding OpId from leader: term: 11 index: 37. (index mismatch)
14:05:03.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.709412 29513 consensus_queue.cc:1048] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 32, Time since last communication: 0.000s
14:05:03.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:03.724539 29531 mvcc.cc:204] Tried to move back new op lower bound from 7287247257733701632 to 7287247254312271872. Current Snapshot: MvccSnapshot[applied={T|T < 7287247257733701632}]
14:05:03.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:03.741901 29211 consensus_peers.cc:597] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e -> Peer d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023): Couldn't send request to peer d1c13f941b874a778af09e1ef1b4a7bb. Status: Network error: Client connection negotiation failed: client connection to 127.26.160.2:37023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:05:04.808 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:05:04.808 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:05:04.809 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=14, sequence_id=28, TimeoutTracker(timeout=30000, elapsed=2597), Traces: [0ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [2ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [2ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [12ms] refreshing cache from master, [13ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [16ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [17ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [19ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [19ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [32ms] refreshing cache from master, [32ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [36ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [37ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [39ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [39ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [72ms] refreshing cache from master, [72ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [76ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [77ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [79ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [79ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [132ms] refreshing cache from master, [132ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [136ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [138ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [139ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [139ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [532ms] refreshing cache from master, [532ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [535ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [537ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [538ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [538ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [952ms] refreshing cache from master, [953ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [956ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [958ms] sending RPC to server d1c13f941b874a778af09e1ef1b4a7bb, [959ms] received response from server d1c13f941b874a778af09e1ef1b4a7bb: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [959ms] delaying RPC due to: Network error: Failed to connect to peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023): Connection refused: /127.26.160.2:37023, [2593ms] refreshing cache from master, [2593ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [2597ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@1246236766(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@7250615d -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@216320225 after retry RPC@1406564746) -> (continuation of Deferred@960073460 after retry RPC@1497606768) -> (continuation of Deferred@270982031 after retry RPC@1425204177) -> (continuation of Deferred@797805406 after retry RPC@1814173973) -> (continuation of Deferred@1928329686 after retry RPC@251247663) -> (continuation of Deferred@1711855285 after retry RPC@634590879), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@41a4bdb -> wakeup thread Time-limited test -> (continuation of Deferred@216320225 after retry RPC@1406564746) -> (continuation of Deferred@960073460 after retry RPC@1497606768) -> (continuation of Deferred@270982031 after retry RPC@1425204177) -> (continuation of Deferred@797805406 after retry RPC@1814173973) -> (continuation of Deferred@1928329686 after retry RPC@251247663) -> (continuation of Deferred@1711855285 after retry RPC@634590879))) after lookup
14:05:04.810 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:05:04.811 [DEBUG - kudu-nio-7] (Connection.java:757) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] Successfully connected to peer
14:05:04.830 [DEBUG - kudu-nio-7] (Negotiator.java:919) Authenticated connection [id: 0xcbe79d2e, L:/127.0.0.1:48446 - R:/127.26.160.3:39969] using TOKEN/null
14:05:04.888 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.26.160.2:37023
14:05:04.888 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 1
  }
}

14:05:04.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:04.890460 27264 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/logs
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data/info.pb
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.2:37023
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.2
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34355
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.2
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:05:04.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:05:05.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:05.285699 29549 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:05:05.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:05.286260 29549 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:05:05.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:05.286727 29549 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:05:05.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:05.326937 29549 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:05:05.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:05.327782 29549 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.160.2
14:05:05.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:05.369621 29549 tablet_server_runner.cc:78] Tablet server non-default flags:
14:05:05.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:05:05.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:05:05.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:05:05.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:05:05.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data
14:05:05.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal
14:05:05.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:05:05.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:05:05.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.2:37023
14:05:05.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:05:05.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:05:05.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:05:05.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data/info.pb
14:05:05.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.2
14:05:05.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=34355
14:05:05.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:05:05.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:05:05.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:05:05.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:05:05.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:05:05.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:05:05.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.2
14:05:05.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/logs
14:05:05.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:05:05.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:05:05.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:05:05.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:05:05.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:05:05.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:05:05.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:05:05.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:22 UTC on e7f111948823
14:05:05.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11959
14:05:05.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:05:05.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:05.371482 29549 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:05:05.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:05.374212 29549 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:05:05.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:05.388594 29559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:06.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:06.205508 29211 consensus_peers.cc:597] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e -> Peer d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023): Couldn't send request to peer d1c13f941b874a778af09e1ef1b4a7bb. Status: Network error: Client connection negotiation failed: client connection to 127.26.160.2:37023: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:05:07.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:05.390758 29560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:07.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:06.792688 29558 debug-util.cc:398] Leaking SignalData structure 0x7b08000271c0 after lost signal to thread 29549
14:05:07.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:07.231352 29549 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.841s	user 0.598s	sys 1.130s
14:05:07.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:07.233503 29549 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.844s	user 0.598s	sys 1.130s
14:05:07.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:07.234445 29562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:07.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:07.238831 29561 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1842 milliseconds
14:05:07.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.238853 29549 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:05:07.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.240535 29549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:05:07.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.243424 29549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:05:07.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.244928 29549 hybrid_clock.cc:648] HybridClock initialized: now 1779113107244831 us; error 98 us; skew 500 ppm
14:05:07.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.246011 29549 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:05:07.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.254858 29549 webserver.cc:533] Webserver started at http://127.26.160.2:34355/ using document root <none> and password file <none>
14:05:07.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.256016 29549 fs_manager.cc:362] Metadata directory not provided
14:05:07.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.256234 29549 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:05:07.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.265229 29549 fs_manager.cc:714] Time spent opening directory manager: real 0.006s	user 0.001s	sys 0.005s
14:05:07.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.270238 29571 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:05:07.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.271623 29549 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.000s	sys 0.003s
14:05:07.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.271952 29549 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data,/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal
14:05:07.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d1c13f941b874a778af09e1ef1b4a7bb"
14:05:07.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:18 on dist-test-slave-8gxc"
14:05:07.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.274153 29549 fs_report.cc:389] FS layout report
14:05:07.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:05:07.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal
14:05:07.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/wal
14:05:07.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data/data
14:05:07.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:05:07.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:05:07.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:05:07.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:05:07.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:05:07.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:05:07.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:05:07.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:05:07.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:05:07.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:05:07.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:05:07.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.360632 29549 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:05:07.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.362720 29549 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:05:07.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.363291 29549 kserver.cc:163] Server-wide thread pool size limit: 3276
14:05:07.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.367041 29549 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:05:07.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.374989 29579 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:05:07.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.385298 29549 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:05:07.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.385572 29549 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.013s	user 0.002s	sys 0.001s
14:05:07.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.385908 29549 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:05:07.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.390954 29549 ts_tablet_manager.cc:616] Registered 1 tablets
14:05:07.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.391331 29549 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s	user 0.004s	sys 0.001s
14:05:07.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.391744 29579 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Bootstrap starting.
14:05:07.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.492107 29579 log.cc:826] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Log is configured to *not* fsync() on all Append() calls
14:05:07.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.592697 29549 rpc_server.cc:307] RPC server started. Bound to: 127.26.160.2:37023
14:05:07.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.592825 29686 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.160.2:37023 every 8 connection(s)
14:05:07.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.596824 29549 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-1/data/info.pb
14:05:07.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.600394 27264 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu as pid 29549
14:05:07.601 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:05:07.651 [DEBUG - kudu-nio-7] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 30 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990, tablet=266c976e4879434f8b95ee412dfafcc5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:05:07.652 [INFO - Time-limited test] (MiniKuduCluster.java:475) Killing tablet server 127.26.160.3:39969
14:05:07.653 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:05:07.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.653656 27264 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu with pid 29185
14:05:07.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.685026 29688 heartbeater.cc:344] Connected to a master server at 127.26.160.61:45315
14:05:07.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.685587 29688 heartbeater.cc:461] Registering TS with master...
14:05:07.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.687251 29688 heartbeater.cc:507] Master 127.26.160.61:45315 requested a full tablet report, sending...
14:05:07.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.693517 27385 ts_manager.cc:194] Re-registered known tserver with Master: d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:05:07.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.698601 29687 heartbeater.cc:344] Connected to a master server at 127.26.160.60:34099
14:05:07.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.698976 29687 heartbeater.cc:461] Registering TS with master...
14:05:07.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.699961 29687 heartbeater.cc:507] Master 127.26.160.60:34099 requested a full tablet report, sending...
14:05:07.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.703755 27467 ts_manager.cc:194] Re-registered known tserver with Master: d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:05:07.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.703961 29579 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=27 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:05:07.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.704985 29579 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Bootstrap complete.
14:05:07.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.707082 29579 ts_tablet_manager.cc:1403] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Time spent bootstrapping tablet: real 0.316s	user 0.198s	sys 0.047s
14:05:07.710 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:05:07.711 [DEBUG - kudu-nio-7] (Connection.java:239) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] handling channelInactive
14:05:07.711 [DEBUG - kudu-nio-7] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state READY due to: connection closed
14:05:07.712 [INFO - kudu-nio-7] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: connection closed
14:05:07.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.715134 27467 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.26.160.2:56509
14:05:07.732 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:05:07.733 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:05:07.733 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=2, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=22), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [1ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [21ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@485422924(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@21347da0 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@495d0832 -> wakeup thread Time-limited test)) after lookup
14:05:07.733 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:05:07.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.730997 29579 raft_consensus.cc:359] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 10 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:05:07.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.734148 29579 raft_consensus.cc:740] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 10 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c13f941b874a778af09e1ef1b4a7bb, State: Initialized, Role: FOLLOWER
14:05:07.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.734735 29692 heartbeater.cc:344] Connected to a master server at 127.26.160.62:43077
14:05:07.735 [INFO - kudu-nio-0] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:05:07.735 [DEBUG - kudu-nio-0] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:05:07.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.735148 29692 heartbeater.cc:461] Registering TS with master...
14:05:07.735 [INFO - kudu-nio-0] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:05:07.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.735096 29579 consensus_queue.cc:260] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 10.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:05:07.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.736099 29692 heartbeater.cc:507] Master 127.26.160.62:43077 requested a full tablet report, sending...
14:05:07.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.738723 29687 heartbeater.cc:499] Master 127.26.160.60:34099 was elected leader, sending a full tablet report...
14:05:07.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.740288 27309 ts_manager.cc:194] Re-registered known tserver with Master: d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:05:07.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:07.742326 29579 ts_tablet_manager.cc:1434] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb: Time spent starting tablet: real 0.035s	user 0.027s	sys 0.001s
14:05:07.751 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:05:07.751 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:05:07.752 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=4, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=41), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [1ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [21ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [22ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [24ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [24ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [40ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@485422924(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@21347da0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@91712394 after retry RPC@617279374), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@495d0832 -> wakeup thread Time-limited test -> (continuation of Deferred@91712394 after retry RPC@617279374))) after lookup
14:05:07.753 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:05:07.756 [INFO - kudu-nio-1] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:05:07.756 [DEBUG - kudu-nio-1] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:05:07.756 [INFO - kudu-nio-1] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:05:07.789 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:05:07.789 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:05:07.790 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=6, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=79), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [1ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [21ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [22ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [24ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [24ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [40ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [42ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [45ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [45ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [77ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@485422924(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@21347da0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@91712394 after retry RPC@617279374) -> (continuation of Deferred@1037326997 after retry RPC@498123705), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@495d0832 -> wakeup thread Time-limited test -> (continuation of Deferred@91712394 after retry RPC@617279374) -> (continuation of Deferred@1037326997 after retry RPC@498123705))) after lookup
14:05:07.791 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:05:07.794 [INFO - kudu-nio-2] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:05:07.794 [DEBUG - kudu-nio-2] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:05:07.794 [INFO - kudu-nio-2] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:05:07.907 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:05:07.908 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:05:07.908 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=8, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=197), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [1ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [21ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [22ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [24ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [24ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [40ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [42ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [45ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [45ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [77ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [79ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [83ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [83ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [192ms] refreshing cache from master, [192ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [196ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@485422924(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@21347da0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@91712394 after retry RPC@617279374) -> (continuation of Deferred@1037326997 after retry RPC@498123705) -> (continuation of Deferred@1484234040 after retry RPC@1136730244), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@495d0832 -> wakeup thread Time-limited test -> (continuation of Deferred@91712394 after retry RPC@617279374) -> (continuation of Deferred@1037326997 after retry RPC@498123705) -> (continuation of Deferred@1484234040 after retry RPC@1136730244))) after lookup
14:05:07.909 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:05:07.910 [INFO - kudu-nio-3] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:05:07.910 [DEBUG - kudu-nio-3] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:05:07.910 [INFO - kudu-nio-3] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:05:08.148 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:05:08.148 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)[L],9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635),d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:05:08.149 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=10, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=437), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [1ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [21ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [22ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [24ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [24ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [40ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [42ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [45ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [45ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [77ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [79ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [83ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [83ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [192ms] refreshing cache from master, [192ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [196ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [198ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [199ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [199ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [433ms] refreshing cache from master, [433ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [437ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@485422924(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@21347da0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@91712394 after retry RPC@617279374) -> (continuation of Deferred@1037326997 after retry RPC@498123705) -> (continuation of Deferred@1484234040 after retry RPC@1136730244) -> (continuation of Deferred@389117843 after retry RPC@1564385521), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@495d0832 -> wakeup thread Time-limited test -> (continuation of Deferred@91712394 after retry RPC@617279374) -> (continuation of Deferred@1037326997 after retry RPC@498123705) -> (continuation of Deferred@1484234040 after retry RPC@1136730244) -> (continuation of Deferred@389117843 after retry RPC@1564385521))) after lookup
14:05:08.149 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] connecting to peer
14:05:08.151 [INFO - kudu-nio-4] (Connection.java:429) Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:05:08.151 [DEBUG - kudu-nio-4] (Connection.java:709) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] cleaning up while in state CONNECTING due to: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:05:08.151 [INFO - kudu-nio-4] (AsyncKuduClient.java:2355) Invalidating location 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969) for tablet 266c976e4879434f8b95ee412dfafcc5: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969
14:05:08.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:08.921824 29701 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
14:05:08.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:08.922333 29701 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:05:08.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:08.924682 29701 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1:40635), 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:05:08.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:08.931252 29576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111)
14:05:08.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:08.936159 29576 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969): Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111)
14:05:08.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:08.938959 29457 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" candidate_term: 11 candidate_status { last_received { term: 10 index: 36 } } ignore_live_leader: false dest_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" is_pre_election: true
14:05:08.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:08.940675 29576 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1c13f941b874a778af09e1ef1b4a7bb; no voters: 353498eabc2240a29da45c31f7cc3c4e, 9f9c16da37ad440eb0d7a7a590be07a0
14:05:08.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:08.942217 29701 raft_consensus.cc:2749] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
14:05:09.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.126915 29706 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 353498eabc2240a29da45c31f7cc3c4e)
14:05:09.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.127691 29706 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:05:09.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.130980 29706 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969), d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:05:09.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:09.135715 29391 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111)
14:05:09.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:09.142045 29391 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969): Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111)
14:05:09.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.149117 29642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" candidate_term: 12 candidate_status { last_received { term: 11 index: 40 } } ignore_live_leader: false dest_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" is_pre_election: true
14:05:09.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.150158 29642 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f9c16da37ad440eb0d7a7a590be07a0 in term 10.
14:05:09.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.152232 29390 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9f9c16da37ad440eb0d7a7a590be07a0, d1c13f941b874a778af09e1ef1b4a7bb; no voters: 353498eabc2240a29da45c31f7cc3c4e
14:05:09.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.153556 29706 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 11 FOLLOWER]: Leader pre-election won for term 12
14:05:09.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.153945 29706 raft_consensus.cc:493] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 353498eabc2240a29da45c31f7cc3c4e)
14:05:09.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.154565 29706 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 11 FOLLOWER]: Advancing to term 12
14:05:09.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.160065 29706 raft_consensus.cc:515] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:05:09.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.163244 29706 leader_election.cc:290] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 12 election: Requested vote from peers 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969), d1c13f941b874a778af09e1ef1b4a7bb (127.26.160.2:37023)
14:05:09.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.164784 29642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "266c976e4879434f8b95ee412dfafcc5" candidate_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" candidate_term: 12 candidate_status { last_received { term: 11 index: 40 } } ignore_live_leader: false dest_uuid: "d1c13f941b874a778af09e1ef1b4a7bb"
14:05:09.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.165324 29642 raft_consensus.cc:3060] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 10 FOLLOWER]: Advancing to term 12
14:05:09.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:09.168147 29391 leader_election.cc:336] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969): Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111)
14:05:09.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.175066 29642 raft_consensus.cc:2468] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f9c16da37ad440eb0d7a7a590be07a0 in term 12.
14:05:09.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.176462 29390 leader_election.cc:304] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9f9c16da37ad440eb0d7a7a590be07a0, d1c13f941b874a778af09e1ef1b4a7bb; no voters: 353498eabc2240a29da45c31f7cc3c4e
14:05:09.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.177134 29706 raft_consensus.cc:2804] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 12 FOLLOWER]: Leader election won for term 12
14:05:09.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.178809 29706 raft_consensus.cc:697] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [term 12 LEADER]: Becoming Leader. State: Replica: 9f9c16da37ad440eb0d7a7a590be07a0, State: Running, Role: LEADER
14:05:09.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.179975 29706 consensus_queue.cc:237] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 11.40, Last appended by leader: 40, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:05:09.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.193044 27467 catalog_manager.cc:5696] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 reported cstate change: term changed from 11 to 12, leader changed from 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3) to 9f9c16da37ad440eb0d7a7a590be07a0 (127.26.160.1). New cstate: current_term: 12 leader_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } health_report { overall_health: UNKNOWN } } }
14:05:09.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.567662 29642 raft_consensus.cc:1275] T 266c976e4879434f8b95ee412dfafcc5 P d1c13f941b874a778af09e1ef1b4a7bb [term 12 FOLLOWER]: Refusing update from remote peer 9f9c16da37ad440eb0d7a7a590be07a0: Log matching property violated. Preceding OpId in replica: term: 10 index: 36. Preceding OpId from leader: term: 12 index: 41. (index mismatch)
14:05:09.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.569311 29706 consensus_queue.cc:1048] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
14:05:09.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:09.604588 29391 consensus_peers.cc:597] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 -> Peer 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969): Couldn't send request to peer 353498eabc2240a29da45c31f7cc3c4e. Status: Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
14:05:09.709 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2538) Learned about tablet 266c976e4879434f8b95ee412dfafcc5 for table 'org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990' with partition [<start>, <end>)
14:05:09.710 [DEBUG - kudu-nio-1] (TableLocationsCache.java:192) Discovered table locations:	[Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=266c976e4879434f8b95ee412dfafcc5@[353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969),9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)[L],d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)]}]
14:05:09.710 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1488) Retrying sending RPC KuduRpc(method=Write, tablet=266c976e4879434f8b95ee412dfafcc5, attempt=12, sequence_id=31, TimeoutTracker(timeout=30000, elapsed=1999), Traces: [0ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [1ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: connection closed, [1ms] delaying RPC due to: Network error: connection closed, [12ms] refreshing cache from master, [12ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [21ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [22ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [24ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [24ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [33ms] refreshing cache from master, [33ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [40ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [42ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [45ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [45ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [73ms] refreshing cache from master, [73ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [77ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [79ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [83ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [83ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [192ms] refreshing cache from master, [192ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [196ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [198ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [199ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [199ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [433ms] refreshing cache from master, [433ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [437ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, [438ms] sending RPC to server 353498eabc2240a29da45c31f7cc3c4e, [440ms] received response from server 353498eabc2240a29da45c31f7cc3c4e: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [440ms] delaying RPC due to: Network error: Failed to connect to peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969): Connection refused: /127.26.160.3:39969, [1992ms] refreshing cache from master, [1992ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.160.60:34099, [1998ms] Sub RPC GetTableLocations: received response from server master-127.26.160.60:34099: OK, deferred=Deferred@485422924(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduSession$$Lambda$236/0x00007ff584336f90@21347da0 -> passthrough -> wakeup thread Time-limited test -> (continuation of Deferred@91712394 after retry RPC@617279374) -> (continuation of Deferred@1037326997 after retry RPC@498123705) -> (continuation of Deferred@1484234040 after retry RPC@1136730244) -> (continuation of Deferred@389117843 after retry RPC@1564385521) -> (continuation of Deferred@1209458902 after retry RPC@1619519501), errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@495d0832 -> wakeup thread Time-limited test -> (continuation of Deferred@91712394 after retry RPC@617279374) -> (continuation of Deferred@1037326997 after retry RPC@498123705) -> (continuation of Deferred@1484234040 after retry RPC@1136730244) -> (continuation of Deferred@389117843 after retry RPC@1564385521) -> (continuation of Deferred@1209458902 after retry RPC@1619519501))) after lookup
14:05:09.711 [DEBUG - kudu-nio-1] (Connection.java:748) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] connecting to peer
14:05:09.712 [DEBUG - kudu-nio-5] (Connection.java:757) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] Successfully connected to peer
14:05:09.750 [DEBUG - kudu-nio-5] (Negotiator.java:919) Authenticated connection [id: 0x3b845ec3, L:/127.0.0.1:32860 - R:/127.26.160.1:40635] using TOKEN/null
14:05:09.808 [INFO - Time-limited test] (MiniKuduCluster.java:456) Starting tablet server 127.26.160.3:39969
14:05:09.808 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
  id {
    type: TSERVER
    index: 2
  }
}

14:05:09.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:09.810025 27264 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:05:09.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu
14:05:09.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:05:09.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data
14:05:09.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/logs
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data/info.pb
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.3:39969
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.3
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42189
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.3
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:05:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
14:05:10.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:10.226917 29734 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
14:05:10.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:10.227530 29734 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
14:05:10.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:10.228474 29734 flags.cc:432] Enabled unsafe flag: --never_fsync=true
14:05:10.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:10.270745 29734 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
14:05:10.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:10.271775 29734 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.160.3
14:05:10.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:10.317480 29734 tablet_server_runner.cc:78] Tablet server non-default flags:
14:05:10.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
14:05:10.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.26.160.20:34421
14:05:10.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
14:05:10.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
14:05:10.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data
14:05:10.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:05:10.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
14:05:10.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
14:05:10.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.26.160.3:39969
14:05:10.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
14:05:10.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data/info.pb
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.26.160.3
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=42189
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.26.160.62:43077,127.26.160.61:45315,127.26.160.60:34099
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.26.160.3
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/logs
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.19.0-SNAPSHOT
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision 66827b15b158c37363c37baa160f29a0ca1f1e63
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by None at 18 May 2026 13:43:22 UTC on e7f111948823
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 11959
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
14:05:10.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:10.319343 29734 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
14:05:10.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:10.321285 29734 file_cache.cc:492] Constructed file cache file cache with capacity 419430
14:05:10.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:10.337941 29740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:11.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:10.340215 29741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:11.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:11.745615 29742 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
14:05:11.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:11.747385 29743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
14:05:11.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.747449 29734 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
14:05:11.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.752399 29734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
14:05:11.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.755470 29734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
14:05:11.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.756968 29734 hybrid_clock.cc:648] HybridClock initialized: now 1779113111756886 us; error 94 us; skew 500 ppm
14:05:11.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.758051 29734 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
14:05:11.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.766722 29734 webserver.cc:533] Webserver started at http://127.26.160.3:42189/ using document root <none> and password file <none>
14:05:11.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.767858 29734 fs_manager.cc:362] Metadata directory not provided
14:05:11.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.768123 29734 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
14:05:11.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.778597 29734 fs_manager.cc:714] Time spent opening directory manager: real 0.007s	user 0.002s	sys 0.004s
14:05:11.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.784366 29751 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
14:05:11.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.786145 29734 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.006s	sys 0.001s
14:05:11.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.786689 29734 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data,/tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:05:11.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "353498eabc2240a29da45c31f7cc3c4e"
14:05:11.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-05-18 14:04:20 on dist-test-slave-8gxc"
14:05:11.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.790037 29734 fs_report.cc:389] FS layout report
14:05:11.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
14:05:11.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:05:11.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/wal
14:05:11.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data/data
14:05:11.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
14:05:11.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
14:05:11.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
14:05:11.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
14:05:11.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
14:05:11.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
14:05:11.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
14:05:11.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
14:05:11.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
14:05:11.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
14:05:11.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
14:05:11.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260518 14:05:11.823391 29391 consensus_peers.cc:597] T 266c976e4879434f8b95ee412dfafcc5 P 9f9c16da37ad440eb0d7a7a590be07a0 -> Peer 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969): Couldn't send request to peer 353498eabc2240a29da45c31f7cc3c4e. Status: Network error: Client connection negotiation failed: client connection to 127.26.160.3:39969: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
14:05:11.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.860185 29734 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
14:05:11.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.861961 29734 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
14:05:11.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.862720 29734 kserver.cc:163] Server-wide thread pool size limit: 3276
14:05:11.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.865932 29734 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
14:05:11.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.873459 29758 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
14:05:11.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.883551 29734 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
14:05:11.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.884157 29734 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.013s	user 0.002s	sys 0.001s
14:05:11.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.884615 29734 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
14:05:11.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.891206 29734 ts_tablet_manager.cc:616] Registered 1 tablets
14:05:11.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.891476 29734 ts_tablet_manager.cc:595] Time spent register tablets: real 0.007s	user 0.007s	sys 0.000s
14:05:11.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.891922 29758 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Bootstrap starting.
14:05:11.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:11.992249 29758 log.cc:826] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Log is configured to *not* fsync() on all Append() calls
14:05:12.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.110363 29734 rpc_server.cc:307] RPC server started. Bound to: 127.26.160.3:39969
14:05:12.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.110507 29865 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.160.3:39969 every 8 connection(s)
14:05:12.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.113487 29734 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskSAAnLi/test-tmp/mini-kudu-cluster10651015935700312152/ts-2/data/info.pb
14:05:12.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.123260 27264 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu as pid 29734
14:05:12.130 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: 
14:05:12.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.161345 29758 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=40 overwritten=0 applied=40 ignored=0} inserts{seen=30 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
14:05:12.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.162935 29758 tablet_bootstrap.cc:492] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Bootstrap complete.
14:05:12.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.169036 29758 ts_tablet_manager.cc:1403] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Time spent bootstrapping tablet: real 0.278s	user 0.205s	sys 0.053s
14:05:12.195 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990, tablet=266c976e4879434f8b95ee412dfafcc5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:05:12.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.195168 29866 heartbeater.cc:344] Connected to a master server at 127.26.160.60:34099
14:05:12.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.195641 29866 heartbeater.cc:461] Registering TS with master...
14:05:12.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.197082 29866 heartbeater.cc:507] Master 127.26.160.60:34099 requested a full tablet report, sending...
14:05:12.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.197211 29758 raft_consensus.cc:359] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 11 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:05:12.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.200418 29758 raft_consensus.cc:740] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [term 11 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353498eabc2240a29da45c31f7cc3c4e, State: Initialized, Role: FOLLOWER
14:05:12.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.201386 29758 consensus_queue.cc:260] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 11.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9c16da37ad440eb0d7a7a590be07a0" member_type: VOTER last_known_addr { host: "127.26.160.1" port: 40635 } } peers { permanent_uuid: "353498eabc2240a29da45c31f7cc3c4e" member_type: VOTER last_known_addr { host: "127.26.160.3" port: 39969 } } peers { permanent_uuid: "d1c13f941b874a778af09e1ef1b4a7bb" member_type: VOTER last_known_addr { host: "127.26.160.2" port: 37023 } }
14:05:12.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.218335 29868 heartbeater.cc:344] Connected to a master server at 127.26.160.62:43077
14:05:12.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.218855 29868 heartbeater.cc:461] Registering TS with master...
14:05:12.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.219945 29868 heartbeater.cc:507] Master 127.26.160.62:43077 requested a full tablet report, sending...
14:05:12.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.221575 29867 heartbeater.cc:344] Connected to a master server at 127.26.160.61:45315
14:05:12.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.222038 29867 heartbeater.cc:461] Registering TS with master...
14:05:12.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.223274 29867 heartbeater.cc:507] Master 127.26.160.61:45315 requested a full tablet report, sending...
14:05:12.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.225258 27309 ts_manager.cc:194] Re-registered known tserver with Master: 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:05:12.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.228343 27384 ts_manager.cc:194] Re-registered known tserver with Master: 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:05:12.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.230023 29758 ts_tablet_manager.cc:1434] T 266c976e4879434f8b95ee412dfafcc5 P 353498eabc2240a29da45c31f7cc3c4e: Time spent starting tablet: real 0.058s	user 0.036s	sys 0.012s
14:05:12.232 [DEBUG - kudu-nio-5] (AsyncKuduScanner.java:1370) AsyncKuduScanner$Response(scannerId = "", data = RowwiseRowResultIterator for 33 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.TestMultipleLeaderFailover-1779113060990, tablet=266c976e4879434f8b95ee412dfafcc5, scannerId=null, scanRequestTimeout=30000, startPrimaryKey=<start>, endPrimaryKey=<end>, lastPrimaryKey=<last>)
14:05:12.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.236122 27466 ts_manager.cc:194] Re-registered known tserver with Master: 353498eabc2240a29da45c31f7cc3c4e (127.26.160.3:39969)
14:05:12.238 [DEBUG - main] (Connection.java:575) [peer master-127.26.160.60:34099(127.26.160.60:34099)] disconnecting while in state READY
14:05:12.239 [DEBUG - main] (Connection.java:575) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] disconnecting while in state READY
14:05:12.240 [DEBUG - main] (Connection.java:575) [peer master-127.26.160.62:43077(127.26.160.62:43077)] disconnecting while in state READY
14:05:12.240 [DEBUG - main] (Connection.java:575) [peer d1c13f941b874a778af09e1ef1b4a7bb(127.26.160.2:37023)] disconnecting while in state TERMINATED
14:05:12.240 [DEBUG - main] (Connection.java:575) [peer master-127.26.160.61:45315(127.26.160.61:45315)] disconnecting while in state READY
14:05:12.241 [DEBUG - main] (Connection.java:575) [peer 353498eabc2240a29da45c31f7cc3c4e(127.26.160.3:39969)] disconnecting while in state TERMINATED
14:05:12.246 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.26.160.60:34099(127.26.160.60:34099)] handling channelInactive
14:05:12.247 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.26.160.60:34099(127.26.160.60:34099)] cleaning up while in state READY due to: connection closed
14:05:12.247 [DEBUG - kudu-nio-5] (Connection.java:239) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] handling channelInactive
14:05:12.248 [DEBUG - kudu-nio-5] (Connection.java:709) [peer 9f9c16da37ad440eb0d7a7a590be07a0(127.26.160.1:40635)] cleaning up while in state READY due to: connection closed
14:05:12.249 [DEBUG - kudu-nio-2] (Connection.java:239) [peer master-127.26.160.61:45315(127.26.160.61:45315)] handling channelInactive
14:05:12.249 [DEBUG - kudu-nio-2] (Connection.java:709) [peer master-127.26.160.61:45315(127.26.160.61:45315)] cleaning up while in state READY due to: connection closed
14:05:12.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.248832 27466 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.26.160.3:40451
14:05:12.249 [DEBUG - kudu-nio-0] (AsyncKuduClient.java:2704) Releasing all remaining resources
14:05:12.251 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.26.160.62:43077(127.26.160.62:43077)] handling channelInactive
14:05:12.251 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.26.160.62:43077(127.26.160.62:43077)] cleaning up while in state READY due to: connection closed
14:05:12.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.253715 27264 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu with pid 29364
14:05:12.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.258694 29866 heartbeater.cc:499] Master 127.26.160.60:34099 was elected leader, sending a full tablet report...
14:05:12.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.292546 27264 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu with pid 29549
14:05:12.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.326287 27264 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu with pid 29734
14:05:12.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.360503 27264 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu with pid 27277
14:05:12.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.409052 27264 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu with pid 27347
14:05:12.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260518 14:05:12.440797 27264 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskSAAnLi/build/tsan/bin/kudu with pid 27422
14:05:13.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-05-18T14:05:13Z chronyd exiting

Time: 71.696

OK (1 test)

