[==========] Running 8 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 2 tests from PrometheusSpnegoITest
[ RUN      ] PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly
Loading random data
Initializing database '/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 08:01:20 dist-test-slave-1nl6 krb5kdc[5736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 08:01:20 dist-test-slave-1nl6 krb5kdc[5736](info): set up 2 sockets
Aug 12 08:01:20 dist-test-slave-1nl6 krb5kdc[5736](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 08:01:22.339203  5728 mini_kdc.cc:121] Time spent starting KDC: real 2.270s	user 0.000s	sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/test-user.keytab.
Aug 12 08:01:22 dist-test-slave-1nl6 krb5kdc[5736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM: 
2026-08-12T08:01:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T08:01:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:01:22.482602  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.62:37539
--webserver_interface=127.5.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:37539,127.5.152.61:35377,127.5.152.60:42773
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 08:01:22.589107  5757 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:01:22.589357  5757 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:01:22.589423  5757 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:01:22.592926  5757 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:01:22.592985  5757 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:01:22.593007  5757 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:01:22.593026  5757 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:01:22.597563  5757 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:37539,127.5.152.61:35377,127.5.152.60:42773
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.62:37539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.62
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:01:22.598502  5757 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:01:22.599388  5757 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:01:22.606678  5770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:22.606725  5767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:22.606684  5766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:22.607724  5757 server_base.cc:1061] running on GCE node
I20260812 08:01:22.608218  5757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:01:22.609092  5757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:01:22.610245  5757 hybrid_clock.cc:648] HybridClock initialized: now 1786521682610219 us; error 42 us; skew 500 ppm
Aug 12 08:01:22 dist-test-slave-1nl6 krb5kdc[5736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521682, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:01:22.612912  5757 init.cc:377] Logged in from keytab as kudu/127.5.152.62@KRBTEST.COM (short username kudu)
I20260812 08:01:22.621165  5757 webserver.cc:533] Webserver started at https://127.5.152.62:41657/ using document root <none> and password file <none>
I20260812 08:01:22.621789  5757 fs_manager.cc:362] Metadata directory not provided
I20260812 08:01:22.621871  5757 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:01:22.622160  5757 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:01:22.623834  5757 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/data/instance:
uuid: "bd657ef1d06648a7920df9c29832074c"
format_stamp: "Formatted at 2026-08-12 08:01:22 on dist-test-slave-1nl6"
I20260812 08:01:22.624293  5757 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/wal/instance:
uuid: "bd657ef1d06648a7920df9c29832074c"
format_stamp: "Formatted at 2026-08-12 08:01:22 on dist-test-slave-1nl6"
I20260812 08:01:22.627614  5757 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.001s
I20260812 08:01:22.629673  5784 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:22.630635  5757 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:01:22.630769  5757 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/wal
uuid: "bd657ef1d06648a7920df9c29832074c"
format_stamp: "Formatted at 2026-08-12 08:01:22 on dist-test-slave-1nl6"
I20260812 08:01:22.630869  5757 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:01:22.645759  5757 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:01:22.652494  5757 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:01:22.652704  5757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:01:22.661427  5871 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.62:37539 every 8 connection(s)
I20260812 08:01:22.661428  5757 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.62:37539
I20260812 08:01:22.662621  5757 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
I20260812 08:01:22.665901  5875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:01:22.668483  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 5757
I20260812 08:01:22.668637  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/wal/instance
I20260812 08:01:22.669996  5875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.152.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.61@KRBTEST.COM" created.
Aug 12 08:01:22 dist-test-slave-1nl6 krb5kdc[5736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521682, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
I20260812 08:01:22.688069  5875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:01:22.688874  5788 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.152.61:35377: connect: Connection refused (error 111)
W20260812 08:01:22.689935  5875 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:35377: Network error: Client connection negotiation failed: client connection to 127.5.152.61:35377: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:01:22.715734  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.61:35377
--webserver_interface=127.5.152.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:37539,127.5.152.61:35377,127.5.152.60:42773
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 08:01:22.739153  5875 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } attempt: 1
W20260812 08:01:22.740644  5875 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:35377: Network error: Client connection negotiation failed: client connection to 127.5.152.61:35377: connect: Connection refused (error 111)
I20260812 08:01:22.808874  5875 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } attempt: 2
W20260812 08:01:22.810361  5875 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:35377: Network error: Client connection negotiation failed: client connection to 127.5.152.61:35377: connect: Connection refused (error 111)
W20260812 08:01:22.816478  5888 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:01:22.816671  5888 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:01:22.816730  5888 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:01:22.819937  5888 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:01:22.819998  5888 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:01:22.820017  5888 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:01:22.820034  5888 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:01:22.824110  5888 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:37539,127.5.152.61:35377,127.5.152.60:42773
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.61:35377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.61
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:01:22.824937  5888 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:01:22.825676  5888 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 08:01:22.830865  5888 server_base.cc:1061] running on GCE node
W20260812 08:01:22.830783  5897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:22.830798  5900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:22.830783  5898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:22.831550  5888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:01:22.832350  5888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:01:22.833506  5888 hybrid_clock.cc:648] HybridClock initialized: now 1786521682833489 us; error 28 us; skew 500 ppm
Aug 12 08:01:22 dist-test-slave-1nl6 krb5kdc[5736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521682, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:01:22.836292  5888 init.cc:377] Logged in from keytab as kudu/127.5.152.61@KRBTEST.COM (short username kudu)
I20260812 08:01:22.844336  5888 webserver.cc:533] Webserver started at https://127.5.152.61:44053/ using document root <none> and password file <none>
I20260812 08:01:22.844894  5888 fs_manager.cc:362] Metadata directory not provided
I20260812 08:01:22.844945  5888 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:01:22.845156  5888 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:01:22.846694  5888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/data/instance:
uuid: "2164207bb74144d6ae68bc768217b7a7"
format_stamp: "Formatted at 2026-08-12 08:01:22 on dist-test-slave-1nl6"
I20260812 08:01:22.847218  5888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/wal/instance:
uuid: "2164207bb74144d6ae68bc768217b7a7"
format_stamp: "Formatted at 2026-08-12 08:01:22 on dist-test-slave-1nl6"
I20260812 08:01:22.850242  5888 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.001s	sys 0.001s
I20260812 08:01:22.852284  5916 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:22.853199  5888 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.001s
I20260812 08:01:22.853336  5888 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/wal
uuid: "2164207bb74144d6ae68bc768217b7a7"
format_stamp: "Formatted at 2026-08-12 08:01:22 on dist-test-slave-1nl6"
I20260812 08:01:22.853435  5888 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:01:22.869019  5888 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:01:22.871526  5888 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:01:22.871729  5888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:01:22.879098  5888 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.61:35377
I20260812 08:01:22.879106  5996 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.61:35377 every 8 connection(s)
I20260812 08:01:22.880020  5888 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
I20260812 08:01:22.881151  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 5888
I20260812 08:01:22.881245  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/wal/instance
I20260812 08:01:22.882668  5999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:01:22.886308  5999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.152.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.60@KRBTEST.COM" created.
Aug 12 08:01:22 dist-test-slave-1nl6 krb5kdc[5736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521682, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
I20260812 08:01:22.900556  5999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:01:22.901547  5875 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } attempt: 3
Aug 12 08:01:22 dist-test-slave-1nl6 krb5kdc[5736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521682, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
Aug 12 08:01:22 dist-test-slave-1nl6 krb5kdc[5736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521682, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:01:22.915642  5999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:01:22.915745  5875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:01:22.916584  5921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.152.60:42773: connect: Connection refused (error 111)
W20260812 08:01:22.917279  5875 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:42773: Network error: Client connection negotiation failed: client connection to 127.5.152.60:42773: connect: Connection refused (error 111)
W20260812 08:01:22.917599  5999 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:42773: Network error: Client connection negotiation failed: client connection to 127.5.152.60:42773: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.5.152.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:01:22.937383  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.60:42773
--webserver_interface=127.5.152.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:37539,127.5.152.61:35377,127.5.152.60:42773
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 08:01:22.948446  5875 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } attempt: 1
W20260812 08:01:22.949776  5875 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:42773: Network error: Client connection negotiation failed: client connection to 127.5.152.60:42773: connect: Connection refused (error 111)
I20260812 08:01:22.966807  5999 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } attempt: 1
W20260812 08:01:22.968152  5999 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:42773: Network error: Client connection negotiation failed: client connection to 127.5.152.60:42773: connect: Connection refused (error 111)
I20260812 08:01:23.025054  5875 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } attempt: 2
W20260812 08:01:23.026510  5875 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:42773: Network error: Client connection negotiation failed: client connection to 127.5.152.60:42773: connect: Connection refused (error 111)
I20260812 08:01:23.036319  5999 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } attempt: 2
W20260812 08:01:23.037616  5999 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:42773: Network error: Client connection negotiation failed: client connection to 127.5.152.60:42773: connect: Connection refused (error 111)
W20260812 08:01:23.041951  6017 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:01:23.042186  6017 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:01:23.042306  6017 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:01:23.045542  6017 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:01:23.045631  6017 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:01:23.045674  6017 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:01:23.045707  6017 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:01:23.049803  6017 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:37539,127.5.152.61:35377,127.5.152.60:42773
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.60:42773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.60
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:01:23.050679  6017 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:01:23.051478  6017 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:01:23.057166  6023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:23.057255  6017 server_base.cc:1061] running on GCE node
W20260812 08:01:23.057183  6027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:23.057214  6024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:23.058233  6017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:01:23.059424  6017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:01:23.060660  6017 hybrid_clock.cc:648] HybridClock initialized: now 1786521683060622 us; error 52 us; skew 500 ppm
Aug 12 08:01:23 dist-test-slave-1nl6 krb5kdc[5736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521683, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:01:23.063791  6017 init.cc:377] Logged in from keytab as kudu/127.5.152.60@KRBTEST.COM (short username kudu)
I20260812 08:01:23.073278  6017 webserver.cc:533] Webserver started at https://127.5.152.60:38383/ using document root <none> and password file <none>
I20260812 08:01:23.073894  6017 fs_manager.cc:362] Metadata directory not provided
I20260812 08:01:23.073974  6017 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:01:23.074200  6017 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:01:23.075879  6017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/data/instance:
uuid: "2a1de1b354e540098cc4a61b297a677b"
format_stamp: "Formatted at 2026-08-12 08:01:23 on dist-test-slave-1nl6"
I20260812 08:01:23.076365  6017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/wal/instance:
uuid: "2a1de1b354e540098cc4a61b297a677b"
format_stamp: "Formatted at 2026-08-12 08:01:23 on dist-test-slave-1nl6"
I20260812 08:01:23.079725  6017 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 08:01:23.081725  6037 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:23.082659  6017 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:01:23.082798  6017 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/wal
uuid: "2a1de1b354e540098cc4a61b297a677b"
format_stamp: "Formatted at 2026-08-12 08:01:23 on dist-test-slave-1nl6"
I20260812 08:01:23.082904  6017 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:01:23.111634  6017 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:01:23.114485  6017 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:01:23.114682  6017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:01:23.122125  6017 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.60:42773
I20260812 08:01:23.122152  6120 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.60:42773 every 8 connection(s)
I20260812 08:01:23.123186  6017 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
I20260812 08:01:23.125656  5875 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } attempt: 3
I20260812 08:01:23.125733  6121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:01:23.128791  5999 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } attempt: 3
I20260812 08:01:23.130285  6121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:01:23.132807  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 6017
I20260812 08:01:23.132927  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/wal/instance
Aug 12 08:01:23 dist-test-slave-1nl6 krb5kdc[5736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521682, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
Aug 12 08:01:23 dist-test-slave-1nl6 krb5kdc[5736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521682, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
Aug 12 08:01:23 dist-test-slave-1nl6 krb5kdc[5736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521683, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
I20260812 08:01:23.144374  5875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c: Bootstrap starting.
I20260812 08:01:23.144728  5999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7: Bootstrap starting.
Aug 12 08:01:23 dist-test-slave-1nl6 krb5kdc[5736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
I20260812 08:01:23.146868  5875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c: Neither blocks nor log segments found. Creating new log.
I20260812 08:01:23.146888  5999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7: Neither blocks nor log segments found. Creating new log.
I20260812 08:01:23.147643  5875 log.cc:826] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c: Log is configured to *not* fsync() on all Append() calls
I20260812 08:01:23.147814  5999 log.cc:826] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7: Log is configured to *not* fsync() on all Append() calls
I20260812 08:01:23.149075  6121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:01:23.149446  5875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c: No bootstrap required, opened a new log
I20260812 08:01:23.152729  5875 raft_consensus.cc:359] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd657ef1d06648a7920df9c29832074c" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } } peers { permanent_uuid: "2164207bb74144d6ae68bc768217b7a7" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } } peers { permanent_uuid: "2a1de1b354e540098cc4a61b297a677b" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } }
I20260812 08:01:23.152904  5875 raft_consensus.cc:385] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:01:23.152956  5875 raft_consensus.cc:740] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd657ef1d06648a7920df9c29832074c, State: Initialized, Role: FOLLOWER
I20260812 08:01:23.153674  5875 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd657ef1d06648a7920df9c29832074c" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } } peers { permanent_uuid: "2164207bb74144d6ae68bc768217b7a7" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } } peers { permanent_uuid: "2a1de1b354e540098cc4a61b297a677b" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } }
Aug 12 08:01:23 dist-test-slave-1nl6 krb5kdc[5736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521683, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:01:23.154258  6140 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd657ef1d06648a7920df9c29832074c" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } } peers { permanent_uuid: "2164207bb74144d6ae68bc768217b7a7" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } } peers { permanent_uuid: "2a1de1b354e540098cc4a61b297a677b" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } } }
I20260812 08:01:23.154381  6140 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:23.155211  5875 sys_catalog.cc:565] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 08:01:23 dist-test-slave-1nl6 krb5kdc[5736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:01:23.158716  6121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:01:23.159194  5999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7: No bootstrap required, opened a new log
I20260812 08:01:23.162585  5999 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd657ef1d06648a7920df9c29832074c" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } } peers { permanent_uuid: "2164207bb74144d6ae68bc768217b7a7" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } } peers { permanent_uuid: "2a1de1b354e540098cc4a61b297a677b" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } }
I20260812 08:01:23.162740  5999 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:01:23.162787  5999 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2164207bb74144d6ae68bc768217b7a7, State: Initialized, Role: FOLLOWER
Aug 12 08:01:23 dist-test-slave-1nl6 krb5kdc[5736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521683, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:01:23.161130  5757 master_runner.cc:187] Error getting master registration for 127.5.152.61:35377: OK, instance_id { permanent_uuid: "2164207bb74144d6ae68bc768217b7a7" instance_seqno: 1786521682877001 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 08:01:23.163228  5999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd657ef1d06648a7920df9c29832074c" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } } peers { permanent_uuid: "2164207bb74144d6ae68bc768217b7a7" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } } peers { permanent_uuid: "2a1de1b354e540098cc4a61b297a677b" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } }
I20260812 08:01:23.164127  5757 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 08:01:23.165076  5999 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 08:01:23 dist-test-slave-1nl6 krb5kdc[5736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:01:23.168188  6163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd657ef1d06648a7920df9c29832074c" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } } peers { permanent_uuid: "2164207bb74144d6ae68bc768217b7a7" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } } peers { permanent_uuid: "2a1de1b354e540098cc4a61b297a677b" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } } }
I20260812 08:01:23.168304  6163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:23.169992  6121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b: Bootstrap starting.
W20260812 08:01:23.172214  6162 catalog_manager.cc:1594] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:01:23.172286  6162 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:01:23.172479  6121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b: Neither blocks nor log segments found. Creating new log.
I20260812 08:01:23.173333  6121 log.cc:826] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b: Log is configured to *not* fsync() on all Append() calls
W20260812 08:01:23.173722  6180 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:01:23.173779  6180 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:01:23.175056  6121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b: No bootstrap required, opened a new log
I20260812 08:01:23.172768  5888 master_runner.cc:187] Error getting master registration for 127.5.152.60:42773: OK, instance_id { permanent_uuid: "2a1de1b354e540098cc4a61b297a677b" instance_seqno: 1786521683120181 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 08:01:23.175400  5888 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 08:01:23.178117  6121 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd657ef1d06648a7920df9c29832074c" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } } peers { permanent_uuid: "2164207bb74144d6ae68bc768217b7a7" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } } peers { permanent_uuid: "2a1de1b354e540098cc4a61b297a677b" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } }
I20260812 08:01:23.178300  6121 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:01:23.178369  6121 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1de1b354e540098cc4a61b297a677b, State: Initialized, Role: FOLLOWER
I20260812 08:01:23.178787  6121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd657ef1d06648a7920df9c29832074c" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } } peers { permanent_uuid: "2164207bb74144d6ae68bc768217b7a7" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } } peers { permanent_uuid: "2a1de1b354e540098cc4a61b297a677b" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } }
I20260812 08:01:23.179201  6187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd657ef1d06648a7920df9c29832074c" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } } peers { permanent_uuid: "2164207bb74144d6ae68bc768217b7a7" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } } peers { permanent_uuid: "2a1de1b354e540098cc4a61b297a677b" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } } }
I20260812 08:01:23.179351  6187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:23.179987  6121 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:01:23.188393  6017 master_runner.cc:264] No leader master found from master 2a1de1b354e540098cc4a61b297a677b
I20260812 08:01:23.188481  6017 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 08:01:23.197603  6202 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:01:23.197685  6202 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:01:23.361176  6163 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:01:23.361393  6163 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd657ef1d06648a7920df9c29832074c" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } } peers { permanent_uuid: "2164207bb74144d6ae68bc768217b7a7" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } } peers { permanent_uuid: "2a1de1b354e540098cc4a61b297a677b" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } }
I20260812 08:01:23.362560  6163 leader_election.cc:290] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd657ef1d06648a7920df9c29832074c (127.5.152.62:37539), 2a1de1b354e540098cc4a61b297a677b (127.5.152.60:42773)
I20260812 08:01:23.363005  5831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2164207bb74144d6ae68bc768217b7a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd657ef1d06648a7920df9c29832074c" is_pre_election: true
I20260812 08:01:23.363217  6078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2164207bb74144d6ae68bc768217b7a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1de1b354e540098cc4a61b297a677b" is_pre_election: true
I20260812 08:01:23.363274  5831 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2164207bb74144d6ae68bc768217b7a7 in term 0.
I20260812 08:01:23.363459  6078 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2164207bb74144d6ae68bc768217b7a7 in term 0.
I20260812 08:01:23.363860  5919 leader_election.cc:304] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2164207bb74144d6ae68bc768217b7a7, bd657ef1d06648a7920df9c29832074c; no voters: 
I20260812 08:01:23.364099  6163 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:01:23.364162  6163 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:01:23.364214  6163 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:01:23.365103  6163 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd657ef1d06648a7920df9c29832074c" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } } peers { permanent_uuid: "2164207bb74144d6ae68bc768217b7a7" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } } peers { permanent_uuid: "2a1de1b354e540098cc4a61b297a677b" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } }
I20260812 08:01:23.365429  6163 leader_election.cc:290] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [CANDIDATE]: Term 1 election: Requested vote from peers bd657ef1d06648a7920df9c29832074c (127.5.152.62:37539), 2a1de1b354e540098cc4a61b297a677b (127.5.152.60:42773)
I20260812 08:01:23.365718  5831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2164207bb74144d6ae68bc768217b7a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd657ef1d06648a7920df9c29832074c"
I20260812 08:01:23.365836  5831 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:01:23.365900  6078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2164207bb74144d6ae68bc768217b7a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1de1b354e540098cc4a61b297a677b"
I20260812 08:01:23.365998  6078 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:01:23.366966  6078 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2164207bb74144d6ae68bc768217b7a7 in term 1.
I20260812 08:01:23.367089  5831 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2164207bb74144d6ae68bc768217b7a7 in term 1.
W20260812 08:01:23.367216  6047 tablet.cc:2406] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:01:23.367525  5919 leader_election.cc:304] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2164207bb74144d6ae68bc768217b7a7, bd657ef1d06648a7920df9c29832074c; no voters: 
I20260812 08:01:23.367794  6163 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:01:23.368005  6163 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [term 1 LEADER]: Becoming Leader. State: Replica: 2164207bb74144d6ae68bc768217b7a7, State: Running, Role: LEADER
I20260812 08:01:23.368306  6163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd657ef1d06648a7920df9c29832074c" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } } peers { permanent_uuid: "2164207bb74144d6ae68bc768217b7a7" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } } peers { permanent_uuid: "2a1de1b354e540098cc4a61b297a677b" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } }
I20260812 08:01:23.370445  6205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2164207bb74144d6ae68bc768217b7a7. Latest consensus state: current_term: 1 leader_uuid: "2164207bb74144d6ae68bc768217b7a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd657ef1d06648a7920df9c29832074c" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } } peers { permanent_uuid: "2164207bb74144d6ae68bc768217b7a7" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } } peers { permanent_uuid: "2a1de1b354e540098cc4a61b297a677b" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } } }
I20260812 08:01:23.370597  6205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [sys.catalog]: This master's current role is: LEADER
I20260812 08:01:23.372081  6208 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
W20260812 08:01:23.372991  5927 tablet.cc:2406] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:01:23.373286  6208 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:01:23.377326  6078 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b [term 1 FOLLOWER]: Refusing update from remote peer 2164207bb74144d6ae68bc768217b7a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:01:23.377367  5831 raft_consensus.cc:1275] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c [term 1 FOLLOWER]: Refusing update from remote peer 2164207bb74144d6ae68bc768217b7a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:01:23.377907  6205 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1de1b354e540098cc4a61b297a677b" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:01:23.378333  6205 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd657ef1d06648a7920df9c29832074c" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:01:23.382269  6187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2164207bb74144d6ae68bc768217b7a7. Latest consensus state: current_term: 1 leader_uuid: "2164207bb74144d6ae68bc768217b7a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd657ef1d06648a7920df9c29832074c" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } } peers { permanent_uuid: "2164207bb74144d6ae68bc768217b7a7" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } } peers { permanent_uuid: "2a1de1b354e540098cc4a61b297a677b" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } } }
I20260812 08:01:23.382360  6140 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2164207bb74144d6ae68bc768217b7a7. Latest consensus state: current_term: 1 leader_uuid: "2164207bb74144d6ae68bc768217b7a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd657ef1d06648a7920df9c29832074c" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } } peers { permanent_uuid: "2164207bb74144d6ae68bc768217b7a7" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } } peers { permanent_uuid: "2a1de1b354e540098cc4a61b297a677b" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } } }
I20260812 08:01:23.382506  6140 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:23.382507  6187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:23.387871  6163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2164207bb74144d6ae68bc768217b7a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd657ef1d06648a7920df9c29832074c" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } } peers { permanent_uuid: "2164207bb74144d6ae68bc768217b7a7" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } } peers { permanent_uuid: "2a1de1b354e540098cc4a61b297a677b" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } } }
I20260812 08:01:23.387995  6163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [sys.catalog]: This master's current role is: LEADER
I20260812 08:01:23.388756  6140 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2164207bb74144d6ae68bc768217b7a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd657ef1d06648a7920df9c29832074c" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } } peers { permanent_uuid: "2164207bb74144d6ae68bc768217b7a7" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } } peers { permanent_uuid: "2a1de1b354e540098cc4a61b297a677b" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } } }
I20260812 08:01:23.388881  6140 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:23.389535  6205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2164207bb74144d6ae68bc768217b7a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd657ef1d06648a7920df9c29832074c" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } } peers { permanent_uuid: "2164207bb74144d6ae68bc768217b7a7" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } } peers { permanent_uuid: "2a1de1b354e540098cc4a61b297a677b" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } } }
I20260812 08:01:23.389638  6205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7 [sys.catalog]: This master's current role is: LEADER
I20260812 08:01:23.391072  6187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2164207bb74144d6ae68bc768217b7a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd657ef1d06648a7920df9c29832074c" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 37539 } } peers { permanent_uuid: "2164207bb74144d6ae68bc768217b7a7" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 35377 } } peers { permanent_uuid: "2a1de1b354e540098cc4a61b297a677b" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 42773 } } }
I20260812 08:01:23.391188  6187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:23.391321  6210 mvcc.cc:204] Tried to move back new op lower bound from 7317592815107653632 to 7317592815079858176. Current Snapshot: MvccSnapshot[applied={T|T < 7317592815107653632}]
I20260812 08:01:23.391579  6212 mvcc.cc:204] Tried to move back new op lower bound from 7317592815107653632 to 7317592815079858176. Current Snapshot: MvccSnapshot[applied={T|T < 7317592815107653632}]
I20260812 08:01:23.393592  6208 catalog_manager.cc:1383] Generated new cluster ID: a3d626528f1a4356a8ca2c7e627690ec
I20260812 08:01:23.393656  6208 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:01:23.419836  6208 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:01:23.420655  6208 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:01:23.430994  6208 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 2164207bb74144d6ae68bc768217b7a7: Generated new TSK 0
I20260812 08:01:23.431509  6208 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:01:23.489528  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.152.1:0
--local_ip_for_outbound_sockets=127.5.152.1
--webserver_interface=127.5.152.1
--webserver_port=0
--tserver_master_addrs=127.5.152.62:37539,127.5.152.61:35377,127.5.152.60:42773
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 08:01:23.590298  6237 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:01:23.590505  6237 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:01:23.590610  6237 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:01:23.593874  6237 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:01:23.594050  6237 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.152.1
I20260812 08:01:23.598120  6237 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.1
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_require_spnego=true
--tserver_master_addrs=127.5.152.62:37539,127.5.152.61:35377,127.5.152.60:42773
--never_fsync=true
--heap_profile_path=/tmp/kudu.6237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.152.1

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:01:23.599084  6237 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:01:23.599848  6237 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:01:23.605863  6246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:23.605873  6251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:23.605942  6237 server_base.cc:1061] running on GCE node
W20260812 08:01:23.605873  6247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:23.606366  6237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:01:23.606789  6237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:01:23.607935  6237 hybrid_clock.cc:648] HybridClock initialized: now 1786521683607923 us; error 30 us; skew 500 ppm
Aug 12 08:01:23 dist-test-slave-1nl6 krb5kdc[5736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521683, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:01:23.610558  6237 init.cc:377] Logged in from keytab as kudu/127.5.152.1@KRBTEST.COM (short username kudu)
I20260812 08:01:23.617870  6237 webserver.cc:533] Webserver started at https://127.5.152.1:39093/ using document root <none> and password file <none>
I20260812 08:01:23.618547  6237 fs_manager.cc:362] Metadata directory not provided
I20260812 08:01:23.618600  6237 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:01:23.618767  6237 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:01:23.620442  6237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/data/instance:
uuid: "c67b31af0244427990b0f734a7e08660"
format_stamp: "Formatted at 2026-08-12 08:01:23 on dist-test-slave-1nl6"
I20260812 08:01:23.620873  6237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/wal/instance:
uuid: "c67b31af0244427990b0f734a7e08660"
format_stamp: "Formatted at 2026-08-12 08:01:23 on dist-test-slave-1nl6"
I20260812 08:01:23.624166  6237 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 08:01:23.626180  6261 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:23.627170  6237 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260812 08:01:23.627285  6237 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/wal
uuid: "c67b31af0244427990b0f734a7e08660"
format_stamp: "Formatted at 2026-08-12 08:01:23 on dist-test-slave-1nl6"
I20260812 08:01:23.627386  6237 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:01:23.641160  6237 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:01:23.643846  6237 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:01:23.644052  6237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:01:23.644618  6237 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:01:23.645511  6237 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:01:23.645561  6237 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:23.645599  6237 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:01:23.645646  6237 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:23.657092  6237 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.1:37125
I20260812 08:01:23.657196  6436 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.1:37125 every 8 connection(s)
I20260812 08:01:23.658835  6237 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
I20260812 08:01:23.665083  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 6237
I20260812 08:01:23.665206  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/wal/instance
Aug 12 08:01:23 dist-test-slave-1nl6 krb5kdc[5736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521683, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
Aug 12 08:01:23 dist-test-slave-1nl6 krb5kdc[5736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521683, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
Aug 12 08:01:23 dist-test-slave-1nl6 krb5kdc[5736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521683, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:01:23.677210  6439 heartbeater.cc:344] Connected to a master server at 127.5.152.60:42773
I20260812 08:01:23.677594  6439 heartbeater.cc:461] Registering TS with master...
I20260812 08:01:23.678088  6438 heartbeater.cc:344] Connected to a master server at 127.5.152.62:37539
I20260812 08:01:23.678202  6438 heartbeater.cc:461] Registering TS with master...
I20260812 08:01:23.678285  6441 heartbeater.cc:344] Connected to a master server at 127.5.152.61:35377
I20260812 08:01:23.678381  6441 heartbeater.cc:461] Registering TS with master...
I20260812 08:01:23.678496  6439 heartbeater.cc:507] Master 127.5.152.60:42773 requested a full tablet report, sending...
I20260812 08:01:23.678496  6438 heartbeater.cc:507] Master 127.5.152.62:37539 requested a full tablet report, sending...
I20260812 08:01:23.678560  6441 heartbeater.cc:507] Master 127.5.152.61:35377 requested a full tablet report, sending...
I20260812 08:01:23.679956  5943 ts_manager.cc:194] Registered new tserver with Master: c67b31af0244427990b0f734a7e08660 (127.5.152.1:37125)
I20260812 08:01:23.679955  6064 ts_manager.cc:194] Registered new tserver with Master: c67b31af0244427990b0f734a7e08660 (127.5.152.1:37125)
I20260812 08:01:23.679953  5816 ts_manager.cc:194] Registered new tserver with Master: c67b31af0244427990b0f734a7e08660 (127.5.152.1:37125)
I20260812 08:01:23.681963  5943 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.152.1@KRBTEST.COM'} at 127.5.152.1:38609
I20260812 08:01:23.685784  5728 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
time=2026-08-12T08:01:23.850Z level=INFO source=main.go:1678 msg="updated GOGC" old=100 new=75
time=2026-08-12T08:01:23.852Z level=INFO source=main.go:744 msg="Leaving GOMAXPROCS=4: CPU quota undefined" component=automaxprocs
time=2026-08-12T08:01:23.853Z level=INFO source=memlimit.go:198 msg="GOMEMLIMIT is updated" component=automemlimit package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=14142549196 previous=9223372036854775807
time=2026-08-12T08:01:23.853Z level=INFO source=main.go:851 msg="Starting Prometheus Server" mode=server version="(version=3.11.2, branch=HEAD, revision=f0f0fdd679dcd6df320b0558b20919f7cd44c407)"
time=2026-08-12T08:01:23.854Z level=INFO source=main.go:856 msg="operational information" build_context="(go=go1.26.2, platform=linux/amd64, user=root@d2684485c347, date=20260413-11:53:40, tags=netgo,builtinassets)" host_details="(Linux 5.4.0-1157-gcp #166~18.04.1-Ubuntu SMP Wed Jan 21 01:57:46 UTC 2026 x86_64 dist-test-slave-1nl6 (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)"
time=2026-08-12T08:01:24.134Z level=INFO source=web.go:710 msg="Start listening for connections" component=web address=127.0.0.1:0
time=2026-08-12T08:01:24.137Z level=INFO source=main.go:1410 msg="Starting TSDB ..."
time=2026-08-12T08:01:24.148Z level=INFO source=tls_config.go:354 msg="Listening on" component=web address=127.0.0.1:37433
time=2026-08-12T08:01:24.148Z level=INFO source=tls_config.go:357 msg="TLS is disabled." component=web http2=false address=127.0.0.1:37433
time=2026-08-12T08:01:24.154Z level=INFO source=head.go:698 msg="Replaying on-disk memory mappable chunks if any" component=tsdb
time=2026-08-12T08:01:24.154Z level=INFO source=head.go:784 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=107.353µs
time=2026-08-12T08:01:24.155Z level=INFO source=head.go:792 msg="Replaying WAL, this may take a while" component=tsdb
time=2026-08-12T08:01:24.158Z level=INFO source=head.go:865 msg="WAL segment loaded" component=tsdb segment=0 maxSegment=0 duration=3.140663ms
time=2026-08-12T08:01:24.158Z level=INFO source=head.go:902 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=153.072µs wal_replay_duration=3.300811ms wbl_replay_duration=54ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=107.353µs total_replay_duration=3.666194ms
time=2026-08-12T08:01:24.161Z level=INFO source=main.go:1431 msg="filesystem information" fs_type=OVERLAYFS_SUPER_MAGIC
time=2026-08-12T08:01:24.161Z level=INFO source=main.go:1434 msg="TSDB started"
time=2026-08-12T08:01:24.161Z level=INFO source=main.go:1632 msg="Loading configuration file" filename=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/prometheus.yml
time=2026-08-12T08:01:24.162Z level=INFO source=main.go:1048 msg="TSDB retention updated" duration=15d size=0B percentage=0
time=2026-08-12T08:01:24.167Z level=INFO source=main.go:1671 msg="Completed loading of configuration file" db_storage=136.055µs remote_storage=45.985µs web_handler=803ns query_engine=1.328µs scrape=4.790027ms scrape_sd=150.964µs notify=25.888µs notify_sd=1.169µs rules=33.622µs tracing=108.227µs filename=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786521680053065-5728-0/prometheus.yml totalDuration=5.992777ms
time=2026-08-12T08:01:24.167Z level=INFO source=main.go:1395 msg="Server is ready to receive web requests."
time=2026-08-12T08:01:24.167Z level=INFO source=manager.go:209 msg="Starting rule manager..." component="rule manager"
I20260812 08:01:24.173521  6162 catalog_manager.cc:1295] Loaded cluster ID: a3d626528f1a4356a8ca2c7e627690ec
I20260812 08:01:24.173592  6162 catalog_manager.cc:1588] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c: loading cluster ID for follower catalog manager: success
I20260812 08:01:24.175408  6162 catalog_manager.cc:1610] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c: acquiring CA information for follower catalog manager: success
I20260812 08:01:24.176472  6162 catalog_manager.cc:1638] T 00000000000000000000000000000000 P bd657ef1d06648a7920df9c29832074c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 08:01:24.198720  6202 catalog_manager.cc:1295] Loaded cluster ID: a3d626528f1a4356a8ca2c7e627690ec
I20260812 08:01:24.198784  6202 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b: loading cluster ID for follower catalog manager: success
I20260812 08:01:24.200306  6202 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b: acquiring CA information for follower catalog manager: success
I20260812 08:01:24.201253  6202 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 2a1de1b354e540098cc4a61b297a677b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 08:01:24.684057  6441 heartbeater.cc:499] Master 127.5.152.61:35377 was elected leader, sending a full tablet report...
time=2026-08-12T08:01:31.938Z level=WARN source=main.go:1191 msg="Received an OS signal, exiting gracefully..." signal=terminated
time=2026-08-12T08:01:31.938Z level=INFO source=main.go:1216 msg="Stopping scrape discovery manager..."
time=2026-08-12T08:01:31.938Z level=INFO source=main.go:1230 msg="Stopping notify discovery manager..."
time=2026-08-12T08:01:31.938Z level=INFO source=main.go:1245 msg="Stopping rule manager manager..."
time=2026-08-12T08:01:31.938Z level=INFO source=main.go:1212 msg="Scrape discovery manager stopped"
time=2026-08-12T08:01:31.938Z level=INFO source=manager.go:223 msg="Stopping rule manager..." component="rule manager"
time=2026-08-12T08:01:31.938Z level=INFO source=main.go:1226 msg="Notify discovery manager stopped"
time=2026-08-12T08:01:31.938Z level=INFO source=manager.go:239 msg="Rule manager stopped" component="rule manager"
time=2026-08-12T08:01:31.938Z level=INFO source=main.go:1241 msg="Rule manager stopped"
time=2026-08-12T08:01:31.938Z level=INFO source=main.go:1269 msg="Stopping scrape manager..."
time=2026-08-12T08:01:31.938Z level=INFO source=main.go:1284 msg="Stopping tracing manager..."
time=2026-08-12T08:01:31.938Z level=INFO source=main.go:1369 msg="Stopping reloaders..."
time=2026-08-12T08:01:31.938Z level=INFO source=main.go:1261 msg="Scrape manager stopped"
time=2026-08-12T08:01:31.938Z level=INFO source=main.go:1458 msg="Stopping storage..."
time=2026-08-12T08:01:31.939Z level=INFO source=main.go:1361 msg="Reloaders stopped"
time=2026-08-12T08:01:31.939Z level=INFO source=main.go:1280 msg="Tracing manager stopped"
time=2026-08-12T08:01:31.943Z level=INFO source=main.go:1536 msg="Stopping web handler..."
time=2026-08-12T08:01:31.943Z level=INFO source=main.go:1559 msg="Stopping notifier manager..."
time=2026-08-12T08:01:31.943Z level=INFO source=main.go:1454 msg="TSDB stopped"
time=2026-08-12T08:01:31.943Z level=INFO source=manager.go:321 msg="Stopping notification manager..." component=notifier
time=2026-08-12T08:01:31.943Z level=INFO source=main.go:1555 msg="Notifier manager stopped"
time=2026-08-12T08:01:31.944Z level=INFO source=main.go:1532 msg="Web handler stopped"
time=2026-08-12T08:01:31.944Z level=INFO source=main.go:1570 msg="See you next time!"
I20260812 08:01:31.958472  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 6237
I20260812 08:01:31.969328  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 5757
I20260812 08:01:31.980036  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 5888
I20260812 08:01:31.990664  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 6017
2026-08-12T08:01:31Z chronyd exiting
[       OK ] PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly (11954 ms)
[ RUN      ] PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly
Loading random data
Initializing database '/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 08:01:32 dist-test-slave-1nl6 krb5kdc[6490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 08:01:32 dist-test-slave-1nl6 krb5kdc[6490](info): set up 2 sockets
Aug 12 08:01:32 dist-test-slave-1nl6 krb5kdc[6490](info): commencing operation
krb5kdc: starting...
W20260812 08:01:34.199751  5728 mini_kdc.cc:121] Time spent starting KDC: real 2.183s	user 0.002s	sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/test-user.keytab.
Aug 12 08:01:34 dist-test-slave-1nl6 krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM: 
2026-08-12T08:01:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T08:01:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:01:34.337545  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.62:41057
--webserver_interface=127.5.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:37951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:41057,127.5.152.61:45893,127.5.152.60:43441
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 08:01:34.441895  6521 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:01:34.442191  6521 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:01:34.442315  6521 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:01:34.445853  6521 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:01:34.445966  6521 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:01:34.446018  6521 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:01:34.446059  6521 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:01:34.450614  6521 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:37951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:41057,127.5.152.61:45893,127.5.152.60:43441
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.62:41057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.62
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:01:34.451628  6521 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:01:34.452476  6521 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:01:34.457517  6528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:34.457568  6527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:34.457525  6530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:34.457844  6521 server_base.cc:1061] running on GCE node
I20260812 08:01:34.458294  6521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:01:34.459156  6521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:01:34.460386  6521 hybrid_clock.cc:648] HybridClock initialized: now 1786521694460379 us; error 38 us; skew 500 ppm
Aug 12 08:01:34 dist-test-slave-1nl6 krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521694, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:01:34.463255  6521 init.cc:377] Logged in from keytab as kudu/127.5.152.62@KRBTEST.COM (short username kudu)
I20260812 08:01:34.471199  6521 webserver.cc:533] Webserver started at https://127.5.152.62:33575/ using document root <none> and password file <none>
I20260812 08:01:34.471755  6521 fs_manager.cc:362] Metadata directory not provided
I20260812 08:01:34.471834  6521 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:01:34.472047  6521 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:01:34.473799  6521 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/data/instance:
uuid: "d369fbc38fff44b28df87f38cddb26fe"
format_stamp: "Formatted at 2026-08-12 08:01:34 on dist-test-slave-1nl6"
I20260812 08:01:34.474355  6521 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/wal/instance:
uuid: "d369fbc38fff44b28df87f38cddb26fe"
format_stamp: "Formatted at 2026-08-12 08:01:34 on dist-test-slave-1nl6"
I20260812 08:01:34.477622  6521 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.001s	sys 0.004s
I20260812 08:01:34.479703  6543 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:34.480619  6521 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:01:34.480763  6521 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/wal
uuid: "d369fbc38fff44b28df87f38cddb26fe"
format_stamp: "Formatted at 2026-08-12 08:01:34 on dist-test-slave-1nl6"
I20260812 08:01:34.480896  6521 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:01:34.492449  6521 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:01:34.495388  6521 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:01:34.495579  6521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:01:34.502751  6521 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.62:41057
I20260812 08:01:34.502801  6617 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.62:41057 every 8 connection(s)
I20260812 08:01:34.503767  6521 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
I20260812 08:01:34.506824  6618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:01:34.510264  6618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:01:34.513829  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 6521
I20260812 08:01:34.514019  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-0/wal/instance
Aug 12 08:01:34 dist-test-slave-1nl6 krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521694, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.152.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.61@KRBTEST.COM" created.
I20260812 08:01:34.530195  6618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:01:34.531090  6546 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.152.61:45893: connect: Connection refused (error 111)
W20260812 08:01:34.532016  6618 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:45893: Network error: Client connection negotiation failed: client connection to 127.5.152.61:45893: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:01:34.565963  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.61:45893
--webserver_interface=127.5.152.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:37951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:41057,127.5.152.61:45893,127.5.152.60:43441
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 08:01:34.581285  6618 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } attempt: 1
W20260812 08:01:34.582798  6618 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:45893: Network error: Client connection negotiation failed: client connection to 127.5.152.61:45893: connect: Connection refused (error 111)
I20260812 08:01:34.651002  6618 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } attempt: 2
W20260812 08:01:34.652485  6618 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:45893: Network error: Client connection negotiation failed: client connection to 127.5.152.61:45893: connect: Connection refused (error 111)
W20260812 08:01:34.666299  6633 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:01:34.666486  6633 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:01:34.666539  6633 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:01:34.669644  6633 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:01:34.669697  6633 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:01:34.669718  6633 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:01:34.669735  6633 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:01:34.673861  6633 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:37951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:41057,127.5.152.61:45893,127.5.152.60:43441
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.61:45893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.61
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:01:34.674702  6633 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:01:34.675505  6633 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:01:34.681023  6639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:34.681025  6638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:34.681023  6642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:34.681427  6633 server_base.cc:1061] running on GCE node
I20260812 08:01:34.681957  6633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:01:34.682799  6633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:01:34.683966  6633 hybrid_clock.cc:648] HybridClock initialized: now 1786521694683924 us; error 62 us; skew 500 ppm
Aug 12 08:01:34 dist-test-slave-1nl6 krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521694, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:01:34.687500  6633 init.cc:377] Logged in from keytab as kudu/127.5.152.61@KRBTEST.COM (short username kudu)
I20260812 08:01:34.696890  6633 webserver.cc:533] Webserver started at https://127.5.152.61:40095/ using document root <none> and password file <none>
I20260812 08:01:34.697666  6633 fs_manager.cc:362] Metadata directory not provided
I20260812 08:01:34.697737  6633 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:01:34.698012  6633 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:01:34.699810  6633 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/data/instance:
uuid: "ec6dd65955cd43148ab4e7729cfc015a"
format_stamp: "Formatted at 2026-08-12 08:01:34 on dist-test-slave-1nl6"
I20260812 08:01:34.700285  6633 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/wal/instance:
uuid: "ec6dd65955cd43148ab4e7729cfc015a"
format_stamp: "Formatted at 2026-08-12 08:01:34 on dist-test-slave-1nl6"
I20260812 08:01:34.703660  6633 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 08:01:34.705701  6659 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:34.706600  6633 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:01:34.706734  6633 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/wal
uuid: "ec6dd65955cd43148ab4e7729cfc015a"
format_stamp: "Formatted at 2026-08-12 08:01:34 on dist-test-slave-1nl6"
I20260812 08:01:34.706853  6633 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:01:34.724337  6633 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:01:34.726990  6633 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:01:34.727169  6633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:01:34.734366  6742 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.61:45893 every 8 connection(s)
I20260812 08:01:34.734378  6633 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.61:45893
I20260812 08:01:34.735404  6633 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
I20260812 08:01:34.737942  6743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:01:34.740984  6743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:01:34.741298  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 6633
I20260812 08:01:34.741410  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-1/wal/instance
I20260812 08:01:34.743671  6618 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } attempt: 3
Aug 12 08:01:34 dist-test-slave-1nl6 krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521694, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
Aug 12 08:01:34 dist-test-slave-1nl6 krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521694, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.152.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.60@KRBTEST.COM" created.
I20260812 08:01:34.755271  6618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:01:34.755416  6743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:01:34.756770  6618 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:43441: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43441: connect: Connection refused (error 111)
Aug 12 08:01:34 dist-test-slave-1nl6 krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521694, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:01:34.764855  6743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:01:34.765585  6661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43441: connect: Connection refused (error 111)
W20260812 08:01:34.766891  6743 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:43441: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43441: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.60@KRBTEST.COM" created.
I20260812 08:01:34.787935  6618 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } attempt: 1
W20260812 08:01:34.789384  6618 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:43441: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43441: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:01:34.793421  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.60:43441
--webserver_interface=127.5.152.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:37951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:41057,127.5.152.61:45893,127.5.152.60:43441
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 08:01:34.816210  6743 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } attempt: 1
W20260812 08:01:34.817560  6743 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:43441: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43441: connect: Connection refused (error 111)
I20260812 08:01:34.864598  6618 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } attempt: 2
W20260812 08:01:34.866367  6618 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:43441: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43441: connect: Connection refused (error 111)
I20260812 08:01:34.885720  6743 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } attempt: 2
W20260812 08:01:34.887128  6743 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:43441: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43441: connect: Connection refused (error 111)
W20260812 08:01:34.898123  6757 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:01:34.898356  6757 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:01:34.898476  6757 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:01:34.901707  6757 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:01:34.901799  6757 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:01:34.901845  6757 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:01:34.901882  6757 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:01:34.905956  6757 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:37951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:41057,127.5.152.61:45893,127.5.152.60:43441
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.60:43441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.60
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:01:34.906875  6757 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:01:34.907676  6757 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:01:34.912793  6766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:34.912806  6764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:34.912801  6770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:34.913218  6757 server_base.cc:1061] running on GCE node
I20260812 08:01:34.913705  6757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:01:34.914507  6757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:01:34.915685  6757 hybrid_clock.cc:648] HybridClock initialized: now 1786521694915670 us; error 32 us; skew 500 ppm
Aug 12 08:01:34 dist-test-slave-1nl6 krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521694, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:01:34.918322  6757 init.cc:377] Logged in from keytab as kudu/127.5.152.60@KRBTEST.COM (short username kudu)
I20260812 08:01:34.925628  6757 webserver.cc:533] Webserver started at https://127.5.152.60:36411/ using document root <none> and password file <none>
I20260812 08:01:34.926126  6757 fs_manager.cc:362] Metadata directory not provided
I20260812 08:01:34.926169  6757 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:01:34.926333  6757 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:01:34.927860  6757 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/data/instance:
uuid: "51e8d6306dcd433f8cdf61e68b8f1b76"
format_stamp: "Formatted at 2026-08-12 08:01:34 on dist-test-slave-1nl6"
I20260812 08:01:34.928261  6757 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/wal/instance:
uuid: "51e8d6306dcd433f8cdf61e68b8f1b76"
format_stamp: "Formatted at 2026-08-12 08:01:34 on dist-test-slave-1nl6"
I20260812 08:01:34.931361  6757 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.001s
I20260812 08:01:34.933272  6782 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:34.934242  6757 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:01:34.934350  6757 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/wal
uuid: "51e8d6306dcd433f8cdf61e68b8f1b76"
format_stamp: "Formatted at 2026-08-12 08:01:34 on dist-test-slave-1nl6"
I20260812 08:01:34.934424  6757 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:01:34.961565  6757 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:01:34.964551  6757 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:01:34.964732  6757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:01:34.965553  6618 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } attempt: 3
W20260812 08:01:34.967051  6618 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:43441: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43441: connect: Connection refused (error 111)
I20260812 08:01:34.972571  6864 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.60:43441 every 8 connection(s)
I20260812 08:01:34.972579  6757 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.60:43441
I20260812 08:01:34.973575  6757 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
I20260812 08:01:34.977811  6865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:01:34.978272  6743 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } attempt: 3
I20260812 08:01:34.978729  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 6757
I20260812 08:01:34.978842  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/master-2/wal/instance
I20260812 08:01:34.982332  6865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 08:01:34 dist-test-slave-1nl6 krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
Aug 12 08:01:34 dist-test-slave-1nl6 krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521694, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
Aug 12 08:01:34 dist-test-slave-1nl6 krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521694, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
I20260812 08:01:34.994016  6743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a: Bootstrap starting.
Aug 12 08:01:34 dist-test-slave-1nl6 krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:01:34.995421  6865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:01:34.996760  6743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a: Neither blocks nor log segments found. Creating new log.
I20260812 08:01:34.997545  6743 log.cc:826] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a: Log is configured to *not* fsync() on all Append() calls
Aug 12 08:01:34 dist-test-slave-1nl6 krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521694, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:01:35.000079  6743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a: No bootstrap required, opened a new log
Aug 12 08:01:34 dist-test-slave-1nl6 krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:01:35.003237  6743 raft_consensus.cc:359] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } } peers { permanent_uuid: "ec6dd65955cd43148ab4e7729cfc015a" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } } peers { permanent_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } }
I20260812 08:01:35.003418  6743 raft_consensus.cc:385] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:01:35.003495  6743 raft_consensus.cc:740] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec6dd65955cd43148ab4e7729cfc015a, State: Initialized, Role: FOLLOWER
I20260812 08:01:35.004015  6743 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } } peers { permanent_uuid: "ec6dd65955cd43148ab4e7729cfc015a" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } } peers { permanent_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } }
I20260812 08:01:35.004487  6878 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } } peers { permanent_uuid: "ec6dd65955cd43148ab4e7729cfc015a" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } } peers { permanent_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } } }
I20260812 08:01:35.004776  6878 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:35.004923  6865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:01:35.005193  6743 sys_catalog.cc:565] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 08:01:35 dist-test-slave-1nl6 krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521694, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
W20260812 08:01:35.010785  6898 catalog_manager.cc:1594] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:01:35.010852  6898 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:01:35.009302  6633 master_runner.cc:187] Error getting master registration for 127.5.152.62:41057: OK, instance_id { permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" instance_seqno: 1786521694500782 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 08:01:35.012636  6633 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 08:01:35.015862  6865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76: Bootstrap starting.
I20260812 08:01:35.017686  6865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76: Neither blocks nor log segments found. Creating new log.
I20260812 08:01:35.018332  6865 log.cc:826] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76: Log is configured to *not* fsync() on all Append() calls
I20260812 08:01:35.019742  6865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76: No bootstrap required, opened a new log
I20260812 08:01:35.021736  6865 raft_consensus.cc:359] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } } peers { permanent_uuid: "ec6dd65955cd43148ab4e7729cfc015a" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } } peers { permanent_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } }
I20260812 08:01:35.021893  6865 raft_consensus.cc:385] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:01:35.021950  6865 raft_consensus.cc:740] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51e8d6306dcd433f8cdf61e68b8f1b76, State: Initialized, Role: FOLLOWER
I20260812 08:01:35.022346  6865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } } peers { permanent_uuid: "ec6dd65955cd43148ab4e7729cfc015a" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } } peers { permanent_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } }
I20260812 08:01:35.022771  6908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } } peers { permanent_uuid: "ec6dd65955cd43148ab4e7729cfc015a" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } } peers { permanent_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } } }
I20260812 08:01:35.022953  6908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:35.023360  6865 sys_catalog.cc:565] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 08:01:35.028051  6925 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:01:35.028172  6925 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:01:35.026399  6757 master_runner.cc:187] Error getting master registration for 127.5.152.62:41057: OK, instance_id { permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" instance_seqno: 1786521694500782 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 08:01:35.028646  6757 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 08:01:35.131227  6618 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } attempt: 4
Aug 12 08:01:35 dist-test-slave-1nl6 krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521694, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:01:35.145710  6618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe: Bootstrap starting.
I20260812 08:01:35.148619  6618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe: Neither blocks nor log segments found. Creating new log.
I20260812 08:01:35.149536  6618 log.cc:826] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe: Log is configured to *not* fsync() on all Append() calls
I20260812 08:01:35.151557  6618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe: No bootstrap required, opened a new log
I20260812 08:01:35.154986  6618 raft_consensus.cc:359] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } } peers { permanent_uuid: "ec6dd65955cd43148ab4e7729cfc015a" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } } peers { permanent_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } }
I20260812 08:01:35.155193  6618 raft_consensus.cc:385] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:01:35.155253  6618 raft_consensus.cc:740] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d369fbc38fff44b28df87f38cddb26fe, State: Initialized, Role: FOLLOWER
I20260812 08:01:35.155714  6618 consensus_queue.cc:260] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } } peers { permanent_uuid: "ec6dd65955cd43148ab4e7729cfc015a" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } } peers { permanent_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } }
I20260812 08:01:35.156248  6929 sys_catalog.cc:455] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } } peers { permanent_uuid: "ec6dd65955cd43148ab4e7729cfc015a" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } } peers { permanent_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } } }
I20260812 08:01:35.156423  6929 sys_catalog.cc:458] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:35.156965  6618 sys_catalog.cc:565] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe [sys.catalog]: configured and running, proceeding with master startup.
W20260812 08:01:35.164077  6949 catalog_manager.cc:1594] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:01:35.164204  6949 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:01:35.166355  6521 master_runner.cc:264] No leader master found from master d369fbc38fff44b28df87f38cddb26fe
I20260812 08:01:35.166433  6521 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 08:01:35.216025  6789 tablet.cc:2406] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 08:01:35.228500  6668 tablet.cc:2406] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 08:01:35.247073  6549 tablet.cc:2406] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:01:35.332056  6908 raft_consensus.cc:493] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:01:35.332319  6908 raft_consensus.cc:515] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } } peers { permanent_uuid: "ec6dd65955cd43148ab4e7729cfc015a" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } } peers { permanent_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } }
I20260812 08:01:35.333250  6908 leader_election.cc:290] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d369fbc38fff44b28df87f38cddb26fe (127.5.152.62:41057), ec6dd65955cd43148ab4e7729cfc015a (127.5.152.61:45893)
I20260812 08:01:35.333721  6574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d369fbc38fff44b28df87f38cddb26fe" is_pre_election: true
I20260812 08:01:35.333854  6702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec6dd65955cd43148ab4e7729cfc015a" is_pre_election: true
I20260812 08:01:35.333993  6574 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51e8d6306dcd433f8cdf61e68b8f1b76 in term 0.
I20260812 08:01:35.334151  6702 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51e8d6306dcd433f8cdf61e68b8f1b76 in term 0.
I20260812 08:01:35.334465  6784 leader_election.cc:304] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51e8d6306dcd433f8cdf61e68b8f1b76, d369fbc38fff44b28df87f38cddb26fe; no voters: 
I20260812 08:01:35.334702  6908 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:01:35.334782  6908 raft_consensus.cc:493] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:01:35.334823  6908 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:01:35.335755  6908 raft_consensus.cc:515] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } } peers { permanent_uuid: "ec6dd65955cd43148ab4e7729cfc015a" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } } peers { permanent_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } }
I20260812 08:01:35.336084  6908 leader_election.cc:290] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [CANDIDATE]: Term 1 election: Requested vote from peers d369fbc38fff44b28df87f38cddb26fe (127.5.152.62:41057), ec6dd65955cd43148ab4e7729cfc015a (127.5.152.61:45893)
I20260812 08:01:35.336444  6574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d369fbc38fff44b28df87f38cddb26fe"
I20260812 08:01:35.336468  6702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec6dd65955cd43148ab4e7729cfc015a"
I20260812 08:01:35.336539  6574 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:01:35.336593  6702 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:01:35.337347  6574 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51e8d6306dcd433f8cdf61e68b8f1b76 in term 1.
I20260812 08:01:35.337545  6702 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51e8d6306dcd433f8cdf61e68b8f1b76 in term 1.
I20260812 08:01:35.337699  6784 leader_election.cc:304] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51e8d6306dcd433f8cdf61e68b8f1b76, d369fbc38fff44b28df87f38cddb26fe; no voters: 
I20260812 08:01:35.337906  6908 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:01:35.338119  6908 raft_consensus.cc:697] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [term 1 LEADER]: Becoming Leader. State: Replica: 51e8d6306dcd433f8cdf61e68b8f1b76, State: Running, Role: LEADER
I20260812 08:01:35.338413  6908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } } peers { permanent_uuid: "ec6dd65955cd43148ab4e7729cfc015a" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } } peers { permanent_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } }
I20260812 08:01:35.340260  6952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51e8d6306dcd433f8cdf61e68b8f1b76. Latest consensus state: current_term: 1 leader_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } } peers { permanent_uuid: "ec6dd65955cd43148ab4e7729cfc015a" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } } peers { permanent_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } } }
I20260812 08:01:35.340361  6952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [sys.catalog]: This master's current role is: LEADER
I20260812 08:01:35.340706  6955 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:01:35.341517  6955 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:01:35.345944  6574 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe [term 1 FOLLOWER]: Refusing update from remote peer 51e8d6306dcd433f8cdf61e68b8f1b76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:01:35.346469  6952 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:01:35.346558  6702 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a [term 1 FOLLOWER]: Refusing update from remote peer 51e8d6306dcd433f8cdf61e68b8f1b76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:01:35.347100  6952 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec6dd65955cd43148ab4e7729cfc015a" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:01:35.349999  6929 sys_catalog.cc:455] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51e8d6306dcd433f8cdf61e68b8f1b76. Latest consensus state: current_term: 1 leader_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } } peers { permanent_uuid: "ec6dd65955cd43148ab4e7729cfc015a" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } } peers { permanent_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } } }
I20260812 08:01:35.350108  6929 sys_catalog.cc:458] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:35.350322  6878 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51e8d6306dcd433f8cdf61e68b8f1b76. Latest consensus state: current_term: 1 leader_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } } peers { permanent_uuid: "ec6dd65955cd43148ab4e7729cfc015a" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } } peers { permanent_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } } }
I20260812 08:01:35.350432  6878 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:35.352175  6908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } } peers { permanent_uuid: "ec6dd65955cd43148ab4e7729cfc015a" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } } peers { permanent_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } } }
I20260812 08:01:35.352283  6908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [sys.catalog]: This master's current role is: LEADER
I20260812 08:01:35.352412  6908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } } peers { permanent_uuid: "ec6dd65955cd43148ab4e7729cfc015a" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } } peers { permanent_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } } }
I20260812 08:01:35.352495  6908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76 [sys.catalog]: This master's current role is: LEADER
I20260812 08:01:35.353050  6929 sys_catalog.cc:455] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } } peers { permanent_uuid: "ec6dd65955cd43148ab4e7729cfc015a" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } } peers { permanent_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } } }
I20260812 08:01:35.353156  6929 sys_catalog.cc:458] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:35.353323  6878 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d369fbc38fff44b28df87f38cddb26fe" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 41057 } } peers { permanent_uuid: "ec6dd65955cd43148ab4e7729cfc015a" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 45893 } } peers { permanent_uuid: "51e8d6306dcd433f8cdf61e68b8f1b76" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43441 } } }
I20260812 08:01:35.353421  6878 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:35.353894  6955 catalog_manager.cc:1383] Generated new cluster ID: 89b658cad2c14389868f08b84520973b
I20260812 08:01:35.353961  6955 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:01:35.353924  6960 mvcc.cc:204] Tried to move back new op lower bound from 7317592864130424832 to 7317592864108982272. Current Snapshot: MvccSnapshot[applied={T|T < 7317592864130424832 or (T in {7317592864130424832})}]
I20260812 08:01:35.368597  6955 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:01:35.369858  6955 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:01:35.377960  6955 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 51e8d6306dcd433f8cdf61e68b8f1b76: Generated new TSK 0
I20260812 08:01:35.378748  6955 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:01:35.443373  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.152.1:0
--local_ip_for_outbound_sockets=127.5.152.1
--webserver_interface=127.5.152.1
--webserver_port=0
--tserver_master_addrs=127.5.152.62:41057,127.5.152.61:45893,127.5.152.60:43441
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:37951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 08:01:35.545447  6977 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:01:35.545672  6977 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:01:35.545763  6977 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:01:35.549070  6977 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:01:35.549243  6977 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.152.1
I20260812 08:01:35.553339  6977 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:37951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.1
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_require_spnego=true
--tserver_master_addrs=127.5.152.62:41057,127.5.152.61:45893,127.5.152.60:43441
--never_fsync=true
--heap_profile_path=/tmp/kudu.6977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.152.1

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:01:35.554275  6977 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:01:35.555083  6977 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:01:35.561466  6993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:35.561525  6997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:35.561570  6977 server_base.cc:1061] running on GCE node
W20260812 08:01:35.561465  6992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:35.562211  6977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:01:35.562672  6977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:01:35.563827  6977 hybrid_clock.cc:648] HybridClock initialized: now 1786521695563798 us; error 48 us; skew 500 ppm
Aug 12 08:01:35 dist-test-slave-1nl6 krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521695, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:01:35.566581  6977 init.cc:377] Logged in from keytab as kudu/127.5.152.1@KRBTEST.COM (short username kudu)
I20260812 08:01:35.573894  6977 webserver.cc:533] Webserver started at https://127.5.152.1:37269/ using document root <none> and password file <none>
I20260812 08:01:35.574462  6977 fs_manager.cc:362] Metadata directory not provided
I20260812 08:01:35.574514  6977 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:01:35.574752  6977 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:01:35.576385  6977 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/data/instance:
uuid: "e7f337bb7cff418bbb22fe79fd382081"
format_stamp: "Formatted at 2026-08-12 08:01:35 on dist-test-slave-1nl6"
I20260812 08:01:35.576897  6977 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/wal/instance:
uuid: "e7f337bb7cff418bbb22fe79fd382081"
format_stamp: "Formatted at 2026-08-12 08:01:35 on dist-test-slave-1nl6"
I20260812 08:01:35.580173  6977 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 08:01:35.582194  7009 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:35.583212  6977 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 08:01:35.583346  6977 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/wal
uuid: "e7f337bb7cff418bbb22fe79fd382081"
format_stamp: "Formatted at 2026-08-12 08:01:35 on dist-test-slave-1nl6"
I20260812 08:01:35.583446  6977 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:01:35.601078  6977 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:01:35.603559  6977 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:01:35.603763  6977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:01:35.604331  6977 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:01:35.605271  6977 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:01:35.605348  6977 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:35.605424  6977 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:01:35.605484  6977 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:35.616139  6977 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.1:37363
I20260812 08:01:35.616192  7193 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.1:37363 every 8 connection(s)
I20260812 08:01:35.617053  6977 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
I20260812 08:01:35.618829  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 6977
I20260812 08:01:35.619011  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/minicluster-data/ts-0/wal/instance
Aug 12 08:01:35 dist-test-slave-1nl6 krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521695, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
Aug 12 08:01:35 dist-test-slave-1nl6 krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521695, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
Aug 12 08:01:35 dist-test-slave-1nl6 krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521695, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:01:35.633675  7195 heartbeater.cc:344] Connected to a master server at 127.5.152.62:41057
I20260812 08:01:35.633805  7194 heartbeater.cc:344] Connected to a master server at 127.5.152.60:43441
I20260812 08:01:35.633961  7194 heartbeater.cc:461] Registering TS with master...
I20260812 08:01:35.633960  7195 heartbeater.cc:461] Registering TS with master...
I20260812 08:01:35.633996  7196 heartbeater.cc:344] Connected to a master server at 127.5.152.61:45893
I20260812 08:01:35.634140  7196 heartbeater.cc:461] Registering TS with master...
I20260812 08:01:35.634609  7196 heartbeater.cc:507] Master 127.5.152.61:45893 requested a full tablet report, sending...
I20260812 08:01:35.634672  7194 heartbeater.cc:507] Master 127.5.152.60:43441 requested a full tablet report, sending...
I20260812 08:01:35.634672  7195 heartbeater.cc:507] Master 127.5.152.62:41057 requested a full tablet report, sending...
I20260812 08:01:35.636230  6684 ts_manager.cc:194] Registered new tserver with Master: e7f337bb7cff418bbb22fe79fd382081 (127.5.152.1:37363)
I20260812 08:01:35.636230  6562 ts_manager.cc:194] Registered new tserver with Master: e7f337bb7cff418bbb22fe79fd382081 (127.5.152.1:37363)
I20260812 08:01:35.637276  6813 ts_manager.cc:194] Registered new tserver with Master: e7f337bb7cff418bbb22fe79fd382081 (127.5.152.1:37363)
I20260812 08:01:35.638835  6813 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.152.1@KRBTEST.COM'} at 127.5.152.1:60701
I20260812 08:01:35.639703  5728 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
time=2026-08-12T08:01:35.697Z level=INFO source=main.go:1678 msg="updated GOGC" old=100 new=75
time=2026-08-12T08:01:35.699Z level=INFO source=main.go:744 msg="Leaving GOMAXPROCS=4: CPU quota undefined" component=automaxprocs
time=2026-08-12T08:01:35.699Z level=INFO source=memlimit.go:198 msg="GOMEMLIMIT is updated" component=automemlimit package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=14142549196 previous=9223372036854775807
time=2026-08-12T08:01:35.700Z level=INFO source=main.go:851 msg="Starting Prometheus Server" mode=server version="(version=3.11.2, branch=HEAD, revision=f0f0fdd679dcd6df320b0558b20919f7cd44c407)"
time=2026-08-12T08:01:35.700Z level=INFO source=main.go:856 msg="operational information" build_context="(go=go1.26.2, platform=linux/amd64, user=root@d2684485c347, date=20260413-11:53:40, tags=netgo,builtinassets)" host_details="(Linux 5.4.0-1157-gcp #166~18.04.1-Ubuntu SMP Wed Jan 21 01:57:46 UTC 2026 x86_64 dist-test-slave-1nl6 (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)"
time=2026-08-12T08:01:35.947Z level=INFO source=web.go:710 msg="Start listening for connections" component=web address=127.0.0.1:0
time=2026-08-12T08:01:35.948Z level=INFO source=main.go:1410 msg="Starting TSDB ..."
time=2026-08-12T08:01:35.951Z level=INFO source=tls_config.go:354 msg="Listening on" component=web address=127.0.0.1:37541
time=2026-08-12T08:01:35.951Z level=INFO source=tls_config.go:357 msg="TLS is disabled." component=web http2=false address=127.0.0.1:37541
time=2026-08-12T08:01:35.951Z level=INFO source=head.go:698 msg="Replaying on-disk memory mappable chunks if any" component=tsdb
time=2026-08-12T08:01:35.951Z level=INFO source=head.go:784 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=2.691µs
time=2026-08-12T08:01:35.951Z level=INFO source=head.go:792 msg="Replaying WAL, this may take a while" component=tsdb
time=2026-08-12T08:01:35.952Z level=INFO source=head.go:865 msg="WAL segment loaded" component=tsdb segment=0 maxSegment=0 duration=343.067µs
time=2026-08-12T08:01:35.952Z level=INFO source=head.go:902 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=53.589µs wal_replay_duration=387.189µs wbl_replay_duration=166ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=2.691µs total_replay_duration=503.964µs
time=2026-08-12T08:01:35.954Z level=INFO source=main.go:1431 msg="filesystem information" fs_type=OVERLAYFS_SUPER_MAGIC
time=2026-08-12T08:01:35.954Z level=INFO source=main.go:1434 msg="TSDB started"
time=2026-08-12T08:01:35.954Z level=INFO source=main.go:1632 msg="Loading configuration file" filename=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/prometheus.yml
time=2026-08-12T08:01:35.954Z level=INFO source=main.go:1048 msg="TSDB retention updated" duration=15d size=0B percentage=0
time=2026-08-12T08:01:35.957Z level=INFO source=main.go:1671 msg="Completed loading of configuration file" db_storage=59.158µs remote_storage=2.344µs web_handler=677ns query_engine=1.384µs scrape=1.986408ms scrape_sd=357.731µs notify=3.18µs notify_sd=1.426µs rules=3.144µs tracing=7.504µs filename=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786521680053065-5728-0/prometheus.yml totalDuration=2.78023ms
time=2026-08-12T08:01:35.957Z level=INFO source=main.go:1395 msg="Server is ready to receive web requests."
time=2026-08-12T08:01:35.957Z level=INFO source=manager.go:209 msg="Starting rule manager..." component="rule manager"
time=2026-08-12T08:01:35.990Z level=ERROR source=refresh.go:72 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:01:35.994Z level=ERROR source=refresh.go:72 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:01:36.000Z level=ERROR source=refresh.go:72 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
I20260812 08:01:36.012285  6898 catalog_manager.cc:1295] Loaded cluster ID: 89b658cad2c14389868f08b84520973b
I20260812 08:01:36.012406  6898 catalog_manager.cc:1588] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a: loading cluster ID for follower catalog manager: success
I20260812 08:01:36.013994  6898 catalog_manager.cc:1610] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a: acquiring CA information for follower catalog manager: success
I20260812 08:01:36.015057  6898 catalog_manager.cc:1638] T 00000000000000000000000000000000 P ec6dd65955cd43148ab4e7729cfc015a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 08:01:36.165473  6949 catalog_manager.cc:1295] Loaded cluster ID: 89b658cad2c14389868f08b84520973b
I20260812 08:01:36.165586  6949 catalog_manager.cc:1588] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe: loading cluster ID for follower catalog manager: success
I20260812 08:01:36.167281  6949 catalog_manager.cc:1610] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe: acquiring CA information for follower catalog manager: success
I20260812 08:01:36.168334  6949 catalog_manager.cc:1638] T 00000000000000000000000000000000 P d369fbc38fff44b28df87f38cddb26fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 08:01:36.640702  7194 heartbeater.cc:499] Master 127.5.152.60:43441 was elected leader, sending a full tablet report...
time=2026-08-12T08:01:37.991Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:01:37.996Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:01:38.001Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:01:39.992Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:01:39.995Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:01:40.001Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:01:41.991Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:01:41.996Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:01:42.000Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:01:43.901Z level=WARN source=main.go:1191 msg="Received an OS signal, exiting gracefully..." signal=terminated
time=2026-08-12T08:01:43.902Z level=INFO source=main.go:1216 msg="Stopping scrape discovery manager..."
time=2026-08-12T08:01:43.902Z level=INFO source=main.go:1230 msg="Stopping notify discovery manager..."
time=2026-08-12T08:01:43.902Z level=INFO source=main.go:1245 msg="Stopping rule manager manager..."
time=2026-08-12T08:01:43.902Z level=INFO source=main.go:1212 msg="Scrape discovery manager stopped"
time=2026-08-12T08:01:43.902Z level=INFO source=manager.go:223 msg="Stopping rule manager..." component="rule manager"
time=2026-08-12T08:01:43.902Z level=INFO source=manager.go:239 msg="Rule manager stopped" component="rule manager"
time=2026-08-12T08:01:43.902Z level=INFO source=main.go:1269 msg="Stopping scrape manager..."
time=2026-08-12T08:01:43.902Z level=INFO source=main.go:1284 msg="Stopping tracing manager..."
time=2026-08-12T08:01:43.902Z level=INFO source=main.go:1369 msg="Stopping reloaders..."
time=2026-08-12T08:01:43.902Z level=INFO source=main.go:1458 msg="Stopping storage..."
time=2026-08-12T08:01:43.902Z level=INFO source=main.go:1261 msg="Scrape manager stopped"
time=2026-08-12T08:01:43.902Z level=INFO source=main.go:1226 msg="Notify discovery manager stopped"
time=2026-08-12T08:01:43.902Z level=INFO source=main.go:1280 msg="Tracing manager stopped"
time=2026-08-12T08:01:43.902Z level=INFO source=main.go:1241 msg="Rule manager stopped"
time=2026-08-12T08:01:43.902Z level=INFO source=main.go:1361 msg="Reloaders stopped"
time=2026-08-12T08:01:43.903Z level=INFO source=main.go:1536 msg="Stopping web handler..."
time=2026-08-12T08:01:43.903Z level=INFO source=main.go:1559 msg="Stopping notifier manager..."
time=2026-08-12T08:01:43.903Z level=INFO source=manager.go:321 msg="Stopping notification manager..." component=notifier
time=2026-08-12T08:01:43.903Z level=INFO source=main.go:1555 msg="Notifier manager stopped"
time=2026-08-12T08:01:43.903Z level=INFO source=main.go:1454 msg="TSDB stopped"
time=2026-08-12T08:01:43.903Z level=INFO source=main.go:1532 msg="Web handler stopped"
time=2026-08-12T08:01:43.903Z level=INFO source=main.go:1570 msg="See you next time!"
I20260812 08:01:43.912068  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 6977
I20260812 08:01:43.923892  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 6521
I20260812 08:01:43.935177  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 6633
I20260812 08:01:43.948208  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 6757
2026-08-12T08:01:43Z chronyd exiting
[       OK ] PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly (11958 ms)
[----------] 2 tests from PrometheusSpnegoITest (23913 ms total)

[----------] 6 tests from PrometheusTokenITest
[ RUN      ] PrometheusTokenITest.TestCorrectTokenScrapeSucceeds
Loading random data
Initializing database '/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 08:01:44 dist-test-slave-1nl6 krb5kdc[7233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 08:01:44 dist-test-slave-1nl6 krb5kdc[7233](info): set up 2 sockets
Aug 12 08:01:44 dist-test-slave-1nl6 krb5kdc[7233](info): commencing operation
krb5kdc: starting...
W20260812 08:01:46.186431  5728 mini_kdc.cc:121] Time spent starting KDC: real 2.210s	user 0.004s	sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/test-user.keytab.
Aug 12 08:01:46 dist-test-slave-1nl6 krb5kdc[7233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM: 
2026-08-12T08:01:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T08:01:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:01:46.327936  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.62:43997
--webserver_interface=127.5.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:40041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:43997,127.5.152.61:43143,127.5.152.60:43465
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 08:01:46.433000  7258 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:01:46.433271  7258 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:01:46.433368  7258 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:01:46.436770  7258 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:01:46.436861  7258 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:01:46.436899  7258 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:01:46.436959  7258 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:01:46.441123  7258 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:40041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:43997,127.5.152.61:43143,127.5.152.60:43465
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.62:43997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.62
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:01:46.442121  7258 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:01:46.443009  7258 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:01:46.448779  7269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:46.448814  7274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:46.448894  7258 server_base.cc:1061] running on GCE node
W20260812 08:01:46.448814  7270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:46.449740  7258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:01:46.451005  7258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:01:46.452227  7258 hybrid_clock.cc:648] HybridClock initialized: now 1786521706452189 us; error 51 us; skew 500 ppm
Aug 12 08:01:46 dist-test-slave-1nl6 krb5kdc[7233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:01:46.455464  7258 init.cc:377] Logged in from keytab as kudu/127.5.152.62@KRBTEST.COM (short username kudu)
I20260812 08:01:46.467233  7258 webserver.cc:533] Webserver started at https://127.5.152.62:34689/ using document root <none> and password file <none>
I20260812 08:01:46.467868  7258 fs_manager.cc:362] Metadata directory not provided
I20260812 08:01:46.467949  7258 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:01:46.468186  7258 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:01:46.469817  7258 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-0/data/instance:
uuid: "d689065b16e64c50bf34d45d36b59bea"
format_stamp: "Formatted at 2026-08-12 08:01:46 on dist-test-slave-1nl6"
I20260812 08:01:46.470276  7258 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-0/wal/instance:
uuid: "d689065b16e64c50bf34d45d36b59bea"
format_stamp: "Formatted at 2026-08-12 08:01:46 on dist-test-slave-1nl6"
I20260812 08:01:46.473825  7258 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.006s	sys 0.000s
I20260812 08:01:46.475958  7292 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:46.476935  7258 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.001s
I20260812 08:01:46.477077  7258 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-0/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-0/wal
uuid: "d689065b16e64c50bf34d45d36b59bea"
format_stamp: "Formatted at 2026-08-12 08:01:46 on dist-test-slave-1nl6"
I20260812 08:01:46.477173  7258 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:01:46.492736  7258 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:01:46.495242  7258 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:01:46.495446  7258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:01:46.502844  7258 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.62:43997
I20260812 08:01:46.502851  7381 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.62:43997 every 8 connection(s)
I20260812 08:01:46.503896  7258 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
I20260812 08:01:46.506573  7384 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:01:46.509831  7384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:01:46.513698  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 7258
I20260812 08:01:46.513849  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-0/wal/instance
Aug 12 08:01:46 dist-test-slave-1nl6 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.152.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.61@KRBTEST.COM" created.
I20260812 08:01:46.526482  7384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:01:46.527848  7295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.152.61:43143: connect: Connection refused (error 111)
W20260812 08:01:46.529201  7384 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:43143: Network error: Client connection negotiation failed: client connection to 127.5.152.61:43143: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:01:46.566457  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.61:43143
--webserver_interface=127.5.152.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:40041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:43997,127.5.152.61:43143,127.5.152.60:43465
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 08:01:46.578454  7384 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } attempt: 1
W20260812 08:01:46.580094  7384 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:43143: Network error: Client connection negotiation failed: client connection to 127.5.152.61:43143: connect: Connection refused (error 111)
I20260812 08:01:46.648331  7384 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } attempt: 2
W20260812 08:01:46.649883  7384 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:43143: Network error: Client connection negotiation failed: client connection to 127.5.152.61:43143: connect: Connection refused (error 111)
W20260812 08:01:46.673136  7406 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:01:46.673370  7406 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:01:46.673483  7406 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:01:46.676718  7406 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:01:46.676810  7406 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:01:46.676857  7406 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:01:46.676898  7406 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:01:46.680881  7406 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:40041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:43997,127.5.152.61:43143,127.5.152.60:43465
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.61:43143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.61
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:01:46.681722  7406 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:01:46.682442  7406 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:01:46.687479  7415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:46.687534  7419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:46.687556  7416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:46.687718  7406 server_base.cc:1061] running on GCE node
I20260812 08:01:46.688268  7406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:01:46.689013  7406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:01:46.690171  7406 hybrid_clock.cc:648] HybridClock initialized: now 1786521706690158 us; error 31 us; skew 500 ppm
Aug 12 08:01:46 dist-test-slave-1nl6 krb5kdc[7233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:01:46.692926  7406 init.cc:377] Logged in from keytab as kudu/127.5.152.61@KRBTEST.COM (short username kudu)
I20260812 08:01:46.708616  7406 webserver.cc:533] Webserver started at https://127.5.152.61:40263/ using document root <none> and password file <none>
I20260812 08:01:46.709434  7406 fs_manager.cc:362] Metadata directory not provided
I20260812 08:01:46.709504  7406 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:01:46.709740  7406 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:01:46.711930  7406 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-1/data/instance:
uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f"
format_stamp: "Formatted at 2026-08-12 08:01:46 on dist-test-slave-1nl6"
I20260812 08:01:46.712563  7406 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-1/wal/instance:
uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f"
format_stamp: "Formatted at 2026-08-12 08:01:46 on dist-test-slave-1nl6"
I20260812 08:01:46.716949  7406 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.001s	sys 0.004s
I20260812 08:01:46.719617  7434 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:46.720724  7406 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260812 08:01:46.720856  7406 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-1/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-1/wal
uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f"
format_stamp: "Formatted at 2026-08-12 08:01:46 on dist-test-slave-1nl6"
I20260812 08:01:46.720974  7406 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:01:46.733613  7406 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:01:46.736356  7406 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:01:46.736539  7406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:01:46.741044  7384 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } attempt: 3
W20260812 08:01:46.742411  7384 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:43143: Network error: Client connection negotiation failed: client connection to 127.5.152.61:43143: connect: Connection refused (error 111)
I20260812 08:01:46.744545  7406 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.61:43143
I20260812 08:01:46.744545  7517 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.61:43143 every 8 connection(s)
I20260812 08:01:46.745524  7406 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
I20260812 08:01:46.748617  7519 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:01:46.752363  7519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:01:46.752806  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 7406
I20260812 08:01:46.752940  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-1/wal/instance
Aug 12 08:01:46 dist-test-slave-1nl6 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.152.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.60@KRBTEST.COM" created.
I20260812 08:01:46.765321  7519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 08:01:46 dist-test-slave-1nl6 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:01:46.776024  7519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:01:46.776780  7435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43465: connect: Connection refused (error 111)
W20260812 08:01:46.777618  7519 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:43465: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43465: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:01:46.805768  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.60:43465
--webserver_interface=127.5.152.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:40041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:43997,127.5.152.61:43143,127.5.152.60:43465
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 08:01:46.826886  7519 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } attempt: 1
W20260812 08:01:46.828585  7519 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:43465: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43465: connect: Connection refused (error 111)
I20260812 08:01:46.885673  7384 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } attempt: 4
Aug 12 08:01:46 dist-test-slave-1nl6 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:01:46.895411  7384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:01:46.896718  7384 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:43465: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43465: connect: Connection refused (error 111)
I20260812 08:01:46.896749  7519 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } attempt: 2
W20260812 08:01:46.897915  7519 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:43465: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43465: connect: Connection refused (error 111)
W20260812 08:01:46.913559  7535 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:01:46.913779  7535 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:01:46.913903  7535 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:01:46.917284  7535 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:01:46.917379  7535 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:01:46.917429  7535 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:01:46.917465  7535 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:01:46.921664  7535 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:40041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:43997,127.5.152.61:43143,127.5.152.60:43465
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.60:43465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.60
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:01:46.922617  7535 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:01:46.923434  7535 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:01:46.928650  7542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:46.928615  7547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:46.928622  7543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:46.929000  7535 server_base.cc:1061] running on GCE node
I20260812 08:01:46.929430  7535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:01:46.930263  7535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:01:46.931444  7535 hybrid_clock.cc:648] HybridClock initialized: now 1786521706931425 us; error 31 us; skew 500 ppm
Aug 12 08:01:46 dist-test-slave-1nl6 krb5kdc[7233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:01:46.934312  7535 init.cc:377] Logged in from keytab as kudu/127.5.152.60@KRBTEST.COM (short username kudu)
I20260812 08:01:46.945690  7535 webserver.cc:533] Webserver started at https://127.5.152.60:46457/ using document root <none> and password file <none>
I20260812 08:01:46.946357  7535 fs_manager.cc:362] Metadata directory not provided
I20260812 08:01:46.946410  7535 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:01:46.946646  7535 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:01:46.948406  7535 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-2/data/instance:
uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6"
format_stamp: "Formatted at 2026-08-12 08:01:46 on dist-test-slave-1nl6"
I20260812 08:01:46.948915  7535 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-2/wal/instance:
uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6"
format_stamp: "Formatted at 2026-08-12 08:01:46 on dist-test-slave-1nl6"
I20260812 08:01:46.952701  7535 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.000s	sys 0.005s
I20260812 08:01:46.954766  7566 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:46.955785  7535 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.002s
I20260812 08:01:46.955935  7535 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-2/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-2/wal
uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6"
format_stamp: "Formatted at 2026-08-12 08:01:46 on dist-test-slave-1nl6"
I20260812 08:01:46.955930  7384 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } attempt: 1
I20260812 08:01:46.956049  7535 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260812 08:01:46.957360  7384 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:43465: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43465: connect: Connection refused (error 111)
I20260812 08:01:46.971963  7535 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:01:46.975190  7535 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:01:46.975363  7535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:01:46.983505  7642 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.60:43465 every 8 connection(s)
I20260812 08:01:46.983513  7535 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.60:43465
I20260812 08:01:46.984738  7535 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
I20260812 08:01:46.988202  7643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:01:46.989133  7519 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } attempt: 3
I20260812 08:01:46.991911  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 7535
I20260812 08:01:46.992041  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/master-2/wal/instance
I20260812 08:01:46.992729  7643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 08:01:46 dist-test-slave-1nl6 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
Aug 12 08:01:46 dist-test-slave-1nl6 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
Aug 12 08:01:46 dist-test-slave-1nl6 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
I20260812 08:01:47.005481  7643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:01:47.006196  7519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f: Bootstrap starting.
Aug 12 08:01:47 dist-test-slave-1nl6 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:01:47.009866  7519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f: Neither blocks nor log segments found. Creating new log.
Aug 12 08:01:47 dist-test-slave-1nl6 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:01:47.010761  7519 log.cc:826] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f: Log is configured to *not* fsync() on all Append() calls
I20260812 08:01:47.012730  7519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f: No bootstrap required, opened a new log
I20260812 08:01:47.015790  7643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:01:47.016124  7519 raft_consensus.cc:359] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } } peers { permanent_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } } peers { permanent_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } }
I20260812 08:01:47.016324  7519 raft_consensus.cc:385] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:01:47.016381  7519 raft_consensus.cc:740] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd8e7288f4ca4adfbe6b3220dd05cf3f, State: Initialized, Role: FOLLOWER
I20260812 08:01:47.016866  7519 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } } peers { permanent_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } } peers { permanent_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } }
I20260812 08:01:47.017421  7659 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } } peers { permanent_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } } peers { permanent_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } } }
I20260812 08:01:47.017539  7659 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:47.018148  7519 sys_catalog.cc:565] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 08:01:47 dist-test-slave-1nl6 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
Aug 12 08:01:47 dist-test-slave-1nl6 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
W20260812 08:01:47.023528  7680 catalog_manager.cc:1594] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:01:47.023602  7680 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:01:47.024533  7384 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } attempt: 2
I20260812 08:01:47.023885  7406 master_runner.cc:187] Error getting master registration for 127.5.152.62:43997: OK, instance_id { permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" instance_seqno: 1786521706500817 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 08:01:47.026639  7406 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 08:01:47.029515  7643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6: Bootstrap starting.
Aug 12 08:01:47 dist-test-slave-1nl6 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521706, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:01:47.032145  7643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6: Neither blocks nor log segments found. Creating new log.
I20260812 08:01:47.033022  7643 log.cc:826] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6: Log is configured to *not* fsync() on all Append() calls
I20260812 08:01:47.034799  7643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6: No bootstrap required, opened a new log
I20260812 08:01:47.037647  7384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea: Bootstrap starting.
I20260812 08:01:47.038057  7643 raft_consensus.cc:359] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } } peers { permanent_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } } peers { permanent_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } }
I20260812 08:01:47.038249  7643 raft_consensus.cc:385] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:01:47.038301  7643 raft_consensus.cc:740] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8490174cc7849eb9bdc2f8f20c4a1b6, State: Initialized, Role: FOLLOWER
I20260812 08:01:47.038743  7643 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } } peers { permanent_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } } peers { permanent_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } }
I20260812 08:01:47.039316  7683 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } } peers { permanent_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } } peers { permanent_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } } }
I20260812 08:01:47.039510  7683 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:47.039948  7643 sys_catalog.cc:565] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:01:47.040302  7384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea: Neither blocks nor log segments found. Creating new log.
I20260812 08:01:47.041167  7384 log.cc:826] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea: Log is configured to *not* fsync() on all Append() calls
W20260812 08:01:47.046010  7707 catalog_manager.cc:1594] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:01:47.046092  7707 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:01:47.043726  7535 master_runner.cc:187] Error getting master registration for 127.5.152.62:43997: OK, instance_id { permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" instance_seqno: 1786521706500817 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 08:01:47.046751  7535 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 08:01:47.047784  7384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea: No bootstrap required, opened a new log
I20260812 08:01:47.051093  7384 raft_consensus.cc:359] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } } peers { permanent_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } } peers { permanent_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } }
I20260812 08:01:47.051292  7384 raft_consensus.cc:385] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:01:47.051358  7384 raft_consensus.cc:740] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d689065b16e64c50bf34d45d36b59bea, State: Initialized, Role: FOLLOWER
I20260812 08:01:47.051795  7384 consensus_queue.cc:260] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } } peers { permanent_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } } peers { permanent_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } }
I20260812 08:01:47.052448  7709 sys_catalog.cc:455] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } } peers { permanent_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } } peers { permanent_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } } }
I20260812 08:01:47.052776  7709 sys_catalog.cc:458] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:47.053066  7384 sys_catalog.cc:565] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea [sys.catalog]: configured and running, proceeding with master startup.
W20260812 08:01:47.059311  7726 catalog_manager.cc:1594] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:01:47.059516  7726 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:01:47.062623  7258 master_runner.cc:264] No leader master found from master d689065b16e64c50bf34d45d36b59bea
I20260812 08:01:47.062723  7258 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 08:01:47.123100  7683 raft_consensus.cc:493] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:01:47.123315  7683 raft_consensus.cc:515] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } } peers { permanent_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } } peers { permanent_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } }
I20260812 08:01:47.124382  7683 leader_election.cc:290] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d689065b16e64c50bf34d45d36b59bea (127.5.152.62:43997), cd8e7288f4ca4adfbe6b3220dd05cf3f (127.5.152.61:43143)
I20260812 08:01:47.125327  7344 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d689065b16e64c50bf34d45d36b59bea" is_pre_election: true
I20260812 08:01:47.125639  7344 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8490174cc7849eb9bdc2f8f20c4a1b6 in term 0.
I20260812 08:01:47.126113  7567 leader_election.cc:304] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d689065b16e64c50bf34d45d36b59bea, e8490174cc7849eb9bdc2f8f20c4a1b6; no voters: 
I20260812 08:01:47.126536  7477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f" is_pre_election: true
I20260812 08:01:47.126806  7477 raft_consensus.cc:2468] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8490174cc7849eb9bdc2f8f20c4a1b6 in term 0.
I20260812 08:01:47.127393  7683 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:01:47.127466  7683 raft_consensus.cc:493] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:01:47.127507  7683 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:01:47.128686  7683 raft_consensus.cc:515] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } } peers { permanent_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } } peers { permanent_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } }
I20260812 08:01:47.129050  7683 leader_election.cc:290] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [CANDIDATE]: Term 1 election: Requested vote from peers d689065b16e64c50bf34d45d36b59bea (127.5.152.62:43997), cd8e7288f4ca4adfbe6b3220dd05cf3f (127.5.152.61:43143)
I20260812 08:01:47.129689  7344 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d689065b16e64c50bf34d45d36b59bea"
I20260812 08:01:47.129812  7344 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:01:47.131009  7477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f"
I20260812 08:01:47.131134  7477 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:01:47.132454  7477 raft_consensus.cc:2468] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8490174cc7849eb9bdc2f8f20c4a1b6 in term 1.
I20260812 08:01:47.132809  7568 leader_election.cc:304] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd8e7288f4ca4adfbe6b3220dd05cf3f, e8490174cc7849eb9bdc2f8f20c4a1b6; no voters: 
I20260812 08:01:47.133041  7344 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8490174cc7849eb9bdc2f8f20c4a1b6 in term 1.
I20260812 08:01:47.133453  7683 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:01:47.133692  7683 raft_consensus.cc:697] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [term 1 LEADER]: Becoming Leader. State: Replica: e8490174cc7849eb9bdc2f8f20c4a1b6, State: Running, Role: LEADER
I20260812 08:01:47.134016  7683 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } } peers { permanent_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } } peers { permanent_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } }
I20260812 08:01:47.136679  7683 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8490174cc7849eb9bdc2f8f20c4a1b6. Latest consensus state: current_term: 1 leader_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } } peers { permanent_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } } peers { permanent_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } } }
I20260812 08:01:47.136796  7683 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [sys.catalog]: This master's current role is: LEADER
I20260812 08:01:47.138664  7731 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:01:47.139727  7731 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:01:47.146148  7477 raft_consensus.cc:1275] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f [term 1 FOLLOWER]: Refusing update from remote peer e8490174cc7849eb9bdc2f8f20c4a1b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:01:47.146770  7729 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:01:47.150370  7659 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8490174cc7849eb9bdc2f8f20c4a1b6. Latest consensus state: current_term: 1 leader_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } } peers { permanent_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } } peers { permanent_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } } }
I20260812 08:01:47.150501  7659 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:47.154268  7344 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea [term 1 FOLLOWER]: Refusing update from remote peer e8490174cc7849eb9bdc2f8f20c4a1b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:01:47.154866  7729 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:01:47.158749  7709 sys_catalog.cc:455] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8490174cc7849eb9bdc2f8f20c4a1b6. Latest consensus state: current_term: 1 leader_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } } peers { permanent_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } } peers { permanent_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } } }
I20260812 08:01:47.158859  7709 sys_catalog.cc:458] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:47.164660  7729 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } } peers { permanent_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } } peers { permanent_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } } }
I20260812 08:01:47.164784  7729 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [sys.catalog]: This master's current role is: LEADER
I20260812 08:01:47.166051  7709 sys_catalog.cc:455] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } } peers { permanent_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } } peers { permanent_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } } }
I20260812 08:01:47.166349  7709 sys_catalog.cc:458] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:47.168954  7732 mvcc.cc:204] Tried to move back new op lower bound from 7317592912463347712 to 7317592912427978752. Current Snapshot: MvccSnapshot[applied={T|T < 7317592912463347712}]
I20260812 08:01:47.171087  7683 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } } peers { permanent_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } } peers { permanent_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } } }
I20260812 08:01:47.171191  7683 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6 [sys.catalog]: This master's current role is: LEADER
I20260812 08:01:47.172194  7659 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d689065b16e64c50bf34d45d36b59bea" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 43997 } } peers { permanent_uuid: "cd8e7288f4ca4adfbe6b3220dd05cf3f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43143 } } peers { permanent_uuid: "e8490174cc7849eb9bdc2f8f20c4a1b6" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43465 } } }
I20260812 08:01:47.172307  7659 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:47.172940  7736 mvcc.cc:204] Tried to move back new op lower bound from 7317592912463347712 to 7317592912427978752. Current Snapshot: MvccSnapshot[applied={T|T < 7317592912463347712}]
I20260812 08:01:47.173105  7733 mvcc.cc:204] Tried to move back new op lower bound from 7317592912463347712 to 7317592912427978752. Current Snapshot: MvccSnapshot[applied={T|T < 7317592912463347712}]
I20260812 08:01:47.177337  7731 catalog_manager.cc:1383] Generated new cluster ID: 947e30ba0017498fb06edea16282589e
I20260812 08:01:47.177400  7731 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:01:47.225247  7731 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:01:47.226676  7731 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:01:47.244904  7731 catalog_manager.cc:6092] T 00000000000000000000000000000000 P e8490174cc7849eb9bdc2f8f20c4a1b6: Generated new TSK 0
I20260812 08:01:47.245633  7731 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:01:47.314760  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.152.1:0
--local_ip_for_outbound_sockets=127.5.152.1
--webserver_interface=127.5.152.1
--webserver_port=0
--tserver_master_addrs=127.5.152.62:43997,127.5.152.61:43143,127.5.152.60:43465
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:40041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 08:01:47.419862  7753 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:01:47.420058  7753 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:01:47.420172  7753 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:01:47.423770  7753 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:01:47.423923  7753 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.152.1
I20260812 08:01:47.430166  7753 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:40041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.1
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--tserver_master_addrs=127.5.152.62:43997,127.5.152.61:43143,127.5.152.60:43465
--never_fsync=true
--heap_profile_path=/tmp/kudu.7753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.152.1

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:01:47.431495  7753 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:01:47.432269  7753 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:01:47.439391  7760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:47.439429  7763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:47.439430  7761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:47.440608  7753 server_base.cc:1061] running on GCE node
I20260812 08:01:47.441033  7753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:01:47.441526  7753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:01:47.442703  7753 hybrid_clock.cc:648] HybridClock initialized: now 1786521707442679 us; error 41 us; skew 500 ppm
Aug 12 08:01:47 dist-test-slave-1nl6 krb5kdc[7233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521707, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:01:47.446544  7753 init.cc:377] Logged in from keytab as kudu/127.5.152.1@KRBTEST.COM (short username kudu)
I20260812 08:01:47.464066  7753 webserver.cc:533] Webserver started at https://127.5.152.1:34403/ using document root <none> and password file <none>
I20260812 08:01:47.465024  7753 fs_manager.cc:362] Metadata directory not provided
I20260812 08:01:47.465113  7753 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:01:47.465420  7753 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:01:47.468003  7753 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/ts-0/data/instance:
uuid: "e5a19cc490074cc699ff3d83b59d16f9"
format_stamp: "Formatted at 2026-08-12 08:01:47 on dist-test-slave-1nl6"
I20260812 08:01:47.468652  7753 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/ts-0/wal/instance:
uuid: "e5a19cc490074cc699ff3d83b59d16f9"
format_stamp: "Formatted at 2026-08-12 08:01:47 on dist-test-slave-1nl6"
I20260812 08:01:47.473865  7753 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260812 08:01:47.476930  7778 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:47.478161  7753 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260812 08:01:47.478286  7753 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/ts-0/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/ts-0/wal
uuid: "e5a19cc490074cc699ff3d83b59d16f9"
format_stamp: "Formatted at 2026-08-12 08:01:47 on dist-test-slave-1nl6"
I20260812 08:01:47.478410  7753 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:01:47.504469  7753 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:01:47.507519  7753 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:01:47.507731  7753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:01:47.508311  7753 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:01:47.509212  7753 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:01:47.509284  7753 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:47.509351  7753 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:01:47.509387  7753 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:47.519481  7753 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.1:39591
I20260812 08:01:47.519534  7954 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.1:39591 every 8 connection(s)
I20260812 08:01:47.520478  7753 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
I20260812 08:01:47.522220  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 7753
I20260812 08:01:47.522360  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/minicluster-data/ts-0/wal/instance
Aug 12 08:01:47 dist-test-slave-1nl6 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521707, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
Aug 12 08:01:47 dist-test-slave-1nl6 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521707, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
Aug 12 08:01:47 dist-test-slave-1nl6 krb5kdc[7233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521707, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:01:47.536197  7956 heartbeater.cc:344] Connected to a master server at 127.5.152.62:43997
I20260812 08:01:47.536419  7956 heartbeater.cc:461] Registering TS with master...
I20260812 08:01:47.536594  7959 heartbeater.cc:344] Connected to a master server at 127.5.152.61:43143
I20260812 08:01:47.536686  7959 heartbeater.cc:461] Registering TS with master...
I20260812 08:01:47.536954  7956 heartbeater.cc:507] Master 127.5.152.62:43997 requested a full tablet report, sending...
I20260812 08:01:47.537010  7959 heartbeater.cc:507] Master 127.5.152.61:43143 requested a full tablet report, sending...
I20260812 08:01:47.537021  7955 heartbeater.cc:344] Connected to a master server at 127.5.152.60:43465
I20260812 08:01:47.537117  7955 heartbeater.cc:461] Registering TS with master...
I20260812 08:01:47.537286  7955 heartbeater.cc:507] Master 127.5.152.60:43465 requested a full tablet report, sending...
I20260812 08:01:47.538344  7321 ts_manager.cc:194] Registered new tserver with Master: e5a19cc490074cc699ff3d83b59d16f9 (127.5.152.1:39591)
I20260812 08:01:47.538393  7459 ts_manager.cc:194] Registered new tserver with Master: e5a19cc490074cc699ff3d83b59d16f9 (127.5.152.1:39591)
I20260812 08:01:47.539273  7592 ts_manager.cc:194] Registered new tserver with Master: e5a19cc490074cc699ff3d83b59d16f9 (127.5.152.1:39591)
I20260812 08:01:47.540593  7592 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.152.1@KRBTEST.COM'} at 127.5.152.1:46647
I20260812 08:01:47.543380  5728 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
time=2026-08-12T08:01:47.616Z level=INFO source=main.go:1678 msg="updated GOGC" old=100 new=75
time=2026-08-12T08:01:47.617Z level=INFO source=main.go:744 msg="Leaving GOMAXPROCS=4: CPU quota undefined" component=automaxprocs
time=2026-08-12T08:01:47.618Z level=INFO source=memlimit.go:198 msg="GOMEMLIMIT is updated" component=automemlimit package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=14142549196 previous=9223372036854775807
time=2026-08-12T08:01:47.618Z level=INFO source=main.go:851 msg="Starting Prometheus Server" mode=server version="(version=3.11.2, branch=HEAD, revision=f0f0fdd679dcd6df320b0558b20919f7cd44c407)"
time=2026-08-12T08:01:47.618Z level=INFO source=main.go:856 msg="operational information" build_context="(go=go1.26.2, platform=linux/amd64, user=root@d2684485c347, date=20260413-11:53:40, tags=netgo,builtinassets)" host_details="(Linux 5.4.0-1157-gcp #166~18.04.1-Ubuntu SMP Wed Jan 21 01:57:46 UTC 2026 x86_64 dist-test-slave-1nl6 (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)"
time=2026-08-12T08:01:47.884Z level=INFO source=web.go:710 msg="Start listening for connections" component=web address=127.0.0.1:0
time=2026-08-12T08:01:47.885Z level=INFO source=main.go:1410 msg="Starting TSDB ..."
time=2026-08-12T08:01:47.889Z level=INFO source=tls_config.go:354 msg="Listening on" component=web address=127.0.0.1:34175
time=2026-08-12T08:01:47.889Z level=INFO source=tls_config.go:357 msg="TLS is disabled." component=web http2=false address=127.0.0.1:34175
time=2026-08-12T08:01:47.895Z level=INFO source=head.go:698 msg="Replaying on-disk memory mappable chunks if any" component=tsdb
time=2026-08-12T08:01:47.895Z level=INFO source=head.go:784 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=5.425µs
time=2026-08-12T08:01:47.895Z level=INFO source=head.go:792 msg="Replaying WAL, this may take a while" component=tsdb
time=2026-08-12T08:01:47.895Z level=INFO source=head.go:865 msg="WAL segment loaded" component=tsdb segment=0 maxSegment=0 duration=531.396µs
time=2026-08-12T08:01:47.895Z level=INFO source=head.go:902 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=79.03µs wal_replay_duration=586.122µs wbl_replay_duration=370ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=5.425µs total_replay_duration=726.612µs
time=2026-08-12T08:01:47.901Z level=INFO source=main.go:1431 msg="filesystem information" fs_type=OVERLAYFS_SUPER_MAGIC
time=2026-08-12T08:01:47.901Z level=INFO source=main.go:1434 msg="TSDB started"
time=2026-08-12T08:01:47.901Z level=INFO source=main.go:1632 msg="Loading configuration file" filename=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/prometheus.yml
time=2026-08-12T08:01:47.901Z level=INFO source=main.go:1048 msg="TSDB retention updated" duration=15d size=0B percentage=0
time=2026-08-12T08:01:47.904Z level=INFO source=main.go:1671 msg="Completed loading of configuration file" db_storage=133.989µs remote_storage=2.77µs web_handler=744ns query_engine=1.793µs scrape=2.144445ms scrape_sd=71.033µs notify=2.764µs notify_sd=2.084µs rules=4.185µs tracing=16.176µs filename=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786521680053065-5728-0/prometheus.yml totalDuration=3.080199ms
time=2026-08-12T08:01:47.904Z level=INFO source=main.go:1395 msg="Server is ready to receive web requests."
time=2026-08-12T08:01:47.904Z level=INFO source=manager.go:209 msg="Starting rule manager..." component="rule manager"
I20260812 08:01:48.025687  7680 catalog_manager.cc:1295] Loaded cluster ID: 947e30ba0017498fb06edea16282589e
I20260812 08:01:48.025851  7680 catalog_manager.cc:1588] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f: loading cluster ID for follower catalog manager: success
I20260812 08:01:48.029232  7680 catalog_manager.cc:1610] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f: acquiring CA information for follower catalog manager: success
I20260812 08:01:48.031211  7680 catalog_manager.cc:1638] T 00000000000000000000000000000000 P cd8e7288f4ca4adfbe6b3220dd05cf3f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 08:01:48.060755  7726 catalog_manager.cc:1295] Loaded cluster ID: 947e30ba0017498fb06edea16282589e
I20260812 08:01:48.060838  7726 catalog_manager.cc:1588] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea: loading cluster ID for follower catalog manager: success
I20260812 08:01:48.062346  7726 catalog_manager.cc:1610] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea: acquiring CA information for follower catalog manager: success
I20260812 08:01:48.063387  7726 catalog_manager.cc:1638] T 00000000000000000000000000000000 P d689065b16e64c50bf34d45d36b59bea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 08:01:48.542445  7955 heartbeater.cc:499] Master 127.5.152.60:43465 was elected leader, sending a full tablet report...
time=2026-08-12T08:01:55.824Z level=WARN source=main.go:1191 msg="Received an OS signal, exiting gracefully..." signal=terminated
time=2026-08-12T08:01:55.824Z level=INFO source=main.go:1216 msg="Stopping scrape discovery manager..."
time=2026-08-12T08:01:55.824Z level=INFO source=main.go:1230 msg="Stopping notify discovery manager..."
time=2026-08-12T08:01:55.824Z level=INFO source=main.go:1245 msg="Stopping rule manager manager..."
time=2026-08-12T08:01:55.824Z level=INFO source=manager.go:223 msg="Stopping rule manager..." component="rule manager"
time=2026-08-12T08:01:55.824Z level=INFO source=main.go:1212 msg="Scrape discovery manager stopped"
time=2026-08-12T08:01:55.824Z level=INFO source=manager.go:239 msg="Rule manager stopped" component="rule manager"
time=2026-08-12T08:01:55.824Z level=INFO source=main.go:1269 msg="Stopping scrape manager..."
time=2026-08-12T08:01:55.824Z level=INFO source=main.go:1241 msg="Rule manager stopped"
time=2026-08-12T08:01:55.824Z level=INFO source=main.go:1284 msg="Stopping tracing manager..."
time=2026-08-12T08:01:55.824Z level=INFO source=main.go:1369 msg="Stopping reloaders..."
time=2026-08-12T08:01:55.824Z level=INFO source=main.go:1261 msg="Scrape manager stopped"
time=2026-08-12T08:01:55.824Z level=INFO source=main.go:1458 msg="Stopping storage..."
time=2026-08-12T08:01:55.824Z level=INFO source=main.go:1226 msg="Notify discovery manager stopped"
time=2026-08-12T08:01:55.824Z level=INFO source=main.go:1280 msg="Tracing manager stopped"
time=2026-08-12T08:01:55.824Z level=INFO source=main.go:1361 msg="Reloaders stopped"
time=2026-08-12T08:01:55.829Z level=INFO source=main.go:1536 msg="Stopping web handler..."
time=2026-08-12T08:01:55.829Z level=INFO source=main.go:1559 msg="Stopping notifier manager..."
time=2026-08-12T08:01:55.829Z level=INFO source=manager.go:321 msg="Stopping notification manager..." component=notifier
time=2026-08-12T08:01:55.829Z level=INFO source=main.go:1454 msg="TSDB stopped"
time=2026-08-12T08:01:55.829Z level=INFO source=main.go:1555 msg="Notifier manager stopped"
time=2026-08-12T08:01:55.829Z level=INFO source=main.go:1532 msg="Web handler stopped"
time=2026-08-12T08:01:55.830Z level=INFO source=main.go:1570 msg="See you next time!"
I20260812 08:01:55.844501  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 7753
I20260812 08:01:55.857933  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 7258
I20260812 08:01:55.870096  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 7406
I20260812 08:01:55.881690  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 7535
2026-08-12T08:01:55Z chronyd exiting
[       OK ] PrometheusTokenITest.TestCorrectTokenScrapeSucceeds (11933 ms)
[ RUN      ] PrometheusTokenITest.TestWrongTokenScrapeFails
Loading random data
Initializing database '/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 08:01:55 dist-test-slave-1nl6 krb5kdc[8004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 08:01:55 dist-test-slave-1nl6 krb5kdc[8004](info): set up 2 sockets
Aug 12 08:01:55 dist-test-slave-1nl6 krb5kdc[8004](info): commencing operation
krb5kdc: starting...
W20260812 08:01:58.228678  5728 mini_kdc.cc:121] Time spent starting KDC: real 2.318s	user 0.003s	sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/test-user.keytab.
Aug 12 08:01:58 dist-test-slave-1nl6 krb5kdc[8004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM: 
2026-08-12T08:01:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T08:01:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:01:58.383962  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.62:35187
--webserver_interface=127.5.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:45029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:35187,127.5.152.61:33515,127.5.152.60:39943
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 08:01:58.489598  8024 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:01:58.489883  8024 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:01:58.489988  8024 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:01:58.493355  8024 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:01:58.493458  8024 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:01:58.493501  8024 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:01:58.493557  8024 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:01:58.497843  8024 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:45029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:35187,127.5.152.61:33515,127.5.152.60:39943
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.62:35187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.62
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:01:58.498909  8024 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:01:58.499761  8024 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:01:58.505015  8036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:58.505002  8031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:58.505102  8033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:58.505223  8024 server_base.cc:1061] running on GCE node
I20260812 08:01:58.505791  8024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:01:58.506603  8024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:01:58.507781  8024 hybrid_clock.cc:648] HybridClock initialized: now 1786521718507766 us; error 32 us; skew 500 ppm
Aug 12 08:01:58 dist-test-slave-1nl6 krb5kdc[8004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:01:58.510524  8024 init.cc:377] Logged in from keytab as kudu/127.5.152.62@KRBTEST.COM (short username kudu)
I20260812 08:01:58.521821  8024 webserver.cc:533] Webserver started at https://127.5.152.62:44587/ using document root <none> and password file <none>
I20260812 08:01:58.522498  8024 fs_manager.cc:362] Metadata directory not provided
I20260812 08:01:58.522552  8024 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:01:58.522782  8024 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:01:58.524577  8024 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/data/instance:
uuid: "0c905ecc2d934adeb0bc638a73de6c50"
format_stamp: "Formatted at 2026-08-12 08:01:58 on dist-test-slave-1nl6"
I20260812 08:01:58.525055  8024 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/wal/instance:
uuid: "0c905ecc2d934adeb0bc638a73de6c50"
format_stamp: "Formatted at 2026-08-12 08:01:58 on dist-test-slave-1nl6"
I20260812 08:01:58.528617  8024 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.003s
I20260812 08:01:58.530654  8051 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:58.531706  8024 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:01:58.531852  8024 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/wal
uuid: "0c905ecc2d934adeb0bc638a73de6c50"
format_stamp: "Formatted at 2026-08-12 08:01:58 on dist-test-slave-1nl6"
I20260812 08:01:58.531961  8024 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:01:58.552165  8024 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:01:58.555133  8024 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:01:58.555358  8024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:01:58.562860  8024 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.62:35187
I20260812 08:01:58.562860  8131 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.62:35187 every 8 connection(s)
I20260812 08:01:58.563913  8024 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
I20260812 08:01:58.566601  8133 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:01:58.569892  8133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:01:58.570479  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 8024
I20260812 08:01:58.570593  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/wal/instance
Aug 12 08:01:58 dist-test-slave-1nl6 krb5kdc[8004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.152.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.61@KRBTEST.COM" created.
I20260812 08:01:58.583969  8133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:01:58.584771  8055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.152.61:33515: connect: Connection refused (error 111)
W20260812 08:01:58.585603  8133 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:33515: Network error: Client connection negotiation failed: client connection to 127.5.152.61:33515: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:01:58.614471  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.61:33515
--webserver_interface=127.5.152.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:45029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:35187,127.5.152.61:33515,127.5.152.60:39943
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 08:01:58.634856  8133 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } attempt: 1
W20260812 08:01:58.636468  8133 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:33515: Network error: Client connection negotiation failed: client connection to 127.5.152.61:33515: connect: Connection refused (error 111)
I20260812 08:01:58.704666  8133 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } attempt: 2
W20260812 08:01:58.706063  8133 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:33515: Network error: Client connection negotiation failed: client connection to 127.5.152.61:33515: connect: Connection refused (error 111)
W20260812 08:01:58.710418  8152 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:01:58.710573  8152 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:01:58.710681  8152 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:01:58.713837  8152 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:01:58.713887  8152 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:01:58.713903  8152 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:01:58.713917  8152 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:01:58.717767  8152 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:45029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:35187,127.5.152.61:33515,127.5.152.60:39943
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.61:33515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.61
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:01:58.718551  8152 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:01:58.719316  8152 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:01:58.723908  8161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:58.723912  8158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:58.723930  8159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:58.724223  8152 server_base.cc:1061] running on GCE node
I20260812 08:01:58.724728  8152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:01:58.725365  8152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:01:58.726538  8152 hybrid_clock.cc:648] HybridClock initialized: now 1786521718726517 us; error 34 us; skew 500 ppm
Aug 12 08:01:58 dist-test-slave-1nl6 krb5kdc[8004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:01:58.729064  8152 init.cc:377] Logged in from keytab as kudu/127.5.152.61@KRBTEST.COM (short username kudu)
I20260812 08:01:58.739809  8152 webserver.cc:533] Webserver started at https://127.5.152.61:37931/ using document root <none> and password file <none>
I20260812 08:01:58.740340  8152 fs_manager.cc:362] Metadata directory not provided
I20260812 08:01:58.740381  8152 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:01:58.740542  8152 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:01:58.741921  8152 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/data/instance:
uuid: "2ab0c2782d064ab6a2923caba793eacd"
format_stamp: "Formatted at 2026-08-12 08:01:58 on dist-test-slave-1nl6"
I20260812 08:01:58.742337  8152 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/wal/instance:
uuid: "2ab0c2782d064ab6a2923caba793eacd"
format_stamp: "Formatted at 2026-08-12 08:01:58 on dist-test-slave-1nl6"
I20260812 08:01:58.745409  8152 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20260812 08:01:58.747386  8172 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:58.748255  8152 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:01:58.748370  8152 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/wal
uuid: "2ab0c2782d064ab6a2923caba793eacd"
format_stamp: "Formatted at 2026-08-12 08:01:58 on dist-test-slave-1nl6"
I20260812 08:01:58.748453  8152 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:01:58.763701  8152 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:01:58.765882  8152 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:01:58.766171  8152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:01:58.773524  8152 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.61:33515
I20260812 08:01:58.773528  8246 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.61:33515 every 8 connection(s)
I20260812 08:01:58.774359  8152 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
I20260812 08:01:58.776866  8248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:01:58.779798  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 8152
I20260812 08:01:58.779906  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/wal/instance
I20260812 08:01:58.780059  8248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 08:01:58 dist-test-slave-1nl6 krb5kdc[8004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.152.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.60@KRBTEST.COM" created.
I20260812 08:01:58.792591  8248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 08:01:58 dist-test-slave-1nl6 krb5kdc[8004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:01:58.797245  8133 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } attempt: 3
I20260812 08:01:58.801602  8248 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 08:01:58 dist-test-slave-1nl6 krb5kdc[8004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
W20260812 08:01:58.802302  8173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.152.60:39943: connect: Connection refused (error 111)
W20260812 08:01:58.803480  8248 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:39943: Network error: Client connection negotiation failed: client connection to 127.5.152.60:39943: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:01:58.808328  8133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:01:58.809661  8133 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:39943: Network error: Client connection negotiation failed: client connection to 127.5.152.60:39943: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.5.152.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:01:58.831005  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.60:39943
--webserver_interface=127.5.152.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:45029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:35187,127.5.152.61:33515,127.5.152.60:39943
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 08:01:58.840862  8133 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } attempt: 1
W20260812 08:01:58.842703  8133 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:39943: Network error: Client connection negotiation failed: client connection to 127.5.152.60:39943: connect: Connection refused (error 111)
I20260812 08:01:58.852717  8248 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } attempt: 1
W20260812 08:01:58.854882  8248 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:39943: Network error: Client connection negotiation failed: client connection to 127.5.152.60:39943: connect: Connection refused (error 111)
I20260812 08:01:58.918059  8133 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } attempt: 2
W20260812 08:01:58.920238  8133 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:39943: Network error: Client connection negotiation failed: client connection to 127.5.152.60:39943: connect: Connection refused (error 111)
I20260812 08:01:58.923134  8248 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } attempt: 2
W20260812 08:01:58.924752  8248 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:39943: Network error: Client connection negotiation failed: client connection to 127.5.152.60:39943: connect: Connection refused (error 111)
W20260812 08:01:58.956779  8263 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:01:58.957109  8263 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:01:58.957247  8263 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:01:58.961053  8263 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:01:58.961212  8263 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:01:58.961283  8263 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:01:58.961345  8263 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:01:58.966198  8263 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:45029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:35187,127.5.152.61:33515,127.5.152.60:39943
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.60:39943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.60
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:01:58.967494  8263 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:01:58.968381  8263 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:01:58.973902  8271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:58.973892  8274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:58.973892  8272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:58.974339  8263 server_base.cc:1061] running on GCE node
I20260812 08:01:58.974838  8263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:01:58.975822  8263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:01:58.976981  8263 hybrid_clock.cc:648] HybridClock initialized: now 1786521718976964 us; error 30 us; skew 500 ppm
Aug 12 08:01:58 dist-test-slave-1nl6 krb5kdc[8004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:01:58.979856  8263 init.cc:377] Logged in from keytab as kudu/127.5.152.60@KRBTEST.COM (short username kudu)
I20260812 08:01:58.991660  8263 webserver.cc:533] Webserver started at https://127.5.152.60:40863/ using document root <none> and password file <none>
I20260812 08:01:58.992287  8263 fs_manager.cc:362] Metadata directory not provided
I20260812 08:01:58.992339  8263 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:01:58.992522  8263 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:01:58.994174  8263 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/data/instance:
uuid: "e9b01b5c2dc646d4a44dbf166000d858"
format_stamp: "Formatted at 2026-08-12 08:01:58 on dist-test-slave-1nl6"
I20260812 08:01:58.994598  8263 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/wal/instance:
uuid: "e9b01b5c2dc646d4a44dbf166000d858"
format_stamp: "Formatted at 2026-08-12 08:01:58 on dist-test-slave-1nl6"
I20260812 08:01:58.998061  8263 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20260812 08:01:59.000258  8287 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:59.001200  8263 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 08:01:59.001363  8263 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/wal
uuid: "e9b01b5c2dc646d4a44dbf166000d858"
format_stamp: "Formatted at 2026-08-12 08:01:58 on dist-test-slave-1nl6"
I20260812 08:01:59.001462  8263 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:01:59.012460  8263 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:01:59.015237  8263 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:01:59.015443  8263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:01:59.015961  8248 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } attempt: 3
W20260812 08:01:59.017529  8248 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:39943: Network error: Client connection negotiation failed: client connection to 127.5.152.60:39943: connect: Connection refused (error 111)
I20260812 08:01:59.019440  8133 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } attempt: 3
W20260812 08:01:59.020833  8133 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:39943: Network error: Client connection negotiation failed: client connection to 127.5.152.60:39943: connect: Connection refused (error 111)
I20260812 08:01:59.023778  8263 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.60:39943
I20260812 08:01:59.023801  8375 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.60:39943 every 8 connection(s)
I20260812 08:01:59.024746  8263 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
I20260812 08:01:59.027638  8376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:01:59.028209  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 8263
I20260812 08:01:59.028398  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/wal/instance
I20260812 08:01:59.032454  8376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 08:01:59 dist-test-slave-1nl6 krb5kdc[8004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
Aug 12 08:01:59 dist-test-slave-1nl6 krb5kdc[8004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
Aug 12 08:01:59 dist-test-slave-1nl6 krb5kdc[8004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:01:59.049180  8376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 08:01:59 dist-test-slave-1nl6 krb5kdc[8004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
Aug 12 08:01:59 dist-test-slave-1nl6 krb5kdc[8004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:01:59.061414  8376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 08:01:59 dist-test-slave-1nl6 krb5kdc[8004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:01:59.075845  8376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858: Bootstrap starting.
I20260812 08:01:59.078493  8376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858: Neither blocks nor log segments found. Creating new log.
I20260812 08:01:59.079483  8376 log.cc:826] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858: Log is configured to *not* fsync() on all Append() calls
I20260812 08:01:59.081764  8376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858: No bootstrap required, opened a new log
I20260812 08:01:59.084594  8376 raft_consensus.cc:359] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } } peers { permanent_uuid: "2ab0c2782d064ab6a2923caba793eacd" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } } peers { permanent_uuid: "e9b01b5c2dc646d4a44dbf166000d858" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } }
I20260812 08:01:59.084813  8376 raft_consensus.cc:385] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:01:59.084851  8376 raft_consensus.cc:740] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b01b5c2dc646d4a44dbf166000d858, State: Initialized, Role: FOLLOWER
I20260812 08:01:59.085250  8376 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } } peers { permanent_uuid: "2ab0c2782d064ab6a2923caba793eacd" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } } peers { permanent_uuid: "e9b01b5c2dc646d4a44dbf166000d858" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } }
I20260812 08:01:59.085779  8391 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } } peers { permanent_uuid: "2ab0c2782d064ab6a2923caba793eacd" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } } peers { permanent_uuid: "e9b01b5c2dc646d4a44dbf166000d858" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } } }
I20260812 08:01:59.085948  8391 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:59.086519  8376 sys_catalog.cc:565] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:01:59.091957  8263 master_runner.cc:187] Error getting master registration for 127.5.152.62:35187: OK, instance_id { permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" instance_seqno: 1786521718560902 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 08:01:59.094192  8263 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 08:01:59.094728  8408 catalog_manager.cc:1594] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:01:59.094957  8408 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:01:59.160720  8248 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } attempt: 4
Aug 12 08:01:59 dist-test-slave-1nl6 krb5kdc[8004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:01:59.172919  8248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd: Bootstrap starting.
I20260812 08:01:59.175784  8248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd: Neither blocks nor log segments found. Creating new log.
I20260812 08:01:59.176734  8248 log.cc:826] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd: Log is configured to *not* fsync() on all Append() calls
I20260812 08:01:59.178665  8248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd: No bootstrap required, opened a new log
I20260812 08:01:59.182046  8248 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } } peers { permanent_uuid: "2ab0c2782d064ab6a2923caba793eacd" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } } peers { permanent_uuid: "e9b01b5c2dc646d4a44dbf166000d858" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } }
I20260812 08:01:59.182267  8248 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:01:59.182325  8248 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab0c2782d064ab6a2923caba793eacd, State: Initialized, Role: FOLLOWER
I20260812 08:01:59.182783  8248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } } peers { permanent_uuid: "2ab0c2782d064ab6a2923caba793eacd" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } } peers { permanent_uuid: "e9b01b5c2dc646d4a44dbf166000d858" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } }
I20260812 08:01:59.183336  8412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } } peers { permanent_uuid: "2ab0c2782d064ab6a2923caba793eacd" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } } peers { permanent_uuid: "e9b01b5c2dc646d4a44dbf166000d858" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } } }
I20260812 08:01:59.183483  8412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:59.184116  8248 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:01:59.185017  8133 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } attempt: 4
Aug 12 08:01:59 dist-test-slave-1nl6 krb5kdc[8004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521718, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
W20260812 08:01:59.191965  8427 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:01:59.192054  8427 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:01:59.190282  8152 master_runner.cc:187] Error getting master registration for 127.5.152.62:35187: OK, instance_id { permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" instance_seqno: 1786521718560902 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 08:01:59.192960  8152 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 08:01:59.199150  8133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50: Bootstrap starting.
I20260812 08:01:59.201347  8133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50: Neither blocks nor log segments found. Creating new log.
I20260812 08:01:59.202157  8133 log.cc:826] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50: Log is configured to *not* fsync() on all Append() calls
I20260812 08:01:59.204104  8133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50: No bootstrap required, opened a new log
I20260812 08:01:59.207145  8133 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } } peers { permanent_uuid: "2ab0c2782d064ab6a2923caba793eacd" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } } peers { permanent_uuid: "e9b01b5c2dc646d4a44dbf166000d858" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } }
I20260812 08:01:59.207320  8133 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:01:59.207396  8133 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c905ecc2d934adeb0bc638a73de6c50, State: Initialized, Role: FOLLOWER
I20260812 08:01:59.207835  8133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } } peers { permanent_uuid: "2ab0c2782d064ab6a2923caba793eacd" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } } peers { permanent_uuid: "e9b01b5c2dc646d4a44dbf166000d858" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } }
I20260812 08:01:59.208318  8432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } } peers { permanent_uuid: "2ab0c2782d064ab6a2923caba793eacd" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } } peers { permanent_uuid: "e9b01b5c2dc646d4a44dbf166000d858" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } } }
I20260812 08:01:59.208482  8432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:59.209015  8133 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:01:59.218001  8024 master_runner.cc:264] No leader master found from master 0c905ecc2d934adeb0bc638a73de6c50
I20260812 08:01:59.218093  8024 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 08:01:59.218732  8449 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:01:59.218868  8449 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:01:59.244580  8391 raft_consensus.cc:493] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:01:59.244819  8391 raft_consensus.cc:515] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } } peers { permanent_uuid: "2ab0c2782d064ab6a2923caba793eacd" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } } peers { permanent_uuid: "e9b01b5c2dc646d4a44dbf166000d858" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } }
I20260812 08:01:59.245982  8391 leader_election.cc:290] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c905ecc2d934adeb0bc638a73de6c50 (127.5.152.62:35187), 2ab0c2782d064ab6a2923caba793eacd (127.5.152.61:33515)
I20260812 08:01:59.247126  8088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9b01b5c2dc646d4a44dbf166000d858" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c905ecc2d934adeb0bc638a73de6c50" is_pre_election: true
I20260812 08:01:59.247493  8088 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9b01b5c2dc646d4a44dbf166000d858 in term 0.
I20260812 08:01:59.247977  8215 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9b01b5c2dc646d4a44dbf166000d858" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab0c2782d064ab6a2923caba793eacd" is_pre_election: true
I20260812 08:01:59.248262  8215 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9b01b5c2dc646d4a44dbf166000d858 in term 0.
I20260812 08:01:59.249253  8292 leader_election.cc:304] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c905ecc2d934adeb0bc638a73de6c50, e9b01b5c2dc646d4a44dbf166000d858; no voters: 
I20260812 08:01:59.249943  8391 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:01:59.250030  8391 raft_consensus.cc:493] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:01:59.250072  8391 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:01:59.252285  8391 raft_consensus.cc:515] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } } peers { permanent_uuid: "2ab0c2782d064ab6a2923caba793eacd" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } } peers { permanent_uuid: "e9b01b5c2dc646d4a44dbf166000d858" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } }
I20260812 08:01:59.253985  8088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9b01b5c2dc646d4a44dbf166000d858" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c905ecc2d934adeb0bc638a73de6c50"
I20260812 08:01:59.254204  8088 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:01:59.256002  8088 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9b01b5c2dc646d4a44dbf166000d858 in term 1.
I20260812 08:01:59.256835  8292 leader_election.cc:304] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c905ecc2d934adeb0bc638a73de6c50, e9b01b5c2dc646d4a44dbf166000d858; no voters: 
I20260812 08:01:59.258466  8215 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9b01b5c2dc646d4a44dbf166000d858" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab0c2782d064ab6a2923caba793eacd"
I20260812 08:01:59.258679  8215 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:01:59.260264  8215 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9b01b5c2dc646d4a44dbf166000d858 in term 1.
I20260812 08:01:59.260998  8391 leader_election.cc:290] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [CANDIDATE]: Term 1 election: Requested vote from peers 0c905ecc2d934adeb0bc638a73de6c50 (127.5.152.62:35187), 2ab0c2782d064ab6a2923caba793eacd (127.5.152.61:33515)
I20260812 08:01:59.261317  8391 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:01:59.261469  8391 raft_consensus.cc:697] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [term 1 LEADER]: Becoming Leader. State: Replica: e9b01b5c2dc646d4a44dbf166000d858, State: Running, Role: LEADER
I20260812 08:01:59.261883  8391 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } } peers { permanent_uuid: "2ab0c2782d064ab6a2923caba793eacd" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } } peers { permanent_uuid: "e9b01b5c2dc646d4a44dbf166000d858" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } }
I20260812 08:01:59.264217  8391 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9b01b5c2dc646d4a44dbf166000d858. Latest consensus state: current_term: 1 leader_uuid: "e9b01b5c2dc646d4a44dbf166000d858" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } } peers { permanent_uuid: "2ab0c2782d064ab6a2923caba793eacd" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } } peers { permanent_uuid: "e9b01b5c2dc646d4a44dbf166000d858" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } } }
I20260812 08:01:59.264465  8391 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [sys.catalog]: This master's current role is: LEADER
I20260812 08:01:59.265060  8455 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:01:59.266255  8455 catalog_manager.cc:1520] Initializing Kudu cluster ID...
W20260812 08:01:59.266700  8298 tablet.cc:2406] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 08:01:59.267882  8181 tablet.cc:2406] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:01:59.273942  8215 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd [term 1 FOLLOWER]: Refusing update from remote peer e9b01b5c2dc646d4a44dbf166000d858: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:01:59.274940  8453 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab0c2782d064ab6a2923caba793eacd" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:01:59.279739  8412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9b01b5c2dc646d4a44dbf166000d858. Latest consensus state: current_term: 1 leader_uuid: "e9b01b5c2dc646d4a44dbf166000d858" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } } peers { permanent_uuid: "2ab0c2782d064ab6a2923caba793eacd" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } } peers { permanent_uuid: "e9b01b5c2dc646d4a44dbf166000d858" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } } }
I20260812 08:01:59.279915  8412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:59.281543  8088 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50 [term 1 FOLLOWER]: Refusing update from remote peer e9b01b5c2dc646d4a44dbf166000d858: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:01:59.282428  8453 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:01:59.288355  8432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9b01b5c2dc646d4a44dbf166000d858. Latest consensus state: current_term: 1 leader_uuid: "e9b01b5c2dc646d4a44dbf166000d858" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } } peers { permanent_uuid: "2ab0c2782d064ab6a2923caba793eacd" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } } peers { permanent_uuid: "e9b01b5c2dc646d4a44dbf166000d858" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } } }
I20260812 08:01:59.288535  8432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:59.290462  8453 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9b01b5c2dc646d4a44dbf166000d858" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } } peers { permanent_uuid: "2ab0c2782d064ab6a2923caba793eacd" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } } peers { permanent_uuid: "e9b01b5c2dc646d4a44dbf166000d858" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } } }
I20260812 08:01:59.290656  8453 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [sys.catalog]: This master's current role is: LEADER
I20260812 08:01:59.292008  8412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9b01b5c2dc646d4a44dbf166000d858" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } } peers { permanent_uuid: "2ab0c2782d064ab6a2923caba793eacd" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } } peers { permanent_uuid: "e9b01b5c2dc646d4a44dbf166000d858" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } } }
I20260812 08:01:59.292207  8412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:59.299270  8453 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9b01b5c2dc646d4a44dbf166000d858" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } } peers { permanent_uuid: "2ab0c2782d064ab6a2923caba793eacd" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } } peers { permanent_uuid: "e9b01b5c2dc646d4a44dbf166000d858" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } } }
I20260812 08:01:59.299461  8453 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858 [sys.catalog]: This master's current role is: LEADER
I20260812 08:01:59.301002  8432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9b01b5c2dc646d4a44dbf166000d858" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c905ecc2d934adeb0bc638a73de6c50" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35187 } } peers { permanent_uuid: "2ab0c2782d064ab6a2923caba793eacd" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 33515 } } peers { permanent_uuid: "e9b01b5c2dc646d4a44dbf166000d858" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39943 } } }
I20260812 08:01:59.301151  8432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:01:59.302579  8455 catalog_manager.cc:1383] Generated new cluster ID: a0915cf0e842401da5abfdf48a9fd73a
I20260812 08:01:59.302675  8455 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:01:59.331920  8455 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:01:59.333381  8455 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:01:59.352681  8455 catalog_manager.cc:6092] T 00000000000000000000000000000000 P e9b01b5c2dc646d4a44dbf166000d858: Generated new TSK 0
I20260812 08:01:59.353730  8455 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:01:59.430672  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.152.1:0
--local_ip_for_outbound_sockets=127.5.152.1
--webserver_interface=127.5.152.1
--webserver_port=0
--tserver_master_addrs=127.5.152.62:35187,127.5.152.61:33515,127.5.152.60:39943
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:45029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 08:01:59.580446  8479 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:01:59.580675  8479 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:01:59.580790  8479 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:01:59.584337  8479 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:01:59.584533  8479 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.152.1
I20260812 08:01:59.589202  8479 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:45029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.1
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--tserver_master_addrs=127.5.152.62:35187,127.5.152.61:33515,127.5.152.60:39943
--never_fsync=true
--heap_profile_path=/tmp/kudu.8479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.152.1

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:01:59.590363  8479 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:01:59.591269  8479 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:01:59.598057  8487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:59.598042  8486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:01:59.598042  8489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:01:59.598265  8479 server_base.cc:1061] running on GCE node
I20260812 08:01:59.598636  8479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:01:59.599346  8479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:01:59.600553  8479 hybrid_clock.cc:648] HybridClock initialized: now 1786521719600535 us; error 38 us; skew 500 ppm
Aug 12 08:01:59 dist-test-slave-1nl6 krb5kdc[8004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521719, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:01:59.604420  8479 init.cc:377] Logged in from keytab as kudu/127.5.152.1@KRBTEST.COM (short username kudu)
I20260812 08:01:59.622427  8479 webserver.cc:533] Webserver started at https://127.5.152.1:43745/ using document root <none> and password file <none>
I20260812 08:01:59.624117  8479 fs_manager.cc:362] Metadata directory not provided
I20260812 08:01:59.624243  8479 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:01:59.624533  8479 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:01:59.626796  8479 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/data/instance:
uuid: "657758d40d714c28846cc27d196256da"
format_stamp: "Formatted at 2026-08-12 08:01:59 on dist-test-slave-1nl6"
I20260812 08:01:59.627552  8479 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/wal/instance:
uuid: "657758d40d714c28846cc27d196256da"
format_stamp: "Formatted at 2026-08-12 08:01:59 on dist-test-slave-1nl6"
I20260812 08:01:59.632287  8479 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.000s
I20260812 08:01:59.635927  8508 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:59.637519  8479 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260812 08:01:59.637739  8479 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/wal
uuid: "657758d40d714c28846cc27d196256da"
format_stamp: "Formatted at 2026-08-12 08:01:59 on dist-test-slave-1nl6"
I20260812 08:01:59.637894  8479 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:01:59.657898  8479 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:01:59.662256  8479 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:01:59.662750  8479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:01:59.663694  8479 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:01:59.665292  8479 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:01:59.665397  8479 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:59.665479  8479 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:01:59.665527  8479 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:01:59.678745  8479 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.1:34019
I20260812 08:01:59.678767  8689 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.1:34019 every 8 connection(s)
I20260812 08:01:59.680089  8479 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
I20260812 08:01:59.681157  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 8479
I20260812 08:01:59.681311  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/wal/instance
Aug 12 08:01:59 dist-test-slave-1nl6 krb5kdc[8004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521719, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
Aug 12 08:01:59 dist-test-slave-1nl6 krb5kdc[8004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521719, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
Aug 12 08:01:59 dist-test-slave-1nl6 krb5kdc[8004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521719, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
I20260812 08:01:59.701296  8690 heartbeater.cc:344] Connected to a master server at 127.5.152.60:39943
I20260812 08:01:59.701350  8692 heartbeater.cc:344] Connected to a master server at 127.5.152.62:35187
I20260812 08:01:59.701591  8692 heartbeater.cc:461] Registering TS with master...
I20260812 08:01:59.701594  8690 heartbeater.cc:461] Registering TS with master...
I20260812 08:01:59.702195  8692 heartbeater.cc:507] Master 127.5.152.62:35187 requested a full tablet report, sending...
I20260812 08:01:59.702196  8690 heartbeater.cc:507] Master 127.5.152.60:39943 requested a full tablet report, sending...
I20260812 08:01:59.701359  8693 heartbeater.cc:344] Connected to a master server at 127.5.152.61:33515
I20260812 08:01:59.702939  8693 heartbeater.cc:461] Registering TS with master...
I20260812 08:01:59.703143  8693 heartbeater.cc:507] Master 127.5.152.61:33515 requested a full tablet report, sending...
I20260812 08:01:59.703964  8072 ts_manager.cc:194] Registered new tserver with Master: 657758d40d714c28846cc27d196256da (127.5.152.1:34019)
I20260812 08:01:59.703964  8319 ts_manager.cc:194] Registered new tserver with Master: 657758d40d714c28846cc27d196256da (127.5.152.1:34019)
I20260812 08:01:59.704294  8197 ts_manager.cc:194] Registered new tserver with Master: 657758d40d714c28846cc27d196256da (127.5.152.1:34019)
I20260812 08:01:59.705476  8319 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.152.1@KRBTEST.COM'} at 127.5.152.1:35273
I20260812 08:01:59.707170  5728 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
time=2026-08-12T08:01:59.785Z level=INFO source=main.go:1678 msg="updated GOGC" old=100 new=75
time=2026-08-12T08:01:59.789Z level=INFO source=main.go:744 msg="Leaving GOMAXPROCS=4: CPU quota undefined" component=automaxprocs
time=2026-08-12T08:01:59.790Z level=INFO source=memlimit.go:198 msg="GOMEMLIMIT is updated" component=automemlimit package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=14142549196 previous=9223372036854775807
time=2026-08-12T08:01:59.790Z level=INFO source=main.go:851 msg="Starting Prometheus Server" mode=server version="(version=3.11.2, branch=HEAD, revision=f0f0fdd679dcd6df320b0558b20919f7cd44c407)"
time=2026-08-12T08:01:59.790Z level=INFO source=main.go:856 msg="operational information" build_context="(go=go1.26.2, platform=linux/amd64, user=root@d2684485c347, date=20260413-11:53:40, tags=netgo,builtinassets)" host_details="(Linux 5.4.0-1157-gcp #166~18.04.1-Ubuntu SMP Wed Jan 21 01:57:46 UTC 2026 x86_64 dist-test-slave-1nl6 (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)"
time=2026-08-12T08:02:00.059Z level=INFO source=web.go:710 msg="Start listening for connections" component=web address=127.0.0.1:0
time=2026-08-12T08:02:00.060Z level=INFO source=main.go:1410 msg="Starting TSDB ..."
time=2026-08-12T08:02:00.064Z level=INFO source=tls_config.go:354 msg="Listening on" component=web address=127.0.0.1:38627
time=2026-08-12T08:02:00.064Z level=INFO source=tls_config.go:357 msg="TLS is disabled." component=web http2=false address=127.0.0.1:38627
time=2026-08-12T08:02:00.064Z level=INFO source=head.go:698 msg="Replaying on-disk memory mappable chunks if any" component=tsdb
time=2026-08-12T08:02:00.064Z level=INFO source=head.go:784 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=2.21µs
time=2026-08-12T08:02:00.064Z level=INFO source=head.go:792 msg="Replaying WAL, this may take a while" component=tsdb
time=2026-08-12T08:02:00.065Z level=INFO source=head.go:865 msg="WAL segment loaded" component=tsdb segment=0 maxSegment=0 duration=385.295µs
time=2026-08-12T08:02:00.065Z level=INFO source=head.go:902 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=74.723µs wal_replay_duration=446.895µs wbl_replay_duration=177ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=2.21µs total_replay_duration=585.429µs
time=2026-08-12T08:02:00.067Z level=INFO source=main.go:1431 msg="filesystem information" fs_type=OVERLAYFS_SUPER_MAGIC
time=2026-08-12T08:02:00.067Z level=INFO source=main.go:1434 msg="TSDB started"
time=2026-08-12T08:02:00.067Z level=INFO source=main.go:1632 msg="Loading configuration file" filename=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/prometheus.yml
time=2026-08-12T08:02:00.067Z level=INFO source=main.go:1048 msg="TSDB retention updated" duration=15d size=0B percentage=0
time=2026-08-12T08:02:00.073Z level=INFO source=main.go:1671 msg="Completed loading of configuration file" db_storage=50.198µs remote_storage=2.147µs web_handler=778ns query_engine=1.52µs scrape=5.478996ms scrape_sd=115.081µs notify=5.482µs notify_sd=2.348µs rules=3.757µs tracing=18.398µs filename=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786521680053065-5728-0/prometheus.yml totalDuration=6.169142ms
time=2026-08-12T08:02:00.073Z level=INFO source=main.go:1395 msg="Server is ready to receive web requests."
time=2026-08-12T08:02:00.073Z level=INFO source=manager.go:209 msg="Starting rule manager..." component="rule manager"
I20260812 08:02:00.193440  8427 catalog_manager.cc:1295] Loaded cluster ID: a0915cf0e842401da5abfdf48a9fd73a
I20260812 08:02:00.193581  8427 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd: loading cluster ID for follower catalog manager: success
I20260812 08:02:00.195488  8427 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd: acquiring CA information for follower catalog manager: success
I20260812 08:02:00.196643  8427 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 2ab0c2782d064ab6a2923caba793eacd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 08:02:00.220834  8449 catalog_manager.cc:1295] Loaded cluster ID: a0915cf0e842401da5abfdf48a9fd73a
I20260812 08:02:00.221050  8449 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50: loading cluster ID for follower catalog manager: success
I20260812 08:02:00.223555  8449 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50: acquiring CA information for follower catalog manager: success
I20260812 08:02:00.225180  8449 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 0c905ecc2d934adeb0bc638a73de6c50: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 08:02:00.707387  8690 heartbeater.cc:499] Master 127.5.152.60:39943 was elected leader, sending a full tablet report...
W20260812 08:02:05.729746  8719 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:53572: invalid bearer token
W20260812 08:02:06.071888  8720 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:40560: invalid bearer token
W20260812 08:02:06.624598  8722 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:36774: invalid bearer token
W20260812 08:02:06.805759  8723 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:41144: invalid bearer token
time=2026-08-12T08:02:07.057Z level=WARN source=main.go:1191 msg="Received an OS signal, exiting gracefully..." signal=terminated
time=2026-08-12T08:02:07.057Z level=INFO source=main.go:1216 msg="Stopping scrape discovery manager..."
time=2026-08-12T08:02:07.058Z level=INFO source=main.go:1230 msg="Stopping notify discovery manager..."
time=2026-08-12T08:02:07.058Z level=INFO source=main.go:1245 msg="Stopping rule manager manager..."
time=2026-08-12T08:02:07.058Z level=INFO source=manager.go:223 msg="Stopping rule manager..." component="rule manager"
time=2026-08-12T08:02:07.058Z level=INFO source=manager.go:239 msg="Rule manager stopped" component="rule manager"
time=2026-08-12T08:02:07.058Z level=INFO source=main.go:1269 msg="Stopping scrape manager..."
time=2026-08-12T08:02:07.058Z level=INFO source=main.go:1226 msg="Notify discovery manager stopped"
time=2026-08-12T08:02:07.058Z level=INFO source=main.go:1241 msg="Rule manager stopped"
time=2026-08-12T08:02:07.058Z level=INFO source=main.go:1284 msg="Stopping tracing manager..."
time=2026-08-12T08:02:07.058Z level=INFO source=main.go:1369 msg="Stopping reloaders..."
time=2026-08-12T08:02:07.058Z level=INFO source=main.go:1458 msg="Stopping storage..."
time=2026-08-12T08:02:07.058Z level=INFO source=main.go:1261 msg="Scrape manager stopped"
time=2026-08-12T08:02:07.058Z level=INFO source=main.go:1280 msg="Tracing manager stopped"
time=2026-08-12T08:02:07.058Z level=INFO source=main.go:1361 msg="Reloaders stopped"
time=2026-08-12T08:02:07.058Z level=INFO source=main.go:1212 msg="Scrape discovery manager stopped"
time=2026-08-12T08:02:07.063Z level=INFO source=main.go:1536 msg="Stopping web handler..."
time=2026-08-12T08:02:07.063Z level=INFO source=main.go:1559 msg="Stopping notifier manager..."
time=2026-08-12T08:02:07.063Z level=INFO source=manager.go:321 msg="Stopping notification manager..." component=notifier
time=2026-08-12T08:02:07.063Z level=INFO source=main.go:1454 msg="TSDB stopped"
time=2026-08-12T08:02:07.063Z level=INFO source=main.go:1555 msg="Notifier manager stopped"
time=2026-08-12T08:02:07.063Z level=INFO source=main.go:1532 msg="Web handler stopped"
time=2026-08-12T08:02:07.063Z level=INFO source=main.go:1570 msg="See you next time!"
I20260812 08:02:07.078387  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 8479
I20260812 08:02:07.101008  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 8024
I20260812 08:02:07.116851  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 8152
I20260812 08:02:07.134081  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 8263
2026-08-12T08:02:07Z chronyd exiting
[       OK ] PrometheusTokenITest.TestWrongTokenScrapeFails (11255 ms)
[ RUN      ] PrometheusTokenITest.TestMissingTokenScrapeFails
Loading random data
Initializing database '/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 08:02:07 dist-test-slave-1nl6 krb5kdc[8728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 08:02:07 dist-test-slave-1nl6 krb5kdc[8728](info): set up 2 sockets
Aug 12 08:02:07 dist-test-slave-1nl6 krb5kdc[8728](info): commencing operation
krb5kdc: starting...
W20260812 08:02:09.424870  5728 mini_kdc.cc:121] Time spent starting KDC: real 2.259s	user 0.001s	sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/test-user.keytab.
Aug 12 08:02:09 dist-test-slave-1nl6 krb5kdc[8728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM: 
2026-08-12T08:02:09Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T08:02:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:02:09.575826  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.62:35051
--webserver_interface=127.5.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:35813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:35051,127.5.152.61:39973,127.5.152.60:34749
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 08:02:09.685374  8751 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:09.685647  8751 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:09.685752  8751 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:09.689395  8751 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:09.689510  8751 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:09.689564  8751 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:09.689601  8751 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:09.694434  8751 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:35813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:35051,127.5.152.61:39973,127.5.152.60:34749
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.62:35051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.62
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:09.695573  8751 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:09.696780  8751 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:09.704502  8761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:09.704562  8758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:09.704586  8751 server_base.cc:1061] running on GCE node
W20260812 08:02:09.704522  8759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:09.705185  8751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:09.706075  8751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:09.707233  8751 hybrid_clock.cc:648] HybridClock initialized: now 1786521729707212 us; error 34 us; skew 500 ppm
Aug 12 08:02:09 dist-test-slave-1nl6 krb5kdc[8728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521729, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:02:09.709971  8751 init.cc:377] Logged in from keytab as kudu/127.5.152.62@KRBTEST.COM (short username kudu)
I20260812 08:02:09.721442  8751 webserver.cc:533] Webserver started at https://127.5.152.62:33223/ using document root <none> and password file <none>
I20260812 08:02:09.722038  8751 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:09.722085  8751 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:09.722249  8751 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:09.723820  8751 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/data/instance:
uuid: "0d9e2298957149f6bcc9faf65863dd39"
format_stamp: "Formatted at 2026-08-12 08:02:09 on dist-test-slave-1nl6"
I20260812 08:02:09.724241  8751 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/wal/instance:
uuid: "0d9e2298957149f6bcc9faf65863dd39"
format_stamp: "Formatted at 2026-08-12 08:02:09 on dist-test-slave-1nl6"
I20260812 08:02:09.727432  8751 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 08:02:09.729465  8772 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:09.730430  8751 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:02:09.730580  8751 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/wal
uuid: "0d9e2298957149f6bcc9faf65863dd39"
format_stamp: "Formatted at 2026-08-12 08:02:09 on dist-test-slave-1nl6"
I20260812 08:02:09.730688  8751 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:09.741369  8751 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:09.743768  8751 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:09.743971  8751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:09.751330  8751 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.62:35051
I20260812 08:02:09.751335  8863 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.62:35051 every 8 connection(s)
I20260812 08:02:09.752281  8751 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
I20260812 08:02:09.755064  8865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:09.758349  8865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:09.761322  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 8751
I20260812 08:02:09.761464  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-0/wal/instance
Aug 12 08:02:09 dist-test-slave-1nl6 krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521729, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.152.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.61@KRBTEST.COM" created.
I20260812 08:02:09.773582  8865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:02:09.774411  8776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.152.61:39973: connect: Connection refused (error 111)
W20260812 08:02:09.775314  8865 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:39973: Network error: Client connection negotiation failed: client connection to 127.5.152.61:39973: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:02:09.815788  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.61:39973
--webserver_interface=127.5.152.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:35813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:35051,127.5.152.61:39973,127.5.152.60:34749
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 08:02:09.824581  8865 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } attempt: 1
W20260812 08:02:09.826066  8865 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:39973: Network error: Client connection negotiation failed: client connection to 127.5.152.61:39973: connect: Connection refused (error 111)
I20260812 08:02:09.894328  8865 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } attempt: 2
W20260812 08:02:09.896219  8865 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:39973: Network error: Client connection negotiation failed: client connection to 127.5.152.61:39973: connect: Connection refused (error 111)
W20260812 08:02:09.922429  8878 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:09.922660  8878 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:09.922827  8878 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:09.926602  8878 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:09.926671  8878 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:09.926736  8878 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:09.926787  8878 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:09.931671  8878 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:35813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:35051,127.5.152.61:39973,127.5.152.60:34749
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.61:39973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.61
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:09.932765  8878 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:09.933676  8878 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:09.939437  8890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:09.939441  8887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:09.939441  8884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:09.939929  8878 server_base.cc:1061] running on GCE node
I20260812 08:02:09.940423  8878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:09.941298  8878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:09.942498  8878 hybrid_clock.cc:648] HybridClock initialized: now 1786521729942475 us; error 41 us; skew 500 ppm
Aug 12 08:02:09 dist-test-slave-1nl6 krb5kdc[8728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521729, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:02:09.945230  8878 init.cc:377] Logged in from keytab as kudu/127.5.152.61@KRBTEST.COM (short username kudu)
I20260812 08:02:09.957079  8878 webserver.cc:533] Webserver started at https://127.5.152.61:41393/ using document root <none> and password file <none>
I20260812 08:02:09.957657  8878 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:09.957710  8878 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:09.957890  8878 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:09.959571  8878 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/data/instance:
uuid: "a11a66f3ad8948c58caa60e9fd0197b8"
format_stamp: "Formatted at 2026-08-12 08:02:09 on dist-test-slave-1nl6"
I20260812 08:02:09.960001  8878 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/wal/instance:
uuid: "a11a66f3ad8948c58caa60e9fd0197b8"
format_stamp: "Formatted at 2026-08-12 08:02:09 on dist-test-slave-1nl6"
I20260812 08:02:09.963410  8878 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.001s
I20260812 08:02:09.965382  8900 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:09.966392  8878 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:02:09.966519  8878 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/wal
uuid: "a11a66f3ad8948c58caa60e9fd0197b8"
format_stamp: "Formatted at 2026-08-12 08:02:09 on dist-test-slave-1nl6"
I20260812 08:02:09.966619  8878 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:09.987524  8865 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } attempt: 3
W20260812 08:02:09.989464  8865 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:39973: Network error: Client connection negotiation failed: client connection to 127.5.152.61:39973: connect: Connection refused (error 111)
I20260812 08:02:09.997176  8878 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:10.000399  8878 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:10.000605  8878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:10.008615  8878 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.61:39973
I20260812 08:02:10.008612  8980 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.61:39973 every 8 connection(s)
I20260812 08:02:10.009574  8878 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
I20260812 08:02:10.011801  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 8878
I20260812 08:02:10.011909  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-1/wal/instance
I20260812 08:02:10.012431  8982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:10.015969  8982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.152.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.60@KRBTEST.COM" created.
Aug 12 08:02:10 dist-test-slave-1nl6 krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521729, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
I20260812 08:02:10.030951  8982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 08:02:10 dist-test-slave-1nl6 krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521729, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:02:10.045053  8982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:02:10.045856  8904 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.152.60:34749: connect: Connection refused (error 111)
W20260812 08:02:10.046835  8982 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:34749: Network error: Client connection negotiation failed: client connection to 127.5.152.60:34749: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.5.152.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:02:10.065989  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.60:34749
--webserver_interface=127.5.152.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:35813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:35051,127.5.152.61:39973,127.5.152.60:34749
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 08:02:10.096138  8982 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } attempt: 1
W20260812 08:02:10.097719  8982 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:34749: Network error: Client connection negotiation failed: client connection to 127.5.152.60:34749: connect: Connection refused (error 111)
I20260812 08:02:10.132694  8865 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } attempt: 4
Aug 12 08:02:10 dist-test-slave-1nl6 krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521729, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:02:10.143378  8865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:02:10.145301  8865 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:34749: Network error: Client connection negotiation failed: client connection to 127.5.152.60:34749: connect: Connection refused (error 111)
I20260812 08:02:10.165927  8982 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } attempt: 2
W20260812 08:02:10.168156  8982 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:34749: Network error: Client connection negotiation failed: client connection to 127.5.152.60:34749: connect: Connection refused (error 111)
W20260812 08:02:10.181133  8998 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:10.181422  8998 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:10.181548  8998 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:10.186074  8998 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:10.186184  8998 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:10.186209  8998 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:10.186290  8998 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:10.191478  8998 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:35813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:35051,127.5.152.61:39973,127.5.152.60:34749
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.60:34749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.60
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:10.192626  8998 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:10.193531  8998 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:10.199024  9007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:10.199048  9006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:10.199214  8998 server_base.cc:1061] running on GCE node
W20260812 08:02:10.199035  9011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:10.199816  8998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:10.200641  8998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:10.201805  8998 hybrid_clock.cc:648] HybridClock initialized: now 1786521730201788 us; error 29 us; skew 500 ppm
Aug 12 08:02:10 dist-test-slave-1nl6 krb5kdc[8728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521730, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:02:10.204622  8865 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } attempt: 1
I20260812 08:02:10.204959  8998 init.cc:377] Logged in from keytab as kudu/127.5.152.60@KRBTEST.COM (short username kudu)
W20260812 08:02:10.206250  8865 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:34749: Network error: Client connection negotiation failed: client connection to 127.5.152.60:34749: connect: Connection refused (error 111)
I20260812 08:02:10.217656  8998 webserver.cc:533] Webserver started at https://127.5.152.60:34087/ using document root <none> and password file <none>
I20260812 08:02:10.218339  8998 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:10.218436  8998 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:10.218699  8998 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:10.220588  8998 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/data/instance:
uuid: "7c3c021feb3340688f89e0c1b67ec2fc"
format_stamp: "Formatted at 2026-08-12 08:02:10 on dist-test-slave-1nl6"
I20260812 08:02:10.221149  8998 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/wal/instance:
uuid: "7c3c021feb3340688f89e0c1b67ec2fc"
format_stamp: "Formatted at 2026-08-12 08:02:10 on dist-test-slave-1nl6"
I20260812 08:02:10.224939  8998 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20260812 08:02:10.227236  9027 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:10.228255  8998 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.001s
I20260812 08:02:10.228434  8998 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/wal
uuid: "7c3c021feb3340688f89e0c1b67ec2fc"
format_stamp: "Formatted at 2026-08-12 08:02:10 on dist-test-slave-1nl6"
I20260812 08:02:10.228547  8998 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:10.248692  8998 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:10.251084  8998 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:10.251273  8998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:10.259117  8998 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.60:34749
I20260812 08:02:10.259124  9117 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.60:34749 every 8 connection(s)
I20260812 08:02:10.259341  8982 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } attempt: 3
I20260812 08:02:10.260368  8998 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
I20260812 08:02:10.263761  9120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:10.266275  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 8998
I20260812 08:02:10.266431  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/master-2/wal/instance
Aug 12 08:02:10 dist-test-slave-1nl6 krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521729, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:02:10.267372  9120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:10.273442  8865 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } attempt: 2
Aug 12 08:02:10 dist-test-slave-1nl6 krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521730, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
Aug 12 08:02:10 dist-test-slave-1nl6 krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
I20260812 08:02:10.277719  8982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8: Bootstrap starting.
I20260812 08:02:10.280323  9120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:10.280995  8982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:10.281970  8982 log.cc:826] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8: Log is configured to *not* fsync() on all Append() calls
Aug 12 08:02:10 dist-test-slave-1nl6 krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:02:10.284442  8982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8: No bootstrap required, opened a new log
Aug 12 08:02:10 dist-test-slave-1nl6 krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521730, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:02:10.287688  8982 raft_consensus.cc:359] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } } peers { permanent_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } } peers { permanent_uuid: "7c3c021feb3340688f89e0c1b67ec2fc" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } }
I20260812 08:02:10.287906  8982 raft_consensus.cc:385] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:10.287962  8982 raft_consensus.cc:740] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a11a66f3ad8948c58caa60e9fd0197b8, State: Initialized, Role: FOLLOWER
I20260812 08:02:10.288481  8982 consensus_queue.cc:260] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } } peers { permanent_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } } peers { permanent_uuid: "7c3c021feb3340688f89e0c1b67ec2fc" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } }
I20260812 08:02:10.289762  8982 sys_catalog.cc:565] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 08:02:10 dist-test-slave-1nl6 krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521729, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:02:10.291587  9136 sys_catalog.cc:455] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } } peers { permanent_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } } peers { permanent_uuid: "7c3c021feb3340688f89e0c1b67ec2fc" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } } }
I20260812 08:02:10.291855  9136 sys_catalog.cc:458] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:10.292558  9120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:02:10.295128  9157 catalog_manager.cc:1594] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:02:10.295262  9157 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 12 08:02:10 dist-test-slave-1nl6 krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:02:10.294674  8878 master_runner.cc:187] Error getting master registration for 127.5.152.62:35051: OK, instance_id { permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" instance_seqno: 1786521729749303 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 08:02:10.297547  8878 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Aug 12 08:02:10 dist-test-slave-1nl6 krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521730, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:02:10.305303  8865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39: Bootstrap starting.
I20260812 08:02:10.305302  9120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc: Bootstrap starting.
I20260812 08:02:10.307878  9120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:10.308254  8865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:10.308801  9120 log.cc:826] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:10.309073  8865 log.cc:826] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:10.310623  8865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39: No bootstrap required, opened a new log
I20260812 08:02:10.312244  9120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc: No bootstrap required, opened a new log
I20260812 08:02:10.313536  9136 raft_consensus.cc:493] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:10.313700  9136 raft_consensus.cc:515] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } } peers { permanent_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } } peers { permanent_uuid: "7c3c021feb3340688f89e0c1b67ec2fc" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } }
I20260812 08:02:10.313834  8865 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } } peers { permanent_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } } peers { permanent_uuid: "7c3c021feb3340688f89e0c1b67ec2fc" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } }
I20260812 08:02:10.314013  8865 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:10.314062  8865 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9e2298957149f6bcc9faf65863dd39, State: Initialized, Role: FOLLOWER
I20260812 08:02:10.314494  8865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } } peers { permanent_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } } peers { permanent_uuid: "7c3c021feb3340688f89e0c1b67ec2fc" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } }
I20260812 08:02:10.314594  9136 leader_election.cc:290] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d9e2298957149f6bcc9faf65863dd39 (127.5.152.62:35051), 7c3c021feb3340688f89e0c1b67ec2fc (127.5.152.60:34749)
I20260812 08:02:10.315245  9120 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } } peers { permanent_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } } peers { permanent_uuid: "7c3c021feb3340688f89e0c1b67ec2fc" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } }
I20260812 08:02:10.315436  9120 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:10.315497  9120 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c3c021feb3340688f89e0c1b67ec2fc, State: Initialized, Role: FOLLOWER
I20260812 08:02:10.315873  9120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } } peers { permanent_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } } peers { permanent_uuid: "7c3c021feb3340688f89e0c1b67ec2fc" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } }
I20260812 08:02:10.316690  8808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9e2298957149f6bcc9faf65863dd39" is_pre_election: true
I20260812 08:02:10.316912  9120 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:02:10.316964  8808 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a11a66f3ad8948c58caa60e9fd0197b8 in term 0.
I20260812 08:02:10.317006  9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c3c021feb3340688f89e0c1b67ec2fc" is_pre_election: true
I20260812 08:02:10.317241  9068 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a11a66f3ad8948c58caa60e9fd0197b8 in term 0.
I20260812 08:02:10.317394  8904 leader_election.cc:304] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9e2298957149f6bcc9faf65863dd39, a11a66f3ad8948c58caa60e9fd0197b8; no voters: 
I20260812 08:02:10.317860  9136 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:02:10.317970  9136 raft_consensus.cc:493] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:10.318055  9136 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:10.319367  9136 raft_consensus.cc:515] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } } peers { permanent_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } } peers { permanent_uuid: "7c3c021feb3340688f89e0c1b67ec2fc" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } }
I20260812 08:02:10.319759  9136 leader_election.cc:290] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [CANDIDATE]: Term 1 election: Requested vote from peers 0d9e2298957149f6bcc9faf65863dd39 (127.5.152.62:35051), 7c3c021feb3340688f89e0c1b67ec2fc (127.5.152.60:34749)
I20260812 08:02:10.320299  9068 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c3c021feb3340688f89e0c1b67ec2fc"
I20260812 08:02:10.320310  8808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d9e2298957149f6bcc9faf65863dd39"
I20260812 08:02:10.320425  8808 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:10.320425  9068 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:10.321554  8808 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a11a66f3ad8948c58caa60e9fd0197b8 in term 1.
I20260812 08:02:10.321554  9068 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a11a66f3ad8948c58caa60e9fd0197b8 in term 1.
I20260812 08:02:10.316639  8865 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:02:10.321837  8904 leader_election.cc:304] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9e2298957149f6bcc9faf65863dd39, a11a66f3ad8948c58caa60e9fd0197b8; no voters: 
I20260812 08:02:10.321990  9136 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:10.322203  9136 raft_consensus.cc:697] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [term 1 LEADER]: Becoming Leader. State: Replica: a11a66f3ad8948c58caa60e9fd0197b8, State: Running, Role: LEADER
I20260812 08:02:10.322458  9136 consensus_queue.cc:237] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } } peers { permanent_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } } peers { permanent_uuid: "7c3c021feb3340688f89e0c1b67ec2fc" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } }
I20260812 08:02:10.320662  8998 master_runner.cc:187] Error getting master registration for 127.5.152.62:35051: OK, instance_id { permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" instance_seqno: 1786521729749303 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 08:02:10.323606  8998 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 08:02:10.324409  9181 sys_catalog.cc:455] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a11a66f3ad8948c58caa60e9fd0197b8. Latest consensus state: current_term: 1 leader_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } } peers { permanent_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } } peers { permanent_uuid: "7c3c021feb3340688f89e0c1b67ec2fc" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } } }
I20260812 08:02:10.324532  9181 sys_catalog.cc:458] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [sys.catalog]: This master's current role is: LEADER
W20260812 08:02:10.324817  9180 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:02:10.324924  9180 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 08:02:10.329164  9206 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:02:10.329288  9206 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:02:10.331248  9167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } } peers { permanent_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } } peers { permanent_uuid: "7c3c021feb3340688f89e0c1b67ec2fc" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } } }
I20260812 08:02:10.331369  9167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:10.332332  9186 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:02:10.333423  9186 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:02:10.335368  9166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } } peers { permanent_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } } peers { permanent_uuid: "7c3c021feb3340688f89e0c1b67ec2fc" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } } }
I20260812 08:02:10.335485  9166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:10.338193  9068 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc [term 1 FOLLOWER]: Refusing update from remote peer a11a66f3ad8948c58caa60e9fd0197b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:10.338853  9136 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c3c021feb3340688f89e0c1b67ec2fc" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:10.342963  9167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a11a66f3ad8948c58caa60e9fd0197b8. Latest consensus state: current_term: 1 leader_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } } peers { permanent_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } } peers { permanent_uuid: "7c3c021feb3340688f89e0c1b67ec2fc" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } } }
I20260812 08:02:10.343082  9167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:10.343959  8808 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39 [term 1 FOLLOWER]: Refusing update from remote peer a11a66f3ad8948c58caa60e9fd0197b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:10.344466  9136 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:10.348198  9166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a11a66f3ad8948c58caa60e9fd0197b8. Latest consensus state: current_term: 1 leader_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } } peers { permanent_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } } peers { permanent_uuid: "7c3c021feb3340688f89e0c1b67ec2fc" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } } }
I20260812 08:02:10.348320  9166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:10.352193  9136 sys_catalog.cc:455] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } } peers { permanent_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } } peers { permanent_uuid: "7c3c021feb3340688f89e0c1b67ec2fc" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } } }
I20260812 08:02:10.352324  9136 sys_catalog.cc:458] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:10.354816  9167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } } peers { permanent_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } } peers { permanent_uuid: "7c3c021feb3340688f89e0c1b67ec2fc" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } } }
I20260812 08:02:10.354950  9167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:10.355077  9210 mvcc.cc:204] Tried to move back new op lower bound from 7317593007458398208 to 7317593007403659264. Current Snapshot: MvccSnapshot[applied={T|T < 7317593007458398208}]
I20260812 08:02:10.355428  9211 mvcc.cc:204] Tried to move back new op lower bound from 7317593007458398208 to 7317593007403659264. Current Snapshot: MvccSnapshot[applied={T|T < 7317593007458398208}]
I20260812 08:02:10.356633  9181 sys_catalog.cc:455] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } } peers { permanent_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } } peers { permanent_uuid: "7c3c021feb3340688f89e0c1b67ec2fc" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } } }
I20260812 08:02:10.356738  9181 sys_catalog.cc:458] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8 [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:10.357823  9166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d9e2298957149f6bcc9faf65863dd39" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 35051 } } peers { permanent_uuid: "a11a66f3ad8948c58caa60e9fd0197b8" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 39973 } } peers { permanent_uuid: "7c3c021feb3340688f89e0c1b67ec2fc" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 34749 } } }
I20260812 08:02:10.357944  9166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:10.358084  9214 mvcc.cc:204] Tried to move back new op lower bound from 7317593007458398208 to 7317593007403659264. Current Snapshot: MvccSnapshot[applied={T|T < 7317593007458398208}]
I20260812 08:02:10.360836  9186 catalog_manager.cc:1383] Generated new cluster ID: 9957fb218a2a4a02b48cbbb2dac21df4
I20260812 08:02:10.360900  9186 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:02:10.378326  9186 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:02:10.379575  9186 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:02:10.400236  9186 catalog_manager.cc:6092] T 00000000000000000000000000000000 P a11a66f3ad8948c58caa60e9fd0197b8: Generated new TSK 0
I20260812 08:02:10.400997  9186 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:02:10.470108  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.152.1:0
--local_ip_for_outbound_sockets=127.5.152.1
--webserver_interface=127.5.152.1
--webserver_port=0
--tserver_master_addrs=127.5.152.62:35051,127.5.152.61:39973,127.5.152.60:34749
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:35813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 08:02:10.618034  9229 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:10.618232  9229 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:10.618379  9229 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:10.621462  9229 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:10.621630  9229 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.152.1
I20260812 08:02:10.625279  9229 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:35813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.1
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--tserver_master_addrs=127.5.152.62:35051,127.5.152.61:39973,127.5.152.60:34749
--never_fsync=true
--heap_profile_path=/tmp/kudu.9229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.152.1

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:10.626274  9229 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:10.627029  9229 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:10.632872  9244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:10.632968  9249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:10.632941  9229 server_base.cc:1061] running on GCE node
W20260812 08:02:10.633111  9236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:10.633489  9229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:10.633929  9229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:10.635090  9229 hybrid_clock.cc:648] HybridClock initialized: now 1786521730635080 us; error 29 us; skew 500 ppm
Aug 12 08:02:10 dist-test-slave-1nl6 krb5kdc[8728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521730, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:02:10.637686  9229 init.cc:377] Logged in from keytab as kudu/127.5.152.1@KRBTEST.COM (short username kudu)
I20260812 08:02:10.648092  9229 webserver.cc:533] Webserver started at https://127.5.152.1:37753/ using document root <none> and password file <none>
I20260812 08:02:10.648629  9229 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:10.648672  9229 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:10.648818  9229 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:10.650173  9229 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/data/instance:
uuid: "6dab7547432d4b8d8891443b86ea273a"
format_stamp: "Formatted at 2026-08-12 08:02:10 on dist-test-slave-1nl6"
I20260812 08:02:10.650553  9229 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/wal/instance:
uuid: "6dab7547432d4b8d8891443b86ea273a"
format_stamp: "Formatted at 2026-08-12 08:02:10 on dist-test-slave-1nl6"
I20260812 08:02:10.653688  9229 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20260812 08:02:10.655701  9261 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:10.656641  9229 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:02:10.656776  9229 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/wal
uuid: "6dab7547432d4b8d8891443b86ea273a"
format_stamp: "Formatted at 2026-08-12 08:02:10 on dist-test-slave-1nl6"
I20260812 08:02:10.656885  9229 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:10.675559  9229 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:10.678179  9229 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:10.678381  9229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:10.679039  9229 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:02:10.679965  9229 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:10.680013  9229 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:10.680083  9229 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:10.680123  9229 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:10.689685  9229 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.1:46251
I20260812 08:02:10.689787  9442 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.1:46251 every 8 connection(s)
I20260812 08:02:10.690606  9229 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
I20260812 08:02:10.697274  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 9229
I20260812 08:02:10.697424  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/minicluster-data/ts-0/wal/instance
Aug 12 08:02:10 dist-test-slave-1nl6 krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521730, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
Aug 12 08:02:10 dist-test-slave-1nl6 krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521730, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
Aug 12 08:02:10 dist-test-slave-1nl6 krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521730, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
I20260812 08:02:10.709435  9443 heartbeater.cc:344] Connected to a master server at 127.5.152.60:34749
I20260812 08:02:10.709755  9443 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:10.709877  9447 heartbeater.cc:344] Connected to a master server at 127.5.152.61:39973
I20260812 08:02:10.709985  9447 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:10.710427  9443 heartbeater.cc:507] Master 127.5.152.60:34749 requested a full tablet report, sending...
I20260812 08:02:10.710485  9447 heartbeater.cc:507] Master 127.5.152.61:39973 requested a full tablet report, sending...
I20260812 08:02:10.710988  9445 heartbeater.cc:344] Connected to a master server at 127.5.152.62:35051
I20260812 08:02:10.711081  9445 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:10.711301  9445 heartbeater.cc:507] Master 127.5.152.62:35051 requested a full tablet report, sending...
I20260812 08:02:10.712436  9054 ts_manager.cc:194] Registered new tserver with Master: 6dab7547432d4b8d8891443b86ea273a (127.5.152.1:46251)
I20260812 08:02:10.712436  8922 ts_manager.cc:194] Registered new tserver with Master: 6dab7547432d4b8d8891443b86ea273a (127.5.152.1:46251)
I20260812 08:02:10.712788  8795 ts_manager.cc:194] Registered new tserver with Master: 6dab7547432d4b8d8891443b86ea273a (127.5.152.1:46251)
I20260812 08:02:10.713795  8922 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.152.1@KRBTEST.COM'} at 127.5.152.1:47317
I20260812 08:02:10.722779  5728 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
time=2026-08-12T08:02:10.779Z level=INFO source=main.go:1678 msg="updated GOGC" old=100 new=75
time=2026-08-12T08:02:10.780Z level=INFO source=main.go:744 msg="Leaving GOMAXPROCS=4: CPU quota undefined" component=automaxprocs
time=2026-08-12T08:02:10.780Z level=INFO source=memlimit.go:198 msg="GOMEMLIMIT is updated" component=automemlimit package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=14142549196 previous=9223372036854775807
time=2026-08-12T08:02:10.780Z level=INFO source=main.go:851 msg="Starting Prometheus Server" mode=server version="(version=3.11.2, branch=HEAD, revision=f0f0fdd679dcd6df320b0558b20919f7cd44c407)"
time=2026-08-12T08:02:10.780Z level=INFO source=main.go:856 msg="operational information" build_context="(go=go1.26.2, platform=linux/amd64, user=root@d2684485c347, date=20260413-11:53:40, tags=netgo,builtinassets)" host_details="(Linux 5.4.0-1157-gcp #166~18.04.1-Ubuntu SMP Wed Jan 21 01:57:46 UTC 2026 x86_64 dist-test-slave-1nl6 (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)"
time=2026-08-12T08:02:11.051Z level=INFO source=web.go:710 msg="Start listening for connections" component=web address=127.0.0.1:0
time=2026-08-12T08:02:11.051Z level=INFO source=main.go:1410 msg="Starting TSDB ..."
time=2026-08-12T08:02:11.054Z level=INFO source=tls_config.go:354 msg="Listening on" component=web address=127.0.0.1:35539
time=2026-08-12T08:02:11.054Z level=INFO source=tls_config.go:357 msg="TLS is disabled." component=web http2=false address=127.0.0.1:35539
time=2026-08-12T08:02:11.058Z level=INFO source=head.go:698 msg="Replaying on-disk memory mappable chunks if any" component=tsdb
time=2026-08-12T08:02:11.058Z level=INFO source=head.go:784 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=2.055µs
time=2026-08-12T08:02:11.058Z level=INFO source=head.go:792 msg="Replaying WAL, this may take a while" component=tsdb
time=2026-08-12T08:02:11.059Z level=INFO source=head.go:865 msg="WAL segment loaded" component=tsdb segment=0 maxSegment=0 duration=581.449µs
time=2026-08-12T08:02:11.059Z level=INFO source=head.go:902 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=113.402µs wal_replay_duration=702.515µs wbl_replay_duration=111ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=2.055µs total_replay_duration=846.655µs
time=2026-08-12T08:02:11.065Z level=INFO source=main.go:1431 msg="filesystem information" fs_type=OVERLAYFS_SUPER_MAGIC
time=2026-08-12T08:02:11.065Z level=INFO source=main.go:1434 msg="TSDB started"
time=2026-08-12T08:02:11.065Z level=INFO source=main.go:1632 msg="Loading configuration file" filename=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/prometheus.yml
time=2026-08-12T08:02:11.065Z level=INFO source=main.go:1048 msg="TSDB retention updated" duration=15d size=0B percentage=0
time=2026-08-12T08:02:11.067Z level=INFO source=main.go:1671 msg="Completed loading of configuration file" db_storage=39.16µs remote_storage=1.968µs web_handler=393ns query_engine=1.071µs scrape=1.54629ms scrape_sd=77.884µs notify=2.685µs notify_sd=1.498µs rules=1.792µs tracing=11.719µs filename=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786521680053065-5728-0/prometheus.yml totalDuration=1.981808ms
time=2026-08-12T08:02:11.067Z level=INFO source=main.go:1395 msg="Server is ready to receive web requests."
time=2026-08-12T08:02:11.067Z level=INFO source=manager.go:209 msg="Starting rule manager..." component="rule manager"
I20260812 08:02:11.326398  9180 catalog_manager.cc:1295] Loaded cluster ID: 9957fb218a2a4a02b48cbbb2dac21df4
I20260812 08:02:11.326498  9180 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc: loading cluster ID for follower catalog manager: success
I20260812 08:02:11.328118  9180 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc: acquiring CA information for follower catalog manager: success
I20260812 08:02:11.329166  9180 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 7c3c021feb3340688f89e0c1b67ec2fc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 08:02:11.330298  9206 catalog_manager.cc:1295] Loaded cluster ID: 9957fb218a2a4a02b48cbbb2dac21df4
I20260812 08:02:11.330389  9206 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39: loading cluster ID for follower catalog manager: success
I20260812 08:02:11.331935  9206 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39: acquiring CA information for follower catalog manager: success
I20260812 08:02:11.333007  9206 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 0d9e2298957149f6bcc9faf65863dd39: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 08:02:11.715672  9447 heartbeater.cc:499] Master 127.5.152.61:39973 was elected leader, sending a full tablet report...
time=2026-08-12T08:02:17.989Z level=WARN source=main.go:1191 msg="Received an OS signal, exiting gracefully..." signal=terminated
time=2026-08-12T08:02:17.989Z level=INFO source=main.go:1216 msg="Stopping scrape discovery manager..."
time=2026-08-12T08:02:17.989Z level=INFO source=main.go:1230 msg="Stopping notify discovery manager..."
time=2026-08-12T08:02:17.989Z level=INFO source=main.go:1245 msg="Stopping rule manager manager..."
time=2026-08-12T08:02:17.989Z level=INFO source=manager.go:223 msg="Stopping rule manager..." component="rule manager"
time=2026-08-12T08:02:17.989Z level=INFO source=manager.go:239 msg="Rule manager stopped" component="rule manager"
time=2026-08-12T08:02:17.989Z level=INFO source=main.go:1269 msg="Stopping scrape manager..."
time=2026-08-12T08:02:17.989Z level=INFO source=main.go:1212 msg="Scrape discovery manager stopped"
time=2026-08-12T08:02:17.989Z level=INFO source=main.go:1226 msg="Notify discovery manager stopped"
time=2026-08-12T08:02:17.989Z level=INFO source=main.go:1241 msg="Rule manager stopped"
time=2026-08-12T08:02:17.990Z level=INFO source=main.go:1284 msg="Stopping tracing manager..."
time=2026-08-12T08:02:17.990Z level=INFO source=main.go:1369 msg="Stopping reloaders..."
time=2026-08-12T08:02:17.990Z level=INFO source=main.go:1261 msg="Scrape manager stopped"
time=2026-08-12T08:02:17.990Z level=INFO source=main.go:1458 msg="Stopping storage..."
time=2026-08-12T08:02:17.990Z level=INFO source=main.go:1280 msg="Tracing manager stopped"
time=2026-08-12T08:02:17.990Z level=INFO source=main.go:1361 msg="Reloaders stopped"
time=2026-08-12T08:02:17.993Z level=INFO source=main.go:1536 msg="Stopping web handler..."
time=2026-08-12T08:02:17.993Z level=INFO source=main.go:1559 msg="Stopping notifier manager..."
time=2026-08-12T08:02:17.993Z level=INFO source=main.go:1454 msg="TSDB stopped"
time=2026-08-12T08:02:17.993Z level=INFO source=manager.go:321 msg="Stopping notification manager..." component=notifier
time=2026-08-12T08:02:17.993Z level=INFO source=main.go:1555 msg="Notifier manager stopped"
time=2026-08-12T08:02:17.993Z level=INFO source=main.go:1532 msg="Web handler stopped"
time=2026-08-12T08:02:17.994Z level=INFO source=main.go:1570 msg="See you next time!"
I20260812 08:02:18.009927  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 9229
I20260812 08:02:18.028389  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 8751
I20260812 08:02:18.042543  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 8878
I20260812 08:02:18.060916  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 8998
2026-08-12T08:02:18Z chronyd exiting
[       OK ] PrometheusTokenITest.TestMissingTokenScrapeFails (10928 ms)
[ RUN      ] PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed
Loading random data
Initializing database '/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 08:02:18 dist-test-slave-1nl6 krb5kdc[9480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 08:02:18 dist-test-slave-1nl6 krb5kdc[9480](info): set up 2 sockets
Aug 12 08:02:18 dist-test-slave-1nl6 krb5kdc[9480](info): commencing operation
krb5kdc: starting...
W20260812 08:02:20.310058  5728 mini_kdc.cc:121] Time spent starting KDC: real 2.215s	user 0.006s	sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/test-user.keytab.
Aug 12 08:02:20 dist-test-slave-1nl6 krb5kdc[9480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM: 
2026-08-12T08:02:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T08:02:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:02:20.446987  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.62:44801
--webserver_interface=127.5.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:41091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:44801,127.5.152.61:43393,127.5.152.60:43787
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 08:02:20.553732  9504 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:20.554059  9504 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:20.554178  9504 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:20.558251  9504 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:20.558353  9504 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:20.558390  9504 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:20.558451  9504 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:20.563593  9504 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:41091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:44801,127.5.152.61:43393,127.5.152.60:43787
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.62:44801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.62
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:20.564674  9504 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:20.565512  9504 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:20.570829  9517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:20.570859  9514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:20.570832  9512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:20.571199  9504 server_base.cc:1061] running on GCE node
I20260812 08:02:20.571699  9504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:20.572526  9504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:20.573704  9504 hybrid_clock.cc:648] HybridClock initialized: now 1786521740573688 us; error 33 us; skew 500 ppm
Aug 12 08:02:20 dist-test-slave-1nl6 krb5kdc[9480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521740, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:02:20.576457  9504 init.cc:377] Logged in from keytab as kudu/127.5.152.62@KRBTEST.COM (short username kudu)
I20260812 08:02:20.587956  9504 webserver.cc:533] Webserver started at https://127.5.152.62:45177/ using document root <none> and password file <none>
I20260812 08:02:20.588589  9504 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:20.588670  9504 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:20.588896  9504 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:20.590533  9504 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-0/data/instance:
uuid: "f4eb4177e35b4b8c80126a1008493c9b"
format_stamp: "Formatted at 2026-08-12 08:02:20 on dist-test-slave-1nl6"
I20260812 08:02:20.591080  9504 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-0/wal/instance:
uuid: "f4eb4177e35b4b8c80126a1008493c9b"
format_stamp: "Formatted at 2026-08-12 08:02:20 on dist-test-slave-1nl6"
I20260812 08:02:20.594385  9504 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:02:20.596550  9534 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:20.597451  9504 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.001s
I20260812 08:02:20.597594  9504 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-0/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-0/wal
uuid: "f4eb4177e35b4b8c80126a1008493c9b"
format_stamp: "Formatted at 2026-08-12 08:02:20 on dist-test-slave-1nl6"
I20260812 08:02:20.597692  9504 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:20.608477  9504 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:20.611261  9504 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:20.611469  9504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:20.618640  9504 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.62:44801
I20260812 08:02:20.618647  9631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.62:44801 every 8 connection(s)
I20260812 08:02:20.619611  9504 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
I20260812 08:02:20.622265  9632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:20.624575  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 9504
I20260812 08:02:20.624706  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-0/wal/instance
I20260812 08:02:20.626020  9632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.152.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.61@KRBTEST.COM" created.
Aug 12 08:02:20 dist-test-slave-1nl6 krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521740, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
I20260812 08:02:20.645533  9632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:02:20.646497  9544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.152.61:43393: connect: Connection refused (error 111)
W20260812 08:02:20.647778  9632 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:43393: Network error: Client connection negotiation failed: client connection to 127.5.152.61:43393: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:02:20.679145  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.61:43393
--webserver_interface=127.5.152.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:41091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:44801,127.5.152.61:43393,127.5.152.60:43787
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 08:02:20.697129  9632 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } attempt: 1
W20260812 08:02:20.698789  9632 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:43393: Network error: Client connection negotiation failed: client connection to 127.5.152.61:43393: connect: Connection refused (error 111)
I20260812 08:02:20.767004  9632 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } attempt: 2
W20260812 08:02:20.769141  9632 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:43393: Network error: Client connection negotiation failed: client connection to 127.5.152.61:43393: connect: Connection refused (error 111)
W20260812 08:02:20.782335  9645 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:20.782557  9645 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:20.782675  9645 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:20.786310  9645 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:20.786379  9645 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:20.786437  9645 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:20.786487  9645 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:20.790890  9645 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:41091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:44801,127.5.152.61:43393,127.5.152.60:43787
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.61:43393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.61
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:20.791895  9645 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:20.792711  9645 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:20.797871  9658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:20.798193  9662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:20.798210  9657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:20.798256  9645 server_base.cc:1061] running on GCE node
I20260812 08:02:20.798741  9645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:20.799571  9645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:20.800742  9645 hybrid_clock.cc:648] HybridClock initialized: now 1786521740800730 us; error 34 us; skew 500 ppm
Aug 12 08:02:20 dist-test-slave-1nl6 krb5kdc[9480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521740, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:02:20.804431  9645 init.cc:377] Logged in from keytab as kudu/127.5.152.61@KRBTEST.COM (short username kudu)
I20260812 08:02:20.816045  9645 webserver.cc:533] Webserver started at https://127.5.152.61:34159/ using document root <none> and password file <none>
I20260812 08:02:20.816658  9645 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:20.816736  9645 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:20.816983  9645 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:20.818665  9645 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-1/data/instance:
uuid: "f32371f38f7740cf9777099de4da5f5d"
format_stamp: "Formatted at 2026-08-12 08:02:20 on dist-test-slave-1nl6"
I20260812 08:02:20.819188  9645 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-1/wal/instance:
uuid: "f32371f38f7740cf9777099de4da5f5d"
format_stamp: "Formatted at 2026-08-12 08:02:20 on dist-test-slave-1nl6"
I20260812 08:02:20.822382  9645 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.001s
I20260812 08:02:20.824450  9678 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:20.825318  9645 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 08:02:20.825453  9645 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-1/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-1/wal
uuid: "f32371f38f7740cf9777099de4da5f5d"
format_stamp: "Formatted at 2026-08-12 08:02:20 on dist-test-slave-1nl6"
I20260812 08:02:20.825552  9645 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:20.843497  9645 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:20.845956  9645 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:20.846148  9645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:20.853734  9645 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.61:43393
I20260812 08:02:20.853756  9766 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.61:43393 every 8 connection(s)
I20260812 08:02:20.854763  9645 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
I20260812 08:02:20.858773  9771 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:20.860437  9632 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } attempt: 3
I20260812 08:02:20.863281  9771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:20.864579  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 9645
I20260812 08:02:20.864733  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-1/wal/instance
Aug 12 08:02:20 dist-test-slave-1nl6 krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521740, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
Aug 12 08:02:20 dist-test-slave-1nl6 krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521740, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
I20260812 08:02:20.876223  9632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.152.60@KRBTEST.COM; defaulting to no policy
I20260812 08:02:20.876307  9771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:02:20.877861  9632 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:43787: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43787: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.60@KRBTEST.COM" created.
Aug 12 08:02:20 dist-test-slave-1nl6 krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521740, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:02:20.884668  9771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:02:20.885455  9688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43787: connect: Connection refused (error 111)
W20260812 08:02:20.886270  9771 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:43787: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43787: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.60@KRBTEST.COM" created.
I20260812 08:02:20.909040  9632 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } attempt: 1
W20260812 08:02:20.910499  9632 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:43787: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43787: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:02:20.917517  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.60:43787
--webserver_interface=127.5.152.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:41091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:44801,127.5.152.61:43393,127.5.152.60:43787
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 08:02:20.935503  9771 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } attempt: 1
W20260812 08:02:20.936865  9771 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:43787: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43787: connect: Connection refused (error 111)
I20260812 08:02:20.985823  9632 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } attempt: 2
W20260812 08:02:20.987382  9632 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:43787: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43787: connect: Connection refused (error 111)
I20260812 08:02:21.005043  9771 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } attempt: 2
W20260812 08:02:21.006389  9771 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:43787: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43787: connect: Connection refused (error 111)
W20260812 08:02:21.022171  9783 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:21.022367  9783 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:21.022482  9783 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:21.025655  9783 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:21.025712  9783 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:21.025731  9783 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:21.025748  9783 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:21.029956  9783 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:41091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:44801,127.5.152.61:43393,127.5.152.60:43787
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.60:43787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.60
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:21.030857  9783 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:21.031638  9783 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:21.036624  9797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:21.036629  9793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:21.036808  9783 server_base.cc:1061] running on GCE node
W20260812 08:02:21.036629  9794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:21.037359  9783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:21.038084  9783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:21.039300  9783 hybrid_clock.cc:648] HybridClock initialized: now 1786521741039282 us; error 34 us; skew 500 ppm
Aug 12 08:02:21 dist-test-slave-1nl6 krb5kdc[9480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521741, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:02:21.042165  9783 init.cc:377] Logged in from keytab as kudu/127.5.152.60@KRBTEST.COM (short username kudu)
I20260812 08:02:21.055796  9783 webserver.cc:533] Webserver started at https://127.5.152.60:38849/ using document root <none> and password file <none>
I20260812 08:02:21.056469  9783 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:21.056525  9783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:21.056720  9783 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:21.058612  9783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-2/data/instance:
uuid: "2e6d84e4240844e8adac563da95dd8c9"
format_stamp: "Formatted at 2026-08-12 08:02:21 on dist-test-slave-1nl6"
I20260812 08:02:21.059144  9783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-2/wal/instance:
uuid: "2e6d84e4240844e8adac563da95dd8c9"
format_stamp: "Formatted at 2026-08-12 08:02:21 on dist-test-slave-1nl6"
I20260812 08:02:21.063014  9783 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.000s
I20260812 08:02:21.065166  9808 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:21.066180  9783 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:02:21.066282  9783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-2/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-2/wal
uuid: "2e6d84e4240844e8adac563da95dd8c9"
format_stamp: "Formatted at 2026-08-12 08:02:21 on dist-test-slave-1nl6"
I20260812 08:02:21.066359  9783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:21.083400  9783 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:21.086143  9783 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:21.086298  9783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:21.086573  9632 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } attempt: 3
W20260812 08:02:21.088078  9632 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:43787: Network error: Client connection negotiation failed: client connection to 127.5.152.60:43787: connect: Connection refused (error 111)
I20260812 08:02:21.094226  9783 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.60:43787
I20260812 08:02:21.094244  9893 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.60:43787 every 8 connection(s)
I20260812 08:02:21.095340  9783 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
I20260812 08:02:21.097533  9771 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } attempt: 3
I20260812 08:02:21.098230  9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:21.101991  9895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:21.103228  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 9783
I20260812 08:02:21.103380  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/master-2/wal/instance
Aug 12 08:02:21 dist-test-slave-1nl6 krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521740, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
Aug 12 08:02:21 dist-test-slave-1nl6 krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521741, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
Aug 12 08:02:21 dist-test-slave-1nl6 krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
I20260812 08:02:21.115118  9771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d: Bootstrap starting.
I20260812 08:02:21.115911  9895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:21.118044  9771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:21.119004  9771 log.cc:826] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:21.123063  9771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d: No bootstrap required, opened a new log
Aug 12 08:02:21 dist-test-slave-1nl6 krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
Aug 12 08:02:21 dist-test-slave-1nl6 krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521741, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:02:21.126541  9771 raft_consensus.cc:359] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } } peers { permanent_uuid: "f32371f38f7740cf9777099de4da5f5d" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } } peers { permanent_uuid: "2e6d84e4240844e8adac563da95dd8c9" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } }
I20260812 08:02:21.126740  9771 raft_consensus.cc:385] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:21.126807  9771 raft_consensus.cc:740] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f32371f38f7740cf9777099de4da5f5d, State: Initialized, Role: FOLLOWER
I20260812 08:02:21.127312  9771 consensus_queue.cc:260] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } } peers { permanent_uuid: "f32371f38f7740cf9777099de4da5f5d" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } } peers { permanent_uuid: "2e6d84e4240844e8adac563da95dd8c9" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } }
I20260812 08:02:21.127782  9907 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } } peers { permanent_uuid: "f32371f38f7740cf9777099de4da5f5d" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } } peers { permanent_uuid: "2e6d84e4240844e8adac563da95dd8c9" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } } }
I20260812 08:02:21.127926  9907 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:21.128559  9771 sys_catalog.cc:565] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:02:21.130950  9895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:02:21.134418  9922 catalog_manager.cc:1594] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:02:21.134491  9922 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:02:21.132652  9645 master_runner.cc:187] Error getting master registration for 127.5.152.62:44801: OK, instance_id { permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" instance_seqno: 1786521740616626 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 08:02:21.134868  9645 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Aug 12 08:02:21 dist-test-slave-1nl6 krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521741, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
Aug 12 08:02:21 dist-test-slave-1nl6 krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:02:21.142825  9895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9: Bootstrap starting.
I20260812 08:02:21.145308  9895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:21.146121  9895 log.cc:826] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:21.147832  9895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9: No bootstrap required, opened a new log
I20260812 08:02:21.150983  9895 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } } peers { permanent_uuid: "f32371f38f7740cf9777099de4da5f5d" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } } peers { permanent_uuid: "2e6d84e4240844e8adac563da95dd8c9" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } }
I20260812 08:02:21.151189  9895 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:21.151243  9895 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6d84e4240844e8adac563da95dd8c9, State: Initialized, Role: FOLLOWER
I20260812 08:02:21.151682  9895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } } peers { permanent_uuid: "f32371f38f7740cf9777099de4da5f5d" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } } peers { permanent_uuid: "2e6d84e4240844e8adac563da95dd8c9" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } }
I20260812 08:02:21.152185  9926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } } peers { permanent_uuid: "f32371f38f7740cf9777099de4da5f5d" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } } peers { permanent_uuid: "2e6d84e4240844e8adac563da95dd8c9" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } } }
I20260812 08:02:21.152334  9926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:21.152863  9895 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:02:21.155263  9907 raft_consensus.cc:493] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:21.155539  9907 raft_consensus.cc:515] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } } peers { permanent_uuid: "f32371f38f7740cf9777099de4da5f5d" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } } peers { permanent_uuid: "2e6d84e4240844e8adac563da95dd8c9" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } }
I20260812 08:02:21.157480  9907 leader_election.cc:290] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4eb4177e35b4b8c80126a1008493c9b (127.5.152.62:44801), 2e6d84e4240844e8adac563da95dd8c9 (127.5.152.60:43787)
I20260812 08:02:21.158166  9853 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f32371f38f7740cf9777099de4da5f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6d84e4240844e8adac563da95dd8c9" is_pre_election: true
I20260812 08:02:21.159123  9853 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f32371f38f7740cf9777099de4da5f5d in term 0.
I20260812 08:02:21.159603  9688 leader_election.cc:304] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e6d84e4240844e8adac563da95dd8c9, f32371f38f7740cf9777099de4da5f5d; no voters: 
I20260812 08:02:21.159905  9907 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:02:21.159999  9907 raft_consensus.cc:493] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:21.160050  9907 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:21.161459  9907 raft_consensus.cc:515] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } } peers { permanent_uuid: "f32371f38f7740cf9777099de4da5f5d" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } } peers { permanent_uuid: "2e6d84e4240844e8adac563da95dd8c9" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } }
I20260812 08:02:21.161891  9907 leader_election.cc:290] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [CANDIDATE]: Term 1 election: Requested vote from peers f4eb4177e35b4b8c80126a1008493c9b (127.5.152.62:44801), 2e6d84e4240844e8adac563da95dd8c9 (127.5.152.60:43787)
I20260812 08:02:21.160096  9783 master_runner.cc:187] Error getting master registration for 127.5.152.62:44801: OK, instance_id { permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" instance_seqno: 1786521740616626 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 08:02:21.162592  9783 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 08:02:21.162882  9853 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f32371f38f7740cf9777099de4da5f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6d84e4240844e8adac563da95dd8c9"
I20260812 08:02:21.163043  9853 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:21.164844  9853 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f32371f38f7740cf9777099de4da5f5d in term 1.
I20260812 08:02:21.165683  9688 leader_election.cc:304] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e6d84e4240844e8adac563da95dd8c9, f32371f38f7740cf9777099de4da5f5d; no voters: 
I20260812 08:02:21.165998  9907 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:21.166257  9907 raft_consensus.cc:697] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [term 1 LEADER]: Becoming Leader. State: Replica: f32371f38f7740cf9777099de4da5f5d, State: Running, Role: LEADER
I20260812 08:02:21.166641  9907 consensus_queue.cc:237] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } } peers { permanent_uuid: "f32371f38f7740cf9777099de4da5f5d" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } } peers { permanent_uuid: "2e6d84e4240844e8adac563da95dd8c9" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } }
I20260812 08:02:21.169632  9944 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f32371f38f7740cf9777099de4da5f5d. Latest consensus state: current_term: 1 leader_uuid: "f32371f38f7740cf9777099de4da5f5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } } peers { permanent_uuid: "f32371f38f7740cf9777099de4da5f5d" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } } peers { permanent_uuid: "2e6d84e4240844e8adac563da95dd8c9" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } } }
I20260812 08:02:21.169761  9944 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [sys.catalog]: This master's current role is: LEADER
W20260812 08:02:21.171787  9941 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:02:21.171882  9941 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:02:21.178720  9949 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:02:21.179864  9949 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:02:21.185078  9853 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9 [term 1 FOLLOWER]: Refusing update from remote peer f32371f38f7740cf9777099de4da5f5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:21.185737  9944 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6d84e4240844e8adac563da95dd8c9" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:21.189940  9926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f32371f38f7740cf9777099de4da5f5d. Latest consensus state: current_term: 1 leader_uuid: "f32371f38f7740cf9777099de4da5f5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } } peers { permanent_uuid: "f32371f38f7740cf9777099de4da5f5d" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } } peers { permanent_uuid: "2e6d84e4240844e8adac563da95dd8c9" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } } }
I20260812 08:02:21.190056  9926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:21.192647  9944 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f32371f38f7740cf9777099de4da5f5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } } peers { permanent_uuid: "f32371f38f7740cf9777099de4da5f5d" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } } peers { permanent_uuid: "2e6d84e4240844e8adac563da95dd8c9" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } } }
I20260812 08:02:21.192759  9944 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:21.193603  9926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f32371f38f7740cf9777099de4da5f5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } } peers { permanent_uuid: "f32371f38f7740cf9777099de4da5f5d" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } } peers { permanent_uuid: "2e6d84e4240844e8adac563da95dd8c9" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } } }
I20260812 08:02:21.193698  9926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:21.194006  9951 mvcc.cc:204] Tried to move back new op lower bound from 7317593051887169536 to 7317593051823833088. Current Snapshot: MvccSnapshot[applied={T|T < 7317593051887169536}]
I20260812 08:02:21.194803  9952 mvcc.cc:204] Tried to move back new op lower bound from 7317593051887169536 to 7317593051823833088. Current Snapshot: MvccSnapshot[applied={T|T < 7317593051887169536}]
I20260812 08:02:21.200637  9949 catalog_manager.cc:1383] Generated new cluster ID: 56aa986fbe9a44839dd16efe898cec01
I20260812 08:02:21.200747  9949 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:02:21.236796  9949 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:02:21.237903  9949 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:02:21.252264  9632 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } attempt: 4
I20260812 08:02:21.259011  9949 catalog_manager.cc:6092] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d: Generated new TSK 0
I20260812 08:02:21.260109  9949 catalog_manager.cc:1550] Initializing in-progress tserver states...
Aug 12 08:02:21 dist-test-slave-1nl6 krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521740, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:02:21.272562  9632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4eb4177e35b4b8c80126a1008493c9b: Bootstrap starting.
I20260812 08:02:21.275686  9632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4eb4177e35b4b8c80126a1008493c9b: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:21.276671  9632 log.cc:826] T 00000000000000000000000000000000 P f4eb4177e35b4b8c80126a1008493c9b: Log is configured to *not* fsync() on all Append() calls
WARNING: no policy specified for kudu/127.5.152.1@KRBTEST.COM; defaulting to no policy
I20260812 08:02:21.280380  9632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4eb4177e35b4b8c80126a1008493c9b: No bootstrap required, opened a new log
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.1@KRBTEST.COM" created.
I20260812 08:02:21.284049  9632 raft_consensus.cc:359] T 00000000000000000000000000000000 P f4eb4177e35b4b8c80126a1008493c9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } } peers { permanent_uuid: "f32371f38f7740cf9777099de4da5f5d" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } } peers { permanent_uuid: "2e6d84e4240844e8adac563da95dd8c9" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } }
I20260812 08:02:21.284262  9632 raft_consensus.cc:385] T 00000000000000000000000000000000 P f4eb4177e35b4b8c80126a1008493c9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:21.284323  9632 raft_consensus.cc:740] T 00000000000000000000000000000000 P f4eb4177e35b4b8c80126a1008493c9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4eb4177e35b4b8c80126a1008493c9b, State: Initialized, Role: FOLLOWER
I20260812 08:02:21.284864  9632 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4eb4177e35b4b8c80126a1008493c9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } } peers { permanent_uuid: "f32371f38f7740cf9777099de4da5f5d" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } } peers { permanent_uuid: "2e6d84e4240844e8adac563da95dd8c9" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } }
I20260812 08:02:21.286175  9632 sys_catalog.cc:565] T 00000000000000000000000000000000 P f4eb4177e35b4b8c80126a1008493c9b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:02:21.286419  9585 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f4eb4177e35b4b8c80126a1008493c9b [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:21.286959  9586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f32371f38f7740cf9777099de4da5f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4eb4177e35b4b8c80126a1008493c9b"
I20260812 08:02:21.288462  9585 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f4eb4177e35b4b8c80126a1008493c9b [term 1 FOLLOWER]: Refusing update from remote peer f32371f38f7740cf9777099de4da5f5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:21.289145  9907 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:21.287411  9588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f32371f38f7740cf9777099de4da5f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4eb4177e35b4b8c80126a1008493c9b" is_pre_election: true
I20260812 08:02:21.289963  9588 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f4eb4177e35b4b8c80126a1008493c9b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f32371f38f7740cf9777099de4da5f5d in term 1.
I20260812 08:02:21.299381  9977 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4eb4177e35b4b8c80126a1008493c9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f32371f38f7740cf9777099de4da5f5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } } peers { permanent_uuid: "f32371f38f7740cf9777099de4da5f5d" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } } peers { permanent_uuid: "2e6d84e4240844e8adac563da95dd8c9" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } } }
I20260812 08:02:21.299527  9977 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4eb4177e35b4b8c80126a1008493c9b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:21.299641  9977 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4eb4177e35b4b8c80126a1008493c9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f32371f38f7740cf9777099de4da5f5d. Latest consensus state: current_term: 1 leader_uuid: "f32371f38f7740cf9777099de4da5f5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } } peers { permanent_uuid: "f32371f38f7740cf9777099de4da5f5d" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } } peers { permanent_uuid: "2e6d84e4240844e8adac563da95dd8c9" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } } }
I20260812 08:02:21.299729  9977 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4eb4177e35b4b8c80126a1008493c9b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:21.299844  9977 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4eb4177e35b4b8c80126a1008493c9b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f32371f38f7740cf9777099de4da5f5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } } peers { permanent_uuid: "f32371f38f7740cf9777099de4da5f5d" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } } peers { permanent_uuid: "2e6d84e4240844e8adac563da95dd8c9" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } } }
I20260812 08:02:21.299928  9977 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4eb4177e35b4b8c80126a1008493c9b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:21.302682  9978 mvcc.cc:204] Tried to move back new op lower bound from 7317593052169228288 to 7317593051823833088. Current Snapshot: MvccSnapshot[applied={T|T < 7317593051887169536}]
I20260812 08:02:21.305964  9976 catalog_manager.cc:1295] Loaded cluster ID: 56aa986fbe9a44839dd16efe898cec01
I20260812 08:02:21.306022  9976 catalog_manager.cc:1588] T 00000000000000000000000000000000 P f4eb4177e35b4b8c80126a1008493c9b: loading cluster ID for follower catalog manager: success
I20260812 08:02:21.308243  9976 catalog_manager.cc:1610] T 00000000000000000000000000000000 P f4eb4177e35b4b8c80126a1008493c9b: acquiring CA information for follower catalog manager: success
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:02:21.309504  9976 catalog_manager.cc:1638] T 00000000000000000000000000000000 P f4eb4177e35b4b8c80126a1008493c9b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for HTTP/127.5.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:02:21.342365  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.152.1:0
--local_ip_for_outbound_sockets=127.5.152.1
--webserver_interface=127.5.152.1
--webserver_port=0
--tserver_master_addrs=127.5.152.62:44801,127.5.152.61:43393,127.5.152.60:43787
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:41091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 08:02:21.485126  9992 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:21.485328  9992 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:21.485440  9992 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:21.488938  9992 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:21.489104  9992 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.152.1
I20260812 08:02:21.493384  9992 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:41091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.1
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--tserver_master_addrs=127.5.152.62:44801,127.5.152.61:43393,127.5.152.60:43787
--never_fsync=true
--heap_profile_path=/tmp/kudu.9992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.152.1

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:21.494338  9992 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:21.495230  9992 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:21.501400 10000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:21.501389 10001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:21.501543  9992 server_base.cc:1061] running on GCE node
W20260812 08:02:21.501423 10003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:21.502174  9992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:21.502686  9992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:21.503866  9992 hybrid_clock.cc:648] HybridClock initialized: now 1786521741503841 us; error 45 us; skew 500 ppm
Aug 12 08:02:21 dist-test-slave-1nl6 krb5kdc[9480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521741, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:02:21.506764  9992 init.cc:377] Logged in from keytab as kudu/127.5.152.1@KRBTEST.COM (short username kudu)
I20260812 08:02:21.520747  9992 webserver.cc:533] Webserver started at https://127.5.152.1:35745/ using document root <none> and password file <none>
I20260812 08:02:21.521490  9992 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:21.521560  9992 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:21.521786  9992 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:21.523921  9992 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/ts-0/data/instance:
uuid: "6204b090ddbf4220aac3ee8079625d09"
format_stamp: "Formatted at 2026-08-12 08:02:21 on dist-test-slave-1nl6"
I20260812 08:02:21.524451  9992 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/ts-0/wal/instance:
uuid: "6204b090ddbf4220aac3ee8079625d09"
format_stamp: "Formatted at 2026-08-12 08:02:21 on dist-test-slave-1nl6"
I20260812 08:02:21.528499  9992 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260812 08:02:21.531147 10016 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:21.532397  9992 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260812 08:02:21.532509  9992 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/ts-0/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/ts-0/wal
uuid: "6204b090ddbf4220aac3ee8079625d09"
format_stamp: "Formatted at 2026-08-12 08:02:21 on dist-test-slave-1nl6"
I20260812 08:02:21.532606  9992 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:21.567106  9992 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:21.569777  9992 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:21.569939  9992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:21.570488  9992 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:02:21.571607  9992 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:21.571659  9992 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:21.571722  9992 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:21.571759  9992 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:21.581317  9992 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.1:38675
I20260812 08:02:21.581344 10206 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.1:38675 every 8 connection(s)
I20260812 08:02:21.582145  9992 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
I20260812 08:02:21.590013  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 9992
I20260812 08:02:21.590157  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/minicluster-data/ts-0/wal/instance
Aug 12 08:02:21 dist-test-slave-1nl6 krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521741, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
Aug 12 08:02:21 dist-test-slave-1nl6 krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521741, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
Aug 12 08:02:21 dist-test-slave-1nl6 krb5kdc[9480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521741, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:02:21.597287 10209 heartbeater.cc:344] Connected to a master server at 127.5.152.62:44801
I20260812 08:02:21.597532 10209 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:21.597816 10210 heartbeater.cc:344] Connected to a master server at 127.5.152.61:43393
I20260812 08:02:21.597911 10210 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:21.597973 10207 heartbeater.cc:344] Connected to a master server at 127.5.152.60:43787
I20260812 08:02:21.598066 10207 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:21.598124 10209 heartbeater.cc:507] Master 127.5.152.62:44801 requested a full tablet report, sending...
I20260812 08:02:21.598155 10210 heartbeater.cc:507] Master 127.5.152.61:43393 requested a full tablet report, sending...
I20260812 08:02:21.598244 10207 heartbeater.cc:507] Master 127.5.152.60:43787 requested a full tablet report, sending...
I20260812 08:02:21.599543  9574 ts_manager.cc:194] Registered new tserver with Master: 6204b090ddbf4220aac3ee8079625d09 (127.5.152.1:38675)
I20260812 08:02:21.599581  9711 ts_manager.cc:194] Registered new tserver with Master: 6204b090ddbf4220aac3ee8079625d09 (127.5.152.1:38675)
I20260812 08:02:21.599581  9839 ts_manager.cc:194] Registered new tserver with Master: 6204b090ddbf4220aac3ee8079625d09 (127.5.152.1:38675)
I20260812 08:02:21.600809  9711 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.152.1@KRBTEST.COM'} at 127.5.152.1:56939
I20260812 08:02:21.611073  5728 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
time=2026-08-12T08:02:21.679Z level=INFO source=main.go:1678 msg="updated GOGC" old=100 new=75
time=2026-08-12T08:02:21.681Z level=INFO source=main.go:744 msg="Leaving GOMAXPROCS=4: CPU quota undefined" component=automaxprocs
time=2026-08-12T08:02:21.683Z level=INFO source=memlimit.go:198 msg="GOMEMLIMIT is updated" component=automemlimit package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=14142549196 previous=9223372036854775807
time=2026-08-12T08:02:21.683Z level=INFO source=main.go:851 msg="Starting Prometheus Server" mode=server version="(version=3.11.2, branch=HEAD, revision=f0f0fdd679dcd6df320b0558b20919f7cd44c407)"
time=2026-08-12T08:02:21.683Z level=INFO source=main.go:856 msg="operational information" build_context="(go=go1.26.2, platform=linux/amd64, user=root@d2684485c347, date=20260413-11:53:40, tags=netgo,builtinassets)" host_details="(Linux 5.4.0-1157-gcp #166~18.04.1-Ubuntu SMP Wed Jan 21 01:57:46 UTC 2026 x86_64 dist-test-slave-1nl6 (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)"
I20260812 08:02:21.855765  9907 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f32371f38f7740cf9777099de4da5f5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4eb4177e35b4b8c80126a1008493c9b" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 44801 } } peers { permanent_uuid: "f32371f38f7740cf9777099de4da5f5d" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 43393 } } peers { permanent_uuid: "2e6d84e4240844e8adac563da95dd8c9" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 43787 } } }
I20260812 08:02:21.855993  9907 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32371f38f7740cf9777099de4da5f5d [sys.catalog]: This master's current role is: LEADER
time=2026-08-12T08:02:21.909Z level=INFO source=web.go:710 msg="Start listening for connections" component=web address=127.0.0.1:0
time=2026-08-12T08:02:21.910Z level=INFO source=main.go:1410 msg="Starting TSDB ..."
time=2026-08-12T08:02:21.913Z level=INFO source=tls_config.go:354 msg="Listening on" component=web address=127.0.0.1:39587
time=2026-08-12T08:02:21.913Z level=INFO source=tls_config.go:357 msg="TLS is disabled." component=web http2=false address=127.0.0.1:39587
time=2026-08-12T08:02:21.913Z level=INFO source=head.go:698 msg="Replaying on-disk memory mappable chunks if any" component=tsdb
time=2026-08-12T08:02:21.913Z level=INFO source=head.go:784 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=2.917µs
time=2026-08-12T08:02:21.913Z level=INFO source=head.go:792 msg="Replaying WAL, this may take a while" component=tsdb
time=2026-08-12T08:02:21.913Z level=INFO source=head.go:865 msg="WAL segment loaded" component=tsdb segment=0 maxSegment=0 duration=269.415µs
time=2026-08-12T08:02:21.914Z level=INFO source=head.go:902 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=54.09µs wal_replay_duration=320.229µs wbl_replay_duration=176ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=2.917µs total_replay_duration=430.848µs
time=2026-08-12T08:02:21.915Z level=INFO source=main.go:1431 msg="filesystem information" fs_type=OVERLAYFS_SUPER_MAGIC
time=2026-08-12T08:02:21.915Z level=INFO source=main.go:1434 msg="TSDB started"
time=2026-08-12T08:02:21.915Z level=INFO source=main.go:1632 msg="Loading configuration file" filename=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/prometheus.yml
time=2026-08-12T08:02:21.916Z level=INFO source=main.go:1048 msg="TSDB retention updated" duration=15d size=0B percentage=0
time=2026-08-12T08:02:21.917Z level=INFO source=main.go:1671 msg="Completed loading of configuration file" db_storage=64.216µs remote_storage=2.462µs web_handler=526ns query_engine=1.247µs scrape=1.147435ms scrape_sd=261.408µs notify=2.243µs notify_sd=2.122µs rules=2.774µs tracing=6.094µs filename=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786521680053065-5728-0/prometheus.yml totalDuration=1.839157ms
time=2026-08-12T08:02:21.917Z level=INFO source=main.go:1395 msg="Server is ready to receive web requests."
time=2026-08-12T08:02:21.918Z level=INFO source=manager.go:209 msg="Starting rule manager..." component="rule manager"
I20260812 08:02:22.173182  9941 catalog_manager.cc:1295] Loaded cluster ID: 56aa986fbe9a44839dd16efe898cec01
I20260812 08:02:22.173262  9941 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9: loading cluster ID for follower catalog manager: success
I20260812 08:02:22.174878  9941 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9: acquiring CA information for follower catalog manager: success
I20260812 08:02:22.175942  9941 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 2e6d84e4240844e8adac563da95dd8c9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 08:02:22.602366 10210 heartbeater.cc:499] Master 127.5.152.61:43393 was elected leader, sending a full tablet report...
time=2026-08-12T08:02:28.865Z level=WARN source=main.go:1191 msg="Received an OS signal, exiting gracefully..." signal=terminated
time=2026-08-12T08:02:28.865Z level=INFO source=main.go:1216 msg="Stopping scrape discovery manager..."
time=2026-08-12T08:02:28.865Z level=INFO source=main.go:1230 msg="Stopping notify discovery manager..."
time=2026-08-12T08:02:28.865Z level=INFO source=main.go:1245 msg="Stopping rule manager manager..."
time=2026-08-12T08:02:28.865Z level=INFO source=manager.go:223 msg="Stopping rule manager..." component="rule manager"
time=2026-08-12T08:02:28.865Z level=INFO source=manager.go:239 msg="Rule manager stopped" component="rule manager"
time=2026-08-12T08:02:28.865Z level=INFO source=main.go:1269 msg="Stopping scrape manager..."
time=2026-08-12T08:02:28.865Z level=INFO source=main.go:1226 msg="Notify discovery manager stopped"
time=2026-08-12T08:02:28.865Z level=INFO source=main.go:1241 msg="Rule manager stopped"
time=2026-08-12T08:02:28.865Z level=INFO source=main.go:1284 msg="Stopping tracing manager..."
time=2026-08-12T08:02:28.865Z level=INFO source=main.go:1369 msg="Stopping reloaders..."
time=2026-08-12T08:02:28.865Z level=INFO source=main.go:1458 msg="Stopping storage..."
time=2026-08-12T08:02:28.865Z level=INFO source=main.go:1261 msg="Scrape manager stopped"
time=2026-08-12T08:02:28.865Z level=INFO source=main.go:1212 msg="Scrape discovery manager stopped"
time=2026-08-12T08:02:28.865Z level=INFO source=main.go:1280 msg="Tracing manager stopped"
time=2026-08-12T08:02:28.865Z level=INFO source=main.go:1361 msg="Reloaders stopped"
time=2026-08-12T08:02:28.870Z level=INFO source=main.go:1536 msg="Stopping web handler..."
time=2026-08-12T08:02:28.870Z level=INFO source=main.go:1559 msg="Stopping notifier manager..."
time=2026-08-12T08:02:28.870Z level=INFO source=manager.go:321 msg="Stopping notification manager..." component=notifier
time=2026-08-12T08:02:28.870Z level=INFO source=main.go:1454 msg="TSDB stopped"
time=2026-08-12T08:02:28.870Z level=INFO source=main.go:1555 msg="Notifier manager stopped"
time=2026-08-12T08:02:28.871Z level=INFO source=main.go:1532 msg="Web handler stopped"
time=2026-08-12T08:02:28.871Z level=INFO source=main.go:1570 msg="See you next time!"
I20260812 08:02:28.885896  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 9992
I20260812 08:02:28.904377  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 9504
I20260812 08:02:28.923025  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 9645
I20260812 08:02:28.938575  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 9783
2026-08-12T08:02:28Z chronyd exiting
[       OK ] PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed (10871 ms)
[ RUN      ] PrometheusTokenITest.TestWrongTokenSdFails
Loading random data
Initializing database '/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 08:02:29 dist-test-slave-1nl6 krb5kdc[10258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 08:02:29 dist-test-slave-1nl6 krb5kdc[10258](info): set up 2 sockets
Aug 12 08:02:29 dist-test-slave-1nl6 krb5kdc[10258](info): commencing operation
krb5kdc: starting...
W20260812 08:02:31.214591  5728 mini_kdc.cc:121] Time spent starting KDC: real 2.248s	user 0.000s	sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/test-user.keytab.
Aug 12 08:02:31 dist-test-slave-1nl6 krb5kdc[10258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM: 
2026-08-12T08:02:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T08:02:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:02:31.357736  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.62:34319
--webserver_interface=127.5.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:32837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:34319,127.5.152.61:36687,127.5.152.60:41815
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 08:02:31.459141 10279 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:31.459337 10279 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:31.459398 10279 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:31.462445 10279 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:31.462498 10279 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:31.462517 10279 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:31.462534 10279 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:31.466528 10279 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:32837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:34319,127.5.152.61:36687,127.5.152.60:41815
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.62:34319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.62
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:31.467427 10279 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:31.468108 10279 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:31.473502 10285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:31.473507 10286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:31.473564 10279 server_base.cc:1061] running on GCE node
W20260812 08:02:31.473766 10288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:31.474220 10279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:31.475040 10279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:31.476217 10279 hybrid_clock.cc:648] HybridClock initialized: now 1786521751476194 us; error 47 us; skew 500 ppm
Aug 12 08:02:31 dist-test-slave-1nl6 krb5kdc[10258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521751, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:02:31.478811 10279 init.cc:377] Logged in from keytab as kudu/127.5.152.62@KRBTEST.COM (short username kudu)
I20260812 08:02:31.489926 10279 webserver.cc:533] Webserver started at https://127.5.152.62:41189/ using document root <none> and password file <none>
I20260812 08:02:31.490502 10279 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:31.490576 10279 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:31.490793 10279 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:31.492367 10279 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/data/instance:
uuid: "877597133e56464eb892fa75844fccab"
format_stamp: "Formatted at 2026-08-12 08:02:31 on dist-test-slave-1nl6"
I20260812 08:02:31.492851 10279 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/wal/instance:
uuid: "877597133e56464eb892fa75844fccab"
format_stamp: "Formatted at 2026-08-12 08:02:31 on dist-test-slave-1nl6"
I20260812 08:02:31.495868 10279 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.001s	sys 0.003s
I20260812 08:02:31.497709 10303 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:31.498600 10279 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:02:31.498734 10279 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/wal
uuid: "877597133e56464eb892fa75844fccab"
format_stamp: "Formatted at 2026-08-12 08:02:31 on dist-test-slave-1nl6"
I20260812 08:02:31.498840 10279 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:31.549072 10279 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:31.551539 10279 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:31.551774 10279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:31.559176 10279 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.62:34319
I20260812 08:02:31.559228 10395 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.62:34319 every 8 connection(s)
I20260812 08:02:31.560170 10279 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
I20260812 08:02:31.562762 10396 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:31.563645  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 10279
I20260812 08:02:31.563838  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/wal/instance
I20260812 08:02:31.566421 10396 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.152.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.61@KRBTEST.COM" created.
Aug 12 08:02:31 dist-test-slave-1nl6 krb5kdc[10258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521751, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
I20260812 08:02:31.585558 10396 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:02:31.586473 10308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.152.61:36687: connect: Connection refused (error 111)
W20260812 08:02:31.587881 10396 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:36687: Network error: Client connection negotiation failed: client connection to 127.5.152.61:36687: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:02:31.616037  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.61:36687
--webserver_interface=127.5.152.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:32837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:34319,127.5.152.61:36687,127.5.152.60:41815
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 08:02:31.637233 10396 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } attempt: 1
W20260812 08:02:31.638800 10396 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:36687: Network error: Client connection negotiation failed: client connection to 127.5.152.61:36687: connect: Connection refused (error 111)
I20260812 08:02:31.706969 10396 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } attempt: 2
W20260812 08:02:31.708547 10396 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:36687: Network error: Client connection negotiation failed: client connection to 127.5.152.61:36687: connect: Connection refused (error 111)
W20260812 08:02:31.713565 10410 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:31.713809 10410 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:31.713937 10410 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:31.717252 10410 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:31.717342 10410 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:31.717387 10410 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:31.717448 10410 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:31.723081 10410 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:32837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:34319,127.5.152.61:36687,127.5.152.60:41815
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.61:36687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.61
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:31.724124 10410 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:31.724956 10410 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:31.730278 10419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:31.730307 10418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:31.730278 10421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:31.730579 10410 server_base.cc:1061] running on GCE node
I20260812 08:02:31.731107 10410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:31.731912 10410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:31.733094 10410 hybrid_clock.cc:648] HybridClock initialized: now 1786521751733080 us; error 30 us; skew 500 ppm
Aug 12 08:02:31 dist-test-slave-1nl6 krb5kdc[10258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521751, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:02:31.735967 10410 init.cc:377] Logged in from keytab as kudu/127.5.152.61@KRBTEST.COM (short username kudu)
I20260812 08:02:31.747776 10410 webserver.cc:533] Webserver started at https://127.5.152.61:38599/ using document root <none> and password file <none>
I20260812 08:02:31.748394 10410 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:31.748471 10410 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:31.748682 10410 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:31.750334 10410 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/data/instance:
uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f"
format_stamp: "Formatted at 2026-08-12 08:02:31 on dist-test-slave-1nl6"
I20260812 08:02:31.750771 10410 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/wal/instance:
uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f"
format_stamp: "Formatted at 2026-08-12 08:02:31 on dist-test-slave-1nl6"
I20260812 08:02:31.754063 10410 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.001s
I20260812 08:02:31.756134 10434 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:31.757136 10410 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 08:02:31.757278 10410 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/wal
uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f"
format_stamp: "Formatted at 2026-08-12 08:02:31 on dist-test-slave-1nl6"
I20260812 08:02:31.757371 10410 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:31.773054 10410 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:31.775309 10410 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:31.775494 10410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:31.782606 10525 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.61:36687 every 8 connection(s)
I20260812 08:02:31.782603 10410 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.61:36687
I20260812 08:02:31.783562 10410 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
I20260812 08:02:31.786093 10526 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:31.789247 10526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:31.791145  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 10410
I20260812 08:02:31.791298  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/wal/instance
Aug 12 08:02:31 dist-test-slave-1nl6 krb5kdc[10258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521751, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.152.60@KRBTEST.COM; defaulting to no policy
I20260812 08:02:31.799707 10396 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.60@KRBTEST.COM" created.
I20260812 08:02:31.802420 10526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 08:02:31 dist-test-slave-1nl6 krb5kdc[10258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521751, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
Aug 12 08:02:31 dist-test-slave-1nl6 krb5kdc[10258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521751, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:02:31.812243 10396 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:31.812284 10526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:02:31.814041 10396 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:41815: Network error: Client connection negotiation failed: client connection to 127.5.152.60:41815: connect: Connection refused (error 111)
W20260812 08:02:31.814117 10435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.152.60:41815: connect: Connection refused (error 111)
W20260812 08:02:31.815008 10526 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:41815: Network error: Client connection negotiation failed: client connection to 127.5.152.60:41815: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.60@KRBTEST.COM" created.
I20260812 08:02:31.845209 10396 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } attempt: 1
W20260812 08:02:31.846693 10396 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:41815: Network error: Client connection negotiation failed: client connection to 127.5.152.60:41815: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:02:31.853299  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.60:41815
--webserver_interface=127.5.152.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:32837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:34319,127.5.152.61:36687,127.5.152.60:41815
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 08:02:31.864255 10526 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } attempt: 1
W20260812 08:02:31.865723 10526 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:41815: Network error: Client connection negotiation failed: client connection to 127.5.152.60:41815: connect: Connection refused (error 111)
I20260812 08:02:31.921906 10396 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } attempt: 2
W20260812 08:02:31.924050 10396 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:41815: Network error: Client connection negotiation failed: client connection to 127.5.152.60:41815: connect: Connection refused (error 111)
I20260812 08:02:31.933933 10526 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } attempt: 2
W20260812 08:02:31.936755 10526 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:41815: Network error: Client connection negotiation failed: client connection to 127.5.152.60:41815: connect: Connection refused (error 111)
W20260812 08:02:31.965932 10541 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:31.966169 10541 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:31.966275 10541 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:31.969601 10541 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:31.969691 10541 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:31.969735 10541 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:31.969770 10541 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:31.973918 10541 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:32837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:34319,127.5.152.61:36687,127.5.152.60:41815
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.60:41815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.60
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:31.974864 10541 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:31.975677 10541 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:31.980666 10547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:31.980738 10548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:31.980768 10553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:31.981129 10541 server_base.cc:1061] running on GCE node
I20260812 08:02:31.981587 10541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:31.982332 10541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:31.983481 10541 hybrid_clock.cc:648] HybridClock initialized: now 1786521751983460 us; error 31 us; skew 500 ppm
Aug 12 08:02:31 dist-test-slave-1nl6 krb5kdc[10258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521751, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:02:31.986163 10541 init.cc:377] Logged in from keytab as kudu/127.5.152.60@KRBTEST.COM (short username kudu)
I20260812 08:02:31.997706 10541 webserver.cc:533] Webserver started at https://127.5.152.60:39657/ using document root <none> and password file <none>
I20260812 08:02:31.998319 10541 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:31.998370 10541 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:31.998594 10541 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:32.000294 10541 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/data/instance:
uuid: "68a3fc96831943238f99f28717198a95"
format_stamp: "Formatted at 2026-08-12 08:02:31 on dist-test-slave-1nl6"
I20260812 08:02:32.000780 10541 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/wal/instance:
uuid: "68a3fc96831943238f99f28717198a95"
format_stamp: "Formatted at 2026-08-12 08:02:31 on dist-test-slave-1nl6"
I20260812 08:02:32.004186 10541 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 08:02:32.006225 10567 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:32.007295 10541 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 08:02:32.007438 10541 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/wal
uuid: "68a3fc96831943238f99f28717198a95"
format_stamp: "Formatted at 2026-08-12 08:02:31 on dist-test-slave-1nl6"
I20260812 08:02:32.007544 10541 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:32.018687 10541 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:32.021263 10541 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:32.021462 10541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:32.023208 10396 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } attempt: 3
W20260812 08:02:32.024576 10396 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:41815: Network error: Client connection negotiation failed: client connection to 127.5.152.60:41815: connect: Connection refused (error 111)
I20260812 08:02:32.027940 10526 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } attempt: 3
W20260812 08:02:32.029282 10526 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:41815: Network error: Client connection negotiation failed: client connection to 127.5.152.60:41815: connect: Connection refused (error 111)
I20260812 08:02:32.029796 10660 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.60:41815 every 8 connection(s)
I20260812 08:02:32.029803 10541 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.60:41815
I20260812 08:02:32.030782 10541 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
I20260812 08:02:32.033416 10661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:32.036861 10661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:32.039446  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 10541
I20260812 08:02:32.039584  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/wal/instance
Aug 12 08:02:32 dist-test-slave-1nl6 krb5kdc[10258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521751, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
Aug 12 08:02:32 dist-test-slave-1nl6 krb5kdc[10258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
I20260812 08:02:32.055811 10661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 08:02:32 dist-test-slave-1nl6 krb5kdc[10258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521751, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
Aug 12 08:02:32 dist-test-slave-1nl6 krb5kdc[10258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:02:32.067831 10661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 08:02:32 dist-test-slave-1nl6 krb5kdc[10258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521751, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
Aug 12 08:02:32 dist-test-slave-1nl6 krb5kdc[10258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:02:32.080224 10661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95: Bootstrap starting.
I20260812 08:02:32.082298 10661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:32.083150 10661 log.cc:826] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:32.084654 10661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95: No bootstrap required, opened a new log
I20260812 08:02:32.086997 10661 raft_consensus.cc:359] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877597133e56464eb892fa75844fccab" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } } peers { permanent_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } } peers { permanent_uuid: "68a3fc96831943238f99f28717198a95" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } }
I20260812 08:02:32.087153 10661 raft_consensus.cc:385] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:32.087225 10661 raft_consensus.cc:740] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a3fc96831943238f99f28717198a95, State: Initialized, Role: FOLLOWER
I20260812 08:02:32.087620 10661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877597133e56464eb892fa75844fccab" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } } peers { permanent_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } } peers { permanent_uuid: "68a3fc96831943238f99f28717198a95" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } }
I20260812 08:02:32.088058 10672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877597133e56464eb892fa75844fccab" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } } peers { permanent_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } } peers { permanent_uuid: "68a3fc96831943238f99f28717198a95" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } } }
I20260812 08:02:32.088181 10672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:32.088693 10661 sys_catalog.cc:565] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 08:02:32.093514 10689 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:02:32.093600 10689 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:02:32.092399 10541 master_runner.cc:187] Error getting master registration for 127.5.152.62:34319: OK, instance_id { permanent_uuid: "877597133e56464eb892fa75844fccab" instance_seqno: 1786521751557157 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 08:02:32.094156 10541 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 08:02:32.172487 10526 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } attempt: 4
Aug 12 08:02:32 dist-test-slave-1nl6 krb5kdc[10258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521751, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:02:32.185124 10526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f: Bootstrap starting.
I20260812 08:02:32.187580 10526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:32.188396 10526 log.cc:826] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:32.188786 10396 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } attempt: 4
I20260812 08:02:32.191383 10526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f: No bootstrap required, opened a new log
Aug 12 08:02:32 dist-test-slave-1nl6 krb5kdc[10258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521751, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:02:32.194753 10526 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877597133e56464eb892fa75844fccab" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } } peers { permanent_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } } peers { permanent_uuid: "68a3fc96831943238f99f28717198a95" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } }
I20260812 08:02:32.194981 10526 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:32.195032 10526 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad085d9ebfb4a75b4de8b09acdcd92f, State: Initialized, Role: FOLLOWER
I20260812 08:02:32.195525 10526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877597133e56464eb892fa75844fccab" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } } peers { permanent_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } } peers { permanent_uuid: "68a3fc96831943238f99f28717198a95" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } }
I20260812 08:02:32.196069 10694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877597133e56464eb892fa75844fccab" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } } peers { permanent_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } } peers { permanent_uuid: "68a3fc96831943238f99f28717198a95" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } } }
I20260812 08:02:32.196280 10694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:32.196766 10526 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:02:32.201565 10396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab: Bootstrap starting.
I20260812 08:02:32.200314 10410 master_runner.cc:187] Error getting master registration for 127.5.152.62:34319: OK, instance_id { permanent_uuid: "877597133e56464eb892fa75844fccab" instance_seqno: 1786521751557157 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 08:02:32.202520 10410 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 08:02:32.204236 10396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:32.205104 10396 log.cc:826] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:32.206602 10396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab: No bootstrap required, opened a new log
W20260812 08:02:32.208109 10712 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:02:32.208184 10712 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:02:32.210048 10396 raft_consensus.cc:359] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877597133e56464eb892fa75844fccab" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } } peers { permanent_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } } peers { permanent_uuid: "68a3fc96831943238f99f28717198a95" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } }
I20260812 08:02:32.210258 10396 raft_consensus.cc:385] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:32.210312 10396 raft_consensus.cc:740] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 877597133e56464eb892fa75844fccab, State: Initialized, Role: FOLLOWER
I20260812 08:02:32.210745 10396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877597133e56464eb892fa75844fccab" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } } peers { permanent_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } } peers { permanent_uuid: "68a3fc96831943238f99f28717198a95" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } }
I20260812 08:02:32.211262 10718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877597133e56464eb892fa75844fccab" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } } peers { permanent_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } } peers { permanent_uuid: "68a3fc96831943238f99f28717198a95" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } } }
I20260812 08:02:32.211457 10718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:32.212060 10396 sys_catalog.cc:565] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab [sys.catalog]: configured and running, proceeding with master startup.
W20260812 08:02:32.216693 10735 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:02:32.216820 10735 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:02:32.217528 10694 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:32.217679 10694 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877597133e56464eb892fa75844fccab" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } } peers { permanent_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } } peers { permanent_uuid: "68a3fc96831943238f99f28717198a95" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } }
I20260812 08:02:32.218565 10694 leader_election.cc:290] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 877597133e56464eb892fa75844fccab (127.5.152.62:34319), 68a3fc96831943238f99f28717198a95 (127.5.152.60:41815)
I20260812 08:02:32.218909 10351 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "877597133e56464eb892fa75844fccab" is_pre_election: true
I20260812 08:02:32.219095 10615 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3fc96831943238f99f28717198a95" is_pre_election: true
I20260812 08:02:32.219233 10351 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad085d9ebfb4a75b4de8b09acdcd92f in term 0.
I20260812 08:02:32.219329 10615 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad085d9ebfb4a75b4de8b09acdcd92f in term 0.
I20260812 08:02:32.219707 10439 leader_election.cc:304] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 68a3fc96831943238f99f28717198a95, 6ad085d9ebfb4a75b4de8b09acdcd92f, 877597133e56464eb892fa75844fccab; no voters: 
I20260812 08:02:32.219940 10694 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:02:32.220008 10694 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:32.220045 10694 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:32.221267 10694 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877597133e56464eb892fa75844fccab" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } } peers { permanent_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } } peers { permanent_uuid: "68a3fc96831943238f99f28717198a95" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } }
I20260812 08:02:32.221589 10694 leader_election.cc:290] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [CANDIDATE]: Term 1 election: Requested vote from peers 877597133e56464eb892fa75844fccab (127.5.152.62:34319), 68a3fc96831943238f99f28717198a95 (127.5.152.60:41815)
I20260812 08:02:32.221849 10351 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "877597133e56464eb892fa75844fccab"
I20260812 08:02:32.221958 10351 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:32.222088 10615 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a3fc96831943238f99f28717198a95"
I20260812 08:02:32.222188 10615 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:32.223075 10279 master_runner.cc:264] No leader master found from master 877597133e56464eb892fa75844fccab
I20260812 08:02:32.223147 10279 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 08:02:32.223332 10615 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ad085d9ebfb4a75b4de8b09acdcd92f in term 1.
I20260812 08:02:32.223388 10351 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ad085d9ebfb4a75b4de8b09acdcd92f in term 1.
I20260812 08:02:32.223652 10439 leader_election.cc:304] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad085d9ebfb4a75b4de8b09acdcd92f, 877597133e56464eb892fa75844fccab; no voters: 
I20260812 08:02:32.223933 10694 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:32.224149 10694 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [term 1 LEADER]: Becoming Leader. State: Replica: 6ad085d9ebfb4a75b4de8b09acdcd92f, State: Running, Role: LEADER
I20260812 08:02:32.224434 10694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877597133e56464eb892fa75844fccab" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } } peers { permanent_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } } peers { permanent_uuid: "68a3fc96831943238f99f28717198a95" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } }
I20260812 08:02:32.225965 10694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ad085d9ebfb4a75b4de8b09acdcd92f. Latest consensus state: current_term: 1 leader_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877597133e56464eb892fa75844fccab" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } } peers { permanent_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } } peers { permanent_uuid: "68a3fc96831943238f99f28717198a95" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } } }
I20260812 08:02:32.226056 10694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:32.228652 10742 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:02:32.229725 10742 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:02:32.234812 10615 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95 [term 1 FOLLOWER]: Refusing update from remote peer 6ad085d9ebfb4a75b4de8b09acdcd92f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:32.235055 10351 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab [term 1 FOLLOWER]: Refusing update from remote peer 6ad085d9ebfb4a75b4de8b09acdcd92f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:32.235605 10740 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "877597133e56464eb892fa75844fccab" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:32.235801 10694 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a3fc96831943238f99f28717198a95" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:32.239210 10672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ad085d9ebfb4a75b4de8b09acdcd92f. Latest consensus state: current_term: 1 leader_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877597133e56464eb892fa75844fccab" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } } peers { permanent_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } } peers { permanent_uuid: "68a3fc96831943238f99f28717198a95" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } } }
I20260812 08:02:32.239323 10672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:32.239326 10718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ad085d9ebfb4a75b4de8b09acdcd92f. Latest consensus state: current_term: 1 leader_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877597133e56464eb892fa75844fccab" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } } peers { permanent_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } } peers { permanent_uuid: "68a3fc96831943238f99f28717198a95" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } } }
I20260812 08:02:32.239429 10718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:32.242846 10740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877597133e56464eb892fa75844fccab" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } } peers { permanent_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } } peers { permanent_uuid: "68a3fc96831943238f99f28717198a95" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } } }
I20260812 08:02:32.242985 10740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:32.243304 10718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877597133e56464eb892fa75844fccab" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } } peers { permanent_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } } peers { permanent_uuid: "68a3fc96831943238f99f28717198a95" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } } }
I20260812 08:02:32.243395 10718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:32.243567 10744 mvcc.cc:204] Tried to move back new op lower bound from 7317593097146597376 to 7317593097114017792. Current Snapshot: MvccSnapshot[applied={T|T < 7317593097146597376}]
I20260812 08:02:32.243606 10745 mvcc.cc:204] Tried to move back new op lower bound from 7317593097146597376 to 7317593097114017792. Current Snapshot: MvccSnapshot[applied={T|T < 7317593097146597376}]
I20260812 08:02:32.248399 10742 catalog_manager.cc:1383] Generated new cluster ID: e81e7a5e4f274675bee44078d9f808fe
I20260812 08:02:32.248457 10742 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:02:32.252547 10740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877597133e56464eb892fa75844fccab" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } } peers { permanent_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } } peers { permanent_uuid: "68a3fc96831943238f99f28717198a95" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } } }
I20260812 08:02:32.252661 10740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:32.253716 10672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877597133e56464eb892fa75844fccab" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 34319 } } peers { permanent_uuid: "6ad085d9ebfb4a75b4de8b09acdcd92f" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 36687 } } peers { permanent_uuid: "68a3fc96831943238f99f28717198a95" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 41815 } } }
I20260812 08:02:32.253818 10672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:32.254329 10747 mvcc.cc:204] Tried to move back new op lower bound from 7317593097146597376 to 7317593097114017792. Current Snapshot: MvccSnapshot[applied={T|T < 7317593097146597376}]
I20260812 08:02:32.272293 10742 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:02:32.273816 10742 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:02:32.290498 10742 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 6ad085d9ebfb4a75b4de8b09acdcd92f: Generated new TSK 0
I20260812 08:02:32.291424 10742 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:02:32.374061  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.152.1:0
--local_ip_for_outbound_sockets=127.5.152.1
--webserver_interface=127.5.152.1
--webserver_port=0
--tserver_master_addrs=127.5.152.62:34319,127.5.152.61:36687,127.5.152.60:41815
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:32837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 08:02:32.511560 10766 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:32.511797 10766 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:32.511869 10766 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:32.515995 10766 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:32.516125 10766 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.152.1
I20260812 08:02:32.520740 10766 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:32837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.1
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--tserver_master_addrs=127.5.152.62:34319,127.5.152.61:36687,127.5.152.60:41815
--never_fsync=true
--heap_profile_path=/tmp/kudu.10766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.152.1

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:32.521785 10766 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:32.522617 10766 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:32.528761 10777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:32.528823 10779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:32.528759 10776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:32.529083 10766 server_base.cc:1061] running on GCE node
I20260812 08:02:32.529423 10766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:32.529872 10766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:32.531013 10766 hybrid_clock.cc:648] HybridClock initialized: now 1786521752530993 us; error 34 us; skew 500 ppm
Aug 12 08:02:32 dist-test-slave-1nl6 krb5kdc[10258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521752, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:02:32.533694 10766 init.cc:377] Logged in from keytab as kudu/127.5.152.1@KRBTEST.COM (short username kudu)
I20260812 08:02:32.544685 10766 webserver.cc:533] Webserver started at https://127.5.152.1:40369/ using document root <none> and password file <none>
I20260812 08:02:32.545264 10766 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:32.545322 10766 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:32.545498 10766 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:32.547134 10766 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/data/instance:
uuid: "f86060d2c24f479e805857e0c839232d"
format_stamp: "Formatted at 2026-08-12 08:02:32 on dist-test-slave-1nl6"
I20260812 08:02:32.547562 10766 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/wal/instance:
uuid: "f86060d2c24f479e805857e0c839232d"
format_stamp: "Formatted at 2026-08-12 08:02:32 on dist-test-slave-1nl6"
I20260812 08:02:32.550733 10766 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.001s
I20260812 08:02:32.552779 10792 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:32.553696 10766 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:02:32.553791 10766 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/wal
uuid: "f86060d2c24f479e805857e0c839232d"
format_stamp: "Formatted at 2026-08-12 08:02:32 on dist-test-slave-1nl6"
I20260812 08:02:32.553862 10766 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:32.563628 10766 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:32.566242 10766 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:32.566447 10766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:32.567131 10766 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:02:32.568011 10766 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:32.568080 10766 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:32.568142 10766 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:32.568192 10766 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:32.577832 10766 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.1:46397
I20260812 08:02:32.577854 10973 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.1:46397 every 8 connection(s)
I20260812 08:02:32.578681 10766 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
I20260812 08:02:32.581472  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 10766
I20260812 08:02:32.581589  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/wal/instance
Aug 12 08:02:32 dist-test-slave-1nl6 krb5kdc[10258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521752, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
Aug 12 08:02:32 dist-test-slave-1nl6 krb5kdc[10258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521752, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
Aug 12 08:02:32 dist-test-slave-1nl6 krb5kdc[10258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521752, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
I20260812 08:02:32.595587 10977 heartbeater.cc:344] Connected to a master server at 127.5.152.62:34319
I20260812 08:02:32.595607 10975 heartbeater.cc:344] Connected to a master server at 127.5.152.60:41815
I20260812 08:02:32.595870 10977 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:32.595872 10975 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:32.596408 10977 heartbeater.cc:507] Master 127.5.152.62:34319 requested a full tablet report, sending...
I20260812 08:02:32.596408 10975 heartbeater.cc:507] Master 127.5.152.60:41815 requested a full tablet report, sending...
I20260812 08:02:32.596982 10981 heartbeater.cc:344] Connected to a master server at 127.5.152.61:36687
I20260812 08:02:32.597062 10981 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:32.597242 10981 heartbeater.cc:507] Master 127.5.152.61:36687 requested a full tablet report, sending...
I20260812 08:02:32.598061 10592 ts_manager.cc:194] Registered new tserver with Master: f86060d2c24f479e805857e0c839232d (127.5.152.1:46397)
I20260812 08:02:32.598061 10338 ts_manager.cc:194] Registered new tserver with Master: f86060d2c24f479e805857e0c839232d (127.5.152.1:46397)
I20260812 08:02:32.599319 10464 ts_manager.cc:194] Registered new tserver with Master: f86060d2c24f479e805857e0c839232d (127.5.152.1:46397)
I20260812 08:02:32.600602 10464 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.152.1@KRBTEST.COM'} at 127.5.152.1:57251
I20260812 08:02:32.603971  5728 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
time=2026-08-12T08:02:32.680Z level=INFO source=main.go:1678 msg="updated GOGC" old=100 new=75
time=2026-08-12T08:02:32.682Z level=INFO source=main.go:744 msg="Leaving GOMAXPROCS=4: CPU quota undefined" component=automaxprocs
time=2026-08-12T08:02:32.682Z level=INFO source=memlimit.go:198 msg="GOMEMLIMIT is updated" component=automemlimit package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=14142549196 previous=9223372036854775807
time=2026-08-12T08:02:32.682Z level=INFO source=main.go:851 msg="Starting Prometheus Server" mode=server version="(version=3.11.2, branch=HEAD, revision=f0f0fdd679dcd6df320b0558b20919f7cd44c407)"
time=2026-08-12T08:02:32.683Z level=INFO source=main.go:856 msg="operational information" build_context="(go=go1.26.2, platform=linux/amd64, user=root@d2684485c347, date=20260413-11:53:40, tags=netgo,builtinassets)" host_details="(Linux 5.4.0-1157-gcp #166~18.04.1-Ubuntu SMP Wed Jan 21 01:57:46 UTC 2026 x86_64 dist-test-slave-1nl6 (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)"
time=2026-08-12T08:02:32.950Z level=INFO source=web.go:710 msg="Start listening for connections" component=web address=127.0.0.1:0
time=2026-08-12T08:02:32.951Z level=INFO source=main.go:1410 msg="Starting TSDB ..."
time=2026-08-12T08:02:32.952Z level=INFO source=tls_config.go:354 msg="Listening on" component=web address=127.0.0.1:42659
time=2026-08-12T08:02:32.952Z level=INFO source=tls_config.go:357 msg="TLS is disabled." component=web http2=false address=127.0.0.1:42659
time=2026-08-12T08:02:32.954Z level=INFO source=head.go:698 msg="Replaying on-disk memory mappable chunks if any" component=tsdb
time=2026-08-12T08:02:32.954Z level=INFO source=head.go:784 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=1.49µs
time=2026-08-12T08:02:32.954Z level=INFO source=head.go:792 msg="Replaying WAL, this may take a while" component=tsdb
time=2026-08-12T08:02:32.955Z level=INFO source=head.go:865 msg="WAL segment loaded" component=tsdb segment=0 maxSegment=0 duration=366.574µs
time=2026-08-12T08:02:32.955Z level=INFO source=head.go:902 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=67.352µs wal_replay_duration=459.939µs wbl_replay_duration=460ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=1.49µs total_replay_duration=563.413µs
time=2026-08-12T08:02:32.960Z level=INFO source=main.go:1431 msg="filesystem information" fs_type=OVERLAYFS_SUPER_MAGIC
time=2026-08-12T08:02:32.960Z level=INFO source=main.go:1434 msg="TSDB started"
time=2026-08-12T08:02:32.960Z level=INFO source=main.go:1632 msg="Loading configuration file" filename=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/prometheus.yml
time=2026-08-12T08:02:32.962Z level=INFO source=main.go:1048 msg="TSDB retention updated" duration=15d size=0B percentage=0
time=2026-08-12T08:02:32.969Z level=INFO source=main.go:1671 msg="Completed loading of configuration file" db_storage=92.216µs remote_storage=4.443µs web_handler=1.147µs query_engine=1.568µs scrape=6.596963ms scrape_sd=272.962µs notify=2.74µs notify_sd=1.851µs rules=3.364µs tracing=6.989µs filename=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786521680053065-5728-0/prometheus.yml totalDuration=8.878419ms
time=2026-08-12T08:02:32.969Z level=INFO source=main.go:1395 msg="Server is ready to receive web requests."
time=2026-08-12T08:02:32.969Z level=INFO source=manager.go:209 msg="Starting rule manager..." component="rule manager"
W20260812 08:02:32.993184 11009 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:57632: invalid bearer token
time=2026-08-12T08:02:32.994Z level=ERROR source=refresh.go:72 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
W20260812 08:02:32.994513 11008 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:57120: invalid bearer token
time=2026-08-12T08:02:32.995Z level=ERROR source=refresh.go:72 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
W20260812 08:02:33.001121 11010 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:50034: invalid bearer token
time=2026-08-12T08:02:33.001Z level=ERROR source=refresh.go:72 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
I20260812 08:02:33.095038 10689 catalog_manager.cc:1295] Loaded cluster ID: e81e7a5e4f274675bee44078d9f808fe
I20260812 08:02:33.095137 10689 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95: loading cluster ID for follower catalog manager: success
I20260812 08:02:33.096845 10689 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95: acquiring CA information for follower catalog manager: success
I20260812 08:02:33.097954 10689 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 68a3fc96831943238f99f28717198a95: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 08:02:33.218204 10735 catalog_manager.cc:1295] Loaded cluster ID: e81e7a5e4f274675bee44078d9f808fe
I20260812 08:02:33.218328 10735 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab: loading cluster ID for follower catalog manager: success
I20260812 08:02:33.219978 10735 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab: acquiring CA information for follower catalog manager: success
I20260812 08:02:33.221040 10735 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 877597133e56464eb892fa75844fccab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 08:02:33.603385 10981 heartbeater.cc:499] Master 127.5.152.61:36687 was elected leader, sending a full tablet report...
W20260812 08:02:34.994796 11009 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:57632: invalid bearer token
time=2026-08-12T08:02:34.995Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
W20260812 08:02:34.995463 11008 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:57120: invalid bearer token
time=2026-08-12T08:02:34.995Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
W20260812 08:02:35.002301 11010 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:50034: invalid bearer token
time=2026-08-12T08:02:35.002Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
W20260812 08:02:36.995419 11009 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:57632: invalid bearer token
W20260812 08:02:36.995433 11008 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:57120: invalid bearer token
time=2026-08-12T08:02:36.995Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:02:36.995Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
W20260812 08:02:37.002233 11010 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:50034: invalid bearer token
time=2026-08-12T08:02:37.002Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
W20260812 08:02:38.994783 11009 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:57632: invalid bearer token
time=2026-08-12T08:02:38.995Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
W20260812 08:02:38.995545 11008 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:57120: invalid bearer token
time=2026-08-12T08:02:38.995Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
W20260812 08:02:39.002329 11010 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:50034: invalid bearer token
time=2026-08-12T08:02:39.002Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:02:40.917Z level=WARN source=main.go:1191 msg="Received an OS signal, exiting gracefully..." signal=terminated
time=2026-08-12T08:02:40.917Z level=INFO source=main.go:1216 msg="Stopping scrape discovery manager..."
time=2026-08-12T08:02:40.917Z level=INFO source=main.go:1230 msg="Stopping notify discovery manager..."
time=2026-08-12T08:02:40.917Z level=INFO source=main.go:1245 msg="Stopping rule manager manager..."
time=2026-08-12T08:02:40.917Z level=INFO source=manager.go:223 msg="Stopping rule manager..." component="rule manager"
time=2026-08-12T08:02:40.917Z level=INFO source=manager.go:239 msg="Rule manager stopped" component="rule manager"
time=2026-08-12T08:02:40.917Z level=INFO source=main.go:1269 msg="Stopping scrape manager..."
time=2026-08-12T08:02:40.917Z level=INFO source=main.go:1284 msg="Stopping tracing manager..."
time=2026-08-12T08:02:40.917Z level=INFO source=main.go:1369 msg="Stopping reloaders..."
time=2026-08-12T08:02:40.917Z level=INFO source=main.go:1458 msg="Stopping storage..."
time=2026-08-12T08:02:40.918Z level=INFO source=main.go:1212 msg="Scrape discovery manager stopped"
time=2026-08-12T08:02:40.919Z level=INFO source=main.go:1226 msg="Notify discovery manager stopped"
time=2026-08-12T08:02:40.919Z level=INFO source=main.go:1241 msg="Rule manager stopped"
time=2026-08-12T08:02:40.919Z level=INFO source=main.go:1261 msg="Scrape manager stopped"
time=2026-08-12T08:02:40.919Z level=INFO source=main.go:1280 msg="Tracing manager stopped"
time=2026-08-12T08:02:40.919Z level=INFO source=main.go:1361 msg="Reloaders stopped"
time=2026-08-12T08:02:40.919Z level=INFO source=main.go:1536 msg="Stopping web handler..."
time=2026-08-12T08:02:40.919Z level=INFO source=main.go:1559 msg="Stopping notifier manager..."
time=2026-08-12T08:02:40.919Z level=INFO source=manager.go:321 msg="Stopping notification manager..." component=notifier
time=2026-08-12T08:02:40.919Z level=INFO source=main.go:1555 msg="Notifier manager stopped"
time=2026-08-12T08:02:40.919Z level=INFO source=main.go:1454 msg="TSDB stopped"
time=2026-08-12T08:02:40.919Z level=INFO source=main.go:1532 msg="Web handler stopped"
time=2026-08-12T08:02:40.919Z level=INFO source=main.go:1570 msg="See you next time!"
I20260812 08:02:40.938037  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 10766
I20260812 08:02:40.952322  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 10279
I20260812 08:02:40.971540  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 10410
I20260812 08:02:40.991403  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 10541
2026-08-12T08:02:41Z chronyd exiting
[       OK ] PrometheusTokenITest.TestWrongTokenSdFails (12070 ms)
[ RUN      ] PrometheusTokenITest.TestMissingTokenSdFails
Loading random data
Initializing database '/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 08:02:41 dist-test-slave-1nl6 krb5kdc[11016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 08:02:41 dist-test-slave-1nl6 krb5kdc[11016](info): set up 2 sockets
Aug 12 08:02:41 dist-test-slave-1nl6 krb5kdc[11016](info): commencing operation
krb5kdc: starting...
W20260812 08:02:43.473978  5728 mini_kdc.cc:121] Time spent starting KDC: real 2.436s	user 0.000s	sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/test-user.keytab.
Aug 12 08:02:43 dist-test-slave-1nl6 krb5kdc[11016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM: 
2026-08-12T08:02:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T08:02:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:02:43.681375  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.62:38581
--webserver_interface=127.5.152.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:38581,127.5.152.61:34787,127.5.152.60:39373
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 08:02:43.814990 11041 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:43.815460 11041 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:43.815590 11041 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:43.820127 11041 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:43.820277 11041 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:43.820320 11041 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:43.820365 11041 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:43.827039 11041 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:38581,127.5.152.61:34787,127.5.152.60:39373
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.62:38581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.62
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:43.828464 11041 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:43.829473 11041 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:43.839751 11052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:43.839771 11047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:43.839728 11050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:43.840513 11041 server_base.cc:1061] running on GCE node
I20260812 08:02:43.841250 11041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:43.842777 11041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:43.843981 11041 hybrid_clock.cc:648] HybridClock initialized: now 1786521763843954 us; error 49 us; skew 500 ppm
Aug 12 08:02:43 dist-test-slave-1nl6 krb5kdc[11016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521763, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:02:43.848248 11041 init.cc:377] Logged in from keytab as kudu/127.5.152.62@KRBTEST.COM (short username kudu)
I20260812 08:02:43.865064 11041 webserver.cc:533] Webserver started at https://127.5.152.62:34073/ using document root <none> and password file <none>
I20260812 08:02:43.865873 11041 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:43.865937 11041 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:43.866155 11041 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:43.868327 11041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/data/instance:
uuid: "a11b8d701c6e469281f62869f9845601"
format_stamp: "Formatted at 2026-08-12 08:02:43 on dist-test-slave-1nl6"
I20260812 08:02:43.868888 11041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/wal/instance:
uuid: "a11b8d701c6e469281f62869f9845601"
format_stamp: "Formatted at 2026-08-12 08:02:43 on dist-test-slave-1nl6"
I20260812 08:02:43.873312 11041 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260812 08:02:43.876572 11066 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:43.878451 11041 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20260812 08:02:43.878649 11041 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/wal
uuid: "a11b8d701c6e469281f62869f9845601"
format_stamp: "Formatted at 2026-08-12 08:02:43 on dist-test-slave-1nl6"
I20260812 08:02:43.878885 11041 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:43.904031 11041 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:43.908123 11041 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:43.908480 11041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:43.919260 11041 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.62:38581
I20260812 08:02:43.919432 11146 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.62:38581 every 8 connection(s)
I20260812 08:02:43.921195 11041 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/data/info.pb
I20260812 08:02:43.922618  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 11041
I20260812 08:02:43.922886  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-0/wal/instance
I20260812 08:02:43.925849 11150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:43.930502 11150 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.152.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.61@KRBTEST.COM" created.
Aug 12 08:02:43 dist-test-slave-1nl6 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521763, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
I20260812 08:02:43.951151 11150 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:02:43.952672 11067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.152.61:34787: connect: Connection refused (error 111)
W20260812 08:02:43.954252 11150 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:34787: Network error: Client connection negotiation failed: client connection to 127.5.152.61:34787: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:02:43.992304  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.61:34787
--webserver_interface=127.5.152.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:38581,127.5.152.61:34787,127.5.152.60:39373
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 08:02:44.003655 11150 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } attempt: 1
W20260812 08:02:44.006522 11150 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:34787: Network error: Client connection negotiation failed: client connection to 127.5.152.61:34787: connect: Connection refused (error 111)
I20260812 08:02:44.075007 11150 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } attempt: 2
W20260812 08:02:44.077945 11150 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:34787: Network error: Client connection negotiation failed: client connection to 127.5.152.61:34787: connect: Connection refused (error 111)
W20260812 08:02:44.133379 11168 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:44.133693 11168 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:44.133762 11168 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:44.137712 11168 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:44.137840 11168 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:44.137883 11168 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:44.137920 11168 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:44.145864 11168 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:38581,127.5.152.61:34787,127.5.152.60:39373
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.61:34787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.61
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:44.147236 11168 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:44.148211 11168 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:44.156251 11175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:44.156251 11178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:44.156318 11174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:44.156728 11168 server_base.cc:1061] running on GCE node
I20260812 08:02:44.157591 11168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:44.158963 11168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:44.160300 11168 hybrid_clock.cc:648] HybridClock initialized: now 1786521764160310 us; error 69 us; skew 500 ppm
Aug 12 08:02:44 dist-test-slave-1nl6 krb5kdc[11016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521764, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:02:44.165052 11168 init.cc:377] Logged in from keytab as kudu/127.5.152.61@KRBTEST.COM (short username kudu)
I20260812 08:02:44.169248 11150 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } attempt: 3
W20260812 08:02:44.171860 11150 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.61:34787: Network error: Client connection negotiation failed: client connection to 127.5.152.61:34787: connect: Connection refused (error 111)
I20260812 08:02:44.185045 11168 webserver.cc:533] Webserver started at https://127.5.152.61:37819/ using document root <none> and password file <none>
I20260812 08:02:44.185956 11168 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:44.186062 11168 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:44.186349 11168 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:44.188484 11168 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/data/instance:
uuid: "0ec34e03c53a47798cee73b7340f0fa6"
format_stamp: "Formatted at 2026-08-12 08:02:44 on dist-test-slave-1nl6"
I20260812 08:02:44.189083 11168 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/wal/instance:
uuid: "0ec34e03c53a47798cee73b7340f0fa6"
format_stamp: "Formatted at 2026-08-12 08:02:44 on dist-test-slave-1nl6"
I20260812 08:02:44.195014 11168 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.000s
I20260812 08:02:44.198665 11192 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:44.200765 11168 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20260812 08:02:44.201087 11168 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/wal
uuid: "0ec34e03c53a47798cee73b7340f0fa6"
format_stamp: "Formatted at 2026-08-12 08:02:44 on dist-test-slave-1nl6"
I20260812 08:02:44.201298 11168 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:44.230728 11168 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:44.235589 11168 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:44.235937 11168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:44.246179 11168 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.61:34787
I20260812 08:02:44.246179 11277 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.61:34787 every 8 connection(s)
I20260812 08:02:44.247570 11168 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/data/info.pb
I20260812 08:02:44.252604 11282 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:44.253720  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 11168
I20260812 08:02:44.253880  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-1/wal/instance
I20260812 08:02:44.257970 11282 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.152.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.60@KRBTEST.COM" created.
Aug 12 08:02:44 dist-test-slave-1nl6 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521764, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
I20260812 08:02:44.280256 11282 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 08:02:44 dist-test-slave-1nl6 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521764, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:02:44.302456 11282 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:02:44.304301 11193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.152.60:39373: connect: Connection refused (error 111)
W20260812 08:02:44.305649 11282 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:39373: Network error: Client connection negotiation failed: client connection to 127.5.152.60:39373: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.5.152.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.60@KRBTEST.COM" created.
I20260812 08:02:44.315109 11150 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } attempt: 4
Aug 12 08:02:44 dist-test-slave-1nl6 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521763, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:02:44.328284 11150 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } has no permanent_uuid. Determining permanent_uuid...
W20260812 08:02:44.330128 11150 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:39373: Network error: Client connection negotiation failed: client connection to 127.5.152.60:39373: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:02:44.335963  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.60:39373
--webserver_interface=127.5.152.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.152.62:38581,127.5.152.61:34787,127.5.152.60:39373
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 08:02:44.355037 11282 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } attempt: 1
W20260812 08:02:44.357339 11282 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:39373: Network error: Client connection negotiation failed: client connection to 127.5.152.60:39373: connect: Connection refused (error 111)
I20260812 08:02:44.389379 11150 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } attempt: 1
W20260812 08:02:44.391353 11150 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:39373: Network error: Client connection negotiation failed: client connection to 127.5.152.60:39373: connect: Connection refused (error 111)
I20260812 08:02:44.425612 11282 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } attempt: 2
W20260812 08:02:44.428076 11282 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:39373: Network error: Client connection negotiation failed: client connection to 127.5.152.60:39373: connect: Connection refused (error 111)
I20260812 08:02:44.458622 11150 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } attempt: 2
W20260812 08:02:44.460410 11150 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:39373: Network error: Client connection negotiation failed: client connection to 127.5.152.60:39373: connect: Connection refused (error 111)
W20260812 08:02:44.477800 11295 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:44.478241 11295 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:44.478369 11295 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:44.483608 11295 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 08:02:44.483733 11295 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:44.483762 11295 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 08:02:44.483784 11295 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 08:02:44.489668 11295 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.152.62:38581,127.5.152.61:34787,127.5.152.60:39373
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.152.60:39373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.60
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:44.491127 11295 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:44.492635 11295 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:44.500255 11305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:44.500334 11306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:44.500792 11295 server_base.cc:1061] running on GCE node
W20260812 08:02:44.501466 11311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:44.502231 11295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:44.503455 11295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:44.504619 11295 hybrid_clock.cc:648] HybridClock initialized: now 1786521764504600 us; error 47 us; skew 500 ppm
Aug 12 08:02:44 dist-test-slave-1nl6 krb5kdc[11016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521764, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:02:44.508766 11295 init.cc:377] Logged in from keytab as kudu/127.5.152.60@KRBTEST.COM (short username kudu)
I20260812 08:02:44.519454 11282 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } attempt: 3
W20260812 08:02:44.522317 11282 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:39373: Network error: Client connection negotiation failed: client connection to 127.5.152.60:39373: connect: Connection refused (error 111)
I20260812 08:02:44.527004 11295 webserver.cc:533] Webserver started at https://127.5.152.60:33445/ using document root <none> and password file <none>
I20260812 08:02:44.527850 11295 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:44.527922 11295 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:44.528213 11295 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:44.530315 11295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/data/instance:
uuid: "a47f74d27e1942b7a3669ce721d6dbe3"
format_stamp: "Formatted at 2026-08-12 08:02:44 on dist-test-slave-1nl6"
I20260812 08:02:44.531142 11295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/wal/instance:
uuid: "a47f74d27e1942b7a3669ce721d6dbe3"
format_stamp: "Formatted at 2026-08-12 08:02:44 on dist-test-slave-1nl6"
I20260812 08:02:44.536811 11295 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.001s
I20260812 08:02:44.541018 11325 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:44.542729 11295 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.000s
I20260812 08:02:44.543030 11295 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/wal
uuid: "a47f74d27e1942b7a3669ce721d6dbe3"
format_stamp: "Formatted at 2026-08-12 08:02:44 on dist-test-slave-1nl6"
I20260812 08:02:44.543196 11295 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:44.560768 11150 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } attempt: 3
W20260812 08:02:44.562803 11150 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.5.152.60:39373: Network error: Client connection negotiation failed: client connection to 127.5.152.60:39373: connect: Connection refused (error 111)
I20260812 08:02:44.569139 11295 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:44.573639 11295 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:44.573953 11295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:44.586030 11405 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.60:39373 every 8 connection(s)
I20260812 08:02:44.586026 11295 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.60:39373
I20260812 08:02:44.587560 11295 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/data/info.pb
I20260812 08:02:44.591681 11406 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:02:44.596506 11406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } has no permanent_uuid. Determining permanent_uuid...
I20260812 08:02:44.597170  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 11295
I20260812 08:02:44.597357  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/master-2/wal/instance
Aug 12 08:02:44 dist-test-slave-1nl6 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
Aug 12 08:02:44 dist-test-slave-1nl6 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521764, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
Aug 12 08:02:44 dist-test-slave-1nl6 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
I20260812 08:02:44.619468 11406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 08:02:44 dist-test-slave-1nl6 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521764, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
Aug 12 08:02:44 dist-test-slave-1nl6 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:02:44.633944 11406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 08:02:44 dist-test-slave-1nl6 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521764, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.60@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:02:44.648492 11406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3: Bootstrap starting.
I20260812 08:02:44.651654 11406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:44.653035 11406 log.cc:826] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:44.655385 11406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3: No bootstrap required, opened a new log
I20260812 08:02:44.658669 11406 raft_consensus.cc:359] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b8d701c6e469281f62869f9845601" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } } peers { permanent_uuid: "0ec34e03c53a47798cee73b7340f0fa6" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } } peers { permanent_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } }
I20260812 08:02:44.658972 11406 raft_consensus.cc:385] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:44.659082 11406 raft_consensus.cc:740] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47f74d27e1942b7a3669ce721d6dbe3, State: Initialized, Role: FOLLOWER
I20260812 08:02:44.659613 11406 consensus_queue.cc:260] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b8d701c6e469281f62869f9845601" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } } peers { permanent_uuid: "0ec34e03c53a47798cee73b7340f0fa6" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } } peers { permanent_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } }
I20260812 08:02:44.660312 11418 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b8d701c6e469281f62869f9845601" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } } peers { permanent_uuid: "0ec34e03c53a47798cee73b7340f0fa6" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } } peers { permanent_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } } }
I20260812 08:02:44.660470 11418 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:44.661379 11406 sys_catalog.cc:565] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:02:44.665587 11282 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } attempt: 4
W20260812 08:02:44.667369 11440 catalog_manager.cc:1594] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:02:44.667480 11440 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:02:44.665838 11295 master_runner.cc:187] Error getting master registration for 127.5.152.62:38581: OK, instance_id { permanent_uuid: "a11b8d701c6e469281f62869f9845601" instance_seqno: 1786521763915737 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 08:02:44.669349 11295 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Aug 12 08:02:44 dist-test-slave-1nl6 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521764, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.61@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:02:44.680034 11282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6: Bootstrap starting.
I20260812 08:02:44.682536 11282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:44.684487 11282 log.cc:826] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:44.686566 11282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6: No bootstrap required, opened a new log
I20260812 08:02:44.689675 11282 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b8d701c6e469281f62869f9845601" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } } peers { permanent_uuid: "0ec34e03c53a47798cee73b7340f0fa6" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } } peers { permanent_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } }
I20260812 08:02:44.689916 11282 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:44.689994 11282 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec34e03c53a47798cee73b7340f0fa6, State: Initialized, Role: FOLLOWER
I20260812 08:02:44.690513 11282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b8d701c6e469281f62869f9845601" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } } peers { permanent_uuid: "0ec34e03c53a47798cee73b7340f0fa6" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } } peers { permanent_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } }
I20260812 08:02:44.691126 11447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b8d701c6e469281f62869f9845601" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } } peers { permanent_uuid: "0ec34e03c53a47798cee73b7340f0fa6" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } } peers { permanent_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } } }
I20260812 08:02:44.691279 11447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:44.692114 11282 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 08:02:44.699342 11468 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 08:02:44.699586 11468 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 08:02:44.697487 11168 master_runner.cc:187] Error getting master registration for 127.5.152.62:38581: OK, instance_id { permanent_uuid: "a11b8d701c6e469281f62869f9845601" instance_seqno: 1786521763915737 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 08:02:44.701040 11168 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 08:02:44.702005 11418 raft_consensus.cc:493] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 08:02:44.702286 11418 raft_consensus.cc:515] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b8d701c6e469281f62869f9845601" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } } peers { permanent_uuid: "0ec34e03c53a47798cee73b7340f0fa6" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } } peers { permanent_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } }
I20260812 08:02:44.703593 11418 leader_election.cc:290] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a11b8d701c6e469281f62869f9845601 (127.5.152.62:38581), 0ec34e03c53a47798cee73b7340f0fa6 (127.5.152.61:34787)
I20260812 08:02:44.704475 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec34e03c53a47798cee73b7340f0fa6" is_pre_election: true
I20260812 08:02:44.704885 11234 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47f74d27e1942b7a3669ce721d6dbe3 in term 0.
I20260812 08:02:44.705415 11326 leader_election.cc:304] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec34e03c53a47798cee73b7340f0fa6, a47f74d27e1942b7a3669ce721d6dbe3; no voters: 
I20260812 08:02:44.705735 11418 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 08:02:44.705823 11418 raft_consensus.cc:493] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 08:02:44.705864 11418 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:44.707197 11418 raft_consensus.cc:515] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b8d701c6e469281f62869f9845601" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } } peers { permanent_uuid: "0ec34e03c53a47798cee73b7340f0fa6" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } } peers { permanent_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } }
I20260812 08:02:44.708860 11418 leader_election.cc:290] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [CANDIDATE]: Term 1 election: Requested vote from peers a11b8d701c6e469281f62869f9845601 (127.5.152.62:38581), 0ec34e03c53a47798cee73b7340f0fa6 (127.5.152.61:34787)
I20260812 08:02:44.709120 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec34e03c53a47798cee73b7340f0fa6"
I20260812 08:02:44.709268 11234 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:44.710632 11234 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47f74d27e1942b7a3669ce721d6dbe3 in term 1.
I20260812 08:02:44.711663 11326 leader_election.cc:304] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec34e03c53a47798cee73b7340f0fa6, a47f74d27e1942b7a3669ce721d6dbe3; no voters: 
I20260812 08:02:44.711997 11418 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:02:44.712466 11418 raft_consensus.cc:697] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [term 1 LEADER]: Becoming Leader. State: Replica: a47f74d27e1942b7a3669ce721d6dbe3, State: Running, Role: LEADER
I20260812 08:02:44.712913 11418 consensus_queue.cc:237] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b8d701c6e469281f62869f9845601" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } } peers { permanent_uuid: "0ec34e03c53a47798cee73b7340f0fa6" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } } peers { permanent_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } }
I20260812 08:02:44.715487 11470 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a47f74d27e1942b7a3669ce721d6dbe3. Latest consensus state: current_term: 1 leader_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b8d701c6e469281f62869f9845601" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } } peers { permanent_uuid: "0ec34e03c53a47798cee73b7340f0fa6" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } } peers { permanent_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } } }
I20260812 08:02:44.715655 11470 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:44.716347 11472 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:02:44.719502 11472 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:02:44.729439 11234 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6 [term 1 FOLLOWER]: Refusing update from remote peer a47f74d27e1942b7a3669ce721d6dbe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:44.730516 11470 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec34e03c53a47798cee73b7340f0fa6" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:44.732998 11150 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } attempt: 4
W20260812 08:02:44.738224 11202 tablet.cc:2406] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 08:02:44.738188 11447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a47f74d27e1942b7a3669ce721d6dbe3. Latest consensus state: current_term: 1 leader_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b8d701c6e469281f62869f9845601" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } } peers { permanent_uuid: "0ec34e03c53a47798cee73b7340f0fa6" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } } peers { permanent_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } } }
I20260812 08:02:44.740963 11447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6 [sys.catalog]: This master's current role is: FOLLOWER
Aug 12 08:02:44 dist-test-slave-1nl6 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521763, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.62@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:02:44.748760 11418 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b8d701c6e469281f62869f9845601" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } } peers { permanent_uuid: "0ec34e03c53a47798cee73b7340f0fa6" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } } peers { permanent_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } } }
I20260812 08:02:44.748960 11418 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:44.753540 11472 catalog_manager.cc:1383] Generated new cluster ID: 496241a8e0ff4bf585575d486d3dedfe
I20260812 08:02:44.753626 11472 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:02:44.753933 11150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a11b8d701c6e469281f62869f9845601: Bootstrap starting.
I20260812 08:02:44.755420 11447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b8d701c6e469281f62869f9845601" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } } peers { permanent_uuid: "0ec34e03c53a47798cee73b7340f0fa6" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } } peers { permanent_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } } }
I20260812 08:02:44.755586 11447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:44.757189 11473 mvcc.cc:204] Tried to move back new op lower bound from 7317593148320862208 to 7317593148268363776. Current Snapshot: MvccSnapshot[applied={T|T < 7317593148320862208 or (T in {7317593148320862208})}]
I20260812 08:02:44.757498 11150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a11b8d701c6e469281f62869f9845601: Neither blocks nor log segments found. Creating new log.
I20260812 08:02:44.758580 11150 log.cc:826] T 00000000000000000000000000000000 P a11b8d701c6e469281f62869f9845601: Log is configured to *not* fsync() on all Append() calls
I20260812 08:02:44.773126 11150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a11b8d701c6e469281f62869f9845601: No bootstrap required, opened a new log
I20260812 08:02:44.777464 11150 raft_consensus.cc:359] T 00000000000000000000000000000000 P a11b8d701c6e469281f62869f9845601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b8d701c6e469281f62869f9845601" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } } peers { permanent_uuid: "0ec34e03c53a47798cee73b7340f0fa6" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } } peers { permanent_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } }
I20260812 08:02:44.777751 11150 raft_consensus.cc:385] T 00000000000000000000000000000000 P a11b8d701c6e469281f62869f9845601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:02:44.777825 11150 raft_consensus.cc:740] T 00000000000000000000000000000000 P a11b8d701c6e469281f62869f9845601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a11b8d701c6e469281f62869f9845601, State: Initialized, Role: FOLLOWER
I20260812 08:02:44.778492 11150 consensus_queue.cc:260] T 00000000000000000000000000000000 P a11b8d701c6e469281f62869f9845601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b8d701c6e469281f62869f9845601" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } } peers { permanent_uuid: "0ec34e03c53a47798cee73b7340f0fa6" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } } peers { permanent_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } }
I20260812 08:02:44.779302 11480 sys_catalog.cc:455] T 00000000000000000000000000000000 P a11b8d701c6e469281f62869f9845601 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b8d701c6e469281f62869f9845601" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } } peers { permanent_uuid: "0ec34e03c53a47798cee73b7340f0fa6" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } } peers { permanent_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } } }
I20260812 08:02:44.779676 11480 sys_catalog.cc:458] T 00000000000000000000000000000000 P a11b8d701c6e469281f62869f9845601 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:44.780117 11150 sys_catalog.cc:565] T 00000000000000000000000000000000 P a11b8d701c6e469281f62869f9845601 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:02:44.780196 11110 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a11b8d701c6e469281f62869f9845601" is_pre_election: true
I20260812 08:02:44.780326 11107 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a11b8d701c6e469281f62869f9845601"
I20260812 08:02:44.780514 11105 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a11b8d701c6e469281f62869f9845601 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:02:44.785179 11105 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a11b8d701c6e469281f62869f9845601 [term 1 FOLLOWER]: Refusing update from remote peer a47f74d27e1942b7a3669ce721d6dbe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 08:02:44.786283 11470 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a11b8d701c6e469281f62869f9845601" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 08:02:44.791853 11480 sys_catalog.cc:455] T 00000000000000000000000000000000 P a11b8d701c6e469281f62869f9845601 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a47f74d27e1942b7a3669ce721d6dbe3. Latest consensus state: current_term: 1 leader_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b8d701c6e469281f62869f9845601" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } } peers { permanent_uuid: "0ec34e03c53a47798cee73b7340f0fa6" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } } peers { permanent_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } } }
I20260812 08:02:44.792047 11480 sys_catalog.cc:458] T 00000000000000000000000000000000 P a11b8d701c6e469281f62869f9845601 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:44.792207 11480 sys_catalog.cc:455] T 00000000000000000000000000000000 P a11b8d701c6e469281f62869f9845601 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b8d701c6e469281f62869f9845601" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } } peers { permanent_uuid: "0ec34e03c53a47798cee73b7340f0fa6" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } } peers { permanent_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } } }
I20260812 08:02:44.792312 11480 sys_catalog.cc:458] T 00000000000000000000000000000000 P a11b8d701c6e469281f62869f9845601 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 08:02:44.792572 11472 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:02:44.793870 11472 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:02:44.794363 11418 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11b8d701c6e469281f62869f9845601" member_type: VOTER last_known_addr { host: "127.5.152.62" port: 38581 } } peers { permanent_uuid: "0ec34e03c53a47798cee73b7340f0fa6" member_type: VOTER last_known_addr { host: "127.5.152.61" port: 34787 } } peers { permanent_uuid: "a47f74d27e1942b7a3669ce721d6dbe3" member_type: VOTER last_known_addr { host: "127.5.152.60" port: 39373 } } }
I20260812 08:02:44.794478 11418 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3 [sys.catalog]: This master's current role is: LEADER
I20260812 08:02:44.804646 11495 mvcc.cc:204] Tried to move back new op lower bound from 7317593148560728064 to 7317593148268363776. Current Snapshot: MvccSnapshot[applied={T|T < 7317593148320862208}]
I20260812 08:02:44.817991 11500 catalog_manager.cc:1295] Loaded cluster ID: 496241a8e0ff4bf585575d486d3dedfe
I20260812 08:02:44.818403 11500 catalog_manager.cc:1588] T 00000000000000000000000000000000 P a11b8d701c6e469281f62869f9845601: loading cluster ID for follower catalog manager: success
I20260812 08:02:44.824339 11472 catalog_manager.cc:6092] T 00000000000000000000000000000000 P a47f74d27e1942b7a3669ce721d6dbe3: Generated new TSK 0
I20260812 08:02:44.825264 11472 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:02:44.826788 11500 catalog_manager.cc:1610] T 00000000000000000000000000000000 P a11b8d701c6e469281f62869f9845601: acquiring CA information for follower catalog manager: success
I20260812 08:02:44.828351 11500 catalog_manager.cc:1638] T 00000000000000000000000000000000 P a11b8d701c6e469281f62869f9845601: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for kudu/127.5.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.5.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.5.152.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.152.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.152.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.5.152.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab.
I20260812 08:02:44.914886  5728 external_mini_cluster.cc:1372] Running /tmp/dist-test-task_3SyAK/build/debug/bin/kudu
/tmp/dist-test-task_3SyAK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.152.1:0
--local_ip_for_outbound_sockets=127.5.152.1
--webserver_interface=127.5.152.1
--webserver_port=0
--tserver_master_addrs=127.5.152.62:38581,127.5.152.61:34787,127.5.152.60:39373
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.5.152.20:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/test.cert
--webserver_private_key_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 08:02:45.088373 11514 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 08:02:45.088704 11514 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 08:02:45.088824 11514 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 08:02:45.093505 11514 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 08:02:45.094002 11514 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.152.1
I20260812 08:02:45.100646 11514 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.152.20:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/krb5kdc/kudu.keytab
--principal=kudu/127.5.152.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.152.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/test.cert
--webserver_interface=127.5.152.1
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--tserver_master_addrs=127.5.152.62:38581,127.5.152.61:34787,127.5.152.60:39373
--never_fsync=true
--heap_profile_path=/tmp/kudu.11514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.152.1

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 08:02:45.103107 11514 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:02:45.104591 11514 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:02:45.114672 11526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:45.114672 11522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:02:45.114830 11523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:02:45.115418 11514 server_base.cc:1061] running on GCE node
I20260812 08:02:45.116392 11514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 08:02:45.117414 11514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 08:02:45.118657 11514 hybrid_clock.cc:648] HybridClock initialized: now 1786521765118625 us; error 50 us; skew 500 ppm
Aug 12 08:02:45 dist-test-slave-1nl6 krb5kdc[11016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521765, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 08:02:45.124205 11514 init.cc:377] Logged in from keytab as kudu/127.5.152.1@KRBTEST.COM (short username kudu)
I20260812 08:02:45.141050 11514 webserver.cc:533] Webserver started at https://127.5.152.1:33985/ using document root <none> and password file <none>
I20260812 08:02:45.141884 11514 fs_manager.cc:362] Metadata directory not provided
I20260812 08:02:45.141952 11514 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:02:45.142230 11514 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:02:45.144290 11514 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/data/instance:
uuid: "1cc81d6d8b4b4f43b29f97923b317533"
format_stamp: "Formatted at 2026-08-12 08:02:45 on dist-test-slave-1nl6"
I20260812 08:02:45.144850 11514 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/wal/instance:
uuid: "1cc81d6d8b4b4f43b29f97923b317533"
format_stamp: "Formatted at 2026-08-12 08:02:45 on dist-test-slave-1nl6"
I20260812 08:02:45.149127 11514 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260812 08:02:45.152134 11539 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:45.153451 11514 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260812 08:02:45.153717 11514 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/data,/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/wal
uuid: "1cc81d6d8b4b4f43b29f97923b317533"
format_stamp: "Formatted at 2026-08-12 08:02:45 on dist-test-slave-1nl6"
I20260812 08:02:45.153882 11514 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:02:45.173980 11514 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:02:45.177263 11514 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:02:45.177459 11514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:02:45.178062 11514 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:02:45.179270 11514 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:02:45.179330 11514 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:45.179373 11514 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:02:45.179389 11514 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:02:45.190353 11514 rpc_server.cc:307] RPC server started. Bound to: 127.5.152.1:38933
I20260812 08:02:45.190376 11719 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.152.1:38933 every 8 connection(s)
I20260812 08:02:45.191527 11514 server_base.cc:1193] Dumped server information to /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/data/info.pb
I20260812 08:02:45.196346  5728 external_mini_cluster.cc:1434] Started /tmp/dist-test-task_3SyAK/build/debug/bin/kudu as pid 11514
I20260812 08:02:45.196527  5728 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/minicluster-data/ts-0/wal/instance
Aug 12 08:02:45 dist-test-slave-1nl6 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521765, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.61@KRBTEST.COM
Aug 12 08:02:45 dist-test-slave-1nl6 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521765, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.62@KRBTEST.COM
Aug 12 08:02:45 dist-test-slave-1nl6 krb5kdc[11016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786521765, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.152.1@KRBTEST.COM for kudu/127.5.152.60@KRBTEST.COM
I20260812 08:02:45.210301 11720 heartbeater.cc:344] Connected to a master server at 127.5.152.60:39373
I20260812 08:02:45.210305 11721 heartbeater.cc:344] Connected to a master server at 127.5.152.62:38581
I20260812 08:02:45.210428 11722 heartbeater.cc:344] Connected to a master server at 127.5.152.61:34787
I20260812 08:02:45.210587 11720 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:45.210588 11721 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:45.210589 11722 heartbeater.cc:461] Registering TS with master...
I20260812 08:02:45.211280 11721 heartbeater.cc:507] Master 127.5.152.62:38581 requested a full tablet report, sending...
I20260812 08:02:45.211277 11720 heartbeater.cc:507] Master 127.5.152.60:39373 requested a full tablet report, sending...
I20260812 08:02:45.211275 11722 heartbeater.cc:507] Master 127.5.152.61:34787 requested a full tablet report, sending...
I20260812 08:02:45.212868 11096 ts_manager.cc:194] Registered new tserver with Master: 1cc81d6d8b4b4f43b29f97923b317533 (127.5.152.1:38933)
I20260812 08:02:45.212872 11221 ts_manager.cc:194] Registered new tserver with Master: 1cc81d6d8b4b4f43b29f97923b317533 (127.5.152.1:38933)
I20260812 08:02:45.212872 11354 ts_manager.cc:194] Registered new tserver with Master: 1cc81d6d8b4b4f43b29f97923b317533 (127.5.152.1:38933)
I20260812 08:02:45.214272 11354 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.152.1@KRBTEST.COM'} at 127.5.152.1:60305
I20260812 08:02:45.218338  5728 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
time=2026-08-12T08:02:45.293Z level=INFO source=main.go:1678 msg="updated GOGC" old=100 new=75
time=2026-08-12T08:02:45.296Z level=INFO source=main.go:744 msg="Leaving GOMAXPROCS=4: CPU quota undefined" component=automaxprocs
time=2026-08-12T08:02:45.297Z level=INFO source=memlimit.go:198 msg="GOMEMLIMIT is updated" component=automemlimit package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=14142549196 previous=9223372036854775807
time=2026-08-12T08:02:45.298Z level=INFO source=main.go:851 msg="Starting Prometheus Server" mode=server version="(version=3.11.2, branch=HEAD, revision=f0f0fdd679dcd6df320b0558b20919f7cd44c407)"
time=2026-08-12T08:02:45.298Z level=INFO source=main.go:856 msg="operational information" build_context="(go=go1.26.2, platform=linux/amd64, user=root@d2684485c347, date=20260413-11:53:40, tags=netgo,builtinassets)" host_details="(Linux 5.4.0-1157-gcp #166~18.04.1-Ubuntu SMP Wed Jan 21 01:57:46 UTC 2026 x86_64 dist-test-slave-1nl6 (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)"
time=2026-08-12T08:02:45.548Z level=INFO source=web.go:710 msg="Start listening for connections" component=web address=127.0.0.1:0
time=2026-08-12T08:02:45.549Z level=INFO source=main.go:1410 msg="Starting TSDB ..."
time=2026-08-12T08:02:45.551Z level=INFO source=tls_config.go:354 msg="Listening on" component=web address=127.0.0.1:45745
time=2026-08-12T08:02:45.551Z level=INFO source=tls_config.go:357 msg="TLS is disabled." component=web http2=false address=127.0.0.1:45745
time=2026-08-12T08:02:45.553Z level=INFO source=head.go:698 msg="Replaying on-disk memory mappable chunks if any" component=tsdb
time=2026-08-12T08:02:45.553Z level=INFO source=head.go:784 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=1.657µs
time=2026-08-12T08:02:45.553Z level=INFO source=head.go:792 msg="Replaying WAL, this may take a while" component=tsdb
time=2026-08-12T08:02:45.554Z level=INFO source=head.go:865 msg="WAL segment loaded" component=tsdb segment=0 maxSegment=0 duration=506.717µs
time=2026-08-12T08:02:45.554Z level=INFO source=head.go:902 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=69.778µs wal_replay_duration=590.963µs wbl_replay_duration=271ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=1.657µs total_replay_duration=690.975µs
time=2026-08-12T08:02:45.560Z level=INFO source=main.go:1431 msg="filesystem information" fs_type=OVERLAYFS_SUPER_MAGIC
time=2026-08-12T08:02:45.560Z level=INFO source=main.go:1434 msg="TSDB started"
time=2026-08-12T08:02:45.560Z level=INFO source=main.go:1632 msg="Loading configuration file" filename=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/prometheus.yml
time=2026-08-12T08:02:45.561Z level=INFO source=main.go:1048 msg="TSDB retention updated" duration=15d size=0B percentage=0
time=2026-08-12T08:02:45.565Z level=INFO source=main.go:1671 msg="Completed loading of configuration file" db_storage=234.611µs remote_storage=4.077µs web_handler=1.436µs query_engine=41.021µs scrape=2.605357ms scrape_sd=281.62µs notify=3.341µs notify_sd=1.832µs rules=4.078µs tracing=7.756µs filename=/tmp/dist-test-task_3SyAK/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786521680053065-5728-0/prometheus.yml totalDuration=4.407273ms
time=2026-08-12T08:02:45.565Z level=INFO source=main.go:1395 msg="Server is ready to receive web requests."
time=2026-08-12T08:02:45.565Z level=INFO source=manager.go:209 msg="Starting rule manager..." component="rule manager"
time=2026-08-12T08:02:45.591Z level=ERROR source=refresh.go:72 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:02:45.599Z level=ERROR source=refresh.go:72 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:02:45.607Z level=ERROR source=refresh.go:72 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
I20260812 08:02:45.702071 11468 catalog_manager.cc:1295] Loaded cluster ID: 496241a8e0ff4bf585575d486d3dedfe
I20260812 08:02:45.702222 11468 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6: loading cluster ID for follower catalog manager: success
I20260812 08:02:45.706085 11468 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6: acquiring CA information for follower catalog manager: success
I20260812 08:02:45.708454 11468 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 0ec34e03c53a47798cee73b7340f0fa6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 08:02:46.216598 11720 heartbeater.cc:499] Master 127.5.152.60:39373 was elected leader, sending a full tablet report...
time=2026-08-12T08:02:47.592Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:02:47.601Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:02:47.608Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:02:49.593Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:02:49.601Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:02:49.608Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:02:51.592Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:02:51.601Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:02:51.609Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:02:53.593Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:02:53.600Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:02:53.608Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T08:02:53.630Z level=WARN source=main.go:1191 msg="Received an OS signal, exiting gracefully..." signal=terminated
time=2026-08-12T08:02:53.630Z level=INFO source=main.go:1216 msg="Stopping scrape discovery manager..."
time=2026-08-12T08:02:53.630Z level=INFO source=main.go:1230 msg="Stopping notify discovery manager..."
time=2026-08-12T08:02:53.630Z level=INFO source=main.go:1245 msg="Stopping rule manager manager..."
time=2026-08-12T08:02:53.630Z level=INFO source=manager.go:223 msg="Stopping rule manager..." component="rule manager"
time=2026-08-12T08:02:53.630Z level=INFO source=manager.go:239 msg="Rule manager stopped" component="rule manager"
time=2026-08-12T08:02:53.630Z level=INFO source=main.go:1269 msg="Stopping scrape manager..."
time=2026-08-12T08:02:53.630Z level=INFO source=main.go:1284 msg="Stopping tracing manager..."
time=2026-08-12T08:02:53.630Z level=INFO source=main.go:1369 msg="Stopping reloaders..."
time=2026-08-12T08:02:53.630Z level=INFO source=main.go:1458 msg="Stopping storage..."
time=2026-08-12T08:02:53.630Z level=INFO source=main.go:1241 msg="Rule manager stopped"
time=2026-08-12T08:02:53.630Z level=INFO source=main.go:1261 msg="Scrape manager stopped"
time=2026-08-12T08:02:53.630Z level=INFO source=main.go:1226 msg="Notify discovery manager stopped"
time=2026-08-12T08:02:53.630Z level=INFO source=main.go:1212 msg="Scrape discovery manager stopped"
time=2026-08-12T08:02:53.630Z level=INFO source=main.go:1280 msg="Tracing manager stopped"
time=2026-08-12T08:02:53.630Z level=INFO source=main.go:1361 msg="Reloaders stopped"
time=2026-08-12T08:02:53.631Z level=INFO source=main.go:1536 msg="Stopping web handler..."
time=2026-08-12T08:02:53.631Z level=INFO source=main.go:1559 msg="Stopping notifier manager..."
time=2026-08-12T08:02:53.631Z level=INFO source=manager.go:321 msg="Stopping notification manager..." component=notifier
time=2026-08-12T08:02:53.631Z level=INFO source=main.go:1555 msg="Notifier manager stopped"
time=2026-08-12T08:02:53.631Z level=INFO source=main.go:1454 msg="TSDB stopped"
time=2026-08-12T08:02:53.631Z level=INFO source=main.go:1532 msg="Web handler stopped"
time=2026-08-12T08:02:53.631Z level=INFO source=main.go:1570 msg="See you next time!"
I20260812 08:02:53.640231  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 11514
I20260812 08:02:53.653151  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 11041
I20260812 08:02:53.666395  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 11168
I20260812 08:02:53.681699  5728 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task_3SyAK/build/debug/bin/kudu with pid 11295
2026-08-12T08:02:53Z chronyd exiting
[       OK ] PrometheusTokenITest.TestMissingTokenSdFails (12674 ms)
[----------] 6 tests from PrometheusTokenITest (69735 ms total)

[----------] Global test environment tear-down
[==========] 8 tests from 2 test suites ran. (93649 ms total)
[  PASSED  ] 8 tests.
