[==========] Running 8 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 2 tests from PrometheusSpnegoITest
[ RUN      ] PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly
Loading random data
Initializing database '/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 06:35:59 dist-test-slave-gjrz krb5kdc[13412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 06:35:59 dist-test-slave-gjrz krb5kdc[13412](info): set up 2 sockets
Aug 12 06:35:59 dist-test-slave-gjrz krb5kdc[13412](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 06:36:01.983070 13401 mini_kdc.cc:121] Time spent starting KDC: real 2.077s	user 0.000s	sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/test-user.keytab.
Aug 12 06:36:02 dist-test-slave-gjrz krb5kdc[13412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM: 
2026-08-12T06:36:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T06:36:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:02.112176 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.126:39439
--webserver_interface=127.13.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:35891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:39439,127.13.22.125:34785,127.13.22.124:38293
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 06:36:02.214629 13442 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:02.214811 13442 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:02.214865 13442 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:02.218014 13442 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:36:02.218065 13442 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:02.218084 13442 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:36:02.218104 13442 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:36:02.222182 13442 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:35891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:39439,127.13.22.125:34785,127.13.22.124:38293
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.126:39439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.126
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:02.223043 13442 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:02.223824 13442 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 06:36:02.229017 13442 server_base.cc:1061] running on GCE node
W20260812 06:36:02.228967 13450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:02.229038 13456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:02.229244 13459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:02.229686 13442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:02.230497 13442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:02.231642 13442 hybrid_clock.cc:648] HybridClock initialized: now 1786516562231655 us; error 57 us; skew 500 ppm
Aug 12 06:36:02 dist-test-slave-gjrz krb5kdc[13412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516562, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:02.234038 13442 init.cc:377] Logged in from keytab as kudu/127.13.22.126@KRBTEST.COM (short username kudu)
I20260812 06:36:02.241694 13442 webserver.cc:533] Webserver started at https://127.13.22.126:44221/ using document root <none> and password file <none>
I20260812 06:36:02.242252 13442 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:02.242303 13442 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:02.242494 13442 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:02.244086 13442 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/data/instance:
uuid: "805f7715195643dfa911a46679931f90"
format_stamp: "Formatted at 2026-08-12 06:36:02 on dist-test-slave-gjrz"
I20260812 06:36:02.244525 13442 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/wal/instance:
uuid: "805f7715195643dfa911a46679931f90"
format_stamp: "Formatted at 2026-08-12 06:36:02 on dist-test-slave-gjrz"
I20260812 06:36:02.247803 13442 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.001s
I20260812 06:36:02.249774 13474 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:02.250677 13442 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:02.250787 13442 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/wal
uuid: "805f7715195643dfa911a46679931f90"
format_stamp: "Formatted at 2026-08-12 06:36:02 on dist-test-slave-gjrz"
I20260812 06:36:02.250867 13442 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:02.264813 13442 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:02.267032 13442 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:02.267195 13442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:02.274284 13442 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.126:39439
I20260812 06:36:02.274293 13567 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.126:39439 every 8 connection(s)
I20260812 06:36:02.275226 13442 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
I20260812 06:36:02.276997 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 13442
I20260812 06:36:02.277123 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/wal/instance
I20260812 06:36:02.278061 13569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:02.282233 13569 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.22.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.125@KRBTEST.COM" created.
Aug 12 06:36:02 dist-test-slave-gjrz krb5kdc[13412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516562, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:36:02.296201 13569 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } has no permanent_uuid. Determining permanent_uuid...
W20260812 06:36:02.296913 13485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.22.125:34785: connect: Connection refused (error 111)
W20260812 06:36:02.297714 13569 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:34785: Network error: Client connection negotiation failed: client connection to 127.13.22.125:34785: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:02.321448 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.125:34785
--webserver_interface=127.13.22.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:35891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:39439,127.13.22.125:34785,127.13.22.124:38293
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 06:36:02.346941 13569 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } attempt: 1
W20260812 06:36:02.348515 13569 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:34785: Network error: Client connection negotiation failed: client connection to 127.13.22.125:34785: connect: Connection refused (error 111)
I20260812 06:36:02.416690 13569 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } attempt: 2
W20260812 06:36:02.418175 13569 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:34785: Network error: Client connection negotiation failed: client connection to 127.13.22.125:34785: connect: Connection refused (error 111)
W20260812 06:36:02.422631 13587 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:02.422823 13587 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:02.422910 13587 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:02.426210 13587 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:36:02.426275 13587 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:02.426302 13587 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:36:02.426334 13587 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:36:02.430491 13587 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:35891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:39439,127.13.22.125:34785,127.13.22.124:38293
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.125:34785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.125
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:02.431454 13587 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:02.432219 13587 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:02.437377 13594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:02.437438 13599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:02.437557 13597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:02.437594 13587 server_base.cc:1061] running on GCE node
I20260812 06:36:02.438095 13587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:02.438907 13587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:02.440068 13587 hybrid_clock.cc:648] HybridClock initialized: now 1786516562440054 us; error 23 us; skew 500 ppm
Aug 12 06:36:02 dist-test-slave-gjrz krb5kdc[13412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516562, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:02.442624 13587 init.cc:377] Logged in from keytab as kudu/127.13.22.125@KRBTEST.COM (short username kudu)
I20260812 06:36:02.450604 13587 webserver.cc:533] Webserver started at https://127.13.22.125:44491/ using document root <none> and password file <none>
I20260812 06:36:02.451177 13587 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:02.451228 13587 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:02.451464 13587 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:02.453086 13587 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/data/instance:
uuid: "bc1868a29f024d18b2558a4a3eb7dd1a"
format_stamp: "Formatted at 2026-08-12 06:36:02 on dist-test-slave-gjrz"
I20260812 06:36:02.453543 13587 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/wal/instance:
uuid: "bc1868a29f024d18b2558a4a3eb7dd1a"
format_stamp: "Formatted at 2026-08-12 06:36:02 on dist-test-slave-gjrz"
I20260812 06:36:02.456936 13587 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20260812 06:36:02.458951 13618 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:02.459966 13587 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:02.460078 13587 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/wal
uuid: "bc1868a29f024d18b2558a4a3eb7dd1a"
format_stamp: "Formatted at 2026-08-12 06:36:02 on dist-test-slave-gjrz"
I20260812 06:36:02.460162 13587 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:02.509336 13569 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } attempt: 3
W20260812 06:36:02.510878 13569 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:34785: Network error: Client connection negotiation failed: client connection to 127.13.22.125:34785: connect: Connection refused (error 111)
I20260812 06:36:02.526067 13587 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:02.528443 13587 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:02.528614 13587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:02.535696 13587 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.125:34785
I20260812 06:36:02.535704 13707 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.125:34785 every 8 connection(s)
I20260812 06:36:02.536701 13587 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
I20260812 06:36:02.539357 13710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:02.542670 13710 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } has no permanent_uuid. Determining permanent_uuid...
I20260812 06:36:02.546514 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 13587
I20260812 06:36:02.546595 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/wal/instance
Aug 12 06:36:02 dist-test-slave-gjrz krb5kdc[13412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516562, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:36:02.554363 13710 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.22.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.124@KRBTEST.COM" created.
Aug 12 06:36:02 dist-test-slave-gjrz krb5kdc[13412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516562, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
I20260812 06:36:02.563479 13710 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } has no permanent_uuid. Determining permanent_uuid...
W20260812 06:36:02.564116 13619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.22.124:38293: connect: Connection refused (error 111)
W20260812 06:36:02.564857 13710 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:38293: Network error: Client connection negotiation failed: client connection to 127.13.22.124:38293: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:02.590286 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.124:38293
--webserver_interface=127.13.22.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:35891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:39439,127.13.22.125:34785,127.13.22.124:38293
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 06:36:02.614075 13710 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } attempt: 1
W20260812 06:36:02.615411 13710 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:38293: Network error: Client connection negotiation failed: client connection to 127.13.22.124:38293: connect: Connection refused (error 111)
I20260812 06:36:02.654052 13569 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } attempt: 4
Aug 12 06:36:02 dist-test-slave-gjrz krb5kdc[13412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516562, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
I20260812 06:36:02.661952 13569 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } has no permanent_uuid. Determining permanent_uuid...
W20260812 06:36:02.663141 13569 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:38293: Network error: Client connection negotiation failed: client connection to 127.13.22.124:38293: connect: Connection refused (error 111)
I20260812 06:36:02.683563 13710 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } attempt: 2
W20260812 06:36:02.684762 13710 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:38293: Network error: Client connection negotiation failed: client connection to 127.13.22.124:38293: connect: Connection refused (error 111)
W20260812 06:36:02.690462 13732 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:02.690639 13732 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:02.690724 13732 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:02.693874 13732 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:36:02.693938 13732 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:02.693967 13732 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:36:02.693998 13732 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:36:02.698062 13732 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:35891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:39439,127.13.22.125:34785,127.13.22.124:38293
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.124:38293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.124
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:02.698935 13732 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:02.699712 13732 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:02.704531 13742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:02.704593 13739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:02.704560 13744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:02.704711 13732 server_base.cc:1061] running on GCE node
I20260812 06:36:02.705175 13732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:02.705915 13732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:02.707069 13732 hybrid_clock.cc:648] HybridClock initialized: now 1786516562707055 us; error 30 us; skew 500 ppm
Aug 12 06:36:02 dist-test-slave-gjrz krb5kdc[13412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516562, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:02.709472 13732 init.cc:377] Logged in from keytab as kudu/127.13.22.124@KRBTEST.COM (short username kudu)
I20260812 06:36:02.716735 13732 webserver.cc:533] Webserver started at https://127.13.22.124:44905/ using document root <none> and password file <none>
I20260812 06:36:02.717269 13732 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:02.717319 13732 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:02.717502 13732 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:02.718998 13732 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/data/instance:
uuid: "cc8f221432c947fcb2f86cb9e5b6e35a"
format_stamp: "Formatted at 2026-08-12 06:36:02 on dist-test-slave-gjrz"
I20260812 06:36:02.719475 13732 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/wal/instance:
uuid: "cc8f221432c947fcb2f86cb9e5b6e35a"
format_stamp: "Formatted at 2026-08-12 06:36:02 on dist-test-slave-gjrz"
I20260812 06:36:02.722299 13569 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } attempt: 1
I20260812 06:36:02.722616 13732 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.001s	sys 0.001s
W20260812 06:36:02.723599 13569 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:38293: Network error: Client connection negotiation failed: client connection to 127.13.22.124:38293: connect: Connection refused (error 111)
I20260812 06:36:02.724849 13755 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:02.725749 13732 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.001s
I20260812 06:36:02.725862 13732 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/wal
uuid: "cc8f221432c947fcb2f86cb9e5b6e35a"
format_stamp: "Formatted at 2026-08-12 06:36:02 on dist-test-slave-gjrz"
I20260812 06:36:02.725942 13732 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:02.745206 13732 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:02.747444 13732 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:02.747602 13732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:02.754467 13732 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.124:38293
I20260812 06:36:02.754485 13849 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.124:38293 every 8 connection(s)
I20260812 06:36:02.755446 13732 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
I20260812 06:36:02.758090 13851 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:02.761361 13851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } has no permanent_uuid. Determining permanent_uuid...
I20260812 06:36:02.764580 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 13732
I20260812 06:36:02.764706 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/wal/instance
Aug 12 06:36:02 dist-test-slave-gjrz krb5kdc[13412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516562, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:36:02.773729 13851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } has no permanent_uuid. Determining permanent_uuid...
I20260812 06:36:02.775921 13710 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } attempt: 3
Aug 12 06:36:02 dist-test-slave-gjrz krb5kdc[13412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
Aug 12 06:36:02 dist-test-slave-gjrz krb5kdc[13412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516562, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
Aug 12 06:36:02 dist-test-slave-gjrz krb5kdc[13412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516562, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:36:02.783106 13851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:36:02 dist-test-slave-gjrz krb5kdc[13412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
Aug 12 06:36:02 dist-test-slave-gjrz krb5kdc[13412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516562, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:36:02.789463 13710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a: Bootstrap starting.
I20260812 06:36:02.790745 13569 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } attempt: 2
I20260812 06:36:02.792227 13710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:02.793121 13710 log.cc:826] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a: Log is configured to *not* fsync() on all Append() calls
Aug 12 06:36:02 dist-test-slave-gjrz krb5kdc[13412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:36:02.794902 13851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a: Bootstrap starting.
Aug 12 06:36:02 dist-test-slave-gjrz krb5kdc[13412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516562, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:36:02.797240 13851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:02.797973 13851 log.cc:826] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:02.799928 13710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a: No bootstrap required, opened a new log
I20260812 06:36:02.799940 13851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a: No bootstrap required, opened a new log
I20260812 06:36:02.801844 13569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90: Bootstrap starting.
I20260812 06:36:02.802264 13851 raft_consensus.cc:359] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805f7715195643dfa911a46679931f90" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } } peers { permanent_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } } peers { permanent_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } }
I20260812 06:36:02.802416 13851 raft_consensus.cc:385] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:02.802471 13851 raft_consensus.cc:740] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8f221432c947fcb2f86cb9e5b6e35a, State: Initialized, Role: FOLLOWER
I20260812 06:36:02.802853 13851 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805f7715195643dfa911a46679931f90" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } } peers { permanent_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } } peers { permanent_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } }
I20260812 06:36:02.802999 13710 raft_consensus.cc:359] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805f7715195643dfa911a46679931f90" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } } peers { permanent_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } } peers { permanent_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } }
I20260812 06:36:02.803172 13710 raft_consensus.cc:385] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:02.803239 13710 raft_consensus.cc:740] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc1868a29f024d18b2558a4a3eb7dd1a, State: Initialized, Role: FOLLOWER
I20260812 06:36:02.803686 13710 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805f7715195643dfa911a46679931f90" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } } peers { permanent_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } } peers { permanent_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } }
I20260812 06:36:02.804049 13851 sys_catalog.cc:565] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:36:02.804188 13569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:02.804766 13710 sys_catalog.cc:565] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:36:02.804961 13569 log.cc:826] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:02.805708 13865 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805f7715195643dfa911a46679931f90" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } } peers { permanent_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } } peers { permanent_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } } }
I20260812 06:36:02.805873 13865 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:02.806643 13569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90: No bootstrap required, opened a new log
I20260812 06:36:02.810146 13569 raft_consensus.cc:359] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805f7715195643dfa911a46679931f90" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } } peers { permanent_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } } peers { permanent_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } }
I20260812 06:36:02.810312 13569 raft_consensus.cc:385] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:02.810369 13569 raft_consensus.cc:740] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 805f7715195643dfa911a46679931f90, State: Initialized, Role: FOLLOWER
I20260812 06:36:02.808442 13732 master_runner.cc:187] Error getting master registration for 127.13.22.126:39439: OK, instance_id { permanent_uuid: "805f7715195643dfa911a46679931f90" instance_seqno: 1786516562272350 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 06:36:02.810998 13732 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 06:36:02.811491 13895 catalog_manager.cc:1594] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:36:02.811570 13895 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:36:02.811470 13587 master_runner.cc:187] Error getting master registration for 127.13.22.126:39439: OK, instance_id { permanent_uuid: "805f7715195643dfa911a46679931f90" instance_seqno: 1786516562272350 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 06:36:02.813341 13587 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 06:36:02.813479 13569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805f7715195643dfa911a46679931f90" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } } peers { permanent_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } } peers { permanent_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } }
I20260812 06:36:02.813895 13909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805f7715195643dfa911a46679931f90" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } } peers { permanent_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } } peers { permanent_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } } }
I20260812 06:36:02.814049 13909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:02.814499 13569 sys_catalog.cc:565] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:36:02.807482 13866 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805f7715195643dfa911a46679931f90" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } } peers { permanent_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } } peers { permanent_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } } }
W20260812 06:36:02.809756 13904 catalog_manager.cc:1594] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:36:02.816807 13904 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:36:02.816888 13866 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:02.818872 13909 raft_consensus.cc:493] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:36:02.819015 13909 raft_consensus.cc:515] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805f7715195643dfa911a46679931f90" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } } peers { permanent_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } } peers { permanent_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } }
I20260812 06:36:02.819815 13909 leader_election.cc:290] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc1868a29f024d18b2558a4a3eb7dd1a (127.13.22.125:34785), cc8f221432c947fcb2f86cb9e5b6e35a (127.13.22.124:38293)
I20260812 06:36:02.820283 13661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "805f7715195643dfa911a46679931f90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a" is_pre_election: true
I20260812 06:36:02.820372 13797 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "805f7715195643dfa911a46679931f90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a" is_pre_election: true
I20260812 06:36:02.820561 13661 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 805f7715195643dfa911a46679931f90 in term 0.
I20260812 06:36:02.820613 13797 raft_consensus.cc:2468] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 805f7715195643dfa911a46679931f90 in term 0.
I20260812 06:36:02.820996 13482 leader_election.cc:304] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 805f7715195643dfa911a46679931f90, cc8f221432c947fcb2f86cb9e5b6e35a; no voters: 
I20260812 06:36:02.821602 13909 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 06:36:02.821668 13909 raft_consensus.cc:493] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 06:36:02.821705 13909 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:02.822753 13909 raft_consensus.cc:515] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805f7715195643dfa911a46679931f90" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } } peers { permanent_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } } peers { permanent_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } }
I20260812 06:36:02.823102 13909 leader_election.cc:290] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [CANDIDATE]: Term 1 election: Requested vote from peers bc1868a29f024d18b2558a4a3eb7dd1a (127.13.22.125:34785), cc8f221432c947fcb2f86cb9e5b6e35a (127.13.22.124:38293)
I20260812 06:36:02.823539 13797 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "805f7715195643dfa911a46679931f90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a"
I20260812 06:36:02.823647 13797 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:02.823840 13659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "805f7715195643dfa911a46679931f90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a"
I20260812 06:36:02.823920 13659 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:02.824680 13797 raft_consensus.cc:2468] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 805f7715195643dfa911a46679931f90 in term 1.
I20260812 06:36:02.824692 13659 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 805f7715195643dfa911a46679931f90 in term 1.
I20260812 06:36:02.824993 13485 leader_election.cc:304] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 805f7715195643dfa911a46679931f90, bc1868a29f024d18b2558a4a3eb7dd1a; no voters: 
I20260812 06:36:02.825156 13909 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 06:36:02.825340 13909 raft_consensus.cc:697] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [term 1 LEADER]: Becoming Leader. State: Replica: 805f7715195643dfa911a46679931f90, State: Running, Role: LEADER
I20260812 06:36:02.825611 13909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805f7715195643dfa911a46679931f90" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } } peers { permanent_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } } peers { permanent_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } }
I20260812 06:36:02.826122 13442 master_runner.cc:233] Existing masters have differing terms: 0 vs 1
I20260812 06:36:02.826229 13442 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 06:36:02.827510 13930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 805f7715195643dfa911a46679931f90. Latest consensus state: current_term: 1 leader_uuid: "805f7715195643dfa911a46679931f90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805f7715195643dfa911a46679931f90" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } } peers { permanent_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } } peers { permanent_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } } }
I20260812 06:36:02.827612 13930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [sys.catalog]: This master's current role is: LEADER
W20260812 06:36:02.832785 13928 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:36:02.832860 13928 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:36:02.835556 13932 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 06:36:02.836503 13932 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 06:36:02.841753 13797 raft_consensus.cc:1275] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a [term 1 FOLLOWER]: Refusing update from remote peer 805f7715195643dfa911a46679931f90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:36:02.842407 13930 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:36:02.845984 13865 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 805f7715195643dfa911a46679931f90. Latest consensus state: current_term: 1 leader_uuid: "805f7715195643dfa911a46679931f90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805f7715195643dfa911a46679931f90" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } } peers { permanent_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } } peers { permanent_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } } }
I20260812 06:36:02.846082 13865 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:02.846990 13659 raft_consensus.cc:1275] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a [term 1 FOLLOWER]: Refusing update from remote peer 805f7715195643dfa911a46679931f90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:36:02.847566 13930 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:36:02.850744 13866 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 805f7715195643dfa911a46679931f90. Latest consensus state: current_term: 1 leader_uuid: "805f7715195643dfa911a46679931f90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805f7715195643dfa911a46679931f90" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } } peers { permanent_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } } peers { permanent_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } } }
I20260812 06:36:02.850841 13866 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:02.855382 13909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "805f7715195643dfa911a46679931f90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805f7715195643dfa911a46679931f90" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } } peers { permanent_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } } peers { permanent_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } } }
I20260812 06:36:02.855480 13909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [sys.catalog]: This master's current role is: LEADER
I20260812 06:36:02.855726 13936 mvcc.cc:204] Tried to move back new op lower bound from 7317571841391628288 to 7317571841336774656. Current Snapshot: MvccSnapshot[applied={T|T < 7317571841391628288}]
I20260812 06:36:02.855860 13865 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "805f7715195643dfa911a46679931f90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805f7715195643dfa911a46679931f90" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } } peers { permanent_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } } peers { permanent_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } } }
I20260812 06:36:02.855962 13865 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:02.856590 13937 mvcc.cc:204] Tried to move back new op lower bound from 7317571841391628288 to 7317571841336774656. Current Snapshot: MvccSnapshot[applied={T|T < 7317571841391628288}]
I20260812 06:36:02.857973 13930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "805f7715195643dfa911a46679931f90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805f7715195643dfa911a46679931f90" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } } peers { permanent_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } } peers { permanent_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } } }
I20260812 06:36:02.858088 13930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90 [sys.catalog]: This master's current role is: LEADER
I20260812 06:36:02.859057 13866 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "805f7715195643dfa911a46679931f90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "805f7715195643dfa911a46679931f90" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 39439 } } peers { permanent_uuid: "bc1868a29f024d18b2558a4a3eb7dd1a" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 34785 } } peers { permanent_uuid: "cc8f221432c947fcb2f86cb9e5b6e35a" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 38293 } } }
I20260812 06:36:02.859155 13866 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:02.859639 13939 mvcc.cc:204] Tried to move back new op lower bound from 7317571841391628288 to 7317571841336774656. Current Snapshot: MvccSnapshot[applied={T|T < 7317571841391628288}]
I20260812 06:36:02.868456 13932 catalog_manager.cc:1383] Generated new cluster ID: dfc9f9dc80044112a456afa9eba54f66
I20260812 06:36:02.868518 13932 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 06:36:02.892203 13932 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 06:36:02.893311 13932 catalog_manager.cc:1540] Loading token signing keys...
I20260812 06:36:02.916205 13932 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 805f7715195643dfa911a46679931f90: Generated new TSK 0
I20260812 06:36:02.916950 13932 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:02.982338 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.22.65:0
--local_ip_for_outbound_sockets=127.13.22.65
--webserver_interface=127.13.22.65
--webserver_port=0
--tserver_master_addrs=127.13.22.126:39439,127.13.22.125:34785,127.13.22.124:38293
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:35891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 06:36:03.104789 13952 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:03.105065 13952 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:03.105145 13952 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:03.108521 13952 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:03.108646 13952 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.22.65
I20260812 06:36:03.113058 13952 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:35891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.65
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_require_spnego=true
--tserver_master_addrs=127.13.22.126:39439,127.13.22.125:34785,127.13.22.124:38293
--never_fsync=true
--heap_profile_path=/tmp/kudu.13952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.22.65

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:03.114024 13952 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:03.114831 13952 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:03.121219 13965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:03.121250 13959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:03.121470 13960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:03.121706 13952 server_base.cc:1061] running on GCE node
I20260812 06:36:03.122045 13952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:03.122521 13952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:03.123662 13952 hybrid_clock.cc:648] HybridClock initialized: now 1786516563123644 us; error 28 us; skew 500 ppm
Aug 12 06:36:03 dist-test-slave-gjrz krb5kdc[13412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516563, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:03.126185 13952 init.cc:377] Logged in from keytab as kudu/127.13.22.65@KRBTEST.COM (short username kudu)
I20260812 06:36:03.133553 13952 webserver.cc:533] Webserver started at https://127.13.22.65:45837/ using document root <none> and password file <none>
I20260812 06:36:03.134094 13952 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:03.134145 13952 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:03.134335 13952 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:03.135910 13952 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/data/instance:
uuid: "b238708c52b14df8b23a95e1c3eb16cb"
format_stamp: "Formatted at 2026-08-12 06:36:03 on dist-test-slave-gjrz"
I20260812 06:36:03.136343 13952 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/wal/instance:
uuid: "b238708c52b14df8b23a95e1c3eb16cb"
format_stamp: "Formatted at 2026-08-12 06:36:03 on dist-test-slave-gjrz"
I20260812 06:36:03.139634 13952 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.001s
I20260812 06:36:03.141597 13975 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:03.142573 13952 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:03.142680 13952 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/wal
uuid: "b238708c52b14df8b23a95e1c3eb16cb"
format_stamp: "Formatted at 2026-08-12 06:36:03 on dist-test-slave-gjrz"
I20260812 06:36:03.142765 13952 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:03.182519 13952 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:03.185151 13952 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:03.185333 13952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:03.185930 13952 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:03.186883 13952 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:36:03.186932 13952 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:03.186985 13952 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:36:03.187014 13952 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:03.197136 13952 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.65:33675
I20260812 06:36:03.197160 14188 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.65:33675 every 8 connection(s)
I20260812 06:36:03.198042 13952 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
I20260812 06:36:03.198328 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 13952
I20260812 06:36:03.198452 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/wal/instance
Aug 12 06:36:03 dist-test-slave-gjrz krb5kdc[13412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516563, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
Aug 12 06:36:03 dist-test-slave-gjrz krb5kdc[13412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516563, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
Aug 12 06:36:03 dist-test-slave-gjrz krb5kdc[13412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516563, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
I20260812 06:36:03.212742 14191 heartbeater.cc:344] Connected to a master server at 127.13.22.126:39439
I20260812 06:36:03.212819 14190 heartbeater.cc:344] Connected to a master server at 127.13.22.124:38293
I20260812 06:36:03.212971 14191 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:03.212980 14190 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:03.213444 14190 heartbeater.cc:507] Master 127.13.22.124:38293 requested a full tablet report, sending...
I20260812 06:36:03.213444 14191 heartbeater.cc:507] Master 127.13.22.126:39439 requested a full tablet report, sending...
I20260812 06:36:03.213467 14194 heartbeater.cc:344] Connected to a master server at 127.13.22.125:34785
I20260812 06:36:03.213613 14194 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:03.213773 14194 heartbeater.cc:507] Master 127.13.22.125:34785 requested a full tablet report, sending...
I20260812 06:36:03.214874 13515 ts_manager.cc:194] Registered new tserver with Master: b238708c52b14df8b23a95e1c3eb16cb (127.13.22.65:33675)
I20260812 06:36:03.214955 13644 ts_manager.cc:194] Registered new tserver with Master: b238708c52b14df8b23a95e1c3eb16cb (127.13.22.65:33675)
I20260812 06:36:03.214955 13781 ts_manager.cc:194] Registered new tserver with Master: b238708c52b14df8b23a95e1c3eb16cb (127.13.22.65:33675)
I20260812 06:36:03.216341 13515 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.22.65@KRBTEST.COM'} at 127.13.22.65:33187
I20260812 06:36:03.220350 13401 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
time=2026-08-12T06:36:03.411Z level=INFO source=main.go:1678 msg="updated GOGC" old=100 new=75
time=2026-08-12T06:36:03.414Z level=INFO source=main.go:744 msg="Leaving GOMAXPROCS=4: CPU quota undefined" component=automaxprocs
time=2026-08-12T06:36:03.415Z level=INFO source=memlimit.go:198 msg="GOMEMLIMIT is updated" component=automemlimit package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=14142549196 previous=9223372036854775807
time=2026-08-12T06:36:03.415Z level=INFO source=main.go:851 msg="Starting Prometheus Server" mode=server version="(version=3.11.2, branch=HEAD, revision=f0f0fdd679dcd6df320b0558b20919f7cd44c407)"
time=2026-08-12T06:36:03.415Z level=INFO source=main.go:856 msg="operational information" build_context="(go=go1.26.2, platform=linux/amd64, user=root@d2684485c347, date=20260413-11:53:40, tags=netgo,builtinassets)" host_details="(Linux 5.4.0-1157-gcp #166~18.04.1-Ubuntu SMP Wed Jan 21 01:57:46 UTC 2026 x86_64 dist-test-slave-gjrz (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)"
time=2026-08-12T06:36:03.649Z level=INFO source=web.go:710 msg="Start listening for connections" component=web address=127.0.0.1:0
time=2026-08-12T06:36:03.650Z level=INFO source=main.go:1410 msg="Starting TSDB ..."
time=2026-08-12T06:36:03.653Z level=INFO source=tls_config.go:354 msg="Listening on" component=web address=127.0.0.1:35915
time=2026-08-12T06:36:03.654Z level=INFO source=tls_config.go:357 msg="TLS is disabled." component=web http2=false address=127.0.0.1:35915
time=2026-08-12T06:36:03.656Z level=INFO source=head.go:698 msg="Replaying on-disk memory mappable chunks if any" component=tsdb
time=2026-08-12T06:36:03.656Z level=INFO source=head.go:784 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=42.094µs
time=2026-08-12T06:36:03.656Z level=INFO source=head.go:792 msg="Replaying WAL, this may take a while" component=tsdb
time=2026-08-12T06:36:03.657Z level=INFO source=head.go:865 msg="WAL segment loaded" component=tsdb segment=0 maxSegment=0 duration=624.277µs
time=2026-08-12T06:36:03.657Z level=INFO source=head.go:902 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=63.8µs wal_replay_duration=699.802µs wbl_replay_duration=79ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=42.094µs total_replay_duration=873.622µs
time=2026-08-12T06:36:03.660Z level=INFO source=main.go:1431 msg="filesystem information" fs_type=OVERLAYFS_SUPER_MAGIC
time=2026-08-12T06:36:03.660Z level=INFO source=main.go:1434 msg="TSDB started"
time=2026-08-12T06:36:03.660Z level=INFO source=main.go:1632 msg="Loading configuration file" filename=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/prometheus.yml
time=2026-08-12T06:36:03.661Z level=INFO source=main.go:1048 msg="TSDB retention updated" duration=15d size=0B percentage=0
time=2026-08-12T06:36:03.667Z level=INFO source=main.go:1671 msg="Completed loading of configuration file" db_storage=96.801µs remote_storage=26.143µs web_handler=521ns query_engine=1.327µs scrape=5.777762ms scrape_sd=275.336µs notify=32.408µs notify_sd=1.682µs rules=34.735µs tracing=148.621µs filename=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly.1786516559891118-13401-0/prometheus.yml totalDuration=7.095603ms
time=2026-08-12T06:36:03.667Z level=INFO source=main.go:1395 msg="Server is ready to receive web requests."
time=2026-08-12T06:36:03.667Z level=INFO source=manager.go:209 msg="Starting rule manager..." component="rule manager"
I20260812 06:36:03.813392 13895 catalog_manager.cc:1295] Loaded cluster ID: dfc9f9dc80044112a456afa9eba54f66
I20260812 06:36:03.813504 13895 catalog_manager.cc:1588] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a: loading cluster ID for follower catalog manager: success
I20260812 06:36:03.816293 13895 catalog_manager.cc:1610] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a: acquiring CA information for follower catalog manager: success
I20260812 06:36:03.817837 13904 catalog_manager.cc:1295] Loaded cluster ID: dfc9f9dc80044112a456afa9eba54f66
I20260812 06:36:03.817895 13904 catalog_manager.cc:1588] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a: loading cluster ID for follower catalog manager: success
I20260812 06:36:03.817925 13895 catalog_manager.cc:1638] T 00000000000000000000000000000000 P cc8f221432c947fcb2f86cb9e5b6e35a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 06:36:03.819437 13904 catalog_manager.cc:1610] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a: acquiring CA information for follower catalog manager: success
I20260812 06:36:03.820320 13904 catalog_manager.cc:1638] T 00000000000000000000000000000000 P bc1868a29f024d18b2558a4a3eb7dd1a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 06:36:04.218484 14191 heartbeater.cc:499] Master 127.13.22.126:39439 was elected leader, sending a full tablet report...
time=2026-08-12T06:36:10.330Z level=WARN source=main.go:1191 msg="Received an OS signal, exiting gracefully..." signal=terminated
time=2026-08-12T06:36:10.331Z level=INFO source=main.go:1216 msg="Stopping scrape discovery manager..."
time=2026-08-12T06:36:10.331Z level=INFO source=main.go:1230 msg="Stopping notify discovery manager..."
time=2026-08-12T06:36:10.331Z level=INFO source=main.go:1245 msg="Stopping rule manager manager..."
time=2026-08-12T06:36:10.331Z level=INFO source=main.go:1212 msg="Scrape discovery manager stopped"
time=2026-08-12T06:36:10.331Z level=INFO source=manager.go:223 msg="Stopping rule manager..." component="rule manager"
time=2026-08-12T06:36:10.331Z level=INFO source=manager.go:239 msg="Rule manager stopped" component="rule manager"
time=2026-08-12T06:36:10.331Z level=INFO source=main.go:1269 msg="Stopping scrape manager..."
time=2026-08-12T06:36:10.331Z level=INFO source=main.go:1226 msg="Notify discovery manager stopped"
time=2026-08-12T06:36:10.331Z level=INFO source=main.go:1241 msg="Rule manager stopped"
time=2026-08-12T06:36:10.331Z level=INFO source=main.go:1284 msg="Stopping tracing manager..."
time=2026-08-12T06:36:10.331Z level=INFO source=main.go:1369 msg="Stopping reloaders..."
time=2026-08-12T06:36:10.331Z level=INFO source=main.go:1458 msg="Stopping storage..."
time=2026-08-12T06:36:10.331Z level=INFO source=main.go:1261 msg="Scrape manager stopped"
time=2026-08-12T06:36:10.331Z level=INFO source=main.go:1280 msg="Tracing manager stopped"
time=2026-08-12T06:36:10.331Z level=INFO source=main.go:1361 msg="Reloaders stopped"
time=2026-08-12T06:36:10.335Z level=INFO source=main.go:1536 msg="Stopping web handler..."
time=2026-08-12T06:36:10.335Z level=INFO source=main.go:1559 msg="Stopping notifier manager..."
time=2026-08-12T06:36:10.335Z level=INFO source=main.go:1454 msg="TSDB stopped"
time=2026-08-12T06:36:10.335Z level=INFO source=manager.go:321 msg="Stopping notification manager..." component=notifier
time=2026-08-12T06:36:10.335Z level=INFO source=main.go:1555 msg="Notifier manager stopped"
time=2026-08-12T06:36:10.335Z level=INFO source=main.go:1532 msg="Web handler stopped"
time=2026-08-12T06:36:10.335Z level=INFO source=main.go:1570 msg="See you next time!"
I20260812 06:36:10.351246 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 13952
I20260812 06:36:10.367323 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 13442
I20260812 06:36:10.381007 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 13587
I20260812 06:36:10.393057 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 13732
2026-08-12T06:36:10Z chronyd exiting
[       OK ] PrometheusSpnegoITest.TestScrapeFailsWithSpnegoOnly (10524 ms)
[ RUN      ] PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly
Loading random data
Initializing database '/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 06:36:10 dist-test-slave-gjrz krb5kdc[14235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 06:36:10 dist-test-slave-gjrz krb5kdc[14235](info): set up 2 sockets
Aug 12 06:36:10 dist-test-slave-gjrz krb5kdc[14235](info): commencing operation
krb5kdc: starting...
W20260812 06:36:12.478017 13401 mini_kdc.cc:121] Time spent starting KDC: real 2.053s	user 0.003s	sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/test-user.keytab.
Aug 12 06:36:12 dist-test-slave-gjrz krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM: 
2026-08-12T06:36:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T06:36:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:12.602126 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.126:32815
--webserver_interface=127.13.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:32815,127.13.22.125:43377,127.13.22.124:39673
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 06:36:12.703634 14264 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:12.703835 14264 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:12.703913 14264 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:12.707108 14264 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:36:12.707166 14264 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:12.707192 14264 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:36:12.707216 14264 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:36:12.711403 14264 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:32815,127.13.22.125:43377,127.13.22.124:39673
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.126:32815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.126
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:12.712356 14264 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:12.713151 14264 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:12.718191 14274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:12.718171 14281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:12.718166 14276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:12.718453 14264 server_base.cc:1061] running on GCE node
I20260812 06:36:12.718888 14264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:12.719744 14264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:12.720899 14264 hybrid_clock.cc:648] HybridClock initialized: now 1786516572720880 us; error 29 us; skew 500 ppm
Aug 12 06:36:12 dist-test-slave-gjrz krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516572, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:12.723479 14264 init.cc:377] Logged in from keytab as kudu/127.13.22.126@KRBTEST.COM (short username kudu)
I20260812 06:36:12.731423 14264 webserver.cc:533] Webserver started at https://127.13.22.126:39409/ using document root <none> and password file <none>
I20260812 06:36:12.732003 14264 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:12.732057 14264 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:12.732249 14264 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:12.733851 14264 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/data/instance:
uuid: "82f166dda65d49518acb4f37651024c8"
format_stamp: "Formatted at 2026-08-12 06:36:12 on dist-test-slave-gjrz"
I20260812 06:36:12.734301 14264 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/wal/instance:
uuid: "82f166dda65d49518acb4f37651024c8"
format_stamp: "Formatted at 2026-08-12 06:36:12 on dist-test-slave-gjrz"
I20260812 06:36:12.737597 14264 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.001s
I20260812 06:36:12.739609 14294 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:12.740553 14264 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.001s
I20260812 06:36:12.740661 14264 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/wal
uuid: "82f166dda65d49518acb4f37651024c8"
format_stamp: "Formatted at 2026-08-12 06:36:12 on dist-test-slave-gjrz"
I20260812 06:36:12.740762 14264 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:12.776240 14264 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:12.778704 14264 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:12.778877 14264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:12.786002 14386 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.126:32815 every 8 connection(s)
I20260812 06:36:12.786008 14264 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.126:32815
I20260812 06:36:12.787007 14264 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
I20260812 06:36:12.789731 14391 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:12.793027 14391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } has no permanent_uuid. Determining permanent_uuid...
I20260812 06:36:12.796774 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 14264
I20260812 06:36:12.796865 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-0/wal/instance
Aug 12 06:36:12 dist-test-slave-gjrz krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516572, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.22.125@KRBTEST.COM; defaulting to no policy
I20260812 06:36:12.806133 14391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.125@KRBTEST.COM" created.
W20260812 06:36:12.806825 14302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.22.125:43377: connect: Connection refused (error 111)
W20260812 06:36:12.807711 14391 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:43377: Network error: Client connection negotiation failed: client connection to 127.13.22.125:43377: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:12.838331 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.125:43377
--webserver_interface=127.13.22.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:32815,127.13.22.125:43377,127.13.22.124:39673
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 06:36:12.856940 14391 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } attempt: 1
W20260812 06:36:12.858458 14391 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:43377: Network error: Client connection negotiation failed: client connection to 127.13.22.125:43377: connect: Connection refused (error 111)
I20260812 06:36:12.926625 14391 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } attempt: 2
W20260812 06:36:12.928103 14391 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:43377: Network error: Client connection negotiation failed: client connection to 127.13.22.125:43377: connect: Connection refused (error 111)
W20260812 06:36:12.939363 14406 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:12.939553 14406 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:12.939623 14406 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:12.942806 14406 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:36:12.942857 14406 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:12.942888 14406 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:36:12.942919 14406 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:36:12.947093 14406 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:32815,127.13.22.125:43377,127.13.22.124:39673
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.125:43377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.125
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:12.948009 14406 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:12.948777 14406 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:12.953724 14414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:12.953729 14415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:12.953855 14406 server_base.cc:1061] running on GCE node
W20260812 06:36:12.953969 14417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:12.954423 14406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:12.955190 14406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:12.956342 14406 hybrid_clock.cc:648] HybridClock initialized: now 1786516572956315 us; error 38 us; skew 500 ppm
Aug 12 06:36:12 dist-test-slave-gjrz krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516572, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:12.958845 14406 init.cc:377] Logged in from keytab as kudu/127.13.22.125@KRBTEST.COM (short username kudu)
I20260812 06:36:12.966635 14406 webserver.cc:533] Webserver started at https://127.13.22.125:46867/ using document root <none> and password file <none>
I20260812 06:36:12.967206 14406 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:12.967257 14406 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:12.967475 14406 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:12.969055 14406 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/data/instance:
uuid: "e30d4ea34f9f40edb062914e40c612ca"
format_stamp: "Formatted at 2026-08-12 06:36:12 on dist-test-slave-gjrz"
I20260812 06:36:12.969494 14406 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/wal/instance:
uuid: "e30d4ea34f9f40edb062914e40c612ca"
format_stamp: "Formatted at 2026-08-12 06:36:12 on dist-test-slave-gjrz"
I20260812 06:36:12.972762 14406 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 06:36:12.974689 14434 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:12.975633 14406 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 06:36:12.975739 14406 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/wal
uuid: "e30d4ea34f9f40edb062914e40c612ca"
format_stamp: "Formatted at 2026-08-12 06:36:12 on dist-test-slave-gjrz"
I20260812 06:36:12.975816 14406 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:12.989897 14406 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:12.992075 14406 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:12.992229 14406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:12.999116 14406 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.125:43377
I20260812 06:36:12.999123 14527 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.125:43377 every 8 connection(s)
I20260812 06:36:13.000110 14406 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
I20260812 06:36:13.002588 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 14406
I20260812 06:36:13.002699 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-1/wal/instance
I20260812 06:36:13.002863 14528 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:13.006222 14528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.22.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.124@KRBTEST.COM" created.
Aug 12 06:36:13 dist-test-slave-gjrz krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516572, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:36:13.018826 14528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } has no permanent_uuid. Determining permanent_uuid...
I20260812 06:36:13.019297 14391 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } attempt: 3
Aug 12 06:36:13 dist-test-slave-gjrz krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516572, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
Aug 12 06:36:13 dist-test-slave-gjrz krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516572, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:13.033177 14528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } has no permanent_uuid. Determining permanent_uuid...
W20260812 06:36:13.034322 14437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.22.124:39673: connect: Connection refused (error 111)
W20260812 06:36:13.035142 14528 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:39673: Network error: Client connection negotiation failed: client connection to 127.13.22.124:39673: connect: Connection refused (error 111)
I20260812 06:36:13.035979 14391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for HTTP/127.13.22.124@KRBTEST.COM; defaulting to no policy
W20260812 06:36:13.037168 14391 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:39673: Network error: Client connection negotiation failed: client connection to 127.13.22.124:39673: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:13.050050 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.124:39673
--webserver_interface=127.13.22.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:32815,127.13.22.125:43377,127.13.22.124:39673
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 06:36:13.068333 14391 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } attempt: 1
W20260812 06:36:13.069694 14391 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:39673: Network error: Client connection negotiation failed: client connection to 127.13.22.124:39673: connect: Connection refused (error 111)
I20260812 06:36:13.084353 14528 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } attempt: 1
W20260812 06:36:13.085561 14528 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:39673: Network error: Client connection negotiation failed: client connection to 127.13.22.124:39673: connect: Connection refused (error 111)
I20260812 06:36:13.144850 14391 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } attempt: 2
W20260812 06:36:13.146216 14391 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:39673: Network error: Client connection negotiation failed: client connection to 127.13.22.124:39673: connect: Connection refused (error 111)
W20260812 06:36:13.152613 14542 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:13.152786 14542 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:13.152849 14542 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260812 06:36:13.153695 14528 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } attempt: 2
W20260812 06:36:13.154834 14528 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:39673: Network error: Client connection negotiation failed: client connection to 127.13.22.124:39673: connect: Connection refused (error 111)
W20260812 06:36:13.156312 14542 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:36:13.156371 14542 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:13.156399 14542 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:36:13.156430 14542 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:36:13.160557 14542 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:32815,127.13.22.125:43377,127.13.22.124:39673
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.124:39673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.124
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:13.161672 14542 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:13.162483 14542 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:13.167418 14561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:13.167474 14553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:13.167424 14554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:13.167658 14542 server_base.cc:1061] running on GCE node
I20260812 06:36:13.168149 14542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:13.168902 14542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:13.170050 14542 hybrid_clock.cc:648] HybridClock initialized: now 1786516573170034 us; error 30 us; skew 500 ppm
Aug 12 06:36:13 dist-test-slave-gjrz krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516573, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:13.172529 14542 init.cc:377] Logged in from keytab as kudu/127.13.22.124@KRBTEST.COM (short username kudu)
I20260812 06:36:13.180176 14542 webserver.cc:533] Webserver started at https://127.13.22.124:36917/ using document root <none> and password file <none>
I20260812 06:36:13.180742 14542 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:13.180792 14542 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:13.180977 14542 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:13.182566 14542 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/data/instance:
uuid: "ab8851d8325a4748914d3f0e0cd4abbe"
format_stamp: "Formatted at 2026-08-12 06:36:13 on dist-test-slave-gjrz"
I20260812 06:36:13.183001 14542 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/wal/instance:
uuid: "ab8851d8325a4748914d3f0e0cd4abbe"
format_stamp: "Formatted at 2026-08-12 06:36:13 on dist-test-slave-gjrz"
I20260812 06:36:13.186120 14542 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20260812 06:36:13.188153 14573 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:13.189291 14542 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:13.189412 14542 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/wal
uuid: "ab8851d8325a4748914d3f0e0cd4abbe"
format_stamp: "Formatted at 2026-08-12 06:36:13 on dist-test-slave-gjrz"
I20260812 06:36:13.189492 14542 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:13.201753 14542 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:13.204244 14542 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:13.204404 14542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:13.211058 14686 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.124:39673 every 8 connection(s)
I20260812 06:36:13.211056 14542 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.124:39673
I20260812 06:36:13.212060 14542 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
I20260812 06:36:13.214617 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 14542
I20260812 06:36:13.214730 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/master-2/wal/instance
I20260812 06:36:13.214799 14687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:13.218818 14687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:36:13 dist-test-slave-gjrz krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
Aug 12 06:36:13 dist-test-slave-gjrz krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516573, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
Aug 12 06:36:13 dist-test-slave-gjrz krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
I20260812 06:36:13.231722 14687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:36:13 dist-test-slave-gjrz krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
Aug 12 06:36:13 dist-test-slave-gjrz krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516573, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
I20260812 06:36:13.238750 14687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:36:13 dist-test-slave-gjrz krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516573, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:36:13.245362 14391 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } attempt: 3
I20260812 06:36:13.245985 14528 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } attempt: 3
I20260812 06:36:13.247418 14687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe: Bootstrap starting.
Aug 12 06:36:13 dist-test-slave-gjrz krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516572, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:36:13.250319 14687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe: Neither blocks nor log segments found. Creating new log.
Aug 12 06:36:13 dist-test-slave-gjrz krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516572, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:36:13.251185 14687 log.cc:826] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:13.252807 14687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe: No bootstrap required, opened a new log
I20260812 06:36:13.255795 14687 raft_consensus.cc:359] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f166dda65d49518acb4f37651024c8" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } } peers { permanent_uuid: "e30d4ea34f9f40edb062914e40c612ca" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } } peers { permanent_uuid: "ab8851d8325a4748914d3f0e0cd4abbe" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } }
I20260812 06:36:13.255985 14687 raft_consensus.cc:385] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:13.256044 14687 raft_consensus.cc:740] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab8851d8325a4748914d3f0e0cd4abbe, State: Initialized, Role: FOLLOWER
I20260812 06:36:13.256157 14391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8: Bootstrap starting.
I20260812 06:36:13.256471 14687 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f166dda65d49518acb4f37651024c8" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } } peers { permanent_uuid: "e30d4ea34f9f40edb062914e40c612ca" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } } peers { permanent_uuid: "ab8851d8325a4748914d3f0e0cd4abbe" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } }
I20260812 06:36:13.256767 14528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca: Bootstrap starting.
I20260812 06:36:13.257599 14687 sys_catalog.cc:565] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:36:13.258379 14391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:13.259001 14528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:13.259130 14391 log.cc:826] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:13.259519 14703 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f166dda65d49518acb4f37651024c8" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } } peers { permanent_uuid: "e30d4ea34f9f40edb062914e40c612ca" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } } peers { permanent_uuid: "ab8851d8325a4748914d3f0e0cd4abbe" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } } }
I20260812 06:36:13.259687 14703 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:13.259855 14528 log.cc:826] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:13.261425 14528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca: No bootstrap required, opened a new log
I20260812 06:36:13.261183 14542 master_runner.cc:187] Error getting master registration for 127.13.22.126:32815: OK, instance_id { permanent_uuid: "82f166dda65d49518acb4f37651024c8" instance_seqno: 1786516572784067 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 06:36:13.263348 14542 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 06:36:13.263389 14723 catalog_manager.cc:1594] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:36:13.263442 14723 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:36:13.264430 14528 raft_consensus.cc:359] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f166dda65d49518acb4f37651024c8" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } } peers { permanent_uuid: "e30d4ea34f9f40edb062914e40c612ca" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } } peers { permanent_uuid: "ab8851d8325a4748914d3f0e0cd4abbe" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } }
I20260812 06:36:13.264611 14528 raft_consensus.cc:385] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:13.264671 14528 raft_consensus.cc:740] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30d4ea34f9f40edb062914e40c612ca, State: Initialized, Role: FOLLOWER
I20260812 06:36:13.265095 14528 consensus_queue.cc:260] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f166dda65d49518acb4f37651024c8" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } } peers { permanent_uuid: "e30d4ea34f9f40edb062914e40c612ca" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } } peers { permanent_uuid: "ab8851d8325a4748914d3f0e0cd4abbe" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } }
I20260812 06:36:13.266209 14528 sys_catalog.cc:565] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:36:13.267472 14728 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f166dda65d49518acb4f37651024c8" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } } peers { permanent_uuid: "e30d4ea34f9f40edb062914e40c612ca" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } } peers { permanent_uuid: "ab8851d8325a4748914d3f0e0cd4abbe" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } } }
I20260812 06:36:13.267561 14728 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:13.268311 14391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8: No bootstrap required, opened a new log
I20260812 06:36:13.271369 14391 raft_consensus.cc:359] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f166dda65d49518acb4f37651024c8" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } } peers { permanent_uuid: "e30d4ea34f9f40edb062914e40c612ca" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } } peers { permanent_uuid: "ab8851d8325a4748914d3f0e0cd4abbe" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } }
I20260812 06:36:13.271540 14391 raft_consensus.cc:385] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:13.271615 14391 raft_consensus.cc:740] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f166dda65d49518acb4f37651024c8, State: Initialized, Role: FOLLOWER
I20260812 06:36:13.272032 14391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f166dda65d49518acb4f37651024c8" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } } peers { permanent_uuid: "e30d4ea34f9f40edb062914e40c612ca" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } } peers { permanent_uuid: "ab8851d8325a4748914d3f0e0cd4abbe" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } }
I20260812 06:36:13.273092 14391 sys_catalog.cc:565] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 06:36:13.276129 14749 catalog_manager.cc:1594] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:36:13.276194 14749 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:36:13.274376 14406 master_runner.cc:187] Error getting master registration for 127.13.22.126:32815: OK, instance_id { permanent_uuid: "82f166dda65d49518acb4f37651024c8" instance_seqno: 1786516572784067 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 06:36:13.277002 14406 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 06:36:13.280222 14305 tablet.cc:2406] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 06:36:13.281847 14772 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:36:13.281932 14772 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:36:13.282472 14264 master_runner.cc:264] No leader master found from master 82f166dda65d49518acb4f37651024c8
I20260812 06:36:13.282534 14264 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 06:36:13.287480 14738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f166dda65d49518acb4f37651024c8" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } } peers { permanent_uuid: "e30d4ea34f9f40edb062914e40c612ca" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } } peers { permanent_uuid: "ab8851d8325a4748914d3f0e0cd4abbe" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } } }
I20260812 06:36:13.287665 14738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 06:36:13.455546 14586 tablet.cc:2406] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 06:36:13.462764 14728 raft_consensus.cc:493] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:36:13.462976 14728 raft_consensus.cc:515] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f166dda65d49518acb4f37651024c8" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } } peers { permanent_uuid: "e30d4ea34f9f40edb062914e40c612ca" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } } peers { permanent_uuid: "ab8851d8325a4748914d3f0e0cd4abbe" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } }
I20260812 06:36:13.464133 14728 leader_election.cc:290] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82f166dda65d49518acb4f37651024c8 (127.13.22.126:32815), ab8851d8325a4748914d3f0e0cd4abbe (127.13.22.124:39673)
I20260812 06:36:13.464643 14347 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e30d4ea34f9f40edb062914e40c612ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f166dda65d49518acb4f37651024c8" is_pre_election: true
I20260812 06:36:13.464740 14638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e30d4ea34f9f40edb062914e40c612ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab8851d8325a4748914d3f0e0cd4abbe" is_pre_election: true
I20260812 06:36:13.464954 14347 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e30d4ea34f9f40edb062914e40c612ca in term 0.
I20260812 06:36:13.465011 14638 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e30d4ea34f9f40edb062914e40c612ca in term 0.
I20260812 06:36:13.465430 14437 leader_election.cc:304] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab8851d8325a4748914d3f0e0cd4abbe, e30d4ea34f9f40edb062914e40c612ca; no voters: 
I20260812 06:36:13.465724 14728 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 06:36:13.465827 14728 raft_consensus.cc:493] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 06:36:13.465895 14728 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:13.467142 14728 raft_consensus.cc:515] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f166dda65d49518acb4f37651024c8" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } } peers { permanent_uuid: "e30d4ea34f9f40edb062914e40c612ca" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } } peers { permanent_uuid: "ab8851d8325a4748914d3f0e0cd4abbe" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } }
I20260812 06:36:13.467666 14728 leader_election.cc:290] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [CANDIDATE]: Term 1 election: Requested vote from peers 82f166dda65d49518acb4f37651024c8 (127.13.22.126:32815), ab8851d8325a4748914d3f0e0cd4abbe (127.13.22.124:39673)
I20260812 06:36:13.467947 14347 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e30d4ea34f9f40edb062914e40c612ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f166dda65d49518acb4f37651024c8"
I20260812 06:36:13.468055 14347 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:13.468374 14638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e30d4ea34f9f40edb062914e40c612ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab8851d8325a4748914d3f0e0cd4abbe"
I20260812 06:36:13.468501 14638 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:13.469126 14347 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e30d4ea34f9f40edb062914e40c612ca in term 1.
I20260812 06:36:13.469431 14435 leader_election.cc:304] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82f166dda65d49518acb4f37651024c8, e30d4ea34f9f40edb062914e40c612ca; no voters: 
I20260812 06:36:13.469594 14728 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [term 1 FOLLOWER]: Leader election won for term 1
I20260812 06:36:13.469658 14638 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e30d4ea34f9f40edb062914e40c612ca in term 1.
I20260812 06:36:13.469794 14728 raft_consensus.cc:697] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [term 1 LEADER]: Becoming Leader. State: Replica: e30d4ea34f9f40edb062914e40c612ca, State: Running, Role: LEADER
I20260812 06:36:13.470077 14728 consensus_queue.cc:237] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f166dda65d49518acb4f37651024c8" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } } peers { permanent_uuid: "e30d4ea34f9f40edb062914e40c612ca" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } } peers { permanent_uuid: "ab8851d8325a4748914d3f0e0cd4abbe" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } }
I20260812 06:36:13.472239 14774 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader e30d4ea34f9f40edb062914e40c612ca. Latest consensus state: current_term: 1 leader_uuid: "e30d4ea34f9f40edb062914e40c612ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f166dda65d49518acb4f37651024c8" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } } peers { permanent_uuid: "e30d4ea34f9f40edb062914e40c612ca" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } } peers { permanent_uuid: "ab8851d8325a4748914d3f0e0cd4abbe" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } } }
I20260812 06:36:13.472404 14774 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [sys.catalog]: This master's current role is: LEADER
I20260812 06:36:13.472999 14777 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 06:36:13.474057 14777 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 06:36:13.480103 14638 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe [term 1 FOLLOWER]: Refusing update from remote peer e30d4ea34f9f40edb062914e40c612ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:36:13.481076 14347 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8 [term 1 FOLLOWER]: Refusing update from remote peer e30d4ea34f9f40edb062914e40c612ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:36:13.481091 14774 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab8851d8325a4748914d3f0e0cd4abbe" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:36:13.481684 14774 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f166dda65d49518acb4f37651024c8" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:36:13.485448 14703 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader e30d4ea34f9f40edb062914e40c612ca. Latest consensus state: current_term: 1 leader_uuid: "e30d4ea34f9f40edb062914e40c612ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f166dda65d49518acb4f37651024c8" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } } peers { permanent_uuid: "e30d4ea34f9f40edb062914e40c612ca" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } } peers { permanent_uuid: "ab8851d8325a4748914d3f0e0cd4abbe" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } } }
I20260812 06:36:13.485570 14703 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:13.486594 14738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e30d4ea34f9f40edb062914e40c612ca. Latest consensus state: current_term: 1 leader_uuid: "e30d4ea34f9f40edb062914e40c612ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f166dda65d49518acb4f37651024c8" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } } peers { permanent_uuid: "e30d4ea34f9f40edb062914e40c612ca" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } } peers { permanent_uuid: "ab8851d8325a4748914d3f0e0cd4abbe" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } } }
I20260812 06:36:13.486711 14738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:13.488457 14774 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e30d4ea34f9f40edb062914e40c612ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f166dda65d49518acb4f37651024c8" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } } peers { permanent_uuid: "e30d4ea34f9f40edb062914e40c612ca" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } } peers { permanent_uuid: "ab8851d8325a4748914d3f0e0cd4abbe" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } } }
I20260812 06:36:13.488552 14774 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [sys.catalog]: This master's current role is: LEADER
I20260812 06:36:13.488770 14779 mvcc.cc:204] Tried to move back new op lower bound from 7317571884964257792 to 7317571884936253440. Current Snapshot: MvccSnapshot[applied={T|T < 7317571884964257792}]
I20260812 06:36:13.489459 14703 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e30d4ea34f9f40edb062914e40c612ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f166dda65d49518acb4f37651024c8" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } } peers { permanent_uuid: "e30d4ea34f9f40edb062914e40c612ca" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } } peers { permanent_uuid: "ab8851d8325a4748914d3f0e0cd4abbe" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } } }
I20260812 06:36:13.489529 14785 mvcc.cc:204] Tried to move back new op lower bound from 7317571884964257792 to 7317571884936253440. Current Snapshot: MvccSnapshot[applied={T|T < 7317571884964257792}]
I20260812 06:36:13.489562 14703 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:13.489656 14728 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e30d4ea34f9f40edb062914e40c612ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f166dda65d49518acb4f37651024c8" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } } peers { permanent_uuid: "e30d4ea34f9f40edb062914e40c612ca" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } } peers { permanent_uuid: "ab8851d8325a4748914d3f0e0cd4abbe" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } } }
I20260812 06:36:13.489753 14728 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca [sys.catalog]: This master's current role is: LEADER
I20260812 06:36:13.490764 14738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e30d4ea34f9f40edb062914e40c612ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f166dda65d49518acb4f37651024c8" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32815 } } peers { permanent_uuid: "e30d4ea34f9f40edb062914e40c612ca" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43377 } } peers { permanent_uuid: "ab8851d8325a4748914d3f0e0cd4abbe" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 39673 } } }
I20260812 06:36:13.490855 14738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:13.491493 14789 mvcc.cc:204] Tried to move back new op lower bound from 7317571884964257792 to 7317571884936253440. Current Snapshot: MvccSnapshot[applied={T|T < 7317571884964257792}]
I20260812 06:36:13.491971 14777 catalog_manager.cc:1383] Generated new cluster ID: 0fc8a6b568f04944bbe27ff330ed042b
I20260812 06:36:13.492020 14777 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 06:36:13.509502 14777 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 06:36:13.510347 14777 catalog_manager.cc:1540] Loading token signing keys...
I20260812 06:36:13.516772 14777 catalog_manager.cc:6092] T 00000000000000000000000000000000 P e30d4ea34f9f40edb062914e40c612ca: Generated new TSK 0
I20260812 06:36:13.517365 14777 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:13.577682 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.22.65:0
--local_ip_for_outbound_sockets=127.13.22.65
--webserver_interface=127.13.22.65
--webserver_port=0
--tserver_master_addrs=127.13.22.126:32815,127.13.22.125:43377,127.13.22.124:39673
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 06:36:13.678151 14808 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:13.678331 14808 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:13.678397 14808 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:13.681672 14808 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:13.681790 14808 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.22.65
I20260812 06:36:13.685864 14808 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.65
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_require_spnego=true
--tserver_master_addrs=127.13.22.126:32815,127.13.22.125:43377,127.13.22.124:39673
--never_fsync=true
--heap_profile_path=/tmp/kudu.14808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.22.65

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:13.686836 14808 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:13.687655 14808 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:13.693840 14818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:13.693935 14808 server_base.cc:1061] running on GCE node
W20260812 06:36:13.693876 14819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:13.693842 14823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:13.694422 14808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:13.694921 14808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:13.696087 14808 hybrid_clock.cc:648] HybridClock initialized: now 1786516573696064 us; error 33 us; skew 500 ppm
Aug 12 06:36:13 dist-test-slave-gjrz krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516573, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:13.698700 14808 init.cc:377] Logged in from keytab as kudu/127.13.22.65@KRBTEST.COM (short username kudu)
I20260812 06:36:13.706303 14808 webserver.cc:533] Webserver started at https://127.13.22.65:41823/ using document root <none> and password file <none>
I20260812 06:36:13.706872 14808 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:13.706921 14808 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:13.707113 14808 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:13.708742 14808 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/data/instance:
uuid: "79bff81f60f04a108985866e8be383d2"
format_stamp: "Formatted at 2026-08-12 06:36:13 on dist-test-slave-gjrz"
I20260812 06:36:13.709177 14808 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/wal/instance:
uuid: "79bff81f60f04a108985866e8be383d2"
format_stamp: "Formatted at 2026-08-12 06:36:13 on dist-test-slave-gjrz"
I20260812 06:36:13.712450 14808 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 06:36:13.714362 14838 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:13.715327 14808 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 06:36:13.715417 14808 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/wal
uuid: "79bff81f60f04a108985866e8be383d2"
format_stamp: "Formatted at 2026-08-12 06:36:13 on dist-test-slave-gjrz"
I20260812 06:36:13.715488 14808 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:13.739429 14808 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:13.742066 14808 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:13.742241 14808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:13.742805 14808 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:13.743783 14808 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:36:13.743831 14808 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:13.743873 14808 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:36:13.743901 14808 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:13.752671 14808 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.65:32791
I20260812 06:36:13.752707 15068 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.65:32791 every 8 connection(s)
I20260812 06:36:13.753578 14808 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
Aug 12 06:36:13 dist-test-slave-gjrz krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516573, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
Aug 12 06:36:13 dist-test-slave-gjrz krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516573, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
I20260812 06:36:13.762621 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 14808
I20260812 06:36:13.762720 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/minicluster-data/ts-0/wal/instance
Aug 12 06:36:13 dist-test-slave-gjrz krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516573, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:36:13.767962 15069 heartbeater.cc:344] Connected to a master server at 127.13.22.126:32815
I20260812 06:36:13.768219 15069 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:13.768329 15071 heartbeater.cc:344] Connected to a master server at 127.13.22.125:43377
I20260812 06:36:13.768417 15071 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:13.768776 15069 heartbeater.cc:507] Master 127.13.22.126:32815 requested a full tablet report, sending...
I20260812 06:36:13.768775 15071 heartbeater.cc:507] Master 127.13.22.125:43377 requested a full tablet report, sending...
I20260812 06:36:13.770151 14474 ts_manager.cc:194] Registered new tserver with Master: 79bff81f60f04a108985866e8be383d2 (127.13.22.65:32791)
I20260812 06:36:13.770195 14323 ts_manager.cc:194] Registered new tserver with Master: 79bff81f60f04a108985866e8be383d2 (127.13.22.65:32791)
I20260812 06:36:13.771289 14474 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.22.65@KRBTEST.COM'} at 127.13.22.65:48861
I20260812 06:36:13.771469 15070 heartbeater.cc:344] Connected to a master server at 127.13.22.124:39673
I20260812 06:36:13.771554 15070 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:13.771730 15070 heartbeater.cc:507] Master 127.13.22.124:39673 requested a full tablet report, sending...
I20260812 06:36:13.772624 14608 ts_manager.cc:194] Registered new tserver with Master: 79bff81f60f04a108985866e8be383d2 (127.13.22.65:32791)
I20260812 06:36:13.782539 13401 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
time=2026-08-12T06:36:13.843Z level=INFO source=main.go:1678 msg="updated GOGC" old=100 new=75
time=2026-08-12T06:36:13.845Z level=INFO source=main.go:744 msg="Leaving GOMAXPROCS=4: CPU quota undefined" component=automaxprocs
time=2026-08-12T06:36:13.845Z level=INFO source=memlimit.go:198 msg="GOMEMLIMIT is updated" component=automemlimit package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=14142549196 previous=9223372036854775807
time=2026-08-12T06:36:13.845Z level=INFO source=main.go:851 msg="Starting Prometheus Server" mode=server version="(version=3.11.2, branch=HEAD, revision=f0f0fdd679dcd6df320b0558b20919f7cd44c407)"
time=2026-08-12T06:36:13.845Z level=INFO source=main.go:856 msg="operational information" build_context="(go=go1.26.2, platform=linux/amd64, user=root@d2684485c347, date=20260413-11:53:40, tags=netgo,builtinassets)" host_details="(Linux 5.4.0-1157-gcp #166~18.04.1-Ubuntu SMP Wed Jan 21 01:57:46 UTC 2026 x86_64 dist-test-slave-gjrz (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)"
time=2026-08-12T06:36:14.078Z level=INFO source=web.go:710 msg="Start listening for connections" component=web address=127.0.0.1:0
time=2026-08-12T06:36:14.078Z level=INFO source=main.go:1410 msg="Starting TSDB ..."
time=2026-08-12T06:36:14.080Z level=INFO source=tls_config.go:354 msg="Listening on" component=web address=127.0.0.1:34281
time=2026-08-12T06:36:14.080Z level=INFO source=tls_config.go:357 msg="TLS is disabled." component=web http2=false address=127.0.0.1:34281
time=2026-08-12T06:36:14.082Z level=INFO source=head.go:698 msg="Replaying on-disk memory mappable chunks if any" component=tsdb
time=2026-08-12T06:36:14.082Z level=INFO source=head.go:784 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=2.477µs
time=2026-08-12T06:36:14.082Z level=INFO source=head.go:792 msg="Replaying WAL, this may take a while" component=tsdb
time=2026-08-12T06:36:14.083Z level=INFO source=head.go:865 msg="WAL segment loaded" component=tsdb segment=0 maxSegment=0 duration=305.739µs
time=2026-08-12T06:36:14.083Z level=INFO source=head.go:902 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=50.428µs wal_replay_duration=337.154µs wbl_replay_duration=150ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=2.477µs total_replay_duration=425.802µs
time=2026-08-12T06:36:14.085Z level=INFO source=main.go:1431 msg="filesystem information" fs_type=OVERLAYFS_SUPER_MAGIC
time=2026-08-12T06:36:14.085Z level=INFO source=main.go:1434 msg="TSDB started"
time=2026-08-12T06:36:14.085Z level=INFO source=main.go:1632 msg="Loading configuration file" filename=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/prometheus.yml
time=2026-08-12T06:36:14.085Z level=INFO source=main.go:1048 msg="TSDB retention updated" duration=15d size=0B percentage=0
time=2026-08-12T06:36:14.087Z level=INFO source=main.go:1671 msg="Completed loading of configuration file" db_storage=39.699µs remote_storage=2.346µs web_handler=547ns query_engine=987ns scrape=1.056302ms scrape_sd=342.608µs notify=2.111µs notify_sd=1.094µs rules=2.315µs tracing=5.671µs filename=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly.1786516559891118-13401-0/prometheus.yml totalDuration=1.835326ms
time=2026-08-12T06:36:14.087Z level=INFO source=main.go:1395 msg="Server is ready to receive web requests."
time=2026-08-12T06:36:14.087Z level=INFO source=manager.go:209 msg="Starting rule manager..." component="rule manager"
time=2026-08-12T06:36:14.110Z level=ERROR source=refresh.go:72 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:36:14.116Z level=ERROR source=refresh.go:72 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:36:14.118Z level=ERROR source=refresh.go:72 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
I20260812 06:36:14.264665 14723 catalog_manager.cc:1295] Loaded cluster ID: 0fc8a6b568f04944bbe27ff330ed042b
I20260812 06:36:14.264748 14723 catalog_manager.cc:1588] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe: loading cluster ID for follower catalog manager: success
I20260812 06:36:14.266465 14723 catalog_manager.cc:1610] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe: acquiring CA information for follower catalog manager: success
I20260812 06:36:14.267668 14723 catalog_manager.cc:1638] T 00000000000000000000000000000000 P ab8851d8325a4748914d3f0e0cd4abbe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 06:36:14.283440 14772 catalog_manager.cc:1295] Loaded cluster ID: 0fc8a6b568f04944bbe27ff330ed042b
I20260812 06:36:14.283524 14772 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8: loading cluster ID for follower catalog manager: success
I20260812 06:36:14.286017 14772 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8: acquiring CA information for follower catalog manager: success
I20260812 06:36:14.287520 14772 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 82f166dda65d49518acb4f37651024c8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 06:36:14.772966 15071 heartbeater.cc:499] Master 127.13.22.125:43377 was elected leader, sending a full tablet report...
time=2026-08-12T06:36:16.112Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:36:16.118Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:36:16.119Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:36:18.112Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:36:18.117Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:36:18.119Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:36:20.112Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:36:20.118Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:36:20.119Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:36:21.864Z level=WARN source=main.go:1191 msg="Received an OS signal, exiting gracefully..." signal=terminated
time=2026-08-12T06:36:21.864Z level=INFO source=main.go:1216 msg="Stopping scrape discovery manager..."
time=2026-08-12T06:36:21.864Z level=INFO source=main.go:1230 msg="Stopping notify discovery manager..."
time=2026-08-12T06:36:21.864Z level=INFO source=main.go:1245 msg="Stopping rule manager manager..."
time=2026-08-12T06:36:21.864Z level=INFO source=manager.go:223 msg="Stopping rule manager..." component="rule manager"
time=2026-08-12T06:36:21.864Z level=INFO source=manager.go:239 msg="Rule manager stopped" component="rule manager"
time=2026-08-12T06:36:21.864Z level=INFO source=main.go:1269 msg="Stopping scrape manager..."
time=2026-08-12T06:36:21.864Z level=INFO source=main.go:1284 msg="Stopping tracing manager..."
time=2026-08-12T06:36:21.864Z level=INFO source=main.go:1369 msg="Stopping reloaders..."
time=2026-08-12T06:36:21.864Z level=INFO source=main.go:1458 msg="Stopping storage..."
time=2026-08-12T06:36:21.864Z level=INFO source=main.go:1261 msg="Scrape manager stopped"
time=2026-08-12T06:36:21.864Z level=INFO source=main.go:1226 msg="Notify discovery manager stopped"
time=2026-08-12T06:36:21.864Z level=INFO source=main.go:1280 msg="Tracing manager stopped"
time=2026-08-12T06:36:21.864Z level=INFO source=main.go:1241 msg="Rule manager stopped"
time=2026-08-12T06:36:21.864Z level=INFO source=main.go:1212 msg="Scrape discovery manager stopped"
time=2026-08-12T06:36:21.864Z level=INFO source=main.go:1361 msg="Reloaders stopped"
time=2026-08-12T06:36:21.865Z level=INFO source=main.go:1536 msg="Stopping web handler..."
time=2026-08-12T06:36:21.865Z level=INFO source=main.go:1559 msg="Stopping notifier manager..."
time=2026-08-12T06:36:21.865Z level=INFO source=manager.go:321 msg="Stopping notification manager..." component=notifier
time=2026-08-12T06:36:21.866Z level=INFO source=main.go:1555 msg="Notifier manager stopped"
time=2026-08-12T06:36:21.866Z level=INFO source=main.go:1454 msg="TSDB stopped"
time=2026-08-12T06:36:21.866Z level=INFO source=main.go:1532 msg="Web handler stopped"
time=2026-08-12T06:36:21.866Z level=INFO source=main.go:1570 msg="See you next time!"
I20260812 06:36:21.874805 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 14808
I20260812 06:36:21.887511 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 14264
I20260812 06:36:21.905799 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 14406
I20260812 06:36:21.917642 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 14542
2026-08-12T06:36:21Z chronyd exiting
[       OK ] PrometheusSpnegoITest.TestSdDiscoveryFailsWithSpnegoOnly (11523 ms)
[----------] 2 tests from PrometheusSpnegoITest (22047 ms total)

[----------] 6 tests from PrometheusTokenITest
[ RUN      ] PrometheusTokenITest.TestCorrectTokenScrapeSucceeds
Loading random data
Initializing database '/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 06:36:21 dist-test-slave-gjrz krb5kdc[15125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 06:36:21 dist-test-slave-gjrz krb5kdc[15125](info): set up 2 sockets
Aug 12 06:36:21 dist-test-slave-gjrz krb5kdc[15125](info): commencing operation
krb5kdc: starting...
W20260812 06:36:24.007718 13401 mini_kdc.cc:121] Time spent starting KDC: real 2.060s	user 0.000s	sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/test-user.keytab.
Aug 12 06:36:24 dist-test-slave-gjrz krb5kdc[15125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM: 
2026-08-12T06:36:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T06:36:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:24.135960 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.126:32771
--webserver_interface=127.13.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:46633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:32771,127.13.22.125:37667,127.13.22.124:46479
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 06:36:24.237953 15153 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:24.238147 15153 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:24.238215 15153 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:24.241415 15153 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:36:24.241474 15153 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:24.241508 15153 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:36:24.241539 15153 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:36:24.245638 15153 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:46633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:32771,127.13.22.125:37667,127.13.22.124:46479
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.126:32771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.126
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:24.246520 15153 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:24.247325 15153 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 06:36:24.252687 15153 server_base.cc:1061] running on GCE node
W20260812 06:36:24.252717 15162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:24.252718 15158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:24.252744 15159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:24.253373 15153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:24.254112 15153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:24.255282 15153 hybrid_clock.cc:648] HybridClock initialized: now 1786516584255238 us; error 33 us; skew 500 ppm
Aug 12 06:36:24 dist-test-slave-gjrz krb5kdc[15125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516584, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:24.257679 15153 init.cc:377] Logged in from keytab as kudu/127.13.22.126@KRBTEST.COM (short username kudu)
I20260812 06:36:24.269176 15153 webserver.cc:533] Webserver started at https://127.13.22.126:35471/ using document root <none> and password file <none>
I20260812 06:36:24.269740 15153 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:24.269786 15153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:24.269959 15153 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:24.271600 15153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-0/data/instance:
uuid: "b4898a5f400c48ebb5675aa20e77ffb6"
format_stamp: "Formatted at 2026-08-12 06:36:24 on dist-test-slave-gjrz"
I20260812 06:36:24.272008 15153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-0/wal/instance:
uuid: "b4898a5f400c48ebb5675aa20e77ffb6"
format_stamp: "Formatted at 2026-08-12 06:36:24 on dist-test-slave-gjrz"
I20260812 06:36:24.275180 15153 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.001s
I20260812 06:36:24.277174 15176 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:24.278119 15153 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:24.278214 15153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-0/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-0/wal
uuid: "b4898a5f400c48ebb5675aa20e77ffb6"
format_stamp: "Formatted at 2026-08-12 06:36:24 on dist-test-slave-gjrz"
I20260812 06:36:24.278287 15153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:24.287949 15153 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:24.290390 15153 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:24.290524 15153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:24.297456 15153 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.126:32771
I20260812 06:36:24.297463 15272 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.126:32771 every 8 connection(s)
I20260812 06:36:24.298393 15153 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
I20260812 06:36:24.300282 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 15153
I20260812 06:36:24.300395 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-0/wal/instance
I20260812 06:36:24.301453 15273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:24.305182 15273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.22.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.125@KRBTEST.COM" created.
Aug 12 06:36:24 dist-test-slave-gjrz krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516584, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:36:24.318955 15273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } has no permanent_uuid. Determining permanent_uuid...
W20260812 06:36:24.319756 15177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.22.125:37667: connect: Connection refused (error 111)
W20260812 06:36:24.320544 15273 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:37667: Network error: Client connection negotiation failed: client connection to 127.13.22.125:37667: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:24.343303 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.125:37667
--webserver_interface=127.13.22.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:46633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:32771,127.13.22.125:37667,127.13.22.124:46479
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 06:36:24.369762 15273 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } attempt: 1
W20260812 06:36:24.371165 15273 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:37667: Network error: Client connection negotiation failed: client connection to 127.13.22.125:37667: connect: Connection refused (error 111)
I20260812 06:36:24.439337 15273 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } attempt: 2
W20260812 06:36:24.440796 15273 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:37667: Network error: Client connection negotiation failed: client connection to 127.13.22.125:37667: connect: Connection refused (error 111)
W20260812 06:36:24.443712 15294 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:24.443898 15294 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:24.443986 15294 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:24.447155 15294 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:36:24.447211 15294 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:24.447239 15294 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:36:24.447285 15294 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:36:24.451393 15294 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:46633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:32771,127.13.22.125:37667,127.13.22.124:46479
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.125:37667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.125
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:24.452327 15294 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:24.453078 15294 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:24.458266 15300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:24.458285 15302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:24.458262 15307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:24.458539 15294 server_base.cc:1061] running on GCE node
I20260812 06:36:24.459026 15294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:24.459919 15294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:24.461077 15294 hybrid_clock.cc:648] HybridClock initialized: now 1786516584461058 us; error 34 us; skew 500 ppm
Aug 12 06:36:24 dist-test-slave-gjrz krb5kdc[15125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516584, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:24.463698 15294 init.cc:377] Logged in from keytab as kudu/127.13.22.125@KRBTEST.COM (short username kudu)
I20260812 06:36:24.474711 15294 webserver.cc:533] Webserver started at https://127.13.22.125:42255/ using document root <none> and password file <none>
I20260812 06:36:24.475405 15294 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:24.475463 15294 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:24.475647 15294 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:24.477252 15294 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-1/data/instance:
uuid: "717affc521284f6191d50cb43ea76673"
format_stamp: "Formatted at 2026-08-12 06:36:24 on dist-test-slave-gjrz"
I20260812 06:36:24.477676 15294 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-1/wal/instance:
uuid: "717affc521284f6191d50cb43ea76673"
format_stamp: "Formatted at 2026-08-12 06:36:24 on dist-test-slave-gjrz"
I20260812 06:36:24.480937 15294 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20260812 06:36:24.483013 15325 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:24.484005 15294 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:24.484113 15294 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-1/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-1/wal
uuid: "717affc521284f6191d50cb43ea76673"
format_stamp: "Formatted at 2026-08-12 06:36:24 on dist-test-slave-gjrz"
I20260812 06:36:24.484206 15294 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:24.494428 15294 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:24.496616 15294 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:24.496773 15294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:24.503832 15407 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.125:37667 every 8 connection(s)
I20260812 06:36:24.503831 15294 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.125:37667
I20260812 06:36:24.504813 15294 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
I20260812 06:36:24.507453 15408 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:24.507642 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 15294
I20260812 06:36:24.507742 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-1/wal/instance
I20260812 06:36:24.510661 15408 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.22.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.124@KRBTEST.COM" created.
Aug 12 06:36:24 dist-test-slave-gjrz krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516584, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:36:24.523375 15408 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:36:24 dist-test-slave-gjrz krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516584, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:24.531984 15273 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } attempt: 3
I20260812 06:36:24.534744 15408 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } has no permanent_uuid. Determining permanent_uuid...
W20260812 06:36:24.535450 15333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.22.124:46479: connect: Connection refused (error 111)
W20260812 06:36:24.536228 15408 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:46479: Network error: Client connection negotiation failed: client connection to 127.13.22.124:46479: connect: Connection refused (error 111)
Aug 12 06:36:24 dist-test-slave-gjrz krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516584, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
WARNING: no policy specified for HTTP/127.13.22.124@KRBTEST.COM; defaulting to no policy
I20260812 06:36:24.540426 15273 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.124@KRBTEST.COM" created.
W20260812 06:36:24.541651 15273 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:46479: Network error: Client connection negotiation failed: client connection to 127.13.22.124:46479: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:24.552276 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.124:46479
--webserver_interface=127.13.22.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:46633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:32771,127.13.22.125:37667,127.13.22.124:46479
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 06:36:24.572808 15273 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } attempt: 1
W20260812 06:36:24.574230 15273 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:46479: Network error: Client connection negotiation failed: client connection to 127.13.22.124:46479: connect: Connection refused (error 111)
I20260812 06:36:24.585460 15408 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } attempt: 1
W20260812 06:36:24.586706 15408 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:46479: Network error: Client connection negotiation failed: client connection to 127.13.22.124:46479: connect: Connection refused (error 111)
I20260812 06:36:24.649395 15273 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } attempt: 2
W20260812 06:36:24.650862 15273 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:46479: Network error: Client connection negotiation failed: client connection to 127.13.22.124:46479: connect: Connection refused (error 111)
W20260812 06:36:24.652731 15426 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:24.652907 15426 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:24.652964 15426 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260812 06:36:24.654852 15408 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } attempt: 2
W20260812 06:36:24.656103 15408 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:46479: Network error: Client connection negotiation failed: client connection to 127.13.22.124:46479: connect: Connection refused (error 111)
W20260812 06:36:24.656394 15426 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:36:24.656451 15426 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:24.656484 15426 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:36:24.656517 15426 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:36:24.660946 15426 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:46633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:32771,127.13.22.125:37667,127.13.22.124:46479
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.124:46479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.124
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:24.661813 15426 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:24.662559 15426 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:24.667497 15436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:24.667486 15439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:24.667477 15434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:24.667722 15426 server_base.cc:1061] running on GCE node
I20260812 06:36:24.668159 15426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:24.668951 15426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:24.670120 15426 hybrid_clock.cc:648] HybridClock initialized: now 1786516584670100 us; error 31 us; skew 500 ppm
Aug 12 06:36:24 dist-test-slave-gjrz krb5kdc[15125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516584, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:24.672667 15426 init.cc:377] Logged in from keytab as kudu/127.13.22.124@KRBTEST.COM (short username kudu)
I20260812 06:36:24.683238 15426 webserver.cc:533] Webserver started at https://127.13.22.124:42355/ using document root <none> and password file <none>
I20260812 06:36:24.683878 15426 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:24.683938 15426 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:24.684154 15426 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:24.685745 15426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-2/data/instance:
uuid: "094f17a4ce6743159195c075d51dc837"
format_stamp: "Formatted at 2026-08-12 06:36:24 on dist-test-slave-gjrz"
I20260812 06:36:24.686196 15426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-2/wal/instance:
uuid: "094f17a4ce6743159195c075d51dc837"
format_stamp: "Formatted at 2026-08-12 06:36:24 on dist-test-slave-gjrz"
I20260812 06:36:24.689517 15426 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20260812 06:36:24.691507 15457 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:24.692487 15426 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:24.692603 15426 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-2/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-2/wal
uuid: "094f17a4ce6743159195c075d51dc837"
format_stamp: "Formatted at 2026-08-12 06:36:24 on dist-test-slave-gjrz"
I20260812 06:36:24.692703 15426 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:24.710672 15426 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:24.712929 15426 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:24.713106 15426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:24.720359 15426 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.124:46479
I20260812 06:36:24.720362 15560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.124:46479 every 8 connection(s)
I20260812 06:36:24.721318 15426 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
I20260812 06:36:24.724273 15561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:24.726883 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 15426
I20260812 06:36:24.727133 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/master-2/wal/instance
I20260812 06:36:24.727957 15561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:36:24 dist-test-slave-gjrz krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
Aug 12 06:36:24 dist-test-slave-gjrz krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516584, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:36:24.747303 15561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } has no permanent_uuid. Determining permanent_uuid...
I20260812 06:36:24.747628 15408 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } attempt: 3
Aug 12 06:36:24 dist-test-slave-gjrz krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
I20260812 06:36:24.750113 15273 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } attempt: 3
Aug 12 06:36:24 dist-test-slave-gjrz krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516584, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
Aug 12 06:36:24 dist-test-slave-gjrz krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516584, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
Aug 12 06:36:24 dist-test-slave-gjrz krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516584, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:36:24.760522 15561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:36:24 dist-test-slave-gjrz krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:36:24.766692 15408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673: Bootstrap starting.
I20260812 06:36:24.768889 15273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6: Bootstrap starting.
I20260812 06:36:24.769402 15408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:24.770267 15408 log.cc:826] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:24.771412 15273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:24.772200 15408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673: No bootstrap required, opened a new log
I20260812 06:36:24.772236 15273 log.cc:826] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:24.773978 15273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6: No bootstrap required, opened a new log
I20260812 06:36:24.775588 15408 raft_consensus.cc:359] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4898a5f400c48ebb5675aa20e77ffb6" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } } peers { permanent_uuid: "717affc521284f6191d50cb43ea76673" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } } peers { permanent_uuid: "094f17a4ce6743159195c075d51dc837" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } }
I20260812 06:36:24.775787 15408 raft_consensus.cc:385] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:24.775859 15408 raft_consensus.cc:740] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717affc521284f6191d50cb43ea76673, State: Initialized, Role: FOLLOWER
I20260812 06:36:24.776810 15273 raft_consensus.cc:359] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4898a5f400c48ebb5675aa20e77ffb6" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } } peers { permanent_uuid: "717affc521284f6191d50cb43ea76673" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } } peers { permanent_uuid: "094f17a4ce6743159195c075d51dc837" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } }
I20260812 06:36:24.776961 15273 raft_consensus.cc:385] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:24.777022 15273 raft_consensus.cc:740] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4898a5f400c48ebb5675aa20e77ffb6, State: Initialized, Role: FOLLOWER
I20260812 06:36:24.777431 15408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4898a5f400c48ebb5675aa20e77ffb6" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } } peers { permanent_uuid: "717affc521284f6191d50cb43ea76673" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } } peers { permanent_uuid: "094f17a4ce6743159195c075d51dc837" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } }
I20260812 06:36:24.777498 15273 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4898a5f400c48ebb5675aa20e77ffb6" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } } peers { permanent_uuid: "717affc521284f6191d50cb43ea76673" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } } peers { permanent_uuid: "094f17a4ce6743159195c075d51dc837" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } }
I20260812 06:36:24.777878 15584 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4898a5f400c48ebb5675aa20e77ffb6" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } } peers { permanent_uuid: "717affc521284f6191d50cb43ea76673" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } } peers { permanent_uuid: "094f17a4ce6743159195c075d51dc837" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } } }
I20260812 06:36:24.778039 15584 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:24.778568 15273 sys_catalog.cc:565] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:36:24.778661 15408 sys_catalog.cc:565] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673 [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 06:36:24 dist-test-slave-gjrz krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516584, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:36:24.783242 15153 master_runner.cc:187] Error getting master registration for 127.13.22.125:37667: OK, instance_id { permanent_uuid: "717affc521284f6191d50cb43ea76673" instance_seqno: 1786516584501917 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 06:36:24.785423 15153 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 06:36:24.785459 15609 catalog_manager.cc:1594] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:36:24.785599 15609 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:36:24.786913 15561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837: Bootstrap starting.
I20260812 06:36:24.788889 15585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4898a5f400c48ebb5675aa20e77ffb6" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } } peers { permanent_uuid: "717affc521284f6191d50cb43ea76673" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } } peers { permanent_uuid: "094f17a4ce6743159195c075d51dc837" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } } }
I20260812 06:36:24.789327 15585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:24.789417 15561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:24.790249 15561 log.cc:826] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837: Log is configured to *not* fsync() on all Append() calls
W20260812 06:36:24.792204 15187 tablet.cc:2406] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 06:36:24.792541 15624 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:36:24.792601 15624 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:36:24.792649 15561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837: No bootstrap required, opened a new log
I20260812 06:36:24.796162 15561 raft_consensus.cc:359] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4898a5f400c48ebb5675aa20e77ffb6" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } } peers { permanent_uuid: "717affc521284f6191d50cb43ea76673" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } } peers { permanent_uuid: "094f17a4ce6743159195c075d51dc837" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } }
I20260812 06:36:24.796353 15561 raft_consensus.cc:385] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:24.796409 15561 raft_consensus.cc:740] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 094f17a4ce6743159195c075d51dc837, State: Initialized, Role: FOLLOWER
I20260812 06:36:24.796881 15561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4898a5f400c48ebb5675aa20e77ffb6" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } } peers { permanent_uuid: "717affc521284f6191d50cb43ea76673" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } } peers { permanent_uuid: "094f17a4ce6743159195c075d51dc837" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } }
I20260812 06:36:24.797353 15632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4898a5f400c48ebb5675aa20e77ffb6" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } } peers { permanent_uuid: "717affc521284f6191d50cb43ea76673" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } } peers { permanent_uuid: "094f17a4ce6743159195c075d51dc837" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } } }
I20260812 06:36:24.794741 15294 master_runner.cc:187] Error getting master registration for 127.13.22.124:46479: OK, instance_id { permanent_uuid: "094f17a4ce6743159195c075d51dc837" instance_seqno: 1786516584718388 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 06:36:24.797523 15294 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 06:36:24.797524 15632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:24.798017 15561 sys_catalog.cc:565] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 06:36:24.805507 15645 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:36:24.805577 15645 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:36:24.808283 15426 master_runner.cc:264] No leader master found from master 094f17a4ce6743159195c075d51dc837
I20260812 06:36:24.808349 15426 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 06:36:24.814870 15632 raft_consensus.cc:493] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:36:24.815054 15632 raft_consensus.cc:515] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4898a5f400c48ebb5675aa20e77ffb6" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } } peers { permanent_uuid: "717affc521284f6191d50cb43ea76673" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } } peers { permanent_uuid: "094f17a4ce6743159195c075d51dc837" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } }
I20260812 06:36:24.815992 15632 leader_election.cc:290] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4898a5f400c48ebb5675aa20e77ffb6 (127.13.22.126:32771), 717affc521284f6191d50cb43ea76673 (127.13.22.125:37667)
I20260812 06:36:24.816586 15229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "094f17a4ce6743159195c075d51dc837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4898a5f400c48ebb5675aa20e77ffb6" is_pre_election: true
I20260812 06:36:24.816869 15229 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 094f17a4ce6743159195c075d51dc837 in term 0.
I20260812 06:36:24.817437 15465 leader_election.cc:304] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 094f17a4ce6743159195c075d51dc837, b4898a5f400c48ebb5675aa20e77ffb6; no voters: 
I20260812 06:36:24.817817 15632 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 06:36:24.817889 15632 raft_consensus.cc:493] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 06:36:24.817934 15632 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:24.818127 15368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "094f17a4ce6743159195c075d51dc837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717affc521284f6191d50cb43ea76673" is_pre_election: true
I20260812 06:36:24.818359 15368 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 094f17a4ce6743159195c075d51dc837 in term 0.
I20260812 06:36:24.819072 15632 raft_consensus.cc:515] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4898a5f400c48ebb5675aa20e77ffb6" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } } peers { permanent_uuid: "717affc521284f6191d50cb43ea76673" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } } peers { permanent_uuid: "094f17a4ce6743159195c075d51dc837" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } }
I20260812 06:36:24.819474 15632 leader_election.cc:290] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [CANDIDATE]: Term 1 election: Requested vote from peers b4898a5f400c48ebb5675aa20e77ffb6 (127.13.22.126:32771), 717affc521284f6191d50cb43ea76673 (127.13.22.125:37667)
I20260812 06:36:24.819828 15368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "094f17a4ce6743159195c075d51dc837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717affc521284f6191d50cb43ea76673"
I20260812 06:36:24.819933 15368 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:24.821043 15368 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 094f17a4ce6743159195c075d51dc837 in term 1.
I20260812 06:36:24.821408 15459 leader_election.cc:304] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 094f17a4ce6743159195c075d51dc837, 717affc521284f6191d50cb43ea76673; no voters: 
I20260812 06:36:24.821470 15229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "094f17a4ce6743159195c075d51dc837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4898a5f400c48ebb5675aa20e77ffb6"
I20260812 06:36:24.821573 15229 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:24.821702 15632 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 06:36:24.821957 15632 raft_consensus.cc:697] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [term 1 LEADER]: Becoming Leader. State: Replica: 094f17a4ce6743159195c075d51dc837, State: Running, Role: LEADER
I20260812 06:36:24.822295 15632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4898a5f400c48ebb5675aa20e77ffb6" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } } peers { permanent_uuid: "717affc521284f6191d50cb43ea76673" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } } peers { permanent_uuid: "094f17a4ce6743159195c075d51dc837" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } }
I20260812 06:36:24.822636 15229 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 094f17a4ce6743159195c075d51dc837 in term 1.
I20260812 06:36:24.824199 15632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 094f17a4ce6743159195c075d51dc837. Latest consensus state: current_term: 1 leader_uuid: "094f17a4ce6743159195c075d51dc837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4898a5f400c48ebb5675aa20e77ffb6" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } } peers { permanent_uuid: "717affc521284f6191d50cb43ea76673" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } } peers { permanent_uuid: "094f17a4ce6743159195c075d51dc837" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } } }
I20260812 06:36:24.824347 15632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [sys.catalog]: This master's current role is: LEADER
I20260812 06:36:24.833465 15650 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 06:36:24.834534 15650 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 06:36:24.839661 15368 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673 [term 1 FOLLOWER]: Refusing update from remote peer 094f17a4ce6743159195c075d51dc837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:36:24.841638 15647 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717affc521284f6191d50cb43ea76673" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:36:24.845217 15585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 094f17a4ce6743159195c075d51dc837. Latest consensus state: current_term: 1 leader_uuid: "094f17a4ce6743159195c075d51dc837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4898a5f400c48ebb5675aa20e77ffb6" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } } peers { permanent_uuid: "717affc521284f6191d50cb43ea76673" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } } peers { permanent_uuid: "094f17a4ce6743159195c075d51dc837" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } } }
I20260812 06:36:24.845312 15585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:24.846256 15229 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6 [term 1 FOLLOWER]: Refusing update from remote peer 094f17a4ce6743159195c075d51dc837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:36:24.847173 15647 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4898a5f400c48ebb5675aa20e77ffb6" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:36:24.850726 15584 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 094f17a4ce6743159195c075d51dc837. Latest consensus state: current_term: 1 leader_uuid: "094f17a4ce6743159195c075d51dc837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4898a5f400c48ebb5675aa20e77ffb6" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } } peers { permanent_uuid: "717affc521284f6191d50cb43ea76673" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } } peers { permanent_uuid: "094f17a4ce6743159195c075d51dc837" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } } }
I20260812 06:36:24.850837 15584 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:24.856978 15647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "094f17a4ce6743159195c075d51dc837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4898a5f400c48ebb5675aa20e77ffb6" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } } peers { permanent_uuid: "717affc521284f6191d50cb43ea76673" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } } peers { permanent_uuid: "094f17a4ce6743159195c075d51dc837" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } } }
I20260812 06:36:24.857086 15647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [sys.catalog]: This master's current role is: LEADER
I20260812 06:36:24.857197 15651 mvcc.cc:204] Tried to move back new op lower bound from 7317571931496247296 to 7317571931435913216. Current Snapshot: MvccSnapshot[applied={T|T < 7317571931496247296}]
I20260812 06:36:24.858335 15584 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "094f17a4ce6743159195c075d51dc837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4898a5f400c48ebb5675aa20e77ffb6" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } } peers { permanent_uuid: "717affc521284f6191d50cb43ea76673" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } } peers { permanent_uuid: "094f17a4ce6743159195c075d51dc837" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } } }
I20260812 06:36:24.858435 15584 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:24.858801 15655 mvcc.cc:204] Tried to move back new op lower bound from 7317571931496247296 to 7317571931435913216. Current Snapshot: MvccSnapshot[applied={T|T < 7317571931496247296}]
I20260812 06:36:24.861662 15650 catalog_manager.cc:1383] Generated new cluster ID: e644297561854165a693ce7062546c7f
I20260812 06:36:24.861726 15650 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 06:36:24.876729 15650 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 06:36:24.877770 15650 catalog_manager.cc:1540] Loading token signing keys...
I20260812 06:36:24.880765 15647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "094f17a4ce6743159195c075d51dc837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4898a5f400c48ebb5675aa20e77ffb6" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } } peers { permanent_uuid: "717affc521284f6191d50cb43ea76673" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } } peers { permanent_uuid: "094f17a4ce6743159195c075d51dc837" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } } }
I20260812 06:36:24.880875 15647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837 [sys.catalog]: This master's current role is: LEADER
I20260812 06:36:24.882360 15585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "094f17a4ce6743159195c075d51dc837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4898a5f400c48ebb5675aa20e77ffb6" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 32771 } } peers { permanent_uuid: "717affc521284f6191d50cb43ea76673" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 37667 } } peers { permanent_uuid: "094f17a4ce6743159195c075d51dc837" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 46479 } } }
I20260812 06:36:24.882480 15585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:24.882360 15652 mvcc.cc:204] Tried to move back new op lower bound from 7317571931496247296 to 7317571931435913216. Current Snapshot: MvccSnapshot[applied={T|T < 7317571931496247296}]
I20260812 06:36:24.896126 15650 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 094f17a4ce6743159195c075d51dc837: Generated new TSK 0
I20260812 06:36:24.898790 15650 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:25.065644 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.22.65:0
--local_ip_for_outbound_sockets=127.13.22.65
--webserver_interface=127.13.22.65
--webserver_port=0
--tserver_master_addrs=127.13.22.126:32771,127.13.22.125:37667,127.13.22.124:46479
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:46633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 06:36:25.211117 15678 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:25.211397 15678 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:25.211480 15678 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:25.215812 15678 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:25.216001 15678 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.22.65
I20260812 06:36:25.221434 15678 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:46633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.65
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--tserver_master_addrs=127.13.22.126:32771,127.13.22.125:37667,127.13.22.124:46479
--never_fsync=true
--heap_profile_path=/tmp/kudu.15678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.22.65

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:25.222760 15678 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:25.223802 15678 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:25.231233 15691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:25.231233 15693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:25.231249 15690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:25.231755 15678 server_base.cc:1061] running on GCE node
I20260812 06:36:25.232088 15678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:25.232589 15678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:25.233736 15678 hybrid_clock.cc:648] HybridClock initialized: now 1786516585233719 us; error 35 us; skew 500 ppm
Aug 12 06:36:25 dist-test-slave-gjrz krb5kdc[15125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516585, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:25.236686 15678 init.cc:377] Logged in from keytab as kudu/127.13.22.65@KRBTEST.COM (short username kudu)
I20260812 06:36:25.249526 15678 webserver.cc:533] Webserver started at https://127.13.22.65:33249/ using document root <none> and password file <none>
I20260812 06:36:25.250319 15678 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:25.250391 15678 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:25.250667 15678 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:25.252931 15678 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/ts-0/data/instance:
uuid: "0ecc9a0f7ff74b26a3f0d7884022cb86"
format_stamp: "Formatted at 2026-08-12 06:36:25 on dist-test-slave-gjrz"
I20260812 06:36:25.253535 15678 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/ts-0/wal/instance:
uuid: "0ecc9a0f7ff74b26a3f0d7884022cb86"
format_stamp: "Formatted at 2026-08-12 06:36:25 on dist-test-slave-gjrz"
I20260812 06:36:25.258143 15678 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260812 06:36:25.260864 15718 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:25.262120 15678 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:36:25.262243 15678 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/ts-0/wal
uuid: "0ecc9a0f7ff74b26a3f0d7884022cb86"
format_stamp: "Formatted at 2026-08-12 06:36:25 on dist-test-slave-gjrz"
I20260812 06:36:25.262346 15678 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:25.281273 15678 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:25.284399 15678 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:25.284585 15678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:25.285362 15678 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:25.286648 15678 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:36:25.286710 15678 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:25.286780 15678 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:36:25.286813 15678 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:25.300217 15678 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.65:43429
I20260812 06:36:25.300257 15897 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.65:43429 every 8 connection(s)
I20260812 06:36:25.301256 15678 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
I20260812 06:36:25.305706 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 15678
I20260812 06:36:25.305819 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/minicluster-data/ts-0/wal/instance
Aug 12 06:36:25 dist-test-slave-gjrz krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516585, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:36:25.316223 15898 heartbeater.cc:344] Connected to a master server at 127.13.22.124:46479
I20260812 06:36:25.316457 15898 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:25.317022 15898 heartbeater.cc:507] Master 127.13.22.124:46479 requested a full tablet report, sending...
I20260812 06:36:25.318459 15489 ts_manager.cc:194] Registered new tserver with Master: 0ecc9a0f7ff74b26a3f0d7884022cb86 (127.13.22.65:43429)
Aug 12 06:36:25 dist-test-slave-gjrz krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516585, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
Aug 12 06:36:25 dist-test-slave-gjrz krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516585, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:36:25.319980 15489 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.22.65@KRBTEST.COM'} at 127.13.22.65:44085
I20260812 06:36:25.324757 15900 heartbeater.cc:344] Connected to a master server at 127.13.22.126:32771
I20260812 06:36:25.324848 15900 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:25.325016 15900 heartbeater.cc:507] Master 127.13.22.126:32771 requested a full tablet report, sending...
I20260812 06:36:25.326010 15901 heartbeater.cc:344] Connected to a master server at 127.13.22.125:37667
I20260812 06:36:25.326102 15901 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:25.326133 15210 ts_manager.cc:194] Registered new tserver with Master: 0ecc9a0f7ff74b26a3f0d7884022cb86 (127.13.22.65:43429)
I20260812 06:36:25.326234 15901 heartbeater.cc:507] Master 127.13.22.125:37667 requested a full tablet report, sending...
I20260812 06:36:25.327353 15354 ts_manager.cc:194] Registered new tserver with Master: 0ecc9a0f7ff74b26a3f0d7884022cb86 (127.13.22.65:43429)
I20260812 06:36:25.338480 13401 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
time=2026-08-12T06:36:25.425Z level=INFO source=main.go:1678 msg="updated GOGC" old=100 new=75
time=2026-08-12T06:36:25.427Z level=INFO source=main.go:744 msg="Leaving GOMAXPROCS=4: CPU quota undefined" component=automaxprocs
time=2026-08-12T06:36:25.428Z level=INFO source=memlimit.go:198 msg="GOMEMLIMIT is updated" component=automemlimit package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=14142549196 previous=9223372036854775807
time=2026-08-12T06:36:25.428Z level=INFO source=main.go:851 msg="Starting Prometheus Server" mode=server version="(version=3.11.2, branch=HEAD, revision=f0f0fdd679dcd6df320b0558b20919f7cd44c407)"
time=2026-08-12T06:36:25.428Z level=INFO source=main.go:856 msg="operational information" build_context="(go=go1.26.2, platform=linux/amd64, user=root@d2684485c347, date=20260413-11:53:40, tags=netgo,builtinassets)" host_details="(Linux 5.4.0-1157-gcp #166~18.04.1-Ubuntu SMP Wed Jan 21 01:57:46 UTC 2026 x86_64 dist-test-slave-gjrz (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)"
time=2026-08-12T06:36:25.678Z level=INFO source=web.go:710 msg="Start listening for connections" component=web address=127.0.0.1:0
time=2026-08-12T06:36:25.679Z level=INFO source=main.go:1410 msg="Starting TSDB ..."
time=2026-08-12T06:36:25.683Z level=INFO source=tls_config.go:354 msg="Listening on" component=web address=127.0.0.1:37075
time=2026-08-12T06:36:25.683Z level=INFO source=tls_config.go:357 msg="TLS is disabled." component=web http2=false address=127.0.0.1:37075
time=2026-08-12T06:36:25.684Z level=INFO source=head.go:698 msg="Replaying on-disk memory mappable chunks if any" component=tsdb
time=2026-08-12T06:36:25.684Z level=INFO source=head.go:784 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=1.969µs
time=2026-08-12T06:36:25.684Z level=INFO source=head.go:792 msg="Replaying WAL, this may take a while" component=tsdb
time=2026-08-12T06:36:25.685Z level=INFO source=head.go:865 msg="WAL segment loaded" component=tsdb segment=0 maxSegment=0 duration=726.573µs
time=2026-08-12T06:36:25.685Z level=INFO source=head.go:902 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=53.112µs wal_replay_duration=780.083µs wbl_replay_duration=144ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=1.969µs total_replay_duration=870.318µs
time=2026-08-12T06:36:25.691Z level=INFO source=main.go:1431 msg="filesystem information" fs_type=OVERLAYFS_SUPER_MAGIC
time=2026-08-12T06:36:25.691Z level=INFO source=main.go:1434 msg="TSDB started"
time=2026-08-12T06:36:25.691Z level=INFO source=main.go:1632 msg="Loading configuration file" filename=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/prometheus.yml
time=2026-08-12T06:36:25.691Z level=INFO source=main.go:1048 msg="TSDB retention updated" duration=15d size=0B percentage=0
time=2026-08-12T06:36:25.693Z level=INFO source=main.go:1671 msg="Completed loading of configuration file" db_storage=77.208µs remote_storage=2.035µs web_handler=487ns query_engine=883ns scrape=1.405189ms scrape_sd=55.543µs notify=2.233µs notify_sd=1.182µs rules=2.705µs tracing=13.527µs filename=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenScrapeSucceeds.1786516559891118-13401-0/prometheus.yml totalDuration=2.358276ms
time=2026-08-12T06:36:25.693Z level=INFO source=main.go:1395 msg="Server is ready to receive web requests."
time=2026-08-12T06:36:25.693Z level=INFO source=manager.go:209 msg="Starting rule manager..." component="rule manager"
I20260812 06:36:25.786916 15609 catalog_manager.cc:1295] Loaded cluster ID: e644297561854165a693ce7062546c7f
I20260812 06:36:25.787003 15609 catalog_manager.cc:1588] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6: loading cluster ID for follower catalog manager: success
I20260812 06:36:25.788874 15609 catalog_manager.cc:1610] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6: acquiring CA information for follower catalog manager: success
I20260812 06:36:25.789924 15609 catalog_manager.cc:1638] T 00000000000000000000000000000000 P b4898a5f400c48ebb5675aa20e77ffb6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 06:36:25.793612 15624 catalog_manager.cc:1295] Loaded cluster ID: e644297561854165a693ce7062546c7f
I20260812 06:36:25.793670 15624 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673: loading cluster ID for follower catalog manager: success
I20260812 06:36:25.795163 15624 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673: acquiring CA information for follower catalog manager: success
I20260812 06:36:25.796141 15624 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 717affc521284f6191d50cb43ea76673: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 06:36:26.322480 15898 heartbeater.cc:499] Master 127.13.22.124:46479 was elected leader, sending a full tablet report...
time=2026-08-12T06:36:33.436Z level=WARN source=main.go:1191 msg="Received an OS signal, exiting gracefully..." signal=terminated
time=2026-08-12T06:36:33.436Z level=INFO source=main.go:1216 msg="Stopping scrape discovery manager..."
time=2026-08-12T06:36:33.436Z level=INFO source=main.go:1230 msg="Stopping notify discovery manager..."
time=2026-08-12T06:36:33.436Z level=INFO source=main.go:1245 msg="Stopping rule manager manager..."
time=2026-08-12T06:36:33.436Z level=INFO source=manager.go:223 msg="Stopping rule manager..." component="rule manager"
time=2026-08-12T06:36:33.436Z level=INFO source=manager.go:239 msg="Rule manager stopped" component="rule manager"
time=2026-08-12T06:36:33.436Z level=INFO source=main.go:1269 msg="Stopping scrape manager..."
time=2026-08-12T06:36:33.436Z level=INFO source=main.go:1226 msg="Notify discovery manager stopped"
time=2026-08-12T06:36:33.436Z level=INFO source=main.go:1241 msg="Rule manager stopped"
time=2026-08-12T06:36:33.436Z level=INFO source=main.go:1212 msg="Scrape discovery manager stopped"
time=2026-08-12T06:36:33.465Z level=INFO source=main.go:1261 msg="Scrape manager stopped"
time=2026-08-12T06:36:33.465Z level=INFO source=main.go:1284 msg="Stopping tracing manager..."
time=2026-08-12T06:36:33.465Z level=INFO source=main.go:1369 msg="Stopping reloaders..."
time=2026-08-12T06:36:33.465Z level=INFO source=main.go:1458 msg="Stopping storage..."
time=2026-08-12T06:36:33.465Z level=INFO source=main.go:1361 msg="Reloaders stopped"
time=2026-08-12T06:36:33.465Z level=INFO source=main.go:1280 msg="Tracing manager stopped"
time=2026-08-12T06:36:33.469Z level=INFO source=main.go:1536 msg="Stopping web handler..."
time=2026-08-12T06:36:33.469Z level=INFO source=main.go:1454 msg="TSDB stopped"
time=2026-08-12T06:36:33.469Z level=INFO source=main.go:1559 msg="Stopping notifier manager..."
time=2026-08-12T06:36:33.469Z level=INFO source=manager.go:321 msg="Stopping notification manager..." component=notifier
time=2026-08-12T06:36:33.469Z level=INFO source=main.go:1532 msg="Web handler stopped"
time=2026-08-12T06:36:33.469Z level=INFO source=main.go:1555 msg="Notifier manager stopped"
time=2026-08-12T06:36:33.469Z level=INFO source=main.go:1570 msg="See you next time!"
I20260812 06:36:33.492756 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 15678
I20260812 06:36:33.509380 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 15153
I20260812 06:36:33.526132 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 15294
I20260812 06:36:33.537108 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 15426
2026-08-12T06:36:33Z chronyd exiting
[       OK ] PrometheusTokenITest.TestCorrectTokenScrapeSucceeds (11615 ms)
[ RUN      ] PrometheusTokenITest.TestWrongTokenScrapeFails
Loading random data
Initializing database '/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 06:36:33 dist-test-slave-gjrz krb5kdc[15950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 06:36:33 dist-test-slave-gjrz krb5kdc[15950](info): set up 2 sockets
Aug 12 06:36:33 dist-test-slave-gjrz krb5kdc[15950](info): commencing operation
krb5kdc: starting...
W20260812 06:36:35.624339 13401 mini_kdc.cc:121] Time spent starting KDC: real 2.061s	user 0.001s	sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/test-user.keytab.
Aug 12 06:36:35 dist-test-slave-gjrz krb5kdc[15950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM: 
2026-08-12T06:36:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T06:36:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:35.750650 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.126:45751
--webserver_interface=127.13.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:45787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:45751,127.13.22.125:43333,127.13.22.124:35393
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 06:36:35.853176 15977 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:35.853367 15977 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:35.853431 15977 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:35.856632 15977 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:36:35.856688 15977 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:35.856706 15977 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:36:35.856721 15977 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:36:35.860858 15977 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:45787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:45751,127.13.22.125:43333,127.13.22.124:35393
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.126:45751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.126
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:35.861778 15977 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:35.862596 15977 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:35.867942 15988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:35.868054 15989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:35.867949 15991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:35.868162 15977 server_base.cc:1061] running on GCE node
I20260812 06:36:35.868624 15977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:35.869410 15977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:35.870566 15977 hybrid_clock.cc:648] HybridClock initialized: now 1786516595870554 us; error 24 us; skew 500 ppm
Aug 12 06:36:35 dist-test-slave-gjrz krb5kdc[15950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516595, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:35.873140 15977 init.cc:377] Logged in from keytab as kudu/127.13.22.126@KRBTEST.COM (short username kudu)
I20260812 06:36:35.884276 15977 webserver.cc:533] Webserver started at https://127.13.22.126:41425/ using document root <none> and password file <none>
I20260812 06:36:35.884886 15977 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:35.884938 15977 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:35.885136 15977 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:35.886736 15977 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/data/instance:
uuid: "2d2eab862c384cea807403727e020709"
format_stamp: "Formatted at 2026-08-12 06:36:35 on dist-test-slave-gjrz"
I20260812 06:36:35.887182 15977 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/wal/instance:
uuid: "2d2eab862c384cea807403727e020709"
format_stamp: "Formatted at 2026-08-12 06:36:35 on dist-test-slave-gjrz"
I20260812 06:36:35.890772 15977 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.000s
I20260812 06:36:35.892859 16010 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:35.893842 15977 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.001s
I20260812 06:36:35.893961 15977 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/wal
uuid: "2d2eab862c384cea807403727e020709"
format_stamp: "Formatted at 2026-08-12 06:36:35 on dist-test-slave-gjrz"
I20260812 06:36:35.894043 15977 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:35.928646 15977 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:35.931421 15977 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:35.931598 15977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:35.938810 16092 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.126:45751 every 8 connection(s)
I20260812 06:36:35.938818 15977 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.126:45751
I20260812 06:36:35.939874 15977 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
I20260812 06:36:35.942655 16093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:35.945406 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 15977
I20260812 06:36:35.945518 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/wal/instance
I20260812 06:36:35.946046 16093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.22.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.125@KRBTEST.COM" created.
Aug 12 06:36:35 dist-test-slave-gjrz krb5kdc[15950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516595, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:36:35.962097 16093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } has no permanent_uuid. Determining permanent_uuid...
W20260812 06:36:35.962837 16015 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.22.125:43333: connect: Connection refused (error 111)
W20260812 06:36:35.963675 16093 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:43333: Network error: Client connection negotiation failed: client connection to 127.13.22.125:43333: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:35.990036 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.125:43333
--webserver_interface=127.13.22.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:45787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:45751,127.13.22.125:43333,127.13.22.124:35393
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 06:36:36.012915 16093 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } attempt: 1
W20260812 06:36:36.014533 16093 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:43333: Network error: Client connection negotiation failed: client connection to 127.13.22.125:43333: connect: Connection refused (error 111)
I20260812 06:36:36.082698 16093 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } attempt: 2
W20260812 06:36:36.084187 16093 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:43333: Network error: Client connection negotiation failed: client connection to 127.13.22.125:43333: connect: Connection refused (error 111)
W20260812 06:36:36.092701 16111 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:36.092900 16111 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:36.092988 16111 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:36.096195 16111 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:36:36.096253 16111 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:36.096282 16111 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:36:36.096313 16111 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:36:36.100528 16111 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:45787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:45751,127.13.22.125:43333,127.13.22.124:35393
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.125:43333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.125
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:36.101464 16111 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:36.102273 16111 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:36.107286 16122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:36.107293 16123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:36.107299 16126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:36.107517 16111 server_base.cc:1061] running on GCE node
I20260812 06:36:36.108078 16111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:36.108839 16111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:36.110014 16111 hybrid_clock.cc:648] HybridClock initialized: now 1786516596109977 us; error 35 us; skew 500 ppm
Aug 12 06:36:36 dist-test-slave-gjrz krb5kdc[15950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516596, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:36.112607 16111 init.cc:377] Logged in from keytab as kudu/127.13.22.125@KRBTEST.COM (short username kudu)
I20260812 06:36:36.123675 16111 webserver.cc:533] Webserver started at https://127.13.22.125:41815/ using document root <none> and password file <none>
I20260812 06:36:36.124329 16111 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:36.124382 16111 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:36.124583 16111 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:36.126191 16111 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/data/instance:
uuid: "72f186d66f4549b7859a28d6f0ebb87b"
format_stamp: "Formatted at 2026-08-12 06:36:36 on dist-test-slave-gjrz"
I20260812 06:36:36.126627 16111 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/wal/instance:
uuid: "72f186d66f4549b7859a28d6f0ebb87b"
format_stamp: "Formatted at 2026-08-12 06:36:36 on dist-test-slave-gjrz"
I20260812 06:36:36.129854 16111 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20260812 06:36:36.131817 16143 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:36.132730 16111 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:36.132838 16111 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/wal
uuid: "72f186d66f4549b7859a28d6f0ebb87b"
format_stamp: "Formatted at 2026-08-12 06:36:36 on dist-test-slave-gjrz"
I20260812 06:36:36.132918 16111 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:36.142697 16111 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:36.144997 16111 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:36.145150 16111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:36.152019 16111 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.125:43333
I20260812 06:36:36.152017 16239 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.125:43333 every 8 connection(s)
I20260812 06:36:36.152928 16111 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
I20260812 06:36:36.154624 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 16111
I20260812 06:36:36.154742 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/wal/instance
I20260812 06:36:36.155642 16240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:36.159755 16240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.22.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.124@KRBTEST.COM" created.
Aug 12 06:36:36 dist-test-slave-gjrz krb5kdc[15950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516596, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:36:36.172514 16240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } has no permanent_uuid. Determining permanent_uuid...
I20260812 06:36:36.175318 16093 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } attempt: 3
Aug 12 06:36:36 dist-test-slave-gjrz krb5kdc[15950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516596, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
Aug 12 06:36:36 dist-test-slave-gjrz krb5kdc[15950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516595, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:36.185134 16093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } has no permanent_uuid. Determining permanent_uuid...
I20260812 06:36:36.185211 16240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } has no permanent_uuid. Determining permanent_uuid...
W20260812 06:36:36.186221 16147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.22.124:35393: connect: Connection refused (error 111)
W20260812 06:36:36.186394 16093 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:35393: Network error: Client connection negotiation failed: client connection to 127.13.22.124:35393: connect: Connection refused (error 111)
W20260812 06:36:36.187059 16240 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:35393: Network error: Client connection negotiation failed: client connection to 127.13.22.124:35393: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.13.22.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:36.201702 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.124:35393
--webserver_interface=127.13.22.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:45787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:45751,127.13.22.125:43333,127.13.22.124:35393
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 06:36:36.217549 16093 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } attempt: 1
W20260812 06:36:36.218940 16093 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:35393: Network error: Client connection negotiation failed: client connection to 127.13.22.124:35393: connect: Connection refused (error 111)
I20260812 06:36:36.236274 16240 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } attempt: 1
W20260812 06:36:36.237679 16240 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:35393: Network error: Client connection negotiation failed: client connection to 127.13.22.124:35393: connect: Connection refused (error 111)
I20260812 06:36:36.294112 16093 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } attempt: 2
W20260812 06:36:36.295614 16093 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:35393: Network error: Client connection negotiation failed: client connection to 127.13.22.124:35393: connect: Connection refused (error 111)
W20260812 06:36:36.303347 16260 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:36.303534 16260 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:36.303609 16260 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260812 06:36:36.305830 16240 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } attempt: 2
W20260812 06:36:36.307075 16240 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:35393: Network error: Client connection negotiation failed: client connection to 127.13.22.124:35393: connect: Connection refused (error 111)
W20260812 06:36:36.307139 16260 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:36:36.307194 16260 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:36.307219 16260 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:36:36.307240 16260 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:36:36.311379 16260 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:45787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:45751,127.13.22.125:43333,127.13.22.124:35393
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.124:35393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.124
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:36.312268 16260 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:36.312984 16260 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:36.317862 16269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:36.317884 16272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:36.317880 16268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:36.318239 16260 server_base.cc:1061] running on GCE node
I20260812 06:36:36.318648 16260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:36.319453 16260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:36.320622 16260 hybrid_clock.cc:648] HybridClock initialized: now 1786516596320606 us; error 30 us; skew 500 ppm
Aug 12 06:36:36 dist-test-slave-gjrz krb5kdc[15950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516596, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:36.323249 16260 init.cc:377] Logged in from keytab as kudu/127.13.22.124@KRBTEST.COM (short username kudu)
I20260812 06:36:36.333743 16260 webserver.cc:533] Webserver started at https://127.13.22.124:39539/ using document root <none> and password file <none>
I20260812 06:36:36.334338 16260 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:36.334390 16260 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:36.334594 16260 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:36.336287 16260 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/data/instance:
uuid: "3f7947b8a8f14a84a66c96cd7613b8e7"
format_stamp: "Formatted at 2026-08-12 06:36:36 on dist-test-slave-gjrz"
I20260812 06:36:36.336825 16260 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/wal/instance:
uuid: "3f7947b8a8f14a84a66c96cd7613b8e7"
format_stamp: "Formatted at 2026-08-12 06:36:36 on dist-test-slave-gjrz"
I20260812 06:36:36.340167 16260 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 06:36:36.342100 16288 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:36.343006 16260 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 06:36:36.343119 16260 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/wal
uuid: "3f7947b8a8f14a84a66c96cd7613b8e7"
format_stamp: "Formatted at 2026-08-12 06:36:36 on dist-test-slave-gjrz"
I20260812 06:36:36.343199 16260 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:36.354842 16260 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:36.357084 16260 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:36.357246 16260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:36.364156 16380 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.124:35393 every 8 connection(s)
I20260812 06:36:36.364166 16260 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.124:35393
I20260812 06:36:36.365116 16260 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
I20260812 06:36:36.366374 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 16260
I20260812 06:36:36.366480 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/wal/instance
I20260812 06:36:36.367933 16381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:36.371495 16381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:36:36 dist-test-slave-gjrz krb5kdc[15950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
Aug 12 06:36:36 dist-test-slave-gjrz krb5kdc[15950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516596, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
Aug 12 06:36:36 dist-test-slave-gjrz krb5kdc[15950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
I20260812 06:36:36.383734 16381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:36:36 dist-test-slave-gjrz krb5kdc[15950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
Aug 12 06:36:36 dist-test-slave-gjrz krb5kdc[15950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516596, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
I20260812 06:36:36.391841 16381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } has no permanent_uuid. Determining permanent_uuid...
I20260812 06:36:36.394776 16093 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } attempt: 3
Aug 12 06:36:36 dist-test-slave-gjrz krb5kdc[15950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516596, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:36:36.398247 16240 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } attempt: 3
Aug 12 06:36:36 dist-test-slave-gjrz krb5kdc[15950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516595, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:36:36.401353 16381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7: Bootstrap starting.
Aug 12 06:36:36 dist-test-slave-gjrz krb5kdc[15950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516596, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:36:36.404062 16381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:36.404980 16381 log.cc:826] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:36.406685 16381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7: No bootstrap required, opened a new log
I20260812 06:36:36.406719 16093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709: Bootstrap starting.
I20260812 06:36:36.408550 16093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:36.408730 16240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b: Bootstrap starting.
I20260812 06:36:36.409166 16093 log.cc:826] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:36.409749 16381 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2eab862c384cea807403727e020709" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } } peers { permanent_uuid: "72f186d66f4549b7859a28d6f0ebb87b" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } } peers { permanent_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } }
I20260812 06:36:36.409931 16381 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:36.409996 16381 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f7947b8a8f14a84a66c96cd7613b8e7, State: Initialized, Role: FOLLOWER
I20260812 06:36:36.410431 16381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2eab862c384cea807403727e020709" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } } peers { permanent_uuid: "72f186d66f4549b7859a28d6f0ebb87b" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } } peers { permanent_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } }
I20260812 06:36:36.410621 16093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709: No bootstrap required, opened a new log
I20260812 06:36:36.411523 16240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:36.411612 16381 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:36:36.412489 16240 log.cc:826] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:36.413081 16399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2eab862c384cea807403727e020709" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } } peers { permanent_uuid: "72f186d66f4549b7859a28d6f0ebb87b" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } } peers { permanent_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } } }
I20260812 06:36:36.413255 16399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:36.413511 16093 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2eab862c384cea807403727e020709" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } } peers { permanent_uuid: "72f186d66f4549b7859a28d6f0ebb87b" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } } peers { permanent_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } }
I20260812 06:36:36.413681 16093 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:36.413736 16093 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d2eab862c384cea807403727e020709, State: Initialized, Role: FOLLOWER
I20260812 06:36:36.414143 16240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b: No bootstrap required, opened a new log
I20260812 06:36:36.414160 16093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2eab862c384cea807403727e020709" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } } peers { permanent_uuid: "72f186d66f4549b7859a28d6f0ebb87b" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } } peers { permanent_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } }
I20260812 06:36:36.414636 16424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2eab862c384cea807403727e020709" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } } peers { permanent_uuid: "72f186d66f4549b7859a28d6f0ebb87b" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } } peers { permanent_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } } }
I20260812 06:36:36.414758 16424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:36.415220 16093 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:36:36.415162 16260 master_runner.cc:187] Error getting master registration for 127.13.22.126:45751: OK, instance_id { permanent_uuid: "2d2eab862c384cea807403727e020709" instance_seqno: 1786516595936843 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 06:36:36.417169 16240 raft_consensus.cc:359] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2eab862c384cea807403727e020709" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } } peers { permanent_uuid: "72f186d66f4549b7859a28d6f0ebb87b" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } } peers { permanent_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } }
I20260812 06:36:36.417326 16260 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 06:36:36.417339 16240 raft_consensus.cc:385] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:36.417390 16240 raft_consensus.cc:740] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f186d66f4549b7859a28d6f0ebb87b, State: Initialized, Role: FOLLOWER
W20260812 06:36:36.417775 16421 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260812 06:36:36.417783 16240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2eab862c384cea807403727e020709" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } } peers { permanent_uuid: "72f186d66f4549b7859a28d6f0ebb87b" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } } peers { permanent_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } }
W20260812 06:36:36.417881 16421 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:36:36.418782 16240 sys_catalog.cc:565] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b [sys.catalog]: configured and running, proceeding with master startup.
W20260812 06:36:36.421545 16441 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:36:36.421600 16441 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:36:36.419714 15977 master_runner.cc:187] Error getting master registration for 127.13.22.125:43333: OK, instance_id { permanent_uuid: "72f186d66f4549b7859a28d6f0ebb87b" instance_seqno: 1786516596150095 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 06:36:36.421808 15977 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 06:36:36.428426 16111 master_runner.cc:264] No leader master found from master 72f186d66f4549b7859a28d6f0ebb87b
I20260812 06:36:36.428501 16111 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 06:36:36.429579 16466 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:36:36.429649 16466 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:36:36.430114 16444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2eab862c384cea807403727e020709" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } } peers { permanent_uuid: "72f186d66f4549b7859a28d6f0ebb87b" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } } peers { permanent_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } } }
I20260812 06:36:36.430220 16444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b [sys.catalog]: This master's current role is: FOLLOWER
W20260812 06:36:36.432955 16020 tablet.cc:2406] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 06:36:36.567724 16424 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:36:36.567935 16424 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2eab862c384cea807403727e020709" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } } peers { permanent_uuid: "72f186d66f4549b7859a28d6f0ebb87b" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } } peers { permanent_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } }
I20260812 06:36:36.569054 16424 leader_election.cc:290] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72f186d66f4549b7859a28d6f0ebb87b (127.13.22.125:43333), 3f7947b8a8f14a84a66c96cd7613b8e7 (127.13.22.124:35393)
I20260812 06:36:36.569828 16191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d2eab862c384cea807403727e020709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f186d66f4549b7859a28d6f0ebb87b" is_pre_election: true
I20260812 06:36:36.570132 16191 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d2eab862c384cea807403727e020709 in term 0.
I20260812 06:36:36.570638 16015 leader_election.cc:304] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d2eab862c384cea807403727e020709, 72f186d66f4549b7859a28d6f0ebb87b; no voters: 
I20260812 06:36:36.570963 16424 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 06:36:36.571039 16424 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 06:36:36.571075 16424 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:36.572283 16424 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2eab862c384cea807403727e020709" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } } peers { permanent_uuid: "72f186d66f4549b7859a28d6f0ebb87b" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } } peers { permanent_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } }
I20260812 06:36:36.572654 16424 leader_election.cc:290] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [CANDIDATE]: Term 1 election: Requested vote from peers 72f186d66f4549b7859a28d6f0ebb87b (127.13.22.125:43333), 3f7947b8a8f14a84a66c96cd7613b8e7 (127.13.22.124:35393)
I20260812 06:36:36.573184 16191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d2eab862c384cea807403727e020709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f186d66f4549b7859a28d6f0ebb87b"
I20260812 06:36:36.573307 16191 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:36.574975 16191 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d2eab862c384cea807403727e020709 in term 1.
I20260812 06:36:36.575832 16330 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d2eab862c384cea807403727e020709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7" is_pre_election: true
I20260812 06:36:36.576100 16330 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d2eab862c384cea807403727e020709 in term 0.
I20260812 06:36:36.576347 16329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d2eab862c384cea807403727e020709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7"
I20260812 06:36:36.576426 16329 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:36.577404 16329 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d2eab862c384cea807403727e020709 in term 1.
I20260812 06:36:36.577906 16015 leader_election.cc:304] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d2eab862c384cea807403727e020709, 72f186d66f4549b7859a28d6f0ebb87b; no voters: 
I20260812 06:36:36.578421 16424 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 06:36:36.578684 16424 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [term 1 LEADER]: Becoming Leader. State: Replica: 2d2eab862c384cea807403727e020709, State: Running, Role: LEADER
I20260812 06:36:36.579046 16424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2eab862c384cea807403727e020709" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } } peers { permanent_uuid: "72f186d66f4549b7859a28d6f0ebb87b" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } } peers { permanent_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } }
I20260812 06:36:36.580727 16424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d2eab862c384cea807403727e020709. Latest consensus state: current_term: 1 leader_uuid: "2d2eab862c384cea807403727e020709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2eab862c384cea807403727e020709" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } } peers { permanent_uuid: "72f186d66f4549b7859a28d6f0ebb87b" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } } peers { permanent_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } } }
I20260812 06:36:36.580921 16424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [sys.catalog]: This master's current role is: LEADER
I20260812 06:36:36.583596 16471 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 06:36:36.584731 16471 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 06:36:36.590420 16329 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7 [term 1 FOLLOWER]: Refusing update from remote peer 2d2eab862c384cea807403727e020709: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:36:36.591323 16191 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b [term 1 FOLLOWER]: Refusing update from remote peer 2d2eab862c384cea807403727e020709: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:36:36.591657 16468 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:36:36.592034 16468 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72f186d66f4549b7859a28d6f0ebb87b" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:36:36.595445 16399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d2eab862c384cea807403727e020709. Latest consensus state: current_term: 1 leader_uuid: "2d2eab862c384cea807403727e020709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2eab862c384cea807403727e020709" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } } peers { permanent_uuid: "72f186d66f4549b7859a28d6f0ebb87b" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } } peers { permanent_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } } }
I20260812 06:36:36.595561 16399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:36.598469 16444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d2eab862c384cea807403727e020709. Latest consensus state: current_term: 1 leader_uuid: "2d2eab862c384cea807403727e020709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2eab862c384cea807403727e020709" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } } peers { permanent_uuid: "72f186d66f4549b7859a28d6f0ebb87b" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } } peers { permanent_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } } }
I20260812 06:36:36.598582 16444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:36.600780 16468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d2eab862c384cea807403727e020709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2eab862c384cea807403727e020709" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } } peers { permanent_uuid: "72f186d66f4549b7859a28d6f0ebb87b" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } } peers { permanent_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } } }
I20260812 06:36:36.600886 16468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [sys.catalog]: This master's current role is: LEADER
I20260812 06:36:36.602520 16444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d2eab862c384cea807403727e020709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2eab862c384cea807403727e020709" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } } peers { permanent_uuid: "72f186d66f4549b7859a28d6f0ebb87b" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } } peers { permanent_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } } }
I20260812 06:36:36.602620 16444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:36.603444 16473 mvcc.cc:204] Tried to move back new op lower bound from 7317571979628646400 to 7317571979590971392. Current Snapshot: MvccSnapshot[applied={T|T < 7317571979628646400}]
I20260812 06:36:36.604950 16424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d2eab862c384cea807403727e020709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2eab862c384cea807403727e020709" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } } peers { permanent_uuid: "72f186d66f4549b7859a28d6f0ebb87b" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } } peers { permanent_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } } }
I20260812 06:36:36.605054 16424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709 [sys.catalog]: This master's current role is: LEADER
I20260812 06:36:36.606011 16399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d2eab862c384cea807403727e020709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2eab862c384cea807403727e020709" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45751 } } peers { permanent_uuid: "72f186d66f4549b7859a28d6f0ebb87b" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 43333 } } peers { permanent_uuid: "3f7947b8a8f14a84a66c96cd7613b8e7" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 35393 } } }
I20260812 06:36:36.606110 16399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:36.606092 16479 mvcc.cc:204] Tried to move back new op lower bound from 7317571979628646400 to 7317571979590971392. Current Snapshot: MvccSnapshot[applied={T|T < 7317571979628646400}]
I20260812 06:36:36.606851 16471 catalog_manager.cc:1383] Generated new cluster ID: 55ec9d23bdb948e68a104c3dd6f6bf59
I20260812 06:36:36.606909 16471 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 06:36:36.623003 16471 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 06:36:36.623839 16471 catalog_manager.cc:1540] Loading token signing keys...
I20260812 06:36:36.631191 16471 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 2d2eab862c384cea807403727e020709: Generated new TSK 0
I20260812 06:36:36.631893 16471 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:36.686565 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.22.65:0
--local_ip_for_outbound_sockets=127.13.22.65
--webserver_interface=127.13.22.65
--webserver_port=0
--tserver_master_addrs=127.13.22.126:45751,127.13.22.125:43333,127.13.22.124:35393
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:45787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 06:36:36.788878 16506 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:36.789063 16506 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:36.789121 16506 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:36.792227 16506 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:36.792331 16506 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.22.65
I20260812 06:36:36.796347 16506 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:45787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.65
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--tserver_master_addrs=127.13.22.126:45751,127.13.22.125:43333,127.13.22.124:35393
--never_fsync=true
--heap_profile_path=/tmp/kudu.16506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.22.65

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:36.797210 16506 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:36.797940 16506 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:36.803678 16523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:36.803755 16516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:36.803697 16518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:36.804000 16506 server_base.cc:1061] running on GCE node
I20260812 06:36:36.804440 16506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:36.804924 16506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:36.806072 16506 hybrid_clock.cc:648] HybridClock initialized: now 1786516596806058 us; error 30 us; skew 500 ppm
Aug 12 06:36:36 dist-test-slave-gjrz krb5kdc[15950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516596, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:36.808656 16506 init.cc:377] Logged in from keytab as kudu/127.13.22.65@KRBTEST.COM (short username kudu)
I20260812 06:36:36.819459 16506 webserver.cc:533] Webserver started at https://127.13.22.65:41147/ using document root <none> and password file <none>
I20260812 06:36:36.820061 16506 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:36.820111 16506 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:36.820299 16506 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:36.822001 16506 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/data/instance:
uuid: "30da96398adf4616a2fb67f94b9315d0"
format_stamp: "Formatted at 2026-08-12 06:36:36 on dist-test-slave-gjrz"
I20260812 06:36:36.822432 16506 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/wal/instance:
uuid: "30da96398adf4616a2fb67f94b9315d0"
format_stamp: "Formatted at 2026-08-12 06:36:36 on dist-test-slave-gjrz"
I20260812 06:36:36.825625 16506 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.000s	sys 0.003s
I20260812 06:36:36.827574 16533 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:36.828481 16506 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.001s
I20260812 06:36:36.828581 16506 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/wal
uuid: "30da96398adf4616a2fb67f94b9315d0"
format_stamp: "Formatted at 2026-08-12 06:36:36 on dist-test-slave-gjrz"
I20260812 06:36:36.828666 16506 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:36.849088 16506 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:36.851300 16506 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:36.851440 16506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:36.851929 16506 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:36.852782 16506 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:36:36.852828 16506 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:36.852862 16506 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:36:36.852882 16506 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:36.861047 16506 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.65:44205
I20260812 06:36:36.861114 16742 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.65:44205 every 8 connection(s)
I20260812 06:36:36.861876 16506 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
Aug 12 06:36:36 dist-test-slave-gjrz krb5kdc[15950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516596, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
Aug 12 06:36:36 dist-test-slave-gjrz krb5kdc[15950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516596, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
Aug 12 06:36:36 dist-test-slave-gjrz krb5kdc[15950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516596, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:36:36.871485 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 16506
I20260812 06:36:36.871626 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/wal/instance
I20260812 06:36:36.876593 16746 heartbeater.cc:344] Connected to a master server at 127.13.22.124:35393
I20260812 06:36:36.876662 16743 heartbeater.cc:344] Connected to a master server at 127.13.22.126:45751
I20260812 06:36:36.876703 16749 heartbeater.cc:344] Connected to a master server at 127.13.22.125:43333
I20260812 06:36:36.876832 16749 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:36.876832 16743 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:36.876830 16746 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:36.877405 16746 heartbeater.cc:507] Master 127.13.22.124:35393 requested a full tablet report, sending...
I20260812 06:36:36.877401 16749 heartbeater.cc:507] Master 127.13.22.125:43333 requested a full tablet report, sending...
I20260812 06:36:36.877399 16743 heartbeater.cc:507] Master 127.13.22.126:45751 requested a full tablet report, sending...
I20260812 06:36:36.878825 16039 ts_manager.cc:194] Registered new tserver with Master: 30da96398adf4616a2fb67f94b9315d0 (127.13.22.65:44205)
I20260812 06:36:36.878815 16311 ts_manager.cc:194] Registered new tserver with Master: 30da96398adf4616a2fb67f94b9315d0 (127.13.22.65:44205)
I20260812 06:36:36.880030 16039 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.22.65@KRBTEST.COM'} at 127.13.22.65:43097
I20260812 06:36:36.880939 16168 ts_manager.cc:194] Registered new tserver with Master: 30da96398adf4616a2fb67f94b9315d0 (127.13.22.65:44205)
I20260812 06:36:36.891419 13401 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
time=2026-08-12T06:36:36.941Z level=INFO source=main.go:1678 msg="updated GOGC" old=100 new=75
time=2026-08-12T06:36:36.943Z level=INFO source=main.go:744 msg="Leaving GOMAXPROCS=4: CPU quota undefined" component=automaxprocs
time=2026-08-12T06:36:36.943Z level=INFO source=memlimit.go:198 msg="GOMEMLIMIT is updated" component=automemlimit package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=14142549196 previous=9223372036854775807
time=2026-08-12T06:36:36.944Z level=INFO source=main.go:851 msg="Starting Prometheus Server" mode=server version="(version=3.11.2, branch=HEAD, revision=f0f0fdd679dcd6df320b0558b20919f7cd44c407)"
time=2026-08-12T06:36:36.944Z level=INFO source=main.go:856 msg="operational information" build_context="(go=go1.26.2, platform=linux/amd64, user=root@d2684485c347, date=20260413-11:53:40, tags=netgo,builtinassets)" host_details="(Linux 5.4.0-1157-gcp #166~18.04.1-Ubuntu SMP Wed Jan 21 01:57:46 UTC 2026 x86_64 dist-test-slave-gjrz (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)"
time=2026-08-12T06:36:37.165Z level=INFO source=web.go:710 msg="Start listening for connections" component=web address=127.0.0.1:0
time=2026-08-12T06:36:37.166Z level=INFO source=main.go:1410 msg="Starting TSDB ..."
time=2026-08-12T06:36:37.171Z level=INFO source=tls_config.go:354 msg="Listening on" component=web address=127.0.0.1:35367
time=2026-08-12T06:36:37.171Z level=INFO source=tls_config.go:357 msg="TLS is disabled." component=web http2=false address=127.0.0.1:35367
time=2026-08-12T06:36:37.171Z level=INFO source=head.go:698 msg="Replaying on-disk memory mappable chunks if any" component=tsdb
time=2026-08-12T06:36:37.171Z level=INFO source=head.go:784 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=2.03µs
time=2026-08-12T06:36:37.171Z level=INFO source=head.go:792 msg="Replaying WAL, this may take a while" component=tsdb
time=2026-08-12T06:36:37.172Z level=INFO source=head.go:865 msg="WAL segment loaded" component=tsdb segment=0 maxSegment=0 duration=539.531µs
time=2026-08-12T06:36:37.172Z level=INFO source=head.go:902 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=69.719µs wal_replay_duration=591.757µs wbl_replay_duration=167ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=2.03µs total_replay_duration=707.805µs
time=2026-08-12T06:36:37.177Z level=INFO source=main.go:1431 msg="filesystem information" fs_type=OVERLAYFS_SUPER_MAGIC
time=2026-08-12T06:36:37.177Z level=INFO source=main.go:1434 msg="TSDB started"
time=2026-08-12T06:36:37.177Z level=INFO source=main.go:1632 msg="Loading configuration file" filename=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/prometheus.yml
time=2026-08-12T06:36:37.177Z level=INFO source=main.go:1048 msg="TSDB retention updated" duration=15d size=0B percentage=0
time=2026-08-12T06:36:37.180Z level=INFO source=main.go:1671 msg="Completed loading of configuration file" db_storage=54.209µs remote_storage=2.901µs web_handler=773ns query_engine=1.844µs scrape=3.225662ms scrape_sd=53.956µs notify=2.271µs notify_sd=1.302µs rules=2.594µs tracing=13.156µs filename=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenScrapeFails.1786516559891118-13401-0/prometheus.yml totalDuration=3.813665ms
time=2026-08-12T06:36:37.181Z level=INFO source=main.go:1395 msg="Server is ready to receive web requests."
time=2026-08-12T06:36:37.181Z level=INFO source=manager.go:209 msg="Starting rule manager..." component="rule manager"
I20260812 06:36:37.419174 16421 catalog_manager.cc:1295] Loaded cluster ID: 55ec9d23bdb948e68a104c3dd6f6bf59
I20260812 06:36:37.419248 16421 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7: loading cluster ID for follower catalog manager: success
I20260812 06:36:37.421008 16421 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7: acquiring CA information for follower catalog manager: success
I20260812 06:36:37.422147 16421 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 3f7947b8a8f14a84a66c96cd7613b8e7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 06:36:37.430680 16466 catalog_manager.cc:1295] Loaded cluster ID: 55ec9d23bdb948e68a104c3dd6f6bf59
I20260812 06:36:37.430740 16466 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b: loading cluster ID for follower catalog manager: success
I20260812 06:36:37.432314 16466 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b: acquiring CA information for follower catalog manager: success
I20260812 06:36:37.433323 16466 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 72f186d66f4549b7859a28d6f0ebb87b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 06:36:37.881614 16743 heartbeater.cc:499] Master 127.13.22.126:45751 was elected leader, sending a full tablet report...
W20260812 06:36:42.488967 16784 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:60756: invalid bearer token
W20260812 06:36:43.013590 16790 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:45848: invalid bearer token
W20260812 06:36:44.044646 16792 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:59960: invalid bearer token
W20260812 06:36:44.181591 16793 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:57312: invalid bearer token
W20260812 06:36:44.477166 16784 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:60756: invalid bearer token
time=2026-08-12T06:36:44.962Z level=WARN source=main.go:1191 msg="Received an OS signal, exiting gracefully..." signal=terminated
time=2026-08-12T06:36:44.962Z level=INFO source=main.go:1216 msg="Stopping scrape discovery manager..."
time=2026-08-12T06:36:44.962Z level=INFO source=main.go:1230 msg="Stopping notify discovery manager..."
time=2026-08-12T06:36:44.962Z level=INFO source=main.go:1245 msg="Stopping rule manager manager..."
time=2026-08-12T06:36:44.962Z level=INFO source=manager.go:223 msg="Stopping rule manager..." component="rule manager"
time=2026-08-12T06:36:44.962Z level=INFO source=main.go:1212 msg="Scrape discovery manager stopped"
time=2026-08-12T06:36:44.962Z level=INFO source=manager.go:239 msg="Rule manager stopped" component="rule manager"
time=2026-08-12T06:36:44.962Z level=INFO source=main.go:1269 msg="Stopping scrape manager..."
time=2026-08-12T06:36:44.962Z level=INFO source=main.go:1226 msg="Notify discovery manager stopped"
time=2026-08-12T06:36:44.962Z level=INFO source=main.go:1241 msg="Rule manager stopped"
time=2026-08-12T06:36:44.962Z level=INFO source=main.go:1284 msg="Stopping tracing manager..."
time=2026-08-12T06:36:44.962Z level=INFO source=main.go:1369 msg="Stopping reloaders..."
time=2026-08-12T06:36:44.962Z level=INFO source=main.go:1261 msg="Scrape manager stopped"
time=2026-08-12T06:36:44.962Z level=INFO source=main.go:1458 msg="Stopping storage..."
time=2026-08-12T06:36:44.962Z level=INFO source=main.go:1280 msg="Tracing manager stopped"
time=2026-08-12T06:36:44.962Z level=INFO source=main.go:1361 msg="Reloaders stopped"
time=2026-08-12T06:36:44.965Z level=INFO source=main.go:1536 msg="Stopping web handler..."
time=2026-08-12T06:36:44.965Z level=INFO source=main.go:1559 msg="Stopping notifier manager..."
time=2026-08-12T06:36:44.965Z level=INFO source=manager.go:321 msg="Stopping notification manager..." component=notifier
time=2026-08-12T06:36:44.965Z level=INFO source=main.go:1454 msg="TSDB stopped"
time=2026-08-12T06:36:44.965Z level=INFO source=main.go:1555 msg="Notifier manager stopped"
time=2026-08-12T06:36:44.965Z level=INFO source=main.go:1532 msg="Web handler stopped"
time=2026-08-12T06:36:44.965Z level=INFO source=main.go:1570 msg="See you next time!"
I20260812 06:36:44.972157 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 16506
I20260812 06:36:44.990945 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 15977
I20260812 06:36:45.001295 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 16111
I20260812 06:36:45.014503 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 16260
2026-08-12T06:36:45Z chronyd exiting
[       OK ] PrometheusTokenITest.TestWrongTokenScrapeFails (11476 ms)
[ RUN      ] PrometheusTokenITest.TestMissingTokenScrapeFails
Loading random data
Initializing database '/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 06:36:45 dist-test-slave-gjrz krb5kdc[16796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 06:36:45 dist-test-slave-gjrz krb5kdc[16796](info): set up 2 sockets
Aug 12 06:36:45 dist-test-slave-gjrz krb5kdc[16796](info): commencing operation
krb5kdc: starting...
W20260812 06:36:47.067543 13401 mini_kdc.cc:121] Time spent starting KDC: real 2.027s	user 0.003s	sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/test-user.keytab.
Aug 12 06:36:47 dist-test-slave-gjrz krb5kdc[16796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM: 
2026-08-12T06:36:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T06:36:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:47.194911 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.126:33495
--webserver_interface=127.13.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:33495,127.13.22.125:42575,127.13.22.124:41715
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 06:36:47.295194 16823 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:47.295426 16823 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:47.295498 16823 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:47.298673 16823 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:36:47.298727 16823 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:47.298745 16823 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:36:47.298776 16823 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:36:47.303522 16823 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:33495,127.13.22.125:42575,127.13.22.124:41715
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.126:33495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.126
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:47.304459 16823 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:47.305207 16823 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:47.310339 16835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:47.310330 16840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:47.310379 16837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:47.310452 16823 server_base.cc:1061] running on GCE node
I20260812 06:36:47.310945 16823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:47.311757 16823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:47.312899 16823 hybrid_clock.cc:648] HybridClock initialized: now 1786516607312886 us; error 31 us; skew 500 ppm
Aug 12 06:36:47 dist-test-slave-gjrz krb5kdc[16796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516607, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:47.315339 16823 init.cc:377] Logged in from keytab as kudu/127.13.22.126@KRBTEST.COM (short username kudu)
I20260812 06:36:47.325892 16823 webserver.cc:533] Webserver started at https://127.13.22.126:33185/ using document root <none> and password file <none>
I20260812 06:36:47.326454 16823 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:47.326498 16823 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:47.326675 16823 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:47.328254 16823 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/data/instance:
uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9"
format_stamp: "Formatted at 2026-08-12 06:36:47 on dist-test-slave-gjrz"
I20260812 06:36:47.328675 16823 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/wal/instance:
uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9"
format_stamp: "Formatted at 2026-08-12 06:36:47 on dist-test-slave-gjrz"
I20260812 06:36:47.331887 16823 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 06:36:47.333880 16858 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:47.334991 16823 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:47.335124 16823 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/wal
uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9"
format_stamp: "Formatted at 2026-08-12 06:36:47 on dist-test-slave-gjrz"
I20260812 06:36:47.335216 16823 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:47.351974 16823 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:47.354130 16823 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:47.354287 16823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:47.360925 16953 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.126:33495 every 8 connection(s)
I20260812 06:36:47.360926 16823 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.126:33495
I20260812 06:36:47.361869 16823 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
I20260812 06:36:47.364590 16955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:47.367887 16955 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } has no permanent_uuid. Determining permanent_uuid...
I20260812 06:36:47.369407 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 16823
I20260812 06:36:47.369508 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-0/wal/instance
Aug 12 06:36:47 dist-test-slave-gjrz krb5kdc[16796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516607, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.22.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.125@KRBTEST.COM" created.
I20260812 06:36:47.381621 16955 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } has no permanent_uuid. Determining permanent_uuid...
W20260812 06:36:47.382323 16867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.22.125:42575: connect: Connection refused (error 111)
W20260812 06:36:47.383164 16955 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:42575: Network error: Client connection negotiation failed: client connection to 127.13.22.125:42575: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:47.411902 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.125:42575
--webserver_interface=127.13.22.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:33495,127.13.22.125:42575,127.13.22.124:41715
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 06:36:47.432349 16955 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } attempt: 1
W20260812 06:36:47.433701 16955 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:42575: Network error: Client connection negotiation failed: client connection to 127.13.22.125:42575: connect: Connection refused (error 111)
I20260812 06:36:47.501843 16955 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } attempt: 2
W20260812 06:36:47.503929 16955 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:42575: Network error: Client connection negotiation failed: client connection to 127.13.22.125:42575: connect: Connection refused (error 111)
W20260812 06:36:47.516207 16976 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:47.516389 16976 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:47.516461 16976 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:47.519654 16976 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:36:47.519706 16976 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:47.519737 16976 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:36:47.519770 16976 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:36:47.523917 16976 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:33495,127.13.22.125:42575,127.13.22.124:41715
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.125:42575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.125
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:47.524807 16976 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:47.525579 16976 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:47.530401 16984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:47.530411 16983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:47.530543 16986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:47.530815 16976 server_base.cc:1061] running on GCE node
I20260812 06:36:47.531252 16976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:47.532079 16976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:47.533228 16976 hybrid_clock.cc:648] HybridClock initialized: now 1786516607533215 us; error 23 us; skew 500 ppm
Aug 12 06:36:47 dist-test-slave-gjrz krb5kdc[16796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516607, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:47.535665 16976 init.cc:377] Logged in from keytab as kudu/127.13.22.125@KRBTEST.COM (short username kudu)
I20260812 06:36:47.546640 16976 webserver.cc:533] Webserver started at https://127.13.22.125:34805/ using document root <none> and password file <none>
I20260812 06:36:47.547204 16976 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:47.547255 16976 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:47.547475 16976 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:47.549012 16976 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/data/instance:
uuid: "17d009c95de34ba288eaad9d5a44961d"
format_stamp: "Formatted at 2026-08-12 06:36:47 on dist-test-slave-gjrz"
I20260812 06:36:47.549448 16976 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/wal/instance:
uuid: "17d009c95de34ba288eaad9d5a44961d"
format_stamp: "Formatted at 2026-08-12 06:36:47 on dist-test-slave-gjrz"
I20260812 06:36:47.552752 16976 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.000s	sys 0.005s
I20260812 06:36:47.554672 17001 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:47.555608 16976 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 06:36:47.555723 16976 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/wal
uuid: "17d009c95de34ba288eaad9d5a44961d"
format_stamp: "Formatted at 2026-08-12 06:36:47 on dist-test-slave-gjrz"
I20260812 06:36:47.555802 16976 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:47.595108 16955 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } attempt: 3
W20260812 06:36:47.596442 16955 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:42575: Network error: Client connection negotiation failed: client connection to 127.13.22.125:42575: connect: Connection refused (error 111)
I20260812 06:36:47.598685 16976 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:47.600870 16976 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:47.601035 16976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:47.608374 17094 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.125:42575 every 8 connection(s)
I20260812 06:36:47.608371 16976 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.125:42575
I20260812 06:36:47.609349 16976 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
I20260812 06:36:47.611785 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 16976
I20260812 06:36:47.611904 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-1/wal/instance
I20260812 06:36:47.612226 17095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:47.615907 17095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.22.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.124@KRBTEST.COM" created.
Aug 12 06:36:47 dist-test-slave-gjrz krb5kdc[16796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516607, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:36:47.627811 17095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:36:47 dist-test-slave-gjrz krb5kdc[16796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516607, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:47.640194 17095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } has no permanent_uuid. Determining permanent_uuid...
W20260812 06:36:47.640879 17015 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.22.124:41715: connect: Connection refused (error 111)
W20260812 06:36:47.641700 17095 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:41715: Network error: Client connection negotiation failed: client connection to 127.13.22.124:41715: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.13.22.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:47.657290 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.124:41715
--webserver_interface=127.13.22.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:33495,127.13.22.125:42575,127.13.22.124:41715
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 06:36:47.690948 17095 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } attempt: 1
W20260812 06:36:47.692416 17095 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:41715: Network error: Client connection negotiation failed: client connection to 127.13.22.124:41715: connect: Connection refused (error 111)
I20260812 06:36:47.739601 16955 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } attempt: 4
Aug 12 06:36:47 dist-test-slave-gjrz krb5kdc[16796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516607, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
I20260812 06:36:47.748023 16955 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } has no permanent_uuid. Determining permanent_uuid...
W20260812 06:36:47.749104 16955 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:41715: Network error: Client connection negotiation failed: client connection to 127.13.22.124:41715: connect: Connection refused (error 111)
W20260812 06:36:47.757359 17109 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:47.757539 17109 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:47.757632 17109 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260812 06:36:47.760584 17095 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } attempt: 2
W20260812 06:36:47.760886 17109 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:36:47.760941 17109 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:47.761018 17109 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:36:47.761056 17109 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260812 06:36:47.761667 17095 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:41715: Network error: Client connection negotiation failed: client connection to 127.13.22.124:41715: connect: Connection refused (error 111)
I20260812 06:36:47.765211 17109 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:33495,127.13.22.125:42575,127.13.22.124:41715
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.124:41715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.124
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:47.766108 17109 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:47.766906 17109 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:47.771762 17124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:47.771804 17121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:47.771956 17122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:47.772032 17109 server_base.cc:1061] running on GCE node
I20260812 06:36:47.772552 17109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:47.773322 17109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:47.774475 17109 hybrid_clock.cc:648] HybridClock initialized: now 1786516607774459 us; error 32 us; skew 500 ppm
Aug 12 06:36:47 dist-test-slave-gjrz krb5kdc[16796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516607, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:47.776953 17109 init.cc:377] Logged in from keytab as kudu/127.13.22.124@KRBTEST.COM (short username kudu)
I20260812 06:36:47.787374 17109 webserver.cc:533] Webserver started at https://127.13.22.124:46247/ using document root <none> and password file <none>
I20260812 06:36:47.787948 17109 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:47.787997 17109 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:47.788189 17109 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:47.789752 17109 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/data/instance:
uuid: "c0202fb44b1b4a4aa94689f376b42f80"
format_stamp: "Formatted at 2026-08-12 06:36:47 on dist-test-slave-gjrz"
I20260812 06:36:47.790184 17109 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/wal/instance:
uuid: "c0202fb44b1b4a4aa94689f376b42f80"
format_stamp: "Formatted at 2026-08-12 06:36:47 on dist-test-slave-gjrz"
I20260812 06:36:47.793483 17109 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20260812 06:36:47.795442 17146 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:47.796357 17109 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:47.796469 17109 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/wal
uuid: "c0202fb44b1b4a4aa94689f376b42f80"
format_stamp: "Formatted at 2026-08-12 06:36:47 on dist-test-slave-gjrz"
I20260812 06:36:47.796555 17109 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:47.808247 16955 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } attempt: 1
W20260812 06:36:47.809449 16955 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:41715: Network error: Client connection negotiation failed: client connection to 127.13.22.124:41715: connect: Connection refused (error 111)
I20260812 06:36:47.814383 17109 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:47.816972 17109 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:47.817134 17109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:47.824219 17232 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.124:41715 every 8 connection(s)
I20260812 06:36:47.824230 17109 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.124:41715
I20260812 06:36:47.825248 17109 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
I20260812 06:36:47.828140 17233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:47.831619 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 17109
I20260812 06:36:47.831734 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/master-2/wal/instance
I20260812 06:36:47.832000 17233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:36:47 dist-test-slave-gjrz krb5kdc[16796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
Aug 12 06:36:47 dist-test-slave-gjrz krb5kdc[16796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516607, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:36:47.844318 17233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:36:47 dist-test-slave-gjrz krb5kdc[16796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
Aug 12 06:36:47 dist-test-slave-gjrz krb5kdc[16796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516607, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
I20260812 06:36:47.852826 17095 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } attempt: 3
I20260812 06:36:47.852978 17233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:36:47 dist-test-slave-gjrz krb5kdc[16796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
Aug 12 06:36:47 dist-test-slave-gjrz krb5kdc[16796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516607, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
Aug 12 06:36:47 dist-test-slave-gjrz krb5kdc[16796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516607, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:36:47.863941 17233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80: Bootstrap starting.
I20260812 06:36:47.864187 17095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d: Bootstrap starting.
I20260812 06:36:47.866468 17095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:47.866468 17233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:47.867342 17233 log.cc:826] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:47.867347 17095 log.cc:826] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:47.869004 17095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d: No bootstrap required, opened a new log
I20260812 06:36:47.869004 17233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80: No bootstrap required, opened a new log
I20260812 06:36:47.872072 17095 raft_consensus.cc:359] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } } peers { permanent_uuid: "17d009c95de34ba288eaad9d5a44961d" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } } peers { permanent_uuid: "c0202fb44b1b4a4aa94689f376b42f80" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } }
I20260812 06:36:47.872069 17233 raft_consensus.cc:359] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } } peers { permanent_uuid: "17d009c95de34ba288eaad9d5a44961d" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } } peers { permanent_uuid: "c0202fb44b1b4a4aa94689f376b42f80" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } }
I20260812 06:36:47.872247 17095 raft_consensus.cc:385] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:47.872247 17233 raft_consensus.cc:385] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:47.872303 17095 raft_consensus.cc:740] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d009c95de34ba288eaad9d5a44961d, State: Initialized, Role: FOLLOWER
I20260812 06:36:47.872313 17233 raft_consensus.cc:740] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0202fb44b1b4a4aa94689f376b42f80, State: Initialized, Role: FOLLOWER
I20260812 06:36:47.872730 17233 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } } peers { permanent_uuid: "17d009c95de34ba288eaad9d5a44961d" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } } peers { permanent_uuid: "c0202fb44b1b4a4aa94689f376b42f80" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } }
I20260812 06:36:47.872730 17095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } } peers { permanent_uuid: "17d009c95de34ba288eaad9d5a44961d" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } } peers { permanent_uuid: "c0202fb44b1b4a4aa94689f376b42f80" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } }
I20260812 06:36:47.873153 17250 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } } peers { permanent_uuid: "17d009c95de34ba288eaad9d5a44961d" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } } peers { permanent_uuid: "c0202fb44b1b4a4aa94689f376b42f80" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } } }
I20260812 06:36:47.873256 17250 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:47.873481 17253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } } peers { permanent_uuid: "17d009c95de34ba288eaad9d5a44961d" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } } peers { permanent_uuid: "c0202fb44b1b4a4aa94689f376b42f80" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } } }
I20260812 06:36:47.873580 17253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:47.873859 17233 sys_catalog.cc:565] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:36:47.873859 17095 sys_catalog.cc:565] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:36:47.876579 16955 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } attempt: 2
W20260812 06:36:47.879307 17287 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:36:47.879432 17287 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:36:47.877493 17109 master_runner.cc:187] Error getting master registration for 127.13.22.126:33495: OK, instance_id { permanent_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" instance_seqno: 1786516607359127 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 06:36:47.879650 17109 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 06:36:47.879395 16976 master_runner.cc:187] Error getting master registration for 127.13.22.126:33495: OK, instance_id { permanent_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" instance_seqno: 1786516607359127 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 06:36:47.881078 16976 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Aug 12 06:36:47 dist-test-slave-gjrz krb5kdc[16796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516607, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
W20260812 06:36:47.883809 17286 catalog_manager.cc:1594] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:36:47.883942 17286 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:36:47.890086 16955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9: Bootstrap starting.
I20260812 06:36:47.892988 16955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:47.893846 16955 log.cc:826] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:47.895678 16955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9: No bootstrap required, opened a new log
I20260812 06:36:47.898769 16955 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } } peers { permanent_uuid: "17d009c95de34ba288eaad9d5a44961d" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } } peers { permanent_uuid: "c0202fb44b1b4a4aa94689f376b42f80" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } }
I20260812 06:36:47.898948 16955 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:47.899046 16955 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e0ed5c10d2d437e953af4a01c3bb5b9, State: Initialized, Role: FOLLOWER
I20260812 06:36:47.899510 16955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } } peers { permanent_uuid: "17d009c95de34ba288eaad9d5a44961d" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } } peers { permanent_uuid: "c0202fb44b1b4a4aa94689f376b42f80" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } }
I20260812 06:36:47.899976 17291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } } peers { permanent_uuid: "17d009c95de34ba288eaad9d5a44961d" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } } peers { permanent_uuid: "c0202fb44b1b4a4aa94689f376b42f80" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } } }
I20260812 06:36:47.900149 17291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:47.900621 17291 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:36:47.900807 17291 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } } peers { permanent_uuid: "17d009c95de34ba288eaad9d5a44961d" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } } peers { permanent_uuid: "c0202fb44b1b4a4aa94689f376b42f80" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } }
I20260812 06:36:47.900866 16955 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:36:47.902706 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d009c95de34ba288eaad9d5a44961d" is_pre_election: true
I20260812 06:36:47.902973 17056 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e0ed5c10d2d437e953af4a01c3bb5b9 in term 0.
I20260812 06:36:47.904582 16867 leader_election.cc:304] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d009c95de34ba288eaad9d5a44961d, 1e0ed5c10d2d437e953af4a01c3bb5b9; no voters: 
I20260812 06:36:47.905794 17185 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0202fb44b1b4a4aa94689f376b42f80" is_pre_election: true
I20260812 06:36:47.906028 17185 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e0ed5c10d2d437e953af4a01c3bb5b9 in term 0.
W20260812 06:36:47.908407 17311 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:36:47.908592 17311 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:36:47.909307 16823 master_runner.cc:264] No leader master found from master 1e0ed5c10d2d437e953af4a01c3bb5b9
I20260812 06:36:47.909363 16823 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 06:36:47.909456 17291 leader_election.cc:290] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17d009c95de34ba288eaad9d5a44961d (127.13.22.125:42575), c0202fb44b1b4a4aa94689f376b42f80 (127.13.22.124:41715)
I20260812 06:36:47.909709 17291 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 06:36:47.909812 17291 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 06:36:47.909900 17291 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:47.911034 17291 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } } peers { permanent_uuid: "17d009c95de34ba288eaad9d5a44961d" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } } peers { permanent_uuid: "c0202fb44b1b4a4aa94689f376b42f80" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } }
I20260812 06:36:47.911743 17056 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d009c95de34ba288eaad9d5a44961d"
I20260812 06:36:47.911854 17056 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:47.912878 17056 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e0ed5c10d2d437e953af4a01c3bb5b9 in term 1.
I20260812 06:36:47.913232 16867 leader_election.cc:304] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d009c95de34ba288eaad9d5a44961d, 1e0ed5c10d2d437e953af4a01c3bb5b9; no voters: 
I20260812 06:36:47.913780 17185 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0202fb44b1b4a4aa94689f376b42f80"
I20260812 06:36:47.913889 17185 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:47.914902 17185 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e0ed5c10d2d437e953af4a01c3bb5b9 in term 1.
I20260812 06:36:47.916560 17291 leader_election.cc:290] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [CANDIDATE]: Term 1 election: Requested vote from peers 17d009c95de34ba288eaad9d5a44961d (127.13.22.125:42575), c0202fb44b1b4a4aa94689f376b42f80 (127.13.22.124:41715)
I20260812 06:36:47.916759 17291 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 06:36:47.916875 17291 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1e0ed5c10d2d437e953af4a01c3bb5b9, State: Running, Role: LEADER
I20260812 06:36:47.917182 17291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } } peers { permanent_uuid: "17d009c95de34ba288eaad9d5a44961d" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } } peers { permanent_uuid: "c0202fb44b1b4a4aa94689f376b42f80" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } }
I20260812 06:36:47.918808 17291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e0ed5c10d2d437e953af4a01c3bb5b9. Latest consensus state: current_term: 1 leader_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } } peers { permanent_uuid: "17d009c95de34ba288eaad9d5a44961d" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } } peers { permanent_uuid: "c0202fb44b1b4a4aa94689f376b42f80" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } } }
I20260812 06:36:47.919021 17291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [sys.catalog]: This master's current role is: LEADER
I20260812 06:36:47.923535 17318 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 06:36:47.924577 17318 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 06:36:47.929771 17185 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80 [term 1 FOLLOWER]: Refusing update from remote peer 1e0ed5c10d2d437e953af4a01c3bb5b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:36:47.932354 17291 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0202fb44b1b4a4aa94689f376b42f80" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:36:47.936789 17250 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e0ed5c10d2d437e953af4a01c3bb5b9. Latest consensus state: current_term: 1 leader_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } } peers { permanent_uuid: "17d009c95de34ba288eaad9d5a44961d" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } } peers { permanent_uuid: "c0202fb44b1b4a4aa94689f376b42f80" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } } }
I20260812 06:36:47.936908 17250 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:47.937139 17056 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d [term 1 FOLLOWER]: Refusing update from remote peer 1e0ed5c10d2d437e953af4a01c3bb5b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:36:47.937716 17291 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d009c95de34ba288eaad9d5a44961d" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:36:47.941468 17253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e0ed5c10d2d437e953af4a01c3bb5b9. Latest consensus state: current_term: 1 leader_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } } peers { permanent_uuid: "17d009c95de34ba288eaad9d5a44961d" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } } peers { permanent_uuid: "c0202fb44b1b4a4aa94689f376b42f80" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } } }
I20260812 06:36:47.941588 17253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:47.943177 17291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } } peers { permanent_uuid: "17d009c95de34ba288eaad9d5a44961d" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } } peers { permanent_uuid: "c0202fb44b1b4a4aa94689f376b42f80" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } } }
I20260812 06:36:47.943288 17291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [sys.catalog]: This master's current role is: LEADER
I20260812 06:36:47.944180 17253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } } peers { permanent_uuid: "17d009c95de34ba288eaad9d5a44961d" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } } peers { permanent_uuid: "c0202fb44b1b4a4aa94689f376b42f80" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } } }
I20260812 06:36:47.944276 17253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:47.944815 17320 mvcc.cc:204] Tried to move back new op lower bound from 7317572026072973312 to 7317572026031755264. Current Snapshot: MvccSnapshot[applied={T|T < 7317572026072973312}]
I20260812 06:36:47.949756 17324 mvcc.cc:204] Tried to move back new op lower bound from 7317572026072973312 to 7317572026031755264. Current Snapshot: MvccSnapshot[applied={T|T < 7317572026072973312}]
I20260812 06:36:47.950496 17312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } } peers { permanent_uuid: "17d009c95de34ba288eaad9d5a44961d" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } } peers { permanent_uuid: "c0202fb44b1b4a4aa94689f376b42f80" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } } }
I20260812 06:36:47.950613 17312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9 [sys.catalog]: This master's current role is: LEADER
I20260812 06:36:47.951444 17250 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0ed5c10d2d437e953af4a01c3bb5b9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 33495 } } peers { permanent_uuid: "17d009c95de34ba288eaad9d5a44961d" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42575 } } peers { permanent_uuid: "c0202fb44b1b4a4aa94689f376b42f80" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 41715 } } }
I20260812 06:36:47.951550 17250 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:47.954526 17322 mvcc.cc:204] Tried to move back new op lower bound from 7317572026072973312 to 7317572026031755264. Current Snapshot: MvccSnapshot[applied={T|T < 7317572026072973312}]
I20260812 06:36:47.954998 17318 catalog_manager.cc:1383] Generated new cluster ID: 495e2ef3109b4bf9b00d0c0d43b2827c
I20260812 06:36:47.955050 17318 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 06:36:48.004042 17318 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 06:36:48.005295 17318 catalog_manager.cc:1540] Loading token signing keys...
I20260812 06:36:48.031407 17318 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 1e0ed5c10d2d437e953af4a01c3bb5b9: Generated new TSK 0
I20260812 06:36:48.032171 17318 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:48.092468 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.22.65:0
--local_ip_for_outbound_sockets=127.13.22.65
--webserver_interface=127.13.22.65
--webserver_port=0
--tserver_master_addrs=127.13.22.126:33495,127.13.22.125:42575,127.13.22.124:41715
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 06:36:48.196920 17337 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:48.197116 17337 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:48.197184 17337 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:48.200460 17337 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:48.200594 17337 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.22.65
I20260812 06:36:48.204885 17337 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.65
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--tserver_master_addrs=127.13.22.126:33495,127.13.22.125:42575,127.13.22.124:41715
--never_fsync=true
--heap_profile_path=/tmp/kudu.17337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.22.65

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:48.205801 17337 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:48.206557 17337 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:48.212442 17345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:48.212440 17350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:48.212488 17344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:48.212670 17337 server_base.cc:1061] running on GCE node
I20260812 06:36:48.213030 17337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:48.213522 17337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:48.214665 17337 hybrid_clock.cc:648] HybridClock initialized: now 1786516608214657 us; error 26 us; skew 500 ppm
Aug 12 06:36:48 dist-test-slave-gjrz krb5kdc[16796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516608, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:48.217164 17337 init.cc:377] Logged in from keytab as kudu/127.13.22.65@KRBTEST.COM (short username kudu)
I20260812 06:36:48.227733 17337 webserver.cc:533] Webserver started at https://127.13.22.65:40243/ using document root <none> and password file <none>
I20260812 06:36:48.228278 17337 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:48.228327 17337 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:48.228513 17337 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:48.230046 17337 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/data/instance:
uuid: "dc0bf6b61c3c4dad8bcde4fc039283f8"
format_stamp: "Formatted at 2026-08-12 06:36:48 on dist-test-slave-gjrz"
I20260812 06:36:48.230451 17337 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/wal/instance:
uuid: "dc0bf6b61c3c4dad8bcde4fc039283f8"
format_stamp: "Formatted at 2026-08-12 06:36:48 on dist-test-slave-gjrz"
I20260812 06:36:48.233623 17337 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.001s	sys 0.004s
I20260812 06:36:48.235563 17371 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:48.236486 17337 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.001s
I20260812 06:36:48.236582 17337 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/wal
uuid: "dc0bf6b61c3c4dad8bcde4fc039283f8"
format_stamp: "Formatted at 2026-08-12 06:36:48 on dist-test-slave-gjrz"
I20260812 06:36:48.236668 17337 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:48.252486 17337 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:48.254573 17337 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:48.254731 17337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:48.255254 17337 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:48.256155 17337 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:36:48.256201 17337 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:48.256240 17337 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:36:48.256270 17337 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:48.264869 17337 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.65:40201
I20260812 06:36:48.264899 17570 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.65:40201 every 8 connection(s)
I20260812 06:36:48.265727 17337 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
I20260812 06:36:48.268219 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 17337
I20260812 06:36:48.268347 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/minicluster-data/ts-0/wal/instance
Aug 12 06:36:48 dist-test-slave-gjrz krb5kdc[16796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516608, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
Aug 12 06:36:48 dist-test-slave-gjrz krb5kdc[16796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516608, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
Aug 12 06:36:48 dist-test-slave-gjrz krb5kdc[16796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516608, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:36:48.279887 17575 heartbeater.cc:344] Connected to a master server at 127.13.22.125:42575
I20260812 06:36:48.279933 17571 heartbeater.cc:344] Connected to a master server at 127.13.22.124:41715
I20260812 06:36:48.279889 17572 heartbeater.cc:344] Connected to a master server at 127.13.22.126:33495
I20260812 06:36:48.280145 17575 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:48.280143 17571 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:48.280141 17572 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:48.280628 17571 heartbeater.cc:507] Master 127.13.22.124:41715 requested a full tablet report, sending...
I20260812 06:36:48.280634 17572 heartbeater.cc:507] Master 127.13.22.126:33495 requested a full tablet report, sending...
I20260812 06:36:48.280634 17575 heartbeater.cc:507] Master 127.13.22.125:42575 requested a full tablet report, sending...
I20260812 06:36:48.282070 17033 ts_manager.cc:194] Registered new tserver with Master: dc0bf6b61c3c4dad8bcde4fc039283f8 (127.13.22.65:40201)
I20260812 06:36:48.282094 17173 ts_manager.cc:194] Registered new tserver with Master: dc0bf6b61c3c4dad8bcde4fc039283f8 (127.13.22.65:40201)
I20260812 06:36:48.282104 16896 ts_manager.cc:194] Registered new tserver with Master: dc0bf6b61c3c4dad8bcde4fc039283f8 (127.13.22.65:40201)
I20260812 06:36:48.283340 16896 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.22.65@KRBTEST.COM'} at 127.13.22.65:34471
I20260812 06:36:48.287557 13401 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
time=2026-08-12T06:36:48.349Z level=INFO source=main.go:1678 msg="updated GOGC" old=100 new=75
time=2026-08-12T06:36:48.350Z level=INFO source=main.go:744 msg="Leaving GOMAXPROCS=4: CPU quota undefined" component=automaxprocs
time=2026-08-12T06:36:48.350Z level=INFO source=memlimit.go:198 msg="GOMEMLIMIT is updated" component=automemlimit package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=14142549196 previous=9223372036854775807
time=2026-08-12T06:36:48.350Z level=INFO source=main.go:851 msg="Starting Prometheus Server" mode=server version="(version=3.11.2, branch=HEAD, revision=f0f0fdd679dcd6df320b0558b20919f7cd44c407)"
time=2026-08-12T06:36:48.350Z level=INFO source=main.go:856 msg="operational information" build_context="(go=go1.26.2, platform=linux/amd64, user=root@d2684485c347, date=20260413-11:53:40, tags=netgo,builtinassets)" host_details="(Linux 5.4.0-1157-gcp #166~18.04.1-Ubuntu SMP Wed Jan 21 01:57:46 UTC 2026 x86_64 dist-test-slave-gjrz (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)"
time=2026-08-12T06:36:48.587Z level=INFO source=web.go:710 msg="Start listening for connections" component=web address=127.0.0.1:0
time=2026-08-12T06:36:48.587Z level=INFO source=main.go:1410 msg="Starting TSDB ..."
time=2026-08-12T06:36:48.603Z level=INFO source=tls_config.go:354 msg="Listening on" component=web address=127.0.0.1:33547
time=2026-08-12T06:36:48.603Z level=INFO source=tls_config.go:357 msg="TLS is disabled." component=web http2=false address=127.0.0.1:33547
time=2026-08-12T06:36:48.605Z level=INFO source=head.go:698 msg="Replaying on-disk memory mappable chunks if any" component=tsdb
time=2026-08-12T06:36:48.605Z level=INFO source=head.go:784 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=2.606µs
time=2026-08-12T06:36:48.605Z level=INFO source=head.go:792 msg="Replaying WAL, this may take a while" component=tsdb
time=2026-08-12T06:36:48.606Z level=INFO source=head.go:865 msg="WAL segment loaded" component=tsdb segment=0 maxSegment=0 duration=414.298µs
time=2026-08-12T06:36:48.606Z level=INFO source=head.go:902 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=63.76µs wal_replay_duration=497.623µs wbl_replay_duration=439ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=2.606µs total_replay_duration=687.418µs
time=2026-08-12T06:36:48.610Z level=INFO source=main.go:1431 msg="filesystem information" fs_type=OVERLAYFS_SUPER_MAGIC
time=2026-08-12T06:36:48.610Z level=INFO source=main.go:1434 msg="TSDB started"
time=2026-08-12T06:36:48.610Z level=INFO source=main.go:1632 msg="Loading configuration file" filename=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/prometheus.yml
time=2026-08-12T06:36:48.610Z level=INFO source=main.go:1048 msg="TSDB retention updated" duration=15d size=0B percentage=0
time=2026-08-12T06:36:48.613Z level=INFO source=main.go:1671 msg="Completed loading of configuration file" db_storage=96.359µs remote_storage=3.865µs web_handler=696ns query_engine=1.147µs scrape=1.882426ms scrape_sd=41.505µs notify=2.146µs notify_sd=1.558µs rules=2.248µs tracing=13.929µs filename=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenScrapeFails.1786516559891118-13401-0/prometheus.yml totalDuration=2.543065ms
time=2026-08-12T06:36:48.613Z level=INFO source=main.go:1395 msg="Server is ready to receive web requests."
time=2026-08-12T06:36:48.613Z level=INFO source=manager.go:209 msg="Starting rule manager..." component="rule manager"
I20260812 06:36:48.881254 17287 catalog_manager.cc:1295] Loaded cluster ID: 495e2ef3109b4bf9b00d0c0d43b2827c
I20260812 06:36:48.881376 17287 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d: loading cluster ID for follower catalog manager: success
I20260812 06:36:48.884189 17287 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d: acquiring CA information for follower catalog manager: success
I20260812 06:36:48.885106 17286 catalog_manager.cc:1295] Loaded cluster ID: 495e2ef3109b4bf9b00d0c0d43b2827c
I20260812 06:36:48.885164 17286 catalog_manager.cc:1588] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80: loading cluster ID for follower catalog manager: success
I20260812 06:36:48.885823 17287 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 17d009c95de34ba288eaad9d5a44961d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 06:36:48.886977 17286 catalog_manager.cc:1610] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80: acquiring CA information for follower catalog manager: success
I20260812 06:36:48.888072 17286 catalog_manager.cc:1638] T 00000000000000000000000000000000 P c0202fb44b1b4a4aa94689f376b42f80: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 06:36:49.284953 17572 heartbeater.cc:499] Master 127.13.22.126:33495 was elected leader, sending a full tablet report...
time=2026-08-12T06:36:55.364Z level=WARN source=main.go:1191 msg="Received an OS signal, exiting gracefully..." signal=terminated
time=2026-08-12T06:36:55.364Z level=INFO source=main.go:1216 msg="Stopping scrape discovery manager..."
time=2026-08-12T06:36:55.364Z level=INFO source=main.go:1230 msg="Stopping notify discovery manager..."
time=2026-08-12T06:36:55.364Z level=INFO source=main.go:1245 msg="Stopping rule manager manager..."
time=2026-08-12T06:36:55.364Z level=INFO source=manager.go:223 msg="Stopping rule manager..." component="rule manager"
time=2026-08-12T06:36:55.364Z level=INFO source=manager.go:239 msg="Rule manager stopped" component="rule manager"
time=2026-08-12T06:36:55.364Z level=INFO source=main.go:1269 msg="Stopping scrape manager..."
time=2026-08-12T06:36:55.364Z level=INFO source=main.go:1212 msg="Scrape discovery manager stopped"
time=2026-08-12T06:36:55.366Z level=INFO source=main.go:1241 msg="Rule manager stopped"
time=2026-08-12T06:36:55.366Z level=INFO source=main.go:1226 msg="Notify discovery manager stopped"
time=2026-08-12T06:36:55.366Z level=INFO source=main.go:1284 msg="Stopping tracing manager..."
time=2026-08-12T06:36:55.366Z level=INFO source=main.go:1369 msg="Stopping reloaders..."
time=2026-08-12T06:36:55.366Z level=INFO source=main.go:1458 msg="Stopping storage..."
time=2026-08-12T06:36:55.366Z level=INFO source=main.go:1261 msg="Scrape manager stopped"
time=2026-08-12T06:36:55.367Z level=INFO source=main.go:1280 msg="Tracing manager stopped"
time=2026-08-12T06:36:55.367Z level=INFO source=main.go:1361 msg="Reloaders stopped"
time=2026-08-12T06:36:55.370Z level=INFO source=main.go:1536 msg="Stopping web handler..."
time=2026-08-12T06:36:55.370Z level=INFO source=main.go:1559 msg="Stopping notifier manager..."
time=2026-08-12T06:36:55.370Z level=INFO source=manager.go:321 msg="Stopping notification manager..." component=notifier
time=2026-08-12T06:36:55.370Z level=INFO source=main.go:1555 msg="Notifier manager stopped"
time=2026-08-12T06:36:55.370Z level=INFO source=main.go:1532 msg="Web handler stopped"
time=2026-08-12T06:36:55.370Z level=INFO source=main.go:1454 msg="TSDB stopped"
time=2026-08-12T06:36:55.370Z level=INFO source=main.go:1570 msg="See you next time!"
I20260812 06:36:55.384634 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 17337
I20260812 06:36:55.399572 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 16823
I20260812 06:36:55.409912 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 16976
I20260812 06:36:55.426946 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 17109
2026-08-12T06:36:55Z chronyd exiting
[       OK ] PrometheusTokenITest.TestMissingTokenScrapeFails (10414 ms)
[ RUN      ] PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed
Loading random data
Initializing database '/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 06:36:55 dist-test-slave-gjrz krb5kdc[17629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 06:36:55 dist-test-slave-gjrz krb5kdc[17629](info): set up 2 sockets
Aug 12 06:36:55 dist-test-slave-gjrz krb5kdc[17629](info): commencing operation
krb5kdc: starting...
W20260812 06:36:57.510607 13401 mini_kdc.cc:121] Time spent starting KDC: real 2.056s	user 0.000s	sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/test-user.keytab.
Aug 12 06:36:57 dist-test-slave-gjrz krb5kdc[17629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM: 
2026-08-12T06:36:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T06:36:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:57.634084 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.126:40957
--webserver_interface=127.13.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:40957,127.13.22.125:41489,127.13.22.124:33657
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 06:36:57.734714 17657 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:57.734915 17657 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:57.734993 17657 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:57.738185 17657 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:36:57.738247 17657 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:57.738273 17657 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:36:57.738296 17657 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:36:57.742460 17657 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:40957,127.13.22.125:41489,127.13.22.124:33657
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.126:40957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.126
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:57.743433 17657 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:57.744227 17657 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:57.749567 17680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:57.749555 17676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:57.749636 17684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:57.749706 17657 server_base.cc:1061] running on GCE node
I20260812 06:36:57.750263 17657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:57.751060 17657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:57.752209 17657 hybrid_clock.cc:648] HybridClock initialized: now 1786516617752173 us; error 49 us; skew 500 ppm
Aug 12 06:36:57 dist-test-slave-gjrz krb5kdc[17629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516617, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:57.754693 17657 init.cc:377] Logged in from keytab as kudu/127.13.22.126@KRBTEST.COM (short username kudu)
I20260812 06:36:57.765771 17657 webserver.cc:533] Webserver started at https://127.13.22.126:42315/ using document root <none> and password file <none>
I20260812 06:36:57.766345 17657 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:57.766396 17657 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:57.766592 17657 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:57.768244 17657 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-0/data/instance:
uuid: "54ea9288ee8848e7a3ffe866d561d3f2"
format_stamp: "Formatted at 2026-08-12 06:36:57 on dist-test-slave-gjrz"
I20260812 06:36:57.768698 17657 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-0/wal/instance:
uuid: "54ea9288ee8848e7a3ffe866d561d3f2"
format_stamp: "Formatted at 2026-08-12 06:36:57 on dist-test-slave-gjrz"
I20260812 06:36:57.771951 17657 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 06:36:57.773942 17699 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:57.774868 17657 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 06:36:57.774982 17657 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-0/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-0/wal
uuid: "54ea9288ee8848e7a3ffe866d561d3f2"
format_stamp: "Formatted at 2026-08-12 06:36:57 on dist-test-slave-gjrz"
I20260812 06:36:57.775063 17657 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:57.785956 17657 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:57.788553 17657 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:57.788719 17657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:57.795708 17657 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.126:40957
I20260812 06:36:57.795725 17794 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.126:40957 every 8 connection(s)
I20260812 06:36:57.796607 17657 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
I20260812 06:36:57.798650 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 17657
I20260812 06:36:57.798766 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-0/wal/instance
I20260812 06:36:57.799538 17795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:57.803053 17795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.22.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.125@KRBTEST.COM" created.
Aug 12 06:36:57 dist-test-slave-gjrz krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516617, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:36:57.816427 17795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } has no permanent_uuid. Determining permanent_uuid...
W20260812 06:36:57.817144 17703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.22.125:41489: connect: Connection refused (error 111)
W20260812 06:36:57.817940 17795 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:41489: Network error: Client connection negotiation failed: client connection to 127.13.22.125:41489: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:57.843444 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.125:41489
--webserver_interface=127.13.22.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:40957,127.13.22.125:41489,127.13.22.124:33657
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 06:36:57.867151 17795 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } attempt: 1
W20260812 06:36:57.868649 17795 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:41489: Network error: Client connection negotiation failed: client connection to 127.13.22.125:41489: connect: Connection refused (error 111)
I20260812 06:36:57.936810 17795 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } attempt: 2
W20260812 06:36:57.938301 17795 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:41489: Network error: Client connection negotiation failed: client connection to 127.13.22.125:41489: connect: Connection refused (error 111)
W20260812 06:36:57.945829 17818 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:57.946035 17818 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:57.946118 17818 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:57.949350 17818 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:36:57.949407 17818 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:57.949433 17818 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:36:57.949465 17818 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:36:57.953605 17818 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:40957,127.13.22.125:41489,127.13.22.124:33657
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.125:41489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.125
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:57.954550 17818 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:57.955365 17818 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:57.960294 17832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:57.960287 17825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:57.960269 17827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:57.960489 17818 server_base.cc:1061] running on GCE node
I20260812 06:36:57.960935 17818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:57.961740 17818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:57.962882 17818 hybrid_clock.cc:648] HybridClock initialized: now 1786516617962871 us; error 30 us; skew 500 ppm
Aug 12 06:36:57 dist-test-slave-gjrz krb5kdc[17629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516617, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:57.965366 17818 init.cc:377] Logged in from keytab as kudu/127.13.22.125@KRBTEST.COM (short username kudu)
I20260812 06:36:57.976281 17818 webserver.cc:533] Webserver started at https://127.13.22.125:45879/ using document root <none> and password file <none>
I20260812 06:36:57.976850 17818 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:57.976902 17818 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:57.977097 17818 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:57.978735 17818 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-1/data/instance:
uuid: "97e6f35670894884bbe8899b91720639"
format_stamp: "Formatted at 2026-08-12 06:36:57 on dist-test-slave-gjrz"
I20260812 06:36:57.979174 17818 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-1/wal/instance:
uuid: "97e6f35670894884bbe8899b91720639"
format_stamp: "Formatted at 2026-08-12 06:36:57 on dist-test-slave-gjrz"
I20260812 06:36:57.982371 17818 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.000s
I20260812 06:36:57.984402 17847 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:57.985288 17818 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:36:57.985397 17818 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-1/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-1/wal
uuid: "97e6f35670894884bbe8899b91720639"
format_stamp: "Formatted at 2026-08-12 06:36:57 on dist-test-slave-gjrz"
I20260812 06:36:57.985476 17818 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:58.000751 17818 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:58.003010 17818 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:58.003170 17818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:58.010142 17937 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.125:41489 every 8 connection(s)
I20260812 06:36:58.010142 17818 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.125:41489
I20260812 06:36:58.011111 17818 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
I20260812 06:36:58.013816 17939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:58.017105 17939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } has no permanent_uuid. Determining permanent_uuid...
I20260812 06:36:58.018057 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 17818
I20260812 06:36:58.018167 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-1/wal/instance
Aug 12 06:36:58 dist-test-slave-gjrz krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516617, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.22.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.124@KRBTEST.COM" created.
I20260812 06:36:58.029453 17795 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } attempt: 3
I20260812 06:36:58.029500 17939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:36:58 dist-test-slave-gjrz krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516617, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
Aug 12 06:36:58 dist-test-slave-gjrz krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516617, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
I20260812 06:36:58.040675 17939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } has no permanent_uuid. Determining permanent_uuid...
I20260812 06:36:58.040694 17795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab.
W20260812 06:36:58.041805 17850 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.22.124:33657: connect: Connection refused (error 111)
W20260812 06:36:58.041985 17795 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:33657: Network error: Client connection negotiation failed: client connection to 127.13.22.124:33657: connect: Connection refused (error 111)
W20260812 06:36:58.042660 17939 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:33657: Network error: Client connection negotiation failed: client connection to 127.13.22.124:33657: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.13.22.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:58.061658 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.124:33657
--webserver_interface=127.13.22.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:40957,127.13.22.125:41489,127.13.22.124:33657
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 06:36:58.073150 17795 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } attempt: 1
W20260812 06:36:58.074494 17795 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:33657: Network error: Client connection negotiation failed: client connection to 127.13.22.124:33657: connect: Connection refused (error 111)
I20260812 06:36:58.091856 17939 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } attempt: 1
W20260812 06:36:58.093158 17939 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:33657: Network error: Client connection negotiation failed: client connection to 127.13.22.124:33657: connect: Connection refused (error 111)
I20260812 06:36:58.149663 17795 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } attempt: 2
W20260812 06:36:58.151149 17795 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:33657: Network error: Client connection negotiation failed: client connection to 127.13.22.124:33657: connect: Connection refused (error 111)
I20260812 06:36:58.161331 17939 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } attempt: 2
W20260812 06:36:58.162608 17939 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:33657: Network error: Client connection negotiation failed: client connection to 127.13.22.124:33657: connect: Connection refused (error 111)
W20260812 06:36:58.163172 17960 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:58.163388 17960 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:58.163476 17960 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:58.166639 17960 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:36:58.166693 17960 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:58.166714 17960 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:36:58.166743 17960 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:36:58.170866 17960 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:40957,127.13.22.125:41489,127.13.22.124:33657
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.124:33657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.124
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:58.171795 17960 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:58.172559 17960 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:58.177462 17966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:58.177453 17973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:58.177488 17967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:58.177812 17960 server_base.cc:1061] running on GCE node
I20260812 06:36:58.178234 17960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:58.179071 17960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:58.180219 17960 hybrid_clock.cc:648] HybridClock initialized: now 1786516618180210 us; error 43 us; skew 500 ppm
Aug 12 06:36:58 dist-test-slave-gjrz krb5kdc[17629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516618, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:58.182812 17960 init.cc:377] Logged in from keytab as kudu/127.13.22.124@KRBTEST.COM (short username kudu)
I20260812 06:36:58.193545 17960 webserver.cc:533] Webserver started at https://127.13.22.124:44557/ using document root <none> and password file <none>
I20260812 06:36:58.194125 17960 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:58.194180 17960 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:58.194372 17960 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:58.196040 17960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-2/data/instance:
uuid: "20632bf5bd8e488a919f4e49a3eddf5d"
format_stamp: "Formatted at 2026-08-12 06:36:58 on dist-test-slave-gjrz"
I20260812 06:36:58.196476 17960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-2/wal/instance:
uuid: "20632bf5bd8e488a919f4e49a3eddf5d"
format_stamp: "Formatted at 2026-08-12 06:36:58 on dist-test-slave-gjrz"
I20260812 06:36:58.199756 17960 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 06:36:58.201774 17991 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:58.202767 17960 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:36:58.202888 17960 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-2/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-2/wal
uuid: "20632bf5bd8e488a919f4e49a3eddf5d"
format_stamp: "Formatted at 2026-08-12 06:36:58 on dist-test-slave-gjrz"
I20260812 06:36:58.202970 17960 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:58.218199 17960 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:58.220458 17960 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:58.220610 17960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:58.227603 17960 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.124:33657
I20260812 06:36:58.227615 18096 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.124:33657 every 8 connection(s)
I20260812 06:36:58.228545 17960 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
I20260812 06:36:58.231251 18099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:36:58.234512 18099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } has no permanent_uuid. Determining permanent_uuid...
I20260812 06:36:58.236213 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 17960
I20260812 06:36:58.236338 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/master-2/wal/instance
Aug 12 06:36:58 dist-test-slave-gjrz krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
Aug 12 06:36:58 dist-test-slave-gjrz krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516618, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:36:58.247601 18099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:36:58 dist-test-slave-gjrz krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
I20260812 06:36:58.250308 17795 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } attempt: 3
Aug 12 06:36:58 dist-test-slave-gjrz krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516618, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
I20260812 06:36:58.253904 17939 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } attempt: 3
Aug 12 06:36:58 dist-test-slave-gjrz krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516617, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:36:58.256444 18099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:36:58 dist-test-slave-gjrz krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516617, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
Aug 12 06:36:58 dist-test-slave-gjrz krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
Aug 12 06:36:58 dist-test-slave-gjrz krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516618, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:36:58.264642 17795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2: Bootstrap starting.
I20260812 06:36:58.267243 17795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:58.268175 17795 log.cc:826] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:58.268718 18099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d: Bootstrap starting.
I20260812 06:36:58.269260 17939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639: Bootstrap starting.
I20260812 06:36:58.270074 17795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2: No bootstrap required, opened a new log
I20260812 06:36:58.270967 18099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:58.271066 17939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639: Neither blocks nor log segments found. Creating new log.
I20260812 06:36:58.271811 18099 log.cc:826] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:58.271909 17939 log.cc:826] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639: Log is configured to *not* fsync() on all Append() calls
I20260812 06:36:58.273578 17939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639: No bootstrap required, opened a new log
I20260812 06:36:58.273846 17795 raft_consensus.cc:359] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ea9288ee8848e7a3ffe866d561d3f2" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } } peers { permanent_uuid: "97e6f35670894884bbe8899b91720639" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } } peers { permanent_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } }
I20260812 06:36:58.274031 17795 raft_consensus.cc:385] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:58.274101 17795 raft_consensus.cc:740] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54ea9288ee8848e7a3ffe866d561d3f2, State: Initialized, Role: FOLLOWER
I20260812 06:36:58.274533 17795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ea9288ee8848e7a3ffe866d561d3f2" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } } peers { permanent_uuid: "97e6f35670894884bbe8899b91720639" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } } peers { permanent_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } }
I20260812 06:36:58.274941 18128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ea9288ee8848e7a3ffe866d561d3f2" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } } peers { permanent_uuid: "97e6f35670894884bbe8899b91720639" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } } peers { permanent_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } } }
I20260812 06:36:58.275051 18128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:58.275701 17795 sys_catalog.cc:565] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:36:58.276371 18099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d: No bootstrap required, opened a new log
I20260812 06:36:58.276616 17939 raft_consensus.cc:359] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ea9288ee8848e7a3ffe866d561d3f2" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } } peers { permanent_uuid: "97e6f35670894884bbe8899b91720639" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } } peers { permanent_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } }
I20260812 06:36:58.276868 17939 raft_consensus.cc:385] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:58.276940 17939 raft_consensus.cc:740] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e6f35670894884bbe8899b91720639, State: Initialized, Role: FOLLOWER
I20260812 06:36:58.277398 17939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ea9288ee8848e7a3ffe866d561d3f2" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } } peers { permanent_uuid: "97e6f35670894884bbe8899b91720639" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } } peers { permanent_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } }
I20260812 06:36:58.278918 17939 sys_catalog.cc:565] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:36:58.279348 18099 raft_consensus.cc:359] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ea9288ee8848e7a3ffe866d561d3f2" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } } peers { permanent_uuid: "97e6f35670894884bbe8899b91720639" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } } peers { permanent_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } }
I20260812 06:36:58.279620 18099 raft_consensus.cc:385] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:36:58.279701 18099 raft_consensus.cc:740] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20632bf5bd8e488a919f4e49a3eddf5d, State: Initialized, Role: FOLLOWER
I20260812 06:36:58.280103 18099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ea9288ee8848e7a3ffe866d561d3f2" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } } peers { permanent_uuid: "97e6f35670894884bbe8899b91720639" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } } peers { permanent_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } }
W20260812 06:36:58.280732 18146 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:36:58.280851 18146 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:36:58.281180 18099 sys_catalog.cc:565] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:36:58.283855 18168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ea9288ee8848e7a3ffe866d561d3f2" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } } peers { permanent_uuid: "97e6f35670894884bbe8899b91720639" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } } peers { permanent_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } } }
I20260812 06:36:58.284067 18168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:58.282017 17657 master_runner.cc:187] Error getting master registration for 127.13.22.125:41489: OK, instance_id { permanent_uuid: "97e6f35670894884bbe8899b91720639" instance_seqno: 1786516618008198 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 06:36:58.284497 17657 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 06:36:58.288331 18176 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:36:58.288453 18176 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:36:58.289296 18142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ea9288ee8848e7a3ffe866d561d3f2" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } } peers { permanent_uuid: "97e6f35670894884bbe8899b91720639" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } } peers { permanent_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } } }
I20260812 06:36:58.289456 18142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 06:36:58.289921 17708 tablet.cc:2406] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 06:36:58.290936 17960 master_runner.cc:264] No leader master found from master 20632bf5bd8e488a919f4e49a3eddf5d
I20260812 06:36:58.290997 17960 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260812 06:36:58.291079 17818 master_runner.cc:264] No leader master found from master 97e6f35670894884bbe8899b91720639
I20260812 06:36:58.291145 17818 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 06:36:58.293718 18192 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:36:58.293845 18192 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:36:58.295195 18168 raft_consensus.cc:493] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:36:58.295370 18168 raft_consensus.cc:515] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ea9288ee8848e7a3ffe866d561d3f2" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } } peers { permanent_uuid: "97e6f35670894884bbe8899b91720639" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } } peers { permanent_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } }
I20260812 06:36:58.296271 18168 leader_election.cc:290] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54ea9288ee8848e7a3ffe866d561d3f2 (127.13.22.126:40957), 97e6f35670894884bbe8899b91720639 (127.13.22.125:41489)
I20260812 06:36:58.297137 17737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54ea9288ee8848e7a3ffe866d561d3f2" is_pre_election: true
I20260812 06:36:58.297375 17737 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20632bf5bd8e488a919f4e49a3eddf5d in term 0.
I20260812 06:36:58.297941 17894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e6f35670894884bbe8899b91720639" is_pre_election: true
I20260812 06:36:58.298182 17894 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20632bf5bd8e488a919f4e49a3eddf5d in term 0.
I20260812 06:36:58.298597 17998 leader_election.cc:304] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 20632bf5bd8e488a919f4e49a3eddf5d, 54ea9288ee8848e7a3ffe866d561d3f2, 97e6f35670894884bbe8899b91720639; no voters: 
I20260812 06:36:58.298840 18168 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 06:36:58.298918 18168 raft_consensus.cc:493] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 06:36:58.298959 18168 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:58.300086 18168 raft_consensus.cc:515] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ea9288ee8848e7a3ffe866d561d3f2" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } } peers { permanent_uuid: "97e6f35670894884bbe8899b91720639" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } } peers { permanent_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } }
I20260812 06:36:58.300486 18168 leader_election.cc:290] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [CANDIDATE]: Term 1 election: Requested vote from peers 54ea9288ee8848e7a3ffe866d561d3f2 (127.13.22.126:40957), 97e6f35670894884bbe8899b91720639 (127.13.22.125:41489)
I20260812 06:36:58.301211 17737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54ea9288ee8848e7a3ffe866d561d3f2"
I20260812 06:36:58.301329 17737 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:58.302444 17737 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20632bf5bd8e488a919f4e49a3eddf5d in term 1.
I20260812 06:36:58.302789 17998 leader_election.cc:304] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20632bf5bd8e488a919f4e49a3eddf5d, 54ea9288ee8848e7a3ffe866d561d3f2; no voters: 
I20260812 06:36:58.302805 17894 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e6f35670894884bbe8899b91720639"
I20260812 06:36:58.302908 17894 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:36:58.303004 18168 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 06:36:58.303181 18168 raft_consensus.cc:697] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [term 1 LEADER]: Becoming Leader. State: Replica: 20632bf5bd8e488a919f4e49a3eddf5d, State: Running, Role: LEADER
I20260812 06:36:58.303486 18168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ea9288ee8848e7a3ffe866d561d3f2" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } } peers { permanent_uuid: "97e6f35670894884bbe8899b91720639" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } } peers { permanent_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } }
I20260812 06:36:58.304071 17894 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20632bf5bd8e488a919f4e49a3eddf5d in term 1.
I20260812 06:36:58.305088 18168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20632bf5bd8e488a919f4e49a3eddf5d. Latest consensus state: current_term: 1 leader_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ea9288ee8848e7a3ffe866d561d3f2" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } } peers { permanent_uuid: "97e6f35670894884bbe8899b91720639" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } } peers { permanent_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } } }
I20260812 06:36:58.305238 18168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [sys.catalog]: This master's current role is: LEADER
I20260812 06:36:58.308796 18199 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 06:36:58.311010 18199 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 06:36:58.318312 17894 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639 [term 1 FOLLOWER]: Refusing update from remote peer 20632bf5bd8e488a919f4e49a3eddf5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:36:58.318939 18168 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "97e6f35670894884bbe8899b91720639" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:36:58.322593 18142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20632bf5bd8e488a919f4e49a3eddf5d. Latest consensus state: current_term: 1 leader_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ea9288ee8848e7a3ffe866d561d3f2" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } } peers { permanent_uuid: "97e6f35670894884bbe8899b91720639" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } } peers { permanent_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } } }
I20260812 06:36:58.322696 18142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:58.325842 17737 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2 [term 1 FOLLOWER]: Refusing update from remote peer 20632bf5bd8e488a919f4e49a3eddf5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:36:58.326433 18168 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "54ea9288ee8848e7a3ffe866d561d3f2" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:36:58.329720 18128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20632bf5bd8e488a919f4e49a3eddf5d. Latest consensus state: current_term: 1 leader_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ea9288ee8848e7a3ffe866d561d3f2" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } } peers { permanent_uuid: "97e6f35670894884bbe8899b91720639" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } } peers { permanent_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } } }
I20260812 06:36:58.329818 18128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:58.339777 18128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ea9288ee8848e7a3ffe866d561d3f2" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } } peers { permanent_uuid: "97e6f35670894884bbe8899b91720639" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } } peers { permanent_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } } }
I20260812 06:36:58.339876 18128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:58.340963 18197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ea9288ee8848e7a3ffe866d561d3f2" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } } peers { permanent_uuid: "97e6f35670894884bbe8899b91720639" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } } peers { permanent_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } } }
I20260812 06:36:58.341075 18197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [sys.catalog]: This master's current role is: LEADER
I20260812 06:36:58.341594 18202 mvcc.cc:204] Tried to move back new op lower bound from 7317572068625113088 to 7317572068574248960. Current Snapshot: MvccSnapshot[applied={T|T < 7317572068625113088}]
I20260812 06:36:58.344347 18168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ea9288ee8848e7a3ffe866d561d3f2" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } } peers { permanent_uuid: "97e6f35670894884bbe8899b91720639" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } } peers { permanent_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } } }
I20260812 06:36:58.344444 18168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d [sys.catalog]: This master's current role is: LEADER
I20260812 06:36:58.345562 18142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54ea9288ee8848e7a3ffe866d561d3f2" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 40957 } } peers { permanent_uuid: "97e6f35670894884bbe8899b91720639" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41489 } } peers { permanent_uuid: "20632bf5bd8e488a919f4e49a3eddf5d" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 33657 } } }
I20260812 06:36:58.345664 18142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:36:58.348507 18210 mvcc.cc:204] Tried to move back new op lower bound from 7317572068625113088 to 7317572068574248960. Current Snapshot: MvccSnapshot[applied={T|T < 7317572068625113088 or (T in {7317572068625113088})}]
I20260812 06:36:58.348781 18205 mvcc.cc:204] Tried to move back new op lower bound from 7317572068625113088 to 7317572068574248960. Current Snapshot: MvccSnapshot[applied={T|T < 7317572068625113088}]
I20260812 06:36:58.351698 18199 catalog_manager.cc:1383] Generated new cluster ID: 11d4b22faf774075bc89287bda501766
I20260812 06:36:58.351758 18199 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 06:36:58.384888 18199 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 06:36:58.386061 18199 catalog_manager.cc:1540] Loading token signing keys...
I20260812 06:36:58.404323 18199 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 20632bf5bd8e488a919f4e49a3eddf5d: Generated new TSK 0
I20260812 06:36:58.405040 18199 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:36:58.472169 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.22.65:0
--local_ip_for_outbound_sockets=127.13.22.65
--webserver_interface=127.13.22.65
--webserver_port=0
--tserver_master_addrs=127.13.22.126:40957,127.13.22.125:41489,127.13.22.124:33657
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 06:36:58.596184 18232 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:36:58.596370 18232 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:36:58.596457 18232 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:36:58.599635 18232 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:36:58.599766 18232 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.22.65
I20260812 06:36:58.603924 18232 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.65
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--tserver_master_addrs=127.13.22.126:40957,127.13.22.125:41489,127.13.22.124:33657
--never_fsync=true
--heap_profile_path=/tmp/kudu.18232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.22.65

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:36:58.604791 18232 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:36:58.605576 18232 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:36:58.611572 18251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:58.611572 18246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:36:58.611596 18247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:36:58.611889 18232 server_base.cc:1061] running on GCE node
I20260812 06:36:58.612311 18232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:36:58.612917 18232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:36:58.614102 18232 hybrid_clock.cc:648] HybridClock initialized: now 1786516618614089 us; error 27 us; skew 500 ppm
Aug 12 06:36:58 dist-test-slave-gjrz krb5kdc[17629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516618, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:36:58.617309 18232 init.cc:377] Logged in from keytab as kudu/127.13.22.65@KRBTEST.COM (short username kudu)
I20260812 06:36:58.634025 18232 webserver.cc:533] Webserver started at https://127.13.22.65:35337/ using document root <none> and password file <none>
I20260812 06:36:58.634611 18232 fs_manager.cc:362] Metadata directory not provided
I20260812 06:36:58.634673 18232 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:36:58.634876 18232 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:36:58.636535 18232 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/ts-0/data/instance:
uuid: "86ca8570acb94b00b1ba4eaaf2eb0eb5"
format_stamp: "Formatted at 2026-08-12 06:36:58 on dist-test-slave-gjrz"
I20260812 06:36:58.636987 18232 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/ts-0/wal/instance:
uuid: "86ca8570acb94b00b1ba4eaaf2eb0eb5"
format_stamp: "Formatted at 2026-08-12 06:36:58 on dist-test-slave-gjrz"
I20260812 06:36:58.640235 18232 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.000s	sys 0.004s
I20260812 06:36:58.642316 18265 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:58.643366 18232 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 06:36:58.643472 18232 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/ts-0/wal
uuid: "86ca8570acb94b00b1ba4eaaf2eb0eb5"
format_stamp: "Formatted at 2026-08-12 06:36:58 on dist-test-slave-gjrz"
I20260812 06:36:58.643550 18232 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:36:58.666877 18232 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:36:58.669375 18232 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:36:58.669545 18232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:36:58.670107 18232 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:36:58.671075 18232 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:36:58.671123 18232 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:58.671176 18232 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:36:58.671206 18232 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:36:58.680922 18232 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.65:43715
I20260812 06:36:58.680943 18464 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.65:43715 every 8 connection(s)
I20260812 06:36:58.681820 18232 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
I20260812 06:36:58.690513 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 18232
I20260812 06:36:58.690637 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/minicluster-data/ts-0/wal/instance
Aug 12 06:36:58 dist-test-slave-gjrz krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516618, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
Aug 12 06:36:58 dist-test-slave-gjrz krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516618, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
Aug 12 06:36:58 dist-test-slave-gjrz krb5kdc[17629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516618, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:36:58.698076 18466 heartbeater.cc:344] Connected to a master server at 127.13.22.126:40957
I20260812 06:36:58.698287 18466 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:58.698698 18466 heartbeater.cc:507] Master 127.13.22.126:40957 requested a full tablet report, sending...
I20260812 06:36:58.699633 18467 heartbeater.cc:344] Connected to a master server at 127.13.22.125:41489
I20260812 06:36:58.699856 18467 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:58.699864 18465 heartbeater.cc:344] Connected to a master server at 127.13.22.124:33657
I20260812 06:36:58.699975 18465 heartbeater.cc:461] Registering TS with master...
I20260812 06:36:58.700070 18467 heartbeater.cc:507] Master 127.13.22.125:41489 requested a full tablet report, sending...
I20260812 06:36:58.700119 17724 ts_manager.cc:194] Registered new tserver with Master: 86ca8570acb94b00b1ba4eaaf2eb0eb5 (127.13.22.65:43715)
I20260812 06:36:58.700162 18465 heartbeater.cc:507] Master 127.13.22.124:33657 requested a full tablet report, sending...
I20260812 06:36:58.701102 18020 ts_manager.cc:194] Registered new tserver with Master: 86ca8570acb94b00b1ba4eaaf2eb0eb5 (127.13.22.65:43715)
I20260812 06:36:58.701102 17877 ts_manager.cc:194] Registered new tserver with Master: 86ca8570acb94b00b1ba4eaaf2eb0eb5 (127.13.22.65:43715)
I20260812 06:36:58.702356 18020 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.22.65@KRBTEST.COM'} at 127.13.22.65:55277
I20260812 06:36:58.711675 13401 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
time=2026-08-12T06:36:58.771Z level=INFO source=main.go:1678 msg="updated GOGC" old=100 new=75
time=2026-08-12T06:36:58.771Z level=INFO source=main.go:744 msg="Leaving GOMAXPROCS=4: CPU quota undefined" component=automaxprocs
time=2026-08-12T06:36:58.772Z level=INFO source=memlimit.go:198 msg="GOMEMLIMIT is updated" component=automemlimit package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=14142549196 previous=9223372036854775807
time=2026-08-12T06:36:58.772Z level=INFO source=main.go:851 msg="Starting Prometheus Server" mode=server version="(version=3.11.2, branch=HEAD, revision=f0f0fdd679dcd6df320b0558b20919f7cd44c407)"
time=2026-08-12T06:36:58.772Z level=INFO source=main.go:856 msg="operational information" build_context="(go=go1.26.2, platform=linux/amd64, user=root@d2684485c347, date=20260413-11:53:40, tags=netgo,builtinassets)" host_details="(Linux 5.4.0-1157-gcp #166~18.04.1-Ubuntu SMP Wed Jan 21 01:57:46 UTC 2026 x86_64 dist-test-slave-gjrz (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)"
time=2026-08-12T06:36:59.009Z level=INFO source=web.go:710 msg="Start listening for connections" component=web address=127.0.0.1:0
time=2026-08-12T06:36:59.010Z level=INFO source=main.go:1410 msg="Starting TSDB ..."
time=2026-08-12T06:36:59.013Z level=INFO source=tls_config.go:354 msg="Listening on" component=web address=127.0.0.1:35965
time=2026-08-12T06:36:59.013Z level=INFO source=tls_config.go:357 msg="TLS is disabled." component=web http2=false address=127.0.0.1:35965
time=2026-08-12T06:36:59.019Z level=INFO source=head.go:698 msg="Replaying on-disk memory mappable chunks if any" component=tsdb
time=2026-08-12T06:36:59.019Z level=INFO source=head.go:784 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=1.683µs
time=2026-08-12T06:36:59.019Z level=INFO source=head.go:792 msg="Replaying WAL, this may take a while" component=tsdb
time=2026-08-12T06:36:59.019Z level=INFO source=head.go:865 msg="WAL segment loaded" component=tsdb segment=0 maxSegment=0 duration=250.569µs
time=2026-08-12T06:36:59.019Z level=INFO source=head.go:902 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=54.54µs wal_replay_duration=284.046µs wbl_replay_duration=156ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=1.683µs total_replay_duration=376.495µs
time=2026-08-12T06:36:59.022Z level=INFO source=main.go:1431 msg="filesystem information" fs_type=OVERLAYFS_SUPER_MAGIC
time=2026-08-12T06:36:59.022Z level=INFO source=main.go:1434 msg="TSDB started"
time=2026-08-12T06:36:59.022Z level=INFO source=main.go:1632 msg="Loading configuration file" filename=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/prometheus.yml
time=2026-08-12T06:36:59.022Z level=INFO source=main.go:1048 msg="TSDB retention updated" duration=15d size=0B percentage=0
time=2026-08-12T06:36:59.024Z level=INFO source=main.go:1671 msg="Completed loading of configuration file" db_storage=45.574µs remote_storage=2.158µs web_handler=447ns query_engine=1.803µs scrape=1.139735ms scrape_sd=142.606µs notify=1.89µs notify_sd=1.137µs rules=1.99µs tracing=5.577µs filename=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed.1786516559891118-13401-0/prometheus.yml totalDuration=1.702114ms
time=2026-08-12T06:36:59.024Z level=INFO source=main.go:1395 msg="Server is ready to receive web requests."
time=2026-08-12T06:36:59.024Z level=INFO source=manager.go:209 msg="Starting rule manager..." component="rule manager"
I20260812 06:36:59.282260 18146 catalog_manager.cc:1295] Loaded cluster ID: 11d4b22faf774075bc89287bda501766
I20260812 06:36:59.282346 18146 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2: loading cluster ID for follower catalog manager: success
I20260812 06:36:59.284148 18146 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2: acquiring CA information for follower catalog manager: success
I20260812 06:36:59.285211 18146 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 54ea9288ee8848e7a3ffe866d561d3f2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 06:36:59.289939 18176 catalog_manager.cc:1295] Loaded cluster ID: 11d4b22faf774075bc89287bda501766
I20260812 06:36:59.290024 18176 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639: loading cluster ID for follower catalog manager: success
I20260812 06:36:59.292407 18176 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639: acquiring CA information for follower catalog manager: success
I20260812 06:36:59.293828 18176 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 97e6f35670894884bbe8899b91720639: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 06:36:59.709062 18465 heartbeater.cc:499] Master 127.13.22.124:33657 was elected leader, sending a full tablet report...
time=2026-08-12T06:37:05.784Z level=WARN source=main.go:1191 msg="Received an OS signal, exiting gracefully..." signal=terminated
time=2026-08-12T06:37:05.784Z level=INFO source=main.go:1216 msg="Stopping scrape discovery manager..."
time=2026-08-12T06:37:05.784Z level=INFO source=main.go:1230 msg="Stopping notify discovery manager..."
time=2026-08-12T06:37:05.784Z level=INFO source=main.go:1245 msg="Stopping rule manager manager..."
time=2026-08-12T06:37:05.784Z level=INFO source=manager.go:223 msg="Stopping rule manager..." component="rule manager"
time=2026-08-12T06:37:05.784Z level=INFO source=manager.go:239 msg="Rule manager stopped" component="rule manager"
time=2026-08-12T06:37:05.784Z level=INFO source=main.go:1269 msg="Stopping scrape manager..."
time=2026-08-12T06:37:05.784Z level=INFO source=main.go:1212 msg="Scrape discovery manager stopped"
time=2026-08-12T06:37:05.784Z level=INFO source=main.go:1226 msg="Notify discovery manager stopped"
time=2026-08-12T06:37:05.784Z level=INFO source=main.go:1241 msg="Rule manager stopped"
time=2026-08-12T06:37:05.784Z level=INFO source=main.go:1284 msg="Stopping tracing manager..."
time=2026-08-12T06:37:05.784Z level=INFO source=main.go:1369 msg="Stopping reloaders..."
time=2026-08-12T06:37:05.784Z level=INFO source=main.go:1458 msg="Stopping storage..."
time=2026-08-12T06:37:05.784Z level=INFO source=main.go:1261 msg="Scrape manager stopped"
time=2026-08-12T06:37:05.784Z level=INFO source=main.go:1280 msg="Tracing manager stopped"
time=2026-08-12T06:37:05.784Z level=INFO source=main.go:1361 msg="Reloaders stopped"
time=2026-08-12T06:37:05.788Z level=INFO source=main.go:1536 msg="Stopping web handler..."
time=2026-08-12T06:37:05.788Z level=INFO source=main.go:1559 msg="Stopping notifier manager..."
time=2026-08-12T06:37:05.788Z level=INFO source=manager.go:321 msg="Stopping notification manager..." component=notifier
time=2026-08-12T06:37:05.788Z level=INFO source=main.go:1555 msg="Notifier manager stopped"
time=2026-08-12T06:37:05.788Z level=INFO source=main.go:1454 msg="TSDB stopped"
time=2026-08-12T06:37:05.788Z level=INFO source=main.go:1532 msg="Web handler stopped"
time=2026-08-12T06:37:05.788Z level=INFO source=main.go:1570 msg="See you next time!"
I20260812 06:37:05.794560 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 18232
I20260812 06:37:05.812577 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 17657
I20260812 06:37:05.825062 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 17818
I20260812 06:37:05.841547 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 17960
2026-08-12T06:37:05Z chronyd exiting
[       OK ] PrometheusTokenITest.TestCorrectTokenSdAndScrapeSucceed (10415 ms)
[ RUN      ] PrometheusTokenITest.TestWrongTokenSdFails
Loading random data
Initializing database '/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 06:37:05 dist-test-slave-gjrz krb5kdc[18538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 06:37:05 dist-test-slave-gjrz krb5kdc[18538](info): set up 2 sockets
Aug 12 06:37:05 dist-test-slave-gjrz krb5kdc[18538](info): commencing operation
krb5kdc: starting...
W20260812 06:37:07.916580 13401 mini_kdc.cc:121] Time spent starting KDC: real 2.047s	user 0.000s	sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/test-user.keytab.
Aug 12 06:37:07 dist-test-slave-gjrz krb5kdc[18538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM: 
2026-08-12T06:37:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T06:37:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:37:08.044185 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.126:45587
--webserver_interface=127.13.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:38445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:45587,127.13.22.125:41221,127.13.22.124:40481
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 06:37:08.143782 18574 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:08.143997 18574 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:08.144079 18574 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:08.147316 18574 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:37:08.147373 18574 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:08.147403 18574 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:37:08.147425 18574 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:37:08.151571 18574 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:38445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:45587,127.13.22.125:41221,127.13.22.124:40481
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.126:45587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.126
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:08.152534 18574 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:08.153329 18574 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:08.158378 18585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:08.158366 18588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:08.158390 18591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:08.158491 18574 server_base.cc:1061] running on GCE node
I20260812 06:37:08.159106 18574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:08.159929 18574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:08.161075 18574 hybrid_clock.cc:648] HybridClock initialized: now 1786516628161063 us; error 25 us; skew 500 ppm
Aug 12 06:37:08 dist-test-slave-gjrz krb5kdc[18538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516628, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:37:08.163539 18574 init.cc:377] Logged in from keytab as kudu/127.13.22.126@KRBTEST.COM (short username kudu)
I20260812 06:37:08.174146 18574 webserver.cc:533] Webserver started at https://127.13.22.126:38217/ using document root <none> and password file <none>
I20260812 06:37:08.174767 18574 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:08.174830 18574 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:08.175089 18574 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:08.176827 18574 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/data/instance:
uuid: "88935b860e7548029feb03eb0b003ae9"
format_stamp: "Formatted at 2026-08-12 06:37:08 on dist-test-slave-gjrz"
I20260812 06:37:08.177289 18574 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/wal/instance:
uuid: "88935b860e7548029feb03eb0b003ae9"
format_stamp: "Formatted at 2026-08-12 06:37:08 on dist-test-slave-gjrz"
I20260812 06:37:08.180604 18574 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 06:37:08.182585 18617 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:08.183601 18574 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:37:08.183709 18574 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/wal
uuid: "88935b860e7548029feb03eb0b003ae9"
format_stamp: "Formatted at 2026-08-12 06:37:08 on dist-test-slave-gjrz"
I20260812 06:37:08.183791 18574 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:08.206004 18574 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:08.208467 18574 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:08.208609 18574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:08.215499 18710 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.126:45587 every 8 connection(s)
I20260812 06:37:08.215509 18574 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.126:45587
I20260812 06:37:08.216512 18574 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
I20260812 06:37:08.218561 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 18574
I20260812 06:37:08.218660 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/wal/instance
I20260812 06:37:08.219318 18716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:08.222841 18716 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.22.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.125@KRBTEST.COM" created.
Aug 12 06:37:08 dist-test-slave-gjrz krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516628, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:37:08.236647 18716 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } has no permanent_uuid. Determining permanent_uuid...
W20260812 06:37:08.237344 18619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.22.125:41221: connect: Connection refused (error 111)
W20260812 06:37:08.238194 18716 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:41221: Network error: Client connection negotiation failed: client connection to 127.13.22.125:41221: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:37:08.261768 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.125:41221
--webserver_interface=127.13.22.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:38445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:45587,127.13.22.125:41221,127.13.22.124:40481
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 06:37:08.287389 18716 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } attempt: 1
W20260812 06:37:08.288812 18716 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:41221: Network error: Client connection negotiation failed: client connection to 127.13.22.125:41221: connect: Connection refused (error 111)
I20260812 06:37:08.356972 18716 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } attempt: 2
W20260812 06:37:08.358443 18716 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:41221: Network error: Client connection negotiation failed: client connection to 127.13.22.125:41221: connect: Connection refused (error 111)
W20260812 06:37:08.362008 18737 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:08.362183 18737 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:08.362264 18737 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:08.365463 18737 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:37:08.365517 18737 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:08.365554 18737 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:37:08.365588 18737 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:37:08.370250 18737 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:38445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:45587,127.13.22.125:41221,127.13.22.124:40481
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.125:41221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.125
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:08.371330 18737 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:08.372110 18737 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:08.377030 18752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:08.377049 18748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:08.377077 18747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:08.377225 18737 server_base.cc:1061] running on GCE node
I20260812 06:37:08.377770 18737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:08.378571 18737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:08.379709 18737 hybrid_clock.cc:648] HybridClock initialized: now 1786516628379696 us; error 30 us; skew 500 ppm
Aug 12 06:37:08 dist-test-slave-gjrz krb5kdc[18538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516628, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:37:08.382210 18737 init.cc:377] Logged in from keytab as kudu/127.13.22.125@KRBTEST.COM (short username kudu)
I20260812 06:37:08.392773 18737 webserver.cc:533] Webserver started at https://127.13.22.125:44103/ using document root <none> and password file <none>
I20260812 06:37:08.393340 18737 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:08.393390 18737 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:08.393582 18737 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:08.395139 18737 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/data/instance:
uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0"
format_stamp: "Formatted at 2026-08-12 06:37:08 on dist-test-slave-gjrz"
I20260812 06:37:08.395635 18737 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/wal/instance:
uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0"
format_stamp: "Formatted at 2026-08-12 06:37:08 on dist-test-slave-gjrz"
I20260812 06:37:08.398729 18737 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.000s
I20260812 06:37:08.400689 18767 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:08.401584 18737 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:37:08.401692 18737 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/wal
uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0"
format_stamp: "Formatted at 2026-08-12 06:37:08 on dist-test-slave-gjrz"
I20260812 06:37:08.401772 18737 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:08.428723 18737 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:08.431021 18737 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:08.431166 18737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:08.438138 18854 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.125:41221 every 8 connection(s)
I20260812 06:37:08.438140 18737 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.125:41221
I20260812 06:37:08.439136 18737 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
I20260812 06:37:08.441869 18856 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:08.445256 18856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } has no permanent_uuid. Determining permanent_uuid...
I20260812 06:37:08.446337 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 18737
I20260812 06:37:08.446437 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/wal/instance
I20260812 06:37:08.449612 18716 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } attempt: 3
Aug 12 06:37:08 dist-test-slave-gjrz krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516628, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.22.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.124@KRBTEST.COM" created.
Aug 12 06:37:08 dist-test-slave-gjrz krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516628, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
I20260812 06:37:08.458021 18856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:37:08 dist-test-slave-gjrz krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516628, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
I20260812 06:37:08.463181 18716 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } has no permanent_uuid. Determining permanent_uuid...
W20260812 06:37:08.464462 18716 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:40481: Network error: Client connection negotiation failed: client connection to 127.13.22.124:40481: connect: Connection refused (error 111)
I20260812 06:37:08.466166 18856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } has no permanent_uuid. Determining permanent_uuid...
W20260812 06:37:08.466768 18771 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.22.124:40481: connect: Connection refused (error 111)
W20260812 06:37:08.467540 18856 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:40481: Network error: Client connection negotiation failed: client connection to 127.13.22.124:40481: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:37:08.492002 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.124:40481
--webserver_interface=127.13.22.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:38445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:45587,127.13.22.125:41221,127.13.22.124:40481
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 06:37:08.495626 18716 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } attempt: 1
W20260812 06:37:08.497045 18716 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:40481: Network error: Client connection negotiation failed: client connection to 127.13.22.124:40481: connect: Connection refused (error 111)
I20260812 06:37:08.516733 18856 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } attempt: 1
W20260812 06:37:08.518018 18856 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:40481: Network error: Client connection negotiation failed: client connection to 127.13.22.124:40481: connect: Connection refused (error 111)
I20260812 06:37:08.572245 18716 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } attempt: 2
W20260812 06:37:08.573705 18716 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:40481: Network error: Client connection negotiation failed: client connection to 127.13.22.124:40481: connect: Connection refused (error 111)
I20260812 06:37:08.586190 18856 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } attempt: 2
W20260812 06:37:08.587590 18856 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:40481: Network error: Client connection negotiation failed: client connection to 127.13.22.124:40481: connect: Connection refused (error 111)
W20260812 06:37:08.593665 18872 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:08.593840 18872 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:08.593917 18872 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:08.597170 18872 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:37:08.597224 18872 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:08.597244 18872 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:37:08.597258 18872 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:37:08.601349 18872 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:38445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:45587,127.13.22.125:41221,127.13.22.124:40481
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.124:40481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.124
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:08.602253 18872 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:08.603009 18872 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:08.608068 18884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:08.608057 18886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:08.608047 18889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:08.608294 18872 server_base.cc:1061] running on GCE node
I20260812 06:37:08.608814 18872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:08.609570 18872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:08.610718 18872 hybrid_clock.cc:648] HybridClock initialized: now 1786516628610703 us; error 30 us; skew 500 ppm
Aug 12 06:37:08 dist-test-slave-gjrz krb5kdc[18538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516628, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:37:08.613291 18872 init.cc:377] Logged in from keytab as kudu/127.13.22.124@KRBTEST.COM (short username kudu)
I20260812 06:37:08.623962 18872 webserver.cc:533] Webserver started at https://127.13.22.124:32929/ using document root <none> and password file <none>
I20260812 06:37:08.624531 18872 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:08.624576 18872 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:08.624740 18872 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:08.626296 18872 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/data/instance:
uuid: "3b2c92192e044a8f94c70e12111b83f1"
format_stamp: "Formatted at 2026-08-12 06:37:08 on dist-test-slave-gjrz"
I20260812 06:37:08.626693 18872 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/wal/instance:
uuid: "3b2c92192e044a8f94c70e12111b83f1"
format_stamp: "Formatted at 2026-08-12 06:37:08 on dist-test-slave-gjrz"
I20260812 06:37:08.629794 18872 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20260812 06:37:08.631750 18911 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:08.632637 18872 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:37:08.632740 18872 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/wal
uuid: "3b2c92192e044a8f94c70e12111b83f1"
format_stamp: "Formatted at 2026-08-12 06:37:08 on dist-test-slave-gjrz"
I20260812 06:37:08.632810 18872 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:08.641454 18872 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:08.643849 18872 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:08.643980 18872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:08.650571 19001 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.124:40481 every 8 connection(s)
I20260812 06:37:08.650575 18872 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.124:40481
I20260812 06:37:08.651508 18872 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
I20260812 06:37:08.654067 19003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:08.656670 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 18872
I20260812 06:37:08.656792 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/wal/instance
I20260812 06:37:08.657547 19003 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:37:08 dist-test-slave-gjrz krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
Aug 12 06:37:08 dist-test-slave-gjrz krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516628, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:37:08.669690 19003 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:37:08 dist-test-slave-gjrz krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
I20260812 06:37:08.672885 18716 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } attempt: 3
Aug 12 06:37:08 dist-test-slave-gjrz krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516628, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
I20260812 06:37:08.677690 19003 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } has no permanent_uuid. Determining permanent_uuid...
I20260812 06:37:08.678742 18856 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } attempt: 3
Aug 12 06:37:08 dist-test-slave-gjrz krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516628, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
Aug 12 06:37:08 dist-test-slave-gjrz krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
Aug 12 06:37:08 dist-test-slave-gjrz krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516628, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
Aug 12 06:37:08 dist-test-slave-gjrz krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516628, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:37:08.688865 18716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9: Bootstrap starting.
I20260812 06:37:08.688989 19003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1: Bootstrap starting.
I20260812 06:37:08.691421 18856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0: Bootstrap starting.
I20260812 06:37:08.691527 19003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:08.691529 18716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:08.692413 19003 log.cc:826] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:08.692405 18716 log.cc:826] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:08.693666 18856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:08.693908 18716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9: No bootstrap required, opened a new log
I20260812 06:37:08.694278 18856 log.cc:826] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:08.695967 18856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0: No bootstrap required, opened a new log
I20260812 06:37:08.696283 19003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1: No bootstrap required, opened a new log
I20260812 06:37:08.696390 18716 raft_consensus.cc:359] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88935b860e7548029feb03eb0b003ae9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } } peers { permanent_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } } peers { permanent_uuid: "3b2c92192e044a8f94c70e12111b83f1" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } }
I20260812 06:37:08.696557 18716 raft_consensus.cc:385] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:37:08.696625 18716 raft_consensus.cc:740] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88935b860e7548029feb03eb0b003ae9, State: Initialized, Role: FOLLOWER
I20260812 06:37:08.696996 18716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88935b860e7548029feb03eb0b003ae9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } } peers { permanent_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } } peers { permanent_uuid: "3b2c92192e044a8f94c70e12111b83f1" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } }
I20260812 06:37:08.697392 19020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88935b860e7548029feb03eb0b003ae9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } } peers { permanent_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } } peers { permanent_uuid: "3b2c92192e044a8f94c70e12111b83f1" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } } }
I20260812 06:37:08.697556 19020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:37:08.698053 18716 sys_catalog.cc:565] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:37:08.699088 18856 raft_consensus.cc:359] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88935b860e7548029feb03eb0b003ae9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } } peers { permanent_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } } peers { permanent_uuid: "3b2c92192e044a8f94c70e12111b83f1" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } }
I20260812 06:37:08.699091 19003 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88935b860e7548029feb03eb0b003ae9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } } peers { permanent_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } } peers { permanent_uuid: "3b2c92192e044a8f94c70e12111b83f1" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } }
I20260812 06:37:08.699256 18856 raft_consensus.cc:385] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:37:08.699256 19003 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:37:08.699348 19003 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b2c92192e044a8f94c70e12111b83f1, State: Initialized, Role: FOLLOWER
I20260812 06:37:08.699357 18856 raft_consensus.cc:740] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040e1cbbcd2340c7a828a9fd5ab4ddc0, State: Initialized, Role: FOLLOWER
I20260812 06:37:08.699739 19003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88935b860e7548029feb03eb0b003ae9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } } peers { permanent_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } } peers { permanent_uuid: "3b2c92192e044a8f94c70e12111b83f1" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } }
I20260812 06:37:08.699751 18856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88935b860e7548029feb03eb0b003ae9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } } peers { permanent_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } } peers { permanent_uuid: "3b2c92192e044a8f94c70e12111b83f1" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } }
I20260812 06:37:08.700201 19042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88935b860e7548029feb03eb0b003ae9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } } peers { permanent_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } } peers { permanent_uuid: "3b2c92192e044a8f94c70e12111b83f1" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } } }
I20260812 06:37:08.700382 19042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:37:08.700662 19040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88935b860e7548029feb03eb0b003ae9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } } peers { permanent_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } } peers { permanent_uuid: "3b2c92192e044a8f94c70e12111b83f1" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } } }
I20260812 06:37:08.700826 19040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:37:08.700831 19003 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:37:08.702075 18856 sys_catalog.cc:565] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:37:08.702625 18574 master_runner.cc:187] Error getting master registration for 127.13.22.125:41221: OK, instance_id { permanent_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" instance_seqno: 1786516628436242 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 06:37:08.704699 18574 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 06:37:08.706334 19043 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:37:08.706416 19043 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 06:37:08.707005 19091 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:37:08.707121 19091 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:37:08.709795 18872 master_runner.cc:264] No leader master found from master 3b2c92192e044a8f94c70e12111b83f1
I20260812 06:37:08.709862 18872 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 06:37:08.709832 18627 tablet.cc:2406] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 06:37:08.709925 18737 master_runner.cc:264] No leader master found from master 040e1cbbcd2340c7a828a9fd5ab4ddc0
I20260812 06:37:08.709980 18737 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 06:37:08.710412 19078 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:37:08.710470 19078 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:37:08.884236 19040 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:37:08.884469 19040 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88935b860e7548029feb03eb0b003ae9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } } peers { permanent_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } } peers { permanent_uuid: "3b2c92192e044a8f94c70e12111b83f1" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } }
I20260812 06:37:08.885560 19040 leader_election.cc:290] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88935b860e7548029feb03eb0b003ae9 (127.13.22.126:45587), 040e1cbbcd2340c7a828a9fd5ab4ddc0 (127.13.22.125:41221)
I20260812 06:37:08.886106 18665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b2c92192e044a8f94c70e12111b83f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88935b860e7548029feb03eb0b003ae9" is_pre_election: true
I20260812 06:37:08.886209 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b2c92192e044a8f94c70e12111b83f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" is_pre_election: true
I20260812 06:37:08.886396 18665 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b2c92192e044a8f94c70e12111b83f1 in term 0.
I20260812 06:37:08.886503 18812 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b2c92192e044a8f94c70e12111b83f1 in term 0.
I20260812 06:37:08.886943 18914 leader_election.cc:304] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 040e1cbbcd2340c7a828a9fd5ab4ddc0, 3b2c92192e044a8f94c70e12111b83f1; no voters: 
I20260812 06:37:08.887228 19040 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 06:37:08.887316 19040 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 06:37:08.887362 19040 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:08.888530 19040 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88935b860e7548029feb03eb0b003ae9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } } peers { permanent_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } } peers { permanent_uuid: "3b2c92192e044a8f94c70e12111b83f1" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } }
I20260812 06:37:08.888912 19040 leader_election.cc:290] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [CANDIDATE]: Term 1 election: Requested vote from peers 88935b860e7548029feb03eb0b003ae9 (127.13.22.126:45587), 040e1cbbcd2340c7a828a9fd5ab4ddc0 (127.13.22.125:41221)
I20260812 06:37:08.889312 18665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b2c92192e044a8f94c70e12111b83f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88935b860e7548029feb03eb0b003ae9"
I20260812 06:37:08.889331 18812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b2c92192e044a8f94c70e12111b83f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0"
I20260812 06:37:08.889421 18665 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:08.889452 18812 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:08.890506 18665 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b2c92192e044a8f94c70e12111b83f1 in term 1.
I20260812 06:37:08.890568 18812 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b2c92192e044a8f94c70e12111b83f1 in term 1.
I20260812 06:37:08.890900 18914 leader_election.cc:304] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 040e1cbbcd2340c7a828a9fd5ab4ddc0, 3b2c92192e044a8f94c70e12111b83f1; no voters: 
I20260812 06:37:08.891084 19040 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 06:37:08.891323 19040 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b2c92192e044a8f94c70e12111b83f1, State: Running, Role: LEADER
I20260812 06:37:08.891657 19040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88935b860e7548029feb03eb0b003ae9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } } peers { permanent_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } } peers { permanent_uuid: "3b2c92192e044a8f94c70e12111b83f1" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } }
I20260812 06:37:08.893360 19040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b2c92192e044a8f94c70e12111b83f1. Latest consensus state: current_term: 1 leader_uuid: "3b2c92192e044a8f94c70e12111b83f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88935b860e7548029feb03eb0b003ae9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } } peers { permanent_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } } peers { permanent_uuid: "3b2c92192e044a8f94c70e12111b83f1" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } } }
I20260812 06:37:08.893486 19040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [sys.catalog]: This master's current role is: LEADER
I20260812 06:37:08.893853 19101 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
W20260812 06:37:08.894970 18922 tablet.cc:2406] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 06:37:08.895598 19101 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 06:37:08.901877 18812 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0 [term 1 FOLLOWER]: Refusing update from remote peer 3b2c92192e044a8f94c70e12111b83f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:37:08.901960 18665 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9 [term 1 FOLLOWER]: Refusing update from remote peer 3b2c92192e044a8f94c70e12111b83f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:37:08.902626 19096 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:37:08.902773 19040 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88935b860e7548029feb03eb0b003ae9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:37:08.906792 19020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b2c92192e044a8f94c70e12111b83f1. Latest consensus state: current_term: 1 leader_uuid: "3b2c92192e044a8f94c70e12111b83f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88935b860e7548029feb03eb0b003ae9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } } peers { permanent_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } } peers { permanent_uuid: "3b2c92192e044a8f94c70e12111b83f1" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } } }
I20260812 06:37:08.906903 19020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:37:08.906989 19042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b2c92192e044a8f94c70e12111b83f1. Latest consensus state: current_term: 1 leader_uuid: "3b2c92192e044a8f94c70e12111b83f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88935b860e7548029feb03eb0b003ae9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } } peers { permanent_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } } peers { permanent_uuid: "3b2c92192e044a8f94c70e12111b83f1" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } } }
I20260812 06:37:08.907109 19042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:37:08.909312 19102 mvcc.cc:204] Tried to move back new op lower bound from 7317572111975706624 to 7317572111943364608. Current Snapshot: MvccSnapshot[applied={T|T < 7317572111975706624}]
I20260812 06:37:08.909850 19040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b2c92192e044a8f94c70e12111b83f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88935b860e7548029feb03eb0b003ae9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } } peers { permanent_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } } peers { permanent_uuid: "3b2c92192e044a8f94c70e12111b83f1" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } } }
I20260812 06:37:08.909951 19040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [sys.catalog]: This master's current role is: LEADER
I20260812 06:37:08.910929 19104 mvcc.cc:204] Tried to move back new op lower bound from 7317572111975706624 to 7317572111943364608. Current Snapshot: MvccSnapshot[applied={T|T < 7317572111975706624}]
I20260812 06:37:08.911074 19042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b2c92192e044a8f94c70e12111b83f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88935b860e7548029feb03eb0b003ae9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } } peers { permanent_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } } peers { permanent_uuid: "3b2c92192e044a8f94c70e12111b83f1" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } } }
I20260812 06:37:08.911175 19042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:37:08.911803 19040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b2c92192e044a8f94c70e12111b83f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88935b860e7548029feb03eb0b003ae9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } } peers { permanent_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } } peers { permanent_uuid: "3b2c92192e044a8f94c70e12111b83f1" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } } }
I20260812 06:37:08.911904 19040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1 [sys.catalog]: This master's current role is: LEADER
I20260812 06:37:08.912726 19020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b2c92192e044a8f94c70e12111b83f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88935b860e7548029feb03eb0b003ae9" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 45587 } } peers { permanent_uuid: "040e1cbbcd2340c7a828a9fd5ab4ddc0" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 41221 } } peers { permanent_uuid: "3b2c92192e044a8f94c70e12111b83f1" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 40481 } } }
I20260812 06:37:08.912833 19020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:37:08.913908 19101 catalog_manager.cc:1383] Generated new cluster ID: ff955ccb13654000822121f607b5b6fd
I20260812 06:37:08.913964 19101 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 06:37:08.915481 19103 mvcc.cc:204] Tried to move back new op lower bound from 7317572111975706624 to 7317572111943364608. Current Snapshot: MvccSnapshot[applied={T|T < 7317572111975706624}]
I20260812 06:37:08.955027 19101 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 06:37:08.956138 19101 catalog_manager.cc:1540] Loading token signing keys...
I20260812 06:37:08.968508 19101 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 3b2c92192e044a8f94c70e12111b83f1: Generated new TSK 0
I20260812 06:37:08.969208 19101 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:37:09.028121 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.22.65:0
--local_ip_for_outbound_sockets=127.13.22.65
--webserver_interface=127.13.22.65
--webserver_port=0
--tserver_master_addrs=127.13.22.126:45587,127.13.22.125:41221,127.13.22.124:40481
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:38445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 06:37:09.129673 19126 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:09.129870 19126 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:09.129962 19126 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:09.133186 19126 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:09.133325 19126 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.22.65
I20260812 06:37:09.137470 19126 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:38445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.65
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--tserver_master_addrs=127.13.22.126:45587,127.13.22.125:41221,127.13.22.124:40481
--never_fsync=true
--heap_profile_path=/tmp/kudu.19126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.22.65

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:09.138443 19126 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:09.139286 19126 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:09.145334 19133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:09.145347 19134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:09.145536 19126 server_base.cc:1061] running on GCE node
W20260812 06:37:09.145340 19137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:09.146205 19126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:09.146687 19126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:09.147831 19126 hybrid_clock.cc:648] HybridClock initialized: now 1786516629147813 us; error 35 us; skew 500 ppm
Aug 12 06:37:09 dist-test-slave-gjrz krb5kdc[18538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516629, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:37:09.150408 19126 init.cc:377] Logged in from keytab as kudu/127.13.22.65@KRBTEST.COM (short username kudu)
I20260812 06:37:09.160755 19126 webserver.cc:533] Webserver started at https://127.13.22.65:46001/ using document root <none> and password file <none>
I20260812 06:37:09.161341 19126 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:09.161388 19126 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:09.161562 19126 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:09.163210 19126 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/data/instance:
uuid: "12bfb02e0cd7476fa4e1da076c0c50d9"
format_stamp: "Formatted at 2026-08-12 06:37:09 on dist-test-slave-gjrz"
I20260812 06:37:09.163693 19126 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/wal/instance:
uuid: "12bfb02e0cd7476fa4e1da076c0c50d9"
format_stamp: "Formatted at 2026-08-12 06:37:09 on dist-test-slave-gjrz"
I20260812 06:37:09.167014 19126 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.002s	sys 0.001s
I20260812 06:37:09.169200 19150 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:09.170181 19126 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:37:09.170291 19126 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/wal
uuid: "12bfb02e0cd7476fa4e1da076c0c50d9"
format_stamp: "Formatted at 2026-08-12 06:37:09 on dist-test-slave-gjrz"
I20260812 06:37:09.170375 19126 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:09.199941 19126 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:09.202647 19126 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:09.202816 19126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:09.203459 19126 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:09.204401 19126 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:37:09.204450 19126 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:09.204501 19126 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:37:09.204527 19126 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:09.213666 19126 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.65:33469
I20260812 06:37:09.213701 19359 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.65:33469 every 8 connection(s)
I20260812 06:37:09.214560 19126 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
I20260812 06:37:09.222846 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 19126
I20260812 06:37:09.222960 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/wal/instance
Aug 12 06:37:09 dist-test-slave-gjrz krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516629, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
Aug 12 06:37:09 dist-test-slave-gjrz krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516629, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:37:09.229785 19368 heartbeater.cc:344] Connected to a master server at 127.13.22.125:41221
I20260812 06:37:09.229996 19368 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:09.230435 19368 heartbeater.cc:507] Master 127.13.22.125:41221 requested a full tablet report, sending...
I20260812 06:37:09.230937 19363 heartbeater.cc:344] Connected to a master server at 127.13.22.124:40481
I20260812 06:37:09.231019 19363 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:09.231210 19363 heartbeater.cc:507] Master 127.13.22.124:40481 requested a full tablet report, sending...
I20260812 06:37:09.231982 18795 ts_manager.cc:194] Registered new tserver with Master: 12bfb02e0cd7476fa4e1da076c0c50d9 (127.13.22.65:33469)
I20260812 06:37:09.232326 18941 ts_manager.cc:194] Registered new tserver with Master: 12bfb02e0cd7476fa4e1da076c0c50d9 (127.13.22.65:33469)
I20260812 06:37:09.233742 18941 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.22.65@KRBTEST.COM'} at 127.13.22.65:49065
Aug 12 06:37:09 dist-test-slave-gjrz krb5kdc[18538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516629, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:37:09.238545 19367 heartbeater.cc:344] Connected to a master server at 127.13.22.126:45587
I20260812 06:37:09.238649 19367 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:09.238806 19367 heartbeater.cc:507] Master 127.13.22.126:45587 requested a full tablet report, sending...
I20260812 06:37:09.239837 18642 ts_manager.cc:194] Registered new tserver with Master: 12bfb02e0cd7476fa4e1da076c0c50d9 (127.13.22.65:33469)
I20260812 06:37:09.242416 13401 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
time=2026-08-12T06:37:09.295Z level=INFO source=main.go:1678 msg="updated GOGC" old=100 new=75
time=2026-08-12T06:37:09.296Z level=INFO source=main.go:744 msg="Leaving GOMAXPROCS=4: CPU quota undefined" component=automaxprocs
time=2026-08-12T06:37:09.297Z level=INFO source=memlimit.go:198 msg="GOMEMLIMIT is updated" component=automemlimit package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=14142549196 previous=9223372036854775807
time=2026-08-12T06:37:09.297Z level=INFO source=main.go:851 msg="Starting Prometheus Server" mode=server version="(version=3.11.2, branch=HEAD, revision=f0f0fdd679dcd6df320b0558b20919f7cd44c407)"
time=2026-08-12T06:37:09.297Z level=INFO source=main.go:856 msg="operational information" build_context="(go=go1.26.2, platform=linux/amd64, user=root@d2684485c347, date=20260413-11:53:40, tags=netgo,builtinassets)" host_details="(Linux 5.4.0-1157-gcp #166~18.04.1-Ubuntu SMP Wed Jan 21 01:57:46 UTC 2026 x86_64 dist-test-slave-gjrz (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)"
time=2026-08-12T06:37:09.521Z level=INFO source=web.go:710 msg="Start listening for connections" component=web address=127.0.0.1:0
time=2026-08-12T06:37:09.521Z level=INFO source=main.go:1410 msg="Starting TSDB ..."
time=2026-08-12T06:37:09.523Z level=INFO source=tls_config.go:354 msg="Listening on" component=web address=127.0.0.1:40255
time=2026-08-12T06:37:09.523Z level=INFO source=tls_config.go:357 msg="TLS is disabled." component=web http2=false address=127.0.0.1:40255
time=2026-08-12T06:37:09.527Z level=INFO source=head.go:698 msg="Replaying on-disk memory mappable chunks if any" component=tsdb
time=2026-08-12T06:37:09.527Z level=INFO source=head.go:784 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=2.578µs
time=2026-08-12T06:37:09.527Z level=INFO source=head.go:792 msg="Replaying WAL, this may take a while" component=tsdb
time=2026-08-12T06:37:09.527Z level=INFO source=head.go:865 msg="WAL segment loaded" component=tsdb segment=0 maxSegment=0 duration=385.034µs
time=2026-08-12T06:37:09.527Z level=INFO source=head.go:902 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=58.379µs wal_replay_duration=427.671µs wbl_replay_duration=137ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=2.578µs total_replay_duration=549.211µs
time=2026-08-12T06:37:09.531Z level=INFO source=main.go:1431 msg="filesystem information" fs_type=OVERLAYFS_SUPER_MAGIC
time=2026-08-12T06:37:09.531Z level=INFO source=main.go:1434 msg="TSDB started"
time=2026-08-12T06:37:09.531Z level=INFO source=main.go:1632 msg="Loading configuration file" filename=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/prometheus.yml
time=2026-08-12T06:37:09.532Z level=INFO source=main.go:1048 msg="TSDB retention updated" duration=15d size=0B percentage=0
time=2026-08-12T06:37:09.536Z level=INFO source=main.go:1671 msg="Completed loading of configuration file" db_storage=54.283µs remote_storage=3.271µs web_handler=508ns query_engine=1.091µs scrape=3.707866ms scrape_sd=213.597µs notify=2.215µs notify_sd=1.596µs rules=2.728µs tracing=5.588µs filename=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestWrongTokenSdFails.1786516559891118-13401-0/prometheus.yml totalDuration=4.514907ms
time=2026-08-12T06:37:09.536Z level=INFO source=main.go:1395 msg="Server is ready to receive web requests."
time=2026-08-12T06:37:09.536Z level=INFO source=manager.go:209 msg="Starting rule manager..." component="rule manager"
W20260812 06:37:09.558864 19401 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:48858: invalid bearer token
time=2026-08-12T06:37:09.559Z level=ERROR source=refresh.go:72 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
W20260812 06:37:09.560317 19404 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:59150: invalid bearer token
W20260812 06:37:09.560424 19406 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:40514: invalid bearer token
time=2026-08-12T06:37:09.560Z level=ERROR source=refresh.go:72 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:37:09.560Z level=ERROR source=refresh.go:72 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
I20260812 06:37:09.707904 19043 catalog_manager.cc:1295] Loaded cluster ID: ff955ccb13654000822121f607b5b6fd
I20260812 06:37:09.707984 19043 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9: loading cluster ID for follower catalog manager: success
I20260812 06:37:09.708644 19091 catalog_manager.cc:1295] Loaded cluster ID: ff955ccb13654000822121f607b5b6fd
I20260812 06:37:09.708726 19091 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0: loading cluster ID for follower catalog manager: success
I20260812 06:37:09.709743 19043 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9: acquiring CA information for follower catalog manager: success
I20260812 06:37:09.710919 19043 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 88935b860e7548029feb03eb0b003ae9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 06:37:09.711136 19091 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0: acquiring CA information for follower catalog manager: success
I20260812 06:37:09.712461 19091 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 040e1cbbcd2340c7a828a9fd5ab4ddc0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 06:37:10.235466 19363 heartbeater.cc:499] Master 127.13.22.124:40481 was elected leader, sending a full tablet report...
W20260812 06:37:11.562820 19401 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:48858: invalid bearer token
time=2026-08-12T06:37:11.563Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
W20260812 06:37:11.563468 19404 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:59150: invalid bearer token
W20260812 06:37:11.563470 19406 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:40514: invalid bearer token
time=2026-08-12T06:37:11.563Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:37:11.563Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
W20260812 06:37:13.560593 19401 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:48858: invalid bearer token
time=2026-08-12T06:37:13.560Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
W20260812 06:37:13.561218 19404 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:59150: invalid bearer token
W20260812 06:37:13.561259 19406 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:40514: invalid bearer token
time=2026-08-12T06:37:13.561Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:37:13.561Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
W20260812 06:37:15.560623 19401 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:48858: invalid bearer token
time=2026-08-12T06:37:15.560Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
W20260812 06:37:15.561167 19406 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:40514: invalid bearer token
W20260812 06:37:15.561218 19404 webserver.cc:660] Failed to authenticate Prometheus request from 127.0.0.1:59150: invalid bearer token
time=2026-08-12T06:37:15.561Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:37:15.561Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:37:17.313Z level=WARN source=main.go:1191 msg="Received an OS signal, exiting gracefully..." signal=terminated
time=2026-08-12T06:37:17.313Z level=INFO source=main.go:1216 msg="Stopping scrape discovery manager..."
time=2026-08-12T06:37:17.313Z level=INFO source=main.go:1230 msg="Stopping notify discovery manager..."
time=2026-08-12T06:37:17.313Z level=INFO source=main.go:1245 msg="Stopping rule manager manager..."
time=2026-08-12T06:37:17.313Z level=INFO source=manager.go:223 msg="Stopping rule manager..." component="rule manager"
time=2026-08-12T06:37:17.313Z level=INFO source=manager.go:239 msg="Rule manager stopped" component="rule manager"
time=2026-08-12T06:37:17.313Z level=INFO source=main.go:1269 msg="Stopping scrape manager..."
time=2026-08-12T06:37:17.313Z level=INFO source=main.go:1284 msg="Stopping tracing manager..."
time=2026-08-12T06:37:17.313Z level=INFO source=main.go:1369 msg="Stopping reloaders..."
time=2026-08-12T06:37:17.313Z level=INFO source=main.go:1212 msg="Scrape discovery manager stopped"
time=2026-08-12T06:37:17.313Z level=INFO source=main.go:1458 msg="Stopping storage..."
time=2026-08-12T06:37:17.313Z level=INFO source=main.go:1241 msg="Rule manager stopped"
time=2026-08-12T06:37:17.313Z level=INFO source=main.go:1261 msg="Scrape manager stopped"
time=2026-08-12T06:37:17.313Z level=INFO source=main.go:1280 msg="Tracing manager stopped"
time=2026-08-12T06:37:17.314Z level=INFO source=main.go:1226 msg="Notify discovery manager stopped"
time=2026-08-12T06:37:17.314Z level=INFO source=main.go:1361 msg="Reloaders stopped"
time=2026-08-12T06:37:17.315Z level=INFO source=main.go:1536 msg="Stopping web handler..."
time=2026-08-12T06:37:17.315Z level=INFO source=main.go:1559 msg="Stopping notifier manager..."
time=2026-08-12T06:37:17.315Z level=INFO source=manager.go:321 msg="Stopping notification manager..." component=notifier
time=2026-08-12T06:37:17.315Z level=INFO source=main.go:1454 msg="TSDB stopped"
time=2026-08-12T06:37:17.315Z level=INFO source=main.go:1555 msg="Notifier manager stopped"
time=2026-08-12T06:37:17.315Z level=INFO source=main.go:1532 msg="Web handler stopped"
time=2026-08-12T06:37:17.315Z level=INFO source=main.go:1570 msg="See you next time!"
I20260812 06:37:17.333963 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 19126
I20260812 06:37:17.347657 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 18574
I20260812 06:37:17.360088 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 18737
I20260812 06:37:17.368244 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 18872
2026-08-12T06:37:17Z chronyd exiting
[       OK ] PrometheusTokenITest.TestWrongTokenSdFails (11532 ms)
[ RUN      ] PrometheusTokenITest.TestMissingTokenSdFails
Loading random data
Initializing database '/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 06:37:17 dist-test-slave-gjrz krb5kdc[19425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 06:37:17 dist-test-slave-gjrz krb5kdc[19425](info): set up 2 sockets
Aug 12 06:37:17 dist-test-slave-gjrz krb5kdc[19425](info): commencing operation
krb5kdc: starting...
W20260812 06:37:19.443818 13401 mini_kdc.cc:121] Time spent starting KDC: real 2.042s	user 0.000s	sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/test-user.keytab.
Aug 12 06:37:19 dist-test-slave-gjrz krb5kdc[19425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM: 
2026-08-12T06:37:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-08-12T06:37:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:37:19.568230 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.126:37059
--webserver_interface=127.13.22.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:37059,127.13.22.125:42689,127.13.22.124:45195
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 06:37:19.667529 19454 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:19.667719 19454 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:19.667781 19454 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:19.671355 19454 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:37:19.671427 19454 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:19.671445 19454 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:37:19.671468 19454 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:37:19.675652 19454 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:37059,127.13.22.125:42689,127.13.22.124:45195
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.126:37059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.126
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:19.676574 19454 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:19.677352 19454 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:19.682556 19467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:19.682580 19461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:19.682682 19454 server_base.cc:1061] running on GCE node
W20260812 06:37:19.682580 19463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:19.683209 19454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:19.683985 19454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:19.685132 19454 hybrid_clock.cc:648] HybridClock initialized: now 1786516639685119 us; error 29 us; skew 500 ppm
Aug 12 06:37:19 dist-test-slave-gjrz krb5kdc[19425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516639, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:37:19.687556 19454 init.cc:377] Logged in from keytab as kudu/127.13.22.126@KRBTEST.COM (short username kudu)
I20260812 06:37:19.697971 19454 webserver.cc:533] Webserver started at https://127.13.22.126:34133/ using document root <none> and password file <none>
I20260812 06:37:19.698559 19454 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:19.698612 19454 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:19.698803 19454 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:19.700403 19454 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/data/instance:
uuid: "279282ef48b94cf9bb12065359ed15d5"
format_stamp: "Formatted at 2026-08-12 06:37:19 on dist-test-slave-gjrz"
I20260812 06:37:19.700841 19454 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/wal/instance:
uuid: "279282ef48b94cf9bb12065359ed15d5"
format_stamp: "Formatted at 2026-08-12 06:37:19 on dist-test-slave-gjrz"
I20260812 06:37:19.704051 19454 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.003s	sys 0.001s
I20260812 06:37:19.705951 19491 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:19.706843 19454 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:37:19.706949 19454 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/wal
uuid: "279282ef48b94cf9bb12065359ed15d5"
format_stamp: "Formatted at 2026-08-12 06:37:19 on dist-test-slave-gjrz"
I20260812 06:37:19.707026 19454 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:19.720444 19454 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:19.722549 19454 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:19.722702 19454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:19.729604 19594 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.126:37059 every 8 connection(s)
I20260812 06:37:19.729605 19454 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.126:37059
I20260812 06:37:19.730598 19454 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/data/info.pb
I20260812 06:37:19.732235 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 19454
I20260812 06:37:19.732354 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-0/wal/instance
I20260812 06:37:19.733626 19595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:19.737761 19595 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.22.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.125@KRBTEST.COM" created.
Aug 12 06:37:19 dist-test-slave-gjrz krb5kdc[19425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516639, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:37:19.750813 19595 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } has no permanent_uuid. Determining permanent_uuid...
W20260812 06:37:19.751552 19495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.22.125:42689: connect: Connection refused (error 111)
W20260812 06:37:19.752339 19595 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:42689: Network error: Client connection negotiation failed: client connection to 127.13.22.125:42689: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:37:19.774200 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.125:42689
--webserver_interface=127.13.22.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:37059,127.13.22.125:42689,127.13.22.124:45195
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 06:37:19.801566 19595 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } attempt: 1
W20260812 06:37:19.803048 19595 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:42689: Network error: Client connection negotiation failed: client connection to 127.13.22.125:42689: connect: Connection refused (error 111)
I20260812 06:37:19.871207 19595 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } attempt: 2
W20260812 06:37:19.872632 19595 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.125:42689: Network error: Client connection negotiation failed: client connection to 127.13.22.125:42689: connect: Connection refused (error 111)
W20260812 06:37:19.873941 19611 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:19.874122 19611 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:19.874207 19611 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:19.877415 19611 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:37:19.877468 19611 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:19.877486 19611 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:37:19.877503 19611 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260812 06:37:19.881594 19611 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:37059,127.13.22.125:42689,127.13.22.124:45195
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.125:42689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.125
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:19.882468 19611 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:19.883204 19611 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:19.888055 19627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:19.888055 19622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:19.888244 19611 server_base.cc:1061] running on GCE node
W20260812 06:37:19.888060 19621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:19.888783 19611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:19.889585 19611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:19.890743 19611 hybrid_clock.cc:648] HybridClock initialized: now 1786516639890725 us; error 28 us; skew 500 ppm
Aug 12 06:37:19 dist-test-slave-gjrz krb5kdc[19425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516639, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:37:19.893242 19611 init.cc:377] Logged in from keytab as kudu/127.13.22.125@KRBTEST.COM (short username kudu)
I20260812 06:37:19.904198 19611 webserver.cc:533] Webserver started at https://127.13.22.125:37689/ using document root <none> and password file <none>
I20260812 06:37:19.904795 19611 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:19.904848 19611 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:19.905045 19611 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:19.906637 19611 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/data/instance:
uuid: "1a81b9d98647416ebb8bfaf71962b867"
format_stamp: "Formatted at 2026-08-12 06:37:19 on dist-test-slave-gjrz"
I20260812 06:37:19.907080 19611 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/wal/instance:
uuid: "1a81b9d98647416ebb8bfaf71962b867"
format_stamp: "Formatted at 2026-08-12 06:37:19 on dist-test-slave-gjrz"
I20260812 06:37:19.910344 19611 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.001s	sys 0.001s
I20260812 06:37:19.912369 19637 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:19.913293 19611 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260812 06:37:19.913406 19611 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/wal
uuid: "1a81b9d98647416ebb8bfaf71962b867"
format_stamp: "Formatted at 2026-08-12 06:37:19 on dist-test-slave-gjrz"
I20260812 06:37:19.913491 19611 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:19.926813 19611 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:19.928959 19611 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:19.929116 19611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:19.936026 19611 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.125:42689
I20260812 06:37:19.936030 19749 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.125:42689 every 8 connection(s)
I20260812 06:37:19.936937 19611 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/data/info.pb
I20260812 06:37:19.938412 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 19611
I20260812 06:37:19.938517 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-1/wal/instance
I20260812 06:37:19.939683 19754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:19.943679 19754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.22.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.124@KRBTEST.COM" created.
Aug 12 06:37:19 dist-test-slave-gjrz krb5kdc[19425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516639, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:37:19.955754 19754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:37:19 dist-test-slave-gjrz krb5kdc[19425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516639, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:37:19.963779 19595 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } attempt: 3
I20260812 06:37:19.966322 19754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } has no permanent_uuid. Determining permanent_uuid...
W20260812 06:37:19.967315 19638 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.22.124:45195: connect: Connection refused (error 111)
Aug 12 06:37:19 dist-test-slave-gjrz krb5kdc[19425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516639, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
W20260812 06:37:19.968253 19754 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:45195: Network error: Client connection negotiation failed: client connection to 127.13.22.124:45195: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.13.22.124@KRBTEST.COM; defaulting to no policy
I20260812 06:37:19.971721 19595 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.124@KRBTEST.COM" created.
W20260812 06:37:19.973080 19595 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:45195: Network error: Client connection negotiation failed: client connection to 127.13.22.124:45195: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:37:19.983820 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.124:45195
--webserver_interface=127.13.22.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.22.126:37059,127.13.22.125:42689,127.13.22.124:45195
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 06:37:20.004222 19595 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } attempt: 1
W20260812 06:37:20.005478 19595 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:45195: Network error: Client connection negotiation failed: client connection to 127.13.22.124:45195: connect: Connection refused (error 111)
I20260812 06:37:20.017407 19754 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } attempt: 1
W20260812 06:37:20.018608 19754 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:45195: Network error: Client connection negotiation failed: client connection to 127.13.22.124:45195: connect: Connection refused (error 111)
I20260812 06:37:20.080641 19595 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } attempt: 2
W20260812 06:37:20.082108 19595 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:45195: Network error: Client connection negotiation failed: client connection to 127.13.22.124:45195: connect: Connection refused (error 111)
W20260812 06:37:20.083855 19773 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:20.084036 19773 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:20.084112 19773 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260812 06:37:20.086782 19754 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } attempt: 2
W20260812 06:37:20.087419 19773 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 06:37:20.087474 19773 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:20.087500 19773 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 06:37:20.087571 19773 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260812 06:37:20.088042 19754 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.13.22.124:45195: Network error: Client connection negotiation failed: client connection to 127.13.22.124:45195: connect: Connection refused (error 111)
I20260812 06:37:20.091815 19773 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.22.126:37059,127.13.22.125:42689,127.13.22.124:45195
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.22.124:45195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.124
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false

Master server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:20.092720 19773 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:20.093483 19773 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:20.098521 19782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:20.098483 19783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:20.098483 19788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:20.098762 19773 server_base.cc:1061] running on GCE node
I20260812 06:37:20.099202 19773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:20.100019 19773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:20.101157 19773 hybrid_clock.cc:648] HybridClock initialized: now 1786516640101145 us; error 24 us; skew 500 ppm
Aug 12 06:37:20 dist-test-slave-gjrz krb5kdc[19425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516640, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:37:20.103627 19773 init.cc:377] Logged in from keytab as kudu/127.13.22.124@KRBTEST.COM (short username kudu)
I20260812 06:37:20.114388 19773 webserver.cc:533] Webserver started at https://127.13.22.124:37719/ using document root <none> and password file <none>
I20260812 06:37:20.114967 19773 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:20.115015 19773 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:20.115222 19773 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:20.116820 19773 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/data/instance:
uuid: "2394c37ba2244eea9b830d7f8a07ef8f"
format_stamp: "Formatted at 2026-08-12 06:37:20 on dist-test-slave-gjrz"
I20260812 06:37:20.117271 19773 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/wal/instance:
uuid: "2394c37ba2244eea9b830d7f8a07ef8f"
format_stamp: "Formatted at 2026-08-12 06:37:20 on dist-test-slave-gjrz"
I20260812 06:37:20.120539 19773 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20260812 06:37:20.122470 19803 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:20.123410 19773 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260812 06:37:20.123512 19773 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/wal
uuid: "2394c37ba2244eea9b830d7f8a07ef8f"
format_stamp: "Formatted at 2026-08-12 06:37:20 on dist-test-slave-gjrz"
I20260812 06:37:20.123602 19773 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:20.139241 19773 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:20.144603 19773 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:20.144759 19773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:20.151546 19907 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.124:45195 every 8 connection(s)
I20260812 06:37:20.151551 19773 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.124:45195
I20260812 06:37:20.152494 19773 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/data/info.pb
I20260812 06:37:20.155107 19909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 06:37:20.158159 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 19773
I20260812 06:37:20.158284 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/master-2/wal/instance
I20260812 06:37:20.158457 19909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:37:20 dist-test-slave-gjrz krb5kdc[19425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
Aug 12 06:37:20 dist-test-slave-gjrz krb5kdc[19425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516640, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
Aug 12 06:37:20 dist-test-slave-gjrz krb5kdc[19425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
I20260812 06:37:20.171814 19909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:37:20 dist-test-slave-gjrz krb5kdc[19425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516640, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
I20260812 06:37:20.179196 19754 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } attempt: 3
I20260812 06:37:20.179224 19909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 06:37:20 dist-test-slave-gjrz krb5kdc[19425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:37:20.181259 19595 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } attempt: 3
Aug 12 06:37:20 dist-test-slave-gjrz krb5kdc[19425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516640, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.124@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
Aug 12 06:37:20 dist-test-slave-gjrz krb5kdc[19425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516639, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.125@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
Aug 12 06:37:20 dist-test-slave-gjrz krb5kdc[19425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516639, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.126@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
I20260812 06:37:20.189837 19909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f: Bootstrap starting.
I20260812 06:37:20.192466 19909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:20.192874 19595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5: Bootstrap starting.
I20260812 06:37:20.193210 19754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867: Bootstrap starting.
I20260812 06:37:20.193343 19909 log.cc:826] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:20.194900 19595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:20.195163 19909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f: No bootstrap required, opened a new log
I20260812 06:37:20.195407 19754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867: Neither blocks nor log segments found. Creating new log.
I20260812 06:37:20.196244 19754 log.cc:826] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:20.196406 19595 log.cc:826] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5: Log is configured to *not* fsync() on all Append() calls
I20260812 06:37:20.197676 19754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867: No bootstrap required, opened a new log
I20260812 06:37:20.198024 19595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5: No bootstrap required, opened a new log
I20260812 06:37:20.198266 19909 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } } peers { permanent_uuid: "1a81b9d98647416ebb8bfaf71962b867" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } } peers { permanent_uuid: "2394c37ba2244eea9b830d7f8a07ef8f" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } }
I20260812 06:37:20.198454 19909 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:37:20.198559 19909 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2394c37ba2244eea9b830d7f8a07ef8f, State: Initialized, Role: FOLLOWER
I20260812 06:37:20.198990 19909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } } peers { permanent_uuid: "1a81b9d98647416ebb8bfaf71962b867" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } } peers { permanent_uuid: "2394c37ba2244eea9b830d7f8a07ef8f" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } }
I20260812 06:37:20.199470 19932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } } peers { permanent_uuid: "1a81b9d98647416ebb8bfaf71962b867" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } } peers { permanent_uuid: "2394c37ba2244eea9b830d7f8a07ef8f" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } } }
I20260812 06:37:20.199654 19932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:37:20.200121 19909 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:37:20.200284 19754 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } } peers { permanent_uuid: "1a81b9d98647416ebb8bfaf71962b867" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } } peers { permanent_uuid: "2394c37ba2244eea9b830d7f8a07ef8f" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } }
I20260812 06:37:20.200420 19754 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:37:20.200474 19754 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a81b9d98647416ebb8bfaf71962b867, State: Initialized, Role: FOLLOWER
I20260812 06:37:20.200830 19754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } } peers { permanent_uuid: "1a81b9d98647416ebb8bfaf71962b867" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } } peers { permanent_uuid: "2394c37ba2244eea9b830d7f8a07ef8f" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } }
I20260812 06:37:20.200984 19595 raft_consensus.cc:359] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } } peers { permanent_uuid: "1a81b9d98647416ebb8bfaf71962b867" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } } peers { permanent_uuid: "2394c37ba2244eea9b830d7f8a07ef8f" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } }
I20260812 06:37:20.201237 19595 raft_consensus.cc:385] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 06:37:20.201345 19595 raft_consensus.cc:740] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 279282ef48b94cf9bb12065359ed15d5, State: Initialized, Role: FOLLOWER
I20260812 06:37:20.201838 19754 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:37:20.201787 19595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } } peers { permanent_uuid: "1a81b9d98647416ebb8bfaf71962b867" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } } peers { permanent_uuid: "2394c37ba2244eea9b830d7f8a07ef8f" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } }
I20260812 06:37:20.202862 19595 sys_catalog.cc:565] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 06:37:20.204200 19773 master_runner.cc:187] Error getting master registration for 127.13.22.126:37059: OK, instance_id { permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" instance_seqno: 1786516639727726 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 06:37:20.206293 19773 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 06:37:20.206838 19972 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:37:20.206900 19972 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:37:20.207065 19937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } } peers { permanent_uuid: "1a81b9d98647416ebb8bfaf71962b867" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } } peers { permanent_uuid: "2394c37ba2244eea9b830d7f8a07ef8f" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } } }
I20260812 06:37:20.207160 19937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:37:20.205261 19611 master_runner.cc:187] Error getting master registration for 127.13.22.126:37059: OK, instance_id { permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" instance_seqno: 1786516639727726 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260812 06:37:20.208792 19955 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:37:20.208909 19955 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:37:20.210355 19611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 06:37:20.210666 19986 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 06:37:20.210750 19986 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 06:37:20.211058 19948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } } peers { permanent_uuid: "1a81b9d98647416ebb8bfaf71962b867" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } } peers { permanent_uuid: "2394c37ba2244eea9b830d7f8a07ef8f" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } } }
I20260812 06:37:20.211243 19948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:37:20.213762 19454 master_runner.cc:264] No leader master found from master 279282ef48b94cf9bb12065359ed15d5
I20260812 06:37:20.213830 19454 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260812 06:37:20.223939 19502 tablet.cc:2406] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 06:37:20.286180 19937 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 06:37:20.286379 19937 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } } peers { permanent_uuid: "1a81b9d98647416ebb8bfaf71962b867" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } } peers { permanent_uuid: "2394c37ba2244eea9b830d7f8a07ef8f" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } }
I20260812 06:37:20.287426 19937 leader_election.cc:290] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 279282ef48b94cf9bb12065359ed15d5 (127.13.22.126:37059), 2394c37ba2244eea9b830d7f8a07ef8f (127.13.22.124:45195)
I20260812 06:37:20.288115 19541 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a81b9d98647416ebb8bfaf71962b867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "279282ef48b94cf9bb12065359ed15d5" is_pre_election: true
I20260812 06:37:20.288383 19541 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a81b9d98647416ebb8bfaf71962b867 in term 0.
I20260812 06:37:20.288980 19862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a81b9d98647416ebb8bfaf71962b867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2394c37ba2244eea9b830d7f8a07ef8f" is_pre_election: true
I20260812 06:37:20.289235 19862 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a81b9d98647416ebb8bfaf71962b867 in term 0.
I20260812 06:37:20.289729 19638 leader_election.cc:304] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a81b9d98647416ebb8bfaf71962b867, 279282ef48b94cf9bb12065359ed15d5; no voters: 
I20260812 06:37:20.290352 19937 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 06:37:20.290419 19937 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 06:37:20.290459 19937 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:20.291709 19937 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } } peers { permanent_uuid: "1a81b9d98647416ebb8bfaf71962b867" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } } peers { permanent_uuid: "2394c37ba2244eea9b830d7f8a07ef8f" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } }
I20260812 06:37:20.292042 19937 leader_election.cc:290] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [CANDIDATE]: Term 1 election: Requested vote from peers 279282ef48b94cf9bb12065359ed15d5 (127.13.22.126:37059), 2394c37ba2244eea9b830d7f8a07ef8f (127.13.22.124:45195)
I20260812 06:37:20.292681 19862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a81b9d98647416ebb8bfaf71962b867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2394c37ba2244eea9b830d7f8a07ef8f"
I20260812 06:37:20.292780 19862 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:20.293825 19862 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a81b9d98647416ebb8bfaf71962b867 in term 1.
I20260812 06:37:20.294581 19541 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a81b9d98647416ebb8bfaf71962b867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "279282ef48b94cf9bb12065359ed15d5"
I20260812 06:37:20.294692 19541 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 06:37:20.295431 19638 leader_election.cc:304] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a81b9d98647416ebb8bfaf71962b867, 2394c37ba2244eea9b830d7f8a07ef8f; no voters: 
I20260812 06:37:20.296083 19541 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a81b9d98647416ebb8bfaf71962b867 in term 1.
I20260812 06:37:20.296547 19937 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 06:37:20.296763 19937 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [term 1 LEADER]: Becoming Leader. State: Replica: 1a81b9d98647416ebb8bfaf71962b867, State: Running, Role: LEADER
I20260812 06:37:20.297070 19937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } } peers { permanent_uuid: "1a81b9d98647416ebb8bfaf71962b867" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } } peers { permanent_uuid: "2394c37ba2244eea9b830d7f8a07ef8f" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } }
I20260812 06:37:20.299372 19937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a81b9d98647416ebb8bfaf71962b867. Latest consensus state: current_term: 1 leader_uuid: "1a81b9d98647416ebb8bfaf71962b867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } } peers { permanent_uuid: "1a81b9d98647416ebb8bfaf71962b867" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } } peers { permanent_uuid: "2394c37ba2244eea9b830d7f8a07ef8f" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } } }
I20260812 06:37:20.301671 19937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [sys.catalog]: This master's current role is: LEADER
I20260812 06:37:20.302793 19995 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 06:37:20.304771 19995 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 06:37:20.309964 19862 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f [term 1 FOLLOWER]: Refusing update from remote peer 1a81b9d98647416ebb8bfaf71962b867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:37:20.310606 19992 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2394c37ba2244eea9b830d7f8a07ef8f" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:37:20.314170 19932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a81b9d98647416ebb8bfaf71962b867. Latest consensus state: current_term: 1 leader_uuid: "1a81b9d98647416ebb8bfaf71962b867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } } peers { permanent_uuid: "1a81b9d98647416ebb8bfaf71962b867" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } } peers { permanent_uuid: "2394c37ba2244eea9b830d7f8a07ef8f" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } } }
I20260812 06:37:20.314263 19932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:37:20.315164 19541 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5 [term 1 FOLLOWER]: Refusing update from remote peer 1a81b9d98647416ebb8bfaf71962b867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 06:37:20.315719 19992 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 06:37:20.318923 19948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a81b9d98647416ebb8bfaf71962b867. Latest consensus state: current_term: 1 leader_uuid: "1a81b9d98647416ebb8bfaf71962b867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } } peers { permanent_uuid: "1a81b9d98647416ebb8bfaf71962b867" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } } peers { permanent_uuid: "2394c37ba2244eea9b830d7f8a07ef8f" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } } }
I20260812 06:37:20.319023 19948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:37:20.326056 19992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a81b9d98647416ebb8bfaf71962b867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } } peers { permanent_uuid: "1a81b9d98647416ebb8bfaf71962b867" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } } peers { permanent_uuid: "2394c37ba2244eea9b830d7f8a07ef8f" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } } }
I20260812 06:37:20.326160 19992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [sys.catalog]: This master's current role is: LEADER
I20260812 06:37:20.327502 19948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a81b9d98647416ebb8bfaf71962b867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } } peers { permanent_uuid: "1a81b9d98647416ebb8bfaf71962b867" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } } peers { permanent_uuid: "2394c37ba2244eea9b830d7f8a07ef8f" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } } }
I20260812 06:37:20.327615 19948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:37:20.328114 19997 mvcc.cc:204] Tried to move back new op lower bound from 7317572158703009792 to 7317572158662258688. Current Snapshot: MvccSnapshot[applied={T|T < 7317572158703009792}]
I20260812 06:37:20.329864 20003 mvcc.cc:204] Tried to move back new op lower bound from 7317572158703009792 to 7317572158662258688. Current Snapshot: MvccSnapshot[applied={T|T < 7317572158703009792}]
I20260812 06:37:20.330675 19937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a81b9d98647416ebb8bfaf71962b867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } } peers { permanent_uuid: "1a81b9d98647416ebb8bfaf71962b867" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } } peers { permanent_uuid: "2394c37ba2244eea9b830d7f8a07ef8f" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } } }
I20260812 06:37:20.330771 19937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867 [sys.catalog]: This master's current role is: LEADER
I20260812 06:37:20.332073 19932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a81b9d98647416ebb8bfaf71962b867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279282ef48b94cf9bb12065359ed15d5" member_type: VOTER last_known_addr { host: "127.13.22.126" port: 37059 } } peers { permanent_uuid: "1a81b9d98647416ebb8bfaf71962b867" member_type: VOTER last_known_addr { host: "127.13.22.125" port: 42689 } } peers { permanent_uuid: "2394c37ba2244eea9b830d7f8a07ef8f" member_type: VOTER last_known_addr { host: "127.13.22.124" port: 45195 } } }
I20260812 06:37:20.332175 19932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 06:37:20.332880 19998 mvcc.cc:204] Tried to move back new op lower bound from 7317572158703009792 to 7317572158662258688. Current Snapshot: MvccSnapshot[applied={T|T < 7317572158703009792}]
I20260812 06:37:20.340003 19995 catalog_manager.cc:1383] Generated new cluster ID: de1597f1ac464b41abe408376b7624d5
I20260812 06:37:20.340061 19995 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 06:37:20.371635 19995 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 06:37:20.372741 19995 catalog_manager.cc:1540] Loading token signing keys...
I20260812 06:37:20.391054 19995 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 1a81b9d98647416ebb8bfaf71962b867: Generated new TSK 0
I20260812 06:37:20.391721 19995 catalog_manager.cc:1550] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.22.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.22.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.22.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.22.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab.
I20260812 06:37:20.439757 13401 external_mini_cluster.cc:1372] Running /tmp/dist-test-tasklV_27S/build/debug/bin/kudu
/tmp/dist-test-tasklV_27S/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.22.65:0
--local_ip_for_outbound_sockets=127.13.22.65
--webserver_interface=127.13.22.65
--webserver_port=0
--tserver_master_addrs=127.13.22.126:37059,127.13.22.125:42689,127.13.22.124:45195
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.22.84:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/test.cert
--webserver_private_key_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/test.key
--webserver_private_key_password_cmd=echo test
--webserver_prometheus_token_cmd=echo test-prometheus-bearer-token with env {KRB5CCNAME=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 06:37:20.540845 20019 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 06:37:20.541020 20019 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 06:37:20.541087 20019 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260812 06:37:20.544263 20019 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260812 06:37:20.544387 20019 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.22.65
I20260812 06:37:20.548349 20019 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.22.84:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.22.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.22.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
--webserver_certificate_file=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/test.cert
--webserver_interface=127.13.22.65
--webserver_port=0
--webserver_private_key_file=<redacted>
--webserver_private_key_password_cmd=<redacted>
--webserver_prometheus_token_cmd=<redacted>
--webserver_require_spnego=true
--tserver_master_addrs=127.13.22.126:37059,127.13.22.125:42689,127.13.22.124:45195
--never_fsync=true
--heap_profile_path=/tmp/kudu.20019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.22.65

Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 70c51312f1186a8ae9f46647160b7730123f7601-dirty
build type DEBUG
built by None at 12 Aug 2026 06:06:26 UTC on 39be9dd7121a
I20260812 06:37:20.549170 20019 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 06:37:20.549898 20019 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 06:37:20.555660 20033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:20.555711 20032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 06:37:20.555646 20035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 06:37:20.555847 20019 server_base.cc:1061] running on GCE node
I20260812 06:37:20.556257 20019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 06:37:20.556728 20019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 06:37:20.557869 20019 hybrid_clock.cc:648] HybridClock initialized: now 1786516640557861 us; error 29 us; skew 500 ppm
Aug 12 06:37:20 dist-test-slave-gjrz krb5kdc[19425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516640, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 06:37:20.560403 20019 init.cc:377] Logged in from keytab as kudu/127.13.22.65@KRBTEST.COM (short username kudu)
I20260812 06:37:20.570616 20019 webserver.cc:533] Webserver started at https://127.13.22.65:45087/ using document root <none> and password file <none>
I20260812 06:37:20.571172 20019 fs_manager.cc:362] Metadata directory not provided
I20260812 06:37:20.571219 20019 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 06:37:20.571458 20019 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 06:37:20.572981 20019 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/data/instance:
uuid: "e6ce502227484ad5bf6e9e56041bcb5b"
format_stamp: "Formatted at 2026-08-12 06:37:20 on dist-test-slave-gjrz"
I20260812 06:37:20.573417 20019 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/wal/instance:
uuid: "e6ce502227484ad5bf6e9e56041bcb5b"
format_stamp: "Formatted at 2026-08-12 06:37:20 on dist-test-slave-gjrz"
I20260812 06:37:20.576527 20019 fs_manager.cc:696] Time spent creating directory manager: real 0.003s	user 0.000s	sys 0.004s
I20260812 06:37:20.578406 20048 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:20.579285 20019 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 06:37:20.579388 20019 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/wal
uuid: "e6ce502227484ad5bf6e9e56041bcb5b"
format_stamp: "Formatted at 2026-08-12 06:37:20 on dist-test-slave-gjrz"
I20260812 06:37:20.579470 20019 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 06:37:20.594478 20019 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 06:37:20.596575 20019 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 06:37:20.596728 20019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 06:37:20.597249 20019 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 06:37:20.598089 20019 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 06:37:20.598134 20019 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:20.598174 20019 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 06:37:20.598202 20019 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 06:37:20.607153 20019 rpc_server.cc:307] RPC server started. Bound to: 127.13.22.65:32795
I20260812 06:37:20.607182 20251 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.22.65:32795 every 8 connection(s)
I20260812 06:37:20.608021 20019 server_base.cc:1193] Dumped server information to /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/data/info.pb
I20260812 06:37:20.614426 13401 external_mini_cluster.cc:1434] Started /tmp/dist-test-tasklV_27S/build/debug/bin/kudu as pid 20019
I20260812 06:37:20.614560 13401 external_mini_cluster.cc:1448] Reading /tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/minicluster-data/ts-0/wal/instance
Aug 12 06:37:20 dist-test-slave-gjrz krb5kdc[19425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516640, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.125@KRBTEST.COM
Aug 12 06:37:20 dist-test-slave-gjrz krb5kdc[19425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516640, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.124@KRBTEST.COM
Aug 12 06:37:20 dist-test-slave-gjrz krb5kdc[19425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786516640, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.22.65@KRBTEST.COM for kudu/127.13.22.126@KRBTEST.COM
I20260812 06:37:20.622658 20257 heartbeater.cc:344] Connected to a master server at 127.13.22.125:42689
I20260812 06:37:20.622750 20252 heartbeater.cc:344] Connected to a master server at 127.13.22.124:45195
I20260812 06:37:20.622881 20257 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:20.622881 20252 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:20.623046 20255 heartbeater.cc:344] Connected to a master server at 127.13.22.126:37059
I20260812 06:37:20.623123 20255 heartbeater.cc:461] Registering TS with master...
I20260812 06:37:20.623469 20252 heartbeater.cc:507] Master 127.13.22.124:45195 requested a full tablet report, sending...
I20260812 06:37:20.623481 20257 heartbeater.cc:507] Master 127.13.22.125:42689 requested a full tablet report, sending...
I20260812 06:37:20.623485 20255 heartbeater.cc:507] Master 127.13.22.126:37059 requested a full tablet report, sending...
I20260812 06:37:20.624888 19526 ts_manager.cc:194] Registered new tserver with Master: e6ce502227484ad5bf6e9e56041bcb5b (127.13.22.65:32795)
I20260812 06:37:20.624894 19674 ts_manager.cc:194] Registered new tserver with Master: e6ce502227484ad5bf6e9e56041bcb5b (127.13.22.65:32795)
I20260812 06:37:20.624924 19843 ts_manager.cc:194] Registered new tserver with Master: e6ce502227484ad5bf6e9e56041bcb5b (127.13.22.65:32795)
I20260812 06:37:20.626133 19674 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.22.65@KRBTEST.COM'} at 127.13.22.65:47949
I20260812 06:37:20.634337 13401 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
time=2026-08-12T06:37:20.696Z level=INFO source=main.go:1678 msg="updated GOGC" old=100 new=75
time=2026-08-12T06:37:20.697Z level=INFO source=main.go:744 msg="Leaving GOMAXPROCS=4: CPU quota undefined" component=automaxprocs
time=2026-08-12T06:37:20.697Z level=INFO source=memlimit.go:198 msg="GOMEMLIMIT is updated" component=automemlimit package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=14142549196 previous=9223372036854775807
time=2026-08-12T06:37:20.697Z level=INFO source=main.go:851 msg="Starting Prometheus Server" mode=server version="(version=3.11.2, branch=HEAD, revision=f0f0fdd679dcd6df320b0558b20919f7cd44c407)"
time=2026-08-12T06:37:20.698Z level=INFO source=main.go:856 msg="operational information" build_context="(go=go1.26.2, platform=linux/amd64, user=root@d2684485c347, date=20260413-11:53:40, tags=netgo,builtinassets)" host_details="(Linux 5.4.0-1157-gcp #166~18.04.1-Ubuntu SMP Wed Jan 21 01:57:46 UTC 2026 x86_64 dist-test-slave-gjrz (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)"
time=2026-08-12T06:37:20.961Z level=INFO source=web.go:710 msg="Start listening for connections" component=web address=127.0.0.1:0
time=2026-08-12T06:37:20.962Z level=INFO source=main.go:1410 msg="Starting TSDB ..."
time=2026-08-12T06:37:20.964Z level=INFO source=tls_config.go:354 msg="Listening on" component=web address=127.0.0.1:36357
time=2026-08-12T06:37:20.964Z level=INFO source=tls_config.go:357 msg="TLS is disabled." component=web http2=false address=127.0.0.1:36357
time=2026-08-12T06:37:20.966Z level=INFO source=head.go:698 msg="Replaying on-disk memory mappable chunks if any" component=tsdb
time=2026-08-12T06:37:20.966Z level=INFO source=head.go:784 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=2.437µs
time=2026-08-12T06:37:20.966Z level=INFO source=head.go:792 msg="Replaying WAL, this may take a while" component=tsdb
time=2026-08-12T06:37:20.966Z level=INFO source=head.go:865 msg="WAL segment loaded" component=tsdb segment=0 maxSegment=0 duration=336.614µs
time=2026-08-12T06:37:20.966Z level=INFO source=head.go:902 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=58.279µs wal_replay_duration=375.436µs wbl_replay_duration=181ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=2.437µs total_replay_duration=497.037µs
time=2026-08-12T06:37:20.971Z level=INFO source=main.go:1431 msg="filesystem information" fs_type=OVERLAYFS_SUPER_MAGIC
time=2026-08-12T06:37:20.971Z level=INFO source=main.go:1434 msg="TSDB started"
time=2026-08-12T06:37:20.971Z level=INFO source=main.go:1632 msg="Loading configuration file" filename=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/prometheus.yml
time=2026-08-12T06:37:20.971Z level=INFO source=main.go:1048 msg="TSDB retention updated" duration=15d size=0B percentage=0
time=2026-08-12T06:37:20.972Z level=INFO source=main.go:1671 msg="Completed loading of configuration file" db_storage=43.306µs remote_storage=2.088µs web_handler=544ns query_engine=1.156µs scrape=1.229158ms scrape_sd=114.371µs notify=2.35µs notify_sd=1.464µs rules=1.947µs tracing=4.943µs filename=/tmp/dist-test-tasklV_27S/test-tmp/prometheus_auth-itest.0.PrometheusTokenITest.TestMissingTokenSdFails.1786516559891118-13401-0/prometheus.yml totalDuration=1.816187ms
time=2026-08-12T06:37:20.972Z level=INFO source=main.go:1395 msg="Server is ready to receive web requests."
time=2026-08-12T06:37:20.973Z level=INFO source=manager.go:209 msg="Starting rule manager..." component="rule manager"
time=2026-08-12T06:37:20.995Z level=ERROR source=refresh.go:72 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:37:21.000Z level=ERROR source=refresh.go:72 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:37:21.001Z level=ERROR source=refresh.go:72 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
I20260812 06:37:21.210183 19955 catalog_manager.cc:1295] Loaded cluster ID: de1597f1ac464b41abe408376b7624d5
I20260812 06:37:21.210270 19955 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f: loading cluster ID for follower catalog manager: success
I20260812 06:37:21.212013 19986 catalog_manager.cc:1295] Loaded cluster ID: de1597f1ac464b41abe408376b7624d5
I20260812 06:37:21.212085 19986 catalog_manager.cc:1588] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5: loading cluster ID for follower catalog manager: success
I20260812 06:37:21.212584 19955 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f: acquiring CA information for follower catalog manager: success
I20260812 06:37:21.213835 19955 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 2394c37ba2244eea9b830d7f8a07ef8f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 06:37:21.214052 19986 catalog_manager.cc:1610] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5: acquiring CA information for follower catalog manager: success
I20260812 06:37:21.214959 19986 catalog_manager.cc:1638] T 00000000000000000000000000000000 P 279282ef48b94cf9bb12065359ed15d5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 06:37:21.627650 20257 heartbeater.cc:499] Master 127.13.22.125:42689 was elected leader, sending a full tablet report...
time=2026-08-12T06:37:22.996Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:37:23.001Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:37:23.003Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:37:24.997Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:37:25.002Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:37:25.002Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:37:26.996Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:37:27.002Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:37:27.002Z level=ERROR source=refresh.go:91 msg="Unable to refresh target groups" component="discovery manager scrape" discovery=http config=kudu err="server returned HTTP status 401 Authentication Required"
time=2026-08-12T06:37:28.728Z level=WARN source=main.go:1191 msg="Received an OS signal, exiting gracefully..." signal=terminated
time=2026-08-12T06:37:28.728Z level=INFO source=main.go:1216 msg="Stopping scrape discovery manager..."
time=2026-08-12T06:37:28.728Z level=INFO source=main.go:1230 msg="Stopping notify discovery manager..."
time=2026-08-12T06:37:28.728Z level=INFO source=main.go:1245 msg="Stopping rule manager manager..."
time=2026-08-12T06:37:28.728Z level=INFO source=main.go:1212 msg="Scrape discovery manager stopped"
time=2026-08-12T06:37:28.728Z level=INFO source=manager.go:223 msg="Stopping rule manager..." component="rule manager"
time=2026-08-12T06:37:28.728Z level=INFO source=manager.go:239 msg="Rule manager stopped" component="rule manager"
time=2026-08-12T06:37:28.728Z level=INFO source=main.go:1269 msg="Stopping scrape manager..."
time=2026-08-12T06:37:28.728Z level=INFO source=main.go:1284 msg="Stopping tracing manager..."
time=2026-08-12T06:37:28.728Z level=INFO source=main.go:1369 msg="Stopping reloaders..."
time=2026-08-12T06:37:28.728Z level=INFO source=main.go:1458 msg="Stopping storage..."
time=2026-08-12T06:37:28.728Z level=INFO source=main.go:1226 msg="Notify discovery manager stopped"
time=2026-08-12T06:37:28.729Z level=INFO source=main.go:1241 msg="Rule manager stopped"
time=2026-08-12T06:37:28.729Z level=INFO source=main.go:1261 msg="Scrape manager stopped"
time=2026-08-12T06:37:28.729Z level=INFO source=main.go:1361 msg="Reloaders stopped"
time=2026-08-12T06:37:28.729Z level=INFO source=main.go:1280 msg="Tracing manager stopped"
time=2026-08-12T06:37:28.730Z level=INFO source=main.go:1536 msg="Stopping web handler..."
time=2026-08-12T06:37:28.730Z level=INFO source=main.go:1559 msg="Stopping notifier manager..."
time=2026-08-12T06:37:28.730Z level=INFO source=manager.go:321 msg="Stopping notification manager..." component=notifier
time=2026-08-12T06:37:28.730Z level=INFO source=main.go:1555 msg="Notifier manager stopped"
time=2026-08-12T06:37:28.730Z level=INFO source=main.go:1454 msg="TSDB stopped"
time=2026-08-12T06:37:28.730Z level=INFO source=main.go:1532 msg="Web handler stopped"
time=2026-08-12T06:37:28.730Z level=INFO source=main.go:1570 msg="See you next time!"
I20260812 06:37:28.748999 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 20019
I20260812 06:37:28.761505 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 19454
I20260812 06:37:28.773415 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 19611
I20260812 06:37:28.784377 13401 external_mini_cluster.cc:1664] Killing /tmp/dist-test-tasklV_27S/build/debug/bin/kudu with pid 19773
2026-08-12T06:37:28Z chronyd exiting
[       OK ] PrometheusTokenITest.TestMissingTokenSdFails (11411 ms)
[----------] 6 tests from PrometheusTokenITest (66865 ms total)

[----------] Global test environment tear-down
[==========] 8 tests from 2 test suites ran. (88913 ms total)
[  PASSED  ] 8 tests.
