[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from EvictionPolicyTypes/FullStackInsertScanTestWithEvictionPolicy
[ RUN      ] EvictionPolicyTypes/FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260812 08:03:02.175267 24143 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.147.254:37725
I20260812 08:03:02.176419 24143 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 08:03:02.177064 24143 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:03:02.183771 24151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:03:02.183815 24149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:03:02.184006 24143 server_base.cc:1061] running on GCE node
W20260812 08:03:02.184242 24154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:03:02.184780 24143 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 08:03:02.184926 24143 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 08:03:02.184978 24143 hybrid_clock.cc:648] HybridClock initialized: now 1786521782184974 us; error 0 us; skew 500 ppm
I20260812 08:03:02.186939 24143 webserver.cc:533] Webserver started at http://127.23.147.254:40733/ using document root <none> and password file <none>
I20260812 08:03:02.187740 24143 fs_manager.cc:362] Metadata directory not provided
I20260812 08:03:02.187841 24143 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:03:02.188123 24143 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:03:02.189986 24143 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest_0.1786521782163021-24143-0/minicluster-data/master-0-root/instance:
uuid: "a735ae45e0bd4630aa165ac44d2bd734"
format_stamp: "Formatted at 2026-08-12 08:03:02 on dist-test-slave-ncp9"
I20260812 08:03:02.194064 24143 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260812 08:03:02.196421 24163 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:02.197628 24143 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.001s
I20260812 08:03:02.197748 24143 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest_0.1786521782163021-24143-0/minicluster-data/master-0-root
uuid: "a735ae45e0bd4630aa165ac44d2bd734"
format_stamp: "Formatted at 2026-08-12 08:03:02 on dist-test-slave-ncp9"
I20260812 08:03:02.197852 24143 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest_0.1786521782163021-24143-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest_0.1786521782163021-24143-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest_0.1786521782163021-24143-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:03:02.219650 24143 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:03:02.220324 24143 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 08:03:02.220487 24143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:03:02.221406 24171 maintenance_manager.cc:346] Maintenance manager is disabled (check --enable_maintenance_manager).
I20260812 08:03:02.228053 24275 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.147.254:37725 every 8 connection(s)
I20260812 08:03:02.228058 24143 rpc_server.cc:307] RPC server started. Bound to: 127.23.147.254:37725
I20260812 08:03:02.230402 24276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:03:02.235904 24276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734: Bootstrap starting.
I20260812 08:03:02.238299 24276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734: Neither blocks nor log segments found. Creating new log.
I20260812 08:03:02.239153 24276 log.cc:826] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734: Log is configured to *not* fsync() on all Append() calls
I20260812 08:03:02.240764 24276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734: No bootstrap required, opened a new log
I20260812 08:03:02.243484 24276 raft_consensus.cc:359] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a735ae45e0bd4630aa165ac44d2bd734" member_type: VOTER }
I20260812 08:03:02.243655 24276 raft_consensus.cc:385] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:03:02.243713 24276 raft_consensus.cc:740] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a735ae45e0bd4630aa165ac44d2bd734, State: Initialized, Role: FOLLOWER
I20260812 08:03:02.244251 24276 consensus_queue.cc:260] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a735ae45e0bd4630aa165ac44d2bd734" member_type: VOTER }
I20260812 08:03:02.244378 24276 raft_consensus.cc:399] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 08:03:02.244421 24276 raft_consensus.cc:493] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 08:03:02.244508 24276 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:03:02.245244 24276 raft_consensus.cc:515] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a735ae45e0bd4630aa165ac44d2bd734" member_type: VOTER }
I20260812 08:03:02.245641 24276 leader_election.cc:304] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a735ae45e0bd4630aa165ac44d2bd734; no voters: 
I20260812 08:03:02.245903 24276 leader_election.cc:290] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 08:03:02.246044 24281 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:03:02.246269 24281 raft_consensus.cc:697] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734 [term 1 LEADER]: Becoming Leader. State: Replica: a735ae45e0bd4630aa165ac44d2bd734, State: Running, Role: LEADER
I20260812 08:03:02.246631 24281 consensus_queue.cc:237] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a735ae45e0bd4630aa165ac44d2bd734" member_type: VOTER }
I20260812 08:03:02.246886 24276 sys_catalog.cc:565] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:03:02.248551 24283 sys_catalog.cc:455] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a735ae45e0bd4630aa165ac44d2bd734. Latest consensus state: current_term: 1 leader_uuid: "a735ae45e0bd4630aa165ac44d2bd734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a735ae45e0bd4630aa165ac44d2bd734" member_type: VOTER } }
I20260812 08:03:02.248677 24283 sys_catalog.cc:458] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734 [sys.catalog]: This master's current role is: LEADER
I20260812 08:03:02.248952 24282 sys_catalog.cc:455] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a735ae45e0bd4630aa165ac44d2bd734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a735ae45e0bd4630aa165ac44d2bd734" member_type: VOTER } }
I20260812 08:03:02.249034 24282 sys_catalog.cc:458] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734 [sys.catalog]: This master's current role is: LEADER
I20260812 08:03:02.249056 24305 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:03:02.249136 24143 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260812 08:03:02.251353 24305 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:03:02.256254 24305 catalog_manager.cc:1383] Generated new cluster ID: f7d0653295114f2595763ae14d4e86c1
I20260812 08:03:02.256330 24305 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:03:02.275578 24305 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:03:02.276757 24305 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:03:02.295537 24305 catalog_manager.cc:6092] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734: Generated new TSK 0
I20260812 08:03:02.296406 24305 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:03:02.313962 24143 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:03:02.316673 24316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:03:02.316761 24319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:03:02.316784 24315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:03:02.316881 24143 server_base.cc:1061] running on GCE node
I20260812 08:03:02.317173 24143 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 08:03:02.317219 24143 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 08:03:02.317234 24143 hybrid_clock.cc:648] HybridClock initialized: now 1786521782317235 us; error 0 us; skew 500 ppm
I20260812 08:03:02.318207 24143 webserver.cc:533] Webserver started at http://127.23.147.193:42365/ using document root <none> and password file <none>
I20260812 08:03:02.318389 24143 fs_manager.cc:362] Metadata directory not provided
I20260812 08:03:02.318480 24143 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:03:02.318574 24143 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:03:02.318988 24143 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest_0.1786521782163021-24143-0/minicluster-data/ts-0-root/instance:
uuid: "2ecbf965a2194794b143f84621a1b0d0"
format_stamp: "Formatted at 2026-08-12 08:03:02 on dist-test-slave-ncp9"
I20260812 08:03:02.320602 24143 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.000s	sys 0.002s
I20260812 08:03:02.321678 24328 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:02.321969 24143 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.000s
I20260812 08:03:02.322049 24143 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest_0.1786521782163021-24143-0/minicluster-data/ts-0-root
uuid: "2ecbf965a2194794b143f84621a1b0d0"
format_stamp: "Formatted at 2026-08-12 08:03:02 on dist-test-slave-ncp9"
I20260812 08:03:02.322131 24143 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest_0.1786521782163021-24143-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest_0.1786521782163021-24143-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest_0.1786521782163021-24143-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:03:02.354429 24143 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:03:02.354876 24143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:03:02.355333 24143 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:03:02.356185 24143 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:03:02.356240 24143 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:02.356295 24143 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:03:02.356326 24143 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:02.362341 24143 rpc_server.cc:307] RPC server started. Bound to: 127.23.147.193:34357
I20260812 08:03:02.362375 24445 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.147.193:34357 every 8 connection(s)
I20260812 08:03:02.376741 24448 heartbeater.cc:344] Connected to a master server at 127.23.147.254:37725
I20260812 08:03:02.377041 24448 heartbeater.cc:461] Registering TS with master...
I20260812 08:03:02.377516 24448 heartbeater.cc:507] Master 127.23.147.254:37725 requested a full tablet report, sending...
I20260812 08:03:02.378980 24192 ts_manager.cc:194] Registered new tserver with Master: 2ecbf965a2194794b143f84621a1b0d0 (127.23.147.193:34357)
I20260812 08:03:02.379834 24143 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.016752886s
I20260812 08:03:02.380266 24192 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41686
I20260812 08:03:02.389430 24192 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41700:
name: "full-stack-mrs-test-tbl"
schema {
  columns {
    name: "key"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int32_val1"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int32_val2"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int32_val3"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int32_val4"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int64_val1"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int64_val2"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int64_val3"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int64_val4"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20260812 08:03:02.403118 24386 tablet_service.cc:1511] Processing CreateTablet for tablet a4099d9a819c45f4bfc3f492de85b0ac (DEFAULT_TABLE table=full-stack-mrs-test-tbl [id=833d67f6cff44049ba5d23ab6121c54d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:03:02.403649 24386 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4099d9a819c45f4bfc3f492de85b0ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:03:02.405918 24466 tablet_bootstrap.cc:492] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0: Bootstrap starting.
I20260812 08:03:02.406802 24466 tablet_bootstrap.cc:654] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0: Neither blocks nor log segments found. Creating new log.
I20260812 08:03:02.408058 24466 tablet_bootstrap.cc:492] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0: No bootstrap required, opened a new log
I20260812 08:03:02.408159 24466 ts_tablet_manager.cc:1403] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:03:02.408532 24466 raft_consensus.cc:359] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecbf965a2194794b143f84621a1b0d0" member_type: VOTER last_known_addr { host: "127.23.147.193" port: 34357 } }
I20260812 08:03:02.408635 24466 raft_consensus.cc:385] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:03:02.408667 24466 raft_consensus.cc:740] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecbf965a2194794b143f84621a1b0d0, State: Initialized, Role: FOLLOWER
I20260812 08:03:02.408797 24466 consensus_queue.cc:260] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecbf965a2194794b143f84621a1b0d0" member_type: VOTER last_known_addr { host: "127.23.147.193" port: 34357 } }
I20260812 08:03:02.408869 24466 raft_consensus.cc:399] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 08:03:02.408934 24466 raft_consensus.cc:493] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 08:03:02.408984 24466 raft_consensus.cc:3060] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:03:02.409655 24466 raft_consensus.cc:515] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecbf965a2194794b143f84621a1b0d0" member_type: VOTER last_known_addr { host: "127.23.147.193" port: 34357 } }
I20260812 08:03:02.409792 24466 leader_election.cc:304] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ecbf965a2194794b143f84621a1b0d0; no voters: 
I20260812 08:03:02.409978 24466 leader_election.cc:290] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 08:03:02.410080 24469 raft_consensus.cc:2804] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:03:02.410250 24469 raft_consensus.cc:697] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0 [term 1 LEADER]: Becoming Leader. State: Replica: 2ecbf965a2194794b143f84621a1b0d0, State: Running, Role: LEADER
I20260812 08:03:02.410318 24466 ts_tablet_manager.cc:1434] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:03:02.410825 24448 heartbeater.cc:499] Master 127.23.147.254:37725 was elected leader, sending a full tablet report...
I20260812 08:03:02.410854 24469 consensus_queue.cc:237] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecbf965a2194794b143f84621a1b0d0" member_type: VOTER last_known_addr { host: "127.23.147.193" port: 34357 } }
I20260812 08:03:02.413702 24192 catalog_manager.cc:5719] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ecbf965a2194794b143f84621a1b0d0 (127.23.147.193). New cstate: current_term: 1 leader_uuid: "2ecbf965a2194794b143f84621a1b0d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ecbf965a2194794b143f84621a1b0d0" member_type: VOTER last_known_addr { host: "127.23.147.193" port: 34357 } health_report { overall_health: HEALTHY } } }
I20260812 08:03:02.682459 24477 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 9% done.
I20260812 08:03:02.920576 24477 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 19% done.
I20260812 08:03:02.989953 24482 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 9% done.
I20260812 08:03:02.990989 24492 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 9% done.
I20260812 08:03:03.023036 24483 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 9% done.
I20260812 08:03:03.038859 24485 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 9% done.
I20260812 08:03:03.064640 24491 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 9% done.
I20260812 08:03:03.089313 24488 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 9% done.
I20260812 08:03:03.207890 24478 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 9% done.
I20260812 08:03:03.237547 24480 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 9% done.
I20260812 08:03:03.262863 24490 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 9% done.
I20260812 08:03:03.468039 24492 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 19% done.
I20260812 08:03:03.501838 24482 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 19% done.
I20260812 08:03:03.608582 24478 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 19% done.
I20260812 08:03:03.770524 24491 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 19% done.
W20260812 08:03:03.782467 24457 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:34357, user_credentials={real_user=slave}} blocked reactor thread for 47156us
I20260812 08:03:03.858769 24483 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 19% done.
I20260812 08:03:03.917587 24480 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 19% done.
I20260812 08:03:03.928187 24485 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 19% done.
I20260812 08:03:03.949863 24477 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 29% done.
I20260812 08:03:03.969255 24488 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 19% done.
I20260812 08:03:04.019800 24490 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 19% done.
I20260812 08:03:04.039764 24482 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 29% done.
I20260812 08:03:04.177513 24483 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 29% done.
I20260812 08:03:04.220346 24504 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=16, attempt_no=0}) took 1007 ms. Trace:
I20260812 08:03:04.220508 24504 rpcz_store.cc:276] 0812 08:03:03.212916 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:03.212962 (+    46us) service_pool.cc:224] Handling call
0812 08:03:04.220320 (+1007358us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:03.841480 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:03.841659 (+   179us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:03.841664 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:03.841666 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:03.970640 (+128974us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:03.970653 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:03.970659 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:04.030099 (+ 59440us) tablet.cc:701] Row locks acquired
0812 08:03:04.030103 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:04.030187 (+    84us) write_op.cc:270] Start()
0812 08:03:04.030235 (+    48us) write_op.cc:276] Timestamp: P: 1786521784030167 usec, L: 0
0812 08:03:04.030237 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:04.030623 (+   386us) log.cc:844] Serialized 486351 byte log entry
0812 08:03:04.047896 (+ 17273us) op_driver.cc:464] REPLICATION: finished
0812 08:03:04.048116 (+   220us) write_op.cc:301] APPLY: starting
0812 08:03:04.048133 (+    17us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:04.054004 (+  5871us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:04.054025 (+    21us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:04.211735 (+157710us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:04.214025 (+  2290us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:04.214032 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:04.218676 (+  4644us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:04.218987 (+   311us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:04.219958 (+   971us) write_op.cc:454] Released schema lock
0812 08:03:04.220203 (+   245us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":172,"num_ops":5000,"prepare.queue_time_us":627035,"prepare.run_cpu_time_us":50823,"prepare.run_wall_time_us":206362,"replication_time_us":17621,"spinlock_wait_cycles":20224}]]}
I20260812 08:03:04.248350 24492 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 29% done.
I20260812 08:03:04.331266 24508 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=17, attempt_no=0}) took 1054 ms. Trace:
I20260812 08:03:04.331403 24508 rpcz_store.cc:276] 0812 08:03:03.276476 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:03.276547 (+    71us) service_pool.cc:224] Handling call
0812 08:03:04.331237 (+1054690us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:04.047893 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:04.048055 (+   162us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:04.048059 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:03:04.048061 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:04.105625 (+ 57564us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:04.105640 (+    15us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:04.105648 (+     8us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:04.137536 (+ 31888us) tablet.cc:701] Row locks acquired
0812 08:03:04.137539 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:04.137630 (+    91us) write_op.cc:270] Start()
0812 08:03:04.137677 (+    47us) write_op.cc:276] Timestamp: P: 1786521784137606 usec, L: 0
0812 08:03:04.137680 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:04.138049 (+   369us) log.cc:844] Serialized 486570 byte log entry
0812 08:03:04.140556 (+  2507us) op_driver.cc:464] REPLICATION: finished
0812 08:03:04.140653 (+    97us) write_op.cc:301] APPLY: starting
0812 08:03:04.140668 (+    15us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:04.146695 (+  6027us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:04.146726 (+    31us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:04.311040 (+164314us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:04.313122 (+  2082us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:04.313129 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:04.329548 (+ 16419us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:04.329638 (+    90us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:04.330860 (+  1222us) write_op.cc:454] Released schema lock
0812 08:03:04.331110 (+   250us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":56,"mutex_wait_us":1247,"num_ops":5000,"prepare.queue_time_us":770589,"prepare.run_cpu_time_us":51117,"prepare.run_wall_time_us":90266,"replication_time_us":2843,"spinlock_wait_cycles":70400}]]}
I20260812 08:03:04.365995 24509 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=18, attempt_no=0}) took 1027 ms. Trace:
I20260812 08:03:04.366204 24509 rpcz_store.cc:276] 0812 08:03:03.338297 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:03.338364 (+    67us) service_pool.cc:224] Handling call
0812 08:03:04.365970 (+1027606us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:04.138209 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:04.138376 (+   167us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:04.138380 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:03:04.138382 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:04.167788 (+ 29406us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:04.167801 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:04.167808 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:04.195702 (+ 27894us) tablet.cc:701] Row locks acquired
0812 08:03:04.195705 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:04.195792 (+    87us) write_op.cc:270] Start()
0812 08:03:04.195838 (+    46us) write_op.cc:276] Timestamp: P: 1786521784195768 usec, L: 0
0812 08:03:04.195840 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:04.196321 (+   481us) log.cc:844] Serialized 485466 byte log entry
0812 08:03:04.198571 (+  2250us) op_driver.cc:464] REPLICATION: finished
0812 08:03:04.213092 (+ 14521us) write_op.cc:301] APPLY: starting
0812 08:03:04.213123 (+    31us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:04.219286 (+  6163us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:04.219312 (+    26us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:04.357382 (+138070us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:04.359590 (+  2208us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:04.359598 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:04.364149 (+  4551us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:04.364448 (+   299us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:04.365601 (+  1153us) write_op.cc:454] Released schema lock
0812 08:03:04.365858 (+   257us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":14432,"mutex_wait_us":11086,"num_ops":5000,"prepare.queue_time_us":798157,"prepare.run_cpu_time_us":50502,"prepare.run_wall_time_us":58202,"replication_time_us":2703,"spinlock_wait_cycles":57216,"thread_start_us":96,"threads_started":1}]]}
I20260812 08:03:04.491679 24504 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=19, attempt_no=0}) took 1144 ms. Trace:
I20260812 08:03:04.491902 24504 rpcz_store.cc:276] 0812 08:03:03.347160 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:03.347212 (+    52us) service_pool.cc:224] Handling call
0812 08:03:04.491651 (+1144439us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:04.196459 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:04.196620 (+   161us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:04.196625 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:04.196626 (+     1us) tablet.cc:662] Decoding operations
0812 08:03:04.251437 (+ 54811us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:04.251449 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:04.251456 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:04.271624 (+ 20168us) tablet.cc:701] Row locks acquired
0812 08:03:04.271627 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:04.271712 (+    85us) write_op.cc:270] Start()
0812 08:03:04.271756 (+    44us) write_op.cc:276] Timestamp: P: 1786521784271691 usec, L: 0
0812 08:03:04.271759 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:04.272827 (+  1068us) log.cc:844] Serialized 486155 byte log entry
0812 08:03:04.275150 (+  2323us) op_driver.cc:464] REPLICATION: finished
0812 08:03:04.275233 (+    83us) write_op.cc:301] APPLY: starting
0812 08:03:04.275250 (+    17us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:04.281308 (+  6058us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:04.281329 (+    21us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:04.483183 (+201854us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:04.485330 (+  2147us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:04.485337 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:04.489872 (+  4535us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:04.490227 (+   355us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:04.491273 (+  1046us) write_op.cc:454] Released schema lock
0812 08:03:04.491525 (+   252us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":33,"mutex_wait_us":15105,"num_ops":5000,"prepare.queue_time_us":847839,"prepare.run_cpu_time_us":50169,"prepare.run_wall_time_us":76477,"replication_time_us":3360,"spinlock_wait_cycles":71040}]]}
I20260812 08:03:04.533758 24485 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 29% done.
I20260812 08:03:04.546469 24511 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=20, attempt_no=0}) took 1180 ms. Trace:
I20260812 08:03:04.547519 24511 rpcz_store.cc:276] 0812 08:03:03.366230 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:03.366291 (+    61us) service_pool.cc:224] Handling call
0812 08:03:04.546445 (+1180154us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:04.272984 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:04.273154 (+   170us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:04.273159 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:04.273161 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:04.301672 (+ 28511us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:04.301685 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:04.301692 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:04.325913 (+ 24221us) tablet.cc:701] Row locks acquired
0812 08:03:04.325917 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:04.326012 (+    95us) write_op.cc:270] Start()
0812 08:03:04.326056 (+    44us) write_op.cc:276] Timestamp: P: 1786521784325992 usec, L: 0
0812 08:03:04.326059 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:04.326400 (+   341us) log.cc:844] Serialized 486090 byte log entry
0812 08:03:04.328734 (+  2334us) op_driver.cc:464] REPLICATION: finished
0812 08:03:04.341053 (+ 12319us) write_op.cc:301] APPLY: starting
0812 08:03:04.341072 (+    19us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:04.346904 (+  5832us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:04.346927 (+    23us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:04.530467 (+183540us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:04.532583 (+  2116us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:04.532591 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:04.545053 (+ 12462us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:04.545134 (+    81us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:04.546078 (+   944us) write_op.cc:454] Released schema lock
0812 08:03:04.546330 (+   252us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":12241,"num_ops":5000,"prepare.queue_time_us":905403,"prepare.run_cpu_time_us":53261,"prepare.run_wall_time_us":55993,"replication_time_us":2641,"spinlock_wait_cycles":32000,"thread_start_us":105,"threads_started":1}]]}
I20260812 08:03:04.590718 24491 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 29% done.
I20260812 08:03:04.655948 24488 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 29% done.
I20260812 08:03:04.725032 24504 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=22, attempt_no=0}) took 1011 ms. Trace:
I20260812 08:03:04.736938 24504 rpcz_store.cc:276] 0812 08:03:03.713822 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:03.713869 (+    47us) service_pool.cc:224] Handling call
0812 08:03:04.725005 (+1011136us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:04.437553 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:04.437715 (+   162us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:04.437721 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:03:04.437723 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:04.501458 (+ 63735us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:04.501483 (+    25us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:04.501490 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:04.541324 (+ 39834us) tablet.cc:701] Row locks acquired
0812 08:03:04.541328 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:04.541409 (+    81us) write_op.cc:270] Start()
0812 08:03:04.541454 (+    45us) write_op.cc:276] Timestamp: P: 1786521784541388 usec, L: 0
0812 08:03:04.541456 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:04.541812 (+   356us) log.cc:844] Serialized 486029 byte log entry
0812 08:03:04.544023 (+  2211us) op_driver.cc:464] REPLICATION: finished
0812 08:03:04.544116 (+    93us) write_op.cc:301] APPLY: starting
0812 08:03:04.544130 (+    14us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:04.558995 (+ 14865us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:04.559016 (+    21us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:04.704667 (+145651us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:04.706812 (+  2145us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:04.706818 (+     6us) write_op.cc:312] APPLY: finished
0812 08:03:04.723355 (+ 16537us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:04.723457 (+   102us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:04.724579 (+  1122us) write_op.cc:454] Released schema lock
0812 08:03:04.724853 (+   274us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":55,"num_ops":5000,"prepare.queue_time_us":722955,"prepare.run_cpu_time_us":50209,"prepare.run_wall_time_us":104350,"replication_time_us":2537,"spinlock_wait_cycles":16512}]]}
I20260812 08:03:04.756783 24509 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=21, attempt_no=0}) took 1128 ms. Trace:
I20260812 08:03:04.772941 24509 rpcz_store.cc:276] 0812 08:03:03.628509 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:03.628569 (+    60us) service_pool.cc:224] Handling call
0812 08:03:04.756758 (+1128189us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:04.329004 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:04.329188 (+   184us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:04.329194 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:03:04.329196 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:04.389322 (+ 60126us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:04.389334 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:04.389340 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:04.424879 (+ 35539us) tablet.cc:701] Row locks acquired
0812 08:03:04.424882 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:04.436994 (+ 12112us) write_op.cc:270] Start()
0812 08:03:04.437044 (+    50us) write_op.cc:276] Timestamp: P: 1786521784436968 usec, L: 0
0812 08:03:04.437046 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:04.437399 (+   353us) log.cc:844] Serialized 486383 byte log entry
0812 08:03:04.439687 (+  2288us) op_driver.cc:464] REPLICATION: finished
0812 08:03:04.439780 (+    93us) write_op.cc:301] APPLY: starting
0812 08:03:04.439793 (+    13us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:04.445757 (+  5964us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:04.445779 (+    22us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:04.697142 (+251363us) spinlock_profiling.cc:243] Waited 3.07 ms on lock 0x55a84f8d1898. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facf3c2fbdd 00007facfc1cfe3b 00007facfc0998df 00007facfc09e60a 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d
0812 08:03:04.732193 (+ 35051us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:04.734354 (+  2161us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:04.734361 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:04.755013 (+ 20652us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:04.755333 (+   320us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:04.756381 (+  1048us) write_op.cc:454] Released schema lock
0812 08:03:04.756632 (+   251us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":55,"num_ops":5000,"prepare.queue_time_us":698827,"prepare.run_cpu_time_us":49262,"prepare.run_wall_time_us":108487,"replication_time_us":2611,"spinlock_wait_cycles":7133440}]]}
I20260812 08:03:04.851064 24477 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 39% done.
I20260812 08:03:04.865476 24480 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 29% done.
I20260812 08:03:04.897568 24478 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 29% done.
I20260812 08:03:04.921644 24490 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 29% done.
I20260812 08:03:05.073242 24511 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=24, attempt_no=0}) took 1062 ms. Trace:
I20260812 08:03:05.080960 24511 rpcz_store.cc:276] 0812 08:03:04.011242 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:04.011301 (+    59us) service_pool.cc:224] Handling call
0812 08:03:05.073214 (+1061913us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:04.708939 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:04.709119 (+   180us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:04.709125 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:03:04.709127 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:04.811733 (+102606us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:04.811745 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:04.811752 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:04.855305 (+ 43553us) tablet.cc:701] Row locks acquired
0812 08:03:04.855309 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:04.855395 (+    86us) write_op.cc:270] Start()
0812 08:03:04.855442 (+    47us) write_op.cc:276] Timestamp: P: 1786521784855373 usec, L: 0
0812 08:03:04.855444 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:04.855827 (+   383us) log.cc:844] Serialized 485953 byte log entry
0812 08:03:04.858212 (+  2385us) op_driver.cc:464] REPLICATION: finished
0812 08:03:04.858308 (+    96us) write_op.cc:301] APPLY: starting
0812 08:03:04.858326 (+    18us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:04.864700 (+  6374us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:04.864719 (+    19us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:05.049157 (+184438us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:05.051153 (+  1996us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:05.051159 (+     6us) write_op.cc:312] APPLY: finished
0812 08:03:05.071649 (+ 20490us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:05.071775 (+   126us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:05.072784 (+  1009us) write_op.cc:454] Released schema lock
0812 08:03:05.073077 (+   293us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":34,"mutex_wait_us":12210,"num_ops":5000,"prepare.queue_time_us":696921,"prepare.run_cpu_time_us":47401,"prepare.run_wall_time_us":146973,"replication_time_us":2738,"spinlock_wait_cycles":7680}]]}
I20260812 08:03:05.089149 24482 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 39% done.
I20260812 08:03:05.097690 24492 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 39% done.
I20260812 08:03:05.129210 24509 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=25, attempt_no=0}) took 1015 ms. Trace:
I20260812 08:03:05.129416 24509 rpcz_store.cc:276] 0812 08:03:04.113413 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:04.113479 (+    66us) service_pool.cc:224] Handling call
0812 08:03:05.129188 (+1015709us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:04.855963 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:04.856125 (+   162us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:04.856130 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:04.856132 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:04.882690 (+ 26558us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:04.882704 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:04.882711 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:04.902585 (+ 19874us) tablet.cc:701] Row locks acquired
0812 08:03:04.902588 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:04.902665 (+    77us) write_op.cc:270] Start()
0812 08:03:04.902710 (+    45us) write_op.cc:276] Timestamp: P: 1786521784902650 usec, L: 0
0812 08:03:04.902712 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:04.903052 (+   340us) log.cc:844] Serialized 485996 byte log entry
0812 08:03:04.905264 (+  2212us) op_driver.cc:464] REPLICATION: finished
0812 08:03:04.905351 (+    87us) write_op.cc:301] APPLY: starting
0812 08:03:04.905366 (+    15us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:04.911195 (+  5829us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:04.911215 (+    20us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:05.121084 (+209869us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:05.123321 (+  2237us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:05.123330 (+     9us) write_op.cc:312] APPLY: finished
0812 08:03:05.127769 (+  4439us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:05.127843 (+    74us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:05.128795 (+   952us) write_op.cc:454] Released schema lock
0812 08:03:05.129072 (+   277us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":34,"num_ops":5000,"prepare.queue_time_us":741674,"prepare.run_cpu_time_us":47085,"prepare.run_wall_time_us":47172,"replication_time_us":2520,"spinlock_wait_cycles":43776}]]}
I20260812 08:03:05.231431 24483 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 39% done.
I20260812 08:03:05.376206 24485 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 39% done.
I20260812 08:03:05.494171 24504 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=28, attempt_no=0}) took 1091 ms. Trace:
I20260812 08:03:05.494293 24504 rpcz_store.cc:276] 0812 08:03:04.402312 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:04.402361 (+    49us) service_pool.cc:224] Handling call
0812 08:03:05.494146 (+1091785us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:05.125562 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:05.125726 (+   164us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:05.125731 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:05.125732 (+     1us) tablet.cc:662] Decoding operations
0812 08:03:05.155497 (+ 29765us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:05.155512 (+    15us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:05.155518 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:05.176307 (+ 20789us) tablet.cc:701] Row locks acquired
0812 08:03:05.176310 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:05.176390 (+    80us) write_op.cc:270] Start()
0812 08:03:05.176435 (+    45us) write_op.cc:276] Timestamp: P: 1786521785176371 usec, L: 0
0812 08:03:05.176437 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:05.176751 (+   314us) log.cc:844] Serialized 486327 byte log entry
0812 08:03:05.179215 (+  2464us) op_driver.cc:464] REPLICATION: finished
0812 08:03:05.179307 (+    92us) write_op.cc:301] APPLY: starting
0812 08:03:05.179325 (+    18us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:05.185232 (+  5907us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:05.185254 (+    22us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:05.474313 (+289059us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:05.476559 (+  2246us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:05.476566 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:05.492592 (+ 16026us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:05.492701 (+   109us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:05.493760 (+  1059us) write_op.cc:454] Released schema lock
0812 08:03:05.494015 (+   255us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":52,"mutex_wait_us":18770,"num_ops":5000,"prepare.queue_time_us":722650,"prepare.run_cpu_time_us":50682,"prepare.run_wall_time_us":51270,"replication_time_us":2746,"spinlock_wait_cycles":40704}]]}
I20260812 08:03:05.498342 24491 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 39% done.
I20260812 08:03:05.568074 24509 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=29, attempt_no=0}) took 1039 ms. Trace:
I20260812 08:03:05.585911 24509 rpcz_store.cc:276] 0812 08:03:04.528335 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:04.528403 (+    68us) service_pool.cc:224] Handling call
0812 08:03:05.568048 (+1039645us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:05.176876 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:05.177061 (+   185us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:05.177066 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:05.177068 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:05.207257 (+ 30189us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:05.207270 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:05.207277 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:05.228434 (+ 21157us) tablet.cc:701] Row locks acquired
0812 08:03:05.228438 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:05.228531 (+    93us) write_op.cc:270] Start()
0812 08:03:05.228593 (+    62us) write_op.cc:276] Timestamp: P: 1786521785228506 usec, L: 0
0812 08:03:05.228595 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:05.229010 (+   415us) log.cc:844] Serialized 486293 byte log entry
0812 08:03:05.231549 (+  2539us) op_driver.cc:464] REPLICATION: finished
0812 08:03:05.231645 (+    96us) write_op.cc:301] APPLY: starting
0812 08:03:05.231663 (+    18us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:05.237520 (+  5857us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:05.237540 (+    20us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:05.395231 (+157691us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x55a84fd49c0c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:05.551032 (+155801us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:05.561836 (+ 10804us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:05.561845 (+     9us) write_op.cc:312] APPLY: finished
0812 08:03:05.566428 (+  4583us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:05.566541 (+   113us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:05.567663 (+  1122us) write_op.cc:454] Released schema lock
0812 08:03:05.567917 (+   254us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":40,"num_ops":5000,"prepare.queue_time_us":647677,"prepare.run_cpu_time_us":50741,"prepare.run_wall_time_us":52221,"replication_time_us":2921,"spinlock_wait_cycles":29309952}]]}
I20260812 08:03:05.587517 24488 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 39% done.
I20260812 08:03:05.718065 24490 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 39% done.
I20260812 08:03:05.783586 24478 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 39% done.
I20260812 08:03:05.859709 24477 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 49% done.
I20260812 08:03:05.911615 24511 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=33, attempt_no=0}) took 1005 ms. Trace:
I20260812 08:03:05.911827 24511 rpcz_store.cc:276] 0812 08:03:04.905896 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:04.905965 (+    69us) service_pool.cc:224] Handling call
0812 08:03:05.911588 (+1005623us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:05.589753 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:05.589916 (+   163us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:05.589920 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:03:05.589922 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:05.687426 (+ 97504us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:05.687441 (+    15us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:05.687448 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:05.732076 (+ 44628us) tablet.cc:701] Row locks acquired
0812 08:03:05.732079 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:05.732176 (+    97us) write_op.cc:270] Start()
0812 08:03:05.732227 (+    51us) write_op.cc:276] Timestamp: P: 1786521785732152 usec, L: 0
0812 08:03:05.732230 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:05.732788 (+   558us) log.cc:844] Serialized 485769 byte log entry
0812 08:03:05.735240 (+  2452us) op_driver.cc:464] REPLICATION: finished
0812 08:03:05.735334 (+    94us) write_op.cc:301] APPLY: starting
0812 08:03:05.735352 (+    18us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:05.741236 (+  5884us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:05.741261 (+    25us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:05.902075 (+160814us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:05.904291 (+  2216us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:05.904300 (+     9us) write_op.cc:312] APPLY: finished
0812 08:03:05.909675 (+  5375us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:05.910015 (+   340us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:05.911205 (+  1190us) write_op.cc:454] Released schema lock
0812 08:03:05.911461 (+   256us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":51,"mutex_wait_us":10327,"num_ops":5000,"prepare.queue_time_us":683164,"prepare.run_cpu_time_us":49804,"prepare.run_wall_time_us":143120,"replication_time_us":2980,"spinlock_wait_cycles":38912}]]}
I20260812 08:03:05.915119 24509 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=32, attempt_no=0}) took 1052 ms. Trace:
I20260812 08:03:05.923718 24509 rpcz_store.cc:276] 0812 08:03:04.862560 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:04.862614 (+    54us) service_pool.cc:224] Handling call
0812 08:03:05.915086 (+1052472us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:05.382071 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:05.393073 (+ 11002us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x55a84fd49c0c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc16a88a 00007facfc155bc3 00007facfc1554cc 00007facfc154d9a 00007facfc15a72a 000055a83d00075d 00007facf8700326 00007facf8700c4e
0812 08:03:05.393090 (+    17us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:05.393096 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:03:05.393098 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:05.529673 (+136575us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:05.529688 (+    15us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:05.529695 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:05.589213 (+ 59518us) tablet.cc:701] Row locks acquired
0812 08:03:05.589216 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:05.589280 (+    64us) write_op.cc:270] Start()
0812 08:03:05.589333 (+    53us) write_op.cc:276] Timestamp: P: 1786521785589266 usec, L: 0
0812 08:03:05.589336 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:05.589640 (+   304us) log.cc:844] Serialized 486407 byte log entry
0812 08:03:05.593041 (+  3401us) op_driver.cc:464] REPLICATION: finished
0812 08:03:05.594252 (+  1211us) write_op.cc:301] APPLY: starting
0812 08:03:05.594272 (+    20us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:05.600089 (+  5817us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:05.600092 (+     3us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:05.899091 (+298999us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:05.909136 (+ 10045us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:05.909144 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:05.913669 (+  4525us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:05.913748 (+    79us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:05.914750 (+  1002us) write_op.cc:454] Released schema lock
0812 08:03:05.914998 (+   248us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1122,"num_ops":5000,"prepare.queue_time_us":518752,"prepare.run_cpu_time_us":49356,"prepare.run_wall_time_us":207641,"replication_time_us":3674,"spinlock_wait_cycles":24602112}]]}
I20260812 08:03:05.947090 24482 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 49% done.
I20260812 08:03:05.986066 24480 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 39% done.
I20260812 08:03:06.418033 24492 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 49% done.
I20260812 08:03:06.456123 24483 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 49% done.
I20260812 08:03:06.504374 24485 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 49% done.
I20260812 08:03:06.541424 24511 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=38, attempt_no=0}) took 1002 ms. Trace:
I20260812 08:03:06.542047 24511 rpcz_store.cc:276] 0812 08:03:05.538684 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:05.538734 (+    50us) service_pool.cc:224] Handling call
0812 08:03:06.541395 (+1002661us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:06.056175 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:06.056345 (+   170us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:06.056351 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:03:06.056352 (+     1us) tablet.cc:662] Decoding operations
0812 08:03:06.085479 (+ 29127us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:06.085493 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:06.085500 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:06.113120 (+ 27620us) tablet.cc:701] Row locks acquired
0812 08:03:06.113124 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:06.113209 (+    85us) write_op.cc:270] Start()
0812 08:03:06.113252 (+    43us) write_op.cc:276] Timestamp: P: 1786521786113187 usec, L: 0
0812 08:03:06.113254 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:06.113581 (+   327us) log.cc:844] Serialized 485463 byte log entry
0812 08:03:06.115952 (+  2371us) op_driver.cc:464] REPLICATION: finished
0812 08:03:06.116057 (+   105us) write_op.cc:301] APPLY: starting
0812 08:03:06.116073 (+    16us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:06.121981 (+  5908us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:06.122002 (+    21us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:06.177070 (+ 55068us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x55a84fd49c0c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:06.421125 (+244055us) spinlock_profiling.cc:243] Waited 7.92 ms on lock 0x55a84fd49c0c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:06.510036 (+ 88911us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:06.511896 (+  1860us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:06.511903 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:06.515881 (+  3978us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:06.516006 (+   125us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:06.541010 (+ 25004us) write_op.cc:454] Released schema lock
0812 08:03:06.541262 (+   252us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":64,"num_ops":5000,"prepare.queue_time_us":516702,"prepare.run_cpu_time_us":57494,"prepare.run_wall_time_us":57497,"replication_time_us":2667,"spinlock_wait_cycles":61561472}]]}
I20260812 08:03:06.553124 24491 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 49% done.
I20260812 08:03:06.583256 24488 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 49% done.
I20260812 08:03:06.603035 24522 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=39, attempt_no=0}) took 1041 ms. Trace:
I20260812 08:03:06.603238 24522 rpcz_store.cc:276] 0812 08:03:05.561927 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:05.561977 (+    50us) service_pool.cc:224] Handling call
0812 08:03:06.603011 (+1041034us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:06.113727 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:06.113889 (+   162us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:06.113894 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:06.113896 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:06.143238 (+ 29342us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:06.143250 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:06.143257 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:06.163944 (+ 20687us) tablet.cc:701] Row locks acquired
0812 08:03:06.163947 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:06.164039 (+    92us) write_op.cc:270] Start()
0812 08:03:06.164248 (+   209us) write_op.cc:276] Timestamp: P: 1786521786164016 usec, L: 0
0812 08:03:06.164252 (+     4us) op_driver.cc:348] REPLICATION: starting
0812 08:03:06.164624 (+   372us) log.cc:844] Serialized 486487 byte log entry
0812 08:03:06.167035 (+  2411us) op_driver.cc:464] REPLICATION: finished
0812 08:03:06.179929 (+ 12894us) write_op.cc:301] APPLY: starting
0812 08:03:06.179948 (+    19us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:06.233256 (+ 53308us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:06.233280 (+    24us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:06.423113 (+189833us) spinlock_profiling.cc:243] Waited 9.86 ms on lock 0x55a84fd49c0c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:06.594513 (+171400us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:06.596644 (+  2131us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:06.596652 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:06.601255 (+  4603us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:06.601602 (+   347us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:06.602658 (+  1056us) write_op.cc:454] Released schema lock
0812 08:03:06.602899 (+   241us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":12811,"num_ops":5000,"prepare.queue_time_us":551117,"prepare.run_cpu_time_us":50757,"prepare.run_wall_time_us":50988,"replication_time_us":2753,"spinlock_wait_cycles":22683008,"thread_start_us":85,"threads_started":1}]]}
I20260812 08:03:06.825819 24504 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=41, attempt_no=0}) took 1116 ms. Trace:
I20260812 08:03:06.825951 24504 rpcz_store.cc:276] 0812 08:03:05.708914 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:05.708973 (+    59us) service_pool.cc:224] Handling call
0812 08:03:06.825792 (+1116819us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:06.215963 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:06.216242 (+   279us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:06.216248 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:03:06.216250 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:06.257550 (+ 41300us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:06.257565 (+    15us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:06.257571 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:06.302239 (+ 44668us) tablet.cc:701] Row locks acquired
0812 08:03:06.302243 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:06.302314 (+    71us) write_op.cc:270] Start()
0812 08:03:06.302356 (+    42us) write_op.cc:276] Timestamp: P: 1786521786302301 usec, L: 0
0812 08:03:06.302359 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:06.302692 (+   333us) log.cc:844] Serialized 485892 byte log entry
0812 08:03:06.304973 (+  2281us) op_driver.cc:464] REPLICATION: finished
0812 08:03:06.405222 (+100249us) write_op.cc:301] APPLY: starting
0812 08:03:06.405241 (+    19us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:06.410985 (+  5744us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:06.411011 (+    26us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:06.795079 (+384068us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:06.797288 (+  2209us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:06.797295 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:06.824000 (+ 26705us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:06.824086 (+    86us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:06.825426 (+  1340us) write_op.cc:454] Released schema lock
0812 08:03:06.825681 (+   255us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":100196,"num_ops":5000,"prepare.queue_time_us":506316,"prepare.run_cpu_time_us":50643,"prepare.run_wall_time_us":86812,"replication_time_us":2578,"spinlock_wait_cycles":26240}]]}
W20260812 08:03:06.889046 24457 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:34357, user_credentials={real_user=slave}} blocked reactor thread for 52824.9us
I20260812 08:03:07.044497 24478 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 49% done.
I20260812 08:03:07.096735 24490 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 49% done.
I20260812 08:03:07.109396 24480 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 49% done.
W20260812 08:03:07.147639 24457 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:34357, user_credentials={real_user=slave}} blocked reactor thread for 51861.8us
I20260812 08:03:07.212723 24477 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 59% done.
I20260812 08:03:07.279846 24504 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=45, attempt_no=0}) took 1127 ms. Trace:
I20260812 08:03:07.279974 24504 rpcz_store.cc:276] 0812 08:03:06.152421 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:06.152488 (+    67us) service_pool.cc:224] Handling call
0812 08:03:07.279822 (+1127334us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:06.614153 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:06.614318 (+   165us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:06.614323 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:06.614325 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:06.641871 (+ 27546us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:06.641882 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:06.641886 (+     4us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:06.662253 (+ 20367us) tablet.cc:701] Row locks acquired
0812 08:03:06.662257 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:06.662357 (+   100us) write_op.cc:270] Start()
0812 08:03:06.662401 (+    44us) write_op.cc:276] Timestamp: P: 1786521786662330 usec, L: 0
0812 08:03:06.662404 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:06.662723 (+   319us) log.cc:844] Serialized 485235 byte log entry
0812 08:03:06.665005 (+  2282us) op_driver.cc:464] REPLICATION: finished
0812 08:03:06.833564 (+168559us) write_op.cc:301] APPLY: starting
0812 08:03:06.833583 (+    19us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:06.839542 (+  5959us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:06.839565 (+    23us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:07.061031 (+221466us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x55a84fd49c0c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:07.223315 (+162284us) spinlock_profiling.cc:243] Waited 14.2 ms on lock 0x55a84fd49c0c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:07.247323 (+ 24008us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:07.249526 (+  2203us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:07.249533 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:07.278252 (+ 28719us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:07.278367 (+   115us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:07.279425 (+  1058us) write_op.cc:454] Released schema lock
0812 08:03:07.279698 (+   273us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":168506,"mutex_wait_us":1128,"num_ops":5000,"prepare.queue_time_us":460930,"prepare.run_cpu_time_us":48652,"prepare.run_wall_time_us":48653,"replication_time_us":2573,"spinlock_wait_cycles":73311232}]]}
I20260812 08:03:07.315579 24482 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 59% done.
I20260812 08:03:07.387401 24492 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 59% done.
W20260812 08:03:07.393865 24457 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:34357, user_credentials={real_user=slave}} blocked reactor thread for 54718.4us
W20260812 08:03:07.457818 24457 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:34357, user_credentials={real_user=slave}} blocked reactor thread for 63638.8us
I20260812 08:03:07.613565 24485 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 59% done.
I20260812 08:03:07.710882 24504 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=50, attempt_no=0}) took 1009 ms. Trace:
I20260812 08:03:07.724936 24504 rpcz_store.cc:276] 0812 08:03:06.701117 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:06.701181 (+    64us) service_pool.cc:224] Handling call
0812 08:03:07.710858 (+1009677us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:07.131298 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:07.131466 (+   168us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:07.131471 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:07.131472 (+     1us) tablet.cc:662] Decoding operations
0812 08:03:07.158539 (+ 27067us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:07.158552 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:07.158559 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:07.184311 (+ 25752us) tablet.cc:701] Row locks acquired
0812 08:03:07.184314 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:07.184399 (+    85us) write_op.cc:270] Start()
0812 08:03:07.184444 (+    45us) write_op.cc:276] Timestamp: P: 1786521787184376 usec, L: 0
0812 08:03:07.184447 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:07.184812 (+   365us) log.cc:844] Serialized 485650 byte log entry
0812 08:03:07.195090 (+ 10278us) op_driver.cc:464] REPLICATION: finished
0812 08:03:07.289498 (+ 94408us) write_op.cc:301] APPLY: starting
0812 08:03:07.289515 (+    17us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:07.295233 (+  5718us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:07.295256 (+    23us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:07.678579 (+383323us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:07.680693 (+  2114us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:07.680700 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:07.709298 (+ 28598us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:07.709416 (+   118us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:07.710460 (+  1044us) write_op.cc:454] Released schema lock
0812 08:03:07.710732 (+   272us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":94358,"mutex_wait_us":186,"num_ops":5000,"prepare.queue_time_us":429451,"prepare.run_cpu_time_us":48255,"prepare.run_wall_time_us":64157,"replication_time_us":10606,"spinlock_wait_cycles":11648}]]}
I20260812 08:03:07.744756 24483 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 59% done.
I20260812 08:03:07.865291 24491 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 59% done.
I20260812 08:03:07.870903 24488 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 59% done.
I20260812 08:03:07.942595 24490 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 59% done.
I20260812 08:03:08.016250 24478 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 59% done.
I20260812 08:03:08.065490 24504 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=54, attempt_no=0}) took 1035 ms. Trace:
I20260812 08:03:08.065620 24504 rpcz_store.cc:276] 0812 08:03:07.030150 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:07.030200 (+    50us) service_pool.cc:224] Handling call
0812 08:03:08.065468 (+1035268us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:07.397638 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:07.397797 (+   159us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:07.397802 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:07.397804 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:07.426223 (+ 28419us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:07.426234 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:07.426241 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:07.446345 (+ 20104us) tablet.cc:701] Row locks acquired
0812 08:03:07.446349 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:07.446436 (+    87us) write_op.cc:270] Start()
0812 08:03:07.446485 (+    49us) write_op.cc:276] Timestamp: P: 1786521787446414 usec, L: 0
0812 08:03:07.446487 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:07.446856 (+   369us) log.cc:844] Serialized 485801 byte log entry
0812 08:03:07.449750 (+  2894us) op_driver.cc:464] REPLICATION: finished
0812 08:03:07.737495 (+287745us) write_op.cc:301] APPLY: starting
0812 08:03:07.737516 (+    21us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:07.747781 (+ 10265us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:07.747806 (+    25us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:08.015386 (+267580us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:08.029462 (+ 14076us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:08.029470 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:08.033907 (+  4437us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:08.034008 (+   101us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:08.065082 (+ 31074us) write_op.cc:454] Released schema lock
0812 08:03:08.065327 (+   245us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":287691,"num_ops":5000,"prepare.queue_time_us":366818,"prepare.run_cpu_time_us":49302,"prepare.run_wall_time_us":49304,"replication_time_us":3234,"spinlock_wait_cycles":59520}]]}
I20260812 08:03:08.246847 24509 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=55, attempt_no=0}) took 1054 ms. Trace:
I20260812 08:03:08.247067 24509 rpcz_store.cc:276] 0812 08:03:07.191987 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:07.192061 (+    74us) service_pool.cc:224] Handling call
0812 08:03:08.246822 (+1054761us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:07.447005 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:07.447175 (+   170us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:07.447180 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:07.447182 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:07.475755 (+ 28573us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:07.475766 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:07.475772 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:07.495815 (+ 20043us) tablet.cc:701] Row locks acquired
0812 08:03:07.495818 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:07.495903 (+    85us) write_op.cc:270] Start()
0812 08:03:07.495945 (+    42us) write_op.cc:276] Timestamp: P: 1786521787495881 usec, L: 0
0812 08:03:07.495948 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:07.496275 (+   327us) log.cc:844] Serialized 485679 byte log entry
0812 08:03:07.498639 (+  2364us) op_driver.cc:464] REPLICATION: finished
0812 08:03:07.809519 (+310880us) write_op.cc:301] APPLY: starting
0812 08:03:07.809537 (+    18us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:07.839447 (+ 29910us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:07.839478 (+    31us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:07.916374 (+ 76896us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x55a84fd49c0c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:08.149073 (+232699us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x55a84fd49c0c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:08.190291 (+ 41218us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:08.192403 (+  2112us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:08.192411 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:08.241082 (+ 48671us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:08.245514 (+  4432us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:08.246444 (+   930us) write_op.cc:454] Released schema lock
0812 08:03:08.246711 (+   267us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":310826,"mutex_wait_us":2104,"num_ops":5000,"prepare.queue_time_us":254273,"prepare.run_cpu_time_us":49344,"prepare.run_wall_time_us":49358,"replication_time_us":2660,"spinlock_wait_cycles":56165888}]]}
I20260812 08:03:08.370488 24480 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 59% done.
I20260812 08:03:08.376490 24483 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 69% done.
I20260812 08:03:08.493857 24477 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 69% done.
I20260812 08:03:08.533296 24511 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=57, attempt_no=0}) took 1074 ms. Trace:
I20260812 08:03:08.540972 24511 rpcz_store.cc:276] 0812 08:03:07.458882 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:07.459654 (+   772us) service_pool.cc:224] Handling call
0812 08:03:08.533270 (+1073616us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:07.546411 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:07.546576 (+   165us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:07.546581 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:07.546583 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:07.575920 (+ 29337us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:07.575933 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:07.575939 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:07.595956 (+ 20017us) tablet.cc:701] Row locks acquired
0812 08:03:07.595959 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:07.596031 (+    72us) write_op.cc:270] Start()
0812 08:03:07.596075 (+    44us) write_op.cc:276] Timestamp: P: 1786521787596014 usec, L: 0
0812 08:03:07.596078 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:07.596392 (+   314us) log.cc:844] Serialized 486334 byte log entry
0812 08:03:07.598680 (+  2288us) op_driver.cc:464] REPLICATION: finished
0812 08:03:07.895722 (+297042us) write_op.cc:301] APPLY: starting
0812 08:03:07.895736 (+    14us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:07.901641 (+  5905us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:07.901669 (+    28us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:07.937040 (+ 35371us) spinlock_profiling.cc:243] Waited 35.2 ms on lock 0x55a84fd49c0c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:08.500621 (+563581us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:08.502827 (+  2206us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:08.502836 (+     9us) write_op.cc:312] APPLY: finished
0812 08:03:08.531617 (+ 28781us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:08.531737 (+   120us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:08.532844 (+  1107us) write_op.cc:454] Released schema lock
0812 08:03:08.533141 (+   297us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":296993,"mutex_wait_us":10358,"num_ops":5000,"prepare.queue_time_us":86230,"prepare.run_cpu_time_us":50039,"prepare.run_wall_time_us":50061,"replication_time_us":2575,"spinlock_wait_cycles":81154944}]]}
I20260812 08:03:08.602105 24485 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 69% done.
I20260812 08:03:08.639639 24478 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 69% done.
W20260812 08:03:08.653127 24457 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:34357, user_credentials={real_user=slave}} blocked reactor thread for 71899.5us
W20260812 08:03:08.958933 24457 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:34357, user_credentials={real_user=slave}} blocked reactor thread for 66773.6us
I20260812 08:03:09.005553 24482 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 69% done.
I20260812 08:03:09.024701 24490 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 69% done.
I20260812 08:03:09.041451 24483 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 79% done.
I20260812 08:03:09.134266 24492 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 69% done.
I20260812 08:03:09.164479 24480 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 69% done.
I20260812 08:03:09.198459 24511 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=63, attempt_no=0}) took 1242 ms. Trace:
I20260812 08:03:09.212940 24511 rpcz_store.cc:276] 0812 08:03:07.955734 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:07.955790 (+    56us) service_pool.cc:224] Handling call
0812 08:03:09.198430 (+1242640us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:08.156834 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:08.169085 (+ 12251us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:08.169091 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:03:08.169093 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:08.212380 (+ 43287us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:08.212393 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:08.212400 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:08.242804 (+ 30404us) tablet.cc:701] Row locks acquired
0812 08:03:08.242807 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:08.242886 (+    79us) write_op.cc:270] Start()
0812 08:03:08.242927 (+    41us) write_op.cc:276] Timestamp: P: 1786521788242868 usec, L: 0
0812 08:03:08.242929 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:08.243280 (+   351us) log.cc:844] Serialized 486012 byte log entry
0812 08:03:08.245408 (+  2128us) op_driver.cc:464] REPLICATION: finished
0812 08:03:08.565533 (+320125us) write_op.cc:301] APPLY: starting
0812 08:03:08.565549 (+    16us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:08.591523 (+ 25974us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:08.591551 (+    28us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:09.148429 (+556878us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:09.162474 (+ 14045us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:09.162482 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:09.187159 (+ 24677us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:09.197026 (+  9867us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:09.198033 (+  1007us) write_op.cc:454] Released schema lock
0812 08:03:09.198289 (+   256us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":320075,"mutex_wait_us":13420,"num_ops":5000,"prepare.queue_time_us":200421,"prepare.run_cpu_time_us":50203,"prepare.run_wall_time_us":86534,"replication_time_us":2453,"spinlock_wait_cycles":42368}]]}
I20260812 08:03:09.303762 24488 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 69% done.
I20260812 08:03:09.330961 24509 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=66, attempt_no=0}) took 1008 ms. Trace:
I20260812 08:03:09.342669 24509 rpcz_store.cc:276] 0812 08:03:08.322646 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:08.322723 (+    77us) service_pool.cc:224] Handling call
0812 08:03:09.330933 (+1008210us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:08.393508 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:08.393699 (+   191us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:08.393705 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:03:08.393707 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:08.451426 (+ 57719us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:08.451440 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:08.451446 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:08.485434 (+ 33988us) tablet.cc:701] Row locks acquired
0812 08:03:08.485437 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:08.485529 (+    92us) write_op.cc:270] Start()
0812 08:03:08.485578 (+    49us) write_op.cc:276] Timestamp: P: 1786521788485505 usec, L: 0
0812 08:03:08.485581 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:08.485933 (+   352us) log.cc:844] Serialized 486173 byte log entry
0812 08:03:08.488337 (+  2404us) op_driver.cc:464] REPLICATION: finished
0812 08:03:08.801523 (+313186us) write_op.cc:301] APPLY: starting
0812 08:03:08.801540 (+    17us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:08.807484 (+  5944us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:08.807505 (+    21us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:09.276875 (+469370us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:09.299117 (+ 22242us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:09.299127 (+    10us) write_op.cc:312] APPLY: finished
0812 08:03:09.329426 (+ 30299us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:09.329535 (+   109us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:09.330541 (+  1006us) write_op.cc:454] Released schema lock
0812 08:03:09.330809 (+   268us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":313134,"mutex_wait_us":14802,"num_ops":5000,"prepare.queue_time_us":70046,"prepare.run_cpu_time_us":50937,"prepare.run_wall_time_us":92525,"replication_time_us":2725,"spinlock_wait_cycles":35584}]]}
I20260812 08:03:09.440184 24478 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 79% done.
W20260812 08:03:09.484611 24457 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:34357, user_credentials={real_user=slave}} blocked reactor thread for 43500.4us
I20260812 08:03:09.525665 24477 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 79% done.
I20260812 08:03:09.704427 24491 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 69% done.
I20260812 08:03:09.736091 24483 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 89% done.
I20260812 08:03:09.927029 24478 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 89% done.
I20260812 08:03:09.936246 24490 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 79% done.
I20260812 08:03:10.017429 24492 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 79% done.
I20260812 08:03:10.026556 24509 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=71, attempt_no=0}) took 1137 ms. Trace:
I20260812 08:03:10.028934 24509 rpcz_store.cc:276] 0812 08:03:08.889464 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:08.889519 (+    55us) service_pool.cc:224] Handling call
0812 08:03:10.026532 (+1137013us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:09.000297 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:09.000456 (+   159us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:09.000461 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:09.000462 (+     1us) tablet.cc:662] Decoding operations
0812 08:03:09.054053 (+ 53591us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:09.054066 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:09.054072 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:09.095099 (+ 41027us) tablet.cc:701] Row locks acquired
0812 08:03:09.095103 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:09.095193 (+    90us) write_op.cc:270] Start()
0812 08:03:09.095243 (+    50us) write_op.cc:276] Timestamp: P: 1786521789095169 usec, L: 0
0812 08:03:09.095245 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:09.095623 (+   378us) log.cc:844] Serialized 485526 byte log entry
0812 08:03:09.098018 (+  2395us) op_driver.cc:464] REPLICATION: finished
0812 08:03:09.360238 (+262220us) write_op.cc:301] APPLY: starting
0812 08:03:09.360252 (+    14us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:09.389494 (+ 29242us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:09.389528 (+    34us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:09.721033 (+331505us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x55a84fd49c0c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:09.877017 (+155984us) spinlock_profiling.cc:243] Waited 7.96 ms on lock 0x55a84fd49c0c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:09.981149 (+104132us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:09.983261 (+  2112us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:09.983269 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:10.007893 (+ 24624us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:10.025019 (+ 17126us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:10.026133 (+  1114us) write_op.cc:454] Released schema lock
0812 08:03:10.026387 (+   254us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":262170,"num_ops":5000,"prepare.queue_time_us":110001,"prepare.run_cpu_time_us":49114,"prepare.run_wall_time_us":95419,"replication_time_us":2740,"spinlock_wait_cycles":57510528}]]}
I20260812 08:03:10.106242 24488 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 79% done.
I20260812 08:03:10.107020 24491 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 79% done.
I20260812 08:03:10.108624 24522 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=72, attempt_no=0}) took 1062 ms. Trace:
I20260812 08:03:10.117022 24522 rpcz_store.cc:276] 0812 08:03:09.045736 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:09.045783 (+    47us) service_pool.cc:224] Handling call
0812 08:03:10.108601 (+1062818us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:09.095772 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:09.095935 (+   163us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:09.095940 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:09.095941 (+     1us) tablet.cc:662] Decoding operations
0812 08:03:09.153525 (+ 57584us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:09.153538 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:09.153545 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:09.178017 (+ 24472us) tablet.cc:701] Row locks acquired
0812 08:03:09.178021 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:09.178106 (+    85us) write_op.cc:270] Start()
0812 08:03:09.178154 (+    48us) write_op.cc:276] Timestamp: P: 1786521789178085 usec, L: 0
0812 08:03:09.178157 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:09.178507 (+   350us) log.cc:844] Serialized 486482 byte log entry
0812 08:03:09.180785 (+  2278us) op_driver.cc:464] REPLICATION: finished
0812 08:03:09.429507 (+248722us) write_op.cc:301] APPLY: starting
0812 08:03:09.429525 (+    18us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:09.435142 (+  5617us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:09.435163 (+    21us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:09.710382 (+275219us) spinlock_profiling.cc:243] Waited 4.23 ms on lock 0x55a84fd49c0c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:10.059332 (+348950us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:10.081481 (+ 22149us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:10.081489 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:10.106448 (+ 24959us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:10.107193 (+   745us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:10.108207 (+  1014us) write_op.cc:454] Released schema lock
0812 08:03:10.108462 (+   255us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":248666,"num_ops":5000,"prepare.queue_time_us":49289,"prepare.run_cpu_time_us":50532,"prepare.run_wall_time_us":82825,"replication_time_us":2601,"spinlock_wait_cycles":9758208}]]}
I20260812 08:03:10.114838 24485 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 79% done.
W20260812 08:03:10.229444 24457 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:34357, user_credentials={real_user=slave}} blocked reactor thread for 71945.3us
I20260812 08:03:10.317073 24480 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 79% done.
I20260812 08:03:10.323280 24482 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 79% done.
I20260812 08:03:10.361872 24477 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 89% done.
I20260812 08:03:10.429478 24490 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 89% done.
I20260812 08:03:10.450799 24511 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=76, attempt_no=0}) took 1109 ms. Trace:
I20260812 08:03:10.451027 24511 rpcz_store.cc:276] 0812 08:03:09.341829 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:09.341893 (+    64us) service_pool.cc:224] Handling call
0812 08:03:10.450773 (+1108880us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:09.360190 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:09.360353 (+   163us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:09.360357 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:03:09.360359 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:09.435088 (+ 74729us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:09.435100 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:09.435105 (+     5us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:09.455991 (+ 20886us) tablet.cc:701] Row locks acquired
0812 08:03:09.455994 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:09.456080 (+    86us) write_op.cc:270] Start()
0812 08:03:09.456126 (+    46us) write_op.cc:276] Timestamp: P: 1786521789456059 usec, L: 0
0812 08:03:09.456129 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:09.456477 (+   348us) log.cc:844] Serialized 486205 byte log entry
0812 08:03:09.458817 (+  2340us) op_driver.cc:464] REPLICATION: finished
0812 08:03:09.843216 (+384399us) write_op.cc:301] APPLY: starting
0812 08:03:09.843234 (+    18us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:09.893313 (+ 50079us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:09.893334 (+    21us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:10.423375 (+530041us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:10.425494 (+  2119us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:10.425500 (+     6us) write_op.cc:312] APPLY: finished
0812 08:03:10.449026 (+ 23526us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:10.449232 (+   206us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:10.450355 (+  1123us) write_op.cc:454] Released schema lock
0812 08:03:10.450633 (+   278us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":384348,"mutex_wait_us":32407,"num_ops":5000,"prepare.queue_time_us":17562,"prepare.run_cpu_time_us":48556,"prepare.run_wall_time_us":96372,"replication_time_us":2664,"spinlock_wait_cycles":13696}]]}
I20260812 08:03:10.527577 24522 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=78, attempt_no=0}) took 1017 ms. Trace:
I20260812 08:03:10.536942 24522 rpcz_store.cc:276] 0812 08:03:09.509945 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:09.509998 (+    53us) service_pool.cc:224] Handling call
0812 08:03:10.527552 (+1017554us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:09.510736 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:09.510893 (+   157us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:09.510897 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:03:09.510899 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:09.610567 (+ 99668us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:09.610582 (+    15us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:09.610589 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:09.703370 (+ 92781us) tablet.cc:701] Row locks acquired
0812 08:03:09.703374 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:09.703451 (+    77us) write_op.cc:270] Start()
0812 08:03:09.703496 (+    45us) write_op.cc:276] Timestamp: P: 1786521789703436 usec, L: 0
0812 08:03:09.703498 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:09.703837 (+   339us) log.cc:844] Serialized 485929 byte log entry
0812 08:03:09.706110 (+  2273us) op_driver.cc:464] REPLICATION: finished
0812 08:03:10.118369 (+412259us) write_op.cc:301] APPLY: starting
0812 08:03:10.118386 (+    17us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:10.138683 (+ 20297us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:10.138713 (+    30us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:10.333052 (+194339us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x55a84f8d1898. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facf3c2fbdd 00007facfc1cfe3b 00007facfc0998df 00007facfc09e60a 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d
0812 08:03:10.465016 (+131964us) spinlock_profiling.cc:243] Waited 6.5 ms on lock 0x55a84f8d1898. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facf3c2fbdd 00007facfc1cfe3b 00007facfc0998df 00007facfc09e60a 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d
0812 08:03:10.506952 (+ 41936us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:10.509183 (+  2231us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:10.509190 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:10.525853 (+ 16663us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:10.526138 (+   285us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:10.527176 (+  1038us) write_op.cc:454] Released schema lock
0812 08:03:10.527433 (+   257us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":412210,"mutex_wait_us":32763,"num_ops":5000,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":49191,"prepare.run_wall_time_us":193192,"replication_time_us":2581,"spinlock_wait_cycles":51561472}]]}
I20260812 08:03:10.570397 24492 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 89% done.
W20260812 08:03:10.757699 24457 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:34357, user_credentials={real_user=slave}} blocked reactor thread for 67965.8us
I20260812 08:03:10.855648 24482 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 89% done.
I20260812 08:03:10.923270 24483 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 99% done.
I20260812 08:03:10.923467 24483 full_stack-insert-scan-test.cc:280] Insertion thread 5 of  10 is 100% done.
I20260812 08:03:10.947278 24485 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 89% done.
I20260812 08:03:11.071758 24488 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 89% done.
I20260812 08:03:11.082926 24511 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=81, attempt_no=0}) took 1252 ms. Trace:
I20260812 08:03:11.096611 24511 rpcz_store.cc:276] 0812 08:03:09.830643 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:09.830712 (+    69us) service_pool.cc:224] Handling call
0812 08:03:11.082903 (+1252191us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:10.051433 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:10.051594 (+   161us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:10.051598 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:03:10.051600 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:10.109106 (+ 57506us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:10.109117 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:10.109122 (+     5us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:10.130156 (+ 21034us) tablet.cc:701] Row locks acquired
0812 08:03:10.130160 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:10.130228 (+    68us) write_op.cc:270] Start()
0812 08:03:10.130270 (+    42us) write_op.cc:276] Timestamp: P: 1786521790130212 usec, L: 0
0812 08:03:10.130272 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:10.130599 (+   327us) log.cc:844] Serialized 485582 byte log entry
0812 08:03:10.132847 (+  2248us) op_driver.cc:464] REPLICATION: finished
0812 08:03:10.452274 (+319427us) write_op.cc:301] APPLY: starting
0812 08:03:10.452292 (+    18us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:10.458235 (+  5943us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:10.458265 (+    30us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:10.485046 (+ 26781us) spinlock_profiling.cc:243] Waited 26.6 ms on lock 0x55a84f8d1898. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facf3c2fbdd 00007facfc1cfe3b 00007facfc0998df 00007facfc09e60a 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d
0812 08:03:11.054225 (+569179us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:11.056444 (+  2219us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:11.056451 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:11.081171 (+ 24720us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:11.081478 (+   307us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:11.082531 (+  1053us) write_op.cc:454] Released schema lock
0812 08:03:11.082788 (+   257us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":319375,"mutex_wait_us":63480,"num_ops":5000,"prepare.queue_time_us":219989,"prepare.run_cpu_time_us":48844,"prepare.run_wall_time_us":79256,"replication_time_us":2550,"spinlock_wait_cycles":61175808}]]}
W20260812 08:03:11.198014 24457 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:34357, user_credentials={real_user=slave}} blocked reactor thread for 68285.5us
I20260812 08:03:11.291981 24478 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 99% done.
I20260812 08:03:11.301926 24478 full_stack-insert-scan-test.cc:280] Insertion thread 2 of  10 is 100% done.
W20260812 08:03:11.297931 24457 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:34357, user_credentials={real_user=slave}} blocked reactor thread for 91874.9us
I20260812 08:03:11.310153 24522 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35682 (ReqId={client: 74099d6faa8e4d6daf24460ab386c8f4, seq_no=85, attempt_no=0}) took 1033 ms. Trace:
I20260812 08:03:11.310376 24522 rpcz_store.cc:276] 0812 08:03:10.276962 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:10.277020 (+    58us) service_pool.cc:224] Handling call
0812 08:03:11.310132 (+1033112us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:10.339698 (+     0us) write_op.cc:183] PREPARE: starting on tablet a4099d9a819c45f4bfc3f492de85b0ac
0812 08:03:10.339857 (+   159us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:10.339862 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:10.339864 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:10.367750 (+ 27886us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:10.367763 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:10.367769 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:10.387701 (+ 19932us) tablet.cc:701] Row locks acquired
0812 08:03:10.387705 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:10.387769 (+    64us) write_op.cc:270] Start()
0812 08:03:10.387807 (+    38us) write_op.cc:276] Timestamp: P: 1786521790387754 usec, L: 0
0812 08:03:10.387809 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:10.388112 (+   303us) log.cc:844] Serialized 485983 byte log entry
0812 08:03:10.390443 (+  2331us) op_driver.cc:464] REPLICATION: finished
0812 08:03:10.845544 (+455101us) write_op.cc:301] APPLY: starting
0812 08:03:10.845564 (+    20us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:10.851378 (+  5814us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:10.851403 (+    25us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:11.278174 (+426771us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:11.304175 (+ 26001us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:11.304182 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:11.308621 (+  4439us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:11.308711 (+    90us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:11.309771 (+  1060us) write_op.cc:454] Released schema lock
0812 08:03:11.310018 (+   247us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":455053,"num_ops":5000,"prepare.queue_time_us":61762,"prepare.run_cpu_time_us":48487,"prepare.run_wall_time_us":48490,"replication_time_us":2610,"spinlock_wait_cycles":40064}]]}
I20260812 08:03:11.358493 24477 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 99% done.
I20260812 08:03:11.358587 24477 full_stack-insert-scan-test.cc:280] Insertion thread 1 of  10 is 100% done.
I20260812 08:03:11.460593 24490 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 99% done.
I20260812 08:03:11.468956 24490 full_stack-insert-scan-test.cc:280] Insertion thread 8 of  10 is 100% done.
W20260812 08:03:11.652758 24457 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:34357, user_credentials={real_user=slave}} blocked reactor thread for 43689.7us
I20260812 08:03:11.658546 24480 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 89% done.
I20260812 08:03:11.788038 24488 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 99% done.
I20260812 08:03:11.788259 24488 full_stack-insert-scan-test.cc:280] Insertion thread 7 of  10 is 100% done.
I20260812 08:03:11.807983 24491 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 89% done.
I20260812 08:03:11.813823 24492 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 99% done.
I20260812 08:03:11.814043 24492 full_stack-insert-scan-test.cc:280] Insertion thread 10 of  10 is 100% done.
I20260812 08:03:11.909860 24485 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 99% done.
I20260812 08:03:11.910033 24485 full_stack-insert-scan-test.cc:280] Insertion thread 6 of  10 is 100% done.
I20260812 08:03:11.986881 24482 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 99% done.
I20260812 08:03:12.002045 24482 full_stack-insert-scan-test.cc:280] Insertion thread 4 of  10 is 100% done.
I20260812 08:03:12.163728 24480 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 99% done.
I20260812 08:03:12.163892 24480 full_stack-insert-scan-test.cc:280] Insertion thread 3 of  10 is 100% done.
I20260812 08:03:12.333043 24491 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 99% done.
I20260812 08:03:12.333225 24491 full_stack-insert-scan-test.cc:280] Insertion thread 9 of  10 is 100% done.
I20260812 08:03:12.674026 24143 full_stack-insert-scan-test.cc:333] Time spent concurrent inserts (500000 rows, 10 threads): real 10.239s	user 0.000s	sys 0.000s
I20260812 08:03:12.674147 24143 full_stack-insert-scan-test.cc:346] Doing test scans on table of 500000 rows.
I20260812 08:03:13.186511 24143 full_stack-insert-scan-test.cc:453] Time spent empty projection, 0 col: real 0.240s	user 0.001s	sys 0.000s
I20260812 08:03:13.624060 24143 full_stack-insert-scan-test.cc:453] Time spent key scan, 1 col: real 0.296s	user 0.002s	sys 0.000s
I20260812 08:03:14.514914 24143 full_stack-insert-scan-test.cc:453] Time spent full schema scan, no cache, 10 col: real 0.739s	user 0.035s	sys 0.001s
I20260812 08:03:15.407401 24143 full_stack-insert-scan-test.cc:453] Time spent fault-tolerant full schema scan, no cache, 10 col: real 0.875s	user 0.032s	sys 0.002s
I20260812 08:03:16.149098 24143 full_stack-insert-scan-test.cc:453] Time spent full schema scan, 10 col: real 0.724s	user 0.026s	sys 0.007s
I20260812 08:03:16.669320 24143 full_stack-insert-scan-test.cc:453] Time spent String projection, 1 col: real 0.382s	user 0.027s	sys 0.000s
I20260812 08:03:17.210767 24143 full_stack-insert-scan-test.cc:453] Time spent Int32 projection, 4 col: real 0.398s	user 0.001s	sys 0.002s
I20260812 08:03:17.765395 24143 full_stack-insert-scan-test.cc:453] Time spent Int64 projection, 4 col: real 0.412s	user 0.004s	sys 0.000s
I20260812 08:03:17.766901 24143 tablet_server.cc:179] TabletServer@127.23.147.193:0 shutting down...
I20260812 08:03:17.888796 24143 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260812 08:03:17.889235 24143 tablet_replica.cc:333] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0: stopping tablet replica
I20260812 08:03:17.889446 24143 raft_consensus.cc:2243] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0 [term 1 LEADER]: Raft consensus shutting down.
I20260812 08:03:17.889681 24143 raft_consensus.cc:2272] T a4099d9a819c45f4bfc3f492de85b0ac P 2ecbf965a2194794b143f84621a1b0d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 08:03:17.931864 24143 tablet_server.cc:196] TabletServer@127.23.147.193:0 shutdown complete.
I20260812 08:03:17.935681 24143 master.cc:562] Master@127.23.147.254:37725 shutting down...
I20260812 08:03:17.979790 24143 raft_consensus.cc:2243] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734 [term 1 LEADER]: Raft consensus shutting down.
I20260812 08:03:17.980007 24143 raft_consensus.cc:2272] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 08:03:17.980086 24143 tablet_replica.cc:333] T 00000000000000000000000000000000 P a735ae45e0bd4630aa165ac44d2bd734: stopping tablet replica
I20260812 08:03:17.992506 24143 master.cc:584] Master@127.23.147.254:37725 shutdown complete.
[       OK ] EvictionPolicyTypes/FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest/0 (15831 ms)
[ RUN      ] EvictionPolicyTypes/FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest/1
I20260812 08:03:18.004520 24143 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.147.254:37241
I20260812 08:03:18.004940 24143 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:03:18.007113 24538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:03:18.007253 24143 server_base.cc:1061] running on GCE node
W20260812 08:03:18.007118 24544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:03:18.007174 24540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:03:18.007560 24143 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 08:03:18.007602 24143 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 08:03:18.007618 24143 hybrid_clock.cc:648] HybridClock initialized: now 1786521798007618 us; error 0 us; skew 500 ppm
I20260812 08:03:18.008368 24143 webserver.cc:533] Webserver started at http://127.23.147.254:36701/ using document root <none> and password file <none>
I20260812 08:03:18.008515 24143 fs_manager.cc:362] Metadata directory not provided
I20260812 08:03:18.008562 24143 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:03:18.008649 24143 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:03:18.009056 24143 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest_1.1786521782163021-24143-0/minicluster-data/master-0-root/instance:
uuid: "33b59b09bc0a4465a3e983805d6e8462"
format_stamp: "Formatted at 2026-08-12 08:03:18 on dist-test-slave-ncp9"
I20260812 08:03:18.010449 24143 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20260812 08:03:18.011303 24549 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:18.011518 24143 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
I20260812 08:03:18.011582 24143 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest_1.1786521782163021-24143-0/minicluster-data/master-0-root
uuid: "33b59b09bc0a4465a3e983805d6e8462"
format_stamp: "Formatted at 2026-08-12 08:03:18 on dist-test-slave-ncp9"
I20260812 08:03:18.011660 24143 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest_1.1786521782163021-24143-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest_1.1786521782163021-24143-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest_1.1786521782163021-24143-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:03:18.028388 24143 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:03:18.028760 24143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:03:18.032605 24143 rpc_server.cc:307] RPC server started. Bound to: 127.23.147.254:37241
I20260812 08:03:18.037899 24641 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.147.254:37241 every 8 connection(s)
I20260812 08:03:18.038313 24642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:03:18.040009 24642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462: Bootstrap starting.
I20260812 08:03:18.040800 24642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462: Neither blocks nor log segments found. Creating new log.
I20260812 08:03:18.041762 24642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462: No bootstrap required, opened a new log
I20260812 08:03:18.042137 24642 raft_consensus.cc:359] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b59b09bc0a4465a3e983805d6e8462" member_type: VOTER }
I20260812 08:03:18.042246 24642 raft_consensus.cc:385] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:03:18.042279 24642 raft_consensus.cc:740] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33b59b09bc0a4465a3e983805d6e8462, State: Initialized, Role: FOLLOWER
I20260812 08:03:18.042426 24642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b59b09bc0a4465a3e983805d6e8462" member_type: VOTER }
I20260812 08:03:18.042510 24642 raft_consensus.cc:399] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 08:03:18.042551 24642 raft_consensus.cc:493] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 08:03:18.042599 24642 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:03:18.043362 24642 raft_consensus.cc:515] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b59b09bc0a4465a3e983805d6e8462" member_type: VOTER }
I20260812 08:03:18.043491 24642 leader_election.cc:304] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33b59b09bc0a4465a3e983805d6e8462; no voters: 
I20260812 08:03:18.043668 24642 leader_election.cc:290] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 08:03:18.043766 24646 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:03:18.043949 24646 raft_consensus.cc:697] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462 [term 1 LEADER]: Becoming Leader. State: Replica: 33b59b09bc0a4465a3e983805d6e8462, State: Running, Role: LEADER
I20260812 08:03:18.044085 24642 sys_catalog.cc:565] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:03:18.044088 24646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b59b09bc0a4465a3e983805d6e8462" member_type: VOTER }
I20260812 08:03:18.044510 24647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33b59b09bc0a4465a3e983805d6e8462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b59b09bc0a4465a3e983805d6e8462" member_type: VOTER } }
I20260812 08:03:18.044539 24648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33b59b09bc0a4465a3e983805d6e8462. Latest consensus state: current_term: 1 leader_uuid: "33b59b09bc0a4465a3e983805d6e8462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b59b09bc0a4465a3e983805d6e8462" member_type: VOTER } }
I20260812 08:03:18.044658 24647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462 [sys.catalog]: This master's current role is: LEADER
I20260812 08:03:18.044678 24648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462 [sys.catalog]: This master's current role is: LEADER
I20260812 08:03:18.045207 24653 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:03:18.046097 24653 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:03:18.046283 24143 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260812 08:03:18.047827 24653 catalog_manager.cc:1383] Generated new cluster ID: 4a0272e58348425496f333a6cc07fc15
I20260812 08:03:18.047873 24653 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:03:18.056401 24653 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:03:18.056914 24653 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:03:18.066672 24653 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462: Generated new TSK 0
I20260812 08:03:18.066833 24653 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:03:18.078579 24143 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:03:18.081081 24676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:03:18.081159 24143 server_base.cc:1061] running on GCE node
W20260812 08:03:18.081296 24673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:03:18.081310 24678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:03:18.081571 24143 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 08:03:18.081617 24143 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 08:03:18.081631 24143 hybrid_clock.cc:648] HybridClock initialized: now 1786521798081632 us; error 0 us; skew 500 ppm
I20260812 08:03:18.082454 24143 webserver.cc:533] Webserver started at http://127.23.147.193:33495/ using document root <none> and password file <none>
I20260812 08:03:18.082638 24143 fs_manager.cc:362] Metadata directory not provided
I20260812 08:03:18.082698 24143 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:03:18.082778 24143 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:03:18.083184 24143 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest_1.1786521782163021-24143-0/minicluster-data/ts-0-root/instance:
uuid: "fb942e9d90d34e929ce1b419cbb43066"
format_stamp: "Formatted at 2026-08-12 08:03:18 on dist-test-slave-ncp9"
I20260812 08:03:18.084638 24143 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.002s	sys 0.000s
I20260812 08:03:18.085584 24686 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:18.085937 24143 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.001s
I20260812 08:03:18.086017 24143 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest_1.1786521782163021-24143-0/minicluster-data/ts-0-root
uuid: "fb942e9d90d34e929ce1b419cbb43066"
format_stamp: "Formatted at 2026-08-12 08:03:18 on dist-test-slave-ncp9"
I20260812 08:03:18.086071 24143 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest_1.1786521782163021-24143-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest_1.1786521782163021-24143-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest_1.1786521782163021-24143-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:03:18.097668 24143 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:03:18.098070 24143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:03:18.098426 24143 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:03:18.098871 24143 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:03:18.098912 24143 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:18.098951 24143 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:03:18.098971 24143 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:18.106035 24143 rpc_server.cc:307] RPC server started. Bound to: 127.23.147.193:37893
I20260812 08:03:18.106053 24813 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.147.193:37893 every 8 connection(s)
I20260812 08:03:18.111493 24815 heartbeater.cc:344] Connected to a master server at 127.23.147.254:37241
I20260812 08:03:18.111629 24815 heartbeater.cc:461] Registering TS with master...
I20260812 08:03:18.111879 24815 heartbeater.cc:507] Master 127.23.147.254:37241 requested a full tablet report, sending...
I20260812 08:03:18.112600 24583 ts_manager.cc:194] Registered new tserver with Master: fb942e9d90d34e929ce1b419cbb43066 (127.23.147.193:37893)
I20260812 08:03:18.113404 24583 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40922
I20260812 08:03:18.113543 24143 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.007080052s
I20260812 08:03:18.121941 24583 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40926:
name: "full-stack-mrs-test-tbl"
schema {
  columns {
    name: "key"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int32_val1"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int32_val2"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int32_val3"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int32_val4"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int64_val1"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int64_val2"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int64_val3"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int64_val4"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20260812 08:03:18.131120 24741 tablet_service.cc:1511] Processing CreateTablet for tablet 83af86d24c524d669f5437cf6bae86a6 (DEFAULT_TABLE table=full-stack-mrs-test-tbl [id=d51b070449634b74bb150fa7f1072f29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:03:18.131454 24741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83af86d24c524d669f5437cf6bae86a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:03:18.133739 24840 tablet_bootstrap.cc:492] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066: Bootstrap starting.
I20260812 08:03:18.134681 24840 tablet_bootstrap.cc:654] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066: Neither blocks nor log segments found. Creating new log.
I20260812 08:03:18.135924 24840 tablet_bootstrap.cc:492] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066: No bootstrap required, opened a new log
I20260812 08:03:18.136029 24840 ts_tablet_manager.cc:1403] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066: Time spent bootstrapping tablet: real 0.002s	user 0.002s	sys 0.000s
I20260812 08:03:18.136502 24840 raft_consensus.cc:359] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb942e9d90d34e929ce1b419cbb43066" member_type: VOTER last_known_addr { host: "127.23.147.193" port: 37893 } }
I20260812 08:03:18.136611 24840 raft_consensus.cc:385] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:03:18.136646 24840 raft_consensus.cc:740] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb942e9d90d34e929ce1b419cbb43066, State: Initialized, Role: FOLLOWER
I20260812 08:03:18.136981 24840 consensus_queue.cc:260] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb942e9d90d34e929ce1b419cbb43066" member_type: VOTER last_known_addr { host: "127.23.147.193" port: 37893 } }
I20260812 08:03:18.137099 24840 raft_consensus.cc:399] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 08:03:18.137151 24840 raft_consensus.cc:493] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 08:03:18.137192 24840 raft_consensus.cc:3060] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:03:18.137914 24840 raft_consensus.cc:515] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb942e9d90d34e929ce1b419cbb43066" member_type: VOTER last_known_addr { host: "127.23.147.193" port: 37893 } }
I20260812 08:03:18.138029 24840 leader_election.cc:304] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb942e9d90d34e929ce1b419cbb43066; no voters: 
I20260812 08:03:18.138196 24840 leader_election.cc:290] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 08:03:18.138320 24845 raft_consensus.cc:2804] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:03:18.138509 24840 ts_tablet_manager.cc:1434] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20260812 08:03:18.138536 24845 raft_consensus.cc:697] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066 [term 1 LEADER]: Becoming Leader. State: Replica: fb942e9d90d34e929ce1b419cbb43066, State: Running, Role: LEADER
I20260812 08:03:18.138561 24815 heartbeater.cc:499] Master 127.23.147.254:37241 was elected leader, sending a full tablet report...
I20260812 08:03:18.138716 24845 consensus_queue.cc:237] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb942e9d90d34e929ce1b419cbb43066" member_type: VOTER last_known_addr { host: "127.23.147.193" port: 37893 } }
I20260812 08:03:18.139953 24583 catalog_manager.cc:5719] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb942e9d90d34e929ce1b419cbb43066 (127.23.147.193). New cstate: current_term: 1 leader_uuid: "fb942e9d90d34e929ce1b419cbb43066" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb942e9d90d34e929ce1b419cbb43066" member_type: VOTER last_known_addr { host: "127.23.147.193" port: 37893 } health_report { overall_health: HEALTHY } } }
I20260812 08:03:18.433239 24853 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 9% done.
I20260812 08:03:18.658147 24872 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 9% done.
I20260812 08:03:18.729928 24853 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 19% done.
I20260812 08:03:18.737854 24858 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 9% done.
I20260812 08:03:18.754943 24859 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 9% done.
I20260812 08:03:18.793466 24871 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 9% done.
I20260812 08:03:18.799574 24861 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 9% done.
I20260812 08:03:18.832510 24857 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 9% done.
I20260812 08:03:18.840531 24860 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 9% done.
I20260812 08:03:18.842344 24865 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 9% done.
I20260812 08:03:18.858922 24855 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 9% done.
I20260812 08:03:19.059136 24859 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 19% done.
I20260812 08:03:19.380378 24857 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 19% done.
I20260812 08:03:19.448482 24853 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 29% done.
I20260812 08:03:19.451941 24871 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 19% done.
I20260812 08:03:19.503062 24865 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 19% done.
I20260812 08:03:19.513761 24860 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 19% done.
I20260812 08:03:19.541785 24872 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 19% done.
I20260812 08:03:19.547703 24855 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 19% done.
I20260812 08:03:19.563632 24858 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 19% done.
I20260812 08:03:19.592428 24861 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 19% done.
I20260812 08:03:19.614832 24859 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 29% done.
I20260812 08:03:19.780817 24872 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 29% done.
I20260812 08:03:19.922216 24858 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 29% done.
I20260812 08:03:20.041438 24871 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 29% done.
I20260812 08:03:20.056368 24888 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51100 (ReqId={client: 1a9df0103ee24e97be61c9330a685648, seq_no=17, attempt_no=0}) took 1091 ms. Trace:
I20260812 08:03:20.061434 24888 rpcz_store.cc:276] 0812 08:03:18.965326 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:18.965375 (+    49us) service_pool.cc:224] Handling call
0812 08:03:20.056343 (+1090968us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:19.704159 (+     0us) write_op.cc:183] PREPARE: starting on tablet 83af86d24c524d669f5437cf6bae86a6
0812 08:03:19.704315 (+   156us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:19.704320 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:19.704322 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:19.733434 (+ 29112us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:19.733447 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:19.733454 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:19.753595 (+ 20141us) tablet.cc:701] Row locks acquired
0812 08:03:19.753598 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:19.753703 (+   105us) write_op.cc:270] Start()
0812 08:03:19.753752 (+    49us) write_op.cc:276] Timestamp: P: 1786521799753679 usec, L: 0
0812 08:03:19.753754 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:19.754461 (+   707us) log.cc:844] Serialized 485900 byte log entry
0812 08:03:19.756821 (+  2360us) op_driver.cc:464] REPLICATION: finished
0812 08:03:19.769077 (+ 12256us) write_op.cc:301] APPLY: starting
0812 08:03:19.769099 (+    22us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:19.775333 (+  6234us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:19.775362 (+    29us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:20.021145 (+245783us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:20.023343 (+  2198us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:20.023350 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:20.054637 (+ 31287us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:20.054758 (+   121us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:20.055965 (+  1207us) write_op.cc:454] Released schema lock
0812 08:03:20.056220 (+   255us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":12171,"mutex_wait_us":9534,"num_ops":5000,"prepare.queue_time_us":737651,"prepare.run_cpu_time_us":50233,"prepare.run_wall_time_us":52859,"replication_time_us":3036,"spinlock_wait_cycles":36224,"thread_start_us":99,"threads_started":1}]]}
I20260812 08:03:20.092334 24881 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51100 (ReqId={client: 1a9df0103ee24e97be61c9330a685648, seq_no=19, attempt_no=0}) took 1086 ms. Trace:
I20260812 08:03:20.101768 24881 rpcz_store.cc:276] 0812 08:03:19.006186 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:19.006265 (+    79us) service_pool.cc:224] Handling call
0812 08:03:20.092311 (+1086046us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:19.825505 (+     0us) write_op.cc:183] PREPARE: starting on tablet 83af86d24c524d669f5437cf6bae86a6
0812 08:03:19.825685 (+   180us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:19.825689 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:03:19.825691 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:19.853757 (+ 28066us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:19.853769 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:19.853774 (+     5us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:19.874185 (+ 20411us) tablet.cc:701] Row locks acquired
0812 08:03:19.874188 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:19.874266 (+    78us) write_op.cc:270] Start()
0812 08:03:19.874312 (+    46us) write_op.cc:276] Timestamp: P: 1786521799874246 usec, L: 0
0812 08:03:19.874314 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:19.874660 (+   346us) log.cc:844] Serialized 486153 byte log entry
0812 08:03:19.877048 (+  2388us) op_driver.cc:464] REPLICATION: finished
0812 08:03:19.877143 (+    95us) write_op.cc:301] APPLY: starting
0812 08:03:19.877159 (+    16us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:19.882915 (+  5756us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:19.882933 (+    18us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:20.075248 (+192315us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:20.077391 (+  2143us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:20.077398 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:20.088337 (+ 10939us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:20.090779 (+  2442us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:20.091949 (+  1170us) write_op.cc:454] Released schema lock
0812 08:03:20.092208 (+   259us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":56,"mutex_wait_us":12098,"num_ops":5000,"prepare.queue_time_us":817945,"prepare.run_cpu_time_us":48936,"prepare.run_wall_time_us":71438,"replication_time_us":2702,"spinlock_wait_cycles":44416}]]}
I20260812 08:03:20.110184 24861 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 29% done.
W20260812 08:03:20.158735 24829 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:37893, user_credentials={real_user=slave}} blocked reactor thread for 55532.4us
I20260812 08:03:20.213815 24885 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51100 (ReqId={client: 1a9df0103ee24e97be61c9330a685648, seq_no=18, attempt_no=0}) took 1233 ms. Trace:
I20260812 08:03:20.224936 24885 rpcz_store.cc:276] 0812 08:03:18.980528 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:18.980583 (+    55us) service_pool.cc:224] Handling call
0812 08:03:20.213789 (+1233206us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:19.757042 (+     0us) write_op.cc:183] PREPARE: starting on tablet 83af86d24c524d669f5437cf6bae86a6
0812 08:03:19.757208 (+   166us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:19.757212 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:03:19.757214 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:19.802583 (+ 45369us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:19.802593 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:19.802598 (+     5us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:19.822408 (+ 19810us) tablet.cc:701] Row locks acquired
0812 08:03:19.822411 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:19.822496 (+    85us) write_op.cc:270] Start()
0812 08:03:19.822541 (+    45us) write_op.cc:276] Timestamp: P: 1786521799822476 usec, L: 0
0812 08:03:19.822543 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:19.822914 (+   371us) log.cc:844] Serialized 485649 byte log entry
0812 08:03:19.825397 (+  2483us) op_driver.cc:464] REPLICATION: finished
0812 08:03:19.825591 (+   194us) write_op.cc:301] APPLY: starting
0812 08:03:19.825607 (+    16us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:19.831580 (+  5973us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:19.831601 (+    21us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:20.176259 (+344658us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:20.178451 (+  2192us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:20.178458 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:20.211503 (+ 33045us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:20.212351 (+   848us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:20.213405 (+  1054us) write_op.cc:454] Released schema lock
0812 08:03:20.213660 (+   255us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":150,"mutex_wait_us":19008,"num_ops":5000,"prepare.queue_time_us":775439,"prepare.run_cpu_time_us":49718,"prepare.run_wall_time_us":68446,"replication_time_us":2824,"spinlock_wait_cycles":35328}]]}
I20260812 08:03:20.255622 24865 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 29% done.
I20260812 08:03:20.358024 24860 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 29% done.
I20260812 08:03:20.442752 24891 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51100 (ReqId={client: 1a9df0103ee24e97be61c9330a685648, seq_no=20, attempt_no=0}) took 1384 ms. Trace:
I20260812 08:03:20.442895 24891 rpcz_store.cc:276] 0812 08:03:19.058574 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:19.058629 (+    55us) service_pool.cc:224] Handling call
0812 08:03:20.442724 (+1384095us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:19.897033 (+     0us) write_op.cc:183] PREPARE: starting on tablet 83af86d24c524d669f5437cf6bae86a6
0812 08:03:19.897207 (+   174us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:19.897212 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:19.897214 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:19.947662 (+ 50448us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:19.947675 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:19.947683 (+     8us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:19.995181 (+ 47498us) tablet.cc:701] Row locks acquired
0812 08:03:19.995184 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:19.995269 (+    85us) write_op.cc:270] Start()
0812 08:03:19.995312 (+    43us) write_op.cc:276] Timestamp: P: 1786521799995248 usec, L: 0
0812 08:03:19.995315 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:19.996026 (+   711us) log.cc:844] Serialized 485654 byte log entry
0812 08:03:19.998380 (+  2354us) op_driver.cc:464] REPLICATION: finished
0812 08:03:20.012946 (+ 14566us) write_op.cc:301] APPLY: starting
0812 08:03:20.012974 (+    28us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:20.028512 (+ 15538us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:20.028540 (+    28us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:20.137028 (+108488us) spinlock_profiling.cc:243] Waited 26.3 ms on lock 0x55a84fd4838c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:20.187017 (+ 49989us) spinlock_profiling.cc:243] Waited 1.96 ms on lock 0x55a84fd4838c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:20.389711 (+202694us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:20.391908 (+  2197us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:20.391915 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:20.417461 (+ 25546us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:20.441002 (+ 23541us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:20.442319 (+  1317us) write_op.cc:454] Released schema lock
0812 08:03:20.442573 (+   254us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":14480,"mutex_wait_us":18547,"num_ops":5000,"prepare.queue_time_us":837015,"prepare.run_cpu_time_us":53502,"prepare.run_wall_time_us":101543,"replication_time_us":3035,"spinlock_wait_cycles":65092992,"thread_start_us":82,"threads_started":1}]]}
I20260812 08:03:20.486495 24857 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 29% done.
I20260812 08:03:20.630856 24881 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51100 (ReqId={client: 1a9df0103ee24e97be61c9330a685648, seq_no=22, attempt_no=0}) took 1057 ms. Trace:
I20260812 08:03:20.636932 24881 rpcz_store.cc:276] 0812 08:03:19.573086 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:19.573148 (+    62us) service_pool.cc:224] Handling call
0812 08:03:20.630832 (+1057684us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:20.088607 (+     0us) write_op.cc:183] PREPARE: starting on tablet 83af86d24c524d669f5437cf6bae86a6
0812 08:03:20.088765 (+   158us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:20.088769 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:03:20.088771 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:20.115712 (+ 26941us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:20.115726 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:20.115732 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:20.136469 (+ 20737us) tablet.cc:701] Row locks acquired
0812 08:03:20.136473 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:20.136569 (+    96us) write_op.cc:270] Start()
0812 08:03:20.136619 (+    50us) write_op.cc:276] Timestamp: P: 1786521800136544 usec, L: 0
0812 08:03:20.136621 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:20.137023 (+   402us) log.cc:844] Serialized 486027 byte log entry
0812 08:03:20.139384 (+  2361us) op_driver.cc:464] REPLICATION: finished
0812 08:03:20.139472 (+    88us) write_op.cc:301] APPLY: starting
0812 08:03:20.139489 (+    17us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:20.146968 (+  7479us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:20.146985 (+    17us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:20.569170 (+422185us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:20.571168 (+  1998us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:20.571177 (+     9us) write_op.cc:312] APPLY: finished
0812 08:03:20.610764 (+ 39587us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:20.614511 (+  3747us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:20.630126 (+ 15615us) spinlock_profiling.cc:243] Waited 3.93 ms on lock 0x55a84fc1f0b0. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfc1c860b 00007facfc1c9ecb 00007facfc1c8c17 00007facfc16e26f 00007facfc16d773 00007facfc16be49 00007facfc15916c 00007facfc158b95 00007facfc1579ca 00007facfc15a95b
0812 08:03:20.630435 (+   309us) write_op.cc:454] Released schema lock
0812 08:03:20.630711 (+   276us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":46,"mutex_wait_us":944,"num_ops":5000,"prepare.queue_time_us":514729,"prepare.run_cpu_time_us":48426,"prepare.run_wall_time_us":48502,"replication_time_us":2732,"spinlock_wait_cycles":9086592}]]}
I20260812 08:03:20.644431 24855 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 29% done.
I20260812 08:03:20.700474 24859 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 39% done.
I20260812 08:03:20.722728 24885 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51100 (ReqId={client: 1a9df0103ee24e97be61c9330a685648, seq_no=23, attempt_no=0}) took 1094 ms. Trace:
I20260812 08:03:20.722939 24885 rpcz_store.cc:276] 0812 08:03:19.628362 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:19.628419 (+    57us) service_pool.cc:224] Handling call
0812 08:03:20.722701 (+1094282us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:20.137182 (+     0us) write_op.cc:183] PREPARE: starting on tablet 83af86d24c524d669f5437cf6bae86a6
0812 08:03:20.137353 (+   171us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:20.137358 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:20.137360 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:20.165035 (+ 27675us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:20.165051 (+    16us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:20.165057 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:20.185911 (+ 20854us) tablet.cc:701] Row locks acquired
0812 08:03:20.185914 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:20.185996 (+    82us) write_op.cc:270] Start()
0812 08:03:20.186042 (+    46us) write_op.cc:276] Timestamp: P: 1786521800185977 usec, L: 0
0812 08:03:20.186045 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:20.186497 (+   452us) log.cc:844] Serialized 486291 byte log entry
0812 08:03:20.188766 (+  2269us) op_driver.cc:464] REPLICATION: finished
0812 08:03:20.237571 (+ 48805us) write_op.cc:301] APPLY: starting
0812 08:03:20.237591 (+    20us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:20.271511 (+ 33920us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:20.271531 (+    20us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:20.545207 (+273676us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x55a84fa0c218. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facf3c2fbdd 00007facfc1cfe3b 00007facfc0998df 00007facfc09e60a 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d
0812 08:03:20.664197 (+118990us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:20.666402 (+  2205us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:20.666410 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:20.720128 (+ 53718us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:20.721064 (+   936us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:20.722329 (+  1265us) write_op.cc:454] Released schema lock
0812 08:03:20.722588 (+   259us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":48750,"mutex_wait_us":7482,"num_ops":5000,"prepare.queue_time_us":507464,"prepare.run_cpu_time_us":49288,"prepare.run_wall_time_us":49397,"replication_time_us":2696,"spinlock_wait_cycles":36678400}]]}
W20260812 08:03:20.878237 24829 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:37893, user_credentials={real_user=slave}} blocked reactor thread for 63437.1us
I20260812 08:03:20.934405 24891 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51100 (ReqId={client: 1a9df0103ee24e97be61c9330a685648, seq_no=25, attempt_no=0}) took 1052 ms. Trace:
I20260812 08:03:20.934628 24891 rpcz_store.cc:276] 0812 08:03:19.881551 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:19.881615 (+    64us) service_pool.cc:224] Handling call
0812 08:03:20.934381 (+1052766us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:20.246845 (+     0us) write_op.cc:183] PREPARE: starting on tablet 83af86d24c524d669f5437cf6bae86a6
0812 08:03:20.247004 (+   159us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:20.247008 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:03:20.247010 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:20.274602 (+ 27592us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:20.274615 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:20.274621 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:20.295281 (+ 20660us) tablet.cc:701] Row locks acquired
0812 08:03:20.295285 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:20.295367 (+    82us) write_op.cc:270] Start()
0812 08:03:20.295412 (+    45us) write_op.cc:276] Timestamp: P: 1786521800295346 usec, L: 0
0812 08:03:20.295415 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:20.295954 (+   539us) log.cc:844] Serialized 486894 byte log entry
0812 08:03:20.298621 (+  2667us) op_driver.cc:464] REPLICATION: finished
0812 08:03:20.443814 (+145193us) write_op.cc:301] APPLY: starting
0812 08:03:20.443829 (+    15us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:20.472095 (+ 28266us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:20.472117 (+    22us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:20.540550 (+ 68433us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x55a84fa0c218. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facf3c2fbdd 00007facfc1cfe3b 00007facfc0998df 00007facfc09e60a 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d
0812 08:03:20.905943 (+365393us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:20.908082 (+  2139us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:20.908088 (+     6us) write_op.cc:312] APPLY: finished
0812 08:03:20.912638 (+  4550us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:20.933002 (+ 20364us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:20.933979 (+   977us) write_op.cc:454] Released schema lock
0812 08:03:20.934251 (+   272us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":145141,"num_ops":5000,"prepare.queue_time_us":364482,"prepare.run_cpu_time_us":49127,"prepare.run_wall_time_us":51858,"replication_time_us":3175,"spinlock_wait_cycles":26442624}]]}
I20260812 08:03:21.034636 24865 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 39% done.
I20260812 08:03:21.043488 24872 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 39% done.
I20260812 08:03:21.066867 24853 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 39% done.
I20260812 08:03:21.339608 24858 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 39% done.
I20260812 08:03:21.378434 24885 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51100 (ReqId={client: 1a9df0103ee24e97be61c9330a685648, seq_no=28, attempt_no=0}) took 1160 ms. Trace:
I20260812 08:03:21.392944 24885 rpcz_store.cc:276] 0812 08:03:20.217692 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:20.217750 (+    58us) service_pool.cc:224] Handling call
0812 08:03:21.378408 (+1160658us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:20.408379 (+     0us) write_op.cc:183] PREPARE: starting on tablet 83af86d24c524d669f5437cf6bae86a6
0812 08:03:20.408538 (+   159us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:20.408543 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:20.408544 (+     1us) tablet.cc:662] Decoding operations
0812 08:03:20.436793 (+ 28249us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:20.436806 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:20.436813 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:20.457231 (+ 20418us) tablet.cc:701] Row locks acquired
0812 08:03:20.457234 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:20.457316 (+    82us) write_op.cc:270] Start()
0812 08:03:20.457359 (+    43us) write_op.cc:276] Timestamp: P: 1786521800457295 usec, L: 0
0812 08:03:20.457362 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:20.457711 (+   349us) log.cc:844] Serialized 486428 byte log entry
0812 08:03:20.460106 (+  2395us) op_driver.cc:464] REPLICATION: finished
0812 08:03:20.745623 (+285517us) write_op.cc:301] APPLY: starting
0812 08:03:20.745640 (+    17us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:20.782390 (+ 36750us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:20.782407 (+    17us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:21.320558 (+538151us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:21.350633 (+ 30075us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:21.350640 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:21.371424 (+ 20784us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:21.377005 (+  5581us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:21.377995 (+   990us) write_op.cc:454] Released schema lock
0812 08:03:21.378272 (+   277us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":285463,"mutex_wait_us":21915,"num_ops":5000,"prepare.queue_time_us":189425,"prepare.run_cpu_time_us":49218,"prepare.run_wall_time_us":51811,"replication_time_us":2702,"spinlock_wait_cycles":39680}]]}
I20260812 08:03:21.407902 24857 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 39% done.
I20260812 08:03:21.438359 24881 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51100 (ReqId={client: 1a9df0103ee24e97be61c9330a685648, seq_no=30, attempt_no=0}) took 1056 ms. Trace:
I20260812 08:03:21.438581 24881 rpcz_store.cc:276] 0812 08:03:20.382290 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:20.382351 (+    61us) service_pool.cc:224] Handling call
0812 08:03:21.438333 (+1055982us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:20.514358 (+     0us) write_op.cc:183] PREPARE: starting on tablet 83af86d24c524d669f5437cf6bae86a6
0812 08:03:20.514559 (+   201us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:20.514564 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:20.514567 (+     3us) tablet.cc:662] Decoding operations
0812 08:03:20.552471 (+ 37904us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:20.552482 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:20.552489 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:20.573125 (+ 20636us) tablet.cc:701] Row locks acquired
0812 08:03:20.573128 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:20.573285 (+   157us) write_op.cc:270] Start()
0812 08:03:20.573328 (+    43us) write_op.cc:276] Timestamp: P: 1786521800573260 usec, L: 0
0812 08:03:20.573330 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:20.573743 (+   413us) log.cc:844] Serialized 485628 byte log entry
0812 08:03:20.576101 (+  2358us) op_driver.cc:464] REPLICATION: finished
0812 08:03:20.815458 (+239357us) write_op.cc:301] APPLY: starting
0812 08:03:20.815472 (+    14us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:20.853330 (+ 37858us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:20.853352 (+    22us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:20.873105 (+ 19753us) spinlock_profiling.cc:243] Waited 19.6 ms on lock 0x55a84fd4838c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:21.080454 (+207349us) spinlock_profiling.cc:243] Waited 3.38 ms on lock 0x55a84fd4838c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:21.389749 (+309295us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:21.391971 (+  2222us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:21.391979 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:21.426026 (+ 34047us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:21.436995 (+ 10969us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:21.437975 (+   980us) write_op.cc:454] Released schema lock
0812 08:03:21.438225 (+   250us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":239297,"mutex_wait_us":60366,"num_ops":5000,"prepare.queue_time_us":131045,"prepare.run_cpu_time_us":51148,"prepare.run_wall_time_us":61840,"replication_time_us":2746,"spinlock_wait_cycles":52804480}]]}
W20260812 08:03:21.485729 24829 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:37893, user_credentials={real_user=slave}} blocked reactor thread for 50744.9us
I20260812 08:03:21.558418 24853 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 49% done.
I20260812 08:03:21.673949 24871 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 39% done.
I20260812 08:03:21.681238 24861 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 39% done.
I20260812 08:03:21.743700 24860 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 39% done.
I20260812 08:03:21.890506 24855 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 39% done.
W20260812 08:03:21.999248 24829 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:37893, user_credentials={real_user=slave}} blocked reactor thread for 62894.5us
I20260812 08:03:22.005625 24859 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 49% done.
I20260812 08:03:22.236284 24861 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 49% done.
I20260812 08:03:22.281247 24872 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 49% done.
I20260812 08:03:22.434662 24865 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 49% done.
I20260812 08:03:22.453294 24871 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 49% done.
I20260812 08:03:22.485594 24853 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 59% done.
I20260812 08:03:22.552721 24860 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 49% done.
I20260812 08:03:22.567574 24858 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 49% done.
I20260812 08:03:22.686026 24857 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 49% done.
I20260812 08:03:22.710510 24872 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 59% done.
I20260812 08:03:22.777457 24859 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 59% done.
I20260812 08:03:22.931579 24855 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 49% done.
I20260812 08:03:22.965740 24865 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 59% done.
I20260812 08:03:22.989724 24860 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 59% done.
I20260812 08:03:23.063870 24858 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 59% done.
I20260812 08:03:23.091909 24881 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51100 (ReqId={client: 1a9df0103ee24e97be61c9330a685648, seq_no=48, attempt_no=0}) took 1066 ms. Trace:
I20260812 08:03:23.092119 24881 rpcz_store.cc:276] 0812 08:03:22.025892 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:22.025965 (+    73us) service_pool.cc:224] Handling call
0812 08:03:23.091886 (+1065921us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:22.714691 (+     0us) write_op.cc:183] PREPARE: starting on tablet 83af86d24c524d669f5437cf6bae86a6
0812 08:03:22.714857 (+   166us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:22.714862 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:22.714864 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:22.810245 (+ 95381us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:22.810258 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:22.810265 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:22.861410 (+ 51145us) tablet.cc:701] Row locks acquired
0812 08:03:22.861413 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:22.861516 (+   103us) write_op.cc:270] Start()
0812 08:03:22.861566 (+    50us) write_op.cc:276] Timestamp: P: 1786521802861492 usec, L: 0
0812 08:03:22.861569 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:22.861925 (+   356us) log.cc:844] Serialized 485738 byte log entry
0812 08:03:22.864380 (+  2455us) op_driver.cc:464] REPLICATION: finished
0812 08:03:22.864488 (+   108us) write_op.cc:301] APPLY: starting
0812 08:03:22.864506 (+    18us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:22.870392 (+  5886us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:22.870415 (+    23us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:23.083827 (+213412us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:23.085913 (+  2086us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:23.085920 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:23.090379 (+  4459us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:23.090446 (+    67us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:23.091499 (+  1053us) write_op.cc:454] Released schema lock
0812 08:03:23.091764 (+   265us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":54,"num_ops":5000,"prepare.queue_time_us":688049,"prepare.run_cpu_time_us":48671,"prepare.run_wall_time_us":162287,"replication_time_us":2778,"spinlock_wait_cycles":11904}]]}
I20260812 08:03:23.131100 24871 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 59% done.
I20260812 08:03:23.395833 24857 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 59% done.
I20260812 08:03:23.460602 24885 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51100 (ReqId={client: 1a9df0103ee24e97be61c9330a685648, seq_no=52, attempt_no=0}) took 1018 ms. Trace:
I20260812 08:03:23.460811 24885 rpcz_store.cc:276] 0812 08:03:22.441999 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:22.442066 (+    67us) service_pool.cc:224] Handling call
0812 08:03:23.460578 (+1018512us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:23.172220 (+     0us) write_op.cc:183] PREPARE: starting on tablet 83af86d24c524d669f5437cf6bae86a6
0812 08:03:23.172390 (+   170us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:23.172396 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:03:23.172398 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:23.200838 (+ 28440us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:23.200849 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:23.200855 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:23.220825 (+ 19970us) tablet.cc:701] Row locks acquired
0812 08:03:23.220829 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:23.220979 (+   150us) write_op.cc:270] Start()
0812 08:03:23.221032 (+    53us) write_op.cc:276] Timestamp: P: 1786521803220953 usec, L: 0
0812 08:03:23.221035 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:23.221412 (+   377us) log.cc:844] Serialized 486078 byte log entry
0812 08:03:23.223640 (+  2228us) op_driver.cc:464] REPLICATION: finished
0812 08:03:23.223733 (+    93us) write_op.cc:301] APPLY: starting
0812 08:03:23.223749 (+    16us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:23.229621 (+  5872us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:23.229641 (+    20us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:23.439827 (+210186us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:23.442054 (+  2227us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:23.442061 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:23.458943 (+ 16882us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:23.459047 (+   104us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:23.460198 (+  1151us) write_op.cc:454] Released schema lock
0812 08:03:23.460454 (+   256us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":54,"mutex_wait_us":13354,"num_ops":5000,"prepare.queue_time_us":729446,"prepare.run_cpu_time_us":49160,"prepare.run_wall_time_us":49277,"replication_time_us":2576,"spinlock_wait_cycles":117120}]]}
I20260812 08:03:23.563556 24853 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 69% done.
I20260812 08:03:23.573053 24861 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 59% done.
I20260812 08:03:23.861500 24855 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 59% done.
I20260812 08:03:23.895701 24865 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 69% done.
I20260812 08:03:23.924765 24881 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51100 (ReqId={client: 1a9df0103ee24e97be61c9330a685648, seq_no=56, attempt_no=0}) took 1076 ms. Trace:
I20260812 08:03:23.932934 24881 rpcz_store.cc:276] 0812 08:03:22.847992 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:22.848048 (+    56us) service_pool.cc:224] Handling call
0812 08:03:23.924741 (+1076693us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:23.520190 (+     0us) write_op.cc:183] PREPARE: starting on tablet 83af86d24c524d669f5437cf6bae86a6
0812 08:03:23.520355 (+   165us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:23.520360 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:23.520362 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:23.594676 (+ 74314us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:23.594689 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:23.594696 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:23.636703 (+ 42007us) tablet.cc:701] Row locks acquired
0812 08:03:23.636707 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:23.636787 (+    80us) write_op.cc:270] Start()
0812 08:03:23.636831 (+    44us) write_op.cc:276] Timestamp: P: 1786521803636768 usec, L: 0
0812 08:03:23.636834 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:23.645811 (+  8977us) log.cc:844] Serialized 486321 byte log entry
0812 08:03:23.648205 (+  2394us) op_driver.cc:464] REPLICATION: finished
0812 08:03:23.648309 (+   104us) write_op.cc:301] APPLY: starting
0812 08:03:23.648327 (+    18us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:23.654178 (+  5851us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:23.654197 (+    19us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:23.902204 (+248007us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:23.904268 (+  2064us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:23.904275 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:23.923111 (+ 18836us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:23.923219 (+   108us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:23.924359 (+  1140us) write_op.cc:454] Released schema lock
0812 08:03:23.924595 (+   236us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":62,"num_ops":5000,"prepare.queue_time_us":671395,"prepare.run_cpu_time_us":48771,"prepare.run_wall_time_us":127970,"replication_time_us":11331,"spinlock_wait_cycles":9088}]]}
I20260812 08:03:24.038605 24872 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 69% done.
I20260812 08:03:24.054467 24859 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 69% done.
I20260812 08:03:24.141017 24860 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 69% done.
W20260812 08:03:24.154062 24829 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:37893, user_credentials={real_user=slave}} blocked reactor thread for 46365.2us
I20260812 08:03:24.156816 24858 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 69% done.
I20260812 08:03:24.288951 24871 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 69% done.
I20260812 08:03:24.430699 24855 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 69% done.
I20260812 08:03:24.531463 24857 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 69% done.
W20260812 08:03:24.585254 24829 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:37893, user_credentials={real_user=slave}} blocked reactor thread for 44222.2us
I20260812 08:03:24.626088 24881 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51100 (ReqId={client: 1a9df0103ee24e97be61c9330a685648, seq_no=63, attempt_no=0}) took 1069 ms. Trace:
I20260812 08:03:24.640937 24881 rpcz_store.cc:276] 0812 08:03:23.556382 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:23.556432 (+    50us) service_pool.cc:224] Handling call
0812 08:03:24.626064 (+1069632us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:24.156317 (+     0us) write_op.cc:183] PREPARE: starting on tablet 83af86d24c524d669f5437cf6bae86a6
0812 08:03:24.156478 (+   161us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:24.156483 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:24.156484 (+     1us) tablet.cc:662] Decoding operations
0812 08:03:24.193811 (+ 37327us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:24.193824 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:24.193831 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:24.232661 (+ 38830us) tablet.cc:701] Row locks acquired
0812 08:03:24.232665 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:24.232723 (+    58us) write_op.cc:270] Start()
0812 08:03:24.232760 (+    37us) write_op.cc:276] Timestamp: P: 1786521804232710 usec, L: 0
0812 08:03:24.232762 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:24.233106 (+   344us) log.cc:844] Serialized 485721 byte log entry
0812 08:03:24.235770 (+  2664us) op_driver.cc:464] REPLICATION: finished
0812 08:03:24.235907 (+   137us) write_op.cc:301] APPLY: starting
0812 08:03:24.235923 (+    16us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:24.243917 (+  7994us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:24.243938 (+    21us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:24.473083 (+229145us) spinlock_profiling.cc:243] Waited 8.02 ms on lock 0x55a84fa0c218. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facf3c2fbdd 00007facfc1cfe3b 00007facfc0998df 00007facfc09e60a 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d
0812 08:03:24.597312 (+124229us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:24.599415 (+  2103us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:24.599422 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:24.624293 (+ 24871us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:24.624429 (+   136us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:24.625683 (+  1254us) write_op.cc:454] Released schema lock
0812 08:03:24.625938 (+   255us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":77,"mutex_wait_us":1929,"num_ops":5000,"prepare.queue_time_us":599196,"prepare.run_cpu_time_us":49096,"prepare.run_wall_time_us":76870,"replication_time_us":2978,"spinlock_wait_cycles":18477312}]]}
I20260812 08:03:24.762434 24861 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 69% done.
I20260812 08:03:24.762434 24894 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51100 (ReqId={client: 1a9df0103ee24e97be61c9330a685648, seq_no=64, attempt_no=0}) took 1062 ms. Trace:
I20260812 08:03:24.776976 24894 rpcz_store.cc:276] 0812 08:03:23.700381 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:23.700432 (+    51us) service_pool.cc:224] Handling call
0812 08:03:24.762416 (+1061984us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:24.233241 (+     0us) write_op.cc:183] PREPARE: starting on tablet 83af86d24c524d669f5437cf6bae86a6
0812 08:03:24.233399 (+   158us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:24.233404 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:24.233406 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:24.283285 (+ 49879us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:24.283298 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:24.283306 (+     8us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:24.307983 (+ 24677us) tablet.cc:701] Row locks acquired
0812 08:03:24.307986 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:24.308074 (+    88us) write_op.cc:270] Start()
0812 08:03:24.308120 (+    46us) write_op.cc:276] Timestamp: P: 1786521804308053 usec, L: 0
0812 08:03:24.308123 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:24.308477 (+   354us) log.cc:844] Serialized 486085 byte log entry
0812 08:03:24.310729 (+  2252us) op_driver.cc:464] REPLICATION: finished
0812 08:03:24.310818 (+    89us) write_op.cc:301] APPLY: starting
0812 08:03:24.310832 (+    14us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:24.316726 (+  5894us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:24.316743 (+    17us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:24.737814 (+421071us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:24.739919 (+  2105us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:24.739925 (+     6us) write_op.cc:312] APPLY: finished
0812 08:03:24.744403 (+  4478us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:24.760999 (+ 16596us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:24.762003 (+  1004us) write_op.cc:454] Released schema lock
0812 08:03:24.762280 (+   277us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":51,"mutex_wait_us":1938,"num_ops":5000,"prepare.queue_time_us":531983,"prepare.run_cpu_time_us":48137,"prepare.run_wall_time_us":75321,"replication_time_us":2579,"spinlock_wait_cycles":29696}]]}
I20260812 08:03:24.864722 24853 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 79% done.
W20260812 08:03:24.868543 24829 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:37893, user_credentials={real_user=slave}} blocked reactor thread for 56066.4us
I20260812 08:03:24.933287 24891 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51100 (ReqId={client: 1a9df0103ee24e97be61c9330a685648, seq_no=65, attempt_no=0}) took 1146 ms. Trace:
I20260812 08:03:24.933490 24891 rpcz_store.cc:276] 0812 08:03:23.786702 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:23.786754 (+    52us) service_pool.cc:224] Handling call
0812 08:03:24.933263 (+1146509us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:24.308621 (+     0us) write_op.cc:183] PREPARE: starting on tablet 83af86d24c524d669f5437cf6bae86a6
0812 08:03:24.308798 (+   177us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:24.308803 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:24.308804 (+     1us) tablet.cc:662] Decoding operations
0812 08:03:24.336689 (+ 27885us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:24.336703 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:24.336710 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:24.357702 (+ 20992us) tablet.cc:701] Row locks acquired
0812 08:03:24.357706 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:24.357794 (+    88us) write_op.cc:270] Start()
0812 08:03:24.357841 (+    47us) write_op.cc:276] Timestamp: P: 1786521804357771 usec, L: 0
0812 08:03:24.357843 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:24.358929 (+  1086us) log.cc:844] Serialized 485537 byte log entry
0812 08:03:24.369714 (+ 10785us) op_driver.cc:464] REPLICATION: finished
0812 08:03:24.525540 (+155826us) write_op.cc:301] APPLY: starting
0812 08:03:24.525558 (+    18us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:24.531365 (+  5807us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:24.531388 (+    23us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:24.840993 (+309605us) spinlock_profiling.cc:243] Waited 4.34 ms on lock 0x55a84fd4838c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:24.921269 (+ 80276us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:24.923524 (+  2255us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:24.923532 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:24.931493 (+  7961us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:24.931843 (+   350us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:24.932866 (+  1023us) write_op.cc:454] Released schema lock
0812 08:03:24.933144 (+   278us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":155773,"num_ops":5000,"prepare.queue_time_us":521220,"prepare.run_cpu_time_us":49552,"prepare.run_wall_time_us":50400,"replication_time_us":11840,"spinlock_wait_cycles":10003200}]]}
I20260812 08:03:24.966774 24872 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 79% done.
I20260812 08:03:24.971874 24865 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 79% done.
I20260812 08:03:25.059363 24859 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 79% done.
I20260812 08:03:25.085814 24881 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51100 (ReqId={client: 1a9df0103ee24e97be61c9330a685648, seq_no=67, attempt_no=0}) took 1004 ms. Trace:
I20260812 08:03:25.086030 24881 rpcz_store.cc:276] 0812 08:03:24.081754 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:24.081802 (+    48us) service_pool.cc:224] Handling call
0812 08:03:25.085789 (+1003987us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:24.408247 (+     0us) write_op.cc:183] PREPARE: starting on tablet 83af86d24c524d669f5437cf6bae86a6
0812 08:03:24.408417 (+   170us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:24.408422 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:24.408424 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:24.436849 (+ 28425us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:24.436863 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:24.436870 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:24.468860 (+ 31990us) tablet.cc:701] Row locks acquired
0812 08:03:24.468863 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:24.468981 (+   118us) write_op.cc:270] Start()
0812 08:03:24.469031 (+    50us) write_op.cc:276] Timestamp: P: 1786521804468948 usec, L: 0
0812 08:03:24.469034 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:24.469389 (+   355us) log.cc:844] Serialized 485344 byte log entry
0812 08:03:24.471847 (+  2458us) op_driver.cc:464] REPLICATION: finished
0812 08:03:24.657508 (+185661us) write_op.cc:301] APPLY: starting
0812 08:03:24.657527 (+    19us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:24.671398 (+ 13871us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:24.671424 (+    26us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:25.062750 (+391326us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:25.064797 (+  2047us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:25.064804 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:25.084068 (+ 19264us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:25.084406 (+   338us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:25.085402 (+   996us) write_op.cc:454] Released schema lock
0812 08:03:25.085668 (+   266us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":185605,"mutex_wait_us":5609,"num_ops":5000,"prepare.queue_time_us":325722,"prepare.run_cpu_time_us":50049,"prepare.run_wall_time_us":61231,"replication_time_us":2784,"spinlock_wait_cycles":17920}]]}
I20260812 08:03:25.179749 24871 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 79% done.
I20260812 08:03:25.253078 24894 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51100 (ReqId={client: 1a9df0103ee24e97be61c9330a685648, seq_no=68, attempt_no=0}) took 1020 ms. Trace:
I20260812 08:03:25.253288 24894 rpcz_store.cc:276] 0812 08:03:24.232254 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:24.232316 (+    62us) service_pool.cc:224] Handling call
0812 08:03:25.253053 (+1020737us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:24.469542 (+     0us) write_op.cc:183] PREPARE: starting on tablet 83af86d24c524d669f5437cf6bae86a6
0812 08:03:24.469703 (+   161us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:24.469707 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:03:24.469709 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:24.498311 (+ 28602us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:24.498323 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:24.498330 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:24.538149 (+ 39819us) tablet.cc:701] Row locks acquired
0812 08:03:24.538151 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:03:24.538213 (+    62us) write_op.cc:270] Start()
0812 08:03:24.538248 (+    35us) write_op.cc:276] Timestamp: P: 1786521804538199 usec, L: 0
0812 08:03:24.538250 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:24.538564 (+   314us) log.cc:844] Serialized 486558 byte log entry
0812 08:03:24.540786 (+  2222us) op_driver.cc:464] REPLICATION: finished
0812 08:03:24.777918 (+237132us) write_op.cc:301] APPLY: starting
0812 08:03:24.777936 (+    18us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:24.812930 (+ 34994us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:24.812952 (+    22us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:24.836801 (+ 23849us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x55a84fd4838c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:25.230519 (+393718us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:25.232634 (+  2115us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:25.232641 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:25.251019 (+ 18378us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:25.251344 (+   325us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:25.252625 (+  1281us) write_op.cc:454] Released schema lock
0812 08:03:25.252875 (+   250us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":24832,"child_traces":[["op",{"apply.queue_time_us":237081,"num_ops":5000,"prepare.queue_time_us":236521,"prepare.run_cpu_time_us":49935,"prepare.run_wall_time_us":69099,"replication_time_us":2514,"spinlock_wait_cycles":36365312}]]}
I20260812 08:03:25.455159 24855 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 79% done.
I20260812 08:03:25.506167 24857 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 79% done.
I20260812 08:03:25.600610 24860 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 79% done.
W20260812 08:03:25.670269 24829 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:37893, user_credentials={real_user=slave}} blocked reactor thread for 58036.9us
W20260812 08:03:25.729864 24829 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:37893, user_credentials={real_user=slave}} blocked reactor thread for 59279.4us
I20260812 08:03:25.741163 24861 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 79% done.
I20260812 08:03:25.786969 24858 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 79% done.
I20260812 08:03:25.834959 24865 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 89% done.
I20260812 08:03:25.909456 24853 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 89% done.
I20260812 08:03:26.194350 24871 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 89% done.
I20260812 08:03:26.209908 24872 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 89% done.
I20260812 08:03:26.217233 24859 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 89% done.
I20260812 08:03:26.253693 24860 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 89% done.
I20260812 08:03:26.355013 24857 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 89% done.
I20260812 08:03:26.367208 24858 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 89% done.
I20260812 08:03:26.440201 24894 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51100 (ReqId={client: 1a9df0103ee24e97be61c9330a685648, seq_no=81, attempt_no=0}) took 1061 ms. Trace:
I20260812 08:03:26.445353 24894 rpcz_store.cc:276] 0812 08:03:25.378899 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:25.378954 (+    55us) service_pool.cc:224] Handling call
0812 08:03:26.440178 (+1061224us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:25.986543 (+     0us) write_op.cc:183] PREPARE: starting on tablet 83af86d24c524d669f5437cf6bae86a6
0812 08:03:25.986704 (+   161us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:25.986709 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:25.986711 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:26.051593 (+ 64882us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:26.051607 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:26.051614 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:26.111146 (+ 59532us) tablet.cc:701] Row locks acquired
0812 08:03:26.111160 (+    14us) write_op.cc:260] PREPARE: finished
0812 08:03:26.111244 (+    84us) write_op.cc:270] Start()
0812 08:03:26.111290 (+    46us) write_op.cc:276] Timestamp: P: 1786521806111221 usec, L: 0
0812 08:03:26.111293 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:26.111755 (+   462us) log.cc:844] Serialized 486601 byte log entry
0812 08:03:26.114178 (+  2423us) op_driver.cc:464] REPLICATION: finished
0812 08:03:26.114253 (+    75us) write_op.cc:301] APPLY: starting
0812 08:03:26.114268 (+    15us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:26.122891 (+  8623us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:26.122916 (+    25us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:26.410784 (+287868us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:26.433808 (+ 23024us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:26.433816 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:26.438692 (+  4876us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:26.438795 (+   103us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:26.439818 (+  1023us) write_op.cc:454] Released schema lock
0812 08:03:26.440067 (+   249us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":29,"num_ops":5000,"prepare.queue_time_us":606864,"prepare.run_cpu_time_us":48125,"prepare.run_wall_time_us":125298,"replication_time_us":2858,"spinlock_wait_cycles":27264}]]}
I20260812 08:03:26.478291 24855 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 89% done.
W20260812 08:03:26.529944 24829 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:37893, user_credentials={real_user=slave}} blocked reactor thread for 52093us
I20260812 08:03:26.553650 24861 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 89% done.
I20260812 08:03:26.574580 24853 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 99% done.
I20260812 08:03:26.574749 24853 full_stack-insert-scan-test.cc:280] Insertion thread 1 of  10 is 100% done.
W20260812 08:03:26.726365 24829 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:37893, user_credentials={real_user=slave}} blocked reactor thread for 45762.1us
I20260812 08:03:26.926373 24865 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 99% done.
I20260812 08:03:26.940936 24865 full_stack-insert-scan-test.cc:280] Insertion thread 8 of  10 is 100% done.
I20260812 08:03:26.944815 24872 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 99% done.
I20260812 08:03:26.944976 24872 full_stack-insert-scan-test.cc:280] Insertion thread 10 of  10 is 100% done.
I20260812 08:03:27.030508 24859 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 99% done.
I20260812 08:03:27.035218 24859 full_stack-insert-scan-test.cc:280] Insertion thread 5 of  10 is 100% done.
I20260812 08:03:27.102267 24860 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 99% done.
I20260812 08:03:27.102470 24860 full_stack-insert-scan-test.cc:280] Insertion thread 6 of  10 is 100% done.
I20260812 08:03:27.128065 24858 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 99% done.
I20260812 08:03:27.128288 24858 full_stack-insert-scan-test.cc:280] Insertion thread 4 of  10 is 100% done.
I20260812 08:03:27.159262 24885 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51100 (ReqId={client: 1a9df0103ee24e97be61c9330a685648, seq_no=88, attempt_no=0}) took 1089 ms. Trace:
I20260812 08:03:27.159428 24885 rpcz_store.cc:276] 0812 08:03:26.069813 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:26.069880 (+    67us) service_pool.cc:224] Handling call
0812 08:03:27.159243 (+1089363us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:26.824625 (+     0us) write_op.cc:183] PREPARE: starting on tablet 83af86d24c524d669f5437cf6bae86a6
0812 08:03:26.824801 (+   176us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:26.824806 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:26.824808 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:26.865654 (+ 40846us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:26.865669 (+    15us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:26.865676 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:26.925417 (+ 59741us) tablet.cc:701] Row locks acquired
0812 08:03:26.925420 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:26.925514 (+    94us) write_op.cc:270] Start()
0812 08:03:26.925559 (+    45us) write_op.cc:276] Timestamp: P: 1786521806925491 usec, L: 0
0812 08:03:26.925561 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:26.925929 (+   368us) log.cc:844] Serialized 486781 byte log entry
0812 08:03:26.928195 (+  2266us) op_driver.cc:464] REPLICATION: finished
0812 08:03:26.928270 (+    75us) write_op.cc:301] APPLY: starting
0812 08:03:26.928286 (+    16us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:26.937288 (+  9002us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:26.937315 (+    27us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:27.143755 (+206440us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:27.146008 (+  2253us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:27.146016 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:27.150767 (+  4751us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:27.150862 (+    95us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:27.158922 (+  8060us) write_op.cc:454] Released schema lock
0812 08:03:27.159135 (+   213us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":28,"mutex_wait_us":8101,"num_ops":5000,"prepare.queue_time_us":754022,"prepare.run_cpu_time_us":60885,"prepare.run_wall_time_us":101392,"replication_time_us":2606,"spinlock_wait_cycles":7040}]]}
I20260812 08:03:27.224526 24894 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:51100 (ReqId={client: 1a9df0103ee24e97be61c9330a685648, seq_no=89, attempt_no=0}) took 1010 ms. Trace:
I20260812 08:03:27.224749 24894 rpcz_store.cc:276] 0812 08:03:26.214065 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:26.214127 (+    62us) service_pool.cc:224] Handling call
0812 08:03:27.224501 (+1010374us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:26.926064 (+     0us) write_op.cc:183] PREPARE: starting on tablet 83af86d24c524d669f5437cf6bae86a6
0812 08:03:26.926225 (+   161us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:26.926230 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:26.926232 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:26.997190 (+ 70958us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:26.997202 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:26.997209 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:27.035475 (+ 38266us) tablet.cc:701] Row locks acquired
0812 08:03:27.035478 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:27.035559 (+    81us) write_op.cc:270] Start()
0812 08:03:27.035603 (+    44us) write_op.cc:276] Timestamp: P: 1786521807035539 usec, L: 0
0812 08:03:27.035605 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:27.036077 (+   472us) log.cc:844] Serialized 485917 byte log entry
0812 08:03:27.038424 (+  2347us) op_driver.cc:464] REPLICATION: finished
0812 08:03:27.038529 (+   105us) write_op.cc:301] APPLY: starting
0812 08:03:27.038547 (+    18us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:27.044548 (+  6001us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:27.044573 (+    25us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:27.120107 (+ 75534us) spinlock_profiling.cc:243] Waited 8.67 ms on lock 0x55a84fa0c218. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facf3c2fbdd 00007facfc1cfe3b 00007facfc0998df 00007facfc09e60a 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d
0812 08:03:27.215880 (+ 95773us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:27.217984 (+  2104us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:27.217991 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:27.222570 (+  4579us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:27.222926 (+   356us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:27.224125 (+  1199us) write_op.cc:454] Released schema lock
0812 08:03:27.224376 (+   251us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":62,"num_ops":5000,"prepare.queue_time_us":711158,"prepare.run_cpu_time_us":48740,"prepare.run_wall_time_us":130908,"replication_time_us":2788,"spinlock_wait_cycles":19975936}]]}
I20260812 08:03:27.267674 24857 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 99% done.
I20260812 08:03:27.267884 24857 full_stack-insert-scan-test.cc:280] Insertion thread 3 of  10 is 100% done.
I20260812 08:03:27.281617 24871 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 99% done.
I20260812 08:03:27.281810 24871 full_stack-insert-scan-test.cc:280] Insertion thread 9 of  10 is 100% done.
I20260812 08:03:27.489543 24861 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 99% done.
I20260812 08:03:27.489737 24861 full_stack-insert-scan-test.cc:280] Insertion thread 7 of  10 is 100% done.
I20260812 08:03:27.519119 24855 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 99% done.
I20260812 08:03:27.519304 24855 full_stack-insert-scan-test.cc:280] Insertion thread 2 of  10 is 100% done.
W20260812 08:03:27.614574 24829 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:37893, user_credentials={real_user=slave}} blocked reactor thread for 50814.9us
I20260812 08:03:28.362815 24143 full_stack-insert-scan-test.cc:333] Time spent concurrent inserts (500000 rows, 10 threads): real 10.204s	user 0.000s	sys 0.000s
I20260812 08:03:28.362911 24143 full_stack-insert-scan-test.cc:346] Doing test scans on table of 500000 rows.
I20260812 08:03:28.841332 24143 full_stack-insert-scan-test.cc:453] Time spent empty projection, 0 col: real 0.238s	user 0.001s	sys 0.000s
I20260812 08:03:29.216125 24143 full_stack-insert-scan-test.cc:453] Time spent key scan, 1 col: real 0.295s	user 0.001s	sys 0.000s
I20260812 08:03:29.952232 24143 full_stack-insert-scan-test.cc:453] Time spent full schema scan, no cache, 10 col: real 0.719s	user 0.030s	sys 0.004s
I20260812 08:03:30.841979 24143 full_stack-insert-scan-test.cc:453] Time spent fault-tolerant full schema scan, no cache, 10 col: real 0.872s	user 0.034s	sys 0.001s
I20260812 08:03:31.575932 24143 full_stack-insert-scan-test.cc:453] Time spent full schema scan, 10 col: real 0.716s	user 0.029s	sys 0.006s
I20260812 08:03:31.974416 24143 full_stack-insert-scan-test.cc:453] Time spent String projection, 1 col: real 0.377s	user 0.027s	sys 0.000s
I20260812 08:03:32.423170 24143 full_stack-insert-scan-test.cc:453] Time spent Int32 projection, 4 col: real 0.396s	user 0.002s	sys 0.000s
I20260812 08:03:32.860287 24143 full_stack-insert-scan-test.cc:453] Time spent Int64 projection, 4 col: real 0.408s	user 0.004s	sys 0.000s
I20260812 08:03:32.861008 24143 tablet_server.cc:179] TabletServer@127.23.147.193:0 shutting down...
W20260812 08:03:32.861249 24829 client.cc:2042] Couldn't close scanner 67d9ecbc3bf947e7bf4be9e8ba186682: Service unavailable: reactor is shutting down (error 108)
W20260812 08:03:32.861480 24693 connection.cc:573] server connection from 127.0.0.1:51100 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 08:03:32.872777 24143 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260812 08:03:32.873056 24143 tablet_replica.cc:333] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066: stopping tablet replica
I20260812 08:03:32.873199 24143 raft_consensus.cc:2243] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066 [term 1 LEADER]: Raft consensus shutting down.
I20260812 08:03:32.873386 24143 raft_consensus.cc:2272] T 83af86d24c524d669f5437cf6bae86a6 P fb942e9d90d34e929ce1b419cbb43066 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 08:03:32.905731 24143 tablet_server.cc:196] TabletServer@127.23.147.193:0 shutdown complete.
I20260812 08:03:32.908041 24143 master.cc:562] Master@127.23.147.254:37241 shutting down...
I20260812 08:03:33.046139 24143 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462 [term 1 LEADER]: Raft consensus shutting down.
I20260812 08:03:33.046368 24143 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 08:03:33.046438 24143 tablet_replica.cc:333] T 00000000000000000000000000000000 P 33b59b09bc0a4465a3e983805d6e8462: stopping tablet replica
I20260812 08:03:33.058707 24143 master.cc:584] Master@127.23.147.254:37241 shutdown complete.
[       OK ] EvictionPolicyTypes/FullStackInsertScanTestWithEvictionPolicy.MRSOnlyStressTest/1 (15065 ms)
[ RUN      ] EvictionPolicyTypes/FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest/0
I20260812 08:03:33.070144 24143 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.147.254:37597
I20260812 08:03:33.070499 24143 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 08:03:33.072535 24143 server_base.cc:1061] running on GCE node
W20260812 08:03:33.072646 24905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:03:33.072649 24900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:03:33.072540 24899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:03:33.073045 24143 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 08:03:33.073091 24143 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 08:03:33.073104 24143 hybrid_clock.cc:648] HybridClock initialized: now 1786521813073104 us; error 0 us; skew 500 ppm
I20260812 08:03:33.073951 24143 webserver.cc:533] Webserver started at http://127.23.147.254:35743/ using document root <none> and password file <none>
I20260812 08:03:33.074087 24143 fs_manager.cc:362] Metadata directory not provided
I20260812 08:03:33.074129 24143 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:03:33.074189 24143 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:03:33.074523 24143 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest_0.1786521782163021-24143-0/minicluster-data/master-0-root/instance:
uuid: "fbc968d00ac2415fb374f72e1213355b"
format_stamp: "Formatted at 2026-08-12 08:03:33 on dist-test-slave-ncp9"
I20260812 08:03:33.075917 24143 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.000s
I20260812 08:03:33.076984 24921 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:33.077268 24143 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
I20260812 08:03:33.077339 24143 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest_0.1786521782163021-24143-0/minicluster-data/master-0-root
uuid: "fbc968d00ac2415fb374f72e1213355b"
format_stamp: "Formatted at 2026-08-12 08:03:33 on dist-test-slave-ncp9"
I20260812 08:03:33.077419 24143 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest_0.1786521782163021-24143-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest_0.1786521782163021-24143-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest_0.1786521782163021-24143-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:03:33.096948 24143 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:03:33.097391 24143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:03:33.103528 24143 rpc_server.cc:307] RPC server started. Bound to: 127.23.147.254:37597
I20260812 08:03:33.119791 25013 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.147.254:37597 every 8 connection(s)
I20260812 08:03:33.120345 25014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:03:33.122259 25014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b: Bootstrap starting.
I20260812 08:03:33.123102 25014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b: Neither blocks nor log segments found. Creating new log.
I20260812 08:03:33.124153 25014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b: No bootstrap required, opened a new log
I20260812 08:03:33.124549 25014 raft_consensus.cc:359] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc968d00ac2415fb374f72e1213355b" member_type: VOTER }
I20260812 08:03:33.124640 25014 raft_consensus.cc:385] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:03:33.124661 25014 raft_consensus.cc:740] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc968d00ac2415fb374f72e1213355b, State: Initialized, Role: FOLLOWER
I20260812 08:03:33.124805 25014 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc968d00ac2415fb374f72e1213355b" member_type: VOTER }
I20260812 08:03:33.124913 25014 raft_consensus.cc:399] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 08:03:33.124950 25014 raft_consensus.cc:493] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 08:03:33.125000 25014 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:03:33.125689 25014 raft_consensus.cc:515] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc968d00ac2415fb374f72e1213355b" member_type: VOTER }
I20260812 08:03:33.125808 25014 leader_election.cc:304] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbc968d00ac2415fb374f72e1213355b; no voters: 
I20260812 08:03:33.125998 25014 leader_election.cc:290] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 08:03:33.126153 25022 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:03:33.126374 25022 raft_consensus.cc:697] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b [term 1 LEADER]: Becoming Leader. State: Replica: fbc968d00ac2415fb374f72e1213355b, State: Running, Role: LEADER
I20260812 08:03:33.126416 25014 sys_catalog.cc:565] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:03:33.126540 25022 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc968d00ac2415fb374f72e1213355b" member_type: VOTER }
I20260812 08:03:33.127017 25028 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbc968d00ac2415fb374f72e1213355b. Latest consensus state: current_term: 1 leader_uuid: "fbc968d00ac2415fb374f72e1213355b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc968d00ac2415fb374f72e1213355b" member_type: VOTER } }
I20260812 08:03:33.127041 25023 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbc968d00ac2415fb374f72e1213355b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc968d00ac2415fb374f72e1213355b" member_type: VOTER } }
I20260812 08:03:33.127231 25023 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b [sys.catalog]: This master's current role is: LEADER
I20260812 08:03:33.127223 25028 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b [sys.catalog]: This master's current role is: LEADER
I20260812 08:03:33.127568 25043 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:03:33.128417 25043 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:03:33.128571 24143 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260812 08:03:33.130378 25043 catalog_manager.cc:1383] Generated new cluster ID: 1fdda03c6cde46d3968dd9bd11e011a0
I20260812 08:03:33.130453 25043 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:03:33.136848 25043 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:03:33.137436 25043 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:03:33.146767 25043 catalog_manager.cc:6092] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b: Generated new TSK 0
I20260812 08:03:33.146955 25043 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:03:33.161283 24143 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260812 08:03:33.163650 24143 server_base.cc:1061] running on GCE node
W20260812 08:03:33.163740 25072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:03:33.163877 25066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:03:33.163740 25069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:03:33.164207 24143 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 08:03:33.164259 24143 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 08:03:33.164281 24143 hybrid_clock.cc:648] HybridClock initialized: now 1786521813164281 us; error 0 us; skew 500 ppm
I20260812 08:03:33.165328 24143 webserver.cc:533] Webserver started at http://127.23.147.193:39335/ using document root <none> and password file <none>
I20260812 08:03:33.165496 24143 fs_manager.cc:362] Metadata directory not provided
I20260812 08:03:33.165536 24143 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:03:33.165591 24143 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:03:33.165977 24143 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest_0.1786521782163021-24143-0/minicluster-data/ts-0-root/instance:
uuid: "8a38e20f59a940439e0128fed1c68688"
format_stamp: "Formatted at 2026-08-12 08:03:33 on dist-test-slave-ncp9"
I20260812 08:03:33.167433 24143 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.001s	sys 0.001s
I20260812 08:03:33.168370 25082 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:33.168618 24143 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
I20260812 08:03:33.168690 24143 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest_0.1786521782163021-24143-0/minicluster-data/ts-0-root
uuid: "8a38e20f59a940439e0128fed1c68688"
format_stamp: "Formatted at 2026-08-12 08:03:33 on dist-test-slave-ncp9"
I20260812 08:03:33.168763 24143 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest_0.1786521782163021-24143-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest_0.1786521782163021-24143-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest_0.1786521782163021-24143-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:03:33.187315 24143 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:03:33.187759 24143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:03:33.188086 24143 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:03:33.188584 24143 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:03:33.188627 24143 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:33.188678 24143 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:03:33.188704 24143 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:33.193148 24143 rpc_server.cc:307] RPC server started. Bound to: 127.23.147.193:45861
I20260812 08:03:33.194989 25191 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.147.193:45861 every 8 connection(s)
I20260812 08:03:33.204141 25192 heartbeater.cc:344] Connected to a master server at 127.23.147.254:37597
I20260812 08:03:33.204283 25192 heartbeater.cc:461] Registering TS with master...
I20260812 08:03:33.204524 25192 heartbeater.cc:507] Master 127.23.147.254:37597 requested a full tablet report, sending...
I20260812 08:03:33.205231 24948 ts_manager.cc:194] Registered new tserver with Master: 8a38e20f59a940439e0128fed1c68688 (127.23.147.193:45861)
I20260812 08:03:33.205958 24948 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55088
I20260812 08:03:33.206081 24143 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.012215935s
I20260812 08:03:33.212880 24948 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55092:
name: "full-stack-mrs-test-tbl"
schema {
  columns {
    name: "key"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int32_val1"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int32_val2"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int32_val3"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int32_val4"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int64_val1"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int64_val2"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int64_val3"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int64_val4"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20260812 08:03:33.221382 25130 tablet_service.cc:1511] Processing CreateTablet for tablet 304b715f38bc4a81940b24b40519223d (DEFAULT_TABLE table=full-stack-mrs-test-tbl [id=050258f5933a405bb213c3bcdc44548d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:03:33.221694 25130 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 304b715f38bc4a81940b24b40519223d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:03:33.223845 25216 tablet_bootstrap.cc:492] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688: Bootstrap starting.
I20260812 08:03:33.224753 25216 tablet_bootstrap.cc:654] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688: Neither blocks nor log segments found. Creating new log.
I20260812 08:03:33.225961 25216 tablet_bootstrap.cc:492] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688: No bootstrap required, opened a new log
I20260812 08:03:33.226059 25216 ts_tablet_manager.cc:1403] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688: Time spent bootstrapping tablet: real 0.002s	user 0.000s	sys 0.002s
I20260812 08:03:33.226519 25216 raft_consensus.cc:359] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a38e20f59a940439e0128fed1c68688" member_type: VOTER last_known_addr { host: "127.23.147.193" port: 45861 } }
I20260812 08:03:33.226646 25216 raft_consensus.cc:385] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:03:33.226699 25216 raft_consensus.cc:740] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a38e20f59a940439e0128fed1c68688, State: Initialized, Role: FOLLOWER
I20260812 08:03:33.226855 25216 consensus_queue.cc:260] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a38e20f59a940439e0128fed1c68688" member_type: VOTER last_known_addr { host: "127.23.147.193" port: 45861 } }
I20260812 08:03:33.226979 25216 raft_consensus.cc:399] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 08:03:33.227035 25216 raft_consensus.cc:493] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 08:03:33.227094 25216 raft_consensus.cc:3060] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:03:33.227918 25216 raft_consensus.cc:515] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a38e20f59a940439e0128fed1c68688" member_type: VOTER last_known_addr { host: "127.23.147.193" port: 45861 } }
I20260812 08:03:33.228055 25216 leader_election.cc:304] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a38e20f59a940439e0128fed1c68688; no voters: 
I20260812 08:03:33.228245 25216 leader_election.cc:290] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 08:03:33.228354 25219 raft_consensus.cc:2804] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:03:33.228550 25219 raft_consensus.cc:697] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688 [term 1 LEADER]: Becoming Leader. State: Replica: 8a38e20f59a940439e0128fed1c68688, State: Running, Role: LEADER
I20260812 08:03:33.228600 25216 ts_tablet_manager.cc:1434] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20260812 08:03:33.228688 25192 heartbeater.cc:499] Master 127.23.147.254:37597 was elected leader, sending a full tablet report...
I20260812 08:03:33.228757 25219 consensus_queue.cc:237] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a38e20f59a940439e0128fed1c68688" member_type: VOTER last_known_addr { host: "127.23.147.193" port: 45861 } }
I20260812 08:03:33.230019 24948 catalog_manager.cc:5719] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a38e20f59a940439e0128fed1c68688 (127.23.147.193). New cstate: current_term: 1 leader_uuid: "8a38e20f59a940439e0128fed1c68688" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a38e20f59a940439e0128fed1c68688" member_type: VOTER last_known_addr { host: "127.23.147.193" port: 45861 } health_report { overall_health: HEALTHY } } }
I20260812 08:03:33.743093 25225 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 9% done.
I20260812 08:03:33.767776 25237 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 9% done.
I20260812 08:03:33.806665 25232 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 9% done.
I20260812 08:03:33.813577 25226 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 9% done.
I20260812 08:03:33.823144 25234 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 9% done.
I20260812 08:03:33.897054 25227 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 9% done.
I20260812 08:03:33.960017 25238 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 9% done.
I20260812 08:03:34.039702 25239 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 9% done.
I20260812 08:03:34.053223 25230 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 9% done.
I20260812 08:03:34.054487 25235 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 9% done.
I20260812 08:03:34.269434 25232 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 19% done.
I20260812 08:03:34.413560 25237 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 19% done.
I20260812 08:03:34.492048 25226 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 19% done.
I20260812 08:03:34.508683 25225 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 19% done.
I20260812 08:03:34.518874 25239 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 19% done.
I20260812 08:03:34.550745 25238 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 19% done.
I20260812 08:03:34.611713 25227 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 19% done.
I20260812 08:03:34.715418 25235 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 19% done.
I20260812 08:03:34.725638 25234 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 19% done.
I20260812 08:03:34.747083 25230 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 19% done.
I20260812 08:03:34.850667 25232 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 29% done.
I20260812 08:03:34.971755 25225 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 29% done.
I20260812 08:03:34.976100 25254 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=15, attempt_no=0}) took 1010 ms. Trace:
I20260812 08:03:34.978767 25254 rpcz_store.cc:276] 0812 08:03:33.966127 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:33.966193 (+    66us) service_pool.cc:224] Handling call
0812 08:03:34.976079 (+1009886us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:34.759335 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:34.759504 (+   169us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:34.759509 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:34.759510 (+     1us) tablet.cc:662] Decoding operations
0812 08:03:34.818141 (+ 58631us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:34.818155 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:34.818162 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:34.848853 (+ 30691us) tablet.cc:701] Row locks acquired
0812 08:03:34.848856 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:34.848977 (+   121us) write_op.cc:270] Start()
0812 08:03:34.849025 (+    48us) write_op.cc:276] Timestamp: P: 1786521814848954 usec, L: 0
0812 08:03:34.849027 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:34.849391 (+   364us) log.cc:844] Serialized 486135 byte log entry
0812 08:03:34.851950 (+  2559us) op_driver.cc:464] REPLICATION: finished
0812 08:03:34.873980 (+ 22030us) write_op.cc:301] APPLY: starting
0812 08:03:34.873998 (+    18us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:34.879982 (+  5984us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:34.880004 (+    22us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:34.967889 (+ 87885us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:34.970175 (+  2286us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:34.970183 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:34.974707 (+  4524us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:34.974809 (+   102us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:34.975705 (+   896us) write_op.cc:454] Released schema lock
0812 08:03:34.975955 (+   250us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":21978,"num_ops":5000,"prepare.queue_time_us":791716,"prepare.run_cpu_time_us":50652,"prepare.run_wall_time_us":92827,"replication_time_us":2895,"spinlock_wait_cycles":23680,"thread_start_us":101,"threads_started":1}]]}
I20260812 08:03:35.031221 25237 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 29% done.
I20260812 08:03:35.090536 25255 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=16, attempt_no=0}) took 1035 ms. Trace:
I20260812 08:03:35.090794 25255 rpcz_store.cc:276] 0812 08:03:34.055519 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:34.055583 (+    64us) service_pool.cc:224] Handling call
0812 08:03:35.090488 (+1034905us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:34.852181 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:34.852339 (+   158us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:34.852344 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:34.852346 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:34.914344 (+ 61998us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:34.914356 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:34.914362 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:34.946862 (+ 32500us) tablet.cc:701] Row locks acquired
0812 08:03:34.946864 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:03:34.946935 (+    71us) write_op.cc:270] Start()
0812 08:03:34.946980 (+    45us) write_op.cc:276] Timestamp: P: 1786521814946920 usec, L: 0
0812 08:03:34.946983 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:34.947312 (+   329us) log.cc:844] Serialized 485624 byte log entry
0812 08:03:34.951257 (+  3945us) op_driver.cc:464] REPLICATION: finished
0812 08:03:34.951339 (+    82us) write_op.cc:301] APPLY: starting
0812 08:03:34.951354 (+    15us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:34.957544 (+  6190us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:34.957566 (+    22us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:35.058142 (+100576us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:35.070970 (+ 12828us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:35.070978 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:35.075491 (+  4513us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:35.089018 (+ 13527us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:35.090088 (+  1070us) write_op.cc:454] Released schema lock
0812 08:03:35.090345 (+   257us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":47,"num_ops":5000,"prepare.queue_time_us":795184,"prepare.run_cpu_time_us":48456,"prepare.run_wall_time_us":95213,"replication_time_us":4250,"spinlock_wait_cycles":5120}]]}
I20260812 08:03:35.183483 25234 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 29% done.
I20260812 08:03:35.195681 25254 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=17, attempt_no=0}) took 1095 ms. Trace:
I20260812 08:03:35.195886 25254 rpcz_store.cc:276] 0812 08:03:34.100549 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:34.100620 (+    71us) service_pool.cc:224] Handling call
0812 08:03:35.195657 (+1095037us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:34.947449 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:34.947608 (+   159us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:34.947613 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:34.947615 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:35.014176 (+ 66561us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:35.014189 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:35.014195 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:35.055226 (+ 41031us) tablet.cc:701] Row locks acquired
0812 08:03:35.055230 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:35.055319 (+    89us) write_op.cc:270] Start()
0812 08:03:35.055369 (+    50us) write_op.cc:276] Timestamp: P: 1786521815055296 usec, L: 0
0812 08:03:35.055371 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:35.055837 (+   466us) log.cc:844] Serialized 486045 byte log entry
0812 08:03:35.058153 (+  2316us) op_driver.cc:464] REPLICATION: finished
0812 08:03:35.058239 (+    86us) write_op.cc:301] APPLY: starting
0812 08:03:35.058255 (+    16us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:35.064103 (+  5848us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:35.064124 (+    21us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:35.187096 (+122972us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:35.189324 (+  2228us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:35.189332 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:35.193998 (+  4666us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:35.194331 (+   333us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:35.195287 (+   956us) write_op.cc:454] Released schema lock
0812 08:03:35.195540 (+   253us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":41,"num_ops":5000,"prepare.queue_time_us":845442,"prepare.run_cpu_time_us":49836,"prepare.run_wall_time_us":110672,"replication_time_us":2751,"spinlock_wait_cycles":11264}]]}
I20260812 08:03:35.240553 25226 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 29% done.
I20260812 08:03:35.270582 25255 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=18, attempt_no=0}) took 1075 ms. Trace:
I20260812 08:03:35.270797 25255 rpcz_store.cc:276] 0812 08:03:34.195593 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:34.195660 (+    67us) service_pool.cc:224] Handling call
0812 08:03:35.270555 (+1074895us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:35.058170 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:35.058333 (+   163us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:35.058338 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:35.058340 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:35.111421 (+ 53081us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:35.111434 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:35.111441 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:35.147483 (+ 36042us) tablet.cc:701] Row locks acquired
0812 08:03:35.147486 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:35.147574 (+    88us) write_op.cc:270] Start()
0812 08:03:35.147619 (+    45us) write_op.cc:276] Timestamp: P: 1786521815147552 usec, L: 0
0812 08:03:35.147621 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:35.147964 (+   343us) log.cc:844] Serialized 486451 byte log entry
0812 08:03:35.159298 (+ 11334us) op_driver.cc:464] REPLICATION: finished
0812 08:03:35.159440 (+   142us) write_op.cc:301] APPLY: starting
0812 08:03:35.159460 (+    20us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:35.165747 (+  6287us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:35.165770 (+    23us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:35.261602 (+ 95832us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:35.263806 (+  2204us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:35.263815 (+     9us) write_op.cc:312] APPLY: finished
0812 08:03:35.268731 (+  4916us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:35.268987 (+   256us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:35.270156 (+  1169us) write_op.cc:454] Released schema lock
0812 08:03:35.270412 (+   256us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":56,"num_ops":5000,"prepare.queue_time_us":861042,"prepare.run_cpu_time_us":48836,"prepare.run_wall_time_us":101348,"replication_time_us":11620,"spinlock_wait_cycles":2688}]]}
I20260812 08:03:35.395411 25238 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 29% done.
I20260812 08:03:35.396638 25227 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 29% done.
I20260812 08:03:35.432170 25239 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 29% done.
I20260812 08:03:35.466622 25255 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=19, attempt_no=0}) took 1265 ms. Trace:
I20260812 08:03:35.466840 25255 rpcz_store.cc:276] 0812 08:03:34.200974 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:34.201020 (+    46us) service_pool.cc:224] Handling call
0812 08:03:35.466598 (+1265578us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:35.159545 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:35.159722 (+   177us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:35.159727 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:35.159729 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:35.230267 (+ 70538us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:35.230281 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:35.230287 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:35.273419 (+ 43132us) tablet.cc:701] Row locks acquired
0812 08:03:35.273422 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:35.273492 (+    70us) write_op.cc:270] Start()
0812 08:03:35.273534 (+    42us) write_op.cc:276] Timestamp: P: 1786521815273477 usec, L: 0
0812 08:03:35.273536 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:35.273890 (+   354us) log.cc:844] Serialized 485914 byte log entry
0812 08:03:35.289050 (+ 15160us) op_driver.cc:464] REPLICATION: finished
0812 08:03:35.289158 (+   108us) write_op.cc:301] APPLY: starting
0812 08:03:35.289177 (+    19us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:35.295013 (+  5836us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:35.295034 (+    21us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:35.458204 (+163170us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:35.460393 (+  2189us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:35.460401 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:35.465112 (+  4711us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:35.465206 (+    94us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:35.466234 (+  1028us) write_op.cc:454] Released schema lock
0812 08:03:35.466479 (+   245us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":65,"num_ops":5000,"prepare.queue_time_us":957187,"prepare.run_cpu_time_us":50784,"prepare.run_wall_time_us":114422,"replication_time_us":15474,"spinlock_wait_cycles":66816}]]}
I20260812 08:03:35.646814 25230 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 29% done.
I20260812 08:03:35.675966 25258 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=21, attempt_no=0}) took 1092 ms. Trace:
I20260812 08:03:35.676177 25258 rpcz_store.cc:276] 0812 08:03:34.583430 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:34.583492 (+    62us) service_pool.cc:224] Handling call
0812 08:03:35.675944 (+1092452us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:35.381555 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:35.381735 (+   180us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:35.381740 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:35.381742 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:35.411992 (+ 30250us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:35.412005 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:35.412012 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:35.432515 (+ 20503us) tablet.cc:701] Row locks acquired
0812 08:03:35.432519 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:35.432605 (+    86us) write_op.cc:270] Start()
0812 08:03:35.432652 (+    47us) write_op.cc:276] Timestamp: P: 1786521815432585 usec, L: 0
0812 08:03:35.432654 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:35.433029 (+   375us) log.cc:844] Serialized 486932 byte log entry
0812 08:03:35.435518 (+  2489us) op_driver.cc:464] REPLICATION: finished
0812 08:03:35.449184 (+ 13666us) write_op.cc:301] APPLY: starting
0812 08:03:35.449202 (+    18us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:35.455331 (+  6129us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:35.455355 (+    24us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:35.578773 (+123418us) spinlock_profiling.cc:243] Waited 9.65 ms on lock 0x55a84fd4918c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:35.667686 (+ 88913us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:35.669596 (+  1910us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:35.669603 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:35.674376 (+  4773us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:35.674454 (+    78us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:35.675587 (+  1133us) write_op.cc:454] Released schema lock
0812 08:03:35.675835 (+   248us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":13615,"mutex_wait_us":11755,"num_ops":5000,"prepare.queue_time_us":797299,"prepare.run_cpu_time_us":48709,"prepare.run_wall_time_us":51553,"replication_time_us":2839,"spinlock_wait_cycles":22253696,"thread_start_us":86,"threads_started":1}]]}
I20260812 08:03:35.688944 25255 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=22, attempt_no=0}) took 1036 ms. Trace:
I20260812 08:03:35.689293 25255 rpcz_store.cc:276] 0812 08:03:34.652436 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:34.652504 (+    68us) service_pool.cc:224] Handling call
0812 08:03:35.688920 (+1036416us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:35.433170 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:35.433340 (+   170us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:35.433345 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:35.433347 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:35.466947 (+ 33600us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:35.466957 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:35.466961 (+     4us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:35.487221 (+ 20260us) tablet.cc:701] Row locks acquired
0812 08:03:35.487224 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:35.487306 (+    82us) write_op.cc:270] Start()
0812 08:03:35.487350 (+    44us) write_op.cc:276] Timestamp: P: 1786521815487286 usec, L: 0
0812 08:03:35.487353 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:35.487699 (+   346us) log.cc:844] Serialized 486261 byte log entry
0812 08:03:35.489856 (+  2157us) op_driver.cc:464] REPLICATION: finished
0812 08:03:35.489928 (+    72us) write_op.cc:301] APPLY: starting
0812 08:03:35.489942 (+    14us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:35.495712 (+  5770us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:35.495728 (+    16us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:35.664132 (+168404us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:35.666340 (+  2208us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:35.666347 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:35.687202 (+ 20855us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:35.687310 (+   108us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:35.688544 (+  1234us) write_op.cc:454] Released schema lock
0812 08:03:35.688780 (+   236us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":41,"mutex_wait_us":13345,"num_ops":5000,"prepare.queue_time_us":779780,"prepare.run_cpu_time_us":49323,"prepare.run_wall_time_us":54617,"replication_time_us":2480,"spinlock_wait_cycles":67456}]]}
I20260812 08:03:35.806248 25235 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 29% done.
I20260812 08:03:35.934026 25237 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 39% done.
I20260812 08:03:35.974987 25232 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 39% done.
I20260812 08:03:36.101776 25226 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 39% done.
I20260812 08:03:36.174211 25225 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 39% done.
I20260812 08:03:36.260317 25254 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=26, attempt_no=0}) took 1109 ms. Trace:
I20260812 08:03:36.260526 25254 rpcz_store.cc:276] 0812 08:03:35.150587 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:35.150644 (+    57us) service_pool.cc:224] Handling call
0812 08:03:36.260289 (+1109645us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:35.845390 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:35.845556 (+   166us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:35.845561 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:35.845563 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:35.923830 (+ 78267us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:35.923845 (+    15us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:35.923852 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:35.985101 (+ 61249us) tablet.cc:701] Row locks acquired
0812 08:03:35.985104 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:35.985186 (+    82us) write_op.cc:270] Start()
0812 08:03:35.985232 (+    46us) write_op.cc:276] Timestamp: P: 1786521815985169 usec, L: 0
0812 08:03:35.985234 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:35.985600 (+   366us) log.cc:844] Serialized 486068 byte log entry
0812 08:03:35.987826 (+  2226us) op_driver.cc:464] REPLICATION: finished
0812 08:03:35.987908 (+    82us) write_op.cc:301] APPLY: starting
0812 08:03:35.987923 (+    15us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:35.993928 (+  6005us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:35.993948 (+    20us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:36.235681 (+241733us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:36.237901 (+  2220us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:36.237909 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:36.258768 (+ 20859us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:36.258886 (+   118us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:36.259884 (+   998us) write_op.cc:454] Released schema lock
0812 08:03:36.260159 (+   275us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":48,"mutex_wait_us":8245,"num_ops":5000,"prepare.queue_time_us":694141,"prepare.run_cpu_time_us":52786,"prepare.run_wall_time_us":140294,"replication_time_us":2567,"spinlock_wait_cycles":54400}]]}
I20260812 08:03:36.307765 25258 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=27, attempt_no=0}) took 1063 ms. Trace:
I20260812 08:03:36.312202 25258 rpcz_store.cc:276] 0812 08:03:35.243943 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:35.244001 (+    58us) service_pool.cc:224] Handling call
0812 08:03:36.307737 (+1063736us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:35.985746 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:35.985906 (+   160us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:35.985911 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:35.985913 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:36.012428 (+ 26515us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:36.012441 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:36.012448 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:36.032576 (+ 20128us) tablet.cc:701] Row locks acquired
0812 08:03:36.032579 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:36.032664 (+    85us) write_op.cc:270] Start()
0812 08:03:36.032711 (+    47us) write_op.cc:276] Timestamp: P: 1786521816032642 usec, L: 0
0812 08:03:36.032714 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:36.033089 (+   375us) log.cc:844] Serialized 486086 byte log entry
0812 08:03:36.035452 (+  2363us) op_driver.cc:464] REPLICATION: finished
0812 08:03:36.035527 (+    75us) write_op.cc:301] APPLY: starting
0812 08:03:36.035542 (+    15us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:36.041388 (+  5846us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:36.041411 (+    23us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:36.178767 (+137356us) spinlock_profiling.cc:243] Waited 8.66 ms on lock 0x55a84fa07898. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facf3c2fbdd 00007facfc1cfe3b 00007facfc0998df 00007facfc09e60a 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d
0812 08:03:36.281897 (+103130us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:36.284019 (+  2122us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:36.284025 (+     6us) write_op.cc:312] APPLY: finished
0812 08:03:36.300783 (+ 16758us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:36.306280 (+  5497us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:36.307356 (+  1076us) write_op.cc:454] Released schema lock
0812 08:03:36.307598 (+   242us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":27,"num_ops":5000,"prepare.queue_time_us":741083,"prepare.run_cpu_time_us":47378,"prepare.run_wall_time_us":47431,"replication_time_us":2710,"spinlock_wait_cycles":19960832}]]}
I20260812 08:03:36.319747 25227 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 39% done.
W20260812 08:03:36.421332 25202 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:45861, user_credentials={real_user=slave}} blocked reactor thread for 59579.9us
I20260812 08:03:36.485415 25264 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=28, attempt_no=0}) took 1129 ms. Trace:
I20260812 08:03:36.485632 25264 rpcz_store.cc:276] 0812 08:03:35.355782 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:35.355847 (+    65us) service_pool.cc:224] Handling call
0812 08:03:36.485392 (+1129545us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:36.033223 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:36.033385 (+   162us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:36.033390 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:36.033391 (+     1us) tablet.cc:662] Decoding operations
0812 08:03:36.060312 (+ 26921us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:36.060327 (+    15us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:36.060334 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:36.080727 (+ 20393us) tablet.cc:701] Row locks acquired
0812 08:03:36.080731 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:36.080812 (+    81us) write_op.cc:270] Start()
0812 08:03:36.080860 (+    48us) write_op.cc:276] Timestamp: P: 1786521816080796 usec, L: 0
0812 08:03:36.080863 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:36.081243 (+   380us) log.cc:844] Serialized 486850 byte log entry
0812 08:03:36.083450 (+  2207us) op_driver.cc:464] REPLICATION: finished
0812 08:03:36.083538 (+    88us) write_op.cc:301] APPLY: starting
0812 08:03:36.083553 (+    15us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:36.089522 (+  5969us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:36.089542 (+    20us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:36.427662 (+338120us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:36.462603 (+ 34941us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:36.462614 (+    11us) write_op.cc:312] APPLY: finished
0812 08:03:36.467293 (+  4679us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:36.476995 (+  9702us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:36.484984 (+  7989us) write_op.cc:454] Released schema lock
0812 08:03:36.485257 (+   273us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":50,"mutex_wait_us":8217,"num_ops":5000,"prepare.queue_time_us":676375,"prepare.run_cpu_time_us":48003,"prepare.run_wall_time_us":48101,"replication_time_us":2558,"spinlock_wait_cycles":77184}]]}
I20260812 08:03:36.486824 25234 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 39% done.
I20260812 08:03:36.536373 25238 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 39% done.
I20260812 08:03:36.643215 25254 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=30, attempt_no=0}) took 1085 ms. Trace:
I20260812 08:03:36.652936 25254 rpcz_store.cc:276] 0812 08:03:35.557830 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:35.557886 (+    56us) service_pool.cc:224] Handling call
0812 08:03:36.643191 (+1085305us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:36.132977 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:36.133139 (+   162us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:36.133144 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:36.133146 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:36.162264 (+ 29118us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:36.162278 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:36.162284 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:36.183259 (+ 20975us) tablet.cc:701] Row locks acquired
0812 08:03:36.183262 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:36.183341 (+    79us) write_op.cc:270] Start()
0812 08:03:36.183384 (+    43us) write_op.cc:276] Timestamp: P: 1786521816183323 usec, L: 0
0812 08:03:36.183387 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:36.183716 (+   329us) log.cc:844] Serialized 486163 byte log entry
0812 08:03:36.186021 (+  2305us) op_driver.cc:464] REPLICATION: finished
0812 08:03:36.274476 (+ 88455us) write_op.cc:301] APPLY: starting
0812 08:03:36.274498 (+    22us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:36.296374 (+ 21876us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:36.296395 (+    21us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:36.618930 (+322535us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:36.637152 (+ 18222us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:36.637159 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:36.641598 (+  4439us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:36.641660 (+    62us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:36.642843 (+  1183us) write_op.cc:454] Released schema lock
0812 08:03:36.643071 (+   228us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":88392,"num_ops":5000,"prepare.queue_time_us":574433,"prepare.run_cpu_time_us":50704,"prepare.run_wall_time_us":53130,"replication_time_us":2605,"spinlock_wait_cycles":38912}]]}
I20260812 08:03:36.647465 25235 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 39% done.
W20260812 08:03:36.715059 25202 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:45861, user_credentials={real_user=slave}} blocked reactor thread for 44329.8us
I20260812 08:03:37.020767 25239 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 39% done.
I20260812 08:03:37.154115 25232 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 49% done.
I20260812 08:03:37.155421 25225 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 49% done.
I20260812 08:03:37.203457 25237 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 49% done.
I20260812 08:03:37.300449 25230 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 39% done.
I20260812 08:03:37.407763 25235 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 49% done.
I20260812 08:03:37.498742 25234 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 49% done.
I20260812 08:03:37.519878 25226 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 49% done.
I20260812 08:03:37.564549 25227 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 49% done.
I20260812 08:03:37.622836 25238 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 49% done.
I20260812 08:03:37.700203 25239 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 49% done.
I20260812 08:03:37.802650 25230 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 49% done.
I20260812 08:03:37.911443 25258 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=43, attempt_no=0}) took 1036 ms. Trace:
I20260812 08:03:37.924943 25258 rpcz_store.cc:276] 0812 08:03:36.874808 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:36.874868 (+    60us) service_pool.cc:224] Handling call
0812 08:03:37.911417 (+1036549us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:37.508157 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:37.508324 (+   167us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:37.508329 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:37.508331 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:37.556373 (+ 48042us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:37.556388 (+    15us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:37.556394 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:37.622311 (+ 65917us) tablet.cc:701] Row locks acquired
0812 08:03:37.622315 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:37.622403 (+    88us) write_op.cc:270] Start()
0812 08:03:37.622450 (+    47us) write_op.cc:276] Timestamp: P: 1786521817622380 usec, L: 0
0812 08:03:37.622453 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:37.622812 (+   359us) log.cc:844] Serialized 486238 byte log entry
0812 08:03:37.624999 (+  2187us) op_driver.cc:464] REPLICATION: finished
0812 08:03:37.625101 (+   102us) write_op.cc:301] APPLY: starting
0812 08:03:37.625117 (+    16us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:37.630894 (+  5777us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:37.630918 (+    24us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:37.887157 (+256239us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:37.905288 (+ 18131us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:37.905297 (+     9us) write_op.cc:312] APPLY: finished
0812 08:03:37.909888 (+  4591us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:37.909999 (+   111us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:37.911035 (+  1036us) write_op.cc:454] Released schema lock
0812 08:03:37.911289 (+   254us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":32,"mutex_wait_us":18339,"num_ops":5000,"prepare.queue_time_us":632660,"prepare.run_cpu_time_us":48177,"prepare.run_wall_time_us":114755,"replication_time_us":2522,"spinlock_wait_cycles":26496}]]}
I20260812 08:03:37.959107 25225 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 59% done.
I20260812 08:03:37.987205 25254 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=44, attempt_no=0}) took 1017 ms. Trace:
I20260812 08:03:38.004940 25254 rpcz_store.cc:276] 0812 08:03:36.969414 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:36.969507 (+    93us) service_pool.cc:224] Handling call
0812 08:03:37.987177 (+1017670us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:37.622945 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:37.623109 (+   164us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:37.623115 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:03:37.623116 (+     1us) tablet.cc:662] Decoding operations
0812 08:03:37.650751 (+ 27635us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:37.650762 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:37.650768 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:37.670784 (+ 20016us) tablet.cc:701] Row locks acquired
0812 08:03:37.670787 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:37.670865 (+    78us) write_op.cc:270] Start()
0812 08:03:37.670912 (+    47us) write_op.cc:276] Timestamp: P: 1786521817670851 usec, L: 0
0812 08:03:37.670915 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:37.671278 (+   363us) log.cc:844] Serialized 486174 byte log entry
0812 08:03:37.673453 (+  2175us) op_driver.cc:464] REPLICATION: finished
0812 08:03:37.673535 (+    82us) write_op.cc:301] APPLY: starting
0812 08:03:37.673552 (+    17us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:37.679427 (+  5875us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:37.679446 (+    19us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:37.963027 (+283581us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:37.965226 (+  2199us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:37.965234 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:37.985735 (+ 20501us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:37.985844 (+   109us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:37.986803 (+   959us) write_op.cc:454] Released schema lock
0812 08:03:37.987054 (+   251us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":38,"mutex_wait_us":4306,"num_ops":5000,"prepare.queue_time_us":652768,"prepare.run_cpu_time_us":48310,"prepare.run_wall_time_us":48414,"replication_time_us":2514,"spinlock_wait_cycles":52224}]]}
I20260812 08:03:38.009765 25237 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 59% done.
I20260812 08:03:38.129495 25232 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 59% done.
I20260812 08:03:38.246387 25258 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=47, attempt_no=0}) took 1047 ms. Trace:
I20260812 08:03:38.260948 25258 rpcz_store.cc:276] 0812 08:03:37.198542 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:37.198605 (+    63us) service_pool.cc:224] Handling call
0812 08:03:38.246368 (+1047763us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:37.785290 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:37.785453 (+   163us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:37.785458 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:37.785460 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:37.812585 (+ 27125us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:37.812600 (+    15us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:37.812607 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:37.832854 (+ 20247us) tablet.cc:701] Row locks acquired
0812 08:03:37.832858 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:37.832973 (+   115us) write_op.cc:270] Start()
0812 08:03:37.833020 (+    47us) write_op.cc:276] Timestamp: P: 1786521817832950 usec, L: 0
0812 08:03:37.833022 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:37.833383 (+   361us) log.cc:844] Serialized 486250 byte log entry
0812 08:03:37.835584 (+  2201us) op_driver.cc:464] REPLICATION: finished
0812 08:03:37.926013 (+ 90429us) write_op.cc:301] APPLY: starting
0812 08:03:37.926030 (+    17us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:37.932000 (+  5970us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:37.932029 (+    29us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:38.182369 (+250340us) spinlock_profiling.cc:243] Waited 3.71 ms on lock 0x55a84fa07898. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facf3c2fbdd 00007facfc1cfe3b 00007facfc0998df 00007facfc09e60a 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d
0812 08:03:38.208675 (+ 26306us) spinlock_profiling.cc:243] Waited 7.58 ms on lock 0x55a84fd4918c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:38.214203 (+  5528us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:38.216328 (+  2125us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:38.216336 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:38.244964 (+ 28628us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:38.245061 (+    97us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:38.246016 (+   955us) write_op.cc:454] Released schema lock
0812 08:03:38.246266 (+   250us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":90372,"mutex_wait_us":4174,"num_ops":5000,"prepare.queue_time_us":585906,"prepare.run_cpu_time_us":48045,"prepare.run_wall_time_us":48177,"replication_time_us":2535,"spinlock_wait_cycles":26016512}]]}
I20260812 08:03:38.438573 25234 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 59% done.
I20260812 08:03:38.471658 25227 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 59% done.
I20260812 08:03:38.637508 25238 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 59% done.
I20260812 08:03:38.642052 25235 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 59% done.
I20260812 08:03:38.692771 25239 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 59% done.
I20260812 08:03:38.716768 25226 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 59% done.
W20260812 08:03:38.718242 25202 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:45861, user_credentials={real_user=slave}} blocked reactor thread for 45816.4us
I20260812 08:03:38.964475 25225 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 69% done.
I20260812 08:03:38.989166 25230 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 59% done.
I20260812 08:03:39.036258 25237 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 69% done.
I20260812 08:03:39.080614 25226 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 69% done.
I20260812 08:03:39.194450 25268 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=56, attempt_no=0}) took 1052 ms. Trace:
I20260812 08:03:39.197448 25268 rpcz_store.cc:276] 0812 08:03:38.142102 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:38.142141 (+    39us) service_pool.cc:224] Handling call
0812 08:03:39.194423 (+1052282us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:38.812678 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:38.812839 (+   161us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:38.812844 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:38.812846 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:38.840347 (+ 27501us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:38.840361 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:38.840368 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:38.861362 (+ 20994us) tablet.cc:701] Row locks acquired
0812 08:03:38.861365 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:38.861453 (+    88us) write_op.cc:270] Start()
0812 08:03:38.861504 (+    51us) write_op.cc:276] Timestamp: P: 1786521818861433 usec, L: 0
0812 08:03:38.861507 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:38.861882 (+   375us) log.cc:844] Serialized 486450 byte log entry
0812 08:03:38.864239 (+  2357us) op_driver.cc:464] REPLICATION: finished
0812 08:03:38.864331 (+    92us) write_op.cc:301] APPLY: starting
0812 08:03:38.864348 (+    17us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:38.870543 (+  6195us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:38.870565 (+    22us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:39.169927 (+299362us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:39.172023 (+  2096us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:39.172030 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:39.176604 (+  4574us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:39.176717 (+   113us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:39.193988 (+ 17271us) write_op.cc:454] Released schema lock
0812 08:03:39.194269 (+   281us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":53,"num_ops":5000,"prepare.queue_time_us":669988,"prepare.run_cpu_time_us":48890,"prepare.run_wall_time_us":49295,"replication_time_us":2700,"spinlock_wait_cycles":25728}]]}
I20260812 08:03:39.199981 25258 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=55, attempt_no=0}) took 1058 ms. Trace:
I20260812 08:03:39.200161 25258 rpcz_store.cc:276] 0812 08:03:38.141141 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:38.141205 (+    64us) service_pool.cc:224] Handling call
0812 08:03:39.199964 (+1058759us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:38.732234 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:38.732394 (+   160us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:38.732399 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:38.732401 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:38.791209 (+ 58808us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:38.791224 (+    15us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:38.791231 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:38.811968 (+ 20737us) tablet.cc:701] Row locks acquired
0812 08:03:38.811971 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:38.812060 (+    89us) write_op.cc:270] Start()
0812 08:03:38.812106 (+    46us) write_op.cc:276] Timestamp: P: 1786521818812037 usec, L: 0
0812 08:03:38.812109 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:38.812518 (+   409us) log.cc:844] Serialized 486020 byte log entry
0812 08:03:38.814764 (+  2246us) op_driver.cc:464] REPLICATION: finished
0812 08:03:38.814857 (+    93us) write_op.cc:301] APPLY: starting
0812 08:03:38.814872 (+    15us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:38.820717 (+  5845us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:38.820740 (+    23us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:39.180256 (+359516us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:39.182401 (+  2145us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:39.182409 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:39.196861 (+ 14452us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:39.196985 (+   124us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:39.199607 (+  2622us) write_op.cc:454] Released schema lock
0812 08:03:39.199862 (+   255us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":54,"num_ops":5000,"prepare.queue_time_us":590324,"prepare.run_cpu_time_us":48348,"prepare.run_wall_time_us":80362,"replication_time_us":2627,"spinlock_wait_cycles":27776}]]}
I20260812 08:03:39.233421 25234 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 69% done.
I20260812 08:03:39.235997 25232 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 69% done.
W20260812 08:03:39.245364 25202 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:45861, user_credentials={real_user=slave}} blocked reactor thread for 47085.7us
W20260812 08:03:39.339449 25202 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:45861, user_credentials={real_user=slave}} blocked reactor thread for 53583.3us
W20260812 08:03:39.451372 25202 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:45861, user_credentials={real_user=slave}} blocked reactor thread for 56731us
I20260812 08:03:39.544605 25235 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 69% done.
I20260812 08:03:39.904655 25227 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 69% done.
I20260812 08:03:39.981528 25238 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 69% done.
I20260812 08:03:40.037750 25226 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 79% done.
I20260812 08:03:40.077123 25272 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=64, attempt_no=0}) took 1030 ms. Trace:
I20260812 08:03:40.077374 25272 rpcz_store.cc:276] 0812 08:03:39.046201 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:39.046266 (+    65us) service_pool.cc:224] Handling call
0812 08:03:40.077098 (+1030832us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:39.435076 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:39.435229 (+   153us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:39.435235 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:03:39.435236 (+     1us) tablet.cc:662] Decoding operations
0812 08:03:39.500139 (+ 64903us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:39.500153 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:39.500157 (+     4us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:39.577978 (+ 77821us) tablet.cc:701] Row locks acquired
0812 08:03:39.577981 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:39.578026 (+    45us) write_op.cc:270] Start()
0812 08:03:39.578062 (+    36us) write_op.cc:276] Timestamp: P: 1786521819578015 usec, L: 0
0812 08:03:39.578065 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:39.578355 (+   290us) log.cc:844] Serialized 486150 byte log entry
0812 08:03:39.593228 (+ 14873us) op_driver.cc:464] REPLICATION: finished
0812 08:03:39.593503 (+   275us) write_op.cc:301] APPLY: starting
0812 08:03:39.593518 (+    15us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:39.612271 (+ 18753us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:39.612290 (+    19us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:40.029075 (+416785us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:40.031303 (+  2228us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:40.031310 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:40.051957 (+ 20647us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:40.052075 (+   118us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:40.069206 (+ 17131us) write_op.cc:454] Released schema lock
0812 08:03:40.076950 (+  7744us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":238,"num_ops":5000,"prepare.queue_time_us":388098,"prepare.run_cpu_time_us":49056,"prepare.run_wall_time_us":163117,"replication_time_us":15135,"spinlock_wait_cycles":21632}]]}
I20260812 08:03:40.082438 25239 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 69% done.
W20260812 08:03:40.170200 25202 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:45861, user_credentials={real_user=slave}} blocked reactor thread for 61027.8us
I20260812 08:03:40.176712 25225 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 79% done.
W20260812 08:03:40.225495 25202 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:45861, user_credentials={real_user=slave}} blocked reactor thread for 48386.8us
I20260812 08:03:40.256237 25232 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 79% done.
W20260812 08:03:40.272989 25202 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:45861, user_credentials={real_user=slave}} blocked reactor thread for 44048.2us
I20260812 08:03:40.387137 25230 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 69% done.
I20260812 08:03:40.406111 25237 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 79% done.
I20260812 08:03:40.677744 25264 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=71, attempt_no=0}) took 1101 ms. Trace:
I20260812 08:03:40.684942 25264 rpcz_store.cc:276] 0812 08:03:39.576753 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:39.576816 (+    63us) service_pool.cc:224] Handling call
0812 08:03:40.677722 (+1100906us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:40.147707 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:40.147867 (+   160us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:40.147872 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:40.147874 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:40.210342 (+ 62468us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:40.210356 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:40.210363 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:40.279314 (+ 68951us) tablet.cc:701] Row locks acquired
0812 08:03:40.279317 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:40.279389 (+    72us) write_op.cc:270] Start()
0812 08:03:40.279432 (+    43us) write_op.cc:276] Timestamp: P: 1786521820279376 usec, L: 0
0812 08:03:40.279435 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:40.279789 (+   354us) log.cc:844] Serialized 485524 byte log entry
0812 08:03:40.282259 (+  2470us) op_driver.cc:464] REPLICATION: finished
0812 08:03:40.282338 (+    79us) write_op.cc:301] APPLY: starting
0812 08:03:40.282356 (+    18us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:40.288264 (+  5908us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:40.288286 (+    22us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:40.646514 (+358228us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:40.648728 (+  2214us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:40.648736 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:40.675997 (+ 27261us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:40.676177 (+   180us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:40.677341 (+  1164us) write_op.cc:454] Released schema lock
0812 08:03:40.677612 (+   271us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":29,"num_ops":5000,"prepare.queue_time_us":570229,"prepare.run_cpu_time_us":48376,"prepare.run_wall_time_us":132163,"replication_time_us":2798,"spinlock_wait_cycles":54912}]]}
I20260812 08:03:40.702715 25274 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=73, attempt_no=0}) took 1043 ms. Trace:
I20260812 08:03:40.702906 25274 rpcz_store.cc:276] 0812 08:03:39.658804 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:39.658869 (+    65us) service_pool.cc:224] Handling call
0812 08:03:40.702691 (+1043822us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:40.279915 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:40.280076 (+   161us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:40.280081 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:40.280082 (+     1us) tablet.cc:662] Decoding operations
0812 08:03:40.327276 (+ 47194us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:40.327288 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:40.327294 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:40.347427 (+ 20133us) tablet.cc:701] Row locks acquired
0812 08:03:40.347430 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:40.347517 (+    87us) write_op.cc:270] Start()
0812 08:03:40.347565 (+    48us) write_op.cc:276] Timestamp: P: 1786521820347496 usec, L: 0
0812 08:03:40.347568 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:40.347951 (+   383us) log.cc:844] Serialized 485460 byte log entry
0812 08:03:40.350450 (+  2499us) op_driver.cc:464] REPLICATION: finished
0812 08:03:40.350529 (+    79us) write_op.cc:301] APPLY: starting
0812 08:03:40.350546 (+    17us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:40.356607 (+  6061us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:40.356632 (+    25us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:40.465088 (+108456us) spinlock_profiling.cc:243] Waited 3.59 ms on lock 0x55a84fd4918c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:40.659394 (+194306us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:40.678564 (+ 19170us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:40.678572 (+     8us) write_op.cc:312] APPLY: finished
0812 08:03:40.700949 (+ 22377us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:40.701251 (+   302us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:40.702297 (+  1046us) write_op.cc:454] Released schema lock
0812 08:03:40.702543 (+   246us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":43,"num_ops":5000,"prepare.queue_time_us":620322,"prepare.run_cpu_time_us":49082,"prepare.run_wall_time_us":68127,"replication_time_us":2856,"spinlock_wait_cycles":8324608}]]}
I20260812 08:03:40.725152 25235 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 79% done.
I20260812 08:03:40.960248 25238 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 79% done.
I20260812 08:03:41.050108 25272 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=74, attempt_no=0}) took 1074 ms. Trace:
I20260812 08:03:41.054678 25272 rpcz_store.cc:276] 0812 08:03:39.976002 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:39.976064 (+    62us) service_pool.cc:224] Handling call
0812 08:03:41.050082 (+1074018us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:40.405963 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:40.406114 (+   151us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:40.406120 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:03:40.406121 (+     1us) tablet.cc:662] Decoding operations
0812 08:03:40.550420 (+144299us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:40.550433 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:40.550439 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:40.572972 (+ 22533us) tablet.cc:701] Row locks acquired
0812 08:03:40.572975 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:40.573059 (+    84us) write_op.cc:270] Start()
0812 08:03:40.573104 (+    45us) write_op.cc:276] Timestamp: P: 1786521820573038 usec, L: 0
0812 08:03:40.573107 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:40.573456 (+   349us) log.cc:844] Serialized 486082 byte log entry
0812 08:03:40.575611 (+  2155us) op_driver.cc:464] REPLICATION: finished
0812 08:03:40.575716 (+   105us) write_op.cc:301] APPLY: starting
0812 08:03:40.575731 (+    15us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:40.581702 (+  5971us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:40.581722 (+    20us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:41.028825 (+447103us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:41.031102 (+  2277us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:41.031109 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:41.036078 (+  4969us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:41.036144 (+    66us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:41.049695 (+ 13551us) write_op.cc:454] Released schema lock
0812 08:03:41.049951 (+   256us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":31,"mutex_wait_us":16733,"num_ops":5000,"prepare.queue_time_us":429254,"prepare.run_cpu_time_us":50227,"prepare.run_wall_time_us":167574,"replication_time_us":2480,"spinlock_wait_cycles":25856}]]}
I20260812 08:03:41.070487 25237 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 89% done.
I20260812 08:03:41.094915 25234 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 79% done.
W20260812 08:03:41.113210 25202 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:45861, user_credentials={real_user=slave}} blocked reactor thread for 48909.9us
I20260812 08:03:41.120563 25227 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 79% done.
I20260812 08:03:41.174412 25239 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 79% done.
I20260812 08:03:41.180699 25225 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 89% done.
I20260812 08:03:41.219980 25230 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 79% done.
W20260812 08:03:41.313596 25202 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:45861, user_credentials={real_user=slave}} blocked reactor thread for 60013.5us
I20260812 08:03:41.383316 25235 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 89% done.
I20260812 08:03:41.394789 25258 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=79, attempt_no=0}) took 1008 ms. Trace:
I20260812 08:03:41.397147 25258 rpcz_store.cc:276] 0812 08:03:40.385949 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:40.386013 (+    64us) service_pool.cc:224] Handling call
0812 08:03:41.394764 (+1008751us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:40.770386 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:40.770545 (+   159us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:40.770550 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:40.770552 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:40.800011 (+ 29459us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:40.800025 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:40.800032 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:40.820962 (+ 20930us) tablet.cc:701] Row locks acquired
0812 08:03:40.820966 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:40.821054 (+    88us) write_op.cc:270] Start()
0812 08:03:40.821101 (+    47us) write_op.cc:276] Timestamp: P: 1786521820821031 usec, L: 0
0812 08:03:40.821104 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:40.821580 (+   476us) log.cc:844] Serialized 486443 byte log entry
0812 08:03:40.823794 (+  2214us) op_driver.cc:464] REPLICATION: finished
0812 08:03:41.065518 (+241724us) write_op.cc:301] APPLY: starting
0812 08:03:41.065536 (+    18us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:41.072837 (+  7301us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:41.072874 (+    37us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:41.370510 (+297636us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:41.372618 (+  2108us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:41.372631 (+    13us) write_op.cc:312] APPLY: finished
0812 08:03:41.393229 (+ 20598us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:41.393338 (+   109us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:41.394363 (+  1025us) write_op.cc:454] Released schema lock
0812 08:03:41.394619 (+   256us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":241669,"mutex_wait_us":981,"num_ops":5000,"prepare.queue_time_us":383645,"prepare.run_cpu_time_us":48962,"prepare.run_wall_time_us":51280,"replication_time_us":2663,"spinlock_wait_cycles":49536}]]}
I20260812 08:03:41.397048 25272 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=78, attempt_no=0}) took 1068 ms. Trace:
I20260812 08:03:41.405046 25272 rpcz_store.cc:276] 0812 08:03:40.328522 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:40.328571 (+    49us) service_pool.cc:224] Handling call
0812 08:03:41.397029 (+1068458us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:40.721052 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:40.721205 (+   153us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:40.721210 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:40.721211 (+     1us) tablet.cc:662] Decoding operations
0812 08:03:40.749011 (+ 27800us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:40.749023 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:40.749030 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:40.769786 (+ 20756us) tablet.cc:701] Row locks acquired
0812 08:03:40.769789 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:40.769853 (+    64us) write_op.cc:270] Start()
0812 08:03:40.769893 (+    40us) write_op.cc:276] Timestamp: P: 1786521820769839 usec, L: 0
0812 08:03:40.769896 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:03:40.770270 (+   374us) log.cc:844] Serialized 485910 byte log entry
0812 08:03:40.772588 (+  2318us) op_driver.cc:464] REPLICATION: finished
0812 08:03:41.055866 (+283278us) write_op.cc:301] APPLY: starting
0812 08:03:41.055884 (+    18us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:41.061920 (+  6036us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:41.061946 (+    26us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:41.357018 (+295072us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x55a84fa07898. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facf3c2fbdd 00007facfc1cfe3b 00007facfc0998df 00007facfc09e60a 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d
0812 08:03:41.378607 (+ 21589us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:41.380785 (+  2178us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:41.380792 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:41.395540 (+ 14748us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:41.395642 (+   102us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:41.396644 (+  1002us) write_op.cc:454] Released schema lock
0812 08:03:41.396924 (+   280us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":283231,"mutex_wait_us":14611,"num_ops":5000,"prepare.queue_time_us":391751,"prepare.run_cpu_time_us":48598,"prepare.run_wall_time_us":49301,"replication_time_us":2657,"spinlock_wait_cycles":36605824}]]}
I20260812 08:03:41.484285 25232 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 89% done.
W20260812 08:03:41.502132 25202 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:45861, user_credentials={real_user=slave}} blocked reactor thread for 84072us
I20260812 08:03:41.532441 25226 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 89% done.
I20260812 08:03:41.586912 25264 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=80, attempt_no=0}) took 1132 ms. Trace:
I20260812 08:03:41.587113 25264 rpcz_store.cc:276] 0812 08:03:40.454413 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:40.454478 (+    65us) service_pool.cc:224] Handling call
0812 08:03:41.586891 (+1132413us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:40.821714 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:40.821870 (+   156us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:40.821875 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:40.821876 (+     1us) tablet.cc:662] Decoding operations
0812 08:03:40.850671 (+ 28795us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:40.850684 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:40.850690 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:40.870948 (+ 20258us) tablet.cc:701] Row locks acquired
0812 08:03:40.870951 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:03:40.871022 (+    71us) write_op.cc:270] Start()
0812 08:03:40.871064 (+    42us) write_op.cc:276] Timestamp: P: 1786521820871010 usec, L: 0
0812 08:03:40.871066 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:40.871397 (+   331us) log.cc:844] Serialized 486772 byte log entry
0812 08:03:40.873564 (+  2167us) op_driver.cc:464] REPLICATION: finished
0812 08:03:41.224736 (+351172us) write_op.cc:301] APPLY: starting
0812 08:03:41.224752 (+    16us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:41.230735 (+  5983us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:41.230757 (+    22us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:41.357002 (+126245us) spinlock_profiling.cc:243] Waited 8.51 ms on lock 0x55a84fa07898. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facf3c2fbdd 00007facfc1cfe3b 00007facfc0998df 00007facfc09e60a 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d
0812 08:03:41.439305 (+ 82303us) spinlock_profiling.cc:243] Waited 2.22 ms on lock 0x55a84fd4918c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:41.562606 (+123301us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:41.564699 (+  2093us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:41.564706 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:41.585269 (+ 20563us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:41.585564 (+   295us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:41.586524 (+   960us) write_op.cc:454] Released schema lock
0812 08:03:41.586779 (+   255us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":351127,"mutex_wait_us":9633,"num_ops":5000,"prepare.queue_time_us":366538,"prepare.run_cpu_time_us":49641,"prepare.run_wall_time_us":49764,"replication_time_us":2473,"spinlock_wait_cycles":24733056}]]}
I20260812 08:03:41.635807 25238 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 89% done.
W20260812 08:03:41.693203 25202 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:45861, user_credentials={real_user=slave}} blocked reactor thread for 53614.6us
I20260812 08:03:41.874305 25258 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=82, attempt_no=0}) took 1089 ms. Trace:
I20260812 08:03:41.874433 25258 rpcz_store.cc:276] 0812 08:03:40.784440 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:40.784514 (+    74us) service_pool.cc:224] Handling call
0812 08:03:41.874282 (+1089768us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:40.933593 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:40.933756 (+   163us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:40.933761 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:40.933763 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:40.962694 (+ 28931us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:40.962707 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:40.962714 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:40.984297 (+ 21583us) tablet.cc:701] Row locks acquired
0812 08:03:40.984301 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:40.984382 (+    81us) write_op.cc:270] Start()
0812 08:03:40.984430 (+    48us) write_op.cc:276] Timestamp: P: 1786521820984363 usec, L: 0
0812 08:03:40.984432 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:40.985235 (+   803us) log.cc:844] Serialized 485934 byte log entry
0812 08:03:40.987372 (+  2137us) op_driver.cc:464] REPLICATION: finished
0812 08:03:41.398462 (+411090us) write_op.cc:301] APPLY: starting
0812 08:03:41.398479 (+    17us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:41.404268 (+  5789us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:41.404291 (+    23us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:41.513057 (+108766us) spinlock_profiling.cc:243] Waited 5.38 ms on lock 0x55a84fd4918c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:03:41.813614 (+300557us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:41.826254 (+ 12640us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:41.826263 (+     9us) write_op.cc:312] APPLY: finished
0812 08:03:41.831008 (+  4745us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:41.844999 (+ 13991us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:41.873615 (+ 28616us) spinlock_profiling.cc:243] Waited 27.6 ms on lock 0x55a84fa75290. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfc1c860b 00007facfc1c9ecb 00007facfc1c8c17 00007facfc16e26f 00007facfc16d773 00007facfc16be49 00007facfc15916c 00007facfc158b95 00007facfc1579ca 00007facfc15a95b
0812 08:03:41.873891 (+   276us) write_op.cc:454] Released schema lock
0812 08:03:41.874147 (+   256us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":411040,"mutex_wait_us":3774,"num_ops":5000,"prepare.queue_time_us":148297,"prepare.run_cpu_time_us":51223,"prepare.run_wall_time_us":51718,"replication_time_us":2916,"spinlock_wait_cycles":75925376}]]}
W20260812 08:03:41.893622 25202 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:45861, user_credentials={real_user=slave}} blocked reactor thread for 46762.9us
I20260812 08:03:41.903275 25234 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 89% done.
W20260812 08:03:41.968670 25202 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:45861, user_credentials={real_user=slave}} blocked reactor thread for 74716.1us
W20260812 08:03:42.073108 25202 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:45861, user_credentials={real_user=slave}} blocked reactor thread for 102304us
I20260812 08:03:42.132500 25225 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 99% done.
I20260812 08:03:42.132630 25225 full_stack-insert-scan-test.cc:280] Insertion thread 1 of  10 is 100% done.
I20260812 08:03:42.137511 25232 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 99% done.
I20260812 08:03:42.152946 25232 full_stack-insert-scan-test.cc:280] Insertion thread 5 of  10 is 100% done.
W20260812 08:03:42.166426 25202 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:45861, user_credentials={real_user=slave}} blocked reactor thread for 89972.5us
I20260812 08:03:42.237771 25227 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 89% done.
I20260812 08:03:42.279697 25274 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:37602 (ReqId={client: dece545bc454455c9beda40380c3edc6, seq_no=85, attempt_no=0}) took 1111 ms. Trace:
I20260812 08:03:42.288947 25274 rpcz_store.cc:276] 0812 08:03:41.167994 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:03:41.168063 (+    69us) service_pool.cc:224] Handling call
0812 08:03:42.279671 (+1111608us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:03:41.308531 (+     0us) write_op.cc:183] PREPARE: starting on tablet 304b715f38bc4a81940b24b40519223d
0812 08:03:41.308695 (+   164us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:03:41.308700 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:03:41.308702 (+     2us) tablet.cc:662] Decoding operations
0812 08:03:41.420079 (+111377us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:03:41.420093 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:03:41.420100 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:03:41.451921 (+ 31821us) tablet.cc:701] Row locks acquired
0812 08:03:41.451925 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:03:41.452005 (+    80us) write_op.cc:270] Start()
0812 08:03:41.452048 (+    43us) write_op.cc:276] Timestamp: P: 1786521821451986 usec, L: 0
0812 08:03:41.452050 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:03:41.452378 (+   328us) log.cc:844] Serialized 486064 byte log entry
0812 08:03:41.454650 (+  2272us) op_driver.cc:464] REPLICATION: finished
0812 08:03:41.645540 (+190890us) write_op.cc:301] APPLY: starting
0812 08:03:41.645560 (+    20us) tablet.cc:1367] starting BulkCheckPresence
0812 08:03:41.651581 (+  6021us) tablet.cc:1370] finished BulkCheckPresence
0812 08:03:41.651607 (+    26us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:03:42.216796 (+565189us) spinlock_profiling.cc:243] Waited 7.62 ms on lock 0x55a84fa07898. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facf3c2fbdd 00007facfc1cfe3b 00007facfc0998df 00007facfc09e60a 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d
0812 08:03:42.250750 (+ 33954us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:03:42.252860 (+  2110us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:03:42.252867 (+     7us) write_op.cc:312] APPLY: finished
0812 08:03:42.273518 (+ 20651us) log.cc:844] Serialized 30022 byte log entry
0812 08:03:42.278206 (+  4688us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:03:42.279263 (+  1057us) write_op.cc:454] Released schema lock
0812 08:03:42.279528 (+   265us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":190840,"mutex_wait_us":16047,"num_ops":5000,"prepare.queue_time_us":139847,"prepare.run_cpu_time_us":48517,"prepare.run_wall_time_us":143926,"replication_time_us":2575,"spinlock_wait_cycles":17554560}]]}
I20260812 08:03:42.318377 25239 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 89% done.
I20260812 08:03:42.484560 25230 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 89% done.
I20260812 08:03:42.555620 25238 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 99% done.
I20260812 08:03:42.570128 25238 full_stack-insert-scan-test.cc:280] Insertion thread 9 of  10 is 100% done.
I20260812 08:03:42.561836 25226 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 99% done.
I20260812 08:03:42.570349 25226 full_stack-insert-scan-test.cc:280] Insertion thread 2 of  10 is 100% done.
I20260812 08:03:42.665231 25227 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 99% done.
I20260812 08:03:42.665436 25227 full_stack-insert-scan-test.cc:280] Insertion thread 3 of  10 is 100% done.
I20260812 08:03:42.680932 25234 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 99% done.
I20260812 08:03:42.681144 25234 full_stack-insert-scan-test.cc:280] Insertion thread 6 of  10 is 100% done.
I20260812 08:03:42.729346 25239 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 99% done.
I20260812 08:03:42.729552 25239 full_stack-insert-scan-test.cc:280] Insertion thread 10 of  10 is 100% done.
I20260812 08:03:42.773702 25237 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 99% done.
I20260812 08:03:42.773890 25237 full_stack-insert-scan-test.cc:280] Insertion thread 8 of  10 is 100% done.
I20260812 08:03:42.887418 25235 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 99% done.
I20260812 08:03:42.887600 25235 full_stack-insert-scan-test.cc:280] Insertion thread 7 of  10 is 100% done.
I20260812 08:03:42.947664 25230 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 99% done.
I20260812 08:03:42.947866 25230 full_stack-insert-scan-test.cc:280] Insertion thread 4 of  10 is 100% done.
W20260812 08:03:43.087263 25202 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:45861, user_credentials={real_user=slave}} blocked reactor thread for 51344.1us
W20260812 08:03:43.152805 25202 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:45861, user_credentials={real_user=slave}} blocked reactor thread for 51519.7us
I20260812 08:03:43.577342 24143 full_stack-insert-scan-test.cc:333] Time spent concurrent inserts (500000 rows, 10 threads): real 10.329s	user 0.000s	sys 0.000s
I20260812 08:03:57.246469 24143 full_stack-insert-scan-test.cc:346] Doing test scans on table of 500000 rows.
I20260812 08:03:57.346292 24143 full_stack-insert-scan-test.cc:453] Time spent empty projection, 0 col: real 0.047s	user 0.001s	sys 0.000s
I20260812 08:03:57.501958 24143 full_stack-insert-scan-test.cc:453] Time spent key scan, 1 col: real 0.120s	user 0.002s	sys 0.000s
I20260812 08:03:58.034364 24143 full_stack-insert-scan-test.cc:453] Time spent full schema scan, no cache, 10 col: real 0.516s	user 0.034s	sys 0.000s
I20260812 08:03:58.652038 24143 full_stack-insert-scan-test.cc:453] Time spent fault-tolerant full schema scan, no cache, 10 col: real 0.601s	user 0.035s	sys 0.000s
I20260812 08:03:59.209697 24143 full_stack-insert-scan-test.cc:453] Time spent full schema scan, 10 col: real 0.543s	user 0.031s	sys 0.004s
I20260812 08:03:59.417294 24143 full_stack-insert-scan-test.cc:453] Time spent String projection, 1 col: real 0.196s	user 0.027s	sys 0.000s
I20260812 08:03:59.641279 24143 full_stack-insert-scan-test.cc:453] Time spent Int32 projection, 4 col: real 0.194s	user 0.003s	sys 0.000s
I20260812 08:03:59.873471 24143 full_stack-insert-scan-test.cc:453] Time spent Int64 projection, 4 col: real 0.217s	user 0.005s	sys 0.000s
I20260812 08:03:59.874588 24143 tablet_server.cc:179] TabletServer@127.23.147.193:0 shutting down...
I20260812 08:03:59.877946 24143 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260812 08:03:59.878171 24143 tablet_replica.cc:333] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688: stopping tablet replica
I20260812 08:03:59.878309 24143 raft_consensus.cc:2243] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688 [term 1 LEADER]: Raft consensus shutting down.
I20260812 08:03:59.878513 24143 raft_consensus.cc:2272] T 304b715f38bc4a81940b24b40519223d P 8a38e20f59a940439e0128fed1c68688 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 08:03:59.892796 24143 tablet_server.cc:196] TabletServer@127.23.147.193:0 shutdown complete.
I20260812 08:03:59.895345 24143 master.cc:562] Master@127.23.147.254:37597 shutting down...
I20260812 08:03:59.899379 24143 raft_consensus.cc:2243] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b [term 1 LEADER]: Raft consensus shutting down.
I20260812 08:03:59.899555 24143 raft_consensus.cc:2272] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 08:03:59.899628 24143 tablet_replica.cc:333] T 00000000000000000000000000000000 P fbc968d00ac2415fb374f72e1213355b: stopping tablet replica
I20260812 08:03:59.911917 24143 master.cc:584] Master@127.23.147.254:37597 shutdown complete.
[       OK ] EvictionPolicyTypes/FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest/0 (26860 ms)
[ RUN      ] EvictionPolicyTypes/FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest/1
I20260812 08:03:59.930874 24143 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.147.254:35701
I20260812 08:03:59.931284 24143 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:03:59.933265 25309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:03:59.933362 25305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:03:59.933333 24143 server_base.cc:1061] running on GCE node
W20260812 08:03:59.933270 25307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:03:59.933684 24143 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 08:03:59.933730 24143 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 08:03:59.933744 24143 hybrid_clock.cc:648] HybridClock initialized: now 1786521839933744 us; error 0 us; skew 500 ppm
I20260812 08:03:59.934484 24143 webserver.cc:533] Webserver started at http://127.23.147.254:38143/ using document root <none> and password file <none>
I20260812 08:03:59.934617 24143 fs_manager.cc:362] Metadata directory not provided
I20260812 08:03:59.934664 24143 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:03:59.934727 24143 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:03:59.935053 24143 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest_1.1786521782163021-24143-0/minicluster-data/master-0-root/instance:
uuid: "1ae203e7ccd945a394e2df800c38ea01"
format_stamp: "Formatted at 2026-08-12 08:03:59 on dist-test-slave-ncp9"
I20260812 08:03:59.936458 24143 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.002s	sys 0.000s
I20260812 08:03:59.937435 25317 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:03:59.937683 24143 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.000s
I20260812 08:03:59.937747 24143 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest_1.1786521782163021-24143-0/minicluster-data/master-0-root
uuid: "1ae203e7ccd945a394e2df800c38ea01"
format_stamp: "Formatted at 2026-08-12 08:03:59 on dist-test-slave-ncp9"
I20260812 08:03:59.937821 24143 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest_1.1786521782163021-24143-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest_1.1786521782163021-24143-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest_1.1786521782163021-24143-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:03:59.951882 24143 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:03:59.952309 24143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:03:59.956460 24143 rpc_server.cc:307] RPC server started. Bound to: 127.23.147.254:35701
I20260812 08:03:59.965622 25412 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.147.254:35701 every 8 connection(s)
I20260812 08:03:59.965631 25413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:03:59.967635 25413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01: Bootstrap starting.
I20260812 08:03:59.968475 25413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01: Neither blocks nor log segments found. Creating new log.
I20260812 08:03:59.969604 25413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01: No bootstrap required, opened a new log
I20260812 08:03:59.970014 25413 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae203e7ccd945a394e2df800c38ea01" member_type: VOTER }
I20260812 08:03:59.970103 25413 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:03:59.970134 25413 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae203e7ccd945a394e2df800c38ea01, State: Initialized, Role: FOLLOWER
I20260812 08:03:59.970263 25413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae203e7ccd945a394e2df800c38ea01" member_type: VOTER }
I20260812 08:03:59.970333 25413 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 08:03:59.970364 25413 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 08:03:59.970412 25413 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:03:59.971091 25413 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae203e7ccd945a394e2df800c38ea01" member_type: VOTER }
I20260812 08:03:59.971225 25413 leader_election.cc:304] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ae203e7ccd945a394e2df800c38ea01; no voters: 
I20260812 08:03:59.971406 25413 leader_election.cc:290] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 08:03:59.971547 25420 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:03:59.971743 25420 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae203e7ccd945a394e2df800c38ea01, State: Running, Role: LEADER
I20260812 08:03:59.971889 25420 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae203e7ccd945a394e2df800c38ea01" member_type: VOTER }
I20260812 08:03:59.971932 25413 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 08:03:59.972301 25421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ae203e7ccd945a394e2df800c38ea01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae203e7ccd945a394e2df800c38ea01" member_type: VOTER } }
I20260812 08:03:59.972334 25422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ae203e7ccd945a394e2df800c38ea01. Latest consensus state: current_term: 1 leader_uuid: "1ae203e7ccd945a394e2df800c38ea01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ae203e7ccd945a394e2df800c38ea01" member_type: VOTER } }
I20260812 08:03:59.972437 25421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01 [sys.catalog]: This master's current role is: LEADER
I20260812 08:03:59.972517 25422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01 [sys.catalog]: This master's current role is: LEADER
I20260812 08:03:59.972997 25425 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260812 08:03:59.973706 25425 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260812 08:03:59.973932 24143 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260812 08:03:59.975404 25425 catalog_manager.cc:1383] Generated new cluster ID: d4cfe8d492cd40e6af4c32aa9f37b063
I20260812 08:03:59.975447 25425 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260812 08:03:59.985877 25425 catalog_manager.cc:1406] Generated new certificate authority record
I20260812 08:03:59.986394 25425 catalog_manager.cc:1540] Loading token signing keys...
I20260812 08:03:59.998586 25425 catalog_manager.cc:6092] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01: Generated new TSK 0
I20260812 08:03:59.998775 25425 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260812 08:04:00.006201 24143 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260812 08:04:00.008219 25450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:04:00.008361 24143 server_base.cc:1061] running on GCE node
W20260812 08:04:00.008371 25451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 08:04:00.008457 25455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 08:04:00.008718 24143 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260812 08:04:00.008765 24143 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260812 08:04:00.008780 24143 hybrid_clock.cc:648] HybridClock initialized: now 1786521840008780 us; error 0 us; skew 500 ppm
I20260812 08:04:00.009752 24143 webserver.cc:533] Webserver started at http://127.23.147.193:41759/ using document root <none> and password file <none>
I20260812 08:04:00.009892 24143 fs_manager.cc:362] Metadata directory not provided
I20260812 08:04:00.009935 24143 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 08:04:00.009994 24143 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 08:04:00.010345 24143 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest_1.1786521782163021-24143-0/minicluster-data/ts-0-root/instance:
uuid: "7b4cfff33abc4a07bdaff2da21e71dc9"
format_stamp: "Formatted at 2026-08-12 08:04:00 on dist-test-slave-ncp9"
I20260812 08:04:00.011780 24143 fs_manager.cc:696] Time spent creating directory manager: real 0.001s	user 0.002s	sys 0.000s
I20260812 08:04:00.012630 25465 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:04:00.012852 24143 fs_manager.cc:730] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
I20260812 08:04:00.012944 24143 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest_1.1786521782163021-24143-0/minicluster-data/ts-0-root
uuid: "7b4cfff33abc4a07bdaff2da21e71dc9"
format_stamp: "Formatted at 2026-08-12 08:04:00 on dist-test-slave-ncp9"
I20260812 08:04:00.013012 24143 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest_1.1786521782163021-24143-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest_1.1786521782163021-24143-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskWOHThk/test-tmp/full_stack-insert-scan-test.0.EvictionPolicyTypes_FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest_1.1786521782163021-24143-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 08:04:00.023945 24143 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260812 08:04:00.024353 24143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 08:04:00.024657 24143 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260812 08:04:00.025204 24143 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 08:04:00.025246 24143 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:04:00.025291 24143 ts_tablet_manager.cc:616] Registered 0 tablets
I20260812 08:04:00.025319 24143 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260812 08:04:00.029287 24143 rpc_server.cc:307] RPC server started. Bound to: 127.23.147.193:33269
I20260812 08:04:00.030026 25608 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.23.147.193:33269 every 8 connection(s)
I20260812 08:04:00.036873 25612 heartbeater.cc:344] Connected to a master server at 127.23.147.254:35701
I20260812 08:04:00.037014 25612 heartbeater.cc:461] Registering TS with master...
I20260812 08:04:00.037240 25612 heartbeater.cc:507] Master 127.23.147.254:35701 requested a full tablet report, sending...
I20260812 08:04:00.037871 25340 ts_manager.cc:194] Registered new tserver with Master: 7b4cfff33abc4a07bdaff2da21e71dc9 (127.23.147.193:33269)
I20260812 08:04:00.038610 25340 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38142
I20260812 08:04:00.038762 24143 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.008816023s
I20260812 08:04:00.046269 25340 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38144:
name: "full-stack-mrs-test-tbl"
schema {
  columns {
    name: "key"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int32_val1"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int32_val2"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int32_val3"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int32_val4"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int64_val1"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int64_val2"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int64_val3"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int64_val4"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20260812 08:04:00.055028 25527 tablet_service.cc:1511] Processing CreateTablet for tablet 8dd491968e374922ad9febecae3337de (DEFAULT_TABLE table=full-stack-mrs-test-tbl [id=9432ba576762435e8522d5cca9da8492]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 08:04:00.055328 25527 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dd491968e374922ad9febecae3337de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 08:04:00.057540 25634 tablet_bootstrap.cc:492] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9: Bootstrap starting.
I20260812 08:04:00.058406 25634 tablet_bootstrap.cc:654] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9: Neither blocks nor log segments found. Creating new log.
I20260812 08:04:00.059533 25634 tablet_bootstrap.cc:492] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9: No bootstrap required, opened a new log
I20260812 08:04:00.059623 25634 ts_tablet_manager.cc:1403] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9: Time spent bootstrapping tablet: real 0.002s	user 0.001s	sys 0.000s
I20260812 08:04:00.060081 25634 raft_consensus.cc:359] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4cfff33abc4a07bdaff2da21e71dc9" member_type: VOTER last_known_addr { host: "127.23.147.193" port: 33269 } }
I20260812 08:04:00.060197 25634 raft_consensus.cc:385] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 08:04:00.060223 25634 raft_consensus.cc:740] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4cfff33abc4a07bdaff2da21e71dc9, State: Initialized, Role: FOLLOWER
I20260812 08:04:00.060361 25634 consensus_queue.cc:260] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4cfff33abc4a07bdaff2da21e71dc9" member_type: VOTER last_known_addr { host: "127.23.147.193" port: 33269 } }
I20260812 08:04:00.060436 25634 raft_consensus.cc:399] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 08:04:00.060470 25634 raft_consensus.cc:493] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 08:04:00.060518 25634 raft_consensus.cc:3060] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 08:04:00.061383 25634 raft_consensus.cc:515] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4cfff33abc4a07bdaff2da21e71dc9" member_type: VOTER last_known_addr { host: "127.23.147.193" port: 33269 } }
I20260812 08:04:00.061513 25634 leader_election.cc:304] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b4cfff33abc4a07bdaff2da21e71dc9; no voters: 
I20260812 08:04:00.061718 25634 leader_election.cc:290] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260812 08:04:00.061834 25637 raft_consensus.cc:2804] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 08:04:00.062014 25637 raft_consensus.cc:697] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9 [term 1 LEADER]: Becoming Leader. State: Replica: 7b4cfff33abc4a07bdaff2da21e71dc9, State: Running, Role: LEADER
I20260812 08:04:00.062067 25612 heartbeater.cc:499] Master 127.23.147.254:35701 was elected leader, sending a full tablet report...
I20260812 08:04:00.062031 25634 ts_tablet_manager.cc:1434] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20260812 08:04:00.062177 25637 consensus_queue.cc:237] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4cfff33abc4a07bdaff2da21e71dc9" member_type: VOTER last_known_addr { host: "127.23.147.193" port: 33269 } }
I20260812 08:04:00.063463 25340 catalog_manager.cc:5719] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b4cfff33abc4a07bdaff2da21e71dc9 (127.23.147.193). New cstate: current_term: 1 leader_uuid: "7b4cfff33abc4a07bdaff2da21e71dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b4cfff33abc4a07bdaff2da21e71dc9" member_type: VOTER last_known_addr { host: "127.23.147.193" port: 33269 } health_report { overall_health: HEALTHY } } }
I20260812 08:04:00.484122 25657 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 9% done.
I20260812 08:04:00.559821 25647 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 9% done.
I20260812 08:04:00.567067 25659 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 9% done.
I20260812 08:04:00.601555 25645 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 9% done.
I20260812 08:04:00.656029 25649 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 9% done.
I20260812 08:04:00.666247 25653 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 9% done.
I20260812 08:04:00.703679 25662 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 9% done.
I20260812 08:04:00.748661 25655 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 9% done.
I20260812 08:04:00.752429 25661 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 9% done.
I20260812 08:04:00.806033 25650 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 9% done.
I20260812 08:04:01.107209 25647 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 19% done.
I20260812 08:04:01.161108 25657 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 19% done.
I20260812 08:04:01.307159 25649 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 19% done.
I20260812 08:04:01.448846 25645 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 19% done.
I20260812 08:04:01.488569 25653 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 19% done.
I20260812 08:04:01.583892 25659 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 19% done.
I20260812 08:04:01.704871 25650 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 19% done.
I20260812 08:04:01.807233 25657 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 29% done.
I20260812 08:04:01.850772 25662 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 19% done.
I20260812 08:04:01.852669 25671 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=16, attempt_no=0}) took 1029 ms. Trace:
I20260812 08:04:01.861065 25671 rpcz_store.cc:276] 0812 08:04:00.823389 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:00.823458 (+    69us) service_pool.cc:224] Handling call
0812 08:04:01.852645 (+1029187us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:01.215629 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:01.215800 (+   171us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:01.215805 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:04:01.215807 (+     2us) tablet.cc:662] Decoding operations
0812 08:04:01.246079 (+ 30272us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:01.246094 (+    15us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:01.246102 (+     8us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:01.278312 (+ 32210us) tablet.cc:701] Row locks acquired
0812 08:04:01.278315 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:04:01.278393 (+    78us) write_op.cc:270] Start()
0812 08:04:01.278436 (+    43us) write_op.cc:276] Timestamp: P: 1786521841278376 usec, L: 0
0812 08:04:01.278439 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:04:01.278807 (+   368us) log.cc:844] Serialized 486499 byte log entry
0812 08:04:01.281198 (+  2391us) op_driver.cc:464] REPLICATION: finished
0812 08:04:01.309542 (+ 28344us) write_op.cc:301] APPLY: starting
0812 08:04:01.309557 (+    15us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:01.331344 (+ 21787us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:01.331362 (+    18us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:01.800692 (+469330us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:01.818927 (+ 18235us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:01.818935 (+     8us) write_op.cc:312] APPLY: finished
0812 08:04:01.839569 (+ 20634us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:01.851107 (+ 11538us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:01.852240 (+  1133us) write_op.cc:454] Released schema lock
0812 08:04:01.852504 (+   264us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":28294,"mutex_wait_us":13380,"num_ops":5000,"prepare.queue_time_us":390713,"prepare.run_cpu_time_us":63147,"prepare.run_wall_time_us":65653,"replication_time_us":2730,"spinlock_wait_cycles":69248}]]}
I20260812 08:04:02.012568 25661 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 19% done.
I20260812 08:04:02.025395 25655 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 19% done.
I20260812 08:04:02.029341 25673 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=18, attempt_no=0}) took 1017 ms. Trace:
I20260812 08:04:02.029565 25673 rpcz_store.cc:276] 0812 08:04:01.012329 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:01.012382 (+    53us) service_pool.cc:224] Handling call
0812 08:04:02.029317 (+1016935us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:01.333655 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:01.333814 (+   159us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:01.333819 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:04:01.333820 (+     1us) tablet.cc:662] Decoding operations
0812 08:04:01.361294 (+ 27474us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:01.361307 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:01.361314 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:01.381756 (+ 20442us) tablet.cc:701] Row locks acquired
0812 08:04:01.381759 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:04:01.381845 (+    86us) write_op.cc:270] Start()
0812 08:04:01.381891 (+    46us) write_op.cc:276] Timestamp: P: 1786521841381822 usec, L: 0
0812 08:04:01.381893 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:04:01.382265 (+   372us) log.cc:844] Serialized 486014 byte log entry
0812 08:04:01.384529 (+  2264us) op_driver.cc:464] REPLICATION: finished
0812 08:04:01.600432 (+215903us) write_op.cc:301] APPLY: starting
0812 08:04:01.600445 (+    13us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:01.622539 (+ 22094us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:01.622557 (+    18us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:02.005232 (+382675us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:02.007220 (+  1988us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:02.007226 (+     6us) write_op.cc:312] APPLY: finished
0812 08:04:02.027598 (+ 20372us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:02.027931 (+   333us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:02.028861 (+   930us) write_op.cc:454] Released schema lock
0812 08:04:02.029195 (+   334us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":215854,"num_ops":5000,"prepare.queue_time_us":319985,"prepare.run_cpu_time_us":48581,"prepare.run_wall_time_us":50951,"replication_time_us":2605,"spinlock_wait_cycles":54144}]]}
I20260812 08:04:02.070458 25676 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=17, attempt_no=0}) took 1067 ms. Trace:
I20260812 08:04:02.071683 25645 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 29% done.
I20260812 08:04:02.073112 25676 rpcz_store.cc:276] 0812 08:04:01.003453 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:01.003516 (+    63us) service_pool.cc:224] Handling call
0812 08:04:02.070438 (+1066922us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:01.281302 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:01.281485 (+   183us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:01.281490 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:04:01.281492 (+     2us) tablet.cc:662] Decoding operations
0812 08:04:01.310632 (+ 29140us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:01.310643 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:01.310649 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:01.330763 (+ 20114us) tablet.cc:701] Row locks acquired
0812 08:04:01.330766 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:04:01.330845 (+    79us) write_op.cc:270] Start()
0812 08:04:01.330888 (+    43us) write_op.cc:276] Timestamp: P: 1786521841330826 usec, L: 0
0812 08:04:01.330890 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:04:01.331219 (+   329us) log.cc:844] Serialized 485779 byte log entry
0812 08:04:01.333548 (+  2329us) op_driver.cc:464] REPLICATION: finished
0812 08:04:01.505733 (+172185us) write_op.cc:301] APPLY: starting
0812 08:04:01.505777 (+    44us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:01.511284 (+  5507us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:01.511307 (+    23us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:02.020205 (+508898us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:02.039811 (+ 19606us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:02.039819 (+     8us) write_op.cc:312] APPLY: finished
0812 08:04:02.061229 (+ 21410us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:02.069020 (+  7791us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:02.070098 (+  1078us) write_op.cc:454] Released schema lock
0812 08:04:02.070329 (+   231us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":172127,"mutex_wait_us":6783,"num_ops":5000,"prepare.queue_time_us":276499,"prepare.run_cpu_time_us":49694,"prepare.run_wall_time_us":52328,"replication_time_us":2626,"spinlock_wait_cycles":75520}]]}
I20260812 08:04:02.121934 25653 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 29% done.
W20260812 08:04:02.153748 25622 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:33269, user_credentials={real_user=slave}} blocked reactor thread for 79156us
W20260812 08:04:02.205813 25622 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:33269, user_credentials={real_user=slave}} blocked reactor thread for 51733.1us
I20260812 08:04:02.313822 25647 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 29% done.
I20260812 08:04:02.316461 25650 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 29% done.
I20260812 08:04:02.422477 25671 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=20, attempt_no=0}) took 1031 ms. Trace:
I20260812 08:04:02.422727 25671 rpcz_store.cc:276] 0812 08:04:01.391008 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:01.391058 (+    50us) service_pool.cc:224] Handling call
0812 08:04:02.422449 (+1031391us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:01.453557 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:01.453727 (+   170us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:01.453732 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:04:01.453734 (+     2us) tablet.cc:662] Decoding operations
0812 08:04:01.483871 (+ 30137us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:01.483884 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:01.483891 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:01.551414 (+ 67523us) tablet.cc:701] Row locks acquired
0812 08:04:01.551418 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:04:01.551513 (+    95us) write_op.cc:270] Start()
0812 08:04:01.551560 (+    47us) write_op.cc:276] Timestamp: P: 1786521841551492 usec, L: 0
0812 08:04:01.551562 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:04:01.551912 (+   350us) log.cc:844] Serialized 485991 byte log entry
0812 08:04:01.562908 (+ 10996us) op_driver.cc:464] REPLICATION: finished
0812 08:04:01.885563 (+322655us) write_op.cc:301] APPLY: starting
0812 08:04:01.885585 (+    22us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:01.899755 (+ 14170us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:01.899777 (+    22us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:02.380110 (+480333us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:02.398231 (+ 18121us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:02.398239 (+     8us) write_op.cc:312] APPLY: finished
0812 08:04:02.402861 (+  4622us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:02.421017 (+ 18156us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:02.422019 (+  1002us) write_op.cc:454] Released schema lock
0812 08:04:02.422303 (+   284us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":322600,"mutex_wait_us":1442,"num_ops":5000,"prepare.queue_time_us":61781,"prepare.run_cpu_time_us":62946,"prepare.run_wall_time_us":109553,"replication_time_us":11318,"spinlock_wait_cycles":46848}]]}
I20260812 08:04:02.433486 25659 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 29% done.
I20260812 08:04:02.452010 25649 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 29% done.
I20260812 08:04:02.724725 25655 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 29% done.
I20260812 08:04:02.731562 25645 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 39% done.
I20260812 08:04:02.744112 25662 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 29% done.
I20260812 08:04:02.794435 25673 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=25, attempt_no=0}) took 1028 ms. Trace:
I20260812 08:04:02.794662 25673 rpcz_store.cc:276] 0812 08:04:01.765867 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:01.765928 (+    61us) service_pool.cc:224] Handling call
0812 08:04:02.794408 (+1028480us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:02.348358 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:02.348511 (+   153us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:02.348526 (+    15us) write_op.cc:435] Acquired schema lock
0812 08:04:02.348528 (+     2us) tablet.cc:662] Decoding operations
0812 08:04:02.439817 (+ 91289us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:02.439828 (+    11us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:02.439834 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:02.480178 (+ 40344us) tablet.cc:701] Row locks acquired
0812 08:04:02.480181 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:04:02.480252 (+    71us) write_op.cc:270] Start()
0812 08:04:02.480292 (+    40us) write_op.cc:276] Timestamp: P: 1786521842480235 usec, L: 0
0812 08:04:02.480294 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:04:02.480599 (+   305us) log.cc:844] Serialized 486276 byte log entry
0812 08:04:02.493141 (+ 12542us) op_driver.cc:464] REPLICATION: finished
0812 08:04:02.493263 (+   122us) write_op.cc:301] APPLY: starting
0812 08:04:02.493284 (+    21us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:02.499144 (+  5860us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:02.499170 (+    26us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:02.775809 (+276639us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:02.778030 (+  2221us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:02.778039 (+     9us) write_op.cc:312] APPLY: finished
0812 08:04:02.782641 (+  4602us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:02.792993 (+ 10352us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:02.794006 (+  1013us) write_op.cc:454] Released schema lock
0812 08:04:02.794280 (+   274us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":46,"num_ops":5000,"prepare.queue_time_us":581713,"prepare.run_cpu_time_us":48209,"prepare.run_wall_time_us":144995,"replication_time_us":12786,"spinlock_wait_cycles":58112}]]}
I20260812 08:04:02.819576 25661 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 29% done.
I20260812 08:04:02.903592 25657 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 39% done.
I20260812 08:04:02.959715 25647 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 39% done.
I20260812 08:04:03.096932 25649 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 39% done.
I20260812 08:04:03.096982 25659 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 39% done.
I20260812 08:04:03.199375 25662 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 39% done.
I20260812 08:04:03.250326 25673 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=29, attempt_no=0}) took 1025 ms. Trace:
I20260812 08:04:03.250537 25673 rpcz_store.cc:276] 0812 08:04:02.225306 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:02.225357 (+    51us) service_pool.cc:224] Handling call
0812 08:04:03.250302 (+1024945us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:02.895354 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:02.895536 (+   182us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:02.895541 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:04:02.895543 (+     2us) tablet.cc:662] Decoding operations
0812 08:04:02.978564 (+ 83021us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:02.978580 (+    16us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:02.978587 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:03.015375 (+ 36788us) tablet.cc:701] Row locks acquired
0812 08:04:03.015378 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:04:03.015459 (+    81us) write_op.cc:270] Start()
0812 08:04:03.015507 (+    48us) write_op.cc:276] Timestamp: P: 1786521843015443 usec, L: 0
0812 08:04:03.015509 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:04:03.015875 (+   366us) log.cc:844] Serialized 486197 byte log entry
0812 08:04:03.018123 (+  2248us) op_driver.cc:464] REPLICATION: finished
0812 08:04:03.018215 (+    92us) write_op.cc:301] APPLY: starting
0812 08:04:03.018231 (+    16us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:03.024050 (+  5819us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:03.024073 (+    23us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:03.230255 (+206182us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:03.232464 (+  2209us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:03.232472 (+     8us) write_op.cc:312] APPLY: finished
0812 08:04:03.243947 (+ 11475us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:03.248993 (+  5046us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:03.249934 (+   941us) write_op.cc:454] Released schema lock
0812 08:04:03.250185 (+   251us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":37,"num_ops":5000,"prepare.queue_time_us":669313,"prepare.run_cpu_time_us":50496,"prepare.run_wall_time_us":120607,"replication_time_us":2585,"spinlock_wait_cycles":52992}]]}
I20260812 08:04:03.315737 25676 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=30, attempt_no=0}) took 1036 ms. Trace:
I20260812 08:04:03.315868 25676 rpcz_store.cc:276] 0812 08:04:02.279630 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:02.279699 (+    69us) service_pool.cc:224] Handling call
0812 08:04:03.315711 (+1036012us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:03.016019 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:03.016181 (+   162us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:03.016185 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:04:03.016187 (+     2us) tablet.cc:662] Decoding operations
0812 08:04:03.073628 (+ 57441us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:03.073640 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:03.073646 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:03.098802 (+ 25156us) tablet.cc:701] Row locks acquired
0812 08:04:03.098806 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:04:03.098892 (+    86us) write_op.cc:270] Start()
0812 08:04:03.098938 (+    46us) write_op.cc:276] Timestamp: P: 1786521843098869 usec, L: 0
0812 08:04:03.098940 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:04:03.099299 (+   359us) log.cc:844] Serialized 485798 byte log entry
0812 08:04:03.101619 (+  2320us) op_driver.cc:464] REPLICATION: finished
0812 08:04:03.101688 (+    69us) write_op.cc:301] APPLY: starting
0812 08:04:03.101705 (+    17us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:03.107687 (+  5982us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:03.107715 (+    28us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:03.295344 (+187629us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:03.309474 (+ 14130us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:03.309482 (+     8us) write_op.cc:312] APPLY: finished
0812 08:04:03.314047 (+  4565us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:03.314345 (+   298us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:03.315328 (+   983us) write_op.cc:454] Released schema lock
0812 08:04:03.315574 (+   246us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":27,"mutex_wait_us":11330,"num_ops":5000,"prepare.queue_time_us":735420,"prepare.run_cpu_time_us":50153,"prepare.run_wall_time_us":85575,"replication_time_us":2638,"spinlock_wait_cycles":36480}]]}
I20260812 08:04:03.386489 25653 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 39% done.
I20260812 08:04:03.601416 25655 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 39% done.
I20260812 08:04:03.626590 25650 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 39% done.
I20260812 08:04:03.637603 25661 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 39% done.
I20260812 08:04:03.629783 25672 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=33, attempt_no=0}) took 1058 ms. Trace:
I20260812 08:04:03.637838 25672 rpcz_store.cc:276] 0812 08:04:02.571344 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:02.571387 (+    43us) service_pool.cc:224] Handling call
0812 08:04:03.629756 (+1058369us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:03.313017 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:03.313189 (+   172us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:03.313195 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:04:03.313196 (+     1us) tablet.cc:662] Decoding operations
0812 08:04:03.375511 (+ 62315us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:03.375520 (+     9us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:03.375526 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:03.395269 (+ 19743us) tablet.cc:701] Row locks acquired
0812 08:04:03.395271 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:04:03.395352 (+    81us) write_op.cc:270] Start()
0812 08:04:03.395398 (+    46us) write_op.cc:276] Timestamp: P: 1786521843395332 usec, L: 0
0812 08:04:03.395400 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:04:03.395756 (+   356us) log.cc:844] Serialized 486078 byte log entry
0812 08:04:03.398259 (+  2503us) op_driver.cc:464] REPLICATION: finished
0812 08:04:03.398348 (+    89us) write_op.cc:301] APPLY: starting
0812 08:04:03.398364 (+    16us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:03.404232 (+  5868us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:03.404258 (+    26us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:03.609951 (+205693us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:03.612044 (+  2093us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:03.612052 (+     8us) write_op.cc:312] APPLY: finished
0812 08:04:03.627850 (+ 15798us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:03.628192 (+   342us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:03.629367 (+  1175us) write_op.cc:454] Released schema lock
0812 08:04:03.629625 (+   258us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":53,"num_ops":5000,"prepare.queue_time_us":741016,"prepare.run_cpu_time_us":50252,"prepare.run_wall_time_us":95930,"replication_time_us":2827,"spinlock_wait_cycles":20992}]]}
I20260812 08:04:03.775077 25673 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=34, attempt_no=0}) took 1090 ms. Trace:
I20260812 08:04:03.775204 25673 rpcz_store.cc:276] 0812 08:04:02.684345 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:02.684398 (+    53us) service_pool.cc:224] Handling call
0812 08:04:03.775051 (+1090653us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:03.409001 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:03.409171 (+   170us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:03.409177 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:04:03.409178 (+     1us) tablet.cc:662] Decoding operations
0812 08:04:03.464538 (+ 55360us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:03.464552 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:03.464558 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:03.525245 (+ 60687us) tablet.cc:701] Row locks acquired
0812 08:04:03.525249 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:04:03.525344 (+    95us) write_op.cc:270] Start()
0812 08:04:03.525390 (+    46us) write_op.cc:276] Timestamp: P: 1786521843525320 usec, L: 0
0812 08:04:03.525393 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:04:03.525895 (+   502us) log.cc:844] Serialized 485801 byte log entry
0812 08:04:03.528256 (+  2361us) op_driver.cc:464] REPLICATION: finished
0812 08:04:03.528340 (+    84us) write_op.cc:301] APPLY: starting
0812 08:04:03.528360 (+    20us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:03.534046 (+  5686us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:03.534069 (+    23us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:03.750680 (+216611us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:03.752758 (+  2078us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:03.752767 (+     9us) write_op.cc:312] APPLY: finished
0812 08:04:03.756798 (+  4031us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:03.765052 (+  8254us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:03.774371 (+  9319us) spinlock_profiling.cc:243] Waited 8.38 ms on lock 0x55a84fa5fc20. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfc1c860b 00007facfc1c9ecb 00007facfc1c8c17 00007facfc16e26f 00007facfc16d773 00007facfc16be49 00007facfc15916c 00007facfc158b95 00007facfc1579ca 00007facfc15a95b
0812 08:04:03.774666 (+   295us) write_op.cc:454] Released schema lock
0812 08:04:03.774922 (+   256us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":32,"num_ops":5000,"prepare.queue_time_us":723833,"prepare.run_cpu_time_us":54139,"prepare.run_wall_time_us":119373,"replication_time_us":2828,"spinlock_wait_cycles":19283456}]]}
I20260812 08:04:03.779838 25657 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 49% done.
I20260812 08:04:03.845825 25645 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 49% done.
I20260812 08:04:03.927392 25659 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 49% done.
I20260812 08:04:03.966773 25647 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 49% done.
I20260812 08:04:03.995414 25673 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=36, attempt_no=0}) took 1094 ms. Trace:
I20260812 08:04:03.995652 25673 rpcz_store.cc:276] 0812 08:04:02.900491 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:02.900544 (+    53us) service_pool.cc:224] Handling call
0812 08:04:03.995389 (+1094845us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:03.657041 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:03.657219 (+   178us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:03.657225 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:04:03.657227 (+     2us) tablet.cc:662] Decoding operations
0812 08:04:03.701962 (+ 44735us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:03.701975 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:03.701982 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:03.773995 (+ 72013us) tablet.cc:701] Row locks acquired
0812 08:04:03.773999 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:04:03.774089 (+    90us) write_op.cc:270] Start()
0812 08:04:03.774142 (+    53us) write_op.cc:276] Timestamp: P: 1786521843774066 usec, L: 0
0812 08:04:03.774144 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:04:03.774533 (+   389us) log.cc:844] Serialized 485363 byte log entry
0812 08:04:03.776839 (+  2306us) op_driver.cc:464] REPLICATION: finished
0812 08:04:03.792986 (+ 16147us) write_op.cc:301] APPLY: starting
0812 08:04:03.793008 (+    22us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:03.798623 (+  5615us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:03.798650 (+    27us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:03.987031 (+188381us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:03.989160 (+  2129us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:03.989168 (+     8us) write_op.cc:312] APPLY: finished
0812 08:04:03.993679 (+  4511us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:03.994014 (+   335us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:03.995019 (+  1005us) write_op.cc:454] Released schema lock
0812 08:04:03.995264 (+   245us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":16066,"mutex_wait_us":6623,"num_ops":5000,"prepare.queue_time_us":755869,"prepare.run_cpu_time_us":62508,"prepare.run_wall_time_us":119922,"replication_time_us":2669,"spinlock_wait_cycles":38656}]]}
I20260812 08:04:04.080506 25649 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 49% done.
I20260812 08:04:04.223999 25653 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 49% done.
I20260812 08:04:04.393059 25655 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 49% done.
I20260812 08:04:04.418380 25673 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=40, attempt_no=0}) took 1004 ms. Trace:
I20260812 08:04:04.418612 25673 rpcz_store.cc:276] 0812 08:04:03.413569 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:03.413620 (+    51us) service_pool.cc:224] Handling call
0812 08:04:04.418356 (+1004736us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:04.104270 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:04.104429 (+   159us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:04.104434 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:04:04.104436 (+     2us) tablet.cc:662] Decoding operations
0812 08:04:04.132791 (+ 28355us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:04.132804 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:04.132812 (+     8us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:04.153811 (+ 20999us) tablet.cc:701] Row locks acquired
0812 08:04:04.153814 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:04:04.153902 (+    88us) write_op.cc:270] Start()
0812 08:04:04.153947 (+    45us) write_op.cc:276] Timestamp: P: 1786521844153879 usec, L: 0
0812 08:04:04.153949 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:04:04.154309 (+   360us) log.cc:844] Serialized 486238 byte log entry
0812 08:04:04.156549 (+  2240us) op_driver.cc:464] REPLICATION: finished
0812 08:04:04.161821 (+  5272us) write_op.cc:301] APPLY: starting
0812 08:04:04.161840 (+    19us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:04.167778 (+  5938us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:04.167799 (+    21us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:04.363872 (+196073us) spinlock_profiling.cc:243] Waited 4.24 ms on lock 0x55a85171e38c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:04:04.409970 (+ 46098us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:04.411973 (+  2003us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:04.411980 (+     7us) write_op.cc:312] APPLY: finished
0812 08:04:04.416655 (+  4675us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:04.417025 (+   370us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:04.417985 (+   960us) write_op.cc:454] Released schema lock
0812 08:04:04.418237 (+   252us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":5214,"mutex_wait_us":13829,"num_ops":5000,"prepare.queue_time_us":689883,"prepare.run_cpu_time_us":49829,"prepare.run_wall_time_us":52251,"replication_time_us":2566,"spinlock_wait_cycles":9807744,"thread_start_us":97,"threads_started":1}]]}
I20260812 08:04:04.574271 25679 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=41, attempt_no=0}) took 1099 ms. Trace:
I20260812 08:04:04.574410 25679 rpcz_store.cc:276] 0812 08:04:03.474614 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:03.474668 (+    54us) service_pool.cc:224] Handling call
0812 08:04:04.574247 (+1099579us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:04.156560 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:04.156734 (+   174us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:04.156739 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:04:04.156740 (+     1us) tablet.cc:662] Decoding operations
0812 08:04:04.184768 (+ 28028us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:04.184782 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:04.184789 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:04.206922 (+ 22133us) tablet.cc:701] Row locks acquired
0812 08:04:04.206925 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:04:04.207014 (+    89us) write_op.cc:270] Start()
0812 08:04:04.207060 (+    46us) write_op.cc:276] Timestamp: P: 1786521844206991 usec, L: 0
0812 08:04:04.207063 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:04:04.207405 (+   342us) log.cc:844] Serialized 486712 byte log entry
0812 08:04:04.209802 (+  2397us) op_driver.cc:464] REPLICATION: finished
0812 08:04:04.209874 (+    72us) write_op.cc:301] APPLY: starting
0812 08:04:04.209891 (+    17us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:04.215601 (+  5710us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:04.215626 (+    25us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:04.369010 (+153384us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x55a85171e38c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:04:04.541004 (+171994us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:04.543235 (+  2231us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:04.543242 (+     7us) write_op.cc:312] APPLY: finished
0812 08:04:04.547753 (+  4511us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:04.572778 (+ 25025us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:04.573846 (+  1068us) write_op.cc:454] Released schema lock
0812 08:04:04.574115 (+   269us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":27,"mutex_wait_us":13153,"num_ops":5000,"prepare.queue_time_us":681160,"prepare.run_cpu_time_us":49096,"prepare.run_wall_time_us":50933,"replication_time_us":2714,"spinlock_wait_cycles":27548544}]]}
I20260812 08:04:04.579316 25661 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 49% done.
I20260812 08:04:04.583248 25683 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=42, attempt_no=0}) took 1095 ms. Trace:
I20260812 08:04:04.583474 25683 rpcz_store.cc:276] 0812 08:04:03.487581 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:03.487642 (+    61us) service_pool.cc:224] Handling call
0812 08:04:04.583227 (+1095585us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:04.207535 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:04.207705 (+   170us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:04.207711 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:04:04.207712 (+     1us) tablet.cc:662] Decoding operations
0812 08:04:04.238227 (+ 30515us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:04.238240 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:04.238246 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:04.260836 (+ 22590us) tablet.cc:701] Row locks acquired
0812 08:04:04.260840 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:04:04.260949 (+   109us) write_op.cc:270] Start()
0812 08:04:04.260997 (+    48us) write_op.cc:276] Timestamp: P: 1786521844260925 usec, L: 0
0812 08:04:04.261000 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:04:04.261370 (+   370us) log.cc:844] Serialized 486149 byte log entry
0812 08:04:04.263687 (+  2317us) op_driver.cc:464] REPLICATION: finished
0812 08:04:04.273379 (+  9692us) write_op.cc:301] APPLY: starting
0812 08:04:04.273400 (+    21us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:04.279133 (+  5733us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:04.279160 (+    27us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:04.363872 (+ 84712us) spinlock_profiling.cc:243] Waited 6.67 ms on lock 0x55a85171e38c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:04:04.574611 (+210739us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:04.577160 (+  2549us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:04.577168 (+     8us) write_op.cc:312] APPLY: finished
0812 08:04:04.581528 (+  4360us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:04.581836 (+   308us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:04.582886 (+  1050us) write_op.cc:454] Released schema lock
0812 08:04:04.583109 (+   223us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":9612,"num_ops":5000,"prepare.queue_time_us":719264,"prepare.run_cpu_time_us":49088,"prepare.run_wall_time_us":53917,"replication_time_us":2659,"spinlock_wait_cycles":15369600,"thread_start_us":103,"threads_started":1}]]}
I20260812 08:04:04.782554 25672 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=43, attempt_no=0}) took 1084 ms. Trace:
I20260812 08:04:04.800959 25672 rpcz_store.cc:276] 0812 08:04:03.698506 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:03.698573 (+    67us) service_pool.cc:224] Handling call
0812 08:04:04.782528 (+1083955us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:04.261507 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:04.261679 (+   172us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:04.261685 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:04:04.261687 (+     2us) tablet.cc:662] Decoding operations
0812 08:04:04.289851 (+ 28164us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:04.289865 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:04.289872 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:04.310809 (+ 20937us) tablet.cc:701] Row locks acquired
0812 08:04:04.310812 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:04:04.310902 (+    90us) write_op.cc:270] Start()
0812 08:04:04.310948 (+    46us) write_op.cc:276] Timestamp: P: 1786521844310879 usec, L: 0
0812 08:04:04.310951 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:04:04.311317 (+   366us) log.cc:844] Serialized 485991 byte log entry
0812 08:04:04.313615 (+  2298us) op_driver.cc:464] REPLICATION: finished
0812 08:04:04.349542 (+ 35927us) write_op.cc:301] APPLY: starting
0812 08:04:04.349559 (+    17us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:04.355394 (+  5835us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:04.355421 (+    27us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:04.753918 (+398497us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:04.756125 (+  2207us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:04.756133 (+     8us) write_op.cc:312] APPLY: finished
0812 08:04:04.781072 (+ 24939us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:04.781171 (+    99us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:04.782116 (+   945us) write_op.cc:454] Released schema lock
0812 08:04:04.782395 (+   279us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":35874,"num_ops":5000,"prepare.queue_time_us":562173,"prepare.run_cpu_time_us":49827,"prepare.run_wall_time_us":49909,"replication_time_us":2636,"spinlock_wait_cycles":35200}]]}
I20260812 08:04:04.792227 25662 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 49% done.
I20260812 08:04:04.832263 25645 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 59% done.
I20260812 08:04:04.850225 25650 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 49% done.
I20260812 08:04:04.862090 25657 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 59% done.
W20260812 08:04:04.899749 25622 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:33269, user_credentials={real_user=slave}} blocked reactor thread for 67717.3us
I20260812 08:04:04.956015 25659 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 59% done.
I20260812 08:04:04.997655 25679 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=45, attempt_no=0}) took 1038 ms. Trace:
I20260812 08:04:05.008930 25679 rpcz_store.cc:276] 0812 08:04:03.959642 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:03.959702 (+    60us) service_pool.cc:224] Handling call
0812 08:04:04.997628 (+1037926us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:04.391412 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:04.391572 (+   160us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:04.391577 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:04:04.391579 (+     2us) tablet.cc:662] Decoding operations
0812 08:04:04.419392 (+ 27813us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:04.419402 (+    10us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:04.419406 (+     4us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:04.463885 (+ 44479us) tablet.cc:701] Row locks acquired
0812 08:04:04.463889 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:04:04.463975 (+    86us) write_op.cc:270] Start()
0812 08:04:04.464021 (+    46us) write_op.cc:276] Timestamp: P: 1786521844463954 usec, L: 0
0812 08:04:04.464023 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:04:04.464363 (+   340us) log.cc:844] Serialized 486596 byte log entry
0812 08:04:04.466709 (+  2346us) op_driver.cc:464] REPLICATION: finished
0812 08:04:04.575397 (+108688us) write_op.cc:301] APPLY: starting
0812 08:04:04.575413 (+    16us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:04.613049 (+ 37636us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:04.613072 (+    23us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:04.933143 (+320071us) spinlock_profiling.cc:243] Waited 8.04 ms on lock 0x55a85171e38c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:04:04.989591 (+ 56448us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:04.991686 (+  2095us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:04.991692 (+     6us) write_op.cc:312] APPLY: finished
0812 08:04:04.996157 (+  4465us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:04.996243 (+    86us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:04.997243 (+  1000us) write_op.cc:454] Released schema lock
0812 08:04:04.997505 (+   262us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":108633,"mutex_wait_us":12622,"num_ops":5000,"prepare.queue_time_us":431038,"prepare.run_cpu_time_us":48983,"prepare.run_wall_time_us":73032,"replication_time_us":2652,"spinlock_wait_cycles":18506880}]]}
I20260812 08:04:05.071573 25647 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 59% done.
I20260812 08:04:05.268748 25653 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 59% done.
I20260812 08:04:05.281944 25673 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=48, attempt_no=0}) took 1072 ms. Trace:
I20260812 08:04:05.292941 25673 rpcz_store.cc:276] 0812 08:04:04.209068 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:04.209129 (+    61us) service_pool.cc:224] Handling call
0812 08:04:05.281922 (+1072793us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:04.785198 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:04.785356 (+   158us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:04.785361 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:04:04.785362 (+     1us) tablet.cc:662] Decoding operations
0812 08:04:04.856429 (+ 71067us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:04.856442 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:04.856449 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:04.914647 (+ 58198us) tablet.cc:701] Row locks acquired
0812 08:04:04.914649 (+     2us) write_op.cc:260] PREPARE: finished
0812 08:04:04.914709 (+    60us) write_op.cc:270] Start()
0812 08:04:04.914749 (+    40us) write_op.cc:276] Timestamp: P: 1786521844914697 usec, L: 0
0812 08:04:04.914752 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:04:04.915064 (+   312us) log.cc:844] Serialized 485717 byte log entry
0812 08:04:04.917278 (+  2214us) op_driver.cc:464] REPLICATION: finished
0812 08:04:04.917353 (+    75us) write_op.cc:301] APPLY: starting
0812 08:04:04.917366 (+    13us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:04.923299 (+  5933us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:04.923316 (+    17us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:04.938956 (+ 15640us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x55a85171e38c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:04:05.069036 (+130080us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x55a85171e38c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:04:05.273310 (+204274us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:05.275571 (+  2261us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:05.275578 (+     7us) write_op.cc:312] APPLY: finished
0812 08:04:05.280301 (+  4723us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:05.280410 (+   109us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:05.281550 (+  1140us) write_op.cc:454] Released schema lock
0812 08:04:05.281801 (+   251us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":43,"num_ops":5000,"prepare.queue_time_us":575412,"prepare.run_cpu_time_us":48453,"prepare.run_wall_time_us":129947,"replication_time_us":2502,"spinlock_wait_cycles":58781568}]]}
W20260812 08:04:05.550127 25622 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:33269, user_credentials={real_user=slave}} blocked reactor thread for 120951us
I20260812 08:04:05.578145 25662 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 59% done.
I20260812 08:04:05.656913 25650 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 59% done.
I20260812 08:04:05.720475 25649 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 59% done.
I20260812 08:04:05.743142 25673 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=52, attempt_no=0}) took 1090 ms. Trace:
I20260812 08:04:05.743350 25673 rpcz_store.cc:276] 0812 08:04:04.652580 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:04.652636 (+    56us) service_pool.cc:224] Handling call
0812 08:04:05.743122 (+1090486us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:05.161023 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:05.161397 (+   374us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:05.161402 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:04:05.161404 (+     2us) tablet.cc:662] Decoding operations
0812 08:04:05.211584 (+ 50180us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:05.211596 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:05.211602 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:05.254089 (+ 42487us) tablet.cc:701] Row locks acquired
0812 08:04:05.254093 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:04:05.254192 (+    99us) write_op.cc:270] Start()
0812 08:04:05.254238 (+    46us) write_op.cc:276] Timestamp: P: 1786521845254170 usec, L: 0
0812 08:04:05.254241 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:04:05.254594 (+   353us) log.cc:844] Serialized 485388 byte log entry
0812 08:04:05.261551 (+  6957us) op_driver.cc:464] REPLICATION: finished
0812 08:04:05.321564 (+ 60013us) write_op.cc:301] APPLY: starting
0812 08:04:05.321581 (+    17us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:05.327554 (+  5973us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:05.327579 (+    25us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:05.593069 (+265490us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x55a85171e38c. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facfcdadce7 00007facfcdadd46 00007facfc09e2c8 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d 00007facf8700326
0812 08:04:05.722025 (+128956us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:05.724241 (+  2216us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:05.724248 (+     7us) write_op.cc:312] APPLY: finished
0812 08:04:05.741614 (+ 17366us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:05.741699 (+    85us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:05.742741 (+  1042us) write_op.cc:454] Released schema lock
0812 08:04:05.743012 (+   271us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":59958,"mutex_wait_us":11235,"num_ops":5000,"prepare.queue_time_us":507705,"prepare.run_cpu_time_us":49354,"prepare.run_wall_time_us":100758,"replication_time_us":7270,"spinlock_wait_cycles":27689344}]]}
I20260812 08:04:05.876821 25657 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 69% done.
I20260812 08:04:05.890931 25645 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 69% done.
I20260812 08:04:05.947160 25655 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 59% done.
I20260812 08:04:06.002236 25661 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 59% done.
I20260812 08:04:06.302429 25647 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 69% done.
I20260812 08:04:06.344751 25659 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 69% done.
I20260812 08:04:06.352387 25662 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 69% done.
I20260812 08:04:06.415326 25649 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 69% done.
I20260812 08:04:06.473795 25653 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 69% done.
I20260812 08:04:06.575519 25655 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 69% done.
I20260812 08:04:06.578464 25650 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 69% done.
I20260812 08:04:06.765108 25661 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 69% done.
I20260812 08:04:06.864982 25657 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 79% done.
I20260812 08:04:06.965588 25645 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 79% done.
I20260812 08:04:07.021993 25647 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 79% done.
I20260812 08:04:07.111940 25679 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=65, attempt_no=0}) took 1114 ms. Trace:
I20260812 08:04:07.114526 25679 rpcz_store.cc:276] 0812 08:04:05.997616 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:05.997677 (+    61us) service_pool.cc:224] Handling call
0812 08:04:07.111914 (+1114237us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:06.737682 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:06.737854 (+   172us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:06.737860 (+     6us) write_op.cc:435] Acquired schema lock
0812 08:04:06.737861 (+     1us) tablet.cc:662] Decoding operations
0812 08:04:06.783211 (+ 45350us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:06.783226 (+    15us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:06.783234 (+     8us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:06.852158 (+ 68924us) tablet.cc:701] Row locks acquired
0812 08:04:06.852162 (+     4us) write_op.cc:260] PREPARE: finished
0812 08:04:06.852244 (+    82us) write_op.cc:270] Start()
0812 08:04:06.852288 (+    44us) write_op.cc:276] Timestamp: P: 1786521846852224 usec, L: 0
0812 08:04:06.852291 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:04:06.852627 (+   336us) log.cc:844] Serialized 485757 byte log entry
0812 08:04:06.854884 (+  2257us) op_driver.cc:464] REPLICATION: finished
0812 08:04:06.854973 (+    89us) write_op.cc:301] APPLY: starting
0812 08:04:06.854988 (+    15us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:06.860764 (+  5776us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:06.860787 (+    23us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:07.080211 (+219424us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:07.098381 (+ 18170us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:07.098391 (+    10us) write_op.cc:312] APPLY: finished
0812 08:04:07.103068 (+  4677us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:07.110497 (+  7429us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:07.111520 (+  1023us) write_op.cc:454] Released schema lock
0812 08:04:07.111780 (+   260us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":49,"num_ops":5000,"prepare.queue_time_us":739283,"prepare.run_cpu_time_us":56400,"prepare.run_wall_time_us":115031,"replication_time_us":2564,"spinlock_wait_cycles":24576}]]}
I20260812 08:04:07.194401 25673 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=66, attempt_no=0}) took 1186 ms. Trace:
I20260812 08:04:07.194713 25673 rpcz_store.cc:276] 0812 08:04:06.008318 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:06.008370 (+    52us) service_pool.cc:224] Handling call
0812 08:04:07.194376 (+1186006us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:06.852754 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:06.853004 (+   250us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:06.853009 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:04:06.853011 (+     2us) tablet.cc:662] Decoding operations
0812 08:04:06.900941 (+ 47930us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:06.900953 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:06.900959 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:06.920643 (+ 19684us) tablet.cc:701] Row locks acquired
0812 08:04:06.920646 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:04:06.920711 (+    65us) write_op.cc:270] Start()
0812 08:04:06.920740 (+    29us) write_op.cc:276] Timestamp: P: 1786521846920697 usec, L: 0
0812 08:04:06.920743 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:04:06.921068 (+   325us) log.cc:844] Serialized 487431 byte log entry
0812 08:04:06.923220 (+  2152us) op_driver.cc:464] REPLICATION: finished
0812 08:04:06.923296 (+    76us) write_op.cc:301] APPLY: starting
0812 08:04:06.923309 (+    13us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:06.929204 (+  5895us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:06.929224 (+    20us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:07.154287 (+225063us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:07.156489 (+  2202us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:07.156496 (+     7us) write_op.cc:312] APPLY: finished
0812 08:04:07.177302 (+ 20806us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:07.193012 (+ 15710us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:07.193994 (+   982us) write_op.cc:454] Released schema lock
0812 08:04:07.194242 (+   248us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":43,"num_ops":5000,"prepare.queue_time_us":843729,"prepare.run_cpu_time_us":48218,"prepare.run_wall_time_us":68394,"replication_time_us":2454,"spinlock_wait_cycles":183424}]]}
I20260812 08:04:07.218004 25659 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 79% done.
I20260812 08:04:07.253456 25649 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 79% done.
I20260812 08:04:07.274966 25672 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=67, attempt_no=0}) took 1070 ms. Trace:
I20260812 08:04:07.280423 25672 rpcz_store.cc:276] 0812 08:04:06.204749 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:06.204815 (+    66us) service_pool.cc:224] Handling call
0812 08:04:07.274937 (+1070122us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:06.921190 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:06.921339 (+   149us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:06.921343 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:04:06.921345 (+     2us) tablet.cc:662] Decoding operations
0812 08:04:06.962431 (+ 41086us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:06.962443 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:06.962449 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:06.982565 (+ 20116us) tablet.cc:701] Row locks acquired
0812 08:04:06.982568 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:04:06.982658 (+    90us) write_op.cc:270] Start()
0812 08:04:06.982705 (+    47us) write_op.cc:276] Timestamp: P: 1786521846982636 usec, L: 0
0812 08:04:06.982707 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:04:06.983040 (+   333us) log.cc:844] Serialized 486480 byte log entry
0812 08:04:06.985249 (+  2209us) op_driver.cc:464] REPLICATION: finished
0812 08:04:06.985352 (+   103us) write_op.cc:301] APPLY: starting
0812 08:04:06.985371 (+    19us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:06.990676 (+  5305us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:06.990700 (+    24us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:07.227216 (+236516us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:07.239587 (+ 12371us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:07.239595 (+     8us) write_op.cc:312] APPLY: finished
0812 08:04:07.244220 (+  4625us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:07.273401 (+ 29181us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:07.274555 (+  1154us) write_op.cc:454] Released schema lock
0812 08:04:07.274797 (+   242us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":33,"mutex_wait_us":3671,"num_ops":5000,"prepare.queue_time_us":715601,"prepare.run_cpu_time_us":49119,"prepare.run_wall_time_us":61945,"replication_time_us":2515,"spinlock_wait_cycles":51328}]]}
I20260812 08:04:07.281602 25687 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=68, attempt_no=0}) took 1008 ms. Trace:
I20260812 08:04:07.281755 25687 rpcz_store.cc:276] 0812 08:04:06.272834 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:06.272922 (+    88us) service_pool.cc:224] Handling call
0812 08:04:07.281585 (+1008663us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:06.983185 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:06.983343 (+   158us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:06.983348 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:04:06.983350 (+     2us) tablet.cc:662] Decoding operations
0812 08:04:07.012106 (+ 28756us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:07.012120 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:07.012126 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:07.032348 (+ 20222us) tablet.cc:701] Row locks acquired
0812 08:04:07.032351 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:04:07.032429 (+    78us) write_op.cc:270] Start()
0812 08:04:07.032476 (+    47us) write_op.cc:276] Timestamp: P: 1786521847032410 usec, L: 0
0812 08:04:07.032479 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:04:07.032814 (+   335us) log.cc:844] Serialized 487114 byte log entry
0812 08:04:07.034984 (+  2170us) op_driver.cc:464] REPLICATION: finished
0812 08:04:07.040059 (+  5075us) write_op.cc:301] APPLY: starting
0812 08:04:07.040075 (+    16us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:07.046000 (+  5925us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:07.046022 (+    22us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:07.273270 (+227248us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:07.275465 (+  2195us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:07.275472 (+     7us) write_op.cc:312] APPLY: finished
0812 08:04:07.280096 (+  4624us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:07.280220 (+   124us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:07.281227 (+  1007us) write_op.cc:454] Released schema lock
0812 08:04:07.281475 (+   248us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":5023,"mutex_wait_us":10137,"num_ops":5000,"prepare.queue_time_us":709481,"prepare.run_cpu_time_us":49235,"prepare.run_wall_time_us":49713,"replication_time_us":2482,"spinlock_wait_cycles":52864,"thread_start_us":86,"threads_started":1}]]}
W20260812 08:04:07.590348 25622 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:33269, user_credentials={real_user=slave}} blocked reactor thread for 79231.8us
I20260812 08:04:07.635535 25653 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 79% done.
I20260812 08:04:07.636525 25662 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 79% done.
I20260812 08:04:07.674520 25687 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=71, attempt_no=0}) took 1063 ms. Trace:
I20260812 08:04:07.674665 25687 rpcz_store.cc:276] 0812 08:04:06.610848 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:06.610912 (+    64us) service_pool.cc:224] Handling call
0812 08:04:07.674494 (+1063582us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:07.249653 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:07.249818 (+   165us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:07.249822 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:04:07.249824 (+     2us) tablet.cc:662] Decoding operations
0812 08:04:07.279493 (+ 29669us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:07.279507 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:07.279513 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:07.300402 (+ 20889us) tablet.cc:701] Row locks acquired
0812 08:04:07.300405 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:04:07.300490 (+    85us) write_op.cc:270] Start()
0812 08:04:07.300538 (+    48us) write_op.cc:276] Timestamp: P: 1786521847300470 usec, L: 0
0812 08:04:07.300540 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:04:07.300950 (+   410us) log.cc:844] Serialized 485670 byte log entry
0812 08:04:07.303260 (+  2310us) op_driver.cc:464] REPLICATION: finished
0812 08:04:07.303344 (+    84us) write_op.cc:301] APPLY: starting
0812 08:04:07.303359 (+    15us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:07.309108 (+  5749us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:07.309131 (+    23us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:07.414502 (+105371us) spinlock_profiling.cc:243] Waited 9.33 ms on lock 0x55a8516e0218. stack: 00007facf86dcd35 00007facf86dd325 00007facf7bc582d 000055a83cfe6504 000055a83cfe68fd 000055a83cfeb4d0 00007facf3c2fbdd 00007facfc1cfe3b 00007facfc0998df 00007facfc09e60a 00007facfc09db7b 00007facfc16bace 00007facfc1584b7 00007facfc1579ca 00007facfc15a95b 000055a83d00075d
0812 08:04:07.641270 (+226768us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:07.643170 (+  1900us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:07.643179 (+     9us) write_op.cc:312] APPLY: finished
0812 08:04:07.663753 (+ 20574us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:07.672931 (+  9178us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:07.674109 (+  1178us) write_op.cc:454] Released schema lock
0812 08:04:07.674363 (+   254us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":48,"mutex_wait_us":17691,"num_ops":5000,"prepare.queue_time_us":638010,"prepare.run_cpu_time_us":51065,"prepare.run_wall_time_us":51386,"replication_time_us":2691,"spinlock_wait_cycles":21525248}]]}
I20260812 08:04:07.789666 25650 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 79% done.
I20260812 08:04:07.939754 25661 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 79% done.
I20260812 08:04:07.942488 25657 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 89% done.
I20260812 08:04:08.101812 25655 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 79% done.
I20260812 08:04:08.208267 25653 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 89% done.
I20260812 08:04:08.208673 25645 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 89% done.
I20260812 08:04:08.310700 25659 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 89% done.
I20260812 08:04:08.354033 25649 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 89% done.
I20260812 08:04:08.426609 25647 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 89% done.
I20260812 08:04:08.603340 25679 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=78, attempt_no=0}) took 1223 ms. Trace:
I20260812 08:04:08.628077 25673 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=79, attempt_no=0}) took 1005 ms. Trace:
I20260812 08:04:08.632959 25673 rpcz_store.cc:276] 0812 08:04:07.623115 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:07.623189 (+    74us) service_pool.cc:224] Handling call
0812 08:04:08.628051 (+1004862us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:08.211209 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:08.211373 (+   164us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:08.211378 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:04:08.211381 (+     3us) tablet.cc:662] Decoding operations
0812 08:04:08.240826 (+ 29445us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:08.240839 (+    13us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:08.240845 (+     6us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:08.272369 (+ 31524us) tablet.cc:701] Row locks acquired
0812 08:04:08.272372 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:04:08.272461 (+    89us) write_op.cc:270] Start()
0812 08:04:08.272508 (+    47us) write_op.cc:276] Timestamp: P: 1786521848272435 usec, L: 0
0812 08:04:08.272511 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:04:08.272859 (+   348us) log.cc:844] Serialized 485916 byte log entry
0812 08:04:08.275252 (+  2393us) op_driver.cc:464] REPLICATION: finished
0812 08:04:08.275350 (+    98us) write_op.cc:301] APPLY: starting
0812 08:04:08.275366 (+    16us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:08.281095 (+  5729us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:08.281120 (+    25us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:08.603434 (+322314us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:08.605600 (+  2166us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:08.605606 (+     6us) write_op.cc:312] APPLY: finished
0812 08:04:08.626344 (+ 20738us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:08.626645 (+   301us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:08.627671 (+  1026us) write_op.cc:454] Released schema lock
0812 08:04:08.627927 (+   256us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":39,"mutex_wait_us":15985,"num_ops":5000,"prepare.queue_time_us":587318,"prepare.run_cpu_time_us":61613,"prepare.run_wall_time_us":61796,"replication_time_us":2709,"spinlock_wait_cycles":96640}]]}
I20260812 08:04:08.632941 25679 rpcz_store.cc:276] 0812 08:04:07.379919 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:07.379967 (+    48us) service_pool.cc:224] Handling call
0812 08:04:08.603316 (+1223349us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:08.024146 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:08.024321 (+   175us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:08.024326 (+     5us) write_op.cc:435] Acquired schema lock
0812 08:04:08.024328 (+     2us) tablet.cc:662] Decoding operations
0812 08:04:08.159091 (+134763us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:08.159105 (+    14us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:08.159112 (+     7us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:08.210595 (+ 51483us) tablet.cc:701] Row locks acquired
0812 08:04:08.210598 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:04:08.210682 (+    84us) write_op.cc:270] Start()
0812 08:04:08.210729 (+    47us) write_op.cc:276] Timestamp: P: 1786521848210661 usec, L: 0
0812 08:04:08.210731 (+     2us) op_driver.cc:348] REPLICATION: starting
0812 08:04:08.211081 (+   350us) log.cc:844] Serialized 485829 byte log entry
0812 08:04:08.213484 (+  2403us) op_driver.cc:464] REPLICATION: finished
0812 08:04:08.213590 (+   106us) write_op.cc:301] APPLY: starting
0812 08:04:08.213612 (+    22us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:08.219363 (+  5751us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:08.219385 (+    22us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:08.586749 (+367364us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:08.588981 (+  2232us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:08.588989 (+     8us) write_op.cc:312] APPLY: finished
0812 08:04:08.601682 (+ 12693us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:08.601786 (+   104us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:08.602930 (+  1144us) write_op.cc:454] Released schema lock
0812 08:04:08.603202 (+   272us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":33,"mutex_wait_us":15993,"num_ops":5000,"prepare.queue_time_us":643492,"prepare.run_cpu_time_us":53781,"prepare.run_wall_time_us":187021,"replication_time_us":2702,"spinlock_wait_cycles":41600}]]}
W20260812 08:04:08.790723 25622 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:33269, user_credentials={real_user=slave}} blocked reactor thread for 60049.1us
I20260812 08:04:08.852471 25650 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 89% done.
W20260812 08:04:08.889171 25622 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:33269, user_credentials={real_user=slave}} blocked reactor thread for 84141us
I20260812 08:04:08.941323 25655 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 89% done.
W20260812 08:04:08.986467 25622 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:33269, user_credentials={real_user=slave}} blocked reactor thread for 95976.2us
I20260812 08:04:09.129369 25657 full_stack-insert-scan-test.cc:274] Insertion thread 7 of 10 is 99% done.
I20260812 08:04:09.129561 25657 full_stack-insert-scan-test.cc:280] Insertion thread 7 of  10 is 100% done.
I20260812 08:04:09.205442 25662 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 89% done.
I20260812 08:04:09.268606 25659 full_stack-insert-scan-test.cc:274] Insertion thread 8 of 10 is 99% done.
I20260812 08:04:09.268808 25659 full_stack-insert-scan-test.cc:280] Insertion thread 8 of  10 is 100% done.
I20260812 08:04:09.383458 25679 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46986 (ReqId={client: 8040e0c2063e4a67a847bd24133b4c26, seq_no=85, attempt_no=0}) took 1379 ms. Trace:
I20260812 08:04:09.383582 25679 rpcz_store.cc:276] 0812 08:04:08.004488 (+     0us) service_pool.cc:167] Inserting onto call queue
0812 08:04:08.004542 (+    54us) service_pool.cc:224] Handling call
0812 08:04:09.383435 (+1378893us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0812 08:04:08.609147 (+     0us) write_op.cc:183] PREPARE: starting on tablet 8dd491968e374922ad9febecae3337de
0812 08:04:08.609301 (+   154us) write_op.cc:432] Acquiring schema lock in shared mode
0812 08:04:08.609305 (+     4us) write_op.cc:435] Acquired schema lock
0812 08:04:08.609307 (+     2us) tablet.cc:662] Decoding operations
0812 08:04:08.649615 (+ 40308us) write_op.cc:620] Acquiring the partition lock for write op
0812 08:04:08.649627 (+    12us) write_op.cc:641] Partition lock acquired for write op
0812 08:04:08.649632 (+     5us) tablet.cc:685] Acquiring locks for 5000 operations
0812 08:04:08.757106 (+107474us) tablet.cc:701] Row locks acquired
0812 08:04:08.757109 (+     3us) write_op.cc:260] PREPARE: finished
0812 08:04:08.757197 (+    88us) write_op.cc:270] Start()
0812 08:04:08.757243 (+    46us) write_op.cc:276] Timestamp: P: 1786521848757172 usec, L: 0
0812 08:04:08.757246 (+     3us) op_driver.cc:348] REPLICATION: starting
0812 08:04:08.757625 (+   379us) log.cc:844] Serialized 486498 byte log entry
0812 08:04:08.759886 (+  2261us) op_driver.cc:464] REPLICATION: finished
0812 08:04:08.759964 (+    78us) write_op.cc:301] APPLY: starting
0812 08:04:08.759982 (+    18us) tablet.cc:1367] starting BulkCheckPresence
0812 08:04:08.765977 (+  5995us) tablet.cc:1370] finished BulkCheckPresence
0812 08:04:08.766002 (+    25us) tablet.cc:1372] starting ApplyRowOperation cycle
0812 08:04:09.319448 (+553446us) tablet.cc:1383] finished ApplyRowOperation cycle
0812 08:04:09.337624 (+ 18176us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0812 08:04:09.337632 (+     8us) write_op.cc:312] APPLY: finished
0812 08:04:09.356814 (+ 19182us) log.cc:844] Serialized 30022 byte log entry
0812 08:04:09.373051 (+ 16237us) write_op.cc:489] Releasing partition, row and schema locks
0812 08:04:09.383068 (+ 10017us) write_op.cc:454] Released schema lock
0812 08:04:09.383321 (+   253us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":30,"num_ops":5000,"prepare.queue_time_us":603841,"prepare.run_cpu_time_us":59529,"prepare.run_wall_time_us":148568,"replication_time_us":2613,"spinlock_wait_cycles":62976}]]}
I20260812 08:04:09.534050 25650 full_stack-insert-scan-test.cc:274] Insertion thread 4 of 10 is 99% done.
I20260812 08:04:09.534252 25650 full_stack-insert-scan-test.cc:280] Insertion thread 4 of  10 is 100% done.
W20260812 08:04:09.722638 25622 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:33269, user_credentials={real_user=slave}} blocked reactor thread for 51688.9us
I20260812 08:04:09.729983 25647 full_stack-insert-scan-test.cc:274] Insertion thread 2 of 10 is 99% done.
I20260812 08:04:09.737078 25647 full_stack-insert-scan-test.cc:280] Insertion thread 2 of  10 is 100% done.
I20260812 08:04:09.775012 25661 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 89% done.
I20260812 08:04:09.792434 25645 full_stack-insert-scan-test.cc:274] Insertion thread 1 of 10 is 99% done.
I20260812 08:04:09.792642 25645 full_stack-insert-scan-test.cc:280] Insertion thread 1 of  10 is 100% done.
I20260812 08:04:09.914055 25649 full_stack-insert-scan-test.cc:274] Insertion thread 3 of 10 is 99% done.
I20260812 08:04:09.914237 25649 full_stack-insert-scan-test.cc:280] Insertion thread 3 of  10 is 100% done.
I20260812 08:04:10.026980 25662 full_stack-insert-scan-test.cc:274] Insertion thread 10 of 10 is 99% done.
I20260812 08:04:10.027168 25662 full_stack-insert-scan-test.cc:280] Insertion thread 10 of  10 is 100% done.
W20260812 08:04:10.031131 25622 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.23.147.193:33269, user_credentials={real_user=slave}} blocked reactor thread for 65750.5us
I20260812 08:04:10.081329 25653 full_stack-insert-scan-test.cc:274] Insertion thread 5 of 10 is 99% done.
I20260812 08:04:10.081545 25653 full_stack-insert-scan-test.cc:280] Insertion thread 5 of  10 is 100% done.
I20260812 08:04:10.123147 25655 full_stack-insert-scan-test.cc:274] Insertion thread 6 of 10 is 99% done.
I20260812 08:04:10.123389 25655 full_stack-insert-scan-test.cc:280] Insertion thread 6 of  10 is 100% done.
I20260812 08:04:10.254235 25661 full_stack-insert-scan-test.cc:274] Insertion thread 9 of 10 is 99% done.
I20260812 08:04:10.268939 25661 full_stack-insert-scan-test.cc:280] Insertion thread 9 of  10 is 100% done.
I20260812 08:04:10.693408 24143 full_stack-insert-scan-test.cc:333] Time spent concurrent inserts (500000 rows, 10 threads): real 10.610s	user 0.000s	sys 0.000s
I20260812 08:04:24.945202 24143 full_stack-insert-scan-test.cc:346] Doing test scans on table of 500000 rows.
I20260812 08:04:25.060535 24143 full_stack-insert-scan-test.cc:453] Time spent empty projection, 0 col: real 0.047s	user 0.001s	sys 0.000s
I20260812 08:04:25.204540 24143 full_stack-insert-scan-test.cc:453] Time spent key scan, 1 col: real 0.113s	user 0.002s	sys 0.000s
I20260812 08:04:25.753429 24143 full_stack-insert-scan-test.cc:453] Time spent full schema scan, no cache, 10 col: real 0.531s	user 0.034s	sys 0.000s
I20260812 08:04:26.388978 24143 full_stack-insert-scan-test.cc:453] Time spent fault-tolerant full schema scan, no cache, 10 col: real 0.620s	user 0.035s	sys 0.000s
I20260812 08:04:26.937654 24143 full_stack-insert-scan-test.cc:453] Time spent full schema scan, 10 col: real 0.534s	user 0.034s	sys 0.000s
I20260812 08:04:27.141754 24143 full_stack-insert-scan-test.cc:453] Time spent String projection, 1 col: real 0.193s	user 0.027s	sys 0.000s
I20260812 08:04:27.353684 24143 full_stack-insert-scan-test.cc:453] Time spent Int32 projection, 4 col: real 0.183s	user 0.003s	sys 0.000s
I20260812 08:04:27.581017 24143 full_stack-insert-scan-test.cc:453] Time spent Int64 projection, 4 col: real 0.212s	user 0.004s	sys 0.000s
I20260812 08:04:27.582221 24143 tablet_server.cc:179] TabletServer@127.23.147.193:0 shutting down...
I20260812 08:04:27.585937 24143 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260812 08:04:27.586140 24143 tablet_replica.cc:333] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9: stopping tablet replica
I20260812 08:04:27.586264 24143 raft_consensus.cc:2243] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9 [term 1 LEADER]: Raft consensus shutting down.
I20260812 08:04:27.586418 24143 raft_consensus.cc:2272] T 8dd491968e374922ad9febecae3337de P 7b4cfff33abc4a07bdaff2da21e71dc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 08:04:27.600695 24143 tablet_server.cc:196] TabletServer@127.23.147.193:0 shutdown complete.
I20260812 08:04:27.603323 24143 master.cc:562] Master@127.23.147.254:35701 shutting down...
I20260812 08:04:27.606794 24143 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01 [term 1 LEADER]: Raft consensus shutting down.
I20260812 08:04:27.606989 24143 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 08:04:27.607053 24143 tablet_replica.cc:333] T 00000000000000000000000000000000 P 1ae203e7ccd945a394e2df800c38ea01: stopping tablet replica
I20260812 08:04:27.620223 24143 master.cc:584] Master@127.23.147.254:35701 shutdown complete.
[       OK ] EvictionPolicyTypes/FullStackInsertScanTestWithEvictionPolicy.WithDiskStressTest/1 (27708 ms)
[----------] 4 tests from EvictionPolicyTypes/FullStackInsertScanTestWithEvictionPolicy (85466 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (85466 ms total)
[  PASSED  ] 4 tests.
I20260812 08:04:27.639286 24143 logging.cc:436] LogThrottler /workspace/apache/dev/local/kudu/src/kudu/util/maintenance_manager.cc:346: suppressed but not reported on 243 messages since previous log ~85 seconds ago
