Unverified Commit 38a6f191 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

[turbopack] Add a simple tool to inspect serialized SST files and report on their contents (#89310)

Example run
```
$ cargo run -p turbo-persistence --bin sst_inspect --  ~/projects/front/apps/vercel-site/.next/cache/turbopack/v16.2.0-canary.8-207-g8e5467e841e6e-dirty
...
Analyzing 169 SST files in /Users/lukesandberg/projects/front/apps/vercel-site/.next/cache/turbopack/v16.2.0-canary.8-207-g8e5467e841e6e-dirty

═══════════════════════════════════════════════════════════════════════════════
Family 0 (Infra):
═══════════════════════════════════════════════════════════════════════════════
  SST files: 1, Total entries: 2
  Total file size: 44 B

  Averages:
    File size: 44 B
    Keys per file: 2
    Keys per key block: 2.0

  Per-file Overhead (total): 8 B (18.2% of total file size)
    Key compression dictionaries: 0 B
      Average per file: 0 B
    Block directories: 8 B
      Average per file: 8 B

  Block Statistics:
      Index blocks: 1 blocks (uncompressed), 3 B
      Key blocks: 1 blocks (uncompressed), 25 B
      Value blocks: none

  Entry Type Histogram:
    type   9:            1 ( 50.0%) │█████████████████████████│ inline 1 bytes
    type  12:            1 ( 50.0%) │█████████████████████████│ inline 4 bytes

  Value Storage:
    Inline: 2 entries, 5 B total

═══════════════════════════════════════════════════════════════════════════════
Family 1 (TaskMeta):
═══════════════════════════════════════════════════════════════════════════════
  SST files: 36, Total entries: 8,840,774
  Total file size: 670.46 MB

  Averages:
    File size: 18.62 MB
    Keys per file: 245,577
    Keys per key block: 1358.2

  Per-file Overhead (total): 64.09 KB (0.0% of total file size)
    Key compression dictionaries: 0 B
      Average per file: 0 B
    Block directories: 64.09 KB
      Average per file: 1.78 KB

  Block Statistics:
      Index blocks: 36 blocks (uncompressed), 63.32 KB
      Key blocks: 6,509 blocks (6,478 compressed, 31 uncompressed)
          stored: 115.52 MB, actual: 134.92 MB (14% savings)
      Value blocks: 9,862 blocks (9,046 compressed, 816 uncompressed)
          stored: 554.75 MB, actual: 673.49 MB (18% savings)

  Entry Type Histogram:
    type   0:    8,840,005 (100.0%) │████████████████████████████████████████│ small value (in value block)
    type   3:          769 (  0.0%) ││ medium value

  Value Storage:
    Small (value block refs): 8,840,005 entries
    Medium (dedicated blocks): 769 entries

═══════════════════════════════════════════════════════════════════════════════
Family 2 (TaskData):
═══════════════════════════════════════════════════════════════════════════════
  SST files: 96, Total entries: 8,840,774
  Total file size: 2.60 GB

  Averages:
    File size: 27.76 MB
    Keys per file: 92,091
    Keys per key block: 1354.3

  Per-file Overhead (total): 253.12 KB (0.0% of total file size)
    Key compression dictionaries: 0 B
      Average per file: 0 B
    Block directories: 253.12 KB
      Average per file: 2.64 KB

  Block Statistics:
      Index blocks: 96 blocks (uncompressed), 63.09 KB
      Key blocks: 6,528 blocks (uncompressed), 134.84 MB
      Value blocks: 58,175 blocks (57,911 compressed, 264 uncompressed)
          stored: 2.47 GB, actual: 5.82 GB (58% savings)

  Entry Type Histogram:
    type   0:    8,825,784 ( 99.8%) │████████████████████████████████████████│ small value (in value block)
    type   3:       14,990 (  0.2%) ││ medium value

  Value Storage:
    Small (value block refs): 8,825,784 entries
    Medium (dedicated blocks): 14,990 entries

═══════════════════════════════════════════════════════════════════════════════
Family 3 (TaskCache):
═══════════════════════════════════════════════════════════════════════════════
  SST files: 36, Total entries: 8,847,914
  Total file size: 327.29 MB

  Averages:
    File size: 9.09 MB
    Keys per file: 245,775
    Keys per key block: 321.5

  Per-file Overhead (total): 2.36 MB (0.7% of total file size)
    Key compression dictionaries: 2.25 MB
      Average per file: 64.00 KB
    Block directories: 107.63 KB
      Average per file: 2.99 KB

  Block Statistics:
      Index blocks: 36 blocks (uncompressed), 268.47 KB
      Key blocks: 27,517 blocks, stored: 324.57 MB, actual: 495.61 MB (35% savings)
      Value blocks: none

  Entry Type Histogram:
    type  12:    8,847,914 (100.0%) │████████████████████████████████████████│ inline 4 bytes

  Value Storage:
    Inline: 8,847,914 entries, 33.75 MB total
```
parent b6257d77
...@@ -42,6 +42,10 @@ rayon = { workspace = true } ...@@ -42,6 +42,10 @@ rayon = { workspace = true }
tempfile = { workspace = true } tempfile = { workspace = true }
turbo-tasks-malloc = { workspace = true } turbo-tasks-malloc = { workspace = true }
[[bin]]
name = "sst_inspect"
path = "src/bin/sst_inspect.rs"
[lints] [lints]
workspace = true workspace = true
......
...@@ -105,7 +105,7 @@ The hashes are sorted. ...@@ -105,7 +105,7 @@ The hashes are sorted.
- foreach entry - foreach entry
- 1 byte type - 1 byte type
- 3 bytes position in block after header - 3 bytes position in block after header
- Max block size: 16 MB - Max block size: 16 KB
A Key block contains n keys, which specify n key value pairs. A Key block contains n keys, which specify n key value pairs.
...@@ -147,6 +147,9 @@ Depending on the `type` field entry has a different format: ...@@ -147,6 +147,9 @@ Depending on the `type` field entry has a different format:
The entries are sorted by key hash and key. The entries are sorted by key hash and key.
Future:
* Some tables have fixed sized keys with consistent values (4 byte task ids). We could optimize key block representation in this case by skipping offset tables.
#### Value Block #### Value Block
- no header, all bytes are data referenced by other blocks - no header, all bytes are data referenced by other blocks
...@@ -170,6 +173,7 @@ Reading start from the current sequence number and goes downwards. ...@@ -170,6 +173,7 @@ Reading start from the current sequence number and goes downwards.
- not found -> break - not found -> break
- Key Block: find key by binary search - Key Block: find key by binary search
- found -> lookup value from value block, return - found -> lookup value from value block, return
- read value as inline, or by using the block index in the key to find the value elsewhere in the file.
- not found -> break - not found -> break
## Writing ## Writing
......
This diff is collapsed.
...@@ -13,11 +13,11 @@ mod db; ...@@ -13,11 +13,11 @@ mod db;
mod key; mod key;
mod lookup_entry; mod lookup_entry;
mod merge_iter; mod merge_iter;
mod meta_file; pub mod meta_file;
mod meta_file_builder; mod meta_file_builder;
mod parallel_scheduler; mod parallel_scheduler;
mod sst_filter; mod sst_filter;
mod static_sorted_file; pub mod static_sorted_file;
mod static_sorted_file_builder; mod static_sorted_file_builder;
mod value_block_count_tracker; mod value_block_count_tracker;
mod value_buf; mod value_buf;
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment