Unverified Commit 0b9fffb1 authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

Turbopack: add turbo-trace-size CLI to inspect trace file size (#99765)

### What?

Adds `turbopack/crates/turbopack-trace-size`, a CLI (`turbo-trace-size`)
that shows where the bytes in a Turbopack trace file
(`.next-profiles/trace-turbopack.bin`) go. It decodes the postcard
`TraceRow` stream (raw, gzip or zstd) and reports:

- size per row type (`Start`, `Enter`, `Exit`, `AllocationCounters`,
...)
- the components of `Start`/`Event`/`Record` rows (name, target, values,
fixed fields)
- bytes per span name, including its Enter/Exit/End/Record rows and
Events
- bytes per attribute key
- the bytes spent re-writing the same span names, targets, keys and
string values

```sh
cargo run --bin turbo-trace-size --release -- .next-profiles/trace-turbopack.bin --top 30
```

### Why?

`NEXT_TURBOPACK_TRACING` traces get large quickly. This tool shows which
events cause that, so we can decide what to slim down.

### Findings: `bench/heavy-npm-deps`, `NEXT_TURBOPACK_TRACING=1 next
build`

68.42 MiB raw, 4.66 M rows (gzip -1: 21.3 MiB, zstd -3: 21.2 MiB):

| row type           |     count |     bytes |    avg |     % |
| ------------------ | --------: | --------: | -----: | ----: |
| AllocationCounters | 2,280,168 | 45.00 MiB | 20.7 B | 65.8% |
| Exit               | 1,140,084 |  8.40 MiB |  7.7 B | 12.3% |
| Enter              | 1,140,084 |  8.40 MiB |  7.7 B | 12.3% |
| Start              |    39,481 |  5.57 MiB |  148 B |  8.1% |
| Record             |    21,913 |   804 KiB |   38 B |  1.1% |
| End                |    39,480 |   265 KiB |    7 B |  0.4% |
| MemorySample       |       244 |   2.5 KiB |   10 B |  0.0% |
| Event              |         8 |     368 B |   46 B |  0.0% |

- **AllocationCounters take 66% of the file.** `RawTraceLayer` writes
one before every `Enter` and every `Exit`, so there are exactly 2× as
many as Enter rows. Each one holds an absolute timestamp, the thread id
and 4 cumulative counters.
- **Enter/Exit take another 25%.** Each span is entered about 29 times
on average, because async spans are re-entered on every poll. `module`
(module_graph) and `process module` are the biggest, at about 6.3–6.5
MiB each.
- **Start rows are 8%**, mostly attribute values: `name` (2.6 MiB),
`lookup_path` (1 MiB), `reference_type` (0.6 MiB).
- **Repeated span names, targets and keys** come from only 147 distinct
strings. At most about 3% could be saved by writing each one once,
before the cost of referencing them. String values could save at most
another 4%.

So skipping or delta-encoding the allocation counters, and
delta-encoding Enter/Exit timestamps, would cut the size far more than
interning strings would. That is left for a follow-up.

Note: the trace was captured with the published
`@next/swc-linux-x64-gnu@16.4.0-canary.31` native binary that `pnpm
install` put in place, running with JS built from this checkout
(16.5.0-canary.1). It was not a source-built native binding.

### How?

- The crate reuses `TraceRow`/`TraceValue` from `turbopack-trace-utils`
and reads the stream incrementally. Every row byte is counted, so rows +
header add up exactly to the decompressed size.
- Rows are written from per-thread buffers, so Enter/Exit/End rows can
show up before the `Start` of their span. Like the trace server, the
tool queues those rows until the `Start` arrives and keeps id mappings
after `End`.
- `contributing/turbopack/tracing.md` now documents the tool.

### Verification

- `cargo test -p turbopack-trace-size`: unit tests check byte accounting
against postcard, gzip (multi-member) and zstd input, rows split across
reads, headerless files, an incomplete last row, invalid rows,
cross-thread ordering and reused ids.
- `cargo clippy -p turbopack-trace-size --all-targets -- -D warnings`
- Ran it on the trace above: raw, gzip and zstd copies give identical
breakdowns, and rows + header = 71,739,555 B, the exact file size.

<!-- NEXT_JS_LLM -->


<!-- fleet 4a52842b-f1a2-43cb-9779-3655caada6a0 -->
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent aaf0a16f
...@@ -10118,6 +10118,19 @@ dependencies = [ ...@@ -10118,6 +10118,19 @@ dependencies = [
"zstd", "zstd",
] ]
[[package]]
name = "turbopack-trace-size"
version = "0.1.0"
dependencies = [
"anyhow",
"clap",
"flate2",
"postcard",
"rustc-hash 2.1.1",
"turbopack-trace-utils",
"zstd",
]
[[package]] [[package]]
name = "turbopack-trace-utils" name = "turbopack-trace-utils"
version = "0.1.0" version = "0.1.0"
......
...@@ -62,5 +62,16 @@ And there are different value modes: ...@@ -62,5 +62,16 @@ And there are different value modes:
- **Deallocated Memory:** How much memory was deallocated during the span. - **Deallocated Memory:** How much memory was deallocated during the span.
- **Persistently Allocated Memory:** How much memory was allocated but not deallocated during the span. It survives the span. - **Persistently Allocated Memory:** How much memory was allocated but not deallocated during the span. It survives the span.
## Trace file size
To see what takes up the space in a trace file, use `turbo-trace-size`. It breaks the file down by row type (`Start`, `Enter`, `Exit`, `AllocationCounters`, ...), by span name, by attribute key and shows how many bytes are spent on repeated strings. Raw, gzip and zstd compressed files are supported. All sizes refer to the decompressed trace stream.
```sh
cargo run --bin turbo-trace-size --release -- /path/to/your/trace-turbopack.bin
# show more entries in the per span name and per attribute key tables
cargo run --bin turbo-trace-size --release -- /path/to/your/trace-turbopack.bin --top 100
```
[turbo-trace-viewer]: https://turbo-trace-viewer.vercel.app/ [turbo-trace-viewer]: https://turbo-trace-viewer.vercel.app/
[youtube-tutorial]: https://www.youtube.com/watch?v=PGO2szAye7A [youtube-tutorial]: https://www.youtube.com/watch?v=PGO2szAye7A
[package]
name = "turbopack-trace-size"
version = "0.1.0"
description = "Inspect which events take up the space in a Turbopack trace file"
license = "MIT"
edition = "2024"
autobenches = false
[[bin]]
name = "turbo-trace-size"
path = "src/main.rs"
bench = false
[lib]
bench = false
[dependencies]
anyhow = { workspace = true }
clap = { workspace = true, features = ["derive"] }
flate2 = { workspace = true }
postcard = { workspace = true, features = ["alloc", "use-std"] }
rustc-hash = { workspace = true }
turbopack-trace-utils = { workspace = true }
zstd = { workspace = true }
[lints]
workspace = true
This diff is collapsed.
use std::{io::Write, path::PathBuf};
use anyhow::Result;
use clap::Parser;
use turbopack_trace_size::TraceSizeAnalyzer;
/// Shows which events take up the space in a Turbopack trace file (e.g.
/// `.next-profiles/trace-turbopack.bin`). Raw, gzip and zstd compressed files
/// are supported.
#[derive(Parser)]
#[command(version)]
struct Args {
/// Path to the trace file.
path: PathBuf,
/// How many entries to show in the per span name and per field key tables.
#[arg(short = 'n', long, default_value_t = 30)]
top: usize,
}
fn main() -> Result<()> {
let args = Args::parse();
let analyzer = TraceSizeAnalyzer::analyze_file(&args.path)?;
let mut out = std::io::stdout().lock();
analyzer.write_report(&mut out, args.top)?;
out.flush()?;
Ok(())
}
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment