Unverified Commit 4b28e261 authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

Turbopack: add zstd compression flags for NEXT_TURBOPACK_TRACING (#99808)

## Summary

Adds zstd compression flags for `NEXT_TURBOPACK_TRACING`, next to `gz`
and `gz-best`:

- `zstd`: level 3. Recommended: about the same writer CPU as `gz`, about
25% smaller files.
- `zstd-fast`: level 1, the lowest-CPU option, still smaller than `gz`.
- `zstd-best`: level 19.

The trace writer drops the encoder on exit (`auto_finish`), so the zstd
frame is complete. If several compression flags are given, the last one
wins, as before. The default stays uncompressed, and the trace format is
unchanged. `zstd` was already in the dependency graph, so no new crate
is compiled.

`contributing/turbopack/tracing.md` documents the flags. It also notes
that `zstd-best` and `gz-best` can fall behind on large traces: the data
waiting to be compressed is kept in memory, and the process waits for it
on exit.

Stack (on top of #99765):

1. #99777: `no-memory` flag
2. #99778: Merge allocation counters into Enter/Exit rows
3. #99779: Delta-encode allocation counters, skip unchanged ones
4. #99780: Omit Exit+Enter pairs with (almost) no gap
5. #99808: **This PR**: zstd flags
6. #99809: Delta-encoded timestamps
7. #99781: Bump the trace header to `TRACEv1`
8. #99810: `upload-trace` accepts compressed traces

## Verification

`next build` of `bench/heavy-npm-deps` with a release binding built from
this branch:

| flag | `NEXT_TURBOPACK_TRACING=1` trace |
| --- | --- |
| none | 23.8 MB |
| `zstd-fast` | 7.2 MB |
| `zstd` | 6.1 MB |
| `zstd-best` | 5.1 MB |
| `turbo-tasks,zstd` | 27.8 MB (raw: 155.7 MB) |

- Every zstd capture passes `zstd -t`, is fully decoded by
`turbo-trace-size` and loads completely in the trace server.
- `1,gz,zstd` writes zstd, and `1,zstd,gz` writes gzip.

<!-- NEXT_JS_LLM -->


<!-- fleet 4a52842b-f1a2-43cb-9779-3655caada6a0 -->
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent e08d99ac
......@@ -4384,6 +4384,7 @@ dependencies = [
"urlencoding",
"vergen-gitcl",
"windows-sys 0.60.2",
"zstd",
]
[[package]]
......
......@@ -15,12 +15,17 @@ It supports [the following special preset values][presets]:
Alternatively, any directives syntax supported by [`tracing_subscriber::filter::EnvFilter`][directives] can be used.
Additionally, these flags can be added to the comma-separated list (e.g. `NEXT_TURBOPACK_TRACING=1,no-memory,gz`):
Additionally, these flags can be added to the comma-separated list (e.g. `NEXT_TURBOPACK_TRACING=1,no-memory,zstd`):
- **`zstd`:** Compress the trace file with zstd (level 3). Recommended: it needs about the same CPU time as `gz`, but produces files about 25% smaller.
- **`zstd-fast`:** Compress the trace file with zstd (level 1). Needs the least CPU time of all compression options, and the files are still smaller than with `gz`.
- **`zstd-best`:** Compress the trace file with zstd (level 19).
- **`gz`:** Compress the trace file with gzip (fast compression level).
- **`gz-best`:** Compress the trace file with gzip (best compression level).
- **`no-memory`:** Don't track memory. Skips the per-thread allocation counters and the process memory samples, which make up a large part of the trace file size. Allocation and memory value modes in the viewer will be empty.
When multiple compression flags are given, the last one is used. Compression runs on the background thread that writes the trace file. `zstd-best` and `gz-best` compress much slower than the other options and can fall behind on large traces (e.g. the `turbo-tasks` preset). The trace data waiting to be compressed is kept in memory, and the process waits for the remaining data to be written before it exits.
> [!WARNING]
> A normal Next.js canary/stable release only includes the info level tracing. This is the tracing level intended for user-facing tracing.
>
......
......@@ -41,6 +41,7 @@ bincode = { workspace = true }
console-subscriber = { workspace = true, optional = true }
dhat = { workspace = true, optional = true }
flate2 = { workspace = true }
zstd = { workspace = true }
futures-util = { workspace = true }
owo-colors = { workspace = true }
napi = { workspace = true }
......
......@@ -468,6 +468,8 @@ pub fn project_new<'env>(
None,
GzipFast,
GzipBest,
/// zstd with the given compression level
Zstd(i32),
}
let mut compress = Compression::None;
let mut raw_trace_options = RawTraceLayerOptions::default();
......@@ -516,6 +518,18 @@ pub fn project_new<'env>(
compress = Compression::GzipBest;
return None;
}
"zstd" => {
compress = Compression::Zstd(3);
return None;
}
"zstd-fast" => {
compress = Compression::Zstd(1);
return None;
}
"zstd-best" => {
compress = Compression::Zstd(19);
return None;
}
"no-memory" => {
raw_trace_options.memory = false;
return None;
......@@ -562,6 +576,14 @@ pub fn project_new<'env>(
let trace_writer = GzEncoder::new(trace_writer, flate2::Compression::best());
TraceWriter::new(trace_writer)
}
Compression::Zstd(level) => {
let trace_writer = std::fs::File::create(trace_file.clone()).unwrap();
// `auto_finish` completes the zstd frame when the trace writer drops it on exit.
let trace_writer = zstd::Encoder::new(trace_writer, level)
.unwrap()
.auto_finish();
TraceWriter::new(trace_writer)
}
};
let subscriber =
subscriber.with(RawTraceLayer::with_options(trace_writer, raw_trace_options));
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment