Replace lzzzz with lz4_flex (#98799)
## What? Replace the direct `lzzzz` dependency used by Turbopack persistence with `lz4_flex` 0.14. The migration covers persisted blocks, the SST inspection utility, and optional compressed-size diagnostics in the task backend. ## Why? This removes the direct C/liblz4-backed codec dependency from these paths in favor of a Rust implementation. Persistence caches are scoped to the Next.js and Turbopack version, so this does not require a compatibility path for caches written by earlier versions. ## How? Persistence continues to use raw LZ4 blocks and the existing compression selection and checksum layout. A large thread-local `CompressTable` is reused across blocks. Starting with the large table costs about 8 KiB more per compression thread than the small table, but improves compression speed and produces LZ4 streams that are faster for `lz4_flex` to decode. The compressor retains the existing caller-owned buffer API. It clears and reserves the output vector, lets `lz4_flex` write directly into the uninitialized backing allocation, and exposes only the initialized byte count returned by the compressor. This avoids both per-call output allocations and pre-zeroing the maximum output size. Zstandard behavior is unchanged. ### Benchmark vs `lzzzz` The final PR configuration was benchmarked against `lzzzz` 2.0.0 using the repository-pinned Rust nightly. The benchmark used source-code blocks sampled from this repository, pinned each process to one physical CPU, measured thread CPU time, and alternated codec order within every paired round. The table reports the median across four physical cores and 20 paired rounds per process (480 paired rounds total). Each decoder read the output produced by its own encoder, matching how a same-version persistence cache is written and later read. | Source corpus | Block size | Compression | Decompression | Compressed size | | --- | ---: | ---: | ---: | ---: | | Mixed | 12 KiB | **7.1% faster** | 9.9% slower | 0.08% larger | | Rust | 12 KiB | **5.5% faster** | 10.4% slower | 0.09% larger | | TypeScript | 12 KiB | **7.4% faster** | 7.4% slower | 0.23% larger | | Mixed | 16 KiB | **5.9% faster** | 8.8% slower | 0.17% smaller | | Rust | 16 KiB | **4.6% faster** | 10.0% slower | 0.12% smaller | | TypeScript | 16 KiB | **6.7% faster** | 6.1% slower | 0.02% smaller | For these 12–16 KiB source blocks, `lz4_flex` compression was **4.6–7.4% faster**, decompression was **6.1–10.4% slower**, and compressed size changed by **-0.17% to +0.23%**. The benchmark was stable enough to support those codec-level ranges: wall time divided by thread CPU time was 1.000073 at the median and 1.001883 at p99. The median difference between the two codec orders was 0.38 percentage points for compression and 1.45 points for decompression. A separate decoder-only comparison fed identical `lzzzz`-encoded source blocks to both decoders; there, `lz4_flex` was 17–21% slower. The smaller regression in the table above comes from the large Flex compression table producing a different valid LZ4 token stream. The PR already uses `lz4_flex`'s supported unsafe pointer implementation with validation; a local experiment that removed validation recovered only 5–6% and remained slower than liblz4, so this PR does not weaken decoder safety. These are codec microbenchmarks, not an end-to-end Turbopack speedup claim. ## Verification - `cargo fmt --all -- --check` - `cargo clippy -p turbo-persistence --all-targets -- -D warnings` - `cargo test -p turbo-persistence -- --test-threads=1` (110 passed) - `cargo check -p turbo-tasks-backend --features print_cache_item_size_with_compressed` - Pinned-nightly paired codec benchmark: 24 processes / 480 paired rounds <!-- NEXT_JS_LLM --> <!-- fleet 8f69f876-cb41-4fa2-ab60-dc8ccea88055 --> --------- Co-authored-by:vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by:
Benjamin Woodruff <180404+bgw@users.noreply.github.com> Co-authored-by:
Tobias Koppers <1365881+sokra@users.noreply.github.com>
Showing
Please
register
or
sign in
to comment