[turbopack] use zero copy qfilter deserialization (#90574)
## What Switch turbo-persistence AMQF filters from owned `qfilter::Filter` (heap-allocated) to zero-copy `qfilter::FilterRef` that borrows directly from the memory-mapped meta file. ## Why - **Lower memory usage** — no heap copy of filter data; `FilterRef` is just a pointer into the mmap - **Faster open time** — no deserialization/allocation, just pointer math over the mmap - **OS-managed memory** — mmap pages can be cheaply evicted under pressure (free LRU behavior), unlike heap-allocated `Filter` data which needs to be swapped ## How - Update `qfilter` dependency to the latest alpha release with `FilterRef` support - Switch per-entry AMQF serialization from `turbo_bincode` to `pot` format, which supports zero-copy deserialization - Store `qfilter::FilterRef<'static>` directly in `MetaEntry` (lifetime transmuted from the mmap borrow) - Rely on Rust's struct field drop order guarantee: `MetaFile::entries` is declared before `MetaFile::mmap`, so all `FilterRef`s are dropped before the mmap is unmapped - Update compaction code to work with `FilterRef` avoiding many allocations when merging ### Safety invariants The `FilterRef<'static>` lifetime is transmuted🙀 — the actual borrow is from `MetaFile::mmap`. This is safe because: 1. `MetaEntry` is never moved out of `MetaFile` (only accessed by `&` reference via `entries()` / `entry()`) 2. Rust drops struct fields in declaration order, and `entries` is declared before `mmap` 3. This is the same pattern used by `ArcBytes`/`RcBytes` in this crate (raw pointer into backing storage) ## Benchmark Results I ran a number of the Read benchmarks and compaction benchmarks and it is all in the noise, which makes sense. We might get a slight benefit from avoiding the `OnceLock` and lazy initialization, we should also have slightly lower maxrss and offer the OS more flexibility during memory pressure From measuring vercel-site, a warm build saves ~40m of MaxRSS
Showing
This diff is collapsed.
Please
register
or
sign in
to comment