turbo-persistence: intermediate merges take the newest files of similar size (#99433)
## Summary Stacked on #99268. An intermediate merge merged every file above a shard's bottom run. Small commits, such as the snapshots of a development session, write small files: they push a shard over `max_files_above_bottom` quickly, and the merge then rewrote the large files of earlier build-sized commits along with them. Like the size ratio of RocksDB's universal compaction, an intermediate merge now takes the newest file, and then each next older one while it's at most `size_ratio_percent` (default 100%) larger than the files taken so far, with at least the two newest. Small files are merged with each other; a large file only joins once the newer files add up to about its size. Because the merge always takes the newest files, its output can still be placed after all existing files, and the older files it leaves out stay older. The trigger (`max_files_above_bottom`) and bottom merges are unchanged. ## Verification - A selector simulation where every build-like commit is followed by ten tiny commits: write amplification **6.43 → 1.98**, with the same maximum of 5 files per shard. The build-only simulation is unchanged (space amplification 1.34, write amplification 1.25). - Selector tests: small files after a large one merge without it; similar sizes all merge; steeply growing sizes merge at least the two newest; the ratio boundary. - `cargo test -p turbo-persistence`, `cargo test -p turbo-tasks-backend --tests`, and the Miri job's command for turbo-persistence pass. - End to end, 31 consecutive vercel-site builds with GC on (about half the TaskData commits are small), against #99268: | | #99268 | this PR | |---|---|---| | compaction bytes written per byte committed | 1.61× (12.52 GB) | **1.45×** (10.99 GB) | | …by intermediate merges | 3.69 GB | **2.34 GB (−37%)** | | …by bottom merges | 8.82 GB | 8.66 GB | | cache size, final / max | 2.94 / 3.24 GB | 2.92 / 3.13 GB | Intermediate merges run more often (126 vs 104 TaskData jobs) but each rewrites less. A development session, with many small snapshot commits, should benefit more. <!-- NEXT_JS_LLM -->
Showing
This diff is collapsed.
Please
register
or
sign in
to comment