Unverified Commit 7c5dbd19 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

turbo-persistence: intermediate merges take the newest files of similar size (#99433)

## Summary

Stacked on #99268.

An intermediate merge merged every file above a shard's bottom run.
Small commits, such as the snapshots of a development session, write
small files: they push a shard over `max_files_above_bottom` quickly,
and the merge then rewrote the large files of earlier build-sized
commits along with them.

Like the size ratio of RocksDB's universal compaction, an intermediate
merge now takes the newest file, and then each next older one while it's
at most `size_ratio_percent` (default 100%) larger than the files taken
so far, with at least the two newest. Small files are merged with each
other; a large file only joins once the newer files add up to about its
size. Because the merge always takes the newest files, its output can
still be placed after all existing files, and the older files it leaves
out stay older.

The trigger (`max_files_above_bottom`) and bottom merges are unchanged.

## Verification

- A selector simulation where every build-like commit is followed by ten
tiny commits: write amplification **6.43 → 1.98**, with the same maximum
of 5 files per shard. The build-only simulation is unchanged (space
amplification 1.34, write amplification 1.25).
- Selector tests: small files after a large one merge without it;
similar sizes all merge; steeply growing sizes merge at least the two
newest; the ratio boundary.
- `cargo test -p turbo-persistence`, `cargo test -p turbo-tasks-backend
--tests`, and the Miri job's command for turbo-persistence pass.
- End to end, 31 consecutive vercel-site builds with GC on (about half
the TaskData commits are small), against #99268:

  | | #99268 | this PR |
  |---|---|---|
| compaction bytes written per byte committed | 1.61× (12.52 GB) |
**1.45×** (10.99 GB) |
  | …by intermediate merges | 3.69 GB | **2.34 GB (−37%)** |
  | …by bottom merges | 8.82 GB | 8.66 GB |
  | cache size, final / max | 2.94 / 3.24 GB | 2.92 / 3.13 GB |

Intermediate merges run more often (126 vs 104 TaskData jobs) but each
rewrites less. A development session, with many small snapshot commits,
should benefit more.

<!-- NEXT_JS_LLM -->
parent 08d7a524
......@@ -385,7 +385,7 @@ CURRENT: 17
The key space of each family is split into a power-of-two number of shards by the leading bits of the key hash (starting at `FamilyConfig::initial_shard_bits`, then following the size of the family so that a shard holds about `DbConfig::target_shard_size` after compaction). The shard count doubles once a shard's bottom run exceeds 1.5 times the target and halves once it falls below half of it, so it doesn't flip back and forth. The size is estimated from the shards that have a bottom run, and the count is kept until one exists; the current count is recorded in each meta file. Commits and merges split SST files at shard boundaries, so each shard is compacted on its own. A shard has a bottom run (SST files flagged `bottom`, written by merging all files of the shard) and the files written since, above it.
- A bottom merge merges all files of a shard into a new bottom run, dropping superseded entries and tombstones. It runs when the files above the bottom run exceed `max_space_amplification_percent` of the bottom run, counting each tombstone as an average bottom entry since it deletes one. This bounds the space amplification.
- An intermediate merge merges only the files above the bottom run when there are more than `max_files_above_bottom`, to bound the number of files a lookup consults.
- An intermediate merge merges files above the bottom run when there are more than `max_files_above_bottom`, to bound the number of files a lookup consults. Like RocksDB's universal compaction, it takes the newest file and then each next older one that is at most `size_ratio_percent` larger than the files taken so far (but at least the two newest), so small files from small commits are merged with each other without rewriting larger files written before them.
A bottom merge writes the entries of recently read keys (the union of the "used key hashes" AMQFs of all meta files) into separate hot SST files, and the rest into cold ones. Processes tend to read the same keys again, so this packs them into fewer blocks, which matters on storage where reads are expensive. Since a bottom merge always includes the whole shard, hot and cold files are merged together and every version of a key meets. The used keys are never copied into the meta files written by compaction, so they expire once all SST files of the commit that recorded them are merged.
......
......@@ -976,6 +976,7 @@ fn bench_compaction(c: &mut Criterion) {
min_bottom_merge_bytes: 1024 * 1024,
max_files_above_bottom: 6,
rewrite_per_fresh_byte: 3.0,
size_ratio_percent: 100,
max_merge_jobs: 16,
})
.unwrap();
......
......@@ -44,6 +44,7 @@ pub const COMPACT_CONFIG: CompactConfig = CompactConfig {
min_bottom_merge_bytes: 1024 * 1024,
max_files_above_bottom: 6,
rewrite_per_fresh_byte: 3.0,
size_ratio_percent: 100,
max_merge_jobs: 16,
};
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment