Unverified Commit 08d7a524 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

turbo-persistence: space-amplification driven compaction with adaptive hash shards (#99268)

## Summary

Turbo-tasks GC can only shrink the persistent cache if compaction merges
each tombstone with the data it deletes. Canary's compaction algorithm
leaves many tombstones behind so with GC enabled the cache is only 5%
smaller than without it. This PR replaces compaction with a
space-amplification policy on hash shards, modeled on RocksDB universal
compaction and Cassandra UCS. With GC enabled, the cache stays ~flat.

### Cache size over 31 consecutive vercel-site builds

Same canary base and the same commits in every arm. The size is measured
after each build, including its compaction on shutdown.

| build | canary, GC off | canary, GC on | this PR, GC off | this PR, GC
on |
|---|---|---|---|---|
| 1 | 2.59 GB | 2.60 GB | 2.63 GB | 2.64 GB |
| 10 | 3.15 GB | 3.12 GB | 3.24 GB | 3.07 GB |
| 20 | 3.42 GB | 2.92 GB | 3.20 GB | 3.06 GB |
| 22 | 4.57 GB | 3.87 GB | 3.55 GB | 3.18 GB |
| 31 | **4.19 GB** | **3.96 GB** | **3.75 GB** | **3.00 GB** |
| growth, build 1 → 31 | +1.60 GB | +1.36 GB | +1.12 GB | **+0.36 GB** |

- **GC saves 5% with canary's compaction and 20% with this one** (final
size, GC on vs. off). A full compaction of the final cache puts the live
data at about 2.8 GB, so this PR with GC is within about 8% of it.
- **Build 22 rewrites a lot of task data.** Canary grows by about 1 GB
at that build in both modes and never recovers the space. This PR
absorbs it and merges the superseded versions away within a few builds.

### Why canary's compaction can't keep up with GC

- **Old versions and tombstones don't reliably meet the data they
supersede.** Canary wrote recently read keys and the rest into separate
warm and cold files and never merged across them.
- **Its merge triggers couldn't see the duplication.** There were
duplication-byte thresholds and `max_merge_bytes`. Most of the garbage
is superseded versions of a relatively small set of large, frequently
rewritten `TaskData` values, so those triggers missed it and produced
inefficient merge groups.

### New Compaction policy
- **Shards:** each family's key space is split into power-of-two
hash-prefix shards. The count starts at
`FamilyConfig::initial_shard_bits` (TaskData 8 shards, TaskMeta 4,
TaskCache and Infra 1) and then follows the family's size, targeting
about 256 MB per shard.
- It doubles once shards exceed 1.5× the target and halves below 0.5×,
so a size near a boundary doesn't flip it.
- Commits and merges split SST files at shard boundaries, so each shard
is compacted on its own. Files written with an old count are re-split by
their shard's next merge, not by a rewrite.
- Write batches still buffer a family in at most 4 collectors (64 MB
each) as on canary, split by a prefix of the shard bits, so their memory
doesn't grow with the shard count. Families with more than 4 shards
write proportionally smaller commit files.
- **Bottom merge:** when the files above a shard's bottom run exceed 50%
of it, the whole shard is merged into a new bottom run. Tombstones count
as the entries they delete. This is the merge that reliably drops
superseded entries and tombstones.
- This is analogous to a 'full compaction' but it targets a single shard
and is targeting files covering typically <0.5G of data.
- **Intermediate merge:** more than 4 files above the bottom run are
merged together, which bounds the number of files a lookup consults.
- **Budget:** bottom merges per family stop after rewriting 3× the
family's fresh (not yet compacted) bytes, so cost follows new writes and
shards spread over multiple compactions. 3× is how often a written byte
is copied in steady state: a bottom merge at the 50% trigger rewrites
1.5× the bytes above the bottom run for every 0.5× of them.
- **Hard cap:** a shard over twice the trigger (100% space
amplification), or with more than twice `max_files_above_bottom` files
above its bottom run, is bottom-merged even when the budget is spent.
This bounds how far a shard can fall behind, e.g. after a GC purge: its
tombstones count as the entries they delete, but they're small and give
little budget.
- **Tiny families:** a 1 MB floor keeps families like Infra, which
rewrite ~everything on every commit, out of bottom merges.
- **Ranking:** jobs are ranked across families by how far over their
trigger they are, so intermediate merges aren't starved.

**Hot/cold split, only in bottom merges.** A bottom merge writes
recently read keys and the rest into separate hot and cold files. It
always includes every file of its shard, hot and cold, so this does not
interfere with deduplication. Used-key tracking is unchanged from
canary: each commit records the keys it read in its meta file, and
compaction uses the union of all live meta files. The marks are never
copied into meta files written by compaction, so they expire with the
commit that recorded them.
- This Hot/cold split is less aggressive than canary. See below for Read
locality impacts.

**Reads:** a lookup only consults the SST files of its key's shard. A
per-family index keeps each shard's hash ranges densely, newest first,
so lookups scan them sequentially.

## Verification

- The cache size benchmark above: vercel-site, 31 commits, GC with
`rootTtlMs=0`. Earlier runs on the same base reproduce the GC-on results
(canary 3.93–3.99 GB, this PR 3.00–3.05 GB).
- **Read locality**, measured with temporary instrumentation that isn't
part of this PR. It counts the distinct SST blocks each build reads, and
the distinct 128 KiB file regions they touch as an estimate of squashfs
decompression. Same base, same builds, builds 2–31, GC on:

  | | canary | no split | this PR (bottom split) |
  |---|---|---|---|
| distinct blocks read per build, median | 1.35 GB | 1.80 GB | 1.55 GB |
| 128 KiB regions touched per build, median / total | 2.42 / 64.2 GB | —
| 2.63 / 68.8 GB |

The bottom split recovers most of the locality that dropping the split
cost. Light builds still read about a third more than on canary, because
a shard's hot set only refreshes at its bottom merge. Splitting
intermediate merges as well gained about 1%, so it isn't included.
- **Lookup microbenchmark**, 16 shards × 20 uncompacted commits (320
SSTs), alternating before and after: `get` hit −14%, miss −28%,
`batch_get` −5%.
- `cargo test -p turbo-persistence` and `cargo test -p
turbo-tasks-backend --tests` pass. The selector has unit tests, and a
simulation whose sweep supports the 50% threshold.

<!-- NEXT_JS_LLM -->
parent bd932b38
...@@ -37,6 +37,7 @@ fn main() -> Result<()> { ...@@ -37,6 +37,7 @@ fn main() -> Result<()> {
sst_size, sst_size,
flags, flags,
block_count, block_count,
entry_count,
} in meta_file.entries } in meta_file.entries
{ {
println!( println!(
...@@ -46,7 +47,7 @@ fn main() -> Result<()> { ...@@ -46,7 +47,7 @@ fn main() -> Result<()> {
); );
println!(" AMQF {amqf_entries} entries = {} KiB", amqf_size / 1024); println!(" AMQF {amqf_entries} entries = {} KiB", amqf_size / 1024);
println!( println!(
" {} KiB = {block_count} blocks (avg {} bytes/block)", " {} KiB = {block_count} blocks (avg {} bytes/block), {entry_count} entries",
sst_size / 1024, sst_size / 1024,
sst_size / block_count as u64 sst_size / block_count as u64
); );
......
...@@ -19,7 +19,7 @@ auto-hash-map = { workspace = true } ...@@ -19,7 +19,7 @@ auto-hash-map = { workspace = true }
bitfield = { workspace = true } bitfield = { workspace = true }
byteorder = { workspace = true } byteorder = { workspace = true }
crc32fast = { workspace = true } crc32fast = { workspace = true }
dashmap = { workspace = true} dashmap = { workspace = true }
either = { workspace = true } either = { workspace = true }
fs-err = { workspace = true } fs-err = { workspace = true }
jiff = { version = "0.2.10", features = ["serde"] } jiff = { version = "0.2.10", features = ["serde"] }
......
...@@ -57,6 +57,7 @@ A meta file can contain metadata about multiple SST files. The metadata is store ...@@ -57,6 +57,7 @@ A meta file can contain metadata about multiple SST files. The metadata is store
- 4 bytes magic number (0xFE4ADA4A) - 4 bytes magic number (0xFE4ADA4A)
- 4 bytes key family - 4 bytes key family
- 1 byte compression algorithm, which must match the configuration used to open the database - 1 byte compression algorithm, which must match the configuration used to open the database
- 1 byte shard bits: the family had `2^bits` shards when the SST files were split
- 4 bytes count of obsolete SST files - 4 bytes count of obsolete SST files
- foreach obsolete SST file - foreach obsolete SST file
- 4 bytes sequence number of the obsolete SST file - 4 bytes sequence number of the obsolete SST file
...@@ -70,11 +71,14 @@ A meta file can contain metadata about multiple SST files. The metadata is store ...@@ -70,11 +71,14 @@ A meta file can contain metadata about multiple SST files. The metadata is store
- 4 bytes flags - 4 bytes flags
- bit 0: cold (compacted and not recently accessed) - bit 0: cold (compacted and not recently accessed)
- bit 1: fresh (not yet compacted) - bit 1: fresh (not yet compacted)
- bit 2: bottom (part of the bottom run of its shard, written by merging all SST files of the shard; without the cold bit it holds the recently read keys)
- 4 bytes entry count
- 4 bytes tombstone count (entries that delete a key or a key-value pair), used by compaction to estimate reclaimable bytes
- 4 bytes end of AMQF offset relative to start of all AMQF data - 4 bytes end of AMQF offset relative to start of all AMQF data
- 4 bytes end of AMQF offset relative to start of all AMQF data of the "used key hashes" AMQF - 4 bytes end of AMQF offset relative to start of all AMQF data of the "used key hashes" AMQF
- foreach described SST file - foreach described SST file
- serialized AMQF - serialized AMQF
- serialized "used key hashes" AMQF - serialized "used key hashes" AMQF: the keys read since the previous commit (only in meta files written by a commit)
### SST file ### SST file
...@@ -376,10 +380,16 @@ DEL 17: (2, 3, 4, 5, 6, 7, 8, 9) ...@@ -376,10 +380,16 @@ DEL 17: (2, 3, 4, 5, 6, 7, 8, 9)
CURRENT: 17 CURRENT: 17
``` ```
Configuration options for compactions are: ### Choosing merge jobs
- max number of SST files that are merged at once The key space of each family is split into a power-of-two number of shards by the leading bits of the key hash (starting at `FamilyConfig::initial_shard_bits`, then following the size of the family so that a shard holds about `DbConfig::target_shard_size` after compaction). The shard count doubles once a shard's bottom run exceeds 1.5 times the target and halves once it falls below half of it, so it doesn't flip back and forth. The size is estimated from the shards that have a bottom run, and the count is kept until one exists; the current count is recorded in each meta file. Commits and merges split SST files at shard boundaries, so each shard is compacted on its own. A shard has a bottom run (SST files flagged `bottom`, written by merging all files of the shard) and the files written since, above it.
- coverage when compaction is triggered (otherwise calling compact is a noop)
- A bottom merge merges all files of a shard into a new bottom run, dropping superseded entries and tombstones. It runs when the files above the bottom run exceed `max_space_amplification_percent` of the bottom run, counting each tombstone as an average bottom entry since it deletes one. This bounds the space amplification.
- An intermediate merge merges only the files above the bottom run when there are more than `max_files_above_bottom`, to bound the number of files a lookup consults.
A bottom merge writes the entries of recently read keys (the union of the "used key hashes" AMQFs of all meta files) into separate hot SST files, and the rest into cold ones. Processes tend to read the same keys again, so this packs them into fewer blocks, which matters on storage where reads are expensive. Since a bottom merge always includes the whole shard, hot and cold files are merged together and every version of a key meets. The used keys are never copied into the meta files written by compaction, so they expire once all SST files of the commit that recorded them are merged.
Bottom merges are paced by the fresh (not yet compacted) files of the family: they are scheduled until they rewrote `rewrite_per_fresh_byte` times the size of those files (the last one may go past it, and the first one always runs), so compaction cost follows the amount of new data. Since keys are hashes, shards grow at the same rate; the rewrite quota spreads their bottom merges over multiple compactions. A skipped compaction leaves the fresh files in place, which increases the next quota. The default is 3: a bottom merge at the 50% trigger rewrites 1.5 times the bytes above the bottom run for each 0.5 of them, so in steady state a written byte is copied about 3 times. A shard over twice the trigger, or with over twice `max_files_above_bottom` files above its bottom run, is bottom merged even when the quota is reached, which bounds how far a shard can fall behind (e.g. after a GC purge, whose small tombstone files give a small quota).
## Opening ## Opening
......
...@@ -14,7 +14,8 @@ use tempfile::TempDir; ...@@ -14,7 +14,8 @@ use tempfile::TempDir;
use turbo_persistence::{ use turbo_persistence::{
ArcBytes, BlockCache, CompactConfig, Compression, DbConfig as TpDbConfig, Entry, EntryValue, ArcBytes, BlockCache, CompactConfig, Compression, DbConfig as TpDbConfig, Entry, EntryValue,
FamilyConfig, FamilyKind, MetaEntryFlags, SerialScheduler, StaticSortedFile, FamilyConfig, FamilyKind, MetaEntryFlags, SerialScheduler, StaticSortedFile,
StaticSortedFileMetaData, TurboPersistence, hash_key, write_static_stored_file, StaticSortedFileMetaData, TurboPersistence, hash_key, shard::ShardBits,
write_static_stored_file,
}; };
use turbo_tasks_malloc::TurboMalloc; use turbo_tasks_malloc::TurboMalloc;
...@@ -625,6 +626,7 @@ fn prefill_multi_value_database( ...@@ -625,6 +626,7 @@ fn prefill_multi_value_database(
name: "test", name: "test",
kind: FamilyKind::MultiValue, kind: FamilyKind::MultiValue,
compression: Compression::Lz4, compression: Compression::Lz4,
initial_shard_bits: ShardBits::new(0),
}], }],
..TpDbConfig::new() ..TpDbConfig::new()
}; };
...@@ -701,6 +703,7 @@ fn open_multi_value_db(path: &Path) -> TurboPersistence<SerialScheduler, 1> { ...@@ -701,6 +703,7 @@ fn open_multi_value_db(path: &Path) -> TurboPersistence<SerialScheduler, 1> {
name: "test", name: "test",
kind: FamilyKind::MultiValue, kind: FamilyKind::MultiValue,
compression: Compression::Lz4, compression: Compression::Lz4,
initial_shard_bits: ShardBits::new(0),
}], }],
..TpDbConfig::new() ..TpDbConfig::new()
}; };
...@@ -862,6 +865,7 @@ fn bench_family_sharding(c: &mut Criterion) { ...@@ -862,6 +865,7 @@ fn bench_family_sharding(c: &mut Criterion) {
name, name,
kind: FamilyKind::SingleValue, kind: FamilyKind::SingleValue,
compression: Compression::Lz4, compression: Compression::Lz4,
initial_shard_bits: ShardBits::new(0),
} }
}), }),
..TpDbConfig::new() ..TpDbConfig::new()
...@@ -968,13 +972,11 @@ fn bench_compaction(c: &mut Criterion) { ...@@ -968,13 +972,11 @@ fn bench_compaction(c: &mut Criterion) {
|(_tempdir, db)| { |(_tempdir, db)| {
// Timed: run normal compaction // Timed: run normal compaction
db.compact(&CompactConfig { db.compact(&CompactConfig {
min_merge_count: 3, max_space_amplification_percent: 50,
optimal_merge_count: 8, min_bottom_merge_bytes: 1024 * 1024,
max_merge_count: 64, max_files_above_bottom: 6,
max_merge_bytes: 512 * MB, rewrite_per_fresh_byte: 3.0,
min_merge_duplication_bytes: 50 * MB, max_merge_jobs: 16,
optimal_merge_duplication_bytes: 100 * MB,
max_merge_segment_count: 16,
}) })
.unwrap(); .unwrap();
black_box(db) black_box(db)
...@@ -1048,6 +1050,7 @@ fn bench_write_multi_value(c: &mut Criterion) { ...@@ -1048,6 +1050,7 @@ fn bench_write_multi_value(c: &mut Criterion) {
name: "test", name: "test",
kind: FamilyKind::MultiValue, kind: FamilyKind::MultiValue,
compression: Compression::Lz4, compression: Compression::Lz4,
initial_shard_bits: ShardBits::new(0),
}], }],
..TpDbConfig::new() ..TpDbConfig::new()
}; };
......
...@@ -148,7 +148,7 @@ impl<K: StoreKey, const SIZE_SHIFT: usize> Collector<K, SIZE_SHIFT> { ...@@ -148,7 +148,7 @@ impl<K: StoreKey, const SIZE_SHIFT: usize> Collector<K, SIZE_SHIFT> {
/// This method does not deduplicate entries. /// This method does not deduplicate entries.
/// ///
/// In debug builds, asserts that SingleValue families have no duplicate keys. /// In debug builds, asserts that SingleValue families have no duplicate keys.
pub fn sorted(&mut self, family_kind: FamilyKind) -> (&[CollectorEntry<K>], usize) { pub fn sorted(&mut self, family_kind: FamilyKind) -> &[CollectorEntry<K>] {
// We can use unstable sort because the relative order of equal elements // We can use unstable sort because the relative order of equal elements
// doesn't matter — duplicates are either disallowed (SingleValue) or // doesn't matter — duplicates are either disallowed (SingleValue) or
// allowed without deduplication (MultiValue). // allowed without deduplication (MultiValue).
...@@ -181,7 +181,7 @@ impl<K: StoreKey, const SIZE_SHIFT: usize> Collector<K, SIZE_SHIFT> { ...@@ -181,7 +181,7 @@ impl<K: StoreKey, const SIZE_SHIFT: usize> Collector<K, SIZE_SHIFT> {
// Suppress unused variable warning in release builds // Suppress unused variable warning in release builds
let _ = family_kind; let _ = family_kind;
(&self.entries, self.total_key_size) &self.entries
} }
/// Clears the collector. /// Clears the collector.
...@@ -202,7 +202,7 @@ impl<K: StoreKey, const SIZE_SHIFT: usize> Collector<K, SIZE_SHIFT> { ...@@ -202,7 +202,7 @@ impl<K: StoreKey, const SIZE_SHIFT: usize> Collector<K, SIZE_SHIFT> {
} }
/// Clears the collector and drops the capacity /// Clears the collector and drops the capacity
pub fn drop_contents(&mut self) { pub fn clear_and_drop_capacity(&mut self) {
drop(take(&mut self.entries)); drop(take(&mut self.entries));
self.total_key_size = 0; self.total_key_size = 0;
self.total_value_size = 0; self.total_value_size = 0;
......
mod interval_map;
pub mod selector; pub mod selector;
#[cfg(test)]
mod naive_interval_map;
use std::{
fmt::{self, Display},
iter::Peekable,
ops::{RangeBounds, RangeInclusive},
slice,
};
use crate::compaction::interval_map::IntervalBound;
/// An integer with a very limited range to allow for exhaustive unit tests of
/// all possible values.
#[derive(Clone, Copy, Debug, Default, Eq, Ord, PartialEq, PartialOrd)]
pub struct TinyInt(pub u8);
impl TinyInt {
pub const MIN: TinyInt = TinyInt(0);
pub const MAX: TinyInt = TinyInt(6);
}
impl IntervalBound for TinyInt {
fn bound_min() -> Self {
Self::MIN
}
fn bound_max() -> Self {
Self::MAX
}
fn checked_increment(&self) -> Option<Self> {
if self < &Self::bound_max() {
Some(Self(self.0 + 1))
} else {
None
}
}
fn checked_decrement(&self) -> Option<Self> {
if self > &Self::bound_min() {
Some(Self(self.0 - 1))
} else {
None
}
}
}
impl Display for TinyInt {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.0)
}
}
/// An impractically slow but very simple implementation of `IntervalMap` used as a known-good
/// version for testing the correctness of `IntervalMap`.
#[allow(dead_code)]
pub struct NaiveIntervalMap<T, B = TinyInt> {
values: Vec<(B, T)>,
}
impl<T, B> NaiveIntervalMap<T, B>
where
T: Default,
B: IntervalBound,
{
pub fn new() -> Self {
let mut values = Vec::new();
let mut pos = Some(B::bound_min());
while let Some(cur) = &pos {
values.push((*cur, T::default()));
pos = cur.checked_increment();
}
Self { values }
}
}
impl<T, B> NaiveIntervalMap<T, B>
where
B: IntervalBound,
T: Clone,
{
pub fn update(&mut self, bounds: impl RangeBounds<B>, mut update: impl FnMut(&mut T)) {
for (pos, cur_value) in &mut self.values {
if bounds.contains(pos) {
update(cur_value);
}
}
}
pub fn replace(&mut self, bounds: impl RangeBounds<B>, value: T) {
self.update(bounds, |v| *v = value.clone());
}
}
impl<T, B> NaiveIntervalMap<T, B>
where
B: IntervalBound,
{
pub fn iter(&self) -> NaiveIntervalMapIterator<'_, T, B> {
NaiveIntervalMapIterator {
values_iter: self.values.iter().peekable(),
}
}
}
pub struct NaiveIntervalMapIterator<'a, T, B> {
values_iter: Peekable<slice::Iter<'a, (B, T)>>,
}
impl<'a, T, B> Iterator for NaiveIntervalMapIterator<'a, T, B>
where
B: IntervalBound,
T: Eq,
{
type Item = (RangeInclusive<B>, &'a T);
fn next(&mut self) -> Option<Self::Item> {
let (start_pos, cur_value) = self.values_iter.next()?;
let mut end_pos = *start_pos;
while self.values_iter.peek().is_some_and(|(_, v)| v == cur_value) {
end_pos = self.values_iter.next().unwrap().0;
}
Some(((*start_pos)..=end_pos, cur_value))
}
}
...@@ -21,6 +21,7 @@ mod meta_file_builder; ...@@ -21,6 +21,7 @@ mod meta_file_builder;
pub mod mmap_helper; pub mod mmap_helper;
mod parallel_scheduler; mod parallel_scheduler;
mod rc_bytes; mod rc_bytes;
pub mod shard;
mod shared_bytes; mod shared_bytes;
pub mod sst_filter; pub mod sst_filter;
pub mod static_sorted_file; pub mod static_sorted_file;
...@@ -69,6 +70,13 @@ pub struct FamilyConfig { ...@@ -69,6 +70,13 @@ pub struct FamilyConfig {
pub name: &'static str, pub name: &'static str,
pub kind: FamilyKind, pub kind: FamilyKind,
pub compression: Compression, pub compression: Compression,
/// Initial sharding factor for the family. There will be 2^ShardBits shards of each family.
///
/// The shard count follows the size of the family, see [`DbConfig::target_shard_size`].
/// This is just a hint to bootstrap the family. `0` would always work, but families with lots
/// of data would benefit from starting with many shards since this influences how initial
/// commits shard outputs.
pub initial_shard_bits: shard::ShardBits,
} }
/// Database-wide configuration with per-family storage settings. /// Database-wide configuration with per-family storage settings.
...@@ -80,6 +88,10 @@ pub struct DbConfig<const FAMILIES: usize> { ...@@ -80,6 +88,10 @@ pub struct DbConfig<const FAMILIES: usize> {
pub family_configs: [FamilyConfig; FAMILIES], pub family_configs: [FamilyConfig; FAMILIES],
/// How SST and meta files are read from disk. /// How SST and meta files are read from disk.
pub access_mode: AccessMode, pub access_mode: AccessMode,
/// The size a shard of a family should have after compaction.
///
/// This is a rough bound to avoid frequent resharding, See [`shard`] for sizing semantics.
pub target_shard_size: u64,
} }
/// Returns the default access mode for this execution environment. /// Returns the default access mode for this execution environment.
...@@ -130,8 +142,10 @@ impl<const FAMILIES: usize> DbConfig<FAMILIES> { ...@@ -130,8 +142,10 @@ impl<const FAMILIES: usize> DbConfig<FAMILIES> {
name: "unknown", name: "unknown",
kind: FamilyKind::SingleValue, kind: FamilyKind::SingleValue,
compression: Compression::Lz4, compression: Compression::Lz4,
initial_shard_bits: shard::ShardBits::new(0),
}; FAMILIES], }; FAMILIES],
access_mode: default_access_mode(), access_mode: default_access_mode(),
target_shard_size: 256 * 1024 * 1024,
} }
} }
} }
......
...@@ -12,12 +12,15 @@ use zerocopy::IntoBytes; ...@@ -12,12 +12,15 @@ use zerocopy::IntoBytes;
use crate::{ use crate::{
Compression, Compression,
meta_file::{EntryHeader, META_FILE_MAGIC}, meta_file::{EntryHeader, META_FILE_MAGIC},
shard::ShardBits,
static_sorted_file_builder::StaticSortedFileBuilderMeta, static_sorted_file_builder::StaticSortedFileBuilderMeta,
}; };
pub struct MetaFileBuilder<'a> { pub struct MetaFileBuilder<'a> {
family: u32, family: u32,
compression: Compression, compression: Compression,
/// The shards the SST files of this meta file were split with.
shard_bits: ShardBits,
/// Entries in the meta file, tuples of (sequence_number, StaticSortedFileBuilderMetaResult) /// Entries in the meta file, tuples of (sequence_number, StaticSortedFileBuilderMetaResult)
entries: Vec<(u32, StaticSortedFileBuilderMeta<'a>)>, entries: Vec<(u32, StaticSortedFileBuilderMeta<'a>)>,
/// Obsolete SST files, represented by their sequence numbers /// Obsolete SST files, represented by their sequence numbers
...@@ -27,10 +30,11 @@ pub struct MetaFileBuilder<'a> { ...@@ -27,10 +30,11 @@ pub struct MetaFileBuilder<'a> {
} }
impl<'a> MetaFileBuilder<'a> { impl<'a> MetaFileBuilder<'a> {
pub fn new(family: u32, compression: Compression) -> Self { pub fn new(family: u32, compression: Compression, shard_bits: ShardBits) -> Self {
Self { Self {
family, family,
compression, compression,
shard_bits,
entries: Vec::new(), entries: Vec::new(),
obsolete_sst_files: Vec::new(), obsolete_sst_files: Vec::new(),
used_key_hashes_amqf: None, used_key_hashes_amqf: None,
...@@ -63,6 +67,7 @@ impl<'a> MetaFileBuilder<'a> { ...@@ -63,6 +67,7 @@ impl<'a> MetaFileBuilder<'a> {
file.write_u32::<BE>(META_FILE_MAGIC)?; // Magic number file.write_u32::<BE>(META_FILE_MAGIC)?; // Magic number
file.write_u32::<BE>(self.family)?; file.write_u32::<BE>(self.family)?;
file.write_u8(self.compression as u8)?; file.write_u8(self.compression as u8)?;
file.write_u8(self.shard_bits.get())?;
self.obsolete_sst_files.sort(); self.obsolete_sst_files.sort();
file.write_u32::<BE>(self.obsolete_sst_files.len() as u32)?; file.write_u32::<BE>(self.obsolete_sst_files.len() as u32)?;
...@@ -82,6 +87,8 @@ impl<'a> MetaFileBuilder<'a> { ...@@ -82,6 +87,8 @@ impl<'a> MetaFileBuilder<'a> {
sst.max_hash, sst.max_hash,
sst.size, sst.size,
sst.flags, sst.flags,
u32::try_from(sst.entries).unwrap_or(u32::MAX),
u32::try_from(sst.tombstones).unwrap_or(u32::MAX),
amqf_offset as u32, amqf_offset as u32,
); );
file.write_all(header.as_bytes())?; file.write_all(header.as_bytes())?;
......
This diff is collapsed.
...@@ -330,6 +330,8 @@ pub struct StaticSortedFileBuilderMeta<'a> { ...@@ -330,6 +330,8 @@ pub struct StaticSortedFileBuilderMeta<'a> {
pub flags: MetaEntryFlags, pub flags: MetaEntryFlags,
/// The number of entries in the SST file /// The number of entries in the SST file
pub entries: u64, pub entries: u64,
/// The number of tombstone entries (`KeyDeleted` and `KeyValueDeleted`) in the SST file.
pub tombstones: u64,
} }
/// Writes an SST file from a pre-sorted slice of entries. /// Writes an SST file from a pre-sorted slice of entries.
...@@ -609,6 +611,7 @@ pub struct StreamingSstWriter<E: Entry> { ...@@ -609,6 +611,7 @@ pub struct StreamingSstWriter<E: Entry> {
min_hash: u64, min_hash: u64,
max_hash: u64, max_hash: u64,
entry_count: u64, entry_count: u64,
tombstone_count: u64,
flags: MetaEntryFlags, flags: MetaEntryFlags,
// Fullness tracking (for compaction callers) // Fullness tracking (for compaction callers)
...@@ -678,6 +681,7 @@ impl<E: Entry> StreamingSstWriter<E> { ...@@ -678,6 +681,7 @@ impl<E: Entry> StreamingSstWriter<E> {
min_hash: u64::MAX, min_hash: u64::MAX,
max_hash: 0, max_hash: 0,
entry_count: 0, entry_count: 0,
tombstone_count: 0,
flags, flags,
total_key_size: 0, total_key_size: 0,
total_value_size: 0, total_value_size: 0,
...@@ -811,8 +815,12 @@ impl<E: Entry> StreamingSstWriter<E> { ...@@ -811,8 +815,12 @@ impl<E: Entry> StreamingSstWriter<E> {
} }
} }
EntryValue::Large { blob } => ValueRef::Blob { blob_id: blob }, EntryValue::Large { blob } => ValueRef::Blob { blob_id: blob },
EntryValue::KeyDeleted => ValueRef::KeyDeleted, EntryValue::KeyDeleted => {
self.tombstone_count += 1;
ValueRef::KeyDeleted
}
EntryValue::KeyValueDeleted { value } => { EntryValue::KeyValueDeleted { value } => {
self.tombstone_count += 1;
// Enforced by `WriteBatch::delete_value`, which rejects oversized values. // Enforced by `WriteBatch::delete_value`, which rejects oversized values.
debug_assert!(value.len() <= MAX_INLINE_VALUE_SIZE); debug_assert!(value.len() <= MAX_INLINE_VALUE_SIZE);
let mut data = [0u8; MAX_INLINE_VALUE_SIZE]; let mut data = [0u8; MAX_INLINE_VALUE_SIZE];
...@@ -1150,6 +1158,7 @@ impl<E: Entry> StreamingSstWriter<E> { ...@@ -1150,6 +1158,7 @@ impl<E: Entry> StreamingSstWriter<E> {
size: file_size, size: file_size,
flags: self.flags, flags: self.flags,
entries: self.entry_count, entries: self.entry_count,
tombstones: self.tombstone_count,
}; };
let file = file.into_inner()?; let file = file.into_inner()?;
......
use turbo_persistence::{Compression, FamilyConfig, FamilyKind}; use turbo_persistence::{Compression, FamilyConfig, FamilyKind, shard::ShardBits};
#[derive(Debug, Clone, Copy)] #[derive(Debug, Clone, Copy)]
pub enum KeySpace { pub enum KeySpace {
...@@ -39,17 +39,23 @@ impl KeySpace { ...@@ -39,17 +39,23 @@ impl KeySpace {
name: self.name(), name: self.name(),
kind: FamilyKind::SingleValue, kind: FamilyKind::SingleValue,
compression: Compression::Lz4, compression: Compression::Lz4,
initial_shard_bits: ShardBits::new(match self {
KeySpace::Infra => 0,
_ => 2,
}),
}, },
KeySpace::TaskData => FamilyConfig { KeySpace::TaskData => FamilyConfig {
name: self.name(), name: self.name(),
kind: FamilyKind::SingleValue, kind: FamilyKind::SingleValue,
compression: Compression::Zstd3, compression: Compression::Zstd3,
initial_shard_bits: ShardBits::new(3),
}, },
KeySpace::TaskCache => FamilyConfig { KeySpace::TaskCache => FamilyConfig {
name: self.name(), name: self.name(),
// TaskCache uses hash-based lookups with potential collisions. // TaskCache uses hash-based lookups with potential collisions.
kind: FamilyKind::MultiValue, kind: FamilyKind::MultiValue,
compression: Compression::Lz4, compression: Compression::Lz4,
initial_shard_bits: ShardBits::new(0),
}, },
} }
} }
......
...@@ -14,7 +14,7 @@ use turbo_persistence::{ ...@@ -14,7 +14,7 @@ use turbo_persistence::{
use turbo_tasks::{ use turbo_tasks::{
message_queue::{TimingEvent, TraceEvent}, message_queue::{TimingEvent, TraceEvent},
parallel::available_parallelism, parallel::available_parallelism,
turbo_tasks, try_turbo_tasks,
}; };
use crate::{ use crate::{
...@@ -30,8 +30,6 @@ pub const FAMILIES: usize = 4; ...@@ -30,8 +30,6 @@ pub const FAMILIES: usize = 4;
const COMPACTION_MESSAGE: &str = "Finished filesystem cache database compaction"; const COMPACTION_MESSAGE: &str = "Finished filesystem cache database compaction";
const MB: u64 = 1024 * 1024;
/// Returns the database configuration for the Turbopack persistent cache, mapping each /// Returns the database configuration for the Turbopack persistent cache, mapping each
/// [`KeySpace`] to its persistence family config. /// [`KeySpace`] to its persistence family config.
pub fn db_config() -> DbConfig<FAMILIES> { pub fn db_config() -> DbConfig<FAMILIES> {
...@@ -42,13 +40,11 @@ pub fn db_config() -> DbConfig<FAMILIES> { ...@@ -42,13 +40,11 @@ pub fn db_config() -> DbConfig<FAMILIES> {
} }
pub const COMPACT_CONFIG: CompactConfig = CompactConfig { pub const COMPACT_CONFIG: CompactConfig = CompactConfig {
min_merge_count: 3, max_space_amplification_percent: 50,
optimal_merge_count: 8, min_bottom_merge_bytes: 1024 * 1024,
max_merge_count: 64, max_files_above_bottom: 6,
max_merge_bytes: 512 * MB, rewrite_per_fresh_byte: 3.0,
min_merge_duplication_bytes: 50 * MB, max_merge_jobs: 16,
optimal_merge_duplication_bytes: 100 * MB,
max_merge_segment_count: 16,
}; };
pub struct TurboKeyValueDatabase { pub struct TurboKeyValueDatabase {
...@@ -166,24 +162,27 @@ impl TurboKeyValueDatabase { ...@@ -166,24 +162,27 @@ impl TurboKeyValueDatabase {
fn do_compact( fn do_compact(
db: &TurboPersistence<TurboTasksParallelScheduler, FAMILIES>, db: &TurboPersistence<TurboTasksParallelScheduler, FAMILIES>,
message: &'static str, message: &'static str,
max_merge_segment_count: usize, max_merge_jobs: usize,
) -> Result<Option<CommitStats>> { ) -> Result<Option<CommitStats>> {
let start = Instant::now(); let start = Instant::now();
// SystemTime for wall-clock timestamps in trace events (Instant has no // SystemTime for wall-clock timestamps in trace events (Instant has no
// defined epoch so it can't be used for cross-process trace correlation). // defined epoch so it can't be used for cross-process trace correlation).
let wall_start = SystemTime::now(); let wall_start = SystemTime::now();
let stats = db.compact(&CompactConfig { let stats = db.compact(&CompactConfig {
max_merge_segment_count, max_merge_jobs,
..COMPACT_CONFIG ..COMPACT_CONFIG
})?; })?;
if let Some(stats) = stats { // Compaction can run outside of turbo-tasks (e.g. in tests), then there is nobody to report to.
if let Some(stats) = stats
&& let Some(turbo_tasks) = try_turbo_tasks()
{
let elapsed = start.elapsed(); let elapsed = start.elapsed();
// avoid spamming the event queue with information about fast operations // avoid spamming the event queue with information about fast operations
if elapsed > Duration::from_secs(10) { if elapsed > Duration::from_secs(10) {
turbo_tasks() turbo_tasks
.send_compilation_event(Arc::new(TimingEvent::new(message.to_string(), elapsed))); .send_compilation_event(Arc::new(TimingEvent::new(message.to_string(), elapsed)));
} }
turbo_tasks().send_compilation_event(Arc::new(TraceEvent::new_with_duration( turbo_tasks.send_compilation_event(Arc::new(TraceEvent::new_with_duration(
"turbopack-compaction", "turbopack-compaction",
wall_start, wall_start,
elapsed, elapsed,
......
...@@ -98,7 +98,7 @@ pub fn compact_database( ...@@ -98,7 +98,7 @@ pub fn compact_database(
// Fully compact with no segment count limit (unlike the runtime shutdown path // Fully compact with no segment count limit (unlike the runtime shutdown path
// which caps segments based on available parallelism). // which caps segments based on available parallelism).
db.compact(&CompactConfig { db.compact(&CompactConfig {
max_merge_segment_count: usize::MAX, max_merge_jobs: usize::MAX,
..turbo::COMPACT_CONFIG ..turbo::COMPACT_CONFIG
})?; })?;
db.shutdown() db.shutdown()
......
...@@ -105,7 +105,7 @@ pub use crate::{ ...@@ -105,7 +105,7 @@ pub use crate::{
ScheduleKey, TaskPersistence, TaskPriority, TurboTasks, TurboTasksApi, TurboTasksCallApi, ScheduleKey, TaskPersistence, TaskPriority, TurboTasks, TurboTasksApi, TurboTasksCallApi,
Unused, UpdateInfo, dynamic_call, emit, get_serialization_invalidator, mark_finished, Unused, UpdateInfo, dynamic_call, emit, get_serialization_invalidator, mark_finished,
mark_stateful, mark_top_level_task, prevent_gc, run, run_once, run_once_with_reason, mark_stateful, mark_top_level_task, prevent_gc, run, run_once, run_once_with_reason,
trait_call, turbo_tasks, turbo_tasks_scope, turbo_tasks_weak, trait_call, try_turbo_tasks, turbo_tasks, turbo_tasks_scope, turbo_tasks_weak,
unmark_top_level_task_may_leak_eventually_consistent_state, with_turbo_tasks, unmark_top_level_task_may_leak_eventually_consistent_state, with_turbo_tasks,
}, },
mapped_read_ref::MappedReadRef, mapped_read_ref::MappedReadRef,
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment