Unverified Commit 09135201 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

[turbopack] Reduce memory usage during compaction (#90229)

### What?

Optimizes memory usage, I/O patterns, and CPU efficiency during database compaction by:

* clearing block caches and SST mmaps before compaction starts
* opening SST files independently with madvise(SEQUENTIAL) for compaction
* making the SST iterator own the file so mmaps are dropped as as soon as the iterator is are consumed
* moving used-key-hash filter deserialization into per-family scope, 
    * also mering AMQF filters into a single pre-sized filter per family, reducing used-key classification from O(N filters) to O(1) per entry
* flushing entries early once collectors reach 50% capacity to reduce peak overhead while still avoiding small files
* adding a lightweight single-entry value block cache on the iterator (replacing the global `BlockCache` for sequential scans, this reaps all the benefits while not accumulating too many blocks that will never be visited again)
* boxing MergeIter heap elements so BinaryHeap sifts swap 8-byte pointers instead of ~312-byte structs

### Why?

Database compaction was consuming excessive memory by maintaining block caches and memory-mapped files that weren't needed during the sequential merge operations. The existing approach used random access patterns optimized for lookups, but compaction performs sequential scans that benefit from different memory management strategies.

Profiling revealed two additional CPU bottlenecks:
1. **AMQF query overhead (44% of compaction time):** With 128 commits, `compact_internal` iterated all 128 AMQF filters per entry (16M entries × 128 filters = ~2B probes). Most filters were empty in write-only workloads but still deserialized. Filtering empty filters and merging the rest into a single filter eliminates this entirely.
2. **MergeIter heap sift overhead (21% inclusive):** `ActiveIterator<StaticSortedFileIter>` is ~312 bytes. BinaryHeap sifts moved these large structs at each level (log2(128) = 7 levels per pop/push). Boxing reduces each swap to 8 bytes, cutting MergeIter inclusive time from 21% to 12%.

### Performance results

Compaction benchmark (16Mi entries, 128 commits):

| Scenario | Before | After | Improvement |
|---|---|---|---|
| Partial compaction | 6.79s | 3.71s | -45% |
| Full compaction | 8.07s | 3.79s | -53% |
parent 574ac9f0
...@@ -19,28 +19,22 @@ pub fn decompress_into_arc( ...@@ -19,28 +19,22 @@ pub fn decompress_into_arc(
zero-copy mmap path" zero-copy mmap path"
); );
// We directly allocate the buffer in an Arc to avoid copying it into an Arc and avoiding // Allocate directly into an Arc to avoid a copy. The buffer is uninitialized;
// double indirection. This is a dynamically sized arc. // decompression will overwrite it completely (verified by the assert below).
let buffer: Arc<[MaybeUninit<u8>]> = Arc::new_zeroed_slice(uncompressed_length as usize); let buffer: Arc<[MaybeUninit<u8>]> = Arc::new_uninit_slice(uncompressed_length as usize);
// Assume that the buffer is initialized. // Safety: decompression will fully initialize the buffer we verify with an assert
let buffer = Arc::into_raw(buffer); let mut buffer = unsafe { buffer.assume_init() };
// Safety: Assuming that the buffer is initialized is safe because we just created it as // We just created this Arc so refcount is 1; get_mut always succeeds.
// zeroed slice and u8 doesn't require initialization. let decompressed = Arc::get_mut(&mut buffer).expect("Arc refcount should be 1");
let mut buffer = unsafe { Arc::from_raw(buffer as *mut [u8]) }; let bytes_written = if let Some(dict) = compression_dictionary {
// Safety: We know that the buffer is not shared yet.
let decompressed = unsafe { Arc::get_mut_unchecked(&mut buffer) };
let bytes_writes = if let Some(dict) = compression_dictionary {
// Safety: decompress_with_dict will only write to `decompressed` and not read from it.
decompress_with_dict(block, decompressed, dict)? decompress_with_dict(block, decompressed, dict)?
} else { } else {
// Safety: decompress will only write to `decompressed` and not read from it.
decompress(block, decompressed)? decompress(block, decompressed)?
}; };
assert_eq!( assert_eq!(
bytes_writes, uncompressed_length as usize, bytes_written, uncompressed_length as usize,
"Decompressed length does not match expected length" "Decompressed length does not match expected length"
); );
// Safety: The buffer is now fully initialized and can be used.
Ok(buffer) Ok(buffer)
} }
......
#![feature(once_cell_try)] #![feature(once_cell_try)]
#![feature(get_mut_unchecked)]
#![feature(sync_unsafe_cell)] #![feature(sync_unsafe_cell)]
#![feature(iter_collect_into)] #![feature(iter_collect_into)]
......
...@@ -17,17 +17,17 @@ pub enum LookupValue { ...@@ -17,17 +17,17 @@ pub enum LookupValue {
} }
/// A value from a SST file lookup. /// A value from a SST file lookup.
pub enum LazyLookupValue<'l> { pub enum LazyLookupValue {
/// A LookupValue /// A LookupValue
Eager(LookupValue), Eager(LookupValue),
/// A medium sized value that is still compressed. /// A medium sized value that is still compressed.
Medium { Medium {
uncompressed_size: u32, uncompressed_size: u32,
block: &'l [u8], block: ArcBytes,
}, },
} }
impl LazyLookupValue<'_> { impl LazyLookupValue {
/// Returns the size of the value in the SST file. /// Returns the size of the value in the SST file.
pub fn uncompressed_size_in_sst(&self) -> usize { pub fn uncompressed_size_in_sst(&self) -> usize {
match self { match self {
...@@ -74,16 +74,16 @@ impl LazyLookupValue<'_> { ...@@ -74,16 +74,16 @@ impl LazyLookupValue<'_> {
} }
/// An entry from a SST file lookup. /// An entry from a SST file lookup.
pub struct LookupEntry<'l> { pub struct LookupEntry {
/// The hash of the key. /// The hash of the key.
pub hash: u64, pub hash: u64,
/// The key. /// The key.
pub key: ArcBytes, pub key: ArcBytes,
/// The value. /// The value.
pub value: LazyLookupValue<'l>, pub value: LazyLookupValue,
} }
impl Entry for LookupEntry<'_> { impl Entry for LookupEntry {
fn key_hash(&self) -> u64 { fn key_hash(&self) -> u64 {
self.hash self.hash
} }
...@@ -116,7 +116,7 @@ impl Entry for LookupEntry<'_> { ...@@ -116,7 +116,7 @@ impl Entry for LookupEntry<'_> {
block, block,
} => EntryValue::MediumRaw { } => EntryValue::MediumRaw {
uncompressed_size: *uncompressed_size, uncompressed_size: *uncompressed_size,
block, block: block.as_ref(),
}, },
} }
} }
......
...@@ -6,27 +6,30 @@ use crate::lookup_entry::LookupEntry; ...@@ -6,27 +6,30 @@ use crate::lookup_entry::LookupEntry;
/// An active iterator that is being merged. It has peeked the next element and can be compared /// An active iterator that is being merged. It has peeked the next element and can be compared
/// according to that element. The `order` is used when multiple iterators have the same key. /// according to that element. The `order` is used when multiple iterators have the same key.
struct ActiveIterator<'l, T: Iterator<Item = Result<LookupEntry<'l>>>> { ///
/// Boxed so that BinaryHeap sift operations only move a pointer (8 bytes) instead of the full
/// struct (~312 bytes for StaticSortedFileIter), which is significant with 128 iterators.
struct ActiveIterator<T: Iterator<Item = Result<LookupEntry>>> {
iter: T, iter: T,
order: usize, order: usize,
entry: LookupEntry<'l>, entry: LookupEntry,
} }
impl<'l, T: Iterator<Item = Result<LookupEntry<'l>>>> PartialEq for ActiveIterator<'l, T> { impl<T: Iterator<Item = Result<LookupEntry>>> PartialEq for Box<ActiveIterator<T>> {
fn eq(&self, other: &Self) -> bool { fn eq(&self, other: &Self) -> bool {
self.entry.hash == other.entry.hash && *self.entry.key == *other.entry.key self.entry.hash == other.entry.hash && *self.entry.key == *other.entry.key
} }
} }
impl<'l, T: Iterator<Item = Result<LookupEntry<'l>>>> Eq for ActiveIterator<'l, T> {} impl<T: Iterator<Item = Result<LookupEntry>>> Eq for Box<ActiveIterator<T>> {}
impl<'l, T: Iterator<Item = Result<LookupEntry<'l>>>> PartialOrd for ActiveIterator<'l, T> { impl<T: Iterator<Item = Result<LookupEntry>>> PartialOrd for Box<ActiveIterator<T>> {
fn partial_cmp(&self, other: &Self) -> Option<Ordering> { fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
Some(self.cmp(other)) Some(self.cmp(other))
} }
} }
impl<'l, T: Iterator<Item = Result<LookupEntry<'l>>>> Ord for ActiveIterator<'l, T> { impl<T: Iterator<Item = Result<LookupEntry>>> Ord for Box<ActiveIterator<T>> {
fn cmp(&self, other: &Self) -> Ordering { fn cmp(&self, other: &Self) -> Ordering {
self.entry self.entry
.hash .hash
...@@ -39,41 +42,34 @@ impl<'l, T: Iterator<Item = Result<LookupEntry<'l>>>> Ord for ActiveIterator<'l, ...@@ -39,41 +42,34 @@ impl<'l, T: Iterator<Item = Result<LookupEntry<'l>>>> Ord for ActiveIterator<'l,
/// An iterator that merges multiple sorted iterators into a single sorted iterator. Internal it /// An iterator that merges multiple sorted iterators into a single sorted iterator. Internal it
/// uses an heap of iterators to iterate them in order. /// uses an heap of iterators to iterate them in order.
pub struct MergeIter<'l, T: Iterator<Item = Result<LookupEntry<'l>>>> { pub struct MergeIter<T: Iterator<Item = Result<LookupEntry>>> {
heap: BinaryHeap<ActiveIterator<'l, T>>, heap: BinaryHeap<Box<ActiveIterator<T>>>,
} }
impl<'l, T: Iterator<Item = Result<LookupEntry<'l>>>> MergeIter<'l, T> { impl<T: Iterator<Item = Result<LookupEntry>>> MergeIter<T> {
pub fn new(iters: impl Iterator<Item = T>) -> Result<Self> { pub fn new(iters: impl Iterator<Item = T>) -> Result<Self> {
let mut heap = BinaryHeap::new(); let mut heap = BinaryHeap::new();
for (order, mut iter) in iters.enumerate() { for (order, mut iter) in iters.enumerate() {
if let Some(entry) = iter.next() { if let Some(entry) = iter.next() {
let entry = entry?; let entry = entry?;
heap.push(ActiveIterator { iter, order, entry }); heap.push(Box::new(ActiveIterator { iter, order, entry }));
} }
} }
Ok(Self { heap }) Ok(Self { heap })
} }
} }
impl<'l, T: Iterator<Item = Result<LookupEntry<'l>>>> Iterator for MergeIter<'l, T> { impl<T: Iterator<Item = Result<LookupEntry>>> Iterator for MergeIter<T> {
type Item = Result<LookupEntry<'l>>; type Item = Result<LookupEntry>;
fn next(&mut self) -> Option<Self::Item> { fn next(&mut self) -> Option<Self::Item> {
let ActiveIterator { let mut active = self.heap.pop()?;
mut iter, let entry = match active.iter.next() {
order, None => return Some(Ok(active.entry)),
entry,
} = self.heap.pop()?;
match iter.next() {
None => {}
Some(Err(e)) => return Some(Err(e)), Some(Err(e)) => return Some(Err(e)),
Some(Ok(next)) => self.heap.push(ActiveIterator { Some(Ok(next)) => std::mem::replace(&mut active.entry, next),
iter, };
order, self.heap.push(active);
entry: next,
}),
}
Some(Ok(entry)) Some(Ok(entry))
} }
} }
...@@ -129,7 +129,7 @@ impl MetaEntry { ...@@ -129,7 +129,7 @@ impl MetaEntry {
pub fn sst(&self, meta: &MetaFile) -> Result<&StaticSortedFile> { pub fn sst(&self, meta: &MetaFile) -> Result<&StaticSortedFile> {
self.sst.get_or_try_init(|| { self.sst.get_or_try_init(|| {
StaticSortedFile::open(&meta.db_path, self.sst_data.clone()).with_context(|| { StaticSortedFile::open(&meta.db_path, self.sst_data).with_context(|| {
format!( format!(
"Unable to open static sorted file referenced from {:08}.meta", "Unable to open static sorted file referenced from {:08}.meta",
meta.sequence_number() meta.sequence_number()
...@@ -162,6 +162,12 @@ impl MetaEntry { ...@@ -162,6 +162,12 @@ impl MetaEntry {
pub fn block_count(&self) -> u16 { pub fn block_count(&self) -> u16 {
self.sst_data.block_count self.sst_data.block_count
} }
/// Returns the SST metadata needed to open the file independently.
/// Used during compaction to avoid caching mmaps on the MetaEntry.
pub fn sst_metadata(&self) -> StaticSortedFileMetaData {
self.sst_data
}
} }
/// The result of a lookup operation. /// The result of a lookup operation.
......
...@@ -33,6 +33,12 @@ impl ValueBlockCountTracker { ...@@ -33,6 +33,12 @@ impl ValueBlockCountTracker {
>= MAX_VALUE_BLOCK_COUNT >= MAX_VALUE_BLOCK_COUNT
} }
/// Returns true if the tracked value block count has reached half of the maximum.
pub fn is_half_full(&self) -> bool {
self.value_block_count + (self.current_small_value_block_size > 0) as usize
>= MAX_VALUE_BLOCK_COUNT / 2
}
/// Reset the tracker to empty. /// Reset the tracker to empty.
pub fn reset(&mut self) { pub fn reset(&mut self) {
self.value_block_count = 0; self.value_block_count = 0;
......
...@@ -1271,7 +1271,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> { ...@@ -1271,7 +1271,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
persisted_task_cache_log, persisted_task_cache_log,
task_snapshots, task_snapshots,
) { ) {
println!("Persisting failed: {err:?}"); eprintln!("Persisting failed: {err:?}");
return None; return None;
} }
#[cfg(feature = "print_cache_item_size")] #[cfg(feature = "print_cache_item_size")]
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment