Unverified Commit 09135201 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

[turbopack] Reduce memory usage during compaction (#90229)

### What?

Optimizes memory usage, I/O patterns, and CPU efficiency during database compaction by:

* clearing block caches and SST mmaps before compaction starts
* opening SST files independently with madvise(SEQUENTIAL) for compaction
* making the SST iterator own the file so mmaps are dropped as as soon as the iterator is are consumed
* moving used-key-hash filter deserialization into per-family scope, 
    * also mering AMQF filters into a single pre-sized filter per family, reducing used-key classification from O(N filters) to O(1) per entry
* flushing entries early once collectors reach 50% capacity to reduce peak overhead while still avoiding small files
* adding a lightweight single-entry value block cache on the iterator (replacing the global `BlockCache` for sequential scans, this reaps all the benefits while not accumulating too many blocks that will never be visited again)
* boxing MergeIter heap elements so BinaryHeap sifts swap 8-byte pointers instead of ~312-byte structs

### Why?

Database compaction was consuming excessive memory by maintaining block caches and memory-mapped files that weren't needed during the sequential merge operations. The existing approach used random access patterns optimized for lookups, but compaction performs sequential scans that benefit from different memory management strategies.

Profiling revealed two additional CPU bottlenecks:
1. **AMQF query overhead (44% of compaction time):** With 128 commits, `compact_internal` iterated all 128 AMQF filters per entry (16M entries × 128 filters = ~2B probes). Most filters were empty in write-only workloads but still deserialized. Filtering empty filters and merging the rest into a single filter eliminates this entirely.
2. **MergeIter heap sift overhead (21% inclusive):** `ActiveIterator<StaticSortedFileIter>` is ~312 bytes. BinaryHeap sifts moved these large structs at each level (log2(128) = 7 levels per pop/push). Boxing reduces each swap to 8 bytes, cutting MergeIter inclusive time from 21% to 12%.

### Performance results

Compaction benchmark (16Mi entries, 128 commits):

| Scenario | Before | After | Improvement |
|---|---|---|---|
| Partial compaction | 6.79s | 3.71s | -45% |
| Full compaction | 8.07s | 3.79s | -53% |
parent 574ac9f0
......@@ -19,28 +19,22 @@ pub fn decompress_into_arc(
zero-copy mmap path"
);
// We directly allocate the buffer in an Arc to avoid copying it into an Arc and avoiding
// double indirection. This is a dynamically sized arc.
let buffer: Arc<[MaybeUninit<u8>]> = Arc::new_zeroed_slice(uncompressed_length as usize);
// Assume that the buffer is initialized.
let buffer = Arc::into_raw(buffer);
// Safety: Assuming that the buffer is initialized is safe because we just created it as
// zeroed slice and u8 doesn't require initialization.
let mut buffer = unsafe { Arc::from_raw(buffer as *mut [u8]) };
// Safety: We know that the buffer is not shared yet.
let decompressed = unsafe { Arc::get_mut_unchecked(&mut buffer) };
let bytes_writes = if let Some(dict) = compression_dictionary {
// Safety: decompress_with_dict will only write to `decompressed` and not read from it.
// Allocate directly into an Arc to avoid a copy. The buffer is uninitialized;
// decompression will overwrite it completely (verified by the assert below).
let buffer: Arc<[MaybeUninit<u8>]> = Arc::new_uninit_slice(uncompressed_length as usize);
// Safety: decompression will fully initialize the buffer we verify with an assert
let mut buffer = unsafe { buffer.assume_init() };
// We just created this Arc so refcount is 1; get_mut always succeeds.
let decompressed = Arc::get_mut(&mut buffer).expect("Arc refcount should be 1");
let bytes_written = if let Some(dict) = compression_dictionary {
decompress_with_dict(block, decompressed, dict)?
} else {
// Safety: decompress will only write to `decompressed` and not read from it.
decompress(block, decompressed)?
};
assert_eq!(
bytes_writes, uncompressed_length as usize,
bytes_written, uncompressed_length as usize,
"Decompressed length does not match expected length"
);
// Safety: The buffer is now fully initialized and can be used.
Ok(buffer)
}
......
#![feature(once_cell_try)]
#![feature(get_mut_unchecked)]
#![feature(sync_unsafe_cell)]
#![feature(iter_collect_into)]
......
......@@ -17,17 +17,17 @@ pub enum LookupValue {
}
/// A value from a SST file lookup.
pub enum LazyLookupValue<'l> {
pub enum LazyLookupValue {
/// A LookupValue
Eager(LookupValue),
/// A medium sized value that is still compressed.
Medium {
uncompressed_size: u32,
block: &'l [u8],
block: ArcBytes,
},
}
impl LazyLookupValue<'_> {
impl LazyLookupValue {
/// Returns the size of the value in the SST file.
pub fn uncompressed_size_in_sst(&self) -> usize {
match self {
......@@ -74,16 +74,16 @@ impl LazyLookupValue<'_> {
}
/// An entry from a SST file lookup.
pub struct LookupEntry<'l> {
pub struct LookupEntry {
/// The hash of the key.
pub hash: u64,
/// The key.
pub key: ArcBytes,
/// The value.
pub value: LazyLookupValue<'l>,
pub value: LazyLookupValue,
}
impl Entry for LookupEntry<'_> {
impl Entry for LookupEntry {
fn key_hash(&self) -> u64 {
self.hash
}
......@@ -116,7 +116,7 @@ impl Entry for LookupEntry<'_> {
block,
} => EntryValue::MediumRaw {
uncompressed_size: *uncompressed_size,
block,
block: block.as_ref(),
},
}
}
......
......@@ -6,27 +6,30 @@ use crate::lookup_entry::LookupEntry;
/// An active iterator that is being merged. It has peeked the next element and can be compared
/// according to that element. The `order` is used when multiple iterators have the same key.
struct ActiveIterator<'l, T: Iterator<Item = Result<LookupEntry<'l>>>> {
///
/// Boxed so that BinaryHeap sift operations only move a pointer (8 bytes) instead of the full
/// struct (~312 bytes for StaticSortedFileIter), which is significant with 128 iterators.
struct ActiveIterator<T: Iterator<Item = Result<LookupEntry>>> {
iter: T,
order: usize,
entry: LookupEntry<'l>,
entry: LookupEntry,
}
impl<'l, T: Iterator<Item = Result<LookupEntry<'l>>>> PartialEq for ActiveIterator<'l, T> {
impl<T: Iterator<Item = Result<LookupEntry>>> PartialEq for Box<ActiveIterator<T>> {
fn eq(&self, other: &Self) -> bool {
self.entry.hash == other.entry.hash && *self.entry.key == *other.entry.key
}
}
impl<'l, T: Iterator<Item = Result<LookupEntry<'l>>>> Eq for ActiveIterator<'l, T> {}
impl<T: Iterator<Item = Result<LookupEntry>>> Eq for Box<ActiveIterator<T>> {}
impl<'l, T: Iterator<Item = Result<LookupEntry<'l>>>> PartialOrd for ActiveIterator<'l, T> {
impl<T: Iterator<Item = Result<LookupEntry>>> PartialOrd for Box<ActiveIterator<T>> {
fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
Some(self.cmp(other))
}
}
impl<'l, T: Iterator<Item = Result<LookupEntry<'l>>>> Ord for ActiveIterator<'l, T> {
impl<T: Iterator<Item = Result<LookupEntry>>> Ord for Box<ActiveIterator<T>> {
fn cmp(&self, other: &Self) -> Ordering {
self.entry
.hash
......@@ -39,41 +42,34 @@ impl<'l, T: Iterator<Item = Result<LookupEntry<'l>>>> Ord for ActiveIterator<'l,
/// An iterator that merges multiple sorted iterators into a single sorted iterator. Internal it
/// uses an heap of iterators to iterate them in order.
pub struct MergeIter<'l, T: Iterator<Item = Result<LookupEntry<'l>>>> {
heap: BinaryHeap<ActiveIterator<'l, T>>,
pub struct MergeIter<T: Iterator<Item = Result<LookupEntry>>> {
heap: BinaryHeap<Box<ActiveIterator<T>>>,
}
impl<'l, T: Iterator<Item = Result<LookupEntry<'l>>>> MergeIter<'l, T> {
impl<T: Iterator<Item = Result<LookupEntry>>> MergeIter<T> {
pub fn new(iters: impl Iterator<Item = T>) -> Result<Self> {
let mut heap = BinaryHeap::new();
for (order, mut iter) in iters.enumerate() {
if let Some(entry) = iter.next() {
let entry = entry?;
heap.push(ActiveIterator { iter, order, entry });
heap.push(Box::new(ActiveIterator { iter, order, entry }));
}
}
Ok(Self { heap })
}
}
impl<'l, T: Iterator<Item = Result<LookupEntry<'l>>>> Iterator for MergeIter<'l, T> {
type Item = Result<LookupEntry<'l>>;
impl<T: Iterator<Item = Result<LookupEntry>>> Iterator for MergeIter<T> {
type Item = Result<LookupEntry>;
fn next(&mut self) -> Option<Self::Item> {
let ActiveIterator {
mut iter,
order,
entry,
} = self.heap.pop()?;
match iter.next() {
None => {}
let mut active = self.heap.pop()?;
let entry = match active.iter.next() {
None => return Some(Ok(active.entry)),
Some(Err(e)) => return Some(Err(e)),
Some(Ok(next)) => self.heap.push(ActiveIterator {
iter,
order,
entry: next,
}),
}
Some(Ok(next)) => std::mem::replace(&mut active.entry, next),
};
self.heap.push(active);
Some(Ok(entry))
}
}
......@@ -129,7 +129,7 @@ impl MetaEntry {
pub fn sst(&self, meta: &MetaFile) -> Result<&StaticSortedFile> {
self.sst.get_or_try_init(|| {
StaticSortedFile::open(&meta.db_path, self.sst_data.clone()).with_context(|| {
StaticSortedFile::open(&meta.db_path, self.sst_data).with_context(|| {
format!(
"Unable to open static sorted file referenced from {:08}.meta",
meta.sequence_number()
......@@ -162,6 +162,12 @@ impl MetaEntry {
pub fn block_count(&self) -> u16 {
self.sst_data.block_count
}
/// Returns the SST metadata needed to open the file independently.
/// Used during compaction to avoid caching mmaps on the MetaEntry.
pub fn sst_metadata(&self) -> StaticSortedFileMetaData {
self.sst_data
}
}
/// The result of a lookup operation.
......
......@@ -33,6 +33,12 @@ impl ValueBlockCountTracker {
>= MAX_VALUE_BLOCK_COUNT
}
/// Returns true if the tracked value block count has reached half of the maximum.
pub fn is_half_full(&self) -> bool {
self.value_block_count + (self.current_small_value_block_size > 0) as usize
>= MAX_VALUE_BLOCK_COUNT / 2
}
/// Reset the tracker to empty.
pub fn reset(&mut self) {
self.value_block_count = 0;
......
......@@ -1271,7 +1271,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
persisted_task_cache_log,
task_snapshots,
) {
println!("Persisting failed: {err:?}");
eprintln!("Persisting failed: {err:?}");
return None;
}
#[cfg(feature = "print_cache_item_size")]
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment