Unverified Commit 8e9f9514 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

[turbopack] Optimize compaction cpu usage (#91468)

## Summary

Optimizes the turbo-persistence compaction and iteration paths with several targeted improvements:

### Iterator optimizations

- **Flatten index block iteration** — The iterator previously used a `Vec<CurrentIndexBlock>` stack, but SST files have exactly one index level. Inline the index block fields (`index_entries`, `index_block_count`, `index_pos`) directly into `StaticSortedFileIter`, eliminating the stack allocation and `Option` overhead.

- **Non-optional `CurrentKeyBlock`** — Parse the first key block during `try_into_iter()` construction so `current_key_block` is always populated, removing the `Option<CurrentKeyBlock>` wrapper and its `take()`/`Some()` ceremony in the hot loop.

- **Replace `ReadBytesExt` with direct byte indexing** — In `handle_key_match`, `parse_key_block`, and `next_internal`, replace `val.read_u16::<BE>()` etc. with `u16::from_be_bytes(val[0..2].try_into().unwrap())`. This eliminates th mutable slice pointer advancement.

- **Extract `read_offset_entry` helper** — Read type + offset from the key block offset table in a single `u32` load + shift, replacing two separate `ReadBytesExt` calls.


### Refcounting optimization

- **Introduce `RcBytes`** — Thread-local byte slice type using `Rc` instead of `Arc`, eliminating atomic refcount overhead during single-threaded SST iteration. The iteration path (`StaticSortedFileIter`) now produces `RcBytes` slices backed by an `Rc<Mmap>`, so per-entry clone/drop operations are plain integer increments rather than atomic operations.


### Merge iterator simplification

- **Optimize `MergeIter::next`** — Replaced the straightforwards `pop/push` pattern with `PeekMut`-based replace-top pattern, which means we only need to adjust the heap once per iteration instead of twice.

## Benchmark results

### Compaction (`key_8/value_4/entries_16.00Mi/commits_128`)

| Benchmark | Canary | Optimized | Change |
|-----------|--------|-----------|--------|
| partial compaction | 1.949 s | 1.515 s | **-22%** |
| full compaction | 2.051 s | 1.542 s | **-25%** |

### Read path (`static_sorted_file_lookup/entries_1000.00Ki`)

No read regression — the branch is neutral to slightly faster:

| Benchmark | Canary | Optimized | Change |
|-----------|--------|-----------|--------|
| hit/uncached | 6.73 µs | 6.59 µs | **-2%** |
| hit/cached | 140.8 ns | 130.7 ns | **-7%** |
| miss/uncached | 5.10 µs | 5.02 µs | **-2%** |
| miss/cached | 230.1 ns | 233.1 ns | ~+1% (noise) |

## Test plan

- [x] `cargo test -p turbo-persistence` — 60/60 tests passing
- [x] Compaction benchmarks run and compared against canary baseline
- [x] Read path (lookup) benchmarks verified no regression
parent af97e756
...@@ -2,13 +2,16 @@ use std::{ ...@@ -2,13 +2,16 @@ use std::{
borrow::Borrow, borrow::Borrow,
fmt::{self, Debug, Formatter}, fmt::{self, Debug, Formatter},
hash::{Hash, Hasher}, hash::{Hash, Hasher},
io::{self, Read},
ops::{Deref, Range}, ops::{Deref, Range},
sync::Arc, sync::Arc,
}; };
use memmap2::Mmap; use memmap2::Mmap;
use crate::{
compression::decompress_into_arc,
shared_bytes::{SharedBytes, is_subslice_of},
};
/// The backing storage for an `ArcBytes`. /// The backing storage for an `ArcBytes`.
/// ///
/// The inner values are never read directly — they exist solely to keep the /// The inner values are never read directly — they exist solely to keep the
...@@ -78,27 +81,17 @@ impl Debug for ArcBytes { ...@@ -78,27 +81,17 @@ impl Debug for ArcBytes {
impl Eq for ArcBytes {} impl Eq for ArcBytes {}
impl Read for ArcBytes { impl ArcBytes {
fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> { /// Returns `true` if this `ArcBytes` is backed by a memory-mapped file.
let available = &**self; pub fn is_mmap_backed(&self) -> bool {
let len = std::cmp::min(buf.len(), available.len()); matches!(self.backing, Backing::Mmap { .. })
buf[..len].copy_from_slice(&available[..len]);
// Advance the slice view
self.data = &available[len..] as *const [u8];
Ok(len)
} }
} }
/// Returns `true` if `subslice` lies entirely within `backing`. impl SharedBytes for ArcBytes {
fn is_subslice_of(subslice: &[u8], backing: &[u8]) -> bool { type MmapHandle = Arc<Mmap>;
let backing = backing.as_ptr_range();
let sub = subslice.as_ptr_range();
sub.start >= backing.start && sub.end <= backing.end
}
impl ArcBytes { fn slice(self, range: Range<usize>) -> Self {
/// Returns a new `ArcBytes` that points to a sub-range of the current slice.
pub fn slice(self, range: Range<usize>) -> ArcBytes {
let data = &*self; let data = &*self;
let data = &data[range] as *const [u8]; let data = &data[range] as *const [u8];
Self { Self {
...@@ -107,14 +100,7 @@ impl ArcBytes { ...@@ -107,14 +100,7 @@ impl ArcBytes {
} }
} }
/// Creates a sub-slice from a slice reference that points into this ArcBytes' backing data. unsafe fn slice_from_subslice(&self, subslice: &[u8]) -> Self {
///
/// # Safety
///
/// The caller must ensure that `subslice` points to memory within this ArcBytes'
/// backing storage (not just within the current slice view, but anywhere in the original
/// backing data).
pub unsafe fn slice_from_subslice(&self, subslice: &[u8]) -> ArcBytes {
debug_assert!( debug_assert!(
is_subslice_of( is_subslice_of(
subslice, subslice,
...@@ -131,24 +117,23 @@ impl ArcBytes { ...@@ -131,24 +117,23 @@ impl ArcBytes {
} }
} }
/// Creates an `ArcBytes` backed by a memory-mapped file. unsafe fn from_mmap(mmap: &Arc<Mmap>, subslice: &[u8]) -> Self {
///
/// # Safety
///
/// The caller must ensure that `subslice` points to memory within the given `mmap`.
pub unsafe fn from_mmap(mmap: Arc<Mmap>, subslice: &[u8]) -> ArcBytes {
debug_assert!( debug_assert!(
is_subslice_of(subslice, &mmap), is_subslice_of(subslice, mmap),
"from_mmap: subslice is not within the mmap" "from_mmap: subslice is not within the mmap"
); );
ArcBytes { ArcBytes {
data: subslice as *const [u8], data: subslice as *const [u8],
backing: Backing::Mmap { _backing: mmap }, backing: Backing::Mmap {
_backing: mmap.clone(),
},
} }
} }
/// Returns `true` if this `ArcBytes` is backed by a memory-mapped file. fn from_decompressed(uncompressed_length: u32, block: &[u8]) -> anyhow::Result<Self> {
pub fn is_mmap_backed(&self) -> bool { Ok(ArcBytes::from(decompress_into_arc(
matches!(self.backing, Backing::Mmap { .. }) uncompressed_length,
block,
)?))
} }
} }
//! Big-endian integer parsing helpers.
//!
//! Each function reads a fixed-width big-endian integer from the prefix of
//! a byte slice. Panics if the slice is too short (single bounds check via
//! `first_chunk`).
#[inline(always)]
pub fn read_u8(s: &[u8]) -> u8 {
s[0]
}
#[inline(always)]
pub fn read_u16(s: &[u8]) -> u16 {
u16::from_be_bytes(*s.first_chunk().unwrap())
}
#[inline(always)]
pub fn read_u24(s: &[u8]) -> u32 {
let &[a, b, c] = s.first_chunk().unwrap();
u32::from_be_bytes([0, a, b, c])
}
#[inline(always)]
pub fn read_u32(s: &[u8]) -> u32 {
u32::from_be_bytes(*s.first_chunk().unwrap())
}
#[inline(always)]
pub fn read_u64(s: &[u8]) -> u64 {
u64::from_be_bytes(*s.first_chunk().unwrap())
}
use std::{mem::MaybeUninit, sync::Arc}; use std::{mem::MaybeUninit, rc::Rc, sync::Arc};
use anyhow::{Context, Result}; use anyhow::{Context, Result};
use lzzzz::lz4::{self, decompress}; use lzzzz::lz4::{self, decompress};
/// Decompresses a block into an Arc allocation. /// Decompresses `block` into `dest`, verifying the output length matches `expected_len`.
/// fn decompress_block(block: &[u8], dest: &mut [u8], expected_len: u32) -> Result<()> {
/// The caller must ensure `uncompressed_length > 0` (i.e., the block is actually compressed).
/// Uncompressed blocks should be handled via zero-copy mmap slices before calling this.
pub fn decompress_into_arc(uncompressed_length: u32, block: &[u8]) -> Result<Arc<[u8]>> {
debug_assert!( debug_assert!(
uncompressed_length > 0, expected_len > 0,
"decompress_into_arc called with uncompressed_length=0; uncompressed blocks should use \ "decompress_block called with uncompressed_length=0; uncompressed blocks should use \
zero-copy mmap path" zero-copy mmap path"
); );
let bytes_written = decompress(block, dest).with_context(|| {
// Allocate directly into an Arc to avoid a copy. The buffer is uninitialized;
// decompression will overwrite it completely (verified by the assert below).
let buffer: Arc<[MaybeUninit<u8>]> = Arc::new_uninit_slice(uncompressed_length as usize);
// Safety: decompression will fully initialize the buffer we verify with an assert
let mut buffer = unsafe { buffer.assume_init() };
// We just created this Arc so refcount is 1; get_mut always succeeds.
let decompressed = Arc::get_mut(&mut buffer).expect("Arc refcount should be 1");
let bytes_written = decompress(block, decompressed).with_context(|| {
format!( format!(
"Failed to decompress block ({} bytes compressed, {} bytes uncompressed)", "Failed to decompress block ({} bytes compressed, {} bytes uncompressed)",
block.len(), block.len(),
uncompressed_length expected_len
) )
})?; })?;
assert_eq!( assert_eq!(
bytes_written, uncompressed_length as usize, bytes_written, expected_len as usize,
"Decompressed length does not match expected length" "Decompressed length does not match expected length"
); );
Ok(())
}
/// Decompresses a block into an Arc allocation.
///
/// The caller must ensure `uncompressed_length > 0` (i.e., the block is actually compressed).
/// Uncompressed blocks should be handled via zero-copy mmap slices before calling this.
pub fn decompress_into_arc(uncompressed_length: u32, block: &[u8]) -> Result<Arc<[u8]>> {
// Allocate directly into an Arc to avoid a copy. The buffer is uninitialized;
// decompression will overwrite it completely (verified by decompress_block).
let buffer: Arc<[MaybeUninit<u8>]> = Arc::new_uninit_slice(uncompressed_length as usize);
// Safety: decompression will fully initialize the buffer (verified by the assert in
// decompress_block).
let mut buffer = unsafe { buffer.assume_init() };
// We just created this Arc so refcount is 1; get_mut always succeeds.
let dest = Arc::get_mut(&mut buffer).expect("Arc refcount should be 1");
decompress_block(block, dest, uncompressed_length)?;
Ok(buffer)
}
/// Like [`decompress_into_arc`] but returns an `Rc<[u8]>` for thread-local use.
pub fn decompress_into_rc(uncompressed_length: u32, block: &[u8]) -> Result<Rc<[u8]>> {
let buffer: Rc<[MaybeUninit<u8>]> = Rc::new_uninit_slice(uncompressed_length as usize);
// Safety: decompression will fully initialize the buffer (verified by the assert in
// decompress_block).
let mut buffer = unsafe { buffer.assume_init() };
let dest = Rc::get_mut(&mut buffer).expect("Rc refcount should be 1");
decompress_block(block, dest, uncompressed_length)?;
Ok(buffer) Ok(buffer)
} }
......
...@@ -30,14 +30,15 @@ use crate::{ ...@@ -30,14 +30,15 @@ use crate::{
MAX_ENTRIES_PER_COMPACTED_FILE, VALUE_BLOCK_AVG_SIZE, VALUE_BLOCK_CACHE_SIZE, MAX_ENTRIES_PER_COMPACTED_FILE, VALUE_BLOCK_AVG_SIZE, VALUE_BLOCK_CACHE_SIZE,
}, },
key::{StoreKey, hash_key}, key::{StoreKey, hash_key},
lookup_entry::{LazyLookupValue, LookupEntry, LookupValue}, lookup_entry::{IterValue, LookupEntry, LookupValue},
merge_iter::MergeIter, merge_iter::MergeIter,
meta_file::{MetaEntryFlags, MetaFile, MetaLookupResult, StaticSortedFileRange}, meta_file::{MetaEntryFlags, MetaFile, MetaLookupResult, StaticSortedFileRange},
meta_file_builder::MetaFileBuilder, meta_file_builder::MetaFileBuilder,
mmap_helper::advise_mmap_for_persistence, mmap_helper::advise_mmap_for_persistence,
parallel_scheduler::ParallelScheduler, parallel_scheduler::ParallelScheduler,
rc_bytes::RcBytes,
sst_filter::SstFilter, sst_filter::SstFilter,
static_sorted_file::{BlockCache, SstLookupResult, StaticSortedFile}, static_sorted_file::{BlockCache, SstLookupResult, StaticSortedFileIter},
static_sorted_file_builder::{StaticSortedFileBuilderMeta, StreamingSstWriter}, static_sorted_file_builder::{StaticSortedFileBuilderMeta, StreamingSstWriter},
write_batch::{FinishResult, WriteBatch}, write_batch::{FinishResult, WriteBatch},
}; };
...@@ -1075,11 +1076,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES> ...@@ -1075,11 +1076,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES>
let meta_index = ssts_with_ranges[index].meta_index; let meta_index = ssts_with_ranges[index].meta_index;
let index_in_meta = ssts_with_ranges[index].index_in_meta; let index_in_meta = ssts_with_ranges[index].index_in_meta;
let entry = meta_files[meta_index].entry(index_in_meta); let entry = meta_files[meta_index].entry(index_in_meta);
StaticSortedFile::open_for_compaction( StaticSortedFileIter::open(path, entry.sst_metadata())
path,
entry.sst_metadata(),
)?
.try_into_iter()
}) })
.collect::<Result<Vec<_>>>()?; .collect::<Result<Vec<_>>>()?;
...@@ -1188,7 +1185,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES> ...@@ -1188,7 +1185,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES>
} }
let mut used_collector = Collector::new(MetaEntryFlags::WARM); let mut used_collector = Collector::new(MetaEntryFlags::WARM);
let mut unused_collector = Collector::new(MetaEntryFlags::COLD); let mut unused_collector = Collector::new(MetaEntryFlags::COLD);
let mut current_key: Option<ArcBytes> = None; let mut current_key: Option<RcBytes> = None;
let mut keys_written = 0; let mut keys_written = 0;
// MergeIter yields entries from newer SSTs first (by SST sequence // MergeIter yields entries from newer SSTs first (by SST sequence
...@@ -1220,10 +1217,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES> ...@@ -1220,10 +1217,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES>
FamilyKind::MultiValue => { FamilyKind::MultiValue => {
// For MultiValue families we only skip remaining if we // For MultiValue families we only skip remaining if we
// see a tombstone // see a tombstone
if matches!( if matches!(entry.value, IterValue::Deleted) {
entry.value,
LazyLookupValue::Eager(LookupValue::Deleted)
) {
skip_remaining_for_this_key = true; skip_remaining_for_this_key = true;
} }
} }
...@@ -1243,10 +1237,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES> ...@@ -1243,10 +1237,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES>
// Entry is being dropped (superseded by newer entry or // Entry is being dropped (superseded by newer entry or
// pruned by tombstone). If it references a blob file, // pruned by tombstone). If it references a blob file,
// mark that blob for deletion. // mark that blob for deletion.
if let LazyLookupValue::Eager(LookupValue::Blob { if let IterValue::Blob { sequence_number } = &entry.value {
sequence_number,
}) = &entry.value
{
blob_seq_numbers_to_delete.push(*sequence_number); blob_seq_numbers_to_delete.push(*sequence_number);
} }
} }
......
...@@ -3,6 +3,7 @@ ...@@ -3,6 +3,7 @@
#![feature(iter_collect_into)] #![feature(iter_collect_into)]
mod arc_bytes; mod arc_bytes;
pub(crate) mod be;
mod collector; mod collector;
mod collector_entry; mod collector_entry;
mod compaction; mod compaction;
...@@ -16,6 +17,8 @@ pub mod meta_file; ...@@ -16,6 +17,8 @@ pub mod meta_file;
mod meta_file_builder; mod meta_file_builder;
pub mod mmap_helper; pub mod mmap_helper;
mod parallel_scheduler; mod parallel_scheduler;
mod rc_bytes;
mod shared_bytes;
mod sst_filter; mod sst_filter;
pub mod static_sorted_file; pub mod static_sorted_file;
mod static_sorted_file_builder; mod static_sorted_file_builder;
......
use crate::{ use crate::{
ArcBytes, ArcBytes,
constants::{MAX_INLINE_VALUE_SIZE, MAX_SMALL_VALUE_SIZE}, constants::{MAX_INLINE_VALUE_SIZE, MAX_SMALL_VALUE_SIZE},
rc_bytes::RcBytes,
static_sorted_file_builder::{Entry, EntryValue}, static_sorted_file_builder::{Entry, EntryValue},
}; };
/// A value from a SST file lookup. /// A value from a SST file. Generic over the byte representation, defaulting to
/// `ArcBytes` for the lookup path. The compaction/iteration path uses
/// `LookupValue<RcBytes>` which is convertible to `IterValue`.
#[derive(PartialEq)] #[derive(PartialEq)]
pub enum LookupValue { pub enum LookupValue<B = ArcBytes> {
/// The value was deleted. /// The value was deleted.
Deleted, Deleted,
/// The value is stored in the SST file. /// The value is stored in the SST file.
/// ///
/// The ArcBytes will be pointing either at a keyblock or a value block in the SST /// The bytes will be pointing either at a keyblock or a value block in the SST
Slice { value: ArcBytes }, Slice { value: B },
/// The value is stored in a blob file. /// The value is stored in a blob file.
Blob { sequence_number: u32 }, Blob { sequence_number: u32 },
} }
/// A value from a SST file lookup. /// A value from SST file iteration (compaction path, uses RcBytes for
pub enum LazyLookupValue { /// non-atomic refcounting).
/// A LookupValue pub enum IterValue {
Eager(LookupValue), /// The value was deleted.
Deleted,
/// The value is stored in the SST file.
Slice { value: RcBytes },
/// The value is stored in a blob file.
Blob { sequence_number: u32 },
/// A medium sized value that is still compressed. /// A medium sized value that is still compressed.
Medium { Medium {
uncompressed_size: u32, uncompressed_size: u32,
checksum: u32, checksum: u32,
block: ArcBytes, block: RcBytes,
}, },
} }
impl From<LookupValue<RcBytes>> for IterValue {
impl LazyLookupValue { fn from(v: LookupValue<RcBytes>) -> Self {
/// Returns the size of the value in the SST file. match v {
pub fn uncompressed_size_in_sst(&self) -> usize { LookupValue::Deleted => IterValue::Deleted,
match self { LookupValue::Slice { value } => IterValue::Slice { value },
LazyLookupValue::Eager(LookupValue::Slice { value }) => value.len(), LookupValue::Blob { sequence_number } => IterValue::Blob { sequence_number },
LazyLookupValue::Eager(LookupValue::Deleted) => 0,
LazyLookupValue::Eager(LookupValue::Blob { .. }) => 0,
LazyLookupValue::Medium {
uncompressed_size,
block,
..
} => {
if *uncompressed_size == 0 {
block.len()
} else {
*uncompressed_size as usize
}
}
}
}
/// Returns true if this value gets its own dedicated value block.
pub fn is_medium_value(&self) -> bool {
match self {
LazyLookupValue::Eager(LookupValue::Slice { value })
if value.len() > MAX_SMALL_VALUE_SIZE =>
{
true
}
LazyLookupValue::Medium { .. } => true,
_ => false,
}
}
/// Returns the value size if it will be packed into a small value block, or 0 otherwise.
pub fn small_value_size(&self) -> usize {
match self {
LazyLookupValue::Eager(LookupValue::Slice { value })
if value.len() > MAX_INLINE_VALUE_SIZE && value.len() <= MAX_SMALL_VALUE_SIZE =>
{
value.len()
}
_ => 0,
} }
} }
} }
/// An entry from SST file iteration (compaction path, uses RcBytes).
/// An entry from a SST file lookup.
pub struct LookupEntry { pub struct LookupEntry {
/// The hash of the key. /// The hash of the key.
pub hash: u64, pub hash: u64,
/// The key. /// The key.
pub key: ArcBytes, pub key: RcBytes,
/// The value. /// The value.
pub value: LazyLookupValue, pub value: IterValue,
} }
impl Entry for LookupEntry { impl Entry for LookupEntry {
...@@ -101,8 +70,8 @@ impl Entry for LookupEntry { ...@@ -101,8 +70,8 @@ impl Entry for LookupEntry {
fn value(&self) -> EntryValue<'_> { fn value(&self) -> EntryValue<'_> {
match &self.value { match &self.value {
LazyLookupValue::Eager(LookupValue::Deleted) => EntryValue::Deleted, IterValue::Deleted => EntryValue::Deleted,
LazyLookupValue::Eager(LookupValue::Slice { value }) => { IterValue::Slice { value } => {
if value.len() <= MAX_INLINE_VALUE_SIZE { if value.len() <= MAX_INLINE_VALUE_SIZE {
EntryValue::Inline { value } EntryValue::Inline { value }
} else if value.len() > MAX_SMALL_VALUE_SIZE { } else if value.len() > MAX_SMALL_VALUE_SIZE {
...@@ -111,10 +80,10 @@ impl Entry for LookupEntry { ...@@ -111,10 +80,10 @@ impl Entry for LookupEntry {
EntryValue::Small { value } EntryValue::Small { value }
} }
} }
LazyLookupValue::Eager(LookupValue::Blob { sequence_number }) => EntryValue::Large { IterValue::Blob { sequence_number } => EntryValue::Large {
blob: *sequence_number, blob: *sequence_number,
}, },
LazyLookupValue::Medium { IterValue::Medium {
uncompressed_size, uncompressed_size,
checksum, checksum,
block, block,
......
use std::{cmp::Ordering, collections::BinaryHeap}; use std::{
cmp::Ordering,
collections::{BinaryHeap, binary_heap::PeekMut},
};
use anyhow::Result; use anyhow::Result;
...@@ -6,45 +9,52 @@ use crate::lookup_entry::LookupEntry; ...@@ -6,45 +9,52 @@ use crate::lookup_entry::LookupEntry;
/// An active iterator that is being merged. It has peeked the next element and can be compared /// An active iterator that is being merged. It has peeked the next element and can be compared
/// according to that element. The `order` is used when multiple iterators have the same key. /// according to that element. The `order` is used when multiple iterators have the same key.
///
/// Boxed so that BinaryHeap sift operations only move a pointer (8 bytes) instead of the full
/// struct (~312 bytes for StaticSortedFileIter), which is significant with 128 iterators.
struct ActiveIterator<T: Iterator<Item = Result<LookupEntry>>> { struct ActiveIterator<T: Iterator<Item = Result<LookupEntry>>> {
iter: T, iter: T,
order: usize, order: usize,
entry: LookupEntry, entry: LookupEntry,
} }
impl<T: Iterator<Item = Result<LookupEntry>>> PartialEq for Box<ActiveIterator<T>> { /// A heap node that keeps the hash inline alongside a boxed `ActiveIterator`. By hoisting the hash
/// out of the Box we avoid a pointer chase on every comparison when the heap is re-ordered Only
/// when hashes collide (extremely rare with u64) do we dereference the Box to compare keys.
/// Note: we cannot use [Prehashed] because we need an non-trivial `Ord` implementation for the heap
/// (see below)
struct HeapNode<T: Iterator<Item = Result<LookupEntry>>> {
/// Cached copy of the current entry's hash, kept in sync with `inner.entry.hash`.
hash: u64,
inner: Box<ActiveIterator<T>>,
}
impl<T: Iterator<Item = Result<LookupEntry>>> PartialEq for HeapNode<T> {
fn eq(&self, other: &Self) -> bool { fn eq(&self, other: &Self) -> bool {
self.entry.hash == other.entry.hash && *self.entry.key == *other.entry.key self.hash == other.hash && *self.inner.entry.key == *other.inner.entry.key
} }
} }
impl<T: Iterator<Item = Result<LookupEntry>>> Eq for Box<ActiveIterator<T>> {} impl<T: Iterator<Item = Result<LookupEntry>>> Eq for HeapNode<T> {}
impl<T: Iterator<Item = Result<LookupEntry>>> PartialOrd for Box<ActiveIterator<T>> { impl<T: Iterator<Item = Result<LookupEntry>>> PartialOrd for HeapNode<T> {
fn partial_cmp(&self, other: &Self) -> Option<Ordering> { fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
Some(self.cmp(other)) Some(self.cmp(other))
} }
} }
impl<T: Iterator<Item = Result<LookupEntry>>> Ord for Box<ActiveIterator<T>> { impl<T: Iterator<Item = Result<LookupEntry>>> Ord for HeapNode<T> {
fn cmp(&self, other: &Self) -> Ordering { fn cmp(&self, other: &Self) -> Ordering {
self.entry self.hash
.hash .cmp(&other.hash)
.cmp(&other.entry.hash) .then_with(|| (*self.inner.entry.key).cmp(&other.inner.entry.key))
.then_with(|| (*self.entry.key).cmp(&other.entry.key))
// Reverse order comparison to yield newest-first // Reverse order comparison to yield newest-first
.then_with(|| other.order.cmp(&self.order)) .then_with(|| other.inner.order.cmp(&self.inner.order))
.reverse() .reverse()
} }
} }
/// An iterator that merges multiple sorted iterators into a single sorted iterator. Internally it /// An iterator that merges multiple sorted iterators into a single sorted iterator. Internally it
/// uses an heap of iterators to iterate them in order. /// uses a heap of iterators to iterate them in order.
pub struct MergeIter<T: Iterator<Item = Result<LookupEntry>>> { pub struct MergeIter<T: Iterator<Item = Result<LookupEntry>>> {
heap: BinaryHeap<Box<ActiveIterator<T>>>, heap: BinaryHeap<HeapNode<T>>,
} }
impl<T: Iterator<Item = Result<LookupEntry>>> MergeIter<T> { impl<T: Iterator<Item = Result<LookupEntry>>> MergeIter<T> {
...@@ -53,7 +63,11 @@ impl<T: Iterator<Item = Result<LookupEntry>>> MergeIter<T> { ...@@ -53,7 +63,11 @@ impl<T: Iterator<Item = Result<LookupEntry>>> MergeIter<T> {
for (order, mut iter) in iters.enumerate() { for (order, mut iter) in iters.enumerate() {
if let Some(entry) = iter.next() { if let Some(entry) = iter.next() {
let entry = entry?; let entry = entry?;
heap.push(Box::new(ActiveIterator { iter, order, entry })); let hash = entry.hash;
heap.push(HeapNode {
hash,
inner: Box::new(ActiveIterator { iter, order, entry }),
});
} }
} }
Ok(Self { heap }) Ok(Self { heap })
...@@ -64,13 +78,25 @@ impl<T: Iterator<Item = Result<LookupEntry>>> Iterator for MergeIter<T> { ...@@ -64,13 +78,25 @@ impl<T: Iterator<Item = Result<LookupEntry>>> Iterator for MergeIter<T> {
type Item = Result<LookupEntry>; type Item = Result<LookupEntry>;
fn next(&mut self) -> Option<Self::Item> { fn next(&mut self) -> Option<Self::Item> {
let mut active = self.heap.pop()?; let mut peek = self.heap.peek_mut()?;
let entry = match active.iter.next() { let node = &mut *peek;
None => return Some(Ok(active.entry)), match node.inner.iter.next() {
Some(Err(e)) => return Some(Err(e)), None => {
Some(Ok(next)) => std::mem::replace(&mut active.entry, next), // This iterator is exhausted, drop it and return the last entry
}; let node = PeekMut::pop(peek);
self.heap.push(active); Some(Ok(node.inner.entry))
Some(Ok(entry)) }
Some(Err(e)) => {
PeekMut::pop(peek);
Some(Err(e))
}
Some(Ok(next)) => {
let entry = std::mem::replace(&mut node.inner.entry, next);
// Update the cached hash before dropping the PeekMut (which triggers sift-down)
node.hash = node.inner.entry.hash;
drop(peek);
Some(Ok(entry))
}
}
} }
} }
use anyhow::Context;
/// Apply Linux-specific mmap advice flags that should be set on all persistent mmaps. /// Apply Linux-specific mmap advice flags that should be set on all persistent mmaps.
/// ///
/// - `DontFork`: prevents mmap regions from being copied into child processes on `fork()`, avoiding /// - `DontFork`: prevents mmap regions from being copied into child processes on `fork()`, avoiding
...@@ -8,6 +6,7 @@ use anyhow::Context; ...@@ -8,6 +6,7 @@ use anyhow::Context;
/// compressed content that won't benefit from deduplication scanning. /// compressed content that won't benefit from deduplication scanning.
#[cfg(target_os = "linux")] #[cfg(target_os = "linux")]
pub fn advise_mmap_for_persistence(mmap: &memmap2::Mmap) -> anyhow::Result<()> { pub fn advise_mmap_for_persistence(mmap: &memmap2::Mmap) -> anyhow::Result<()> {
use anyhow::Context;
mmap.advise(memmap2::Advice::DontFork) mmap.advise(memmap2::Advice::DontFork)
.context("Failed to advise mmap DontFork")?; .context("Failed to advise mmap DontFork")?;
mmap.advise(memmap2::Advice::Unmergeable) mmap.advise(memmap2::Advice::Unmergeable)
......
use std::{
borrow::Borrow,
fmt::{self, Debug, Formatter},
ops::{Deref, Range},
rc::Rc,
};
use memmap2::Mmap;
use crate::{
compression::decompress_into_rc,
shared_bytes::{SharedBytes, is_subslice_of},
};
/// The backing storage for an `RcBytes`.
///
/// Uses `Rc` for all refcounting, eliminating atomic operations.
#[derive(Clone)]
enum Backing {
Rc { _backing: Rc<[u8]> },
Mmap { _backing: Rc<Mmap> },
}
/// An owned byte slice backed by either an `Rc<[u8]>` or a memory-mapped file.
///
/// Identical to `ArcBytes` but uses `Rc` instead of `Arc`, eliminating atomic
/// refcount overhead. Use this in single-threaded contexts like SST iteration
/// during compaction.
#[derive(Clone)]
pub struct RcBytes {
data: *const [u8],
backing: Backing,
}
impl From<Rc<[u8]>> for RcBytes {
fn from(rc: Rc<[u8]>) -> Self {
Self {
data: &*rc as *const [u8],
backing: Backing::Rc { _backing: rc },
}
}
}
impl From<Box<[u8]>> for RcBytes {
fn from(b: Box<[u8]>) -> Self {
Self::from(Rc::from(b))
}
}
impl Deref for RcBytes {
type Target = [u8];
fn deref(&self) -> &Self::Target {
unsafe { &*self.data }
}
}
impl Borrow<[u8]> for RcBytes {
fn borrow(&self) -> &[u8] {
self
}
}
impl PartialEq for RcBytes {
fn eq(&self, other: &Self) -> bool {
self.deref().eq(other.deref())
}
}
impl Debug for RcBytes {
fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
Debug::fmt(&**self, f)
}
}
impl Eq for RcBytes {}
impl SharedBytes for RcBytes {
type MmapHandle = Rc<Mmap>;
fn slice(self, range: Range<usize>) -> Self {
let data = &*self;
let data = &data[range] as *const [u8];
Self {
data,
backing: self.backing,
}
}
unsafe fn slice_from_subslice(&self, subslice: &[u8]) -> Self {
debug_assert!(
is_subslice_of(
subslice,
match &self.backing {
Backing::Rc { _backing } => _backing,
Backing::Mmap { _backing } => _backing,
}
),
"slice_from_subslice: subslice is not within the backing storage"
);
Self {
data: subslice as *const [u8],
backing: self.backing.clone(),
}
}
unsafe fn from_mmap(mmap: &Rc<Mmap>, subslice: &[u8]) -> Self {
debug_assert!(
is_subslice_of(subslice, mmap),
"from_mmap: subslice is not within the mmap"
);
RcBytes {
data: subslice as *const [u8],
backing: Backing::Mmap {
_backing: mmap.clone(),
},
}
}
fn from_decompressed(uncompressed_length: u32, block: &[u8]) -> anyhow::Result<Self> {
Ok(RcBytes::from(decompress_into_rc(
uncompressed_length,
block,
)?))
}
}
use std::ops::{Deref, Range};
use memmap2::Mmap;
/// Trait abstracting over `ArcBytes` and `RcBytes`.
///
/// Both types are owned byte slices backed by either a ref-counted heap
/// allocation or a memory-mapped file. The only difference is the refcount
/// flavor: `Arc` (atomic, thread-safe) vs `Rc` (non-atomic, single-threaded).
///
/// This trait captures the shared interface so that code in
/// `static_sorted_file.rs` (block reading, key matching, etc.) can be written
/// once and used for both lookup (ArcBytes) and iteration (RcBytes) paths.
pub trait SharedBytes: Clone + Deref<Target = [u8]> + Sized {
/// The ref-counted handle to the memory-mapped file.
type MmapHandle: Deref<Target = Mmap>;
/// Returns a new instance that points to a sub-range of the current slice.
fn slice(self, range: Range<usize>) -> Self;
/// Creates a sub-slice from a reference that points into this instance's
/// backing data.
///
/// # Safety
///
/// The caller must ensure that `subslice` points to memory within this
/// instance's backing storage.
unsafe fn slice_from_subslice(&self, subslice: &[u8]) -> Self;
/// Creates an instance backed by a memory-mapped file.
///
/// # Safety
///
/// The caller must ensure that `subslice` points to memory within the
/// given `mmap`.
unsafe fn from_mmap(mmap: &Self::MmapHandle, subslice: &[u8]) -> Self;
/// Creates an instance from a decompressed block.
fn from_decompressed(uncompressed_length: u32, block: &[u8]) -> anyhow::Result<Self>;
}
/// Returns `true` if `subslice` lies entirely within `backing`.
pub(crate) fn is_subslice_of(subslice: &[u8], backing: &[u8]) -> bool {
let backing = backing.as_ptr_range();
let sub = subslice.as_ptr_range();
sub.start >= backing.start && sub.end <= backing.end
}
...@@ -1196,10 +1196,10 @@ struct IndexBlockBuilder<W: Write> { ...@@ -1196,10 +1196,10 @@ struct IndexBlockBuilder<W: Write> {
} }
/// Size of a single index block entry (u64 hash + u16 block index). /// Size of a single index block entry (u64 hash + u16 block index).
const INDEX_BLOCK_ENTRY_SIZE: usize = size_of::<u64>() + size_of::<u16>(); pub(crate) const INDEX_BLOCK_ENTRY_SIZE: usize = size_of::<u64>() + size_of::<u16>();
/// Size of the index block header (u8 type + u16 first_block). /// Size of the index block header (u8 type + u16 first_block).
const INDEX_BLOCK_HEADER_SIZE: usize = size_of::<u8>() + size_of::<u16>(); pub(crate) const INDEX_BLOCK_HEADER_SIZE: usize = size_of::<u8>() + size_of::<u16>();
impl<W: Write> IndexBlockBuilder<W> { impl<W: Write> IndexBlockBuilder<W> {
/// Creates a new builder for an index block with the specified number of entries and a pointer /// Creates a new builder for an index block with the specified number of entries and a pointer
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment