Unverified Commit 8e9f9514 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

[turbopack] Optimize compaction cpu usage (#91468)

## Summary

Optimizes the turbo-persistence compaction and iteration paths with several targeted improvements:

### Iterator optimizations

- **Flatten index block iteration** — The iterator previously used a `Vec<CurrentIndexBlock>` stack, but SST files have exactly one index level. Inline the index block fields (`index_entries`, `index_block_count`, `index_pos`) directly into `StaticSortedFileIter`, eliminating the stack allocation and `Option` overhead.

- **Non-optional `CurrentKeyBlock`** — Parse the first key block during `try_into_iter()` construction so `current_key_block` is always populated, removing the `Option<CurrentKeyBlock>` wrapper and its `take()`/`Some()` ceremony in the hot loop.

- **Replace `ReadBytesExt` with direct byte indexing** — In `handle_key_match`, `parse_key_block`, and `next_internal`, replace `val.read_u16::<BE>()` etc. with `u16::from_be_bytes(val[0..2].try_into().unwrap())`. This eliminates th mutable slice pointer advancement.

- **Extract `read_offset_entry` helper** — Read type + offset from the key block offset table in a single `u32` load + shift, replacing two separate `ReadBytesExt` calls.


### Refcounting optimization

- **Introduce `RcBytes`** — Thread-local byte slice type using `Rc` instead of `Arc`, eliminating atomic refcount overhead during single-threaded SST iteration. The iteration path (`StaticSortedFileIter`) now produces `RcBytes` slices backed by an `Rc<Mmap>`, so per-entry clone/drop operations are plain integer increments rather than atomic operations.


### Merge iterator simplification

- **Optimize `MergeIter::next`** — Replaced the straightforwards `pop/push` pattern with `PeekMut`-based replace-top pattern, which means we only need to adjust the heap once per iteration instead of twice.

## Benchmark results

### Compaction (`key_8/value_4/entries_16.00Mi/commits_128`)

| Benchmark | Canary | Optimized | Change |
|-----------|--------|-----------|--------|
| partial compaction | 1.949 s | 1.515 s | **-22%** |
| full compaction | 2.051 s | 1.542 s | **-25%** |

### Read path (`static_sorted_file_lookup/entries_1000.00Ki`)

No read regression — the branch is neutral to slightly faster:

| Benchmark | Canary | Optimized | Change |
|-----------|--------|-----------|--------|
| hit/uncached | 6.73 µs | 6.59 µs | **-2%** |
| hit/cached | 140.8 ns | 130.7 ns | **-7%** |
| miss/uncached | 5.10 µs | 5.02 µs | **-2%** |
| miss/cached | 230.1 ns | 233.1 ns | ~+1% (noise) |

## Test plan

- [x] `cargo test -p turbo-persistence` — 60/60 tests passing
- [x] Compaction benchmarks run and compared against canary baseline
- [x] Read path (lookup) benchmarks verified no regression
parent af97e756
......@@ -2,13 +2,16 @@ use std::{
borrow::Borrow,
fmt::{self, Debug, Formatter},
hash::{Hash, Hasher},
io::{self, Read},
ops::{Deref, Range},
sync::Arc,
};
use memmap2::Mmap;
use crate::{
compression::decompress_into_arc,
shared_bytes::{SharedBytes, is_subslice_of},
};
/// The backing storage for an `ArcBytes`.
///
/// The inner values are never read directly — they exist solely to keep the
......@@ -78,27 +81,17 @@ impl Debug for ArcBytes {
impl Eq for ArcBytes {}
impl Read for ArcBytes {
fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> {
let available = &**self;
let len = std::cmp::min(buf.len(), available.len());
buf[..len].copy_from_slice(&available[..len]);
// Advance the slice view
self.data = &available[len..] as *const [u8];
Ok(len)
impl ArcBytes {
/// Returns `true` if this `ArcBytes` is backed by a memory-mapped file.
pub fn is_mmap_backed(&self) -> bool {
matches!(self.backing, Backing::Mmap { .. })
}
}
/// Returns `true` if `subslice` lies entirely within `backing`.
fn is_subslice_of(subslice: &[u8], backing: &[u8]) -> bool {
let backing = backing.as_ptr_range();
let sub = subslice.as_ptr_range();
sub.start >= backing.start && sub.end <= backing.end
}
impl SharedBytes for ArcBytes {
type MmapHandle = Arc<Mmap>;
impl ArcBytes {
/// Returns a new `ArcBytes` that points to a sub-range of the current slice.
pub fn slice(self, range: Range<usize>) -> ArcBytes {
fn slice(self, range: Range<usize>) -> Self {
let data = &*self;
let data = &data[range] as *const [u8];
Self {
......@@ -107,14 +100,7 @@ impl ArcBytes {
}
}
/// Creates a sub-slice from a slice reference that points into this ArcBytes' backing data.
///
/// # Safety
///
/// The caller must ensure that `subslice` points to memory within this ArcBytes'
/// backing storage (not just within the current slice view, but anywhere in the original
/// backing data).
pub unsafe fn slice_from_subslice(&self, subslice: &[u8]) -> ArcBytes {
unsafe fn slice_from_subslice(&self, subslice: &[u8]) -> Self {
debug_assert!(
is_subslice_of(
subslice,
......@@ -131,24 +117,23 @@ impl ArcBytes {
}
}
/// Creates an `ArcBytes` backed by a memory-mapped file.
///
/// # Safety
///
/// The caller must ensure that `subslice` points to memory within the given `mmap`.
pub unsafe fn from_mmap(mmap: Arc<Mmap>, subslice: &[u8]) -> ArcBytes {
unsafe fn from_mmap(mmap: &Arc<Mmap>, subslice: &[u8]) -> Self {
debug_assert!(
is_subslice_of(subslice, &mmap),
is_subslice_of(subslice, mmap),
"from_mmap: subslice is not within the mmap"
);
ArcBytes {
data: subslice as *const [u8],
backing: Backing::Mmap { _backing: mmap },
backing: Backing::Mmap {
_backing: mmap.clone(),
},
}
}
/// Returns `true` if this `ArcBytes` is backed by a memory-mapped file.
pub fn is_mmap_backed(&self) -> bool {
matches!(self.backing, Backing::Mmap { .. })
fn from_decompressed(uncompressed_length: u32, block: &[u8]) -> anyhow::Result<Self> {
Ok(ArcBytes::from(decompress_into_arc(
uncompressed_length,
block,
)?))
}
}
//! Big-endian integer parsing helpers.
//!
//! Each function reads a fixed-width big-endian integer from the prefix of
//! a byte slice. Panics if the slice is too short (single bounds check via
//! `first_chunk`).
#[inline(always)]
pub fn read_u8(s: &[u8]) -> u8 {
s[0]
}
#[inline(always)]
pub fn read_u16(s: &[u8]) -> u16 {
u16::from_be_bytes(*s.first_chunk().unwrap())
}
#[inline(always)]
pub fn read_u24(s: &[u8]) -> u32 {
let &[a, b, c] = s.first_chunk().unwrap();
u32::from_be_bytes([0, a, b, c])
}
#[inline(always)]
pub fn read_u32(s: &[u8]) -> u32 {
u32::from_be_bytes(*s.first_chunk().unwrap())
}
#[inline(always)]
pub fn read_u64(s: &[u8]) -> u64 {
u64::from_be_bytes(*s.first_chunk().unwrap())
}
use std::{mem::MaybeUninit, sync::Arc};
use std::{mem::MaybeUninit, rc::Rc, sync::Arc};
use anyhow::{Context, Result};
use lzzzz::lz4::{self, decompress};
/// Decompresses a block into an Arc allocation.
///
/// The caller must ensure `uncompressed_length > 0` (i.e., the block is actually compressed).
/// Uncompressed blocks should be handled via zero-copy mmap slices before calling this.
pub fn decompress_into_arc(uncompressed_length: u32, block: &[u8]) -> Result<Arc<[u8]>> {
/// Decompresses `block` into `dest`, verifying the output length matches `expected_len`.
fn decompress_block(block: &[u8], dest: &mut [u8], expected_len: u32) -> Result<()> {
debug_assert!(
uncompressed_length > 0,
"decompress_into_arc called with uncompressed_length=0; uncompressed blocks should use \
expected_len > 0,
"decompress_block called with uncompressed_length=0; uncompressed blocks should use \
zero-copy mmap path"
);
// Allocate directly into an Arc to avoid a copy. The buffer is uninitialized;
// decompression will overwrite it completely (verified by the assert below).
let buffer: Arc<[MaybeUninit<u8>]> = Arc::new_uninit_slice(uncompressed_length as usize);
// Safety: decompression will fully initialize the buffer we verify with an assert
let mut buffer = unsafe { buffer.assume_init() };
// We just created this Arc so refcount is 1; get_mut always succeeds.
let decompressed = Arc::get_mut(&mut buffer).expect("Arc refcount should be 1");
let bytes_written = decompress(block, decompressed).with_context(|| {
let bytes_written = decompress(block, dest).with_context(|| {
format!(
"Failed to decompress block ({} bytes compressed, {} bytes uncompressed)",
block.len(),
uncompressed_length
expected_len
)
})?;
assert_eq!(
bytes_written, uncompressed_length as usize,
bytes_written, expected_len as usize,
"Decompressed length does not match expected length"
);
Ok(())
}
/// Decompresses a block into an Arc allocation.
///
/// The caller must ensure `uncompressed_length > 0` (i.e., the block is actually compressed).
/// Uncompressed blocks should be handled via zero-copy mmap slices before calling this.
pub fn decompress_into_arc(uncompressed_length: u32, block: &[u8]) -> Result<Arc<[u8]>> {
// Allocate directly into an Arc to avoid a copy. The buffer is uninitialized;
// decompression will overwrite it completely (verified by decompress_block).
let buffer: Arc<[MaybeUninit<u8>]> = Arc::new_uninit_slice(uncompressed_length as usize);
// Safety: decompression will fully initialize the buffer (verified by the assert in
// decompress_block).
let mut buffer = unsafe { buffer.assume_init() };
// We just created this Arc so refcount is 1; get_mut always succeeds.
let dest = Arc::get_mut(&mut buffer).expect("Arc refcount should be 1");
decompress_block(block, dest, uncompressed_length)?;
Ok(buffer)
}
/// Like [`decompress_into_arc`] but returns an `Rc<[u8]>` for thread-local use.
pub fn decompress_into_rc(uncompressed_length: u32, block: &[u8]) -> Result<Rc<[u8]>> {
let buffer: Rc<[MaybeUninit<u8>]> = Rc::new_uninit_slice(uncompressed_length as usize);
// Safety: decompression will fully initialize the buffer (verified by the assert in
// decompress_block).
let mut buffer = unsafe { buffer.assume_init() };
let dest = Rc::get_mut(&mut buffer).expect("Rc refcount should be 1");
decompress_block(block, dest, uncompressed_length)?;
Ok(buffer)
}
......
......@@ -30,14 +30,15 @@ use crate::{
MAX_ENTRIES_PER_COMPACTED_FILE, VALUE_BLOCK_AVG_SIZE, VALUE_BLOCK_CACHE_SIZE,
},
key::{StoreKey, hash_key},
lookup_entry::{LazyLookupValue, LookupEntry, LookupValue},
lookup_entry::{IterValue, LookupEntry, LookupValue},
merge_iter::MergeIter,
meta_file::{MetaEntryFlags, MetaFile, MetaLookupResult, StaticSortedFileRange},
meta_file_builder::MetaFileBuilder,
mmap_helper::advise_mmap_for_persistence,
parallel_scheduler::ParallelScheduler,
rc_bytes::RcBytes,
sst_filter::SstFilter,
static_sorted_file::{BlockCache, SstLookupResult, StaticSortedFile},
static_sorted_file::{BlockCache, SstLookupResult, StaticSortedFileIter},
static_sorted_file_builder::{StaticSortedFileBuilderMeta, StreamingSstWriter},
write_batch::{FinishResult, WriteBatch},
};
......@@ -1075,11 +1076,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES>
let meta_index = ssts_with_ranges[index].meta_index;
let index_in_meta = ssts_with_ranges[index].index_in_meta;
let entry = meta_files[meta_index].entry(index_in_meta);
StaticSortedFile::open_for_compaction(
path,
entry.sst_metadata(),
)?
.try_into_iter()
StaticSortedFileIter::open(path, entry.sst_metadata())
})
.collect::<Result<Vec<_>>>()?;
......@@ -1188,7 +1185,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES>
}
let mut used_collector = Collector::new(MetaEntryFlags::WARM);
let mut unused_collector = Collector::new(MetaEntryFlags::COLD);
let mut current_key: Option<ArcBytes> = None;
let mut current_key: Option<RcBytes> = None;
let mut keys_written = 0;
// MergeIter yields entries from newer SSTs first (by SST sequence
......@@ -1220,10 +1217,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES>
FamilyKind::MultiValue => {
// For MultiValue families we only skip remaining if we
// see a tombstone
if matches!(
entry.value,
LazyLookupValue::Eager(LookupValue::Deleted)
) {
if matches!(entry.value, IterValue::Deleted) {
skip_remaining_for_this_key = true;
}
}
......@@ -1243,10 +1237,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES>
// Entry is being dropped (superseded by newer entry or
// pruned by tombstone). If it references a blob file,
// mark that blob for deletion.
if let LazyLookupValue::Eager(LookupValue::Blob {
sequence_number,
}) = &entry.value
{
if let IterValue::Blob { sequence_number } = &entry.value {
blob_seq_numbers_to_delete.push(*sequence_number);
}
}
......
......@@ -3,6 +3,7 @@
#![feature(iter_collect_into)]
mod arc_bytes;
pub(crate) mod be;
mod collector;
mod collector_entry;
mod compaction;
......@@ -16,6 +17,8 @@ pub mod meta_file;
mod meta_file_builder;
pub mod mmap_helper;
mod parallel_scheduler;
mod rc_bytes;
mod shared_bytes;
mod sst_filter;
pub mod static_sorted_file;
mod static_sorted_file_builder;
......
use crate::{
ArcBytes,
constants::{MAX_INLINE_VALUE_SIZE, MAX_SMALL_VALUE_SIZE},
rc_bytes::RcBytes,
static_sorted_file_builder::{Entry, EntryValue},
};
/// A value from a SST file lookup.
/// A value from a SST file. Generic over the byte representation, defaulting to
/// `ArcBytes` for the lookup path. The compaction/iteration path uses
/// `LookupValue<RcBytes>` which is convertible to `IterValue`.
#[derive(PartialEq)]
pub enum LookupValue {
pub enum LookupValue<B = ArcBytes> {
/// The value was deleted.
Deleted,
/// The value is stored in the SST file.
///
/// The ArcBytes will be pointing either at a keyblock or a value block in the SST
Slice { value: ArcBytes },
/// The bytes will be pointing either at a keyblock or a value block in the SST
Slice { value: B },
/// The value is stored in a blob file.
Blob { sequence_number: u32 },
}
/// A value from a SST file lookup.
pub enum LazyLookupValue {
/// A LookupValue
Eager(LookupValue),
/// A value from SST file iteration (compaction path, uses RcBytes for
/// non-atomic refcounting).
pub enum IterValue {
/// The value was deleted.
Deleted,
/// The value is stored in the SST file.
Slice { value: RcBytes },
/// The value is stored in a blob file.
Blob { sequence_number: u32 },
/// A medium sized value that is still compressed.
Medium {
uncompressed_size: u32,
checksum: u32,
block: ArcBytes,
block: RcBytes,
},
}
impl LazyLookupValue {
/// Returns the size of the value in the SST file.
pub fn uncompressed_size_in_sst(&self) -> usize {
match self {
LazyLookupValue::Eager(LookupValue::Slice { value }) => value.len(),
LazyLookupValue::Eager(LookupValue::Deleted) => 0,
LazyLookupValue::Eager(LookupValue::Blob { .. }) => 0,
LazyLookupValue::Medium {
uncompressed_size,
block,
..
} => {
if *uncompressed_size == 0 {
block.len()
} else {
*uncompressed_size as usize
}
}
}
}
/// Returns true if this value gets its own dedicated value block.
pub fn is_medium_value(&self) -> bool {
match self {
LazyLookupValue::Eager(LookupValue::Slice { value })
if value.len() > MAX_SMALL_VALUE_SIZE =>
{
true
}
LazyLookupValue::Medium { .. } => true,
_ => false,
}
}
/// Returns the value size if it will be packed into a small value block, or 0 otherwise.
pub fn small_value_size(&self) -> usize {
match self {
LazyLookupValue::Eager(LookupValue::Slice { value })
if value.len() > MAX_INLINE_VALUE_SIZE && value.len() <= MAX_SMALL_VALUE_SIZE =>
{
value.len()
}
_ => 0,
impl From<LookupValue<RcBytes>> for IterValue {
fn from(v: LookupValue<RcBytes>) -> Self {
match v {
LookupValue::Deleted => IterValue::Deleted,
LookupValue::Slice { value } => IterValue::Slice { value },
LookupValue::Blob { sequence_number } => IterValue::Blob { sequence_number },
}
}
}
/// An entry from a SST file lookup.
/// An entry from SST file iteration (compaction path, uses RcBytes).
pub struct LookupEntry {
/// The hash of the key.
pub hash: u64,
/// The key.
pub key: ArcBytes,
pub key: RcBytes,
/// The value.
pub value: LazyLookupValue,
pub value: IterValue,
}
impl Entry for LookupEntry {
......@@ -101,8 +70,8 @@ impl Entry for LookupEntry {
fn value(&self) -> EntryValue<'_> {
match &self.value {
LazyLookupValue::Eager(LookupValue::Deleted) => EntryValue::Deleted,
LazyLookupValue::Eager(LookupValue::Slice { value }) => {
IterValue::Deleted => EntryValue::Deleted,
IterValue::Slice { value } => {
if value.len() <= MAX_INLINE_VALUE_SIZE {
EntryValue::Inline { value }
} else if value.len() > MAX_SMALL_VALUE_SIZE {
......@@ -111,10 +80,10 @@ impl Entry for LookupEntry {
EntryValue::Small { value }
}
}
LazyLookupValue::Eager(LookupValue::Blob { sequence_number }) => EntryValue::Large {
IterValue::Blob { sequence_number } => EntryValue::Large {
blob: *sequence_number,
},
LazyLookupValue::Medium {
IterValue::Medium {
uncompressed_size,
checksum,
block,
......
use std::{cmp::Ordering, collections::BinaryHeap};
use std::{
cmp::Ordering,
collections::{BinaryHeap, binary_heap::PeekMut},
};
use anyhow::Result;
......@@ -6,45 +9,52 @@ use crate::lookup_entry::LookupEntry;
/// An active iterator that is being merged. It has peeked the next element and can be compared
/// according to that element. The `order` is used when multiple iterators have the same key.
///
/// Boxed so that BinaryHeap sift operations only move a pointer (8 bytes) instead of the full
/// struct (~312 bytes for StaticSortedFileIter), which is significant with 128 iterators.
struct ActiveIterator<T: Iterator<Item = Result<LookupEntry>>> {
iter: T,
order: usize,
entry: LookupEntry,
}
impl<T: Iterator<Item = Result<LookupEntry>>> PartialEq for Box<ActiveIterator<T>> {
/// A heap node that keeps the hash inline alongside a boxed `ActiveIterator`. By hoisting the hash
/// out of the Box we avoid a pointer chase on every comparison when the heap is re-ordered Only
/// when hashes collide (extremely rare with u64) do we dereference the Box to compare keys.
/// Note: we cannot use [Prehashed] because we need an non-trivial `Ord` implementation for the heap
/// (see below)
struct HeapNode<T: Iterator<Item = Result<LookupEntry>>> {
/// Cached copy of the current entry's hash, kept in sync with `inner.entry.hash`.
hash: u64,
inner: Box<ActiveIterator<T>>,
}
impl<T: Iterator<Item = Result<LookupEntry>>> PartialEq for HeapNode<T> {
fn eq(&self, other: &Self) -> bool {
self.entry.hash == other.entry.hash && *self.entry.key == *other.entry.key
self.hash == other.hash && *self.inner.entry.key == *other.inner.entry.key
}
}
impl<T: Iterator<Item = Result<LookupEntry>>> Eq for Box<ActiveIterator<T>> {}
impl<T: Iterator<Item = Result<LookupEntry>>> Eq for HeapNode<T> {}
impl<T: Iterator<Item = Result<LookupEntry>>> PartialOrd for Box<ActiveIterator<T>> {
impl<T: Iterator<Item = Result<LookupEntry>>> PartialOrd for HeapNode<T> {
fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
Some(self.cmp(other))
}
}
impl<T: Iterator<Item = Result<LookupEntry>>> Ord for Box<ActiveIterator<T>> {
impl<T: Iterator<Item = Result<LookupEntry>>> Ord for HeapNode<T> {
fn cmp(&self, other: &Self) -> Ordering {
self.entry
.hash
.cmp(&other.entry.hash)
.then_with(|| (*self.entry.key).cmp(&other.entry.key))
self.hash
.cmp(&other.hash)
.then_with(|| (*self.inner.entry.key).cmp(&other.inner.entry.key))
// Reverse order comparison to yield newest-first
.then_with(|| other.order.cmp(&self.order))
.then_with(|| other.inner.order.cmp(&self.inner.order))
.reverse()
}
}
/// An iterator that merges multiple sorted iterators into a single sorted iterator. Internally it
/// uses an heap of iterators to iterate them in order.
/// uses a heap of iterators to iterate them in order.
pub struct MergeIter<T: Iterator<Item = Result<LookupEntry>>> {
heap: BinaryHeap<Box<ActiveIterator<T>>>,
heap: BinaryHeap<HeapNode<T>>,
}
impl<T: Iterator<Item = Result<LookupEntry>>> MergeIter<T> {
......@@ -53,7 +63,11 @@ impl<T: Iterator<Item = Result<LookupEntry>>> MergeIter<T> {
for (order, mut iter) in iters.enumerate() {
if let Some(entry) = iter.next() {
let entry = entry?;
heap.push(Box::new(ActiveIterator { iter, order, entry }));
let hash = entry.hash;
heap.push(HeapNode {
hash,
inner: Box::new(ActiveIterator { iter, order, entry }),
});
}
}
Ok(Self { heap })
......@@ -64,13 +78,25 @@ impl<T: Iterator<Item = Result<LookupEntry>>> Iterator for MergeIter<T> {
type Item = Result<LookupEntry>;
fn next(&mut self) -> Option<Self::Item> {
let mut active = self.heap.pop()?;
let entry = match active.iter.next() {
None => return Some(Ok(active.entry)),
Some(Err(e)) => return Some(Err(e)),
Some(Ok(next)) => std::mem::replace(&mut active.entry, next),
};
self.heap.push(active);
Some(Ok(entry))
let mut peek = self.heap.peek_mut()?;
let node = &mut *peek;
match node.inner.iter.next() {
None => {
// This iterator is exhausted, drop it and return the last entry
let node = PeekMut::pop(peek);
Some(Ok(node.inner.entry))
}
Some(Err(e)) => {
PeekMut::pop(peek);
Some(Err(e))
}
Some(Ok(next)) => {
let entry = std::mem::replace(&mut node.inner.entry, next);
// Update the cached hash before dropping the PeekMut (which triggers sift-down)
node.hash = node.inner.entry.hash;
drop(peek);
Some(Ok(entry))
}
}
}
}
use anyhow::Context;
/// Apply Linux-specific mmap advice flags that should be set on all persistent mmaps.
///
/// - `DontFork`: prevents mmap regions from being copied into child processes on `fork()`, avoiding
......@@ -8,6 +6,7 @@ use anyhow::Context;
/// compressed content that won't benefit from deduplication scanning.
#[cfg(target_os = "linux")]
pub fn advise_mmap_for_persistence(mmap: &memmap2::Mmap) -> anyhow::Result<()> {
use anyhow::Context;
mmap.advise(memmap2::Advice::DontFork)
.context("Failed to advise mmap DontFork")?;
mmap.advise(memmap2::Advice::Unmergeable)
......
use std::{
borrow::Borrow,
fmt::{self, Debug, Formatter},
ops::{Deref, Range},
rc::Rc,
};
use memmap2::Mmap;
use crate::{
compression::decompress_into_rc,
shared_bytes::{SharedBytes, is_subslice_of},
};
/// The backing storage for an `RcBytes`.
///
/// Uses `Rc` for all refcounting, eliminating atomic operations.
#[derive(Clone)]
enum Backing {
Rc { _backing: Rc<[u8]> },
Mmap { _backing: Rc<Mmap> },
}
/// An owned byte slice backed by either an `Rc<[u8]>` or a memory-mapped file.
///
/// Identical to `ArcBytes` but uses `Rc` instead of `Arc`, eliminating atomic
/// refcount overhead. Use this in single-threaded contexts like SST iteration
/// during compaction.
#[derive(Clone)]
pub struct RcBytes {
data: *const [u8],
backing: Backing,
}
impl From<Rc<[u8]>> for RcBytes {
fn from(rc: Rc<[u8]>) -> Self {
Self {
data: &*rc as *const [u8],
backing: Backing::Rc { _backing: rc },
}
}
}
impl From<Box<[u8]>> for RcBytes {
fn from(b: Box<[u8]>) -> Self {
Self::from(Rc::from(b))
}
}
impl Deref for RcBytes {
type Target = [u8];
fn deref(&self) -> &Self::Target {
unsafe { &*self.data }
}
}
impl Borrow<[u8]> for RcBytes {
fn borrow(&self) -> &[u8] {
self
}
}
impl PartialEq for RcBytes {
fn eq(&self, other: &Self) -> bool {
self.deref().eq(other.deref())
}
}
impl Debug for RcBytes {
fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
Debug::fmt(&**self, f)
}
}
impl Eq for RcBytes {}
impl SharedBytes for RcBytes {
type MmapHandle = Rc<Mmap>;
fn slice(self, range: Range<usize>) -> Self {
let data = &*self;
let data = &data[range] as *const [u8];
Self {
data,
backing: self.backing,
}
}
unsafe fn slice_from_subslice(&self, subslice: &[u8]) -> Self {
debug_assert!(
is_subslice_of(
subslice,
match &self.backing {
Backing::Rc { _backing } => _backing,
Backing::Mmap { _backing } => _backing,
}
),
"slice_from_subslice: subslice is not within the backing storage"
);
Self {
data: subslice as *const [u8],
backing: self.backing.clone(),
}
}
unsafe fn from_mmap(mmap: &Rc<Mmap>, subslice: &[u8]) -> Self {
debug_assert!(
is_subslice_of(subslice, mmap),
"from_mmap: subslice is not within the mmap"
);
RcBytes {
data: subslice as *const [u8],
backing: Backing::Mmap {
_backing: mmap.clone(),
},
}
}
fn from_decompressed(uncompressed_length: u32, block: &[u8]) -> anyhow::Result<Self> {
Ok(RcBytes::from(decompress_into_rc(
uncompressed_length,
block,
)?))
}
}
use std::ops::{Deref, Range};
use memmap2::Mmap;
/// Trait abstracting over `ArcBytes` and `RcBytes`.
///
/// Both types are owned byte slices backed by either a ref-counted heap
/// allocation or a memory-mapped file. The only difference is the refcount
/// flavor: `Arc` (atomic, thread-safe) vs `Rc` (non-atomic, single-threaded).
///
/// This trait captures the shared interface so that code in
/// `static_sorted_file.rs` (block reading, key matching, etc.) can be written
/// once and used for both lookup (ArcBytes) and iteration (RcBytes) paths.
pub trait SharedBytes: Clone + Deref<Target = [u8]> + Sized {
/// The ref-counted handle to the memory-mapped file.
type MmapHandle: Deref<Target = Mmap>;
/// Returns a new instance that points to a sub-range of the current slice.
fn slice(self, range: Range<usize>) -> Self;
/// Creates a sub-slice from a reference that points into this instance's
/// backing data.
///
/// # Safety
///
/// The caller must ensure that `subslice` points to memory within this
/// instance's backing storage.
unsafe fn slice_from_subslice(&self, subslice: &[u8]) -> Self;
/// Creates an instance backed by a memory-mapped file.
///
/// # Safety
///
/// The caller must ensure that `subslice` points to memory within the
/// given `mmap`.
unsafe fn from_mmap(mmap: &Self::MmapHandle, subslice: &[u8]) -> Self;
/// Creates an instance from a decompressed block.
fn from_decompressed(uncompressed_length: u32, block: &[u8]) -> anyhow::Result<Self>;
}
/// Returns `true` if `subslice` lies entirely within `backing`.
pub(crate) fn is_subslice_of(subslice: &[u8], backing: &[u8]) -> bool {
let backing = backing.as_ptr_range();
let sub = subslice.as_ptr_range();
sub.start >= backing.start && sub.end <= backing.end
}
......@@ -1196,10 +1196,10 @@ struct IndexBlockBuilder<W: Write> {
}
/// Size of a single index block entry (u64 hash + u16 block index).
const INDEX_BLOCK_ENTRY_SIZE: usize = size_of::<u64>() + size_of::<u16>();
pub(crate) const INDEX_BLOCK_ENTRY_SIZE: usize = size_of::<u64>() + size_of::<u16>();
/// Size of the index block header (u8 type + u16 first_block).
const INDEX_BLOCK_HEADER_SIZE: usize = size_of::<u8>() + size_of::<u16>();
pub(crate) const INDEX_BLOCK_HEADER_SIZE: usize = size_of::<u8>() + size_of::<u16>();
impl<W: Write> IndexBlockBuilder<W> {
/// Creates a new builder for an index block with the specified number of entries and a pointer
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment