Unverified Commit 0d49dd88 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

[turbopack] Make compression in SST files optional (#89309)

Some blocks in our SST files are not compressible. Notably all Index blocks and many Key blocks.

 * For index blocks the issue is that it is a sequence of hash->block-index pairs.  The hashes are high entropy (typically only a few of the uppermost bits will be shared across all hashes) and the indexes should be compressible but based on my measurements of vercel-site this isn't true in practice.  So this PR simply doesn't try.

 * For key blocks the same argument applies, but not quite as dramatically, we are storing high entropy hashes or small keys with similar hashes (and thus distinct values), so we shouldn't generally expect this to be very compressible.
   * Future work would be to mark certain _families_ as incompressible, but this is deferred for now

* Value blocks on the other hand achieve high amounts of compressibility, so we should of course keep trying.

This leverages heuristics from industry and avoid writing data as compressed unless it is smaller than 7/8 of the source document.  This is the pattern used by LevelDB, RocksDB, and ZFS.   When we detect this scenario we set the `uncompressed_size`  to 0 so readers know it is uncompressed. (See README updates for a description)

On the read side use the `mmap` slice directly to avoid copies, instead we make managing the slice the Operating Systems problem.  This is achieved by augmenting the ArcSlice datastructure to be able to store an `Arc<Mmap>` (and renaming it since it is now officially about bytes).
parent b2f193b0
...@@ -79,11 +79,18 @@ The SST file contains only data without any header. ...@@ -79,11 +79,18 @@ The SST file contains only data without any header.
- serialized key Compression Dictionary - serialized key Compression Dictionary
- foreach block - foreach block
- 4 bytes uncompressed block length - 4 bytes block header (uncompressed length or sentinel)
- compressed data - block data (compressed or uncompressed)
- foreach block - foreach block
- 4 bytes end of block offset relative to start of all blocks - 4 bytes end of block offset relative to start of all blocks
#### Block Compression
Blocks can be stored compressed (LZ4) or uncompressed. The 4-byte header distinguishes them:
- **Header > 0**: Block is LZ4 compressed. Header value is the uncompressed length.
- **Header = 0**: Block is stored uncompressed. Actual length is derived from block offsets.
#### Index Block #### Index Block
- 1 byte block type (0: index block) - 1 byte block type (0: index block)
......
...@@ -10,7 +10,7 @@ use quick_cache::sync::GuardResult; ...@@ -10,7 +10,7 @@ use quick_cache::sync::GuardResult;
use rand::{Rng, SeedableRng, rngs::SmallRng, seq::SliceRandom}; use rand::{Rng, SeedableRng, rngs::SmallRng, seq::SliceRandom};
use tempfile::TempDir; use tempfile::TempDir;
use turbo_persistence::{ use turbo_persistence::{
ArcSlice, BlockCache, CompactConfig, Entry, EntryValue, MetaEntryFlags, SerialScheduler, ArcBytes, BlockCache, CompactConfig, Entry, EntryValue, MetaEntryFlags, SerialScheduler,
StaticSortedFile, StaticSortedFileMetaData, TurboPersistence, hash_key, StaticSortedFile, StaticSortedFileMetaData, TurboPersistence, hash_key,
write_static_stored_file, write_static_stored_file,
}; };
...@@ -957,7 +957,7 @@ fn bench_block_cache(c: &mut Criterion) { ...@@ -957,7 +957,7 @@ fn bench_block_cache(c: &mut Criterion) {
let mut block_data = vec![0u8; BLOCK_SIZE]; let mut block_data = vec![0u8; BLOCK_SIZE];
rng.fill(&mut block_data[..]); rng.fill(&mut block_data[..]);
let block = ArcSlice::from(block_data.into_boxed_slice()); let block = ArcBytes::from(block_data.into_boxed_slice());
// Create cache with enough capacity for all entries // Create cache with enough capacity for all entries
let cache: BlockCache = BlockCache::with( let cache: BlockCache = BlockCache::with(
......
use std::{
borrow::Borrow,
fmt::{self, Debug, Formatter},
hash::{Hash, Hasher},
io::{self, Read},
ops::{Deref, Range},
sync::Arc,
};
use memmap2::Mmap;
/// The backing storage for an `ArcBytes`.
///
/// The inner values are never read directly — they exist solely to keep the
/// backing memory alive while the raw `data` pointer in `ArcBytes` references it.
#[derive(Clone)]
enum Backing {
Arc { _backing: Arc<[u8]> },
Mmap { _backing: Arc<Mmap> },
}
/// An owned byte slice backed by either an `Arc<[u8]>` or a memory-mapped file.
#[derive(Clone)]
pub struct ArcBytes {
data: *const [u8],
backing: Backing,
}
unsafe impl Send for ArcBytes {}
unsafe impl Sync for ArcBytes {}
impl From<Arc<[u8]>> for ArcBytes {
fn from(arc: Arc<[u8]>) -> Self {
Self {
data: &*arc as *const [u8],
backing: Backing::Arc { _backing: arc },
}
}
}
impl From<Box<[u8]>> for ArcBytes {
fn from(b: Box<[u8]>) -> Self {
Self::from(Arc::from(b))
}
}
impl Deref for ArcBytes {
type Target = [u8];
fn deref(&self) -> &Self::Target {
unsafe { &*self.data }
}
}
impl Borrow<[u8]> for ArcBytes {
fn borrow(&self) -> &[u8] {
self
}
}
impl Hash for ArcBytes {
fn hash<H: Hasher>(&self, state: &mut H) {
self.deref().hash(state)
}
}
impl PartialEq for ArcBytes {
fn eq(&self, other: &Self) -> bool {
self.deref().eq(other.deref())
}
}
impl Debug for ArcBytes {
fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
Debug::fmt(&**self, f)
}
}
impl Eq for ArcBytes {}
impl Read for ArcBytes {
fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> {
let available = &**self;
let len = std::cmp::min(buf.len(), available.len());
buf[..len].copy_from_slice(&available[..len]);
// Advance the slice view
self.data = &available[len..] as *const [u8];
Ok(len)
}
}
/// Returns `true` if `subslice` lies entirely within `backing`.
fn is_subslice_of(subslice: &[u8], backing: &[u8]) -> bool {
let backing = backing.as_ptr_range();
let sub = subslice.as_ptr_range();
sub.start >= backing.start && sub.end <= backing.end
}
impl ArcBytes {
/// Returns a new `ArcBytes` that points to a sub-range of the current slice.
pub fn slice(self, range: Range<usize>) -> ArcBytes {
let data = &*self;
let data = &data[range] as *const [u8];
Self {
data,
backing: self.backing,
}
}
/// Creates a sub-slice from a slice reference that points into this ArcBytes' backing data.
///
/// # Safety
///
/// The caller must ensure that `subslice` points to memory within this ArcBytes'
/// backing storage (not just within the current slice view, but anywhere in the original
/// backing data).
pub unsafe fn slice_from_subslice(&self, subslice: &[u8]) -> ArcBytes {
debug_assert!(
is_subslice_of(
subslice,
match &self.backing {
Backing::Arc { _backing } => _backing,
Backing::Mmap { _backing } => _backing,
}
),
"slice_from_subslice: subslice is not within the backing storage"
);
Self {
data: subslice as *const [u8],
backing: self.backing.clone(),
}
}
/// Creates an `ArcBytes` backed by a memory-mapped file.
///
/// # Safety
///
/// The caller must ensure that `subslice` points to memory within the given `mmap`.
pub unsafe fn from_mmap(mmap: Arc<Mmap>, subslice: &[u8]) -> ArcBytes {
debug_assert!(
is_subslice_of(subslice, &mmap),
"from_mmap: subslice is not within the mmap"
);
ArcBytes {
data: subslice as *const [u8],
backing: Backing::Mmap { _backing: mmap },
}
}
/// Returns `true` if this `ArcBytes` is backed by a memory-mapped file.
pub fn is_mmap_backed(&self) -> bool {
matches!(self.backing, Backing::Mmap { .. })
}
}
use std::{
borrow::Borrow,
fmt::{self, Debug, Formatter},
hash::{Hash, Hasher},
io::{self, Read},
ops::{Deref, Range},
sync::Arc,
};
/// A owned slice that is backed by an `Arc`.
#[derive(Clone)]
pub struct ArcSlice<T> {
data: *const [T],
arc: Arc<[T]>,
}
unsafe impl<T> Send for ArcSlice<T> {}
unsafe impl<T> Sync for ArcSlice<T> {}
impl<T> From<Arc<[T]>> for ArcSlice<T> {
fn from(arc: Arc<[T]>) -> Self {
Self {
data: &*arc as *const [T],
arc,
}
}
}
impl<T> From<Box<[T]>> for ArcSlice<T> {
fn from(b: Box<[T]>) -> Self {
Self::from(Arc::from(b))
}
}
impl<T> Deref for ArcSlice<T> {
type Target = [T];
fn deref(&self) -> &Self::Target {
unsafe { &*self.data }
}
}
impl<T> Borrow<[T]> for ArcSlice<T> {
fn borrow(&self) -> &[T] {
self
}
}
impl<T: Hash> Hash for ArcSlice<T> {
fn hash<H: Hasher>(&self, state: &mut H) {
self.deref().hash(state)
}
}
impl<T: PartialEq> PartialEq for ArcSlice<T> {
fn eq(&self, other: &Self) -> bool {
self.deref().eq(other.deref())
}
}
impl<T: Debug> Debug for ArcSlice<T> {
fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
Debug::fmt(&**self, f)
}
}
impl<T: Eq> Eq for ArcSlice<T> {}
impl Read for ArcSlice<u8> {
fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> {
let available = &**self;
let len = std::cmp::min(buf.len(), available.len());
buf[..len].copy_from_slice(&available[..len]);
// Advance the slice view
self.data = &available[len..] as *const [u8];
Ok(len)
}
}
impl<T> ArcSlice<T> {
/// Returns a new `ArcSlice` that points to a slice of the current slice.
pub fn slice(self, range: Range<usize>) -> ArcSlice<T> {
let data = &*self;
let data = &data[range] as *const [T];
Self {
data,
arc: self.arc,
}
}
/// Creates a sub-slice from a slice reference that points into this ArcSlice's backing data.
///
/// # Safety
///
/// The caller must ensure that `subslice` points to memory within this ArcSlice's
/// backing Arc (not just within the current slice view, but anywhere in the original Arc).
pub unsafe fn slice_from_subslice(&self, subslice: &[T]) -> ArcSlice<T> {
Self {
data: subslice as *const [T],
arc: self.arc.clone(),
}
}
}
...@@ -3,12 +3,22 @@ use std::{mem::MaybeUninit, sync::Arc}; ...@@ -3,12 +3,22 @@ use std::{mem::MaybeUninit, sync::Arc};
use anyhow::{Context, Result}; use anyhow::{Context, Result};
use lzzzz::lz4::{ACC_LEVEL_DEFAULT, decompress, decompress_with_dict}; use lzzzz::lz4::{ACC_LEVEL_DEFAULT, decompress, decompress_with_dict};
/// Decompresses a block into an Arc allocation.
///
/// The caller must ensure `uncompressed_length > 0` (i.e., the block is actually compressed).
/// Uncompressed blocks should be handled via zero-copy mmap slices before calling this.
pub fn decompress_into_arc( pub fn decompress_into_arc(
uncompressed_length: u32, uncompressed_length: u32,
block: &[u8], block: &[u8],
compression_dictionary: Option<&[u8]>, compression_dictionary: Option<&[u8]>,
_long_term: bool, _long_term: bool,
) -> Result<Arc<[u8]>> { ) -> Result<Arc<[u8]>> {
debug_assert!(
uncompressed_length > 0,
"decompress_into_arc called with uncompressed_length=0; uncompressed blocks should use \
zero-copy mmap path"
);
// We directly allocate the buffer in an Arc to avoid copying it into an Arc and avoiding // We directly allocate the buffer in an Arc to avoid copying it into an Arc and avoiding
// double indirection. This is a dynamically sized arc. // double indirection. This is a dynamically sized arc.
let buffer: Arc<[MaybeUninit<u8>]> = Arc::new_zeroed_slice(uncompressed_length as usize); let buffer: Arc<[MaybeUninit<u8>]> = Arc::new_zeroed_slice(uncompressed_length as usize);
......
...@@ -21,7 +21,7 @@ use smallvec::SmallVec; ...@@ -21,7 +21,7 @@ use smallvec::SmallVec;
pub use crate::compaction::selector::CompactConfig; pub use crate::compaction::selector::CompactConfig;
use crate::{ use crate::{
QueryKey, QueryKey,
arc_slice::ArcSlice, arc_bytes::ArcBytes,
compaction::selector::{Compactable, get_merge_segments}, compaction::selector::{Compactable, get_merge_segments},
compression::decompress_into_arc, compression::decompress_into_arc,
constants::{ constants::{
...@@ -377,7 +377,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES> ...@@ -377,7 +377,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES>
/// Reads and decompresses a blob file. This is not backed by any cache. /// Reads and decompresses a blob file. This is not backed by any cache.
#[tracing::instrument(level = "info", name = "reading database blob", skip_all)] #[tracing::instrument(level = "info", name = "reading database blob", skip_all)]
fn read_blob(&self, seq: u32) -> Result<ArcSlice<u8>> { fn read_blob(&self, seq: u32) -> Result<ArcBytes> {
let path = self.path.join(format!("{seq:08}.blob")); let path = self.path.join(format!("{seq:08}.blob"));
let mmap = unsafe { Mmap::map(&File::open(&path)?)? }; let mmap = unsafe { Mmap::map(&File::open(&path)?)? };
#[cfg(unix)] #[cfg(unix)]
...@@ -392,7 +392,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES> ...@@ -392,7 +392,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES>
let uncompressed_length = compressed.read_u32::<BE>()?; let uncompressed_length = compressed.read_u32::<BE>()?;
let buffer = decompress_into_arc(uncompressed_length, compressed, None, true)?; let buffer = decompress_into_arc(uncompressed_length, compressed, None, true)?;
Ok(ArcSlice::from(buffer)) Ok(ArcBytes::from(buffer))
} }
/// Returns true if the database is empty. /// Returns true if the database is empty.
...@@ -1357,7 +1357,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES> ...@@ -1357,7 +1357,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES>
/// Get a value from the database. Returns None if the key is not found. The returned value /// Get a value from the database. Returns None if the key is not found. The returned value
/// might hold onto a block of the database and it should not be hold long-term. /// might hold onto a block of the database and it should not be hold long-term.
pub fn get<K: QueryKey>(&self, family: usize, key: &K) -> Result<Option<ArcSlice<u8>>> { pub fn get<K: QueryKey>(&self, family: usize, key: &K) -> Result<Option<ArcBytes>> {
debug_assert!(family < FAMILIES, "Family index out of bounds"); debug_assert!(family < FAMILIES, "Family index out of bounds");
let span = tracing::trace_span!( let span = tracing::trace_span!(
"database read", "database read",
...@@ -1429,7 +1429,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES> ...@@ -1429,7 +1429,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES>
&self, &self,
family: usize, family: usize,
keys: &[K], keys: &[K],
) -> Result<Vec<Option<ArcSlice<u8>>>> { ) -> Result<Vec<Option<ArcBytes>>> {
debug_assert!(family < FAMILIES, "Family index out of bounds"); debug_assert!(family < FAMILIES, "Family index out of bounds");
let span = tracing::trace_span!( let span = tracing::trace_span!(
"database batch read", "database batch read",
......
...@@ -3,7 +3,7 @@ ...@@ -3,7 +3,7 @@
#![feature(sync_unsafe_cell)] #![feature(sync_unsafe_cell)]
#![feature(iter_collect_into)] #![feature(iter_collect_into)]
mod arc_slice; mod arc_bytes;
mod collector; mod collector;
mod collector_entry; mod collector_entry;
mod compaction; mod compaction;
...@@ -26,7 +26,7 @@ mod write_batch; ...@@ -26,7 +26,7 @@ mod write_batch;
#[cfg(test)] #[cfg(test)]
mod tests; mod tests;
pub use arc_slice::ArcSlice; pub use arc_bytes::ArcBytes;
pub use db::{CompactConfig, MetaFileEntryInfo, MetaFileInfo, TurboPersistence}; pub use db::{CompactConfig, MetaFileEntryInfo, MetaFileInfo, TurboPersistence};
pub use key::{KeyBase, QueryKey, StoreKey, hash_key}; pub use key::{KeyBase, QueryKey, StoreKey, hash_key};
pub use meta_file::MetaEntryFlags; pub use meta_file::MetaEntryFlags;
......
use crate::{ use crate::{
ArcSlice, ArcBytes,
constants::{MAX_INLINE_VALUE_SIZE, MAX_SMALL_VALUE_SIZE}, constants::{MAX_INLINE_VALUE_SIZE, MAX_SMALL_VALUE_SIZE},
static_sorted_file_builder::{Entry, EntryValue}, static_sorted_file_builder::{Entry, EntryValue},
}; };
...@@ -10,8 +10,8 @@ pub enum LookupValue { ...@@ -10,8 +10,8 @@ pub enum LookupValue {
Deleted, Deleted,
/// The value is stored in the SST file. /// The value is stored in the SST file.
/// ///
/// The ArcSlice will be pointing either at a keyblock or a value block in the SST /// The ArcBytes will be pointing either at a keyblock or a value block in the SST
Slice { value: ArcSlice<u8> }, Slice { value: ArcBytes },
/// The value is stored in a blob file. /// The value is stored in a blob file.
Blob { sequence_number: u32 }, Blob { sequence_number: u32 },
} }
...@@ -35,8 +35,15 @@ impl LazyLookupValue<'_> { ...@@ -35,8 +35,15 @@ impl LazyLookupValue<'_> {
LazyLookupValue::Eager(LookupValue::Deleted) => 0, LazyLookupValue::Eager(LookupValue::Deleted) => 0,
LazyLookupValue::Eager(LookupValue::Blob { .. }) => 0, LazyLookupValue::Eager(LookupValue::Blob { .. }) => 0,
LazyLookupValue::Medium { LazyLookupValue::Medium {
uncompressed_size, .. uncompressed_size,
} => *uncompressed_size as usize, block,
} => {
if *uncompressed_size == 0 {
block.len()
} else {
*uncompressed_size as usize
}
}
} }
} }
...@@ -71,7 +78,7 @@ pub struct LookupEntry<'l> { ...@@ -71,7 +78,7 @@ pub struct LookupEntry<'l> {
/// The hash of the key. /// The hash of the key.
pub hash: u64, pub hash: u64,
/// The key. /// The key.
pub key: ArcSlice<u8>, pub key: ArcBytes,
/// The value. /// The value.
pub value: LazyLookupValue<'l>, pub value: LazyLookupValue<'l>,
} }
...@@ -107,7 +114,7 @@ impl Entry for LookupEntry<'_> { ...@@ -107,7 +114,7 @@ impl Entry for LookupEntry<'_> {
LazyLookupValue::Medium { LazyLookupValue::Medium {
uncompressed_size, uncompressed_size,
block, block,
} => EntryValue::MediumCompressed { } => EntryValue::MediumRaw {
uncompressed_size: *uncompressed_size, uncompressed_size: *uncompressed_size,
block, block,
}, },
......
...@@ -4,6 +4,7 @@ use std::{ ...@@ -4,6 +4,7 @@ use std::{
hash::BuildHasherDefault, hash::BuildHasherDefault,
ops::Range, ops::Range,
path::{Path, PathBuf}, path::{Path, PathBuf},
sync::Arc,
}; };
use anyhow::{Context, Result, bail}; use anyhow::{Context, Result, bail};
...@@ -14,7 +15,7 @@ use rustc_hash::FxHasher; ...@@ -14,7 +15,7 @@ use rustc_hash::FxHasher;
use crate::{ use crate::{
QueryKey, QueryKey,
arc_slice::ArcSlice, arc_bytes::ArcBytes,
compression::decompress_into_arc, compression::decompress_into_arc,
constants::MAX_INLINE_VALUE_SIZE, constants::MAX_INLINE_VALUE_SIZE,
lookup_entry::{LazyLookupValue, LookupEntry, LookupValue}, lookup_entry::{LazyLookupValue, LookupEntry, LookupValue},
...@@ -62,14 +63,21 @@ impl From<LookupValue> for SstLookupResult { ...@@ -62,14 +63,21 @@ impl From<LookupValue> for SstLookupResult {
#[derive(Clone, Default)] #[derive(Clone, Default)]
pub struct BlockWeighter; pub struct BlockWeighter;
impl quick_cache::Weighter<(u32, u16), ArcSlice<u8>> for BlockWeighter { impl quick_cache::Weighter<(u32, u16), ArcBytes> for BlockWeighter {
fn weight(&self, _key: &(u32, u16), val: &ArcSlice<u8>) -> u64 { fn weight(&self, _key: &(u32, u16), val: &ArcBytes) -> u64 {
val.len() as u64 + 8 if val.is_mmap_backed() {
// Mmap-backed blocks are cheap (just a pointer + Arc clone), so we
// assign a small fixed weight. Caching them avoids re-parsing block
// offsets on every lookup.
64
} else {
val.len() as u64 + 8
}
} }
} }
pub type BlockCache = pub type BlockCache =
quick_cache::sync::Cache<(u32, u16), ArcSlice<u8>, BlockWeighter, BuildHasherDefault<FxHasher>>; quick_cache::sync::Cache<(u32, u16), ArcBytes, BlockWeighter, BuildHasherDefault<FxHasher>>;
#[derive(Clone, Debug)] #[derive(Clone, Debug)]
pub struct StaticSortedFileMetaData { pub struct StaticSortedFileMetaData {
...@@ -104,7 +112,9 @@ pub struct StaticSortedFile { ...@@ -104,7 +112,9 @@ pub struct StaticSortedFile {
/// The meta file of this file. /// The meta file of this file.
meta: StaticSortedFileMetaData, meta: StaticSortedFileMetaData,
/// The memory mapped file. /// The memory mapped file.
mmap: Mmap, /// We store as an Arc so we can hand out references (via ArcBytes) that can outlive this
/// struct (not that we expect them to outlive it by very much)
mmap: Arc<Mmap>,
} }
impl StaticSortedFile { impl StaticSortedFile {
...@@ -126,7 +136,10 @@ impl StaticSortedFile { ...@@ -126,7 +136,10 @@ impl StaticSortedFile {
let offset = meta.block_offsets_start(mmap.len()); let offset = meta.block_offsets_start(mmap.len());
let _ = mmap.advise_range(memmap2::Advice::Sequential, offset, mmap.len() - offset); let _ = mmap.advise_range(memmap2::Advice::Sequential, offset, mmap.len() - offset);
} }
let file = Self { meta, mmap }; let file = Self {
meta,
mmap: Arc::new(mmap),
};
Ok(file) Ok(file)
} }
...@@ -229,7 +242,7 @@ impl StaticSortedFile { ...@@ -229,7 +242,7 @@ impl StaticSortedFile {
/// Looks up a key in a key block and the value in a value block. /// Looks up a key in a key block and the value in a value block.
fn lookup_key_block<K: QueryKey>( fn lookup_key_block<K: QueryKey>(
&self, &self,
mut block: ArcSlice<u8>, mut block: ArcBytes,
key_hash: u64, key_hash: u64,
key: &K, key: &K,
has_hash: bool, has_hash: bool,
...@@ -276,7 +289,7 @@ impl StaticSortedFile { ...@@ -276,7 +289,7 @@ impl StaticSortedFile {
&self, &self,
ty: u8, ty: u8,
mut val: &[u8], mut val: &[u8],
key_block_arc: &ArcSlice<u8>, key_block_arc: &ArcBytes,
value_block_cache: &BlockCache, value_block_cache: &BlockCache,
) -> Result<LookupValue> { ) -> Result<LookupValue> {
Ok(match ty { Ok(match ty {
...@@ -285,7 +298,7 @@ impl StaticSortedFile { ...@@ -285,7 +298,7 @@ impl StaticSortedFile {
let size = val.read_u16::<BE>()? as usize; let size = val.read_u16::<BE>()? as usize;
let position = val.read_u32::<BE>()? as usize; let position = val.read_u32::<BE>()? as usize;
let value = self let value = self
.get_value_block(block, value_block_cache)? .get_small_value_block(block, value_block_cache)?
.slice(position..position + size); .slice(position..position + size);
LookupValue::Slice { value } LookupValue::Slice { value }
} }
...@@ -313,7 +326,7 @@ impl StaticSortedFile { ...@@ -313,7 +326,7 @@ impl StaticSortedFile {
&self, &self,
block: u16, block: u16,
key_block_cache: &BlockCache, key_block_cache: &BlockCache,
) -> Result<ArcSlice<u8>, anyhow::Error> { ) -> Result<ArcBytes, anyhow::Error> {
Ok( Ok(
match key_block_cache.get_value_or_guard(&(self.meta.sequence_number, block), None) { match key_block_cache.get_value_or_guard(&(self.meta.sequence_number, block), None) {
GuardResult::Value(block) => block, GuardResult::Value(block) => block,
...@@ -328,7 +341,11 @@ impl StaticSortedFile { ...@@ -328,7 +341,11 @@ impl StaticSortedFile {
} }
/// Gets a value block from the cache or reads it from the file. /// Gets a value block from the cache or reads it from the file.
fn get_value_block(&self, block: u16, value_block_cache: &BlockCache) -> Result<ArcSlice<u8>> { fn get_small_value_block(
&self,
block: u16,
value_block_cache: &BlockCache,
) -> Result<ArcBytes> {
let block = let block =
match value_block_cache.get_value_or_guard(&(self.meta.sequence_number, block), None) { match value_block_cache.get_value_or_guard(&(self.meta.sequence_number, block), None) {
GuardResult::Value(block) => block, GuardResult::Value(block) => block,
...@@ -343,7 +360,7 @@ impl StaticSortedFile { ...@@ -343,7 +360,7 @@ impl StaticSortedFile {
} }
/// Reads a key block from the file. /// Reads a key block from the file.
fn read_key_block(&self, block_index: u16) -> Result<ArcSlice<u8>> { fn read_key_block(&self, block_index: u16) -> Result<ArcBytes> {
self.read_block( self.read_block(
block_index, block_index,
Some(&self.mmap[self.meta.key_compression_dictionary_range()]), Some(&self.mmap[self.meta.key_compression_dictionary_range()]),
...@@ -352,12 +369,12 @@ impl StaticSortedFile { ...@@ -352,12 +369,12 @@ impl StaticSortedFile {
} }
/// Reads a value block from the file. /// Reads a value block from the file.
fn read_small_value_block(&self, block_index: u16) -> Result<ArcSlice<u8>> { fn read_small_value_block(&self, block_index: u16) -> Result<ArcBytes> {
self.read_block(block_index, None, false) self.read_block(block_index, None, false)
} }
/// Reads a value block from the file. /// Reads a value block from the file.
fn read_value_block(&self, block_index: u16) -> Result<ArcSlice<u8>> { fn read_value_block(&self, block_index: u16) -> Result<ArcBytes> {
self.read_block(block_index, None, true) self.read_block(block_index, None, true)
} }
...@@ -368,8 +385,25 @@ impl StaticSortedFile { ...@@ -368,8 +385,25 @@ impl StaticSortedFile {
block_index: u16, block_index: u16,
compression_dictionary: Option<&[u8]>, compression_dictionary: Option<&[u8]>,
long_term: bool, long_term: bool,
) -> Result<ArcSlice<u8>> { ) -> Result<ArcBytes> {
let (uncompressed_length, block) = self.get_compressed_block(block_index)?; let (uncompressed_length, block) = self.get_raw_block(block_index)?;
// 0 means the block was not compressed, just return the slice into the mmap
if uncompressed_length == 0 {
// SAFETY: get_raw_block only returns reference into the mmap.
return Ok(unsafe { ArcBytes::from_mmap(self.mmap.clone(), block) });
}
// Advise Sequential only here: we're about to linearly scan the block
// through the decompressor. For uncompressed blocks (returned above)
// and lazy medium values (which call get_raw_block directly without
// decompressing), the file-level Random advice applies.
#[cfg(unix)]
let _ = self.mmap.advise_range(
memmap2::Advice::Sequential,
block.as_ptr() as usize - self.mmap.as_ptr() as usize,
block.len(),
);
let buffer = decompress_into_arc( let buffer = decompress_into_arc(
uncompressed_length, uncompressed_length,
...@@ -377,11 +411,11 @@ impl StaticSortedFile { ...@@ -377,11 +411,11 @@ impl StaticSortedFile {
compression_dictionary, compression_dictionary,
long_term, long_term,
)?; )?;
Ok(ArcSlice::from(buffer)) Ok(ArcBytes::from(buffer))
} }
/// Gets the slice of the compressed block from the memory mapped file. /// Gets the slice of the compressed block from the memory mapped file.
fn get_compressed_block(&self, block_index: u16) -> Result<(u32, &[u8])> { fn get_raw_block(&self, block_index: u16) -> Result<(u32, &[u8])> {
#[cfg(feature = "strict_checks")] #[cfg(feature = "strict_checks")]
if block_index >= self.meta.block_count { if block_index >= self.meta.block_count {
bail!( bail!(
...@@ -429,13 +463,8 @@ impl StaticSortedFile { ...@@ -429,13 +463,8 @@ impl StaticSortedFile {
self.meta.blocks_start() self.meta.blocks_start()
); );
} }
#[cfg(unix)] let uncompressed_length =
let _ = self.mmap.advise_range( u32::from_be_bytes(self.mmap[block_start..block_start + 4].try_into()?);
memmap2::Advice::Sequential,
block_start,
block_end - block_start,
);
let uncompressed_length = (&self.mmap[block_start..block_start + 4]).read_u32::<BE>()?;
let block = &self.mmap[block_start + 4..block_end]; let block = &self.mmap[block_start + 4..block_end];
Ok((uncompressed_length, block)) Ok((uncompressed_length, block))
} }
...@@ -452,15 +481,15 @@ pub struct StaticSortedFileIter<'l> { ...@@ -452,15 +481,15 @@ pub struct StaticSortedFileIter<'l> {
} }
struct CurrentKeyBlock { struct CurrentKeyBlock {
offsets: ArcSlice<u8>, offsets: ArcBytes,
entries: ArcSlice<u8>, entries: ArcBytes,
entry_count: usize, entry_count: usize,
index: usize, index: usize,
hash_len: u8, hash_len: u8,
} }
struct CurrentIndexBlock { struct CurrentIndexBlock {
entries: ArcSlice<u8>, entries: ArcBytes,
block_indices_count: usize, block_indices_count: usize,
index: usize, index: usize,
} }
...@@ -534,7 +563,7 @@ impl<'l> StaticSortedFileIter<'l> { ...@@ -534,7 +563,7 @@ impl<'l> StaticSortedFileIter<'l> {
let value = if ty == KEY_BLOCK_ENTRY_TYPE_MEDIUM { let value = if ty == KEY_BLOCK_ENTRY_TYPE_MEDIUM {
let mut val = val; let mut val = val;
let block = val.read_u16::<BE>()?; let block = val.read_u16::<BE>()?;
let (uncompressed_size, block) = self.this.get_compressed_block(block)?; let (uncompressed_size, block) = self.this.get_raw_block(block)?;
LazyLookupValue::Medium { LazyLookupValue::Medium {
uncompressed_size, uncompressed_size,
block, block,
......
...@@ -79,8 +79,10 @@ pub enum EntryValue<'l> { ...@@ -79,8 +79,10 @@ pub enum EntryValue<'l> {
Small { value: &'l [u8] }, Small { value: &'l [u8] },
/// Medium-sized value. They are stored in their own value block. /// Medium-sized value. They are stored in their own value block.
Medium { value: &'l [u8] }, Medium { value: &'l [u8] },
/// Medium-sized value. They are stored in their own value block. Precompressed. /// Medium-sized value. They are stored in their own value block. In the raw form as on disk.
MediumCompressed { MediumRaw {
/// The uncompressed size of the block data. `0` means the block is stored uncompressed
/// (and thus the size is the `len` of the block)
uncompressed_size: u32, uncompressed_size: u32,
block: &'l [u8], block: &'l [u8],
}, },
...@@ -224,6 +226,16 @@ fn compute_key_compression_dictionary<E: Entry>( ...@@ -224,6 +226,16 @@ fn compute_key_compression_dictionary<E: Entry>(
Ok(result) Ok(result)
} }
enum CompressionConfig<'a> {
/// Attempt compression; use the result only if it's smaller than the original.
TryCompress {
dict: Option<&'a [u8]>,
long_term: bool,
},
/// Write the block uncompressed.
Uncompressed,
}
struct BlockWriter<'l> { struct BlockWriter<'l> {
buffer: &'l mut Vec<u8>, buffer: &'l mut Vec<u8>,
block_offsets: Vec<u32>, block_offsets: Vec<u32>,
...@@ -255,32 +267,69 @@ impl<'l> BlockWriter<'l> { ...@@ -255,32 +267,69 @@ impl<'l> BlockWriter<'l> {
#[tracing::instrument(level = "trace", skip_all)] #[tracing::instrument(level = "trace", skip_all)]
fn write_key_block(&mut self, block: &[u8], dict: &[u8]) -> Result<()> { fn write_key_block(&mut self, block: &[u8], dict: &[u8]) -> Result<()> {
self.write_block(block, Some(dict), false) self.write_block(
.context("Failed to write key block") block,
CompressionConfig::TryCompress {
dict: Some(dict),
long_term: false,
},
)
.context("Failed to write key block")
} }
#[tracing::instrument(level = "trace", skip_all)] #[tracing::instrument(level = "trace", skip_all)]
fn write_index_block(&mut self, block: &[u8], dict: &[u8]) -> Result<()> { fn write_index_block(&mut self, block: &[u8]) -> Result<()> {
self.write_block(block, Some(dict), false) // Index blocks are minimally compressible so don't try
self.write_block(block, CompressionConfig::Uncompressed)
.context("Failed to write index block") .context("Failed to write index block")
} }
#[tracing::instrument(level = "trace", skip_all)] #[tracing::instrument(level = "trace", skip_all)]
fn write_small_value_block(&mut self, block: &[u8]) -> Result<()> { fn write_small_value_block(&mut self, block: &[u8]) -> Result<()> {
self.write_block(block, None, false) self.write_block(
.context("Failed to write small value block") block,
CompressionConfig::TryCompress {
dict: None,
long_term: false,
},
)
.context("Failed to write small value block")
} }
#[tracing::instrument(level = "trace", skip_all)] #[tracing::instrument(level = "trace", skip_all)]
fn write_value_block(&mut self, block: &[u8]) -> Result<()> { fn write_value_block(&mut self, block: &[u8]) -> Result<()> {
self.write_block(block, None, true) self.write_block(
.context("Failed to write value block") block,
CompressionConfig::TryCompress {
dict: None,
long_term: true,
},
)
.context("Failed to write value block")
} }
fn write_block(&mut self, block: &[u8], dict: Option<&[u8]>, long_term: bool) -> Result<()> { fn write_block(&mut self, block: &[u8], compression: CompressionConfig<'_>) -> Result<()> {
let uncompressed_size = block.len().try_into().unwrap(); let (uncompressed_size, data_to_write): (u32, &[u8]) = match compression {
self.compress_block_into_buffer(block, dict, long_term)?; CompressionConfig::TryCompress { dict, long_term } => {
let len = (self.buffer.len() + 4).try_into().unwrap(); self.compress_block_into_buffer(block, dict, long_term)?;
// Same threshold as LevelDB/RocksDB: require at least 12.5% savings to store
// compressed.
// See https://github.com/google/leveldb/blob/ac691084fdc5546421a55b25e7653d450e5a25fb/table/table_builder.cc#L164
// Uncompressed blocks take more time to read but we can directly leverage the mmap
// on the read side, compressed blocks need to be decompressed and managed in a
// cache. So we should only do it if we expect to save time.
if self.buffer.len() < block.len() - (block.len() / 8) {
// Compression helped - use compressed data
(block.len().try_into().unwrap(), self.buffer.as_slice())
} else {
// Compression didn't help - use uncompressed with sentinel size value
(0, block)
}
}
CompressionConfig::Uncompressed => (0, block),
};
let len: u32 = (data_to_write.len() + 4).try_into().unwrap();
let offset = self let offset = self
.block_offsets .block_offsets
.last() .last()
...@@ -292,10 +341,10 @@ impl<'l> BlockWriter<'l> { ...@@ -292,10 +341,10 @@ impl<'l> BlockWriter<'l> {
self.writer self.writer
.write_u32::<BE>(uncompressed_size) .write_u32::<BE>(uncompressed_size)
.context("Failed to write uncompressed size")?; .context("Failed to write uncompressed_size")?;
self.writer self.writer
.write_all(self.buffer) .write_all(data_to_write)
.context("Failed to write compressed block")?; .context("Failed to write block data")?;
self.buffer.clear(); self.buffer.clear();
Ok(()) Ok(())
} }
...@@ -372,7 +421,7 @@ fn write_value_blocks( ...@@ -372,7 +421,7 @@ fn write_value_blocks(
value_locations.push((block_index, 0)); value_locations.push((block_index, 0));
writer.write_value_block(value)?; writer.write_value_block(value)?;
} }
EntryValue::MediumCompressed { EntryValue::MediumRaw {
uncompressed_size, uncompressed_size,
block, block,
} => { } => {
...@@ -436,7 +485,7 @@ fn write_key_blocks_and_compute_amqf( ...@@ -436,7 +485,7 @@ fn write_key_blocks_and_compute_amqf(
value.len().try_into().unwrap(), value.len().try_into().unwrap(),
); );
} }
EntryValue::Medium { .. } | EntryValue::MediumCompressed { .. } => { EntryValue::Medium { .. } | EntryValue::MediumRaw { .. } => {
block.put_medium(entry, value_location.0); block.put_medium(entry, value_location.0);
} }
EntryValue::Large { blob } => { EntryValue::Large { blob } => {
...@@ -526,7 +575,7 @@ fn write_key_blocks_and_compute_amqf( ...@@ -526,7 +575,7 @@ fn write_key_blocks_and_compute_amqf(
} }
let _ = writer.next_block_index(); let _ = writer.next_block_index();
index_block.finish(); index_block.finish();
writer.write_index_block(buffer, key_compression_dictionary)?; writer.write_index_block(buffer)?;
buffer.clear(); buffer.clear();
Ok(turbo_bincode_encode(&AmqfBincodeWrapper(filter)).expect("AMQF serialization failed")) Ok(turbo_bincode_encode(&AmqfBincodeWrapper(filter)).expect("AMQF serialization failed"))
......
...@@ -9,7 +9,7 @@ use std::{ ...@@ -9,7 +9,7 @@ use std::{
use anyhow::{Ok, Result}; use anyhow::{Ok, Result};
use parking_lot::Mutex; use parking_lot::Mutex;
use turbo_persistence::{ use turbo_persistence::{
ArcSlice, CompactConfig, KeyBase, StoreKey, TurboPersistence, ValueBuffer, ArcBytes, CompactConfig, KeyBase, StoreKey, TurboPersistence, ValueBuffer,
}; };
use turbo_tasks::{JoinHandle, message_queue::TimingEvent, spawn, turbo_tasks}; use turbo_tasks::{JoinHandle, message_queue::TimingEvent, spawn, turbo_tasks};
...@@ -71,7 +71,7 @@ impl KeyValueDatabase for TurboKeyValueDatabase { ...@@ -71,7 +71,7 @@ impl KeyValueDatabase for TurboKeyValueDatabase {
} }
type ValueBuffer<'l> type ValueBuffer<'l>
= ArcSlice<u8> = ArcBytes
where where
Self: 'l; Self: 'l;
...@@ -176,7 +176,7 @@ pub struct TurboWriteBatch<'a> { ...@@ -176,7 +176,7 @@ pub struct TurboWriteBatch<'a> {
impl<'a> BaseWriteBatch<'a> for TurboWriteBatch<'a> { impl<'a> BaseWriteBatch<'a> for TurboWriteBatch<'a> {
type ValueBuffer<'l> type ValueBuffer<'l>
= ArcSlice<u8> = ArcBytes
where where
Self: 'l, Self: 'l,
'a: 'l; 'a: 'l;
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment