Unverified Commit 4550be40 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

turbo-persistence: read compaction inputs with pread instead of mmap (#100336)

## Summary

Compaction iterated its input SST files through an mmap with
`MADV_SEQUENTIAL`. This does not fit the block layout pattern (which
tends to be ValueBlock, ValueBlock, KeyBlock....) because iteration
starts from key blocks, so the readahead isn't well targeted. OS level
reclaim can recover this memory but it isn't eager and we _know_ after
reading a block that we don't need it any more.

`StaticSortedFileIter`, which only compaction uses, now always reads
blocks with `pread` into owned buffers, independent of the overall
database's `AccessMode`, so an open input file holds only the blocks it
currently uses.

This also enhances SharedBytes with a mechanism to be constructed from
an `uninit` buffer (useful for decompression and `pread` targets). This
unifies a few codepaths and allows us to avoid some small buffer copies.

### Measurements

Full compaction on Linux (podman VM, kernel 7.2, XFS on virtio, page
cache dropped before each run, single threaded), 2 rounds:

| | time | max RSS | peak mapped file pages |
|---|---|---|---|
| 1.1 GB zstd database, values 16 B – 8 KiB, mmap (before) | 2.10 / 2.33
s | 315 MB | 301 – 318 MB |
| same, `pread` (this PR) | 2.07 / 2.04 s | 10 MB | 5 MB |
| 230 MB database of 16Mi tiny entries, mmap (before) | 2.73 / 2.64 s |
245 MB | 201 MB |
| same, `pread` (this PR) | 2.35 / 2.37 s | 57 MB | 6 MB |

The cgroup's `memory.peak` is the same in both, since it is dominated by
page cache, which is charged either way and reclaimable: with a 512 MB
cgroup limit both variants complete without OOM kills. What `pread`
removes is the input files from the process' RSS.
### Why not mmap with `MADV_RANDOM` and explicit `WILLNEED`/`DONTNEED`

That would let the iterator prefetch the blocks it is about to read and
drop those behind it, but at best it reaches the memory profile `pread`
already has, and adds costs:

- `MADV_DONTNEED` on a file mapping only unmaps the pages, they stay in
page cache like with `pread`.
- With `MADV_RANDOM` there is no readahead, and `WILLNEED` only starts
the I/O, it doesn't map pages, so every 4 KiB page still takes a (minor)
fault. `pread` copies ~10 KB per call without page table work.
- Unmapping requires a TLB shootdown on every CPU running the process,
and compaction runs merges on many threads in the same address space as
turbo-tasks. This cost grows with concurrency.
- `WILLNEED` and `DONTNEED` take kernel level locks, which can lead to
contention in addition to general syscall overhead.


<!-- NEXT_JS_LLM -->

---------
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: 's avatarLuke Sandberg <210140+lukesandberg@users.noreply.github.com>
parent 404d55e1
...@@ -2,6 +2,7 @@ use std::{ ...@@ -2,6 +2,7 @@ use std::{
borrow::Borrow, borrow::Borrow,
fmt::{self, Debug, Formatter}, fmt::{self, Debug, Formatter},
hash::{Hash, Hasher}, hash::{Hash, Hasher},
io::{self, BorrowedCursor},
ops::{Deref, Range}, ops::{Deref, Range},
sync::Arc, sync::Arc,
}; };
...@@ -12,7 +13,7 @@ use memmap2::Mmap; ...@@ -12,7 +13,7 @@ use memmap2::Mmap;
use crate::{ use crate::{
Compression, Compression,
compression::decompress_into_arc, compression::decompress_into_arc,
shared_bytes::{INLINE_CAPACITY, SharedBytes, is_subslice_of}, shared_bytes::{INLINE_CAPACITY, SharedBytes, fill_uninit, is_subslice_of},
}; };
/// The representation of an `ArcBytes`. /// The representation of an `ArcBytes`.
/// ///
...@@ -219,6 +220,19 @@ impl SharedBytes for ArcBytes { ...@@ -219,6 +220,19 @@ impl SharedBytes for ArcBytes {
)?)) )?))
} }
fn read_filled(
len: usize,
fill: impl FnOnce(BorrowedCursor<'_, u8>) -> io::Result<()>,
) -> io::Result<Self> {
let mut bytes = Arc::<[u8]>::new_uninit_slice(len);
fill_uninit(
Arc::get_mut(&mut bytes).expect("the Arc was just created"),
fill,
)?;
// SAFETY: `fill_uninit` returned `Ok`, so all bytes were written.
Ok(Self::from(unsafe { bytes.assume_init() }))
}
#[inline] #[inline]
fn from_inline(bytes: &[u8]) -> Self { fn from_inline(bytes: &[u8]) -> Self {
assert!( assert!(
......
...@@ -1385,10 +1385,9 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES> ...@@ -1385,10 +1385,9 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES>
} }
let mut guard = self.acquire_write_operation("compaction")?; let mut guard = self.acquire_write_operation("compaction")?;
// Free block caches and SST mmaps before compaction. The block caches // Free block caches and SST mmaps before compaction. Neither is used during
// are not used during compaction (we iterate uncached, through separate // compaction (we iterate uncached with `pread`), so clearing them upfront frees
// `MADV_SEQUENTIAL` mmaps), so clearing them upfront frees memory for // memory for the merge work.
// the merge work.
self.clear_cache(); self.clear_cache();
let mut sequence_number; let mut sequence_number;
...@@ -1697,9 +1696,10 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES> ...@@ -1697,9 +1696,10 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES>
}) })
} }
}; };
// Open SST files independently for compaction. // Open SST files independently for compaction. They are read
// Uses MADV_SEQUENTIAL for better OS page management // with `pread`, which keeps the memory of an open file to the
// and avoids caching mmaps on MetaEntry's OnceLock. // blocks it currently holds, and doesn't cache an mmap on
// MetaEntry's OnceLock.
let iters = indices let iters = indices
.iter() .iter()
.map(|&index| { .map(|&index| {
...@@ -1711,7 +1711,6 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES> ...@@ -1711,7 +1711,6 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES>
path, path,
entry.sst_metadata(), entry.sst_metadata(),
meta_file.compression(), meta_file.compression(),
self.config.access_mode,
) )
}) })
.collect::<Result<Vec<_>>>()?; .collect::<Result<Vec<_>>>()?;
......
#![cfg_attr(target_os = "wasi", feature(wasi_ext))] #![cfg_attr(target_os = "wasi", feature(wasi_ext))]
#![feature(core_io_borrowed_buf)]
#![feature(once_cell_try)] #![feature(once_cell_try)]
// The positional `BorrowedCursor` reads `pread_buf` uses: unix gates them behind `read_buf_at`,
// Windows behind `seek_read_exact_seek_write_all`, WASI behind `wasi_ext`.
#![cfg_attr(unix, feature(read_buf_at))]
#![cfg_attr(windows, feature(seek_read_exact_seek_write_all))]
#![feature(sync_unsafe_cell)] #![feature(sync_unsafe_cell)]
// Miri compiles a reduced test subset, leaving helpers from disabled tests intentionally unused. // Miri compiles a reduced test subset, leaving helpers from disabled tests intentionally unused.
#![cfg_attr(miri, allow(dead_code, unused_imports))] #![cfg_attr(miri, allow(dead_code, unused_imports))]
......
...@@ -2,6 +2,7 @@ use std::{ ...@@ -2,6 +2,7 @@ use std::{
borrow::Borrow, borrow::Borrow,
fmt::{self, Debug, Formatter}, fmt::{self, Debug, Formatter},
hash::{Hash, Hasher}, hash::{Hash, Hasher},
io::{self, BorrowedCursor},
ops::{Deref, Range}, ops::{Deref, Range},
rc::Rc, rc::Rc,
}; };
...@@ -12,7 +13,7 @@ use memmap2::Mmap; ...@@ -12,7 +13,7 @@ use memmap2::Mmap;
use crate::{ use crate::{
Compression, Compression,
compression::decompress_into_rc, compression::decompress_into_rc,
shared_bytes::{INLINE_CAPACITY, SharedBytes, is_subslice_of}, shared_bytes::{INLINE_CAPACITY, SharedBytes, fill_uninit, is_subslice_of},
}; };
/// The representation of an `RcBytes`. /// The representation of an `RcBytes`.
...@@ -194,6 +195,19 @@ impl SharedBytes for RcBytes { ...@@ -194,6 +195,19 @@ impl SharedBytes for RcBytes {
)?)) )?))
} }
fn read_filled(
len: usize,
fill: impl FnOnce(BorrowedCursor<'_, u8>) -> io::Result<()>,
) -> io::Result<Self> {
let mut bytes = Rc::<[u8]>::new_uninit_slice(len);
fill_uninit(
Rc::get_mut(&mut bytes).expect("the Rc was just created"),
fill,
)?;
// SAFETY: `fill_uninit` returned `Ok`, so all bytes were written.
Ok(Self::from(unsafe { bytes.assume_init() }))
}
#[inline] #[inline]
fn from_inline(bytes: &[u8]) -> Self { fn from_inline(bytes: &[u8]) -> Self {
assert!( assert!(
......
use std::ops::{Deref, Range}; use std::{
io::{self, BorrowedBuf, BorrowedCursor},
mem::MaybeUninit,
ops::{Deref, Range},
};
#[cfg(feature = "mmap")] #[cfg(feature = "mmap")]
use memmap2::Mmap; use memmap2::Mmap;
...@@ -62,6 +66,14 @@ pub trait SharedBytes: Clone + Deref<Target = [u8]> + Sized { ...@@ -62,6 +66,14 @@ pub trait SharedBytes: Clone + Deref<Target = [u8]> + Sized {
block: &[u8], block: &[u8],
) -> anyhow::Result<Self>; ) -> anyhow::Result<Self>;
/// Allocates `len` uninitialized bytes, lets `fill` write all of them (e.g. by reading from a
/// file) and returns them. The bytes are written in place, so they are neither zeroed before
/// `fill` nor copied after it. Fails if `fill` leaves any of them unwritten.
fn read_filled(
len: usize,
fill: impl FnOnce(BorrowedCursor<'_, u8>) -> io::Result<()>,
) -> io::Result<Self>;
/// Copies `bytes` into an inline buffer, so the result owns them and borrows nothing. /// Copies `bytes` into an inline buffer, so the result owns them and borrows nothing.
/// ///
/// # Panics /// # Panics
...@@ -71,6 +83,23 @@ pub trait SharedBytes: Clone + Deref<Target = [u8]> + Sized { ...@@ -71,6 +83,23 @@ pub trait SharedBytes: Clone + Deref<Target = [u8]> + Sized {
fn from_inline(block: &[u8]) -> Self; fn from_inline(block: &[u8]) -> Self;
} }
/// Lets `fill` write into `buf` through a [`BorrowedCursor`], which tracks the bytes it wrote.
/// Returns `Ok` only if all of `buf` was written, so the caller may then assume it is initialized.
pub(crate) fn fill_uninit(
buf: &mut [MaybeUninit<u8>],
fill: impl FnOnce(BorrowedCursor<'_, u8>) -> io::Result<()>,
) -> io::Result<()> {
let mut buf = BorrowedBuf::from(buf);
fill(buf.unfilled())?;
if buf.len() != buf.capacity() {
return Err(io::Error::new(
io::ErrorKind::UnexpectedEof,
format!("only {} of {} bytes were read", buf.len(), buf.capacity()),
));
}
Ok(())
}
/// Returns `true` if `subslice` lies entirely within `backing`. /// Returns `true` if `subslice` lies entirely within `backing`.
pub(crate) fn is_subslice_of(subslice: &[u8], backing: &[u8]) -> bool { pub(crate) fn is_subslice_of(subslice: &[u8], backing: &[u8]) -> bool {
let backing = backing.as_ptr_range(); let backing = backing.as_ptr_range();
......
...@@ -2497,9 +2497,7 @@ fn count_tombstones( ...@@ -2497,9 +2497,7 @@ fn count_tombstones(
sequence_number: entry.sequence_number, sequence_number: entry.sequence_number,
block_count: entry.block_count, block_count: entry.block_count,
}; };
for item in for item in StaticSortedFileIter::open(path, sst, Compression::Lz4)? {
StaticSortedFileIter::open(path, sst, Compression::Lz4, crate::mmap_access_mode())?
{
if matches!( if matches!(
item?.value, item?.value,
IterValue::KeyDeleted | IterValue::KeyValueDeleted { .. } IterValue::KeyDeleted | IterValue::KeyValueDeleted { .. }
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment