Unverified Commit e2e7393c authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

turbo-persistence: prefetch multi-page SST blocks with MADV_WILLNEED (#100283)

### What?

turbo-persistence SST lookup mappings keep `MADV_RANDOM`, and also
prefetch whole multi-page blocks:

- **Block hint:** before a block's first full read in an SST open (CRC
check and decompression), the block's page-aligned range gets
`MADV_WILLNEED`. The hint is issued before any byte of the block is
read, header included, and only when the block spans more than one page
(using the system page size).
- Medium value blocks bypass the block cache, so they get the hint on
every read.
  - Already-verified blocks and single-page blocks get no syscall.
- **Block-offset table:** gets `MADV_WILLNEED` at open, instead of
`MADV_SEQUENTIAL`.
- **Unknown page size:** if `sysconf(_SC_PAGESIZE)` fails, block hints
can't be computed, so `MADV_RANDOM` isn't applied either and the mapping
keeps default advice (read-around still prefetches). The offset-table
hint is applied in both cases.
- **Unchanged:** compaction iteration (`MADV_SEQUENTIAL`), meta/blob
files, File access mode, non-Unix platforms and the on-disk format.

### Why?

See #100250.
- With `MADV_RANDOM` alone, a cold multi-page block is faulted in one
page at a time.
- Dropping the advice (the first version of this PR) lets kernel
read-around fix that, but it also reads neighboring blocks that are
often never used.

The two variants were compared on a real workload:
- **Setup:** warm `next build`s of
`bench/nested-deps-app-router-many-pages`, with a new
`NEXT_DEPLOYMENT_ID` each build. The cache snapshot contained fresh,
compacted, hot_bottom and cold_bottom SSTs.
- **Runs:** cold guest page cache, Linux, 4 KiB pages, 5 rotated trials
per variant.

| Variant | Median build | Major faults | Cache resident after |
| --- | --- | --- | --- |
| canary (`Random`) | 40.0 s | ~101k | 527 MiB |
| default advice | 39.6 s | ~14.2k | 545 MiB |
| `Random` + `WillNeed` everywhere (this PR) | 38.2 s | ~13.4k | 527 MiB
|
| `WillNeed` for cold_bottom only, default for the rest | 38.8 s |
~13.8k | 533 MiB |

- Build times are within run-to-run noise. Cache reads are a small part
of this build, so this is not a claimed wall-time win.
- The targeted hints match default advice's fault reduction, with less
data pulled into the page cache.
- Reading unneeded pages costs more on slow or compressed backing
storage such as squashfs. That is another reason to avoid read-around,
though it wasn't benchmarked here.
- Not measured: memory pressure, macOS and 16 KiB pages. With 16 KiB
pages more blocks fit in one page and get no hint.

### How?

- `get_raw_block_slice` takes a `will_need` flag and issues the hint
after locating the block, before reading its header.
- Lookups pass `true` when the block's verified-CRC bit is unset; the
index block and medium values always pass `true`; iteration passes
`false`.
- Adds a `libc` dependency on Unix for `sysconf(_SC_PAGESIZE)`, queried
once and cached. If it can't be determined, both the block hints and
`MADV_RANDOM` are skipped.
- Tests:
- Linux `/proc/self/smaps` check that lookup mappings carry `rr` and not
`sr`;
- a test-only hint log verifying the offset-table hint, multi-page-only
first-read hints, no hints on re-reads, and per-read medium-value hints;
  - pure page-boundary tests for 4 KiB and 16 KiB.
- README documents the advice policy.

<!-- NEXT_JS_LLM -->


<!-- fleet 5f39f2b2-0882-4566-ae42-cf64574c5747 -->

---------
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: 's avatarLuke Sandberg <210140+lukesandberg@users.noreply.github.com>
parent 6d8388a9
...@@ -9100,6 +9100,7 @@ dependencies = [ ...@@ -9100,6 +9100,7 @@ dependencies = [
"either", "either",
"fs-err", "fs-err",
"jiff", "jiff",
"libc",
"lz4_flex", "lz4_flex",
"memmap2", "memmap2",
"nohash-hasher", "nohash-hasher",
......
...@@ -44,6 +44,9 @@ xxhash-rust = { workspace = true } ...@@ -44,6 +44,9 @@ xxhash-rust = { workspace = true }
lz4_flex = { workspace = true } lz4_flex = { workspace = true }
zstd = { workspace = true } zstd = { workspace = true }
[target.'cfg(unix)'.dependencies]
libc = "0.2"
[dev-dependencies] [dev-dependencies]
criterion = { workspace = true } criterion = { workspace = true }
rand = { workspace = true } rand = { workspace = true }
......
...@@ -275,6 +275,24 @@ files, from newest to oldest; there is no ordering dependency between families. ...@@ -275,6 +275,24 @@ files, from newest to oldest; there is no ordering dependency between families.
- read value as inline, or by using the block index in the key to find the value elsewhere in the file. - read value as inline, or by using the block index in the key to find the value elsewhere in the file.
- not found -> break - not found -> break
### Memory map advice
Lookups usually touch only some blocks of an SST file, and a block is read in full the first time it
is used (checksum verification and decompression). On Unix, the lookup mapping therefore uses:
- `MADV_RANDOM` for the whole file, so a fault doesn't read around into neighboring blocks.
- `MADV_WILLNEED` for the block offset table when the file is opened, since every block access
reads it.
- A best-effort `MADV_WILLNEED` for a block that spans more than one page, before any of its bytes is
read: on its first full read in this open, and on every read of a medium value block (those bypass
the block cache). Without the hint, `MADV_RANDOM` would fault such a block in one page at a time.
Blocks within a single page, and blocks whose checksum was already verified, get no hint.
If the system page size can't be determined, block hints are skipped, so `MADV_RANDOM` isn't applied
either and the mapping keeps the default advice.
Compaction reads SST files through separate `MADV_SEQUENTIAL` mappings.
## Writing ## Writing
Writing starts by creating a new WriteBatch. It maintains an atomic counter of the next free sequence number. Writing starts by creating a new WriteBatch. It maintains an atomic counter of the next free sequence number.
......
...@@ -1386,9 +1386,9 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES> ...@@ -1386,9 +1386,9 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES>
let mut guard = self.acquire_write_operation("compaction")?; let mut guard = self.acquire_write_operation("compaction")?;
// Free block caches and SST mmaps before compaction. The block caches // Free block caches and SST mmaps before compaction. The block caches
// are not used during compaction (we iterate uncached), and any cached // are not used during compaction (we iterate uncached, through separate
// SST mmaps would use MADV_RANDOM which is wrong for sequential scans. // `MADV_SEQUENTIAL` mmaps), so clearing them upfront frees memory for
// Clearing them upfront frees memory for the merge work. // the merge work.
self.clear_cache(); self.clear_cache();
let mut sequence_number; let mut sequence_number;
......
...@@ -463,12 +463,24 @@ impl StaticSortedFile { ...@@ -463,12 +463,24 @@ impl StaticSortedFile {
file.metadata().map(|m| m.len()).unwrap_or(0) file.metadata().map(|m| m.len()).unwrap_or(0)
) )
})?; })?;
// Lookups touch only a few blocks of most SSTs, so `Random` stops the kernel from
// reading around every fault. In exchange, `get_raw_block_slice` asks for a
// multi-page block with `WillNeed` before its first full read. Those hints need the
// page size; without it, keep the default advice so read-around still prefetches.
// Every block access reads the block-offset table, so request that up front.
#[cfg(unix)] #[cfg(unix)]
{ {
mmap.advise(memmap2::Advice::Random)?; if page_size().is_some() {
mmap.advise(memmap2::Advice::Random)?;
}
let offset = meta.block_offsets_start(mmap.len()); let offset = meta.block_offsets_start(mmap.len());
let _ = let _ = mmap.advise_range(
mmap.advise_range(memmap2::Advice::Sequential, offset, mmap.len() - offset); memmap2::Advice::WillNeed,
offset,
mmap.len().saturating_sub(offset),
);
#[cfg(test)]
will_need_log::record(offset..mmap.len());
} }
advise_mmap_for_persistence(&mmap)?; advise_mmap_for_persistence(&mmap)?;
StaticSortedFileBacking::Mmap(Arc::new(mmap)) StaticSortedFileBacking::Mmap(Arc::new(mmap))
...@@ -820,8 +832,12 @@ fn get_or_read_block<'l>( ...@@ -820,8 +832,12 @@ fn get_or_read_block<'l>(
let _ = verified_blocks; let _ = verified_blocks;
#[cfg(feature = "mmap")] #[cfg(feature = "mmap")]
let mmap_block = if let StaticSortedFileBacking::Mmap(mmap) = backing { let mmap_block = if let StaticSortedFileBacking::Mmap(mmap) = backing {
// An unverified block is about to be read in full for the first time in this open (CRC
// verification and, if compressed, decompression), so hint it. A verified block is either
// in the block cache or touches only the bytes a lookup needs.
let will_need = !is_block_verified(block_index, verified_blocks);
let (uncompressed_length, checksum, block_data) = let (uncompressed_length, checksum, block_data) =
get_raw_block_slice(mmap, meta, block_index).with_context(|| { get_raw_block_slice(mmap, meta, block_index, will_need).with_context(|| {
format!( format!(
"Failed to read raw block {} from {:08}.sst", "Failed to read raw block {} from {:08}.sst",
block_index, meta.sequence_number block_index, meta.sequence_number
...@@ -891,13 +907,73 @@ fn get_or_read_block<'l>( ...@@ -891,13 +907,73 @@ fn get_or_read_block<'l>(
)) ))
} }
/// Asks the kernel to read `range` of the mapping ahead, if it spans more than one page.
///
/// Lookup mappings have `Random` advice, so without this hint the kernel faults in each page of a
/// multi-page block separately. Readahead is best effort, and single-page ranges gain nothing.
#[cfg(feature = "mmap")]
fn will_need_range(mmap: &Mmap, range: Range<usize>) {
#[cfg(unix)]
if let Some(page_size) = page_size()
&& spans_multiple_pages(&range, page_size)
{
#[cfg(test)]
will_need_log::record(range.clone());
let _ = mmap.advise_range(memmap2::Advice::WillNeed, range.start, range.len());
}
#[cfg(not(unix))]
let _ = (mmap, range);
}
/// Whether the non-empty `range` crosses a page boundary.
#[cfg(all(feature = "mmap", unix))]
fn spans_multiple_pages(range: &Range<usize>, page_size: usize) -> bool {
debug_assert!(!range.is_empty());
range.start / page_size != (range.end - 1) / page_size
}
/// The system page size, queried once. `None` if it can't be determined, in which case block hints
/// are skipped.
#[cfg(all(feature = "mmap", unix))]
fn page_size() -> Option<usize> {
static PAGE_SIZE: std::sync::OnceLock<Option<usize>> = std::sync::OnceLock::new();
*PAGE_SIZE.get_or_init(|| {
// SAFETY: `sysconf` has no preconditions.
let size = unsafe { libc::sysconf(libc::_SC_PAGESIZE) };
usize::try_from(size).ok().filter(|&size| size > 0)
})
}
/// Records the ranges passed to `MADV_WILLNEED` on this thread, for tests.
#[cfg(all(test, feature = "mmap", unix))]
pub(crate) mod will_need_log {
use std::{cell::RefCell, ops::Range};
thread_local! {
static LOG: RefCell<Vec<Range<usize>>> = const { RefCell::new(Vec::new()) };
}
pub(crate) fn record(range: Range<usize>) {
LOG.with(|log| log.borrow_mut().push(range));
}
/// Returns and clears this thread's recorded ranges.
pub(crate) fn take() -> Vec<Range<usize>> {
LOG.with(|log| std::mem::take(&mut *log.borrow_mut()))
}
}
/// Gets the raw block slice directly from a memory-mapped file. /// Gets the raw block slice directly from a memory-mapped file.
/// Returns `(uncompressed_length, checksum, block_data)`. /// Returns `(uncompressed_length, checksum, block_data)`.
///
/// With `will_need`, the caller is about to read the whole block, so the block (header included)
/// is requested with `WillNeed` before any of its bytes is read; see [`will_need_range`].
#[cfg(feature = "mmap")] #[cfg(feature = "mmap")]
fn get_raw_block_slice<'a>( fn get_raw_block_slice<'a>(
mmap: &'a Mmap, mmap: &'a Mmap,
meta: &StaticSortedFileMetaData, meta: &StaticSortedFileMetaData,
block_index: u16, block_index: u16,
will_need: bool,
) -> Result<(u32, u32, &'a [u8])> { ) -> Result<(u32, u32, &'a [u8])> {
#[cfg(feature = "strict_checks")] #[cfg(feature = "strict_checks")]
if block_index >= meta.block_count { if block_index >= meta.block_count {
...@@ -941,11 +1017,18 @@ fn get_raw_block_slice<'a>( ...@@ -941,11 +1017,18 @@ fn get_raw_block_slice<'a>(
); );
} }
ensure!( ensure!(
block_start + BLOCK_HEADER_SIZE <= block_end, block_start
.checked_add(BLOCK_HEADER_SIZE)
.is_some_and(|header_end| header_end <= block_end),
"block {} header truncated in {:08}.sst", "block {} header truncated in {:08}.sst",
block_index, block_index,
meta.sequence_number meta.sequence_number
); );
// Only hint ranges inside the mapping; without `strict_checks` a corrupt end offset isn't
// rejected before the slicing below panics.
if will_need && block_end <= mmap.len() {
will_need_range(mmap, block_start..block_end);
}
let uncompressed_length = be::read_u32(&mmap[block_start..]); let uncompressed_length = be::read_u32(&mmap[block_start..]);
let checksum = be::read_u32(&mmap[block_start + 4..]); let checksum = be::read_u32(&mmap[block_start + 4..]);
let block = &mmap[block_start + BLOCK_HEADER_SIZE..block_end]; let block = &mmap[block_start + BLOCK_HEADER_SIZE..block_end];
...@@ -961,8 +1044,10 @@ fn get_raw_block<'a>( ...@@ -961,8 +1044,10 @@ fn get_raw_block<'a>(
match backing { match backing {
#[cfg(feature = "mmap")] #[cfg(feature = "mmap")]
StaticSortedFileBacking::Mmap(mmap) => { StaticSortedFileBacking::Mmap(mmap) => {
// Every caller with an mmap backing reads the whole block: the index block once at
// open, and medium value blocks, which bypass the block cache, on every read.
let (uncompressed_length, checksum, block) = let (uncompressed_length, checksum, block) =
get_raw_block_slice(mmap, meta, block_index)?; get_raw_block_slice(mmap, meta, block_index, true)?;
Ok((uncompressed_length, checksum, Cow::Borrowed(block))) Ok((uncompressed_length, checksum, Cow::Borrowed(block)))
} }
StaticSortedFileBacking::File { StaticSortedFileBacking::File {
...@@ -1048,6 +1133,17 @@ fn verify_checksum( ...@@ -1048,6 +1133,17 @@ fn verify_checksum(
Ok(()) Ok(())
} }
/// Whether block `block_index`'s CRC has already been verified in this open. An out-of-range index
/// counts as unverified; reading the block reports the error.
#[cfg(feature = "mmap")]
fn is_block_verified(block_index: u16, verified_blocks: &[AtomicU64]) -> bool {
let word_idx = block_index as usize / u64::BITS as usize;
let bit = 1u64 << (block_index as usize % u64::BITS as usize);
verified_blocks
.get(word_idx)
.is_some_and(|word| word.load(AtomicOrdering::Relaxed) & bit != 0)
}
/// Verifies a block's CRC using the `verified_blocks` bitmap to avoid redundant /// Verifies a block's CRC using the `verified_blocks` bitmap to avoid redundant
/// work. In practice each block is verified once, but concurrent first-time /// work. In practice each block is verified once, but concurrent first-time
/// accesses may race and verify the same block more than once — this is harmless /// accesses may race and verify the same block more than once — this is harmless
...@@ -1109,8 +1205,9 @@ fn get_raw_block_iter( ...@@ -1109,8 +1205,9 @@ fn get_raw_block_iter(
match backing { match backing {
#[cfg(feature = "mmap")] #[cfg(feature = "mmap")]
StaticSortedFileIterBacking::Mmap(mmap) => { StaticSortedFileIterBacking::Mmap(mmap) => {
// Iteration mappings use `Sequential` advice, which already reads ahead.
let (uncompressed_length, checksum, block) = let (uncompressed_length, checksum, block) =
get_raw_block_slice(mmap, meta, block_index)?; get_raw_block_slice(mmap, meta, block_index, false)?;
// SAFETY: block points into mmap. // SAFETY: block points into mmap.
Ok((uncompressed_length, checksum, unsafe { Ok((uncompressed_length, checksum, unsafe {
RcBytes::from_mmap(mmap, block) RcBytes::from_mmap(mmap, block)
...@@ -1872,4 +1969,30 @@ mod tests { ...@@ -1872,4 +1969,30 @@ mod tests {
); );
assert_eq!(KeyBlockLayout::KeyOnly.hash_len(), 0); assert_eq!(KeyBlockLayout::KeyOnly.hash_len(), 0);
} }
/// Only ranges that cross a page boundary are hinted, including a small block that straddles
/// one, for both 4 KiB and 16 KiB pages.
#[cfg(all(feature = "mmap", unix))]
#[test]
fn spans_multiple_pages_detects_page_crossings() {
for page in [4096, 16384] {
assert!(!spans_multiple_pages(&(0..page), page), "one full page");
assert!(
!spans_multiple_pages(&(page..page + 100), page),
"inside a page"
);
assert!(spans_multiple_pages(&(0..page + 1), page), "one byte over");
assert!(
spans_multiple_pages(&(page - 8..page + 8), page),
"straddles a boundary"
);
assert!(
spans_multiple_pages(&(page / 2..3 * page), page),
"several pages"
);
}
// A 5 KiB block is one page at 16 KiB but two at 4 KiB.
assert!(spans_multiple_pages(&(0..5120), 4096));
assert!(!spans_multiple_pages(&(0..5120), 16384));
}
} }
...@@ -1804,6 +1804,139 @@ mod tests { ...@@ -1804,6 +1804,139 @@ mod tests {
Ok(()) Ok(())
} }
/// Deterministic, incompressible bytes, so that blocks keep their size on disk.
#[cfg(all(feature = "mmap", unix))]
fn noise(seed: u64, len: usize) -> Vec<u8> {
let mut state = seed.wrapping_mul(0x9E37_79B9_7F4A_7C15) | 1;
(0..len)
.map(|_| {
state ^= state << 13;
state ^= state >> 7;
state ^= state << 17;
state as u8
})
.collect()
}
/// Small values in multi-page blocks plus a few medium values.
#[cfg(all(feature = "mmap", unix))]
fn will_need_entries() -> Vec<TestEntry> {
let mut entries: Vec<TestEntry> = (0..300)
.map(|i| TestEntry::small(format!("key-{i:04}").as_bytes(), &noise(i, 300)))
.chain((0..3).map(|i| {
TestEntry::medium(format!("med-{i}").as_bytes(), &noise(1000 + i, 20_000))
}))
.collect();
sort_entries(&mut entries);
entries
}
/// Lookup mappings carry `MADV_RANDOM` (`rr` in `VmFlags`) and no `MADV_SEQUENTIAL` (`sr`).
#[cfg(all(feature = "mmap", target_os = "linux", not(miri)))]
#[test]
fn lookup_mmap_has_random_advice() -> Result<()> {
let dir = tempfile::tempdir()?;
let entries = will_need_entries();
let meta = write_sst(dir.path(), 1, &entries, MetaEntryFlags::default())?;
let sst = open_sst(dir.path(), 1, &meta)?;
let path = dir.path().join("00000001.sst").canonicalize()?;
let path = path.to_str().unwrap();
let smaps = std::fs::read_to_string("/proc/self/smaps")?;
let mut lines = smaps.lines();
lines
.find(|line| line.ends_with(path))
.expect("SST mapping not found in /proc/self/smaps");
let flags = lines
.find_map(|line| line.strip_prefix("VmFlags:"))
.expect("VmFlags line not found");
let flags: Vec<&str> = flags.split_whitespace().collect();
assert!(flags.contains(&"rr"), "expected MADV_RANDOM, got {flags:?}");
assert!(
!flags.contains(&"sr"),
"unexpected MADV_SEQUENTIAL, got {flags:?}"
);
drop(sst);
Ok(())
}
/// `MADV_WILLNEED` covers the block-offset table at open, and then only multi-page blocks:
/// key and small value blocks on their first read, medium value blocks on every read.
#[cfg(all(feature = "mmap", unix))]
#[test]
fn will_need_hints_multi_page_blocks_on_first_read() -> Result<()> {
use std::ops::Range;
use crate::static_sorted_file::will_need_log;
// SAFETY: `sysconf` has no preconditions.
let page_size = unsafe { libc::sysconf(libc::_SC_PAGESIZE) } as usize;
let multi_page =
|range: &Range<usize>| range.start / page_size != (range.end - 1) / page_size;
let dir = tempfile::tempdir()?;
let entries = will_need_entries();
let meta = write_sst(dir.path(), 1, &entries, MetaEntryFlags::default())?;
let file_len = std::fs::metadata(dir.path().join("00000001.sst"))?.len() as usize;
will_need_log::take();
// Open: the block-offset table, then the index block if it spans several pages.
let sst = open_sst(dir.path(), 1, &meta)?;
let opened = will_need_log::take();
let table_start = file_len - meta.block_count as usize * 4;
assert_eq!(opened.first(), Some(&(table_start..file_len)));
assert!(opened[1..].iter().all(multi_page), "{opened:?}");
// First pass: each multi-page key and small value block is hinted once.
let kc = make_cache();
let vc = make_cache();
let small = || {
entries
.iter()
.filter(|e| matches!(e.value_kind, TestValueKind::Small(_)))
};
for entry in small() {
assert_lookup(&sst, entry, &kc, &vc)?;
}
let first = will_need_log::take();
assert!(!first.is_empty(), "no multi-page block was hinted");
assert!(first.iter().all(multi_page), "{first:?}");
let mut unique = first.clone();
unique.sort_by_key(|range| range.start);
unique.dedup();
assert_eq!(
unique.len(),
first.len(),
"a block was hinted twice: {first:?}"
);
// Second pass, with empty caches: every block is verified already, so no hints.
let kc = make_cache();
let vc = make_cache();
for entry in small() {
assert_lookup(&sst, entry, &kc, &vc)?;
}
assert_eq!(will_need_log::take(), Vec::<Range<usize>>::new());
// Medium value blocks are hinted on every read.
let medium = entries
.iter()
.find(|e| matches!(e.value_kind, TestValueKind::Medium(_)))
.unwrap();
assert_lookup(&sst, medium, &kc, &vc)?;
let first_read = will_need_log::take();
assert_lookup(&sst, medium, &kc, &vc)?;
let second_read = will_need_log::take();
let value_block = second_read
.last()
.expect("medium value block was not hinted")
.clone();
assert!(multi_page(&value_block));
assert!(value_block.len() > 20_000, "{value_block:?}");
assert_eq!(first_read.last(), Some(&value_block));
Ok(())
}
#[test] #[test]
fn many_medium_values() -> Result<()> { fn many_medium_values() -> Result<()> {
let dir = tempfile::tempdir()?; let dir = tempfile::tempdir()?;
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment