Unverified Commit 875663cb authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

turbo-persistence: give key block searches a dense region of their own (#97705)

### What?

Rearranges the bytes inside a key block so that the bytes a lookup's
binary search compares are contiguous, instead of interleaved with key
and value payload the search never reads.


| block kind | before | after |
| --- | --- | --- |
| fixed, `HashThenKey` | `[hash\|key\|val]*n` | `[hash]*n` then
`[key\|val]*n` |
| fixed, `KeyOnly` | `[key\|val]*n` | `[key]*n` then `[val]*n` |
| variable, `HashThenKey` | `[type\|pos]*4B` then `[hash\|key\|val]…` |
`[hash\|type\|pos]*12B` then `[key\|val]…` |


### Why?

The binary search routines are a random access pattern through the
hash/key ranges, we only need to look at value payloads when we find a
plausible match. So by organizing all the 'random access data' in denser
memory range we optimize cache/page fault behavior.


`get` perf impacts are small but real.

| shape | in-block search before → after | as % of a ~1070 ns `get()` |
| --- | --- | --- |
| key=8, value=4 inline | 166 → 146 ns | 1.9% |
| key=4, small value ref | 144 → 119 ns | 2.4% |

The reason the impacts are small is mostly because the 'working sets' of
the benchmarks are not large enough and the page size of macos is quite
large.
parent bf13ab8a
...@@ -118,39 +118,40 @@ The hashes are sorted. ...@@ -118,39 +118,40 @@ The hashes are sorted.
- 1 byte block type (1: key block with hash, 2: key block without hash) - 1 byte block type (1: key block with hash, 2: key block without hash)
- 3 bytes entry count - 3 bytes entry count
- foreach entry - offset table: foreach entry
- 8 bytes key hash (block type 1 only)
- 1 byte type - 1 byte type
- 3 bytes position in block after header - 3 bytes position in block after header
- Max block size: 16 KB - Max block size: 16 KB
A Key block contains n keys, which specify n key value pairs. A Key block contains n keys, which specify n key value pairs.
The block type determines whether the key hash is stored per entry, and with it the order the The block type determines whether the key hash is stored, and with it the order the entries are
entries are stored in: stored in:
- Block type 1 (with hash): Full 8-byte hash stored per entry. Entries are sorted by - Block type 1 (with hash): Full 8-byte hash per entry, stored in the offset table. Entries are
`(key hash, key)`. sorted by `(key hash, key)`.
- Block type 2 (no hash): No hash stored (for keys ≤ 32 bytes). Entries are sorted by **key**. - Block type 2 (no hash): No hash stored (for keys ≤ 32 bytes). Entries are sorted by **key**.
See [Entry ordering](#entry-ordering) for why the two differ. See [Entry ordering](#entry-ordering) for why the two differ.
The hash lives in the offset table rather than beside its key so that a lookup's binary search reads
only that dense array. Fixed-size key blocks apply the same idea; see
[Two regions, not interleaved](#two-regions-not-interleaved).
Depending on the `type` field entry has a different format: Depending on the `type` field entry has a different format:
- 0: normal key (small value) - 0: normal key (small value)
- 8 bytes key hash (if block type 1)
- key data - key data
- 2 byte block index - 2 byte block index
- 2 bytes size - 2 bytes size
- 4 bytes position in block - 4 bytes position in block
- 1: blob reference - 1: blob reference
- 8 bytes key hash (if block type 1)
- key data - key data
- 4 bytes sequence number - 4 bytes sequence number
- 2: deleted key / key tombstone (no data) - 2: deleted key / key tombstone (no data)
- 8 bytes key hash (if block type 1)
- key data - key data
- 3: normal key (medium sized value) - 3: normal key (medium sized value)
- 8 bytes key hash (if block type 1)
- key data - key data
- 2 byte block index - 2 byte block index
- 7: merge key (future) - 7: merge key (future)
...@@ -160,12 +161,10 @@ Depending on the `type` field entry has a different format: ...@@ -160,12 +161,10 @@ Depending on the `type` field entry has a different format:
- 4 bytes position in block - 4 bytes position in block
- 8..=16: inlined value, size = type - 8 (the format supports up to 247, but `MAX_INLINE_VALUE_SIZE` - 8..=16: inlined value, size = type - 8 (the format supports up to 247, but `MAX_INLINE_VALUE_SIZE`
currently caps it at 8) currently caps it at 8)
- 8 bytes key hash (if block type 1)
- key data - key data
- (type - 8) bytes value data (inline, no separate value block) - (type - 8) bytes value data (inline, no separate value block)
- 17..=25: key-value tombstone, deleted value size = type - 17 (mirrors the inline range and shifts - 17..=25: key-value tombstone, deleted value size = type - 17 (mirrors the inline range and shifts
with `MAX_INLINE_VALUE_SIZE`) with `MAX_INLINE_VALUE_SIZE`)
- 8 bytes key hash (if block type 1)
- key data - key data
- (type - 17) bytes of the deleted value, stored inline - (type - 17) bytes of the deleted value, stored inline
...@@ -174,7 +173,7 @@ the inline range. ...@@ -174,7 +173,7 @@ the inline range.
##### Entry ordering ##### Entry ordering
Logically keys are ordered by hash (this is how we chose file and block assignments). However, within a single key block, however, the order is chosen per block type: Logically keys are ordered by hash (this is how we chose file and block assignments). However, within a key block, the order may be different
- **With hash (types 1 and 3):** sorted by `(key hash, key)`. - **With hash (types 1 and 3):** sorted by `(key hash, key)`.
- **No hash (types 2 and 4):** sorted by **key** alone. - **No hash (types 2 and 4):** sorted by **key** alone.
...@@ -211,9 +210,10 @@ during binary search. ...@@ -211,9 +210,10 @@ during binary search.
- 1 byte value type (shared by all entries, same encoding as variable-size type field), or - 1 byte value type (shared by all entries, same encoding as variable-size type field), or
`FIXED_KEY_BLOCK_MIXED_VALUE_TYPE` (4) when entries share a value size but not a value type `FIXED_KEY_BLOCK_MIXED_VALUE_TYPE` (4) when entries share a value size but not a value type
- 1 byte value size — only present when the value type is `FIXED_KEY_BLOCK_MIXED_VALUE_TYPE` - 1 byte value size — only present when the value type is `FIXED_KEY_BLOCK_MIXED_VALUE_TYPE`
- foreach entry (packed at stride = hash_len + key_size + val_size): - search region, foreach entry at stride `search_stride`:
- 8 bytes key hash (if block type 3) - 8 bytes key hash (block type 3), or key data (block type 4, `key_size` bytes)
- key data (key_size bytes) - tail region, foreach entry at stride `tail_stride`:
- key data (block type 3 only, `key_size` bytes)
- 1 byte value type — only present when the block is mixed-type - 1 byte value type — only present when the block is mixed-type
- value data (size determined by the block's or the entry's value type) - value data (size determined by the block's or the entry's value type)
...@@ -221,7 +221,21 @@ The mixed-type form exists so that same-sized inline values and key-value tombst ...@@ -221,7 +221,21 @@ The mixed-type form exists so that same-sized inline values and key-value tombst
fixed-size block: they have equal value sizes but different type bytes. Tag 4 is available as the fixed-size block: they have equal value sizes but different type bytes. Tag 4 is available as the
mixed marker because it is not itself a valid entry type. mixed marker because it is not itself a valid entry type.
Entry position for index `i` is computed as `header_size + i * stride` with no indirection. The writer automatically selects fixed-size format when all entries in a block qualify; otherwise falls back to the variable-size format above. ##### Two regions, not interleaved
Rather than one interleaved record per entry, entries are split into a **search region** and a
**tail region**, indexed by the same entry number. The search region holds only the bytes binary
search compares first — the hash for block type `3`, the key for block type `4` (see
[Entry ordering](#entry-ordering)) — so a probe searches the dense prefix region. The _values_ are then found by index in the **tail**
after the search succeeds.
Entry positions for index `i` are `header_size + i * search_stride` and
`header_size + entry_count * search_stride + i * tail_stride`, both with no indirection.
The search region must be in ascending order for that binary search to be valid. This is inherited
from the `(key hash, key)` order the writer requires of its input, not established per block, so
reordering entries within a block is a format violation rather than a free choice — see
[Entry ordering](#entry-ordering).
#### Value Block #### Value Block
......
...@@ -1201,8 +1201,8 @@ fn bench_static_sorted_file_lookup(c: &mut Criterion) { ...@@ -1201,8 +1201,8 @@ fn bench_static_sorted_file_lookup(c: &mut Criterion) {
}) })
.collect(); .collect();
// Sort by hash (required by write_static_stored_file) // Sort by (hash, key) order, as required by write_static_stored_file
entries.sort_by_key(|e| e.hash); entries.sort_by_key(|e| (e.hash, e.key));
// Create temp directory and write SST file // Create temp directory and write SST file
let tempdir = tempfile::tempdir().unwrap(); let tempdir = tempfile::tempdir().unwrap();
......
...@@ -26,11 +26,11 @@ use turbo_persistence::{ ...@@ -26,11 +26,11 @@ use turbo_persistence::{
read_current_version, read_current_version,
sst_filter::SstFilter, sst_filter::SstFilter,
static_sorted_file::{ static_sorted_file::{
BLOCK_TYPE_FIXED_KEY_NO_HASH, BLOCK_TYPE_FIXED_KEY_WITH_HASH, BLOCK_TYPE_KEY_NO_HASH, FIXED_KEY_BLOCK_MIXED_VALUE_TYPE, FixedRegions, KEY_BLOCK_ENTRY_TYPE_BLOB,
BLOCK_TYPE_KEY_WITH_HASH, FIXED_KEY_BLOCK_MIXED_VALUE_TYPE, KEY_BLOCK_ENTRY_TYPE_BLOB,
KEY_BLOCK_ENTRY_TYPE_INLINE_MIN, KEY_BLOCK_ENTRY_TYPE_KEY_DELETED, KEY_BLOCK_ENTRY_TYPE_INLINE_MIN, KEY_BLOCK_ENTRY_TYPE_KEY_DELETED,
KEY_BLOCK_ENTRY_TYPE_KEY_VALUE_DELETED_MIN, KEY_BLOCK_ENTRY_TYPE_MEDIUM, KEY_BLOCK_ENTRY_TYPE_KEY_VALUE_DELETED_MIN, KEY_BLOCK_ENTRY_TYPE_MEDIUM,
KEY_BLOCK_ENTRY_TYPE_SMALL, KEY_BLOCK_ENTRY_TYPE_SMALL, KEY_BLOCK_TABLE_ENTRY_SIZE_NO_HASH, KeyBlockLayout,
key_block_table_stride,
}, },
}; };
...@@ -394,18 +394,19 @@ fn parse_key_block_indices(index_block: &[u8]) -> HashSet<u16> { ...@@ -394,18 +394,19 @@ fn parse_key_block_indices(index_block: &[u8]) -> HashSet<u16> {
enum KeyBlockHeader { enum KeyBlockHeader {
Variable { Variable {
entry_count: u32, entry_count: u32,
/// Bytes per offset table entry, wider when the block hoists hashes into the table.
table_stride: usize,
}, },
Fixed { Fixed {
entry_count: u32, entry_count: u32,
value_type: u8, value_type: u8,
}, },
/// Fixed-size layout whose entries share a value size but not a value type, so each carries /// Fixed-size layout whose entries share a value size but not a value type, so each carries
/// its own type byte between its key and its value. /// its own type byte ahead of its value in the block's tail region.
FixedMixedType { FixedMixedType {
entry_count: u32, entry_count: u32,
hash_len: usize, /// Where the block's search and tail regions sit, derived by the shared reader helper.
key_size: usize, regions: FixedRegions,
stride: usize,
}, },
} }
...@@ -414,36 +415,34 @@ fn parse_key_block_header(block: &[u8]) -> Result<KeyBlockHeader> { ...@@ -414,36 +415,34 @@ fn parse_key_block_header(block: &[u8]) -> Result<KeyBlockHeader> {
assert!(block.len() >= 4, "Key block too small"); assert!(block.len() >= 4, "Key block too small");
let block_type = block[0]; let block_type = block[0];
let entry_count = ((block[1] as u32) << 16) | ((block[2] as u32) << 8) | (block[3] as u32); let entry_count = ((block[1] as u32) << 16) | ((block[2] as u32) << 8) | (block[3] as u32);
match block_type { let Some((layout, fixed)) = KeyBlockLayout::from_block_type(block_type) else {
BLOCK_TYPE_KEY_WITH_HASH | BLOCK_TYPE_KEY_NO_HASH => { bail!("Invalid key block type: {block_type}");
Ok(KeyBlockHeader::Variable { entry_count }) };
} if !fixed {
BLOCK_TYPE_FIXED_KEY_WITH_HASH | BLOCK_TYPE_FIXED_KEY_NO_HASH => { return Ok(KeyBlockHeader::Variable {
assert!(block.len() >= 6, "Fixed key block header too small"); entry_count,
if block[5] == FIXED_KEY_BLOCK_MIXED_VALUE_TYPE { table_stride: key_block_table_stride(layout.hash_len()),
assert!(block.len() >= 7, "Mixed-type key block header too small"); });
let hash_len = if block_type == BLOCK_TYPE_FIXED_KEY_WITH_HASH { }
8 assert!(block.len() >= 6, "Fixed key block header too small");
} else { if block[5] == FIXED_KEY_BLOCK_MIXED_VALUE_TYPE {
0 assert!(block.len() >= 7, "Mixed-type key block header too small");
}; Ok(KeyBlockHeader::FixedMixedType {
let key_size = block[4] as usize; entry_count,
let val_size = block[6] as usize; // `FixedRegions` owns the search/tail split; `val_size` includes the per-entry type
Ok(KeyBlockHeader::FixedMixedType { // byte, which the header stores separately from the value size.
entry_count, regions: FixedRegions::new(
hash_len, entry_count as usize,
key_size, layout,
// +1 for the per-entry type byte. block[4] as usize,
stride: hash_len + key_size + val_size + 1, block[6] as usize + 1,
}) ),
} else { })
Ok(KeyBlockHeader::Fixed { } else {
entry_count, Ok(KeyBlockHeader::Fixed {
value_type: block[5], entry_count,
}) value_type: block[5],
} })
}
_ => bail!("Invalid key block type: {block_type}"),
} }
} }
...@@ -457,24 +456,23 @@ fn iter_key_block_entry_types( ...@@ -457,24 +456,23 @@ fn iter_key_block_entry_types(
block: &[u8], block: &[u8],
) -> impl Iterator<Item = u8> + '_ { ) -> impl Iterator<Item = u8> + '_ {
let entry_count = match header { let entry_count = match header {
KeyBlockHeader::Variable { entry_count } KeyBlockHeader::Variable { entry_count, .. }
| KeyBlockHeader::Fixed { entry_count, .. } | KeyBlockHeader::Fixed { entry_count, .. }
| KeyBlockHeader::FixedMixedType { entry_count, .. } => entry_count, | KeyBlockHeader::FixedMixedType { entry_count, .. } => entry_count,
}; };
(0..entry_count).map(move |i| match header { (0..entry_count).map(move |i| match header {
// Variable block: offset table starts at byte 4 (after 1B type + 3B count), // Variable block: offset table starts at byte 4 (after 1B type + 3B count). The type byte
// each entry is 4 bytes, first byte is the entry type. // leads the trailing type/position word, which follows any hoisted hash.
KeyBlockHeader::Variable { .. } => block[KEY_BLOCK_HEADER_SIZE + i as usize * 4], KeyBlockHeader::Variable { table_stride, .. } => {
block[KEY_BLOCK_HEADER_SIZE
+ i as usize * table_stride
+ (table_stride - KEY_BLOCK_TABLE_ENTRY_SIZE_NO_HASH)]
}
KeyBlockHeader::Fixed { value_type, .. } => value_type, KeyBlockHeader::Fixed { value_type, .. } => value_type,
KeyBlockHeader::FixedMixedType { KeyBlockHeader::FixedMixedType { regions, .. } => {
hash_len, // Entry data starts after the 7-byte mixed-type header; within the tail region the
key_size, // type byte precedes the value, after the key for `HashThenKey` blocks.
stride, block[7 + regions.total_len(i as usize) + regions.tail_key_size()]
..
} => {
// Entry data starts after the 7-byte mixed-type header; the type byte sits between
// the entry's key and its value.
block[7 + i as usize * stride + hash_len + key_size]
} }
}) })
} }
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment