Unverified Commit 33a5d542 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

Store keys in key order in SST blocks that omit hashes (#97480)

## What

Key blocks for short keys (≤ 32 bytes) don't store a per-entry hash, but
were still sorted by `(hash, key)`. That forced every binary-search
probe to recompute the entry's xxh3 hash just to compare it — roughly
ten hashes per lookup. These blocks now store their entries in **key
order**, so the search compares key bytes directly and hashes nothing.

All the families `turbo-tasks-backend` produces hit this case, so this
is the common path in practice.

## Why this is safe

Hash-based routing is unchanged. Files are still assigned by hash, the
index block still routes to a key block by hash, and
`min_hash`/`max_hash`, the AMQF, and compaction's coverage model all
work exactly as before.

Thus only the order *within* a hash-less block changes. This adds a
small cost to writing, which now must re-sort blocks of keys and a
double cost to compaction which now must re-sort blocks in hash order to
iterate them, and then sort them back to key order when writing the new
file.

## Benchmarks

Measured against `canary`, baseline and comparison run back to back on
the same machine.

**Lookups ** All 16 `static_sorted_file_lookup` configurations improved
| entries | hit/uncached | hit/cached | miss/uncached | miss/cached |
| --- | --- | --- | --- | --- |
| 1 Ki | -30.4% | -24.8% | -29.1% | -29.4% |
| 10 Ki | -25.2% | -19.7% | -22.7% | -23.7% |
| 100 Ki | -17.3% | -19.3% | -15.2% | -16.8% |
| 1000 Ki | -7.0% | -25.7% | -7.0% | -10.9% |


**Commits ** Short keys pay for the added per-block sort.

| config | change |
| --- | --- |
| `key_8` 85Ki entries | +3.3% |
| `key_8` 853Ki entries | +3.1% |
| `key_8` 8.33Mi entries | +8.5% |
| `key_32Ki` / `key_4` (large-key configs) | -2.6% to -19.1% |

Six of thirteen write configurations came out as noise (p >= 0.05) and
are omitted.

**Compaction** `StaticSortedFileIter` must yield `(hash, key)` order
because `MergeIter` merges on it, so it reorders each hash-less block
back into hash order. Plus the additional cost of the commit (above)

| config | change |
| --- | --- |
| 4Mi entries / 8 commits | +6.0% (reproduced at +10.2%) |
| 16Mi entries / 8 commits | +7.5% |
| 4Mi entries / 32 commits | +7.8% |
| 16Mi entries / 32 commits | +5.2% |
| 16Mi entries / 128 commits | -9.5% |
parent 920a7673
...@@ -122,12 +122,14 @@ The hashes are sorted. ...@@ -122,12 +122,14 @@ The hashes are sorted.
A Key block contains n keys, which specify n key value pairs. A Key block contains n keys, which specify n key value pairs.
The block type determines whether the key hash is stored per entry: The block type determines whether the key hash is stored per entry, and with it the order the
entries are stored in:
- Block type 1 (with hash): Full 8-byte hash stored per entry - Block type 1 (with hash): Full 8-byte hash stored per entry. Entries are sorted by
- Block type 2 (no hash): No hash stored (for keys ≤ 32 bytes) `(key hash, key)`.
- Block type 2 (no hash): No hash stored (for keys ≤ 32 bytes). Entries are sorted by **key**.
During lookup, if block type is 2, the full hash is recomputed from the key data. See [Entry ordering](#entry-ordering) for why the two differ.
Depending on the `type` field entry has a different format: Depending on the `type` field entry has a different format:
...@@ -167,7 +169,13 @@ Depending on the `type` field entry has a different format: ...@@ -167,7 +169,13 @@ Depending on the `type` field entry has a different format:
Both ranged kinds are open-ended, so a decoder must test the key-value tombstone range **before** Both ranged kinds are open-ended, so a decoder must test the key-value tombstone range **before**
the inline range. the inline range.
The entries are sorted by key hash and key. ##### Entry ordering
Logically keys are ordered by hash (this is how we chose file and block assignments). However, within a single key block, however, the order is chosen per block type:
- **With hash (types 1 and 3):** sorted by `(key hash, key)`.
- **No hash (types 2 and 4):** sorted by **key** alone.
##### Key-value tombstones ##### Key-value tombstones
...@@ -236,10 +244,11 @@ Reading start from the current sequence number and goes downwards. ...@@ -236,10 +244,11 @@ Reading start from the current sequence number and goes downwards.
- Check AMQF from SST file for key existence -> if not continue - Check AMQF from SST file for key existence -> if not continue
- let block = 0 - let block = 0
- loop - loop
- Index Block: find key range that contains the key by binary search - Index Block: find key range that contains the key by binary search using the **hash** of the key
- found -> set block, continue - found -> set block, continue
- not found -> break - not found -> break
- Key Block: find key by binary search - Key Block: find key by binary search, comparing `(hash, key)` in blocks that store a hash and
the key alone in blocks that do not (see [Entry ordering](#entry-ordering))
- found -> lookup value from value block, return - found -> lookup value from value block, return
- read value as inline, or by using the block index in the key to find the value elsewhere in the file. - read value as inline, or by using the block index in the key to find the value elsewhere in the file.
- not found -> break - not found -> break
......
...@@ -1161,8 +1161,8 @@ impl Entry for BenchEntry { ...@@ -1161,8 +1161,8 @@ impl Entry for BenchEntry {
8 8
} }
fn write_key_to(&self, buf: &mut Vec<u8>) { fn key_bytes(&self) -> &[u8] {
buf.extend_from_slice(&self.key); &self.key
} }
fn value(&self) -> EntryValue<'_> { fn value(&self) -> EntryValue<'_> {
......
...@@ -145,8 +145,8 @@ impl<K: StoreKey> Entry for CollectorEntry<K> { ...@@ -145,8 +145,8 @@ impl<K: StoreKey> Entry for CollectorEntry<K> {
self.key.data.len() self.key.data.len()
} }
fn write_key_to(&self, buf: &mut Vec<u8>) { fn key_bytes(&self) -> &[u8] {
self.key.data.write_to(buf); self.key.data.as_slice()
} }
fn value(&self) -> EntryValue<'_> { fn value(&self) -> EntryValue<'_> {
......
...@@ -118,51 +118,69 @@ impl<T: KeyBase> KeyBase for &'_ T { ...@@ -118,51 +118,69 @@ impl<T: KeyBase> KeyBase for &'_ T {
/// comparison with a byte slice (total order). /// comparison with a byte slice (total order).
pub trait QueryKey: KeyBase { pub trait QueryKey: KeyBase {
fn cmp(&self, key: &[u8]) -> std::cmp::Ordering; fn cmp(&self, key: &[u8]) -> std::cmp::Ordering;
fn eq(&self, key: &[u8]) -> bool;
} }
impl QueryKey for &'_ [u8] { impl QueryKey for &'_ [u8] {
fn cmp(&self, key: &[u8]) -> std::cmp::Ordering { fn cmp(&self, key: &[u8]) -> std::cmp::Ordering {
Ord::cmp(self, &key) Ord::cmp(self, &key)
} }
fn eq(&self, key: &[u8]) -> bool {
PartialEq::eq(*self, key)
}
} }
impl<const N: usize> QueryKey for [u8; N] { impl<const N: usize> QueryKey for [u8; N] {
fn cmp(&self, key: &[u8]) -> std::cmp::Ordering { fn cmp(&self, key: &[u8]) -> std::cmp::Ordering {
Ord::cmp(&self[..], key) Ord::cmp(&self[..], key)
} }
fn eq(&self, key: &[u8]) -> bool {
PartialEq::eq(self, key)
}
} }
impl QueryKey for Vec<u8> { impl QueryKey for Vec<u8> {
fn cmp(&self, key: &[u8]) -> std::cmp::Ordering { fn cmp(&self, key: &[u8]) -> std::cmp::Ordering {
Ord::cmp(&**self, key) Ord::cmp(&**self, key)
} }
fn eq(&self, key: &[u8]) -> bool {
PartialEq::eq(self.as_slice(), key)
}
} }
impl QueryKey for Box<[u8]> { impl QueryKey for Box<[u8]> {
fn cmp(&self, key: &[u8]) -> std::cmp::Ordering { fn cmp(&self, key: &[u8]) -> std::cmp::Ordering {
Ord::cmp(&**self, key) Ord::cmp(&**self, key)
} }
fn eq(&self, key: &[u8]) -> bool {
PartialEq::eq(&**self, key)
}
} }
impl QueryKey for u8 { impl QueryKey for u8 {
fn cmp(&self, key: &[u8]) -> std::cmp::Ordering { fn cmp(&self, key: &[u8]) -> std::cmp::Ordering {
Ord::cmp(&[*self][..], key) Ord::cmp(&[*self][..], key)
} }
fn eq(&self, key: &[u8]) -> bool {
PartialEq::eq(&[*self][..], key)
}
} }
impl<A: QueryKey, B: QueryKey> QueryKey for (A, B) { impl<A: QueryKey, B: QueryKey> QueryKey for (A, B) {
fn cmp(&self, mut key: &[u8]) -> std::cmp::Ordering { fn cmp(&self, key: &[u8]) -> std::cmp::Ordering {
let (a, b) = self; let (a, b) = self;
let len = a.len(); let len = a.len();
let key_len = key.len(); let key_len = key.len();
let key_part = &key[..min(key_len, len)]; let (key_part, value_part) = key.split_at(min(key_len, len));
match a.cmp(key_part) { a.cmp(key_part).then_with(|| b.cmp(value_part))
std::cmp::Ordering::Equal => {
key = &key[len..];
b.cmp(key)
}
ord => ord,
} }
fn eq(&self, key: &[u8]) -> bool {
let (a, b) = self;
let len = a.len();
let key_len = key.len();
let (key_part, value_part) = &key.split_at(min(key_len, len));
a.eq(key_part) && b.eq(value_part)
} }
} }
...@@ -170,53 +188,48 @@ impl<T: QueryKey> QueryKey for &'_ T { ...@@ -170,53 +188,48 @@ impl<T: QueryKey> QueryKey for &'_ T {
fn cmp(&self, key: &[u8]) -> std::cmp::Ordering { fn cmp(&self, key: &[u8]) -> std::cmp::Ordering {
(*self).cmp(key) (*self).cmp(key)
} }
fn eq(&self, key: &[u8]) -> bool {
(*self).eq(key)
}
} }
/// A trait for keys that can be stored in the database. They need to allow hashing and comparison. /// A trait for keys that can be stored in the database. They need to allow hashing and comparison.
pub trait StoreKey: KeyBase + Ord { pub trait StoreKey: KeyBase + Ord {
fn write_to(&self, buf: &mut Vec<u8>); /// The key's bytes.
fn as_slice(&self) -> &[u8];
fn write_to(&self, buf: &mut Vec<u8>) {
buf.extend_from_slice(self.as_slice());
}
} }
impl<const N: usize> StoreKey for [u8; N] { impl<const N: usize> StoreKey for [u8; N] {
fn write_to(&self, buf: &mut Vec<u8>) { fn as_slice(&self) -> &[u8] {
buf.extend_from_slice(&self[..]); &self[..]
} }
} }
impl StoreKey for Vec<u8> { impl StoreKey for Vec<u8> {
fn write_to(&self, buf: &mut Vec<u8>) { fn as_slice(&self) -> &[u8] {
buf.extend_from_slice(self); self
} }
} }
impl StoreKey for Box<[u8]> { impl StoreKey for Box<[u8]> {
fn write_to(&self, buf: &mut Vec<u8>) { fn as_slice(&self) -> &[u8] {
buf.extend_from_slice(self); self
} }
} }
impl StoreKey for &'_ [u8] { impl StoreKey for &'_ [u8] {
fn write_to(&self, buf: &mut Vec<u8>) { fn as_slice(&self) -> &[u8] {
buf.extend_from_slice(self); self
}
}
impl StoreKey for u8 {
fn write_to(&self, buf: &mut Vec<u8>) {
buf.push(*self);
}
}
impl<A: StoreKey, B: StoreKey> StoreKey for (A, B) {
fn write_to(&self, buf: &mut Vec<u8>) {
self.0.write_to(buf);
self.1.write_to(buf);
} }
} }
impl<T: StoreKey> StoreKey for &'_ T { impl<T: StoreKey> StoreKey for &'_ T {
fn write_to(&self, buf: &mut Vec<u8>) { fn as_slice(&self) -> &[u8] {
(*self).write_to(buf); (*self).as_slice()
} }
} }
......
...@@ -82,8 +82,8 @@ pub use key::{KeyBase, QueryKey, StoreKey, hash_key}; ...@@ -82,8 +82,8 @@ pub use key::{KeyBase, QueryKey, StoreKey, hash_key};
pub use meta_file::MetaEntryFlags; pub use meta_file::MetaEntryFlags;
pub use parallel_scheduler::{ParallelScheduler, SerialScheduler}; pub use parallel_scheduler::{ParallelScheduler, SerialScheduler};
pub use static_sorted_file::{ pub use static_sorted_file::{
BlockCache, BlockCacheLifecycle, BlockWeighter, SstLookupResult, StaticSortedFile, BlockCache, BlockCacheLifecycle, BlockWeighter, KeyBlockLayout, SstLookupResult,
StaticSortedFileMetaData, StaticSortedFile, StaticSortedFileMetaData,
}; };
pub use static_sorted_file_builder::{ pub use static_sorted_file_builder::{
BLOCK_HEADER_SIZE, Entry, EntryValue, StreamingSstWriter, write_static_stored_file, BLOCK_HEADER_SIZE, Entry, EntryValue, StreamingSstWriter, write_static_stored_file,
......
...@@ -70,8 +70,8 @@ impl Entry for LookupEntry { ...@@ -70,8 +70,8 @@ impl Entry for LookupEntry {
self.key.len() self.key.len()
} }
fn write_key_to(&self, buf: &mut Vec<u8>) { fn key_bytes(&self) -> &[u8] {
buf.extend_from_slice(&self.key); &self.key
} }
fn value(&self) -> EntryValue<'_> { fn value(&self) -> EntryValue<'_> {
......
...@@ -106,6 +106,13 @@ impl ParallelScheduler for RayonParallelScheduler { ...@@ -106,6 +106,13 @@ impl ParallelScheduler for RayonParallelScheduler {
} }
} }
fn tuple_key(prefix: u8, suffix: [u8; 4]) -> Box<[u8]> {
let mut key = Vec::with_capacity(1 + suffix.len());
key.push(prefix);
key.extend_from_slice(&suffix);
key.into_boxed_slice()
}
#[test] #[test]
fn full_cycle() -> Result<()> { fn full_cycle() -> Result<()> {
let mut test_cases = Vec::new(); let mut test_cases = Vec::new();
...@@ -514,13 +521,9 @@ fn persist_changes() -> Result<()> { ...@@ -514,13 +521,9 @@ fn persist_changes() -> Result<()> {
let path = tempdir.path(); let path = tempdir.path();
const READ_COUNT: u32 = 2_000; // we'll read every 10th value, so writes are 10x this value const READ_COUNT: u32 = 2_000; // we'll read every 10th value, so writes are 10x this value
fn put( fn put(b: &WriteBatch<Box<[u8]>, RayonParallelScheduler, 1>, key: u8, value: u8) -> Result<()> {
b: &WriteBatch<(u8, [u8; 4]), RayonParallelScheduler, 1>,
key: u8,
value: u8,
) -> Result<()> {
for i in 0..(READ_COUNT * 10) { for i in 0..(READ_COUNT * 10) {
b.put(0, (key, i.to_be_bytes()), vec![value].into())?; b.put(0, tuple_key(key, i.to_be_bytes()), vec![value].into())?;
} }
Ok(()) Ok(())
} }
...@@ -646,13 +649,9 @@ fn partial_compaction() -> Result<()> { ...@@ -646,13 +649,9 @@ fn partial_compaction() -> Result<()> {
let path = tempdir.path(); let path = tempdir.path();
const READ_COUNT: u32 = 2_000; // we'll read every 10th value, so writes are 10x this value const READ_COUNT: u32 = 2_000; // we'll read every 10th value, so writes are 10x this value
fn put( fn put(b: &WriteBatch<Box<[u8]>, RayonParallelScheduler, 1>, key: u8, value: u8) -> Result<()> {
b: &WriteBatch<(u8, [u8; 4]), RayonParallelScheduler, 1>,
key: u8,
value: u8,
) -> Result<()> {
for i in 0..(READ_COUNT * 10) { for i in 0..(READ_COUNT * 10) {
b.put(0, (key, i.to_be_bytes()), vec![value].into())?; b.put(0, tuple_key(key, i.to_be_bytes()), vec![value].into())?;
} }
Ok(()) Ok(())
} }
...@@ -747,14 +746,14 @@ fn merge_file_removal() -> Result<()> { ...@@ -747,14 +746,14 @@ fn merge_file_removal() -> Result<()> {
const READ_COUNT: u32 = 2_000; // we'll read every 10th value, so writes are 10x this value const READ_COUNT: u32 = 2_000; // we'll read every 10th value, so writes are 10x this value
fn put( fn put(
b: &WriteBatch<(u8, [u8; 4]), RayonParallelScheduler, 1>, b: &WriteBatch<Box<[u8]>, RayonParallelScheduler, 1>,
key: u8, key: u8,
value: u32, value: u32,
) -> Result<()> { ) -> Result<()> {
for i in 0..(READ_COUNT * 10) { for i in 0..(READ_COUNT * 10) {
b.put( b.put(
0, 0,
(key, i.to_be_bytes()), tuple_key(key, i.to_be_bytes()),
value.to_be_bytes().to_vec().into(), value.to_be_bytes().to_vec().into(),
)?; )?;
} }
......
...@@ -272,8 +272,8 @@ impl KeyBase for WriteBuffer<'_> { ...@@ -272,8 +272,8 @@ impl KeyBase for WriteBuffer<'_> {
} }
impl StoreKey for WriteBuffer<'_> { impl StoreKey for WriteBuffer<'_> {
fn write_to(&self, buf: &mut Vec<u8>) { fn as_slice(&self) -> &[u8] {
buf.extend_from_slice(self); self
} }
} }
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment