Unverified Commit 33a5d542 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

Store keys in key order in SST blocks that omit hashes (#97480)

## What

Key blocks for short keys (≤ 32 bytes) don't store a per-entry hash, but
were still sorted by `(hash, key)`. That forced every binary-search
probe to recompute the entry's xxh3 hash just to compare it — roughly
ten hashes per lookup. These blocks now store their entries in **key
order**, so the search compares key bytes directly and hashes nothing.

All the families `turbo-tasks-backend` produces hit this case, so this
is the common path in practice.

## Why this is safe

Hash-based routing is unchanged. Files are still assigned by hash, the
index block still routes to a key block by hash, and
`min_hash`/`max_hash`, the AMQF, and compaction's coverage model all
work exactly as before.

Thus only the order *within* a hash-less block changes. This adds a
small cost to writing, which now must re-sort blocks of keys and a
double cost to compaction which now must re-sort blocks in hash order to
iterate them, and then sort them back to key order when writing the new
file.

## Benchmarks

Measured against `canary`, baseline and comparison run back to back on
the same machine.

**Lookups ** All 16 `static_sorted_file_lookup` configurations improved
| entries | hit/uncached | hit/cached | miss/uncached | miss/cached |
| --- | --- | --- | --- | --- |
| 1 Ki | -30.4% | -24.8% | -29.1% | -29.4% |
| 10 Ki | -25.2% | -19.7% | -22.7% | -23.7% |
| 100 Ki | -17.3% | -19.3% | -15.2% | -16.8% |
| 1000 Ki | -7.0% | -25.7% | -7.0% | -10.9% |


**Commits ** Short keys pay for the added per-block sort.

| config | change |
| --- | --- |
| `key_8` 85Ki entries | +3.3% |
| `key_8` 853Ki entries | +3.1% |
| `key_8` 8.33Mi entries | +8.5% |
| `key_32Ki` / `key_4` (large-key configs) | -2.6% to -19.1% |

Six of thirteen write configurations came out as noise (p >= 0.05) and
are omitted.

**Compaction** `StaticSortedFileIter` must yield `(hash, key)` order
because `MergeIter` merges on it, so it reorders each hash-less block
back into hash order. Plus the additional cost of the commit (above)

| config | change |
| --- | --- |
| 4Mi entries / 8 commits | +6.0% (reproduced at +10.2%) |
| 16Mi entries / 8 commits | +7.5% |
| 4Mi entries / 32 commits | +7.8% |
| 16Mi entries / 32 commits | +5.2% |
| 16Mi entries / 128 commits | -9.5% |
parent 920a7673
......@@ -122,12 +122,14 @@ The hashes are sorted.
A Key block contains n keys, which specify n key value pairs.
The block type determines whether the key hash is stored per entry:
The block type determines whether the key hash is stored per entry, and with it the order the
entries are stored in:
- Block type 1 (with hash): Full 8-byte hash stored per entry
- Block type 2 (no hash): No hash stored (for keys ≤ 32 bytes)
- Block type 1 (with hash): Full 8-byte hash stored per entry. Entries are sorted by
`(key hash, key)`.
- Block type 2 (no hash): No hash stored (for keys ≤ 32 bytes). Entries are sorted by **key**.
During lookup, if block type is 2, the full hash is recomputed from the key data.
See [Entry ordering](#entry-ordering) for why the two differ.
Depending on the `type` field entry has a different format:
......@@ -167,7 +169,13 @@ Depending on the `type` field entry has a different format:
Both ranged kinds are open-ended, so a decoder must test the key-value tombstone range **before**
the inline range.
The entries are sorted by key hash and key.
##### Entry ordering
Logically keys are ordered by hash (this is how we chose file and block assignments). However, within a single key block, however, the order is chosen per block type:
- **With hash (types 1 and 3):** sorted by `(key hash, key)`.
- **No hash (types 2 and 4):** sorted by **key** alone.
##### Key-value tombstones
......@@ -236,10 +244,11 @@ Reading start from the current sequence number and goes downwards.
- Check AMQF from SST file for key existence -> if not continue
- let block = 0
- loop
- Index Block: find key range that contains the key by binary search
- Index Block: find key range that contains the key by binary search using the **hash** of the key
- found -> set block, continue
- not found -> break
- Key Block: find key by binary search
- Key Block: find key by binary search, comparing `(hash, key)` in blocks that store a hash and
the key alone in blocks that do not (see [Entry ordering](#entry-ordering))
- found -> lookup value from value block, return
- read value as inline, or by using the block index in the key to find the value elsewhere in the file.
- not found -> break
......
......@@ -1161,8 +1161,8 @@ impl Entry for BenchEntry {
8
}
fn write_key_to(&self, buf: &mut Vec<u8>) {
buf.extend_from_slice(&self.key);
fn key_bytes(&self) -> &[u8] {
&self.key
}
fn value(&self) -> EntryValue<'_> {
......
......@@ -145,8 +145,8 @@ impl<K: StoreKey> Entry for CollectorEntry<K> {
self.key.data.len()
}
fn write_key_to(&self, buf: &mut Vec<u8>) {
self.key.data.write_to(buf);
fn key_bytes(&self) -> &[u8] {
self.key.data.as_slice()
}
fn value(&self) -> EntryValue<'_> {
......
......@@ -118,51 +118,69 @@ impl<T: KeyBase> KeyBase for &'_ T {
/// comparison with a byte slice (total order).
pub trait QueryKey: KeyBase {
fn cmp(&self, key: &[u8]) -> std::cmp::Ordering;
fn eq(&self, key: &[u8]) -> bool;
}
impl QueryKey for &'_ [u8] {
fn cmp(&self, key: &[u8]) -> std::cmp::Ordering {
Ord::cmp(self, &key)
}
fn eq(&self, key: &[u8]) -> bool {
PartialEq::eq(*self, key)
}
}
impl<const N: usize> QueryKey for [u8; N] {
fn cmp(&self, key: &[u8]) -> std::cmp::Ordering {
Ord::cmp(&self[..], key)
}
fn eq(&self, key: &[u8]) -> bool {
PartialEq::eq(self, key)
}
}
impl QueryKey for Vec<u8> {
fn cmp(&self, key: &[u8]) -> std::cmp::Ordering {
Ord::cmp(&**self, key)
}
fn eq(&self, key: &[u8]) -> bool {
PartialEq::eq(self.as_slice(), key)
}
}
impl QueryKey for Box<[u8]> {
fn cmp(&self, key: &[u8]) -> std::cmp::Ordering {
Ord::cmp(&**self, key)
}
fn eq(&self, key: &[u8]) -> bool {
PartialEq::eq(&**self, key)
}
}
impl QueryKey for u8 {
fn cmp(&self, key: &[u8]) -> std::cmp::Ordering {
Ord::cmp(&[*self][..], key)
}
fn eq(&self, key: &[u8]) -> bool {
PartialEq::eq(&[*self][..], key)
}
}
impl<A: QueryKey, B: QueryKey> QueryKey for (A, B) {
fn cmp(&self, mut key: &[u8]) -> std::cmp::Ordering {
fn cmp(&self, key: &[u8]) -> std::cmp::Ordering {
let (a, b) = self;
let len = a.len();
let key_len = key.len();
let key_part = &key[..min(key_len, len)];
match a.cmp(key_part) {
std::cmp::Ordering::Equal => {
key = &key[len..];
b.cmp(key)
}
ord => ord,
}
let (key_part, value_part) = key.split_at(min(key_len, len));
a.cmp(key_part).then_with(|| b.cmp(value_part))
}
fn eq(&self, key: &[u8]) -> bool {
let (a, b) = self;
let len = a.len();
let key_len = key.len();
let (key_part, value_part) = &key.split_at(min(key_len, len));
a.eq(key_part) && b.eq(value_part)
}
}
......@@ -170,53 +188,48 @@ impl<T: QueryKey> QueryKey for &'_ T {
fn cmp(&self, key: &[u8]) -> std::cmp::Ordering {
(*self).cmp(key)
}
fn eq(&self, key: &[u8]) -> bool {
(*self).eq(key)
}
}
/// A trait for keys that can be stored in the database. They need to allow hashing and comparison.
pub trait StoreKey: KeyBase + Ord {
fn write_to(&self, buf: &mut Vec<u8>);
/// The key's bytes.
fn as_slice(&self) -> &[u8];
fn write_to(&self, buf: &mut Vec<u8>) {
buf.extend_from_slice(self.as_slice());
}
}
impl<const N: usize> StoreKey for [u8; N] {
fn write_to(&self, buf: &mut Vec<u8>) {
buf.extend_from_slice(&self[..]);
fn as_slice(&self) -> &[u8] {
&self[..]
}
}
impl StoreKey for Vec<u8> {
fn write_to(&self, buf: &mut Vec<u8>) {
buf.extend_from_slice(self);
fn as_slice(&self) -> &[u8] {
self
}
}
impl StoreKey for Box<[u8]> {
fn write_to(&self, buf: &mut Vec<u8>) {
buf.extend_from_slice(self);
fn as_slice(&self) -> &[u8] {
self
}
}
impl StoreKey for &'_ [u8] {
fn write_to(&self, buf: &mut Vec<u8>) {
buf.extend_from_slice(self);
}
}
impl StoreKey for u8 {
fn write_to(&self, buf: &mut Vec<u8>) {
buf.push(*self);
}
}
impl<A: StoreKey, B: StoreKey> StoreKey for (A, B) {
fn write_to(&self, buf: &mut Vec<u8>) {
self.0.write_to(buf);
self.1.write_to(buf);
fn as_slice(&self) -> &[u8] {
self
}
}
impl<T: StoreKey> StoreKey for &'_ T {
fn write_to(&self, buf: &mut Vec<u8>) {
(*self).write_to(buf);
fn as_slice(&self) -> &[u8] {
(*self).as_slice()
}
}
......
......@@ -82,8 +82,8 @@ pub use key::{KeyBase, QueryKey, StoreKey, hash_key};
pub use meta_file::MetaEntryFlags;
pub use parallel_scheduler::{ParallelScheduler, SerialScheduler};
pub use static_sorted_file::{
BlockCache, BlockCacheLifecycle, BlockWeighter, SstLookupResult, StaticSortedFile,
StaticSortedFileMetaData,
BlockCache, BlockCacheLifecycle, BlockWeighter, KeyBlockLayout, SstLookupResult,
StaticSortedFile, StaticSortedFileMetaData,
};
pub use static_sorted_file_builder::{
BLOCK_HEADER_SIZE, Entry, EntryValue, StreamingSstWriter, write_static_stored_file,
......
......@@ -70,8 +70,8 @@ impl Entry for LookupEntry {
self.key.len()
}
fn write_key_to(&self, buf: &mut Vec<u8>) {
buf.extend_from_slice(&self.key);
fn key_bytes(&self) -> &[u8] {
&self.key
}
fn value(&self) -> EntryValue<'_> {
......
......@@ -106,6 +106,13 @@ impl ParallelScheduler for RayonParallelScheduler {
}
}
fn tuple_key(prefix: u8, suffix: [u8; 4]) -> Box<[u8]> {
let mut key = Vec::with_capacity(1 + suffix.len());
key.push(prefix);
key.extend_from_slice(&suffix);
key.into_boxed_slice()
}
#[test]
fn full_cycle() -> Result<()> {
let mut test_cases = Vec::new();
......@@ -514,13 +521,9 @@ fn persist_changes() -> Result<()> {
let path = tempdir.path();
const READ_COUNT: u32 = 2_000; // we'll read every 10th value, so writes are 10x this value
fn put(
b: &WriteBatch<(u8, [u8; 4]), RayonParallelScheduler, 1>,
key: u8,
value: u8,
) -> Result<()> {
fn put(b: &WriteBatch<Box<[u8]>, RayonParallelScheduler, 1>, key: u8, value: u8) -> Result<()> {
for i in 0..(READ_COUNT * 10) {
b.put(0, (key, i.to_be_bytes()), vec![value].into())?;
b.put(0, tuple_key(key, i.to_be_bytes()), vec![value].into())?;
}
Ok(())
}
......@@ -646,13 +649,9 @@ fn partial_compaction() -> Result<()> {
let path = tempdir.path();
const READ_COUNT: u32 = 2_000; // we'll read every 10th value, so writes are 10x this value
fn put(
b: &WriteBatch<(u8, [u8; 4]), RayonParallelScheduler, 1>,
key: u8,
value: u8,
) -> Result<()> {
fn put(b: &WriteBatch<Box<[u8]>, RayonParallelScheduler, 1>, key: u8, value: u8) -> Result<()> {
for i in 0..(READ_COUNT * 10) {
b.put(0, (key, i.to_be_bytes()), vec![value].into())?;
b.put(0, tuple_key(key, i.to_be_bytes()), vec![value].into())?;
}
Ok(())
}
......@@ -747,14 +746,14 @@ fn merge_file_removal() -> Result<()> {
const READ_COUNT: u32 = 2_000; // we'll read every 10th value, so writes are 10x this value
fn put(
b: &WriteBatch<(u8, [u8; 4]), RayonParallelScheduler, 1>,
b: &WriteBatch<Box<[u8]>, RayonParallelScheduler, 1>,
key: u8,
value: u32,
) -> Result<()> {
for i in 0..(READ_COUNT * 10) {
b.put(
0,
(key, i.to_be_bytes()),
tuple_key(key, i.to_be_bytes()),
value.to_be_bytes().to_vec().into(),
)?;
}
......
......@@ -272,8 +272,8 @@ impl KeyBase for WriteBuffer<'_> {
}
impl StoreKey for WriteBuffer<'_> {
fn write_to(&self, buf: &mut Vec<u8>) {
buf.extend_from_slice(self);
fn as_slice(&self) -> &[u8] {
self
}
}
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment