Unverified Commit 34fdcbb4 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

Avoid using TaskTypes as keys in storage instead use hashes (#88904)

Save space in the persisent store by only recording TaskType one time instead of twice.

## What?
Instead of using an encoded TaskType struct as a key in persistent storage just use a 64 bit hash.   Now when looking up we actually need to do a cascading read

* read task ids that match the hash
* restore all the tasks until we find one that matches our CachedTaskType

Full cache misses perform the same and so do cache hits since we always end up reading the TaskData anyway (in the `connect_child` operation that immediately follows).   This just slightly changes _when_ that second read happens, as such we shouldn't really expect it to slow down.

In the case of a hash collisions we do end up doing strictly more work (reading and restoring TaskData entries for the wrong task), but this work is cached and this should be _extremely_ rare assuming a good hash function..

From measuring vercel-site this saves ~231M of data in the persistent cache. (The cache goes from from 3846M ->  3615 or -231M or about -6%).

Finally this also addresses a narrow race condition where two racing calls to `get_persistent_task_id` for the same task could result in two entries being pusshed to the `persistent_task_log`, that is now addressed as well.

## Why?

Currently we encode 2 copies of every `CachedTaskType` in the database. 
1. as the key of the `TaskType`->`TaskId` map (aka `TaskCache` keyspace)
2. as a part of the `TaskStorage` struct stored in the `TaskData` keyspace

This redundancy is wasteful.  Instead we can make the `TaskCache` map much smaller and add a bit of complexity to lookups.



## Future Work

### Better hash functions
Right now to compute the hashes we are just running `encode` and then hashing the bytes.  This is not optimal, but we do not have a hash function that is suitable for this usecase.  So we should create a new `PersistentHash` trait that TaskInputs implement in order to support this without encoding. Or perhaps a custom _encoder_ that accumulates encoding data to a hasher.  This will be addressed in https://github.com/vercel/next.js/pull/89059

### New SST ffile heuristics

 now that the TaskcCache keyspace is smaller our heuristics on 'maxiumum number of keys in a file' need to be rethought since we are now producing lots of 7Mb files for the taskcache.  this will be addressed in https://github.com/vercel/next.js/pull/89058

### New compaction semantics

Right now we tolerate duplicates in the database but compaction will delete them.  This is not too harmful for us since it means if there is a hash coliision we will tend to lose one of the results over time.

A better option would be to change compaction semantics for this KeySpace to either tolerate duplicates, leverage values for comparison, or something wilder where we simply 'recompute' the TaskCache instead of compacting it.   This will be addressed in https://github.com/vercel/next.js/pull/89075
parent 1f45f98a
......@@ -284,21 +284,6 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
ExecuteContextImpl::new(self, turbo_tasks)
}
/// # Safety
///
/// `tx` must be a transaction from this TurboTasksBackendInner instance.
unsafe fn execute_context_with_tx<'e, 'tx>(
&'e self,
tx: Option<&'e B::ReadTransaction<'tx>>,
turbo_tasks: &'e dyn TurboTasksBackendApi<TurboTasksBackend<B>>,
) -> impl ExecuteContext<'e> + use<'e, 'tx, B>
where
'tx: 'e,
{
// Safety: `tx` is from `self`.
unsafe { ExecuteContextImpl::new_with_tx(self, tx, turbo_tasks) }
}
fn suspending_requested(&self) -> bool {
self.should_persist()
&& (self.in_progress_operations.load(Ordering::Relaxed) & SNAPSHOT_REQUESTED_BIT) != 0
......@@ -483,22 +468,6 @@ struct TaskExecutionCompletePrepareResult {
// Operations
impl<B: BackingStorage> TurboTasksBackendInner<B> {
/// # Safety
///
/// `tx` must be a transaction from this TurboTasksBackendInner instance.
unsafe fn connect_child_with_tx<'l, 'tx: 'l>(
&'l self,
tx: Option<&'l B::ReadTransaction<'tx>>,
parent_task: Option<TaskId>,
child_task: TaskId,
task_type: Option<ArcOrOwned<CachedTaskType>>,
turbo_tasks: &'l dyn TurboTasksBackendApi<TurboTasksBackend<B>>,
) {
operation::ConnectChildOperation::run(parent_task, child_task, task_type, unsafe {
self.execute_context_with_tx(tx, turbo_tasks)
});
}
fn connect_child(
&self,
parent_task: Option<TaskId>,
......@@ -1500,65 +1469,52 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
return task_id;
}
let check_backing_storage =
self.should_restore() && self.local_is_partial.load(Ordering::Acquire);
let tx = check_backing_storage
.then(|| self.backing_storage.start_read_transaction())
.flatten();
// Create a single ExecuteContext for both lookup and connect_child
let mut ctx = self.execute_context(turbo_tasks);
let mut is_new = false;
let (task_id, task_type) = {
// Safety: `tx` is a valid transaction from `self.backend.backing_storage`.
if let Some(task_id) = unsafe {
check_backing_storage
.then(|| {
self.backing_storage
.forward_lookup_task_cache(tx.as_ref(), &task_type)
.expect("Failed to lookup task id")
})
.flatten()
} {
// Task exists in backing storage
// So we only need to insert it into the in-memory cache
self.track_cache_hit(&task_type);
let task_type = match raw_entry(&self.task_cache, &task_type) {
RawEntry::Occupied(_) => ArcOrOwned::Owned(task_type),
RawEntry::Vacant(e) => {
let task_type = Arc::new(task_type);
e.insert(task_type.clone(), task_id);
ArcOrOwned::Arc(task_type)
}
};
(task_id, task_type)
} else {
// Task doesn't exist in memory cache or backing storage
// So we might need to create a new task
let (task_id, mut task_type) = match raw_entry(&self.task_cache, &task_type) {
RawEntry::Occupied(e) => {
let task_id = *e.get();
drop(e);
self.track_cache_hit(&task_type);
(task_id, ArcOrOwned::Owned(task_type))
}
RawEntry::Vacant(e) => {
let task_type = Arc::new(task_type);
let task_id = self.persisted_task_id_factory.get();
e.insert(task_type.clone(), task_id);
self.track_cache_miss(&task_type);
is_new = true;
(task_id, ArcOrOwned::Arc(task_type))
let (task_id, task_type) = if let Some(task_id) = ctx.task_by_type(&task_type) {
// Task exists in backing storage
// So we only need to insert it into the in-memory cache
self.track_cache_hit(&task_type);
let task_type = match raw_entry(&self.task_cache, &task_type) {
RawEntry::Occupied(_) => ArcOrOwned::Owned(task_type),
RawEntry::Vacant(e) => {
let task_type = Arc::new(task_type);
e.insert(task_type.clone(), task_id);
ArcOrOwned::Arc(task_type)
}
};
(task_id, task_type)
} else {
// Task doesn't exist in memory cache or backing storage
// So we might need to create a new task
let (task_id, task_type) = match raw_entry(&self.task_cache, &task_type) {
RawEntry::Occupied(e) => {
// Another thread beat us to creating this task - use their task_id.
// They will handle logging to persisted_task_cache_log.
let task_id = *e.get();
drop(e);
self.track_cache_hit(&task_type);
(task_id, ArcOrOwned::Owned(task_type))
}
RawEntry::Vacant(e) => {
// We're creating a new task.
let task_type = Arc::new(task_type);
let task_id = self.persisted_task_id_factory.get();
e.insert(task_type.clone(), task_id);
// insert() consumes e, releasing the lock
self.track_cache_miss(&task_type);
is_new = true;
if let Some(log) = &self.persisted_task_cache_log {
log.lock(task_id).push((task_type.clone(), task_id));
}
};
if let Some(log) = &self.persisted_task_cache_log {
let task_type_arc: Arc<CachedTaskType> = Arc::from(task_type);
log.lock(task_id).push((task_type_arc.clone(), task_id));
task_type = ArcOrOwned::Arc(task_type_arc);
(task_id, ArcOrOwned::Arc(task_type))
}
(task_id, task_type)
}
};
(task_id, task_type)
};
if is_new && is_root {
let mut ctx = self.execute_context(turbo_tasks);
AggregationUpdateQueue::run(
AggregationUpdateJob::UpdateAggregationNumber {
task_id,
......@@ -1568,17 +1524,8 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
&mut ctx,
);
}
// Safety: `tx` is a valid transaction from `self.backend.backing_storage`.
unsafe {
self.connect_child_with_tx(
tx.as_ref(),
parent_task,
task_id,
Some(task_type),
turbo_tasks,
)
};
// Reuse the same ExecuteContext for connect_child
operation::ConnectChildOperation::run(parent_task, task_id, Some(task_type), ctx);
task_id
}
......
......@@ -34,10 +34,8 @@ pub trait Operation: Encode + Decode<()> + Default + TryFrom<AnyOperation, Error
fn execute(self, ctx: &mut impl ExecuteContext<'_>);
}
#[derive(Copy, Clone)]
enum TransactionState<'a, 'tx, B: BackingStorage> {
enum TransactionState<'tx, B: BackingStorage> {
None,
Borrowed(Option<&'a B::ReadTransaction<'tx>>),
Owned(Option<B::ReadTransaction<'tx>>),
}
......@@ -94,6 +92,12 @@ pub trait ExecuteContext<'e>: Sized {
fn should_track_dependencies(&self) -> bool;
fn should_track_activeness(&self) -> bool;
fn turbo_tasks(&self) -> Arc<dyn TurboTasksCallApi>;
/// Look up a TaskId from the backing storage for a given task type.
///
/// Uses hash-based lookup which may return multiple candidates due to hash collisions,
/// then verifies each candidate by comparing the stored `persistent_task_type`.
/// Returns `Some(task_id)` if a matching task is found, `None` otherwise.
fn task_by_type(&mut self, task_type: &CachedTaskType) -> Option<TaskId>;
}
pub trait ChildExecuteContext<'e>: Send + Sized {
......@@ -108,7 +112,7 @@ where
backend: &'e TurboTasksBackendInner<B>,
turbo_tasks: &'e dyn TurboTasksBackendApi<TurboTasksBackend<B>>,
_operation_guard: Option<OperationGuard<'e, B>>,
transaction: TransactionState<'e, 'tx, B>,
transaction: TransactionState<'tx, B>,
#[cfg(debug_assertions)]
active_task_locks: std::sync::Arc<std::sync::atomic::AtomicU8>,
}
......@@ -131,21 +135,6 @@ where
}
}
pub(super) unsafe fn new_with_tx(
backend: &'e TurboTasksBackendInner<B>,
transaction: Option<&'e B::ReadTransaction<'tx>>,
turbo_tasks: &'e dyn TurboTasksBackendApi<TurboTasksBackend<B>>,
) -> Self {
Self {
backend,
turbo_tasks,
_operation_guard: Some(backend.start_operation()),
transaction: TransactionState::Borrowed(transaction),
#[cfg(debug_assertions)]
active_task_locks: std::sync::Arc::new(std::sync::atomic::AtomicU8::new(0)),
}
}
fn ensure_transaction(&mut self) -> bool {
if matches!(self.transaction, TransactionState::None) {
let check_backing_storage = self.backend.should_restore()
......@@ -229,7 +218,6 @@ where
fn get_tx(&self) -> Option<&<B as BackingStorageSealed>::ReadTransaction<'tx>> {
match &self.transaction {
TransactionState::None => unreachable!(),
TransactionState::Borrowed(tx) => *tx,
TransactionState::Owned(tx) => tx.as_ref(),
}
}
......@@ -642,6 +630,35 @@ where
fn turbo_tasks(&self) -> Arc<dyn TurboTasksCallApi> {
self.turbo_tasks.pin()
}
fn task_by_type(&mut self, task_type: &CachedTaskType) -> Option<TaskId> {
// Ensure we have a transaction (this will be reused by subsequent task() calls)
if !self.ensure_transaction() {
return None;
}
let tx = self.get_tx();
// Get candidates from backing storage (hash-based lookup may return multiple)
// Safety: `tx` is a valid transaction from `self.backend.backing_storage`.
let candidates = unsafe {
self.backend
.backing_storage
.lookup_task_candidates(tx, task_type)
.expect("Failed to lookup task ids")
};
// Verify each candidate by comparing the stored persistent_task_type.
// Only rarely is there more than one candidate, so no need for parallelization.
for candidate_id in candidates {
let task = self.task(candidate_id, TaskDataCategory::Data);
if let Some(stored_type) = task.get_persistent_task_type()
&& stored_type.as_ref() == task_type
{
return Some(candidate_id);
}
}
None
}
}
struct ChildExecuteContextImpl<'e, B: BackingStorage> {
......
......@@ -60,14 +60,20 @@ pub trait BackingStorageSealed: 'static + Send + Sync {
> + Send
+ Sync;
fn start_read_transaction(&self) -> Option<Self::ReadTransaction<'_>>;
/// Returns all task IDs that match the given task type (hash collision candidates).
///
/// Since TaskCache uses hash-based keys, multiple task types may (rarely) hash to the same key.
/// The caller must verify each returned TaskId by comparing the stored task type which will
/// require a second database read
///
/// # Safety
///
/// `tx` must be a transaction from this BackingStorage instance.
unsafe fn forward_lookup_task_cache(
unsafe fn lookup_task_candidates(
&self,
tx: Option<&Self::ReadTransaction<'_>>,
key: &CachedTaskType,
) -> Result<Option<TaskId>>;
) -> Result<SmallVec<[TaskId; 1]>>;
/// # Safety
///
/// `tx` must be a transaction from this BackingStorage instance.
......@@ -151,19 +157,19 @@ where
})
}
unsafe fn forward_lookup_task_cache(
unsafe fn lookup_task_candidates(
&self,
tx: Option<&Self::ReadTransaction<'_>>,
key: &CachedTaskType,
) -> Result<Option<TaskId>> {
) -> Result<SmallVec<[TaskId; 1]>> {
match self {
Either::Left(this) => {
let tx = tx.map(|tx| read_transaction_left_or_panic(tx.as_ref()));
unsafe { this.forward_lookup_task_cache(tx, key) }
unsafe { this.lookup_task_candidates(tx, key) }
}
Either::Right(this) => {
let tx = tx.map(|tx| read_transaction_right_or_panic(tx.as_ref()));
unsafe { this.forward_lookup_task_cache(tx, key) }
unsafe { this.lookup_task_candidates(tx, key) }
}
}
}
......
use anyhow::Result;
use smallvec::SmallVec;
use turbo_persistence::{FamilyConfig, FamilyKind};
use crate::database::write_batch::{
ConcurrentWriteBatch, SerialWriteBatch, UnimplementedWriteBatch, WriteBatch,
......@@ -11,6 +13,20 @@ pub enum KeySpace {
TaskData = 2,
TaskCache = 3,
}
impl KeySpace {
/// Returns the persistence configuration for this keyspace.
pub const fn family_config(&self) -> FamilyConfig {
match self {
KeySpace::Infra | KeySpace::TaskMeta | KeySpace::TaskData => FamilyConfig {
kind: FamilyKind::SingleValue,
},
KeySpace::TaskCache => FamilyConfig {
// TaskCache uses hash-based lookups with potential collisions.
kind: FamilyKind::MultiValue,
},
}
}
}
pub trait KeyValueDatabase {
type ReadTransaction<'l>
......@@ -33,6 +49,19 @@ pub trait KeyValueDatabase {
key_space: KeySpace,
key: &[u8],
) -> Result<Option<Self::ValueBuffer<'l>>>;
/// Looks up a key and returns all matching values.
///
/// Useful for keyspaces where keys are hashes and collisions are possible (e.g., TaskCache).
/// The default implementation returns at most one value (from `get`), but implementations
/// that support multiple values per key should override this.
fn get_multiple<'l, 'db: 'l>(
&'l self,
transaction: &'l Self::ReadTransaction<'db>,
key_space: KeySpace,
key: &[u8],
) -> Result<SmallVec<[Self::ValueBuffer<'l>; 1]>> {
Ok(self.get(transaction, key_space, key)?.into_iter().collect())
}
fn batch_get<'l, 'db: 'l>(
&'l self,
......
......@@ -57,22 +57,20 @@ impl<T: KeyValueDatabase> StartupCacheLayer<T> {
pub fn new(database: T, path: PathBuf, fresh_db: bool) -> Result<Self> {
let mut restored = Vec::new();
let mut restored_map = ByKeySpace::new(|_| FxHashMap::default());
if !fresh_db {
if let Result::Ok(mut cache_file) = File::open(&path) {
cache_file.read_to_end(&mut restored)?;
drop(cache_file);
let mut pos = 0;
while pos < restored.len() {
let (key_space, key, value) = read_key_value_pair(&restored, &mut pos)?;
let map = restored_map.get_mut(key_space);
unsafe {
// Safety: This is a self reference, it's valid as long the `restored`
// buffer is alive
map.insert(
transmute::<&'_ [u8], &'static [u8]>(key),
transmute::<&'_ [u8], &'static [u8]>(value),
);
}
if !fresh_db && let Result::Ok(mut cache_file) = File::open(&path) {
cache_file.read_to_end(&mut restored)?;
drop(cache_file);
let mut pos = 0;
while pos < restored.len() {
let (key_space, key, value) = read_key_value_pair(&restored, &mut pos)?;
let map = restored_map.get_mut(key_space);
unsafe {
// Safety: This is a self reference, it's valid as long the `restored`
// buffer is alive
map.insert(
transmute::<&'_ [u8], &'static [u8]>(key),
transmute::<&'_ [u8], &'static [u8]>(value),
);
}
}
}
......
......@@ -8,8 +8,9 @@ use std::{
use anyhow::{Ok, Result};
use parking_lot::Mutex;
use smallvec::SmallVec;
use turbo_persistence::{
ArcBytes, CompactConfig, KeyBase, StoreKey, TurboPersistence, ValueBuffer,
ArcBytes, CompactConfig, DbConfig, KeyBase, StoreKey, TurboPersistence, ValueBuffer,
};
use turbo_tasks::{JoinHandle, message_queue::TimingEvent, spawn, turbo_tasks};
......@@ -45,7 +46,15 @@ pub struct TurboKeyValueDatabase {
impl TurboKeyValueDatabase {
pub fn new(versioned_path: PathBuf, is_ci: bool, is_short_session: bool) -> Result<Self> {
let db = Arc::new(TurboPersistence::open(versioned_path)?);
const CONFIG: DbConfig<FAMILIES> = DbConfig {
family_configs: [
KeySpace::Infra.family_config(),
KeySpace::TaskMeta.family_config(),
KeySpace::TaskData.family_config(),
KeySpace::TaskCache.family_config(),
],
};
let db = Arc::new(TurboPersistence::open_with_config(versioned_path, CONFIG)?);
Ok(Self {
db: db.clone(),
compact_join_handle: Mutex::new(None),
......@@ -93,6 +102,15 @@ impl KeyValueDatabase for TurboKeyValueDatabase {
self.db.batch_get(key_space as usize, keys)
}
fn get_multiple<'l, 'db: 'l>(
&'l self,
_transaction: &'l Self::ReadTransaction<'db>,
key_space: KeySpace,
key: &[u8],
) -> Result<SmallVec<[Self::ValueBuffer<'l>; 1]>> {
self.db.get_multiple(key_space as usize, &key)
}
type ConcurrentWriteBatch<'l>
= TurboWriteBatch<'l>
where
......
......@@ -30,8 +30,8 @@ macro_rules! impl_write_number {
/// Note that the default [`std::hash::Hash`] trait used by Rust allows for hashing that differs
/// across process runs, so it is not suitable for filesystem cache with turbo-tasks.
///
/// It's very important that `Vc`s never implement this, since they cannot be deterministic. The
/// value that they wrap, however, can implement the trait.
/// It's very important that `Vc`s never implement this, since they are not consistent from run to
/// run. The value that they wrap, however, can implement the trait.
pub trait DeterministicHash {
/// Adds `self`'s bytes to the [`DeterministicHasher`]'s state, in a way that is replicatable on
/// any platform or process run.
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment