Unverified Commit d81d5ab7 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

Proof of concept: task eviction after snapshot for turbo-tasks-backend (#91790)

> **Note:** This is a **proof of concept** implementation. It is not yet
ready for production use.

## Summary

Implements memory eviction for the turbo-tasks engine. After a
persistence snapshot completes, tasks that are safe to remove are
evicted from in-memory storage and transparently restored from disk on
next access.

### Eviction levels

- **Full eviction**: Entire task removed from the in-memory map
(restored from disk on access). Only possible when the task has no
meaningful transient state (and other state is already on disk)
- **DataAndMeta eviction**: Both data and meta categories cleared, but
the task stays in the map to preserve transient state (e.g.
`current_session_clean`, aggregated session-clean counts).
- **DataOnly eviction**: Only data-category fields cleared; meta (graph
structure, output, dirty state) stays in memory.
- **MetaOnly eviction**: Only meta-category fields cleared; data stays
in memory.

Data and meta evictability are computed independently — if one category
is modified but the other is clean, the clean category can still be
dropped.

Eviction is gated behind `BackendOptions::evict_after_snapshot` (off by
default), and can be enabled in Next.js via the
`TURBO_ENGINE_EVICT_AFTER_SNAPSHOT=1` env var for testing.

## Key changes

- **Orthogonal eviction decision tree** (`storage_schema.rs`): Data and
meta evictability are computed independently. Full eviction additionally
requires no meaningful transient state (session-clean flags, aggregated
session-clean counts). Replaces the previous sequential bail-out
approach which was too aggressive on full eviction (losing transient
session state on leaf tasks) and not aggressive enough on partial
eviction (blocking all eviction when only one category was modified).

- **`drop_partial()` codegen** (`task_storage_macro.rs`): New generated
methods to drop data

- **`restore_from_*()` codegen changes** (`task_storage_macro.rs`): New
semantics for merging persistent data from the backend with transient
data stored in memory.

- **`task_cache` moved into `Storage`** (`storage.rs`): The
`CachedTaskType → TaskId` deduplication map was previously a separate
field on `TurboTasksBackendInner`. It is now owned by `Storage` so
eviction can remove entries when a task is fully evicted. Because
`task_cache` is a pure performance cache (entries are re-populated by
`task_by_type()` on miss once the task type is persisted to backing
storage), evicting entries is safe. After bulk eviction the map is
shrunk when it is less than half full.

- **Parallel shard eviction** (`storage.rs`): Eviction iterates all
storage shards in parallel after snapshot, applying the appropriate
eviction level per task. Each shard is shrunk after bulk eviction to
reclaim slack capacity.
- In principle this is O(N) work to scan, but because each pass drops
>98% of tasks there isn't wasted work and the logic is fast, taking
<100ms for even the largest applications.


## Design notes

- **SessionDependent tasks**: SessionDependent tasks can still be
evicted but if `current_session_clean` is set we prevent full eviction
to avoid rechecking. Within a session the file-watchers are responsible
for invalidations after setting `current_session_clean`.

## Known limitations (proof of concept)

- No LRU or access-frequency tracking — all eligible tasks are evicted
on every snapshot cycle
- No memory pressure feedback — eviction runs on a timer, not in
response to actual memory pressure
- Only runs after snapshotting which tends to be a high point in memory
- Future work will explore interleaving this logic with snapshotting to
trim the peak



<!-- NEXT_JS_LLM_PR -->
parent ed9a295f
...@@ -9889,6 +9889,7 @@ dependencies = [ ...@@ -9889,6 +9889,7 @@ dependencies = [
"thread_local", "thread_local",
"tokio", "tokio",
"tracing", "tracing",
"triomphe 0.1.12",
"turbo-bincode", "turbo-bincode",
"turbo-persistence", "turbo-persistence",
"turbo-rcstr", "turbo-rcstr",
...@@ -10080,6 +10081,7 @@ dependencies = [ ...@@ -10080,6 +10081,7 @@ dependencies = [
"smallvec", "smallvec",
"tokio", "tokio",
"turbo-tasks", "turbo-tasks",
"turbo-tasks-backend",
] ]
[[package]] [[package]]
......
...@@ -1495,8 +1495,8 @@ impl Project { ...@@ -1495,8 +1495,8 @@ impl Project {
/// Computes the whole app module graph without dropping issues. /// Computes the whole app module graph without dropping issues.
/// ///
/// Use this instead of [`whole_app_module_graphs`] when you need to collect issues from the /// Use this instead of [Self::whole_app_module_graphs] when you need to collect issues from
/// computation (e.g. for the `get_compilation_issues` MCP tool). /// the computation (e.g. for the `get_compilation_issues` MCP tool).
#[turbo_tasks::function] #[turbo_tasks::function]
pub async fn whole_app_module_graphs_without_dropping_issues( pub async fn whole_app_module_graphs_without_dropping_issues(
self: ResolvedVc<Self>, self: ResolvedVc<Self>,
......
...@@ -256,6 +256,8 @@ pub fn create_turbo_tasks( ...@@ -256,6 +256,8 @@ pub fn create_turbo_tasks(
}), }),
dependency_tracking, dependency_tracking,
num_workers: Some(tokio::runtime::Handle::current().metrics().num_workers()), num_workers: Some(tokio::runtime::Handle::current().metrics().num_workers()),
evict_after_snapshot: std::env::var("TURBO_ENGINE_EVICT_AFTER_SNAPSHOT")
.is_ok_and(|v| v == "1" || v == "true"),
..Default::default() ..Default::default()
}, },
Either::Left(backing_storage), Either::Left(backing_storage),
......
import { nextTestSetup, isNextDev } from 'e2e-utils'
import { retry, waitFor } from 'next-test-utils'
// Eviction requires the dev server (HMR) and persistent caching (Turbopack).
// Skip entirely in prod/start mode.
;(isNextDev ? describe : describe.skip)('evict-after-snapshot', () => {
const envVars = [
'ENABLE_CACHING=1',
'TURBO_ENGINE_IGNORE_DIRTY=1',
'TURBO_ENGINE_SNAPSHOT_IDLE_TIMEOUT_MILLIS=1000',
'TURBO_ENGINE_EVICT_AFTER_SNAPSHOT=1',
].join(' ')
const { skipped, next } = nextTestSetup({
files: __dirname,
skipDeployment: true,
packageJson: {
scripts: {
dev: `${envVars} next dev`,
},
},
installCommand: 'npm i',
startCommand: 'npm run dev',
})
if (skipped) {
return
}
async function waitForSnapshotAndEviction() {
// The idle timeout is 1s, give extra time for snapshot + eviction to complete
await waitFor(5000)
}
// Turbopack-only: eviction requires persistent caching
;(process.env.IS_TURBOPACK_TEST ? it : it.skip)(
'should serve correct content after eviction and HMR',
async () => {
const browser = await next.browser('/')
await retry(async () => {
expect(await browser.elementByCss('p').text()).toBe('hello world')
})
let currentContent = 'hello world'
for (let cycle = 1; cycle <= 3; cycle++) {
await waitForSnapshotAndEviction()
const prevContent = currentContent
const nextContent = `cycle ${cycle}`
await next.patchFile('app/page.tsx', (content) =>
content.replace(prevContent, nextContent)
)
currentContent = nextContent
const expected = currentContent
await retry(async () => {
expect(await browser.elementByCss('p').text()).toBe(expected)
}, 10000)
}
await browser.close()
},
90000
)
;(process.env.IS_TURBOPACK_TEST ? it : it.skip)(
'should handle client component HMR after eviction',
async () => {
const browser = await next.browser('/client')
await retry(async () => {
expect(await browser.elementByCss('p').text()).toBe('hello world')
})
await waitForSnapshotAndEviction()
await next.patchFile(
'app/client/page.tsx',
(content) => content.replace('hello world', 'hello eviction'),
async () => {
await retry(async () => {
expect(await browser.elementByCss('p').text()).toBe(
'hello eviction'
)
}, 10000)
}
)
await browser.close()
},
90000
)
})
...@@ -619,8 +619,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES> ...@@ -619,8 +619,7 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES>
/// Clears all caches of the database. /// Clears all caches of the database.
pub fn clear_cache(&self) { pub fn clear_cache(&self) {
self.key_block_cache.clear(); self.clear_block_caches();
self.value_block_cache.clear();
for meta in self.inner.write().meta_files.iter_mut() { for meta in self.inner.write().meta_files.iter_mut() {
meta.clear_cache(); meta.clear_cache();
} }
......
...@@ -97,6 +97,14 @@ impl<K: Hash + Eq, H: BuildHasher + Default, const I: usize> AutoSet<K, H, I> { ...@@ -97,6 +97,14 @@ impl<K: Hash + Eq, H: BuildHasher + Default, const I: usize> AutoSet<K, H, I> {
pub fn contains(&self, key: &K) -> bool { pub fn contains(&self, key: &K) -> bool {
self.map.contains_key(key) self.map.contains_key(key)
} }
/// see [HashSet::retain](https://doc.rust-lang.org/std/collections/hash_set/struct.HashSet.html#method.retain)
pub fn retain<F>(&mut self, mut f: F)
where
F: FnMut(&K) -> bool,
{
self.map.retain(|k, _| f(k));
}
} }
impl<K, H, const I: usize> AutoSet<K, H, I> { impl<K, H, const I: usize> AutoSet<K, H, I> {
......
...@@ -60,6 +60,7 @@ turbo-persistence = { workspace = true } ...@@ -60,6 +60,7 @@ turbo-persistence = { workspace = true }
turbo-rcstr = { workspace = true } turbo-rcstr = { workspace = true }
turbo-tasks = { workspace = true } turbo-tasks = { workspace = true }
turbo-tasks-hash = { workspace = true } turbo-tasks-hash = { workspace = true }
turbo-tasks-malloc = { workspace = true, default-features = false }
thread_local = { workspace = true } thread_local = { workspace = true }
[dev-dependencies] [dev-dependencies]
...@@ -68,6 +69,7 @@ futures = { workspace = true } ...@@ -68,6 +69,7 @@ futures = { workspace = true }
indoc = { workspace = true } indoc = { workspace = true }
regex = { workspace = true } regex = { workspace = true }
tempfile = { workspace = true } tempfile = { workspace = true }
triomphe = { workspace = true }
turbo-tasks-malloc = { workspace = true } turbo-tasks-malloc = { workspace = true }
rstest = { workspace = true } rstest = { workspace = true }
turbo-tasks-testing = { workspace = true } turbo-tasks-testing = { workspace = true }
......
...@@ -2,27 +2,14 @@ ...@@ -2,27 +2,14 @@
//! //!
//! Every task cell — whether its value type is bincode-serializable, hash-only, //! Every task cell — whether its value type is bincode-serializable, hash-only,
//! derivable, or non-reconstructible — lives in a single `CellData` map keyed //! derivable, or non-reconstructible — lives in a single `CellData` map keyed
//! by [`CellId`]. The map's bincode impl decides at encode time which entries //! by [`CellId`].
//! to persist, by consulting the global [`ValueType`] registry: entries whose
//! value type has no bincode function are omitted from the serialized output.
//!
//! This replaces the older split of `persistent_cell_data` /
//! `transient_cell_data` fields which routed every cell write through an
//! `is_serializable_cell_content: bool` that threaded through ~14 call sites.
//! By keying the bincode decision on the value type itself, the routing
//! collapses to an unconditional insert.
//!
//! The inner value is stored as [`SharedReference`] rather than
//! [`TypedSharedReference`] because the `CellId` key already carries the
//! [`ValueTypeId`] — duplicating it in each map entry would waste memory.
//! Encode / decode recover the value type from the key.
use std::{ use std::{
hash::BuildHasherDefault, hash::BuildHasherDefault,
ops::{Deref, DerefMut}, ops::{Deref, DerefMut},
}; };
use auto_hash_map::AutoMap; use auto_hash_map::{AutoMap, map::Entry};
use bincode::{ use bincode::{
Decode, Encode, Decode, Encode,
error::{DecodeError, EncodeError}, error::{DecodeError, EncodeError},
...@@ -32,8 +19,18 @@ use turbo_bincode::{ ...@@ -32,8 +19,18 @@ use turbo_bincode::{
TurboBincodeDecode, TurboBincodeDecoder, TurboBincodeEncode, TurboBincodeEncoder, TurboBincodeDecode, TurboBincodeDecoder, TurboBincodeEncode, TurboBincodeEncoder,
impl_decode_for_turbo_bincode_decode, impl_encode_for_turbo_bincode_encode, impl_decode_for_turbo_bincode_decode, impl_encode_for_turbo_bincode_encode,
}; };
use turbo_tasks::{CellId, SharedReference, ShrinkToFit, ValueTypePersistence, registry}; use turbo_tasks::{
CellId, Evictability, SharedReference, ShrinkToFit, ValueTypePersistence, registry,
};
use crate::backend::storage_schema::{DropPartial, DropPartialOutcome, MergeRestore};
/// The value is stored as [`SharedReference`] rather than
/// [`TypedSharedReference`] because the `CellId` key already carries the
/// [`ValueTypeId`] — duplicating it in each map entry would waste memory.
/// Encode / decode recover the value type from the key.
/// Default inline size is 1 because this optimizes storage layout and a non-trivial number of tasks
/// have <=1 cells
type InnerMap = AutoMap<CellId, SharedReference, BuildHasherDefault<FxHasher>, 1>; type InnerMap = AutoMap<CellId, SharedReference, BuildHasherDefault<FxHasher>, 1>;
/// Map of cell id → shared reference, with bincode that filters out entries /// Map of cell id → shared reference, with bincode that filters out entries
...@@ -47,6 +44,68 @@ impl CellData { ...@@ -47,6 +44,68 @@ impl CellData {
} }
} }
impl MergeRestore for CellData {
type Item = (CellId, SharedReference);
fn merge_restore(&mut self, items: impl IntoIterator<Item = Self::Item>) {
for (k, v) in items {
match self.entry(k) {
Entry::Vacant(e) => {
e.insert(v);
}
Entry::Occupied(_e) => {
// Residue exists for this CellId. Keep it; the in-memory
// value is more authoritative than the decoded one.
debug_assert!(
!matches!(
registry::get_value_type(k.type_id).evictability,
Evictability::Always,
),
"Found an evictable cell in a task we are restoring into: {}",
registry::get_value_type(k.type_id).ty.name,
);
}
}
}
}
}
impl DropPartial for CellData {
/// Drop cells whose value type is freely evictable, retain those that
/// are not. Called by the macro-generated `TaskStorage::drop_partial`
/// on the data-eviction path.
///
/// Dropped (`Evictability::Always`): persistable cells (restored from
/// disk on next access), skip cells (re-derived by re-running the task),
/// and hash-only cells (re-derived; hash gates spurious invalidation).
///
/// Retained:
/// - `Evictability::Expensive` — re-derivation is non-trivial, prefer keeping in memory.
/// - `Evictability::Never` — value type holds session-scoped state that must not leave memory
/// (`State<>` cells, file watchers, worker pools).
fn drop_partial(&mut self) -> DropPartialOutcome {
self.0.retain(
|cell_id, _| match registry::get_value_type(cell_id.type_id).evictability {
Evictability::Always => false,
Evictability::Expensive | Evictability::Never => true,
},
);
if self.0.is_empty() {
return DropPartialOutcome::Empty;
}
self.shrink_to_fit();
DropPartialOutcome::HasResidue
}
}
impl IntoIterator for CellData {
type Item = (CellId, SharedReference);
type IntoIter = <InnerMap as IntoIterator>::IntoIter;
fn into_iter(self) -> Self::IntoIter {
self.0.into_iter()
}
}
impl Deref for CellData { impl Deref for CellData {
type Target = InnerMap; type Target = InnerMap;
...@@ -69,9 +128,9 @@ impl ShrinkToFit for CellData { ...@@ -69,9 +128,9 @@ impl ShrinkToFit for CellData {
impl TurboBincodeEncode for CellData { impl TurboBincodeEncode for CellData {
/// Writes `count-of-persistable-entries` followed by each persistable /// Writes `count-of-persistable-entries` followed by each persistable
/// `(CellId, encoded-value)`. Entries whose value type is `SkipPersist` /// `(CellId, encoded-value)`. Entries whose value type is `Skip` or
/// or `SessionStateful` (no bincode) are skipped; they will be /// `HashOnly` (no bincode codec) are skipped; they will be reconstructed
/// reconstructed on the next task execution after restore. /// on the next task execution after restore.
fn encode(&self, encoder: &mut TurboBincodeEncoder) -> Result<(), EncodeError> { fn encode(&self, encoder: &mut TurboBincodeEncoder) -> Result<(), EncodeError> {
// First pass: count persistable entries. One extra O(N) iteration over // First pass: count persistable entries. One extra O(N) iteration over
// the registry — cold path (snapshot time only) and the registry is a // the registry — cold path (snapshot time only) and the registry is a
...@@ -129,3 +188,101 @@ impl<Context> TurboBincodeDecode<Context> for CellData { ...@@ -129,3 +188,101 @@ impl<Context> TurboBincodeDecode<Context> for CellData {
impl_encode_for_turbo_bincode_encode!(CellData); impl_encode_for_turbo_bincode_encode!(CellData);
impl_decode_for_turbo_bincode_decode!(CellData); impl_decode_for_turbo_bincode_decode!(CellData);
#[cfg(test)]
mod tests {
//! `drop_partial` must partition cells by their `Evictability` — keep
//! the non-evictable ones, drop the rest. Tests below cover every
//! `(persistence, evictability)` combination the macro currently emits,
//! including the `Persistable + Never` combo (e.g. `DiskFileSystem`).
use turbo_tasks::{self as turbo_tasks, VcValueType};
use super::*;
#[turbo_tasks::value]
struct PersistableV(#[allow(dead_code)] u32);
#[turbo_tasks::value(evict = "never")]
struct PersistableNeverV(#[allow(dead_code)] u32);
#[turbo_tasks::value(serialization = "skip")]
struct SkipCheapV(
#[turbo_tasks(trace_ignore)]
#[allow(dead_code)]
u32,
);
#[turbo_tasks::value(serialization = "skip", evict = "last")]
struct SkipExpensiveV(
#[turbo_tasks(trace_ignore)]
#[allow(dead_code)]
u32,
);
#[turbo_tasks::value(serialization = "skip", evict = "never", cell = "new", eq = "manual")]
struct SessionStatefulV;
#[turbo_tasks::value(serialization = "hash")]
struct HashOnlyV(#[allow(dead_code)] u32);
fn cell_of<V: VcValueType>(index: u32) -> CellId {
CellId {
type_id: V::get_value_type_id(),
index,
}
}
fn dummy_ref() -> SharedReference {
// The drop_partial logic only inspects the key's type_id, not the
// value, so any Any + Send + Sync works.
SharedReference::new(triomphe::Arc::new(0u32))
}
#[test]
fn drop_partial_partitions_by_evictability() {
let mut data = CellData::new();
data.insert(cell_of::<PersistableV>(0), dummy_ref());
data.insert(cell_of::<PersistableNeverV>(0), dummy_ref());
data.insert(cell_of::<SkipCheapV>(0), dummy_ref());
data.insert(cell_of::<SkipExpensiveV>(0), dummy_ref());
data.insert(cell_of::<SessionStatefulV>(0), dummy_ref());
data.insert(cell_of::<HashOnlyV>(0), dummy_ref());
assert_eq!(data.drop_partial(), DropPartialOutcome::HasResidue);
assert_eq!(data.len(), 3);
assert!(data.contains_key(&cell_of::<PersistableNeverV>(0)));
assert!(data.contains_key(&cell_of::<SkipExpensiveV>(0)));
assert!(data.contains_key(&cell_of::<SessionStatefulV>(0)));
assert!(!data.contains_key(&cell_of::<PersistableV>(0)));
assert!(!data.contains_key(&cell_of::<SkipCheapV>(0)));
assert!(!data.contains_key(&cell_of::<HashOnlyV>(0)));
}
#[test]
fn drop_partial_fully_empties_when_all_evictable() {
let mut data = CellData::new();
data.insert(cell_of::<PersistableV>(0), dummy_ref());
data.insert(cell_of::<SkipCheapV>(0), dummy_ref());
data.insert(cell_of::<HashOnlyV>(0), dummy_ref());
assert_eq!(data.drop_partial(), DropPartialOutcome::Empty);
assert!(data.is_empty());
}
#[test]
fn drop_partial_keeps_everything_when_all_non_evictable() {
let mut data = CellData::new();
data.insert(cell_of::<PersistableNeverV>(0), dummy_ref());
data.insert(cell_of::<SkipExpensiveV>(0), dummy_ref());
data.insert(cell_of::<SessionStatefulV>(0), dummy_ref());
assert_eq!(data.drop_partial(), DropPartialOutcome::HasResidue);
assert_eq!(data.len(), 3);
}
#[test]
fn drop_partial_on_empty_returns_empty() {
let mut data = CellData::new();
assert_eq!(data.drop_partial(), DropPartialOutcome::Empty);
}
}
...@@ -117,6 +117,34 @@ impl<K, V> CounterMap<K, V> { ...@@ -117,6 +117,34 @@ impl<K, V> CounterMap<K, V> {
{ {
self.0.remove(key) self.0.remove(key)
} }
/// Retain only entries for which `f(key, value)` returns `true`.
pub fn retain<F>(&mut self, mut f: F)
where
F: FnMut(&K, &V) -> bool,
K: Eq + Hash,
{
self.0.retain(|k, v| f(k, v));
}
/// Extend this map with the entries from an iterator. Used by restore paths
/// to merge persistent entries loaded from disk into an existing map that
/// may hold transient residue.
pub fn extend(&mut self, iter: impl IntoIterator<Item = (K, V)>)
where
K: Eq + Hash,
{
self.0.extend(iter);
}
}
impl<K, V> IntoIterator for CounterMap<K, V> {
type Item = (K, V);
type IntoIter = <InnerMap<K, V> as IntoIterator>::IntoIter;
fn into_iter(self) -> Self::IntoIter {
self.0.into_iter()
}
} }
impl<K: Hash + Eq, V: CounterValue> CounterMap<K, V> { impl<K: Hash + Eq, V: CounterValue> CounterMap<K, V> {
......
...@@ -23,7 +23,7 @@ use turbo_tasks::{ ...@@ -23,7 +23,7 @@ use turbo_tasks::{
macro_helpers::NativeFunction, macro_helpers::NativeFunction,
}; };
use self::aggregation_update::ComputeDirtyAndCleanUpdate; pub use self::aggregation_update::ComputeDirtyAndCleanUpdate;
use crate::{ use crate::{
backend::{ backend::{
EventDescription, TaskDataCategory, TurboTasksBackend, TurboTasksBackendInner, EventDescription, TaskDataCategory, TurboTasksBackend, TurboTasksBackendInner,
...@@ -562,6 +562,7 @@ impl<'e, B: BackingStorage> ExecuteContextImpl<'e, B> { ...@@ -562,6 +562,7 @@ impl<'e, B: BackingStorage> ExecuteContextImpl<'e, B> {
if let Some(task_type) = entry.task_type.clone() { if let Some(task_type) = entry.task_type.clone() {
// Insert into the task cache to avoid future lookups // Insert into the task cache to avoid future lookups
self.backend self.backend
.storage
.task_cache .task_cache
.entry(task_type) .entry(task_type)
.or_insert(entry.task_id); .or_insert(entry.task_id);
...@@ -641,7 +642,7 @@ fn apply_restore_result( ...@@ -641,7 +642,7 @@ fn apply_restore_result(
task.flags.set_restoring(task_category, false); task.flags.set_restoring(task_category, false);
return Ok(()); return Ok(());
} }
task.restore_from(storage, task_category); task.restore_from(storage, category);
task.flags.set_restored(task_category); task.flags.set_restored(task_category);
task.flags.set_restoring(task_category, false); task.flags.set_restoring(task_category, false);
Ok(()) Ok(())
...@@ -1053,6 +1054,7 @@ impl Display for TaskTypeRef<'_> { ...@@ -1053,6 +1054,7 @@ impl Display for TaskTypeRef<'_> {
} }
} }
#[derive(Debug)]
pub enum TaskType { pub enum TaskType {
Cached(Arc<CachedTaskType>), Cached(Arc<CachedTaskType>),
Transient(Arc<TransientTask>), Transient(Arc<TransientTask>),
......
...@@ -209,7 +209,7 @@ impl UpdateCellOperation { ...@@ -209,7 +209,7 @@ impl UpdateCellOperation {
task.remove_cell_data(&cell) task.remove_cell_data(&cell)
}; };
// Update cell_data_hash for non-hashonly cells. // Update cell_data_hash for hash-only cells.
if matches!(value_type.persistence, ValueTypePersistence::HashOnly) { if matches!(value_type.persistence, ValueTypePersistence::HashOnly) {
update_cell_data_hash(&mut task, &cell, content_hash); update_cell_data_hash(&mut task, &cell, content_hash);
} }
......
...@@ -64,6 +64,47 @@ pub fn raw_entry_in_shard<'l, K: Eq + Hash, V, S: BuildHasher + Clone>( ...@@ -64,6 +64,47 @@ pub fn raw_entry_in_shard<'l, K: Eq + Hash, V, S: BuildHasher + Clone>(
} }
} }
/// Outcome of [`try_lock_and_remove`].
pub enum TryLockAndRemove {
/// The shard lock was acquired and a matching entry was removed.
Removed,
/// The shard lock was acquired but no matching entry was present.
NotFound,
/// The shard lock was contended; the caller should retry later after releasing
/// any other locks they are holding.
WouldBlock,
}
/// Remove `key` from `map` without blocking on shard contention.
///
/// Intended for call sites that already hold another lock and want to avoid a
/// cyclic wait. On contention (`WouldBlock`), the caller is expected to defer the
/// removal and retry after dropping the other lock.
pub fn try_lock_and_remove<
K: Eq + Hash + AsRef<Q>,
V,
Q: Eq + Hash + ?Sized,
S: BuildHasher + Clone,
>(
map: &DashMap<K, V, S>,
key: &Q,
) -> TryLockAndRemove {
let hasher = map.hasher();
let hash = hasher.hash_one(key);
let shard_idx = map.determine_shard(hash as usize);
let Some(mut shard) = map.shards()[shard_idx].try_write() else {
return TryLockAndRemove::WouldBlock;
};
// SAFETY: we hold the write lock for the duration of the find/erase.
match shard.find(hash, |(k, _v)| k.as_ref() == key) {
Some(bucket) => {
unsafe { shard.erase(bucket) };
TryLockAndRemove::Removed
}
None => TryLockAndRemove::NotFound,
}
}
pub enum RawEntry<'l, K, V> { pub enum RawEntry<'l, K, V> {
Occupied(OccupiedEntry<'l, K, V>), Occupied(OccupiedEntry<'l, K, V>),
Vacant(VacantEntry<'l, K, V>), Vacant(VacantEntry<'l, K, V>),
......
This diff is collapsed.
...@@ -11,6 +11,7 @@ ...@@ -11,6 +11,7 @@
turbo_tasks_backend::BackendOptions { turbo_tasks_backend::BackendOptions {
num_workers: Some(2), num_workers: Some(2),
small_preallocation: true, small_preallocation: true,
storage_mode: Some(turbo_tasks_backend::StorageMode::ReadWriteOnShutdown),
..Default::default() ..Default::default()
}, },
turbo_tasks_backend::turbo_backing_storage( turbo_tasks_backend::turbo_backing_storage(
......
...@@ -10,6 +10,7 @@ ...@@ -10,6 +10,7 @@
turbo_tasks_backend::BackendOptions { turbo_tasks_backend::BackendOptions {
num_workers: Some(2), num_workers: Some(2),
small_preallocation: true, small_preallocation: true,
storage_mode: Some(turbo_tasks_backend::StorageMode::ReadWriteOnShutdown),
..Default::default() ..Default::default()
}, },
turbo_tasks_backend::turbo_backing_storage( turbo_tasks_backend::turbo_backing_storage(
......
...@@ -501,9 +501,14 @@ impl DiskFileSystemInner { ...@@ -501,9 +501,14 @@ impl DiskFileSystemInner {
} }
} }
/// `DiskFileSystem` carries serializable fields (`name`, `root`,
/// `denied_paths`) inside `DiskFileSystemInner` alongside session-scoped
/// state (the `notify` watcher, invalidator maps, weak `TurboTasksApi`,
/// etc.) This is important to maintain invariants in a session and ensure invalidations work, so we
/// never evict this data.
#[derive(Clone, ValueToString)] #[derive(Clone, ValueToString)]
#[value_to_string(self.inner.name)] #[value_to_string(self.inner.name)]
#[turbo_tasks::value(cell = "new", eq = "manual")] #[turbo_tasks::value(cell = "new", eq = "manual", evict = "never")]
pub struct DiskFileSystem { pub struct DiskFileSystem {
inner: Arc<DiskFileSystemInner>, inner: Arc<DiskFileSystemInner>,
} }
......
...@@ -4,6 +4,7 @@ ...@@ -4,6 +4,7 @@
turbo_tasks_backend::BackendOptions { turbo_tasks_backend::BackendOptions {
num_workers: Some(2), num_workers: Some(2),
small_preallocation: true, small_preallocation: true,
storage_mode: Some(turbo_tasks_backend::StorageMode::ReadWriteOnShutdown),
..Default::default() ..Default::default()
}, },
turbo_tasks_backend::noop_backing_storage(), turbo_tasks_backend::noop_backing_storage(),
......
...@@ -53,11 +53,14 @@ pub fn primitive(input: TokenStream) -> TokenStream { ...@@ -53,11 +53,14 @@ pub fn primitive(input: TokenStream) -> TokenStream {
decode_ty, decode_ty,
} = bincode_wrappers; } = bincode_wrappers;
quote! { quote! {
turbo_tasks::ValueType::new_with_bincode_wrappers::<#ty, #encode_ty, #decode_ty>(#name) turbo_tasks::ValueType::new_with_bincode_wrappers::<#ty, #encode_ty, #decode_ty>(
#name,
turbo_tasks::Evictability::Always,
)
} }
} else { } else {
quote! { quote! {
turbo_tasks::ValueType::persistable::<#ty>(#name) turbo_tasks::ValueType::persistable::<#ty>(#name, turbo_tasks::Evictability::Always)
} }
}; };
......
...@@ -286,15 +286,20 @@ pub fn value(args: TokenStream, input: TokenStream) -> TokenStream { ...@@ -286,15 +286,20 @@ pub fn value(args: TokenStream, input: TokenStream) -> TokenStream {
.into(); .into();
} }
// `evict = "last" | "never"` is only valid when `serialization = "skip"`; // `evict = "last"` only makes sense for `serialization = "skip"`: it
// other persistence modes have their own eviction semantics fixed by the // says "re-deriving this cell is expensive", and re-derivation is the
// backend (Persistable: evict-and-restore, HashOnly: evict-with-hash-gate). // recovery path only for skip mode. Persistable cells restore from disk
if !matches!(evict_mode, EvictMode::Always) // (predictable cost), HashOnly cells short-circuit on unchanged hash.
//
// `evict = "never"` is allowed with any serialization mode — a value
// type can be persistable AND hold session-scoped state that must not
// leave memory (e.g. `DiskFileSystem` carrying file watchers).
if matches!(evict_mode, EvictMode::Last)
&& !matches!(serialization_mode, SerializationMode::Skip) && !matches!(serialization_mode, SerializationMode::Skip)
{ {
return syn::Error::new( return syn::Error::new(
proc_macro2::Span::call_site(), proc_macro2::Span::call_site(),
"evict = \"last\" | \"never\" is only valid with serialization = \"skip\"", "evict = \"last\" is only valid with serialization = \"skip\"",
) )
.to_compile_error() .to_compile_error()
.into(); .into();
...@@ -469,24 +474,33 @@ pub fn value(args: TokenStream, input: TokenStream) -> TokenStream { ...@@ -469,24 +474,33 @@ pub fn value(args: TokenStream, input: TokenStream) -> TokenStream {
} }
let name = global_name_for_type(ident); let name = global_name_for_type(ident);
// Dispatch to the constructor whose name reflects the persistence + // `serialization` and `evict` set independent fields on `ValueType`:
// eviction combo. `evict` is only read when `serialization = Skip`; // persistence carries the codec (or marks Skip / HashOnly), evictability
// other modes ignore it (and the parser rejects non-Always values). // controls the in-memory drop policy. The codec-bearing constructor
let new_value_type = match (&serialization_mode, &evict_mode) { // (`persistable`) is kept distinct so its functions inline at the call
(SerializationMode::Auto | SerializationMode::Custom, _) => quote! { // site; non-codec modes go through the generic `new` constructor.
turbo_tasks::ValueType::persistable::<#ident>(#name) let evictability = match evict_mode {
}, EvictMode::Always => quote! { turbo_tasks::Evictability::Always },
(SerializationMode::Hash, _) => quote! { EvictMode::Last => quote! { turbo_tasks::Evictability::Expensive },
turbo_tasks::ValueType::hash_only::<#ident>(#name) EvictMode::Never => quote! { turbo_tasks::Evictability::Never },
}, };
(SerializationMode::Skip, EvictMode::Always) => quote! { let new_value_type = match &serialization_mode {
turbo_tasks::ValueType::skip_persist::<#ident>(#name) SerializationMode::Auto | SerializationMode::Custom => quote! {
turbo_tasks::ValueType::persistable::<#ident>(#name, #evictability)
}, },
(SerializationMode::Skip, EvictMode::Last) => quote! { SerializationMode::Skip => quote! {
turbo_tasks::ValueType::skip_persist_expensive::<#ident>(#name) turbo_tasks::ValueType::new::<#ident>(
#name,
turbo_tasks::ValueTypePersistence::Skip,
#evictability,
)
}, },
(SerializationMode::Skip, EvictMode::Never) => quote! { SerializationMode::Hash => quote! {
turbo_tasks::ValueType::session_stateful::<#ident>(#name) turbo_tasks::ValueType::new::<#ident>(
#name,
turbo_tasks::ValueTypePersistence::HashOnly,
#evictability,
)
}, },
}; };
let has_serialization = match serialization_mode { let has_serialization = match serialization_mode {
...@@ -587,6 +601,7 @@ pub fn value_type_and_register( ...@@ -587,6 +601,7 @@ pub fn value_type_and_register(
fn has_serialization() -> bool { fn has_serialization() -> bool {
#has_serialization #has_serialization
} }
} }
} }
} }
...@@ -95,9 +95,22 @@ impl TurboMalloc { ...@@ -95,9 +95,22 @@ impl TurboMalloc {
} }
pub fn thread_park() { pub fn thread_park() {
Self::collect(false);
}
/// When using mimalloc triggers some cleanup
/// force=false: process threadlocal free lists and other threadlocal deferred work
/// only operates on thread local data and should be fast
/// force=true: do all the work of `process=false` and then process global shared structures and
/// return memory to the OS if possible, this is much slower and should only be done rarely.
pub fn collect(force: bool) {
#[cfg(all(feature = "custom_allocator", not(target_family = "wasm")))] #[cfg(all(feature = "custom_allocator", not(target_family = "wasm")))]
unsafe { unsafe {
libmimalloc_sys::mi_collect(false); libmimalloc_sys::mi_collect(force);
}
#[cfg(not(all(feature = "custom_allocator", not(target_family = "wasm"))))]
{
let _ = force;
} }
} }
......
...@@ -20,3 +20,4 @@ rustc-hash = { workspace = true } ...@@ -20,3 +20,4 @@ rustc-hash = { workspace = true }
smallvec = { workspace = true } smallvec = { workspace = true }
tokio = { workspace = true } tokio = { workspace = true }
turbo-tasks = { workspace = true, features = ["non_operation_vc_strongly_consistent"] } turbo-tasks = { workspace = true, features = ["non_operation_vc_strongly_consistent"] }
turbo-tasks-backend = { workspace = true }
...@@ -28,7 +28,8 @@ use turbo_tasks::{ ...@@ -28,7 +28,8 @@ use turbo_tasks::{
}; };
pub use crate::run::{ pub use crate::run::{
Registration, run, run_once, run_once_without_cache_check, run_with_tt, run_without_cache_check, Registration, TestInstance, run, run_once, run_once_without_cache_check, run_with_tt,
run_without_cache_check, test_instance,
}; };
enum Task { enum Task {
......
use std::{env, fmt::Debug, future::Future, sync::Arc}; use std::{env, fmt::Debug, future::Future, sync::Arc};
use anyhow::Result; use anyhow::Result;
use turbo_tasks::{TurboTasksApi, trace::TraceRawVcs}; use turbo_tasks::{TurboTasks, TurboTasksApi, trace::TraceRawVcs};
use turbo_tasks_backend::{BackingStorage, TurboTasksBackend};
/// A freshly created test instance: the `TurboTasks` handle (type-erased to
/// `Arc<dyn TurboTasksApi>`) and a closure that, when called, takes a
/// snapshot and evicts all evictable tasks on that instance.
///
/// The eviction closure captures the concrete backend type internally so
/// harness code holding an erased `TurboTasksApi` can still reach the
/// `snapshot_and_evict` API.
pub struct TestInstance {
pub tt: Arc<dyn TurboTasksApi>,
pub snapshot_and_evict: Box<dyn Fn() + Send + Sync>,
}
/// Type-erased factory returned by the `register!` macro. Stays non-generic so
/// call sites can write `static REGISTRATION: Registration = register!();`
/// without naming the backing storage type.
pub struct Registration { pub struct Registration {
create_turbo_tasks: fn(&str, bool) -> Arc<dyn TurboTasksApi>, create_turbo_tasks: fn(&str, bool) -> TestInstance,
} }
impl Registration { impl Registration {
#[doc(hidden)] #[doc(hidden)]
pub const fn new(create_turbo_tasks: fn(&str, bool) -> Arc<dyn TurboTasksApi>) -> Self { pub const fn new(create_turbo_tasks: fn(&str, bool) -> TestInstance) -> Self {
Registration { create_turbo_tasks } Registration { create_turbo_tasks }
} }
pub fn create_turbo_tasks(&self, name: &str, initial: bool) -> Arc<dyn TurboTasksApi> { pub fn create_turbo_tasks(&self, name: &str, initial: bool) -> TestInstance {
(self.create_turbo_tasks)(name, initial) (self.create_turbo_tasks)(name, initial)
} }
} }
/// Wrap a concrete `Arc<TurboTasks<TurboTasksBackend<B>>>` into a
/// [`TestInstance`]. Called from the `register!` macro — the `.trs` closure
/// returns a concrete backend-parameterized `TurboTasks`, and this function
/// erases the type while retaining eviction access via a capturing closure.
pub fn test_instance<B>(tt: Arc<TurboTasks<TurboTasksBackend<B>>>) -> TestInstance
where
B: BackingStorage + 'static,
{
let tt_for_evict = tt.clone();
let snapshot_and_evict = Box::new(move || {
let _ = tt_for_evict
.backend()
.snapshot_and_evict_for_testing(&*tt_for_evict);
});
TestInstance {
tt: tt as Arc<dyn TurboTasksApi>,
snapshot_and_evict,
}
}
#[macro_export] #[macro_export]
macro_rules! register { macro_rules! register {
() => {{ () => {{
use std::sync::Arc; fn create_turbo_tasks(name: &str, initial: bool) -> turbo_tasks_testing::TestInstance {
use turbo_tasks::TurboTasksApi;
fn create_turbo_tasks(name: &str, initial: bool) -> Arc<dyn TurboTasksApi> {
let inner = include!(concat!( let inner = include!(concat!(
env!("CARGO_MANIFEST_DIR"), env!("CARGO_MANIFEST_DIR"),
"/tests/test_config.trs" "/tests/test_config.trs"
)); ));
(inner)(name, initial) turbo_tasks_testing::test_instance((inner)(name, initial))
} }
turbo_tasks_testing::Registration::new(create_turbo_tasks) turbo_tasks_testing::Registration::new(create_turbo_tasks)
}}; }};
...@@ -43,8 +76,8 @@ where ...@@ -43,8 +76,8 @@ where
T: TraceRawVcs + Send + 'static, T: TraceRawVcs + Send + 'static,
{ {
let name = closure_to_name(&fut); let name = closure_to_name(&fut);
let tt = registration.create_turbo_tasks(&name, true); let instance = registration.create_turbo_tasks(&name, true);
turbo_tasks::run_once(tt, async move { Ok(fut.await) }) turbo_tasks::run_once(instance.tt, async move { Ok(fut.await) })
.await .await
.unwrap() .unwrap()
} }
...@@ -57,8 +90,8 @@ where ...@@ -57,8 +90,8 @@ where
T: TraceRawVcs + Send + 'static, T: TraceRawVcs + Send + 'static,
{ {
let name = closure_to_name(&fut); let name = closure_to_name(&fut);
let tt = registration.create_turbo_tasks(&name, true); let instance = registration.create_turbo_tasks(&name, true);
turbo_tasks::run(tt, async move { Ok(fut.await) }) turbo_tasks::run(instance.tt, async move { Ok(fut.await) })
.await .await
.unwrap() .unwrap()
} }
...@@ -104,36 +137,41 @@ where ...@@ -104,36 +137,41 @@ where
let name = closure_to_name(&fut); let name = closure_to_name(&fut);
let mut i = 1; let mut i = 1;
loop { loop {
let tt = registration.create_turbo_tasks(&name, true); let instance = registration.create_turbo_tasks(&name, true);
println!("Run #{i} (without cache)"); println!("Run #{i} (without cache)");
let start = std::time::Instant::now(); let start = std::time::Instant::now();
let first = fut(tt.clone()).await?; let first = fut(instance.tt.clone()).await?;
println!("Run #{i} took {:?}", start.elapsed()); println!("Run #{i} took {:?}", start.elapsed());
i += 1; i += 1;
if !single_run { if !single_run {
let max_run = if infinite_memory_runs { usize::MAX } else { 10 }; let max_run = if infinite_memory_runs { usize::MAX } else { 10 };
for _ in 0..max_run { for _ in 0..max_run {
println!("Run #{i} (with memory cache, same TurboTasks instance)"); // Snapshot + evict between runs. Forces every subsequent read to
// go through the restore path instead of the warm in-memory cache,
// so tests exercise persistence on every iteration — not just the
// initial cold run and the post-`stop_and_wait` fs-cache runs.
(instance.snapshot_and_evict)();
println!("Run #{i} (with memory cache, same TurboTasks instance, post-evict)");
let start = std::time::Instant::now(); let start = std::time::Instant::now();
let second = fut(tt.clone()).await?; let second = fut(instance.tt.clone()).await?;
println!("Run #{i} took {:?}", start.elapsed()); println!("Run #{i} took {:?}", start.elapsed());
i += 1; i += 1;
assert_eq!(first, second); assert_eq!(first, second);
} }
} }
let start = std::time::Instant::now(); let start = std::time::Instant::now();
tt.stop_and_wait().await; instance.tt.stop_and_wait().await;
println!("Stopping TurboTasks took {:?}", start.elapsed()); println!("Stopping TurboTasks took {:?}", start.elapsed());
if !single_run { if !single_run {
for _ in 10..20 { for _ in 10..20 {
let tt = registration.create_turbo_tasks(&name, false); let instance = registration.create_turbo_tasks(&name, false);
println!("Run #{i} (with filesystem cache if available, new TurboTasks instance)"); println!("Run #{i} (with filesystem cache if available, new TurboTasks instance)");
let start = std::time::Instant::now(); let start = std::time::Instant::now();
let third = fut(tt.clone()).await?; let third = fut(instance.tt.clone()).await?;
println!("Run #{i} took {:?}", start.elapsed()); println!("Run #{i} took {:?}", start.elapsed());
i += 1; i += 1;
let start = std::time::Instant::now(); let start = std::time::Instant::now();
tt.stop_and_wait().await; instance.tt.stop_and_wait().await;
println!("Stopping TurboTasks took {:?}", start.elapsed()); println!("Stopping TurboTasks took {:?}", start.elapsed());
assert_eq!(first, third); assert_eq!(first, third);
} }
......
...@@ -27,7 +27,7 @@ pub fn get_invalidator() -> Option<Invalidator> { ...@@ -27,7 +27,7 @@ pub fn get_invalidator() -> Option<Invalidator> {
/// A lightweight handle to invalidate a task. Only stores the task ID. /// A lightweight handle to invalidate a task. Only stores the task ID.
/// The caller must provide the `TurboTasksApi` when calling invalidation methods. /// The caller must provide the `TurboTasksApi` when calling invalidation methods.
#[derive(Clone, Copy, Hash, PartialEq, Eq, Encode, Decode)] #[derive(Clone, Copy, Hash, PartialEq, Eq, Encode, Decode, Debug)]
pub struct Invalidator { pub struct Invalidator {
task: TaskId, task: TaskId,
} }
......
...@@ -110,7 +110,7 @@ pub use crate::{ ...@@ -110,7 +110,7 @@ pub use crate::{
task_execution_reason::TaskExecutionReason, task_execution_reason::TaskExecutionReason,
trait_ref::TraitRef, trait_ref::TraitRef,
value::{TransientInstance, TransientValue}, value::{TransientInstance, TransientValue},
value_type::{TraitMethod, TraitType, ValueType, ValueTypePersistence}, value_type::{Evictability, TraitMethod, TraitType, ValueType, ValueTypePersistence},
vc::{ vc::{
Dynamic, NonLocalValue, OperationValue, OperationVc, OptionVcExt, ReadVcFuture, Dynamic, NonLocalValue, OperationValue, OperationVc, OptionVcExt, ReadVcFuture,
ResolveOperationVcFuture, ResolveVcFuture, ResolvedVc, ToResolvedVcFuture, Upcast, ResolveOperationVcFuture, ResolveVcFuture, ResolvedVc, ToResolvedVcFuture, Upcast,
......
...@@ -24,6 +24,14 @@ impl SerializationInvalidator { ...@@ -24,6 +24,14 @@ impl SerializationInvalidator {
with_turbo_tasks(|tt| tt.invalidate_serialization(self.task)); with_turbo_tasks(|tt| tt.invalidate_serialization(self.task));
} }
/// The task whose serialized form should be re-emitted on the next
/// snapshot. Exposed so callers that already hold a `TurboTasksApi`
/// reference can fold the serialization-invalidation into a larger
/// batch without paying for another `with_turbo_tasks` lookup.
pub(crate) fn task(&self) -> TaskId {
self.task
}
pub(crate) fn new(task_id: TaskId) -> Self { pub(crate) fn new(task_id: TaskId) -> Self {
Self { task: task_id } Self { task: task_id }
} }
......
use std::{ use std::{
any::type_name,
fmt::Debug, fmt::Debug,
mem::take, mem::take,
ops::{Deref, DerefMut}, ops::{Deref, DerefMut},
...@@ -33,92 +32,128 @@ impl<T> StateInner<T> { ...@@ -33,92 +32,128 @@ impl<T> StateInner<T> {
self.invalidators.insert(invalidator); self.invalidators.insert(invalidator);
} }
pub fn set_unconditionally(&mut self, value: T) { /// Sets the value and returns the drained invalidators. The caller MUST
/// run them via [`run_invalidators`] *after* dropping the [`Mutex`] guard
/// — calling [`Invalidator::invalidate`] may grab locks in the backend which can lead to cycles
#[must_use]
fn set_unconditionally(&mut self, value: T) -> AutoSet<Invalidator> {
self.value = value; self.value = value;
let _span = trace_span!("state value changed", value_type = type_name::<T>()).entered(); take(&mut self.invalidators)
let invalidators = take(&mut self.invalidators);
if !invalidators.is_empty() {
with_turbo_tasks(|tt| {
for invalidator in invalidators {
invalidator.invalidate(&**tt);
}
});
}
} }
pub fn update_conditionally(&mut self, update: impl FnOnce(&mut T) -> bool) -> bool { /// See [`Self::set_unconditionally`] for the locking contract on the
/// returned invalidators.
#[must_use]
fn update_conditionally(
&mut self,
update: impl FnOnce(&mut T) -> bool,
) -> Option<AutoSet<Invalidator>> {
if !update(&mut self.value) { if !update(&mut self.value) {
return false; return None;
}
let _span = trace_span!("state value changed", value_type = type_name::<T>()).entered();
let invalidators = take(&mut self.invalidators);
if !invalidators.is_empty() {
with_turbo_tasks(|tt| {
for invalidator in invalidators {
invalidator.invalidate(&**tt);
}
});
} }
true Some(take(&mut self.invalidators))
} }
} }
impl<T: PartialEq> StateInner<T> { impl<T: PartialEq> StateInner<T> {
pub fn set(&mut self, value: T) -> bool { /// See [`Self::set_unconditionally`] for the locking contract on the
/// returned invalidators.
#[must_use]
fn set(&mut self, value: T) -> Option<AutoSet<Invalidator>> {
if self.value == value { if self.value == value {
return false; return None;
} }
let _span = trace_span!("state value changed", value_type = type_name::<T>()).entered();
self.value = value; self.value = value;
let invalidators = take(&mut self.invalidators); Some(take(&mut self.invalidators))
if !invalidators.is_empty() { }
with_turbo_tasks(|tt| { }
for invalidator in invalidators {
invalidator.invalidate(&**tt); /// Notifies the backend that the [`State`] has been mutated: runs every
} /// dependent [`Invalidator`] and invalidates the serialized state. Must be
}); /// called *outside* the [`StateInner`] mutex guard; see
} /// [`StateInner::set_unconditionally`] for why.
true ///
/// Both notifications resolve `TURBO_TASKS` from a task-local, so we do them
/// inside a single [`with_turbo_tasks`] call to amortize that lookup.
fn notify_mutated(
invalidators: AutoSet<Invalidator>,
serialization_invalidator: Option<&SerializationInvalidator>,
) {
if invalidators.is_empty() && serialization_invalidator.is_none() {
return;
} }
let _span = trace_span!("state value changed").entered();
with_turbo_tasks(|tt| {
for invalidator in invalidators {
invalidator.invalidate(&**tt);
}
if let Some(serialization_invalidator) = serialization_invalidator {
tt.invalidate_serialization(serialization_invalidator.task());
}
});
} }
pub struct StateRef<'a, T> { pub struct StateRef<'a, T> {
serialization_invalidator: Option<&'a SerializationInvalidator>, serialization_invalidator: Option<&'a SerializationInvalidator>,
inner: MutexGuard<'a, StateInner<T>>, // `Option` so `Drop` can `take()` the guard and release it before running
// invalidators. Always `Some` for the lifetime of the `StateRef` outside
// of `Drop`.
inner: Option<MutexGuard<'a, StateInner<T>>>,
mutated: bool, mutated: bool,
} }
impl<'a, T> StateRef<'a, T> {
fn new(
inner: MutexGuard<'a, StateInner<T>>,
serialization_invalidator: Option<&'a SerializationInvalidator>,
) -> Self {
Self {
serialization_invalidator,
inner: Some(inner),
mutated: false,
}
}
fn inner(&self) -> &StateInner<T> {
self.inner.as_deref().expect("inner only None during Drop")
}
fn inner_mut(&mut self) -> &mut StateInner<T> {
self.inner
.as_deref_mut()
.expect("inner only None during Drop")
}
}
impl<T> Deref for StateRef<'_, T> { impl<T> Deref for StateRef<'_, T> {
type Target = T; type Target = T;
fn deref(&self) -> &Self::Target { fn deref(&self) -> &Self::Target {
&self.inner.value &self.inner().value
} }
} }
impl<T> DerefMut for StateRef<'_, T> { impl<T> DerefMut for StateRef<'_, T> {
fn deref_mut(&mut self) -> &mut Self::Target { fn deref_mut(&mut self) -> &mut Self::Target {
self.mutated = true; self.mutated = true;
&mut self.inner.value &mut self.inner_mut().value
} }
} }
impl<T> Drop for StateRef<'_, T> { impl<T> Drop for StateRef<'_, T> {
fn drop(&mut self) { fn drop(&mut self) {
if self.mutated { if !self.mutated {
let _span = trace_span!("state value changed", value_type = type_name::<T>()).entered(); return;
let invalidators = take(&mut self.inner.invalidators);
if !invalidators.is_empty() {
with_turbo_tasks(|tt| {
for invalidator in invalidators {
invalidator.invalidate(&**tt);
}
});
}
if let Some(serialization_invalidator) = self.serialization_invalidator {
serialization_invalidator.invalidate();
}
} }
// Drain invalidators while we still hold the guard, then drop the
// guard before running them. Running invalidators reaches into the
// backend and acquires task-storage shard locks, and the snapshot
// path takes the State mutex while holding such a shard lock — so
// running them under the guard is a lock-order inversion.
let mut guard = self.inner.take().expect("Drop only called once");
let invalidators = take(&mut guard.invalidators);
drop(guard);
notify_mutated(invalidators, self.serialization_invalidator);
} }
} }
...@@ -235,31 +270,23 @@ impl<T> State<T> { ...@@ -235,31 +270,23 @@ impl<T> State<T> {
if let Some(invalidator) = invalidator { if let Some(invalidator) = invalidator {
inner.add_invalidator(invalidator); inner.add_invalidator(invalidator);
} }
StateRef { StateRef::new(inner, Some(&self.serialization_invalidator))
serialization_invalidator: Some(&self.serialization_invalidator),
inner,
mutated: false,
}
} }
/// Gets the current value of the state. Untracked. /// Gets the current value of the state. Untracked.
pub fn get_untracked(&self) -> StateRef<'_, T> { pub fn get_untracked(&self) -> StateRef<'_, T> {
let inner = self.inner.lock(); let inner = self.inner.lock();
StateRef { StateRef::new(inner, Some(&self.serialization_invalidator))
serialization_invalidator: Some(&self.serialization_invalidator),
inner,
mutated: false,
}
} }
/// Sets the current state without comparing it with the old value. This /// Sets the current state without comparing it with the old value. This
/// should only be used if one is sure that the value has changed. /// should only be used if one is sure that the value has changed.
pub fn set_unconditionally(&self, value: T) { pub fn set_unconditionally(&self, value: T) {
{ let invalidators = {
let mut inner = self.inner.lock(); let mut inner = self.inner.lock();
inner.set_unconditionally(value); inner.set_unconditionally(value)
} };
self.serialization_invalidator.invalidate(); notify_mutated(invalidators, Some(&self.serialization_invalidator));
} }
/// Updates the current state with the `update` function. The `update` /// Updates the current state with the `update` function. The `update`
...@@ -267,13 +294,13 @@ impl<T> State<T> { ...@@ -267,13 +294,13 @@ impl<T> State<T> {
/// the current value from the `update` function is not allowed and will /// the current value from the `update` function is not allowed and will
/// result in incorrect cache invalidation. /// result in incorrect cache invalidation.
pub fn update_conditionally(&self, update: impl FnOnce(&mut T) -> bool) { pub fn update_conditionally(&self, update: impl FnOnce(&mut T) -> bool) {
{ let Some(invalidators) = ({
let mut inner = self.inner.lock(); let mut inner = self.inner.lock();
if !inner.update_conditionally(update) { inner.update_conditionally(update)
return; }) else {
} return;
} };
self.serialization_invalidator.invalidate(); notify_mutated(invalidators, Some(&self.serialization_invalidator));
} }
} }
...@@ -281,12 +308,12 @@ impl<T: PartialEq> State<T> { ...@@ -281,12 +308,12 @@ impl<T: PartialEq> State<T> {
/// Update the current state when the `value` is different from the current /// Update the current state when the `value` is different from the current
/// value. `T` must implement [PartialEq] for this to work. /// value. `T` must implement [PartialEq] for this to work.
pub fn set(&self, value: T) { pub fn set(&self, value: T) {
{ let Some(invalidators) = ({
let mut inner = self.inner.lock(); let mut inner = self.inner.lock();
if !inner.set(value) { inner.set(value)
return; }) else {
} return;
} };
self.serialization_invalidator.invalidate(); notify_mutated(invalidators, Some(&self.serialization_invalidator));
} }
} }
...@@ -10,6 +10,7 @@ ...@@ -10,6 +10,7 @@
turbo_tasks_backend::BackendOptions { turbo_tasks_backend::BackendOptions {
num_workers: Some(2), num_workers: Some(2),
small_preallocation: true, small_preallocation: true,
storage_mode: Some(turbo_tasks_backend::StorageMode::ReadWriteOnShutdown),
..Default::default() ..Default::default()
}, },
turbo_tasks_backend::turbo_backing_storage( turbo_tasks_backend::turbo_backing_storage(
......
...@@ -11,6 +11,7 @@ ...@@ -11,6 +11,7 @@
turbo_tasks_backend::BackendOptions { turbo_tasks_backend::BackendOptions {
num_workers: Some(2), num_workers: Some(2),
small_preallocation: true, small_preallocation: true,
storage_mode: Some(turbo_tasks_backend::StorageMode::ReadWriteOnShutdown),
..Default::default() ..Default::default()
}, },
turbo_tasks_backend::turbo_backing_storage( turbo_tasks_backend::turbo_backing_storage(
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment