Unverified Commit 29045997 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

Optimize how we track data for persistence (#89370)

## Summary

Reworks how the turbo-tasks backend tracks modified tasks for persistence snapshots, reducing overhead and simplifying the snapshot lifecycle.

**Key changes:**

- **Replace `modified` DashMap with per-shard atomic counters + inline flags.** Instead of maintaining a separate `FxDashMap<TaskId, ModifiedState>` that mirrors every modification, track modifications via flags already on `TaskStorage` and use per-shard `AtomicU64` counters to skip unmodified shards during snapshot iteration. This eliminates a major source of  memory overhead.
    - The downside here is needing to scan the map.  per shard counters enable early exits but we will still need to scan entire shards.  For a large site that means scanning thousands of

- **Merge task cache writes into the snapshot pipeline.** New tasks now carry a `new_task` flag and their type hash in `SnapshotItem`, so task cache entries are written in the same batch as task data/meta — removing the separate `persisted_task_cache_log` (`Sharded<ChunkedVec<...>>`) and its associated locking.

- **Remove `local_is_partial` optimization.** The backing storage layer already short-circuits on empty databases, and new tasks eagerly set `restored` flags at allocation time, making this redundant.

- **Simplify `end_snapshot`.** Instead of a multi-pass retain/iterate/update cycle over the `modified` map, `end_snapshot` now just drains the small `snapshots` map (only tasks concurrently accessed during snapshot mode) and promotes their `modified_during_snapshot` flags.

- **Delete unused utilities.** Removes `Sharded`, `ChunkedVec` (from backing_storage), and `swap_retain` import now that they're no longer needed.

**Other cleanups:**
- `initialize_new_task` sets restored + new_task flags at allocation time for both persistent and transient tasks
- Fuzz test updated to use `active_tracking: true` and `StorageMode::ReadWrite`
- New KV storage test for batch write+flush+reopen pattern
- Minor fix: `SmallVec::into_boxed_slice()` instead of `into_vec().into_boxed_slice()`


## Build Benchmark Results

  Measured over 9 runs (1 warm-up discarded), macOS, `TURBOPACK_PERSISTENT_CACHE=1`.

  ### Cold build (`rm -rf .next/`)

  | | Time (avg) | Time (stddev) | MaxRSS (avg) |
  |---|---|---|---|
  | HEAD | 75.48s | 0.72s | 24,231 MiB |
  | This PR | 75.20s | 1.98s | 23,829 MiB |
  | **Delta** | −0.28s | — | **−402 MiB (−1.7%)** |

  ### Warm build (single file edit)

  | | Time (avg) | MaxRSS (avg) |
  |---|---|---|
  | HEAD | 23.79s | 8,764 MiB |
  | This PR | 23.67s | 8,766 MiB |
  | **Delta** | −0.12s | flat |

  Cold build time difference is within noise (< 1 stddev). The meaningful improvement is a **~400 MiB reduction in peak memory on cold builds**, consistent with the fixed overhead this PR targets. Warm builds are unaffected as expected.
parent 18430c7f
...@@ -17,14 +17,10 @@ const nextConfig = { ...@@ -17,14 +17,10 @@ const nextConfig = {
}, },
}, },
}, },
experimental: enableCaching experimental: {
? { turbopackFileSystemCacheForBuild: enableCaching,
turbopackFileSystemCacheForBuild: true, turbopackFileSystemCacheForDev: enableCaching,
} },
: {
turbopackFileSystemCacheForDev: false,
turbopackFileSystemCacheForBuild: false,
},
env: { env: {
NEXT_PUBLIC_CONFIG_ENV: 'hello world', NEXT_PUBLIC_CONFIG_ENV: 'hello world',
}, },
......
...@@ -23,11 +23,9 @@ const runDev = async (config: any, shouldWaitForReady = true) => { ...@@ -23,11 +23,9 @@ const runDev = async (config: any, shouldWaitForReady = true) => {
const port = await findPort() const port = await findPort()
const obj = { port, stdout: '', stderr: '' } const obj = { port, stdout: '', stderr: '' }
app = await launchApp(appDir, port, { app = await launchApp(appDir, port, {
stdout: false,
onStdout(msg: string) { onStdout(msg: string) {
obj.stdout += msg || '' obj.stdout += msg || ''
}, },
stderr: false,
onStderr(msg: string) { onStderr(msg: string) {
obj.stderr += msg || '' obj.stderr += msg || ''
}, },
......
...@@ -13,7 +13,7 @@ impl ValueBuffer<'_> { ...@@ -13,7 +13,7 @@ impl ValueBuffer<'_> {
match self { match self {
ValueBuffer::Borrowed(b) => b.into(), ValueBuffer::Borrowed(b) => b.into(),
ValueBuffer::Vec(v) => v.into_boxed_slice(), ValueBuffer::Vec(v) => v.into_boxed_slice(),
ValueBuffer::SmallVec(sv) => sv.into_vec().into_boxed_slice(), ValueBuffer::SmallVec(sv) => sv.into_boxed_slice(),
} }
} }
} }
......
...@@ -16,7 +16,7 @@ libfuzzer-sys = "0.4.9" ...@@ -16,7 +16,7 @@ libfuzzer-sys = "0.4.9"
once_cell = { workspace = true } once_cell = { workspace = true }
serde = { workspace = true } serde = { workspace = true }
tokio = { workspace = true, features = ["full"] } tokio = { workspace = true, features = ["full"] }
turbo-tasks = { workspace = true } turbo-tasks = { workspace = true, features = ["non_operation_vc_strongly_consistent"] }
turbo-tasks-backend = { workspace = true } turbo-tasks-backend = { workspace = true }
turbo-tasks-malloc = { workspace = true } turbo-tasks-malloc = { workspace = true }
......
...@@ -133,16 +133,17 @@ pub fn run(data: Vec<TaskSpec>) { ...@@ -133,16 +133,17 @@ pub fn run(data: Vec<TaskSpec>) {
struct Iteration(State<usize>); struct Iteration(State<usize>);
fn actual_operation(spec: Arc<Vec<TaskSpec>>, iterations: usize) { fn actual_operation(spec: Arc<Vec<TaskSpec>>, iterations: usize) {
let tt = TurboTasks::new(turbo_tasks_backend::TurboTasksBackend::new(
turbo_tasks_backend::BackendOptions {
storage_mode: None,
small_preallocation: true,
..Default::default()
},
turbo_tasks_backend::noop_backing_storage(),
));
RUNTIME RUNTIME
.block_on(async { .block_on(async {
let tt = TurboTasks::new(turbo_tasks_backend::TurboTasksBackend::new(
turbo_tasks_backend::BackendOptions {
storage_mode: Some(turbo_tasks_backend::StorageMode::ReadWrite),
small_preallocation: false,
active_tracking: true,
..Default::default()
},
turbo_tasks_backend::noop_backing_storage(),
));
for i in 0..iterations { for i in 0..iterations {
let spec = spec.clone(); let spec = spec.clone();
tt.run(async move { tt.run(async move {
......
...@@ -63,7 +63,7 @@ use crate::{ ...@@ -63,7 +63,7 @@ use crate::{
storage::Storage, storage::Storage,
storage_schema::{TaskStorage, TaskStorageAccessors}, storage_schema::{TaskStorage, TaskStorageAccessors},
}, },
backing_storage::{BackingStorage, SnapshotItem}, backing_storage::{BackingStorage, SnapshotItem, compute_task_type_hash},
data::{ data::{
ActivenessState, CellRef, CollectibleRef, CollectiblesRef, Dirtyness, InProgressCellState, ActivenessState, CellRef, CollectibleRef, CollectiblesRef, Dirtyness, InProgressCellState,
InProgressState, InProgressStateInner, OutputValue, TransientTask, InProgressState, InProgressStateInner, OutputValue, TransientTask,
...@@ -71,13 +71,10 @@ use crate::{ ...@@ -71,13 +71,10 @@ use crate::{
error::TaskError, error::TaskError,
utils::{ utils::{
arc_or_owned::ArcOrOwned, arc_or_owned::ArcOrOwned,
chunked_vec::ChunkedVec,
dash_map_drop_contents::drop_contents, dash_map_drop_contents::drop_contents,
dash_map_raw_entry::{RawEntry, raw_entry}, dash_map_raw_entry::{RawEntry, raw_entry},
ptr_eq_arc::PtrEqArc, ptr_eq_arc::PtrEqArc,
shard_amount::compute_shard_amount, shard_amount::compute_shard_amount,
sharded::Sharded,
swap_retain,
}, },
}; };
...@@ -167,8 +164,6 @@ pub enum TurboTasksBackendJob { ...@@ -167,8 +164,6 @@ pub enum TurboTasksBackendJob {
pub struct TurboTasksBackend<B: BackingStorage>(Arc<TurboTasksBackendInner<B>>); pub struct TurboTasksBackend<B: BackingStorage>(Arc<TurboTasksBackendInner<B>>);
type TaskCacheLog = Sharded<ChunkedVec<(Arc<CachedTaskType>, TaskId)>>;
struct TurboTasksBackendInner<B: BackingStorage> { struct TurboTasksBackendInner<B: BackingStorage> {
options: BackendOptions, options: BackendOptions,
...@@ -177,15 +172,10 @@ struct TurboTasksBackendInner<B: BackingStorage> { ...@@ -177,15 +172,10 @@ struct TurboTasksBackendInner<B: BackingStorage> {
persisted_task_id_factory: IdFactoryWithReuse<TaskId>, persisted_task_id_factory: IdFactoryWithReuse<TaskId>,
transient_task_id_factory: IdFactoryWithReuse<TaskId>, transient_task_id_factory: IdFactoryWithReuse<TaskId>,
persisted_task_cache_log: Option<TaskCacheLog>,
task_cache: FxDashMap<Arc<CachedTaskType>, TaskId>, task_cache: FxDashMap<Arc<CachedTaskType>, TaskId>,
storage: Storage, storage: Storage,
/// When true, the backing_storage has data that is not in the local storage.
/// This is determined once at startup and never changes.
local_is_partial: bool,
/// Number of executing operations + Highest bit is set when snapshot is /// Number of executing operations + Highest bit is set when snapshot is
/// requested. When that bit is set, operations should pause until the /// requested. When that bit is set, operations should pause until the
/// snapshot is completed. When the bit is set and in progress counter /// snapshot is completed. When the bit is set and in progress counter
...@@ -235,10 +225,6 @@ impl<B: BackingStorage> TurboTasksBackend<B> { ...@@ -235,10 +225,6 @@ impl<B: BackingStorage> TurboTasksBackend<B> {
impl<B: BackingStorage> TurboTasksBackendInner<B> { impl<B: BackingStorage> TurboTasksBackendInner<B> {
pub fn new(mut options: BackendOptions, backing_storage: B) -> Self { pub fn new(mut options: BackendOptions, backing_storage: B) -> Self {
let shard_amount = compute_shard_amount(options.num_workers, options.small_preallocation); let shard_amount = compute_shard_amount(options.num_workers, options.small_preallocation);
let need_log = matches!(
options.storage_mode,
Some(StorageMode::ReadWrite) | Some(StorageMode::ReadWriteOnShutdown)
);
if !options.dependency_tracking { if !options.dependency_tracking {
options.active_tracking = false; options.active_tracking = false;
} }
...@@ -257,9 +243,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> { ...@@ -257,9 +243,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
TaskId::try_from(TRANSIENT_TASK_BIT).unwrap(), TaskId::try_from(TRANSIENT_TASK_BIT).unwrap(),
TaskId::MAX, TaskId::MAX,
), ),
persisted_task_cache_log: need_log.then(|| Sharded::new(shard_amount)),
task_cache: FxDashMap::default(), task_cache: FxDashMap::default(),
local_is_partial: next_task_id != TaskId::MIN,
storage: Storage::new(shard_amount, small_preallocation), storage: Storage::new(shard_amount, small_preallocation),
in_progress_operations: AtomicUsize::new(0), in_progress_operations: AtomicUsize::new(0),
snapshot_request: Mutex::new(SnapshotRequest::new()), snapshot_request: Mutex::new(SnapshotRequest::new()),
...@@ -1036,12 +1020,9 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> { ...@@ -1036,12 +1020,9 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
.map(|op| op.arc().clone()) .map(|op| op.arc().clone())
.collect::<Vec<_>>(); .collect::<Vec<_>>();
} }
self.storage.start_snapshot(); // Enter snapshot mode, which atomically reads and resets the modified count.
let mut persisted_task_cache_log = self // Checking after start_snapshot ensures no concurrent increments can race.
.persisted_task_cache_log let (snapshot_guard, has_modifications) = self.storage.start_snapshot();
.as_ref()
.map(|l| l.take(|i| i))
.unwrap_or_default();
let mut snapshot_request = self.snapshot_request.lock(); let mut snapshot_request = self.snapshot_request.lock();
snapshot_request.snapshot_requested = false; snapshot_request.snapshot_requested = false;
self.in_progress_operations self.in_progress_operations
...@@ -1050,6 +1031,13 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> { ...@@ -1050,6 +1031,13 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
let snapshot_time = Instant::now(); let snapshot_time = Instant::now();
drop(snapshot_request); drop(snapshot_request);
if !has_modifications {
// No tasks modified since the last snapshot — drop the guard (which
// calls end_snapshot) and skip the expensive O(N) scan.
drop(snapshot_guard);
return Some((start, false));
}
#[cfg(feature = "print_cache_item_size")] #[cfg(feature = "print_cache_item_size")]
#[derive(Default)] #[derive(Default)]
struct TaskCacheStats { struct TaskCacheStats {
...@@ -1278,35 +1266,49 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> { ...@@ -1278,35 +1266,49 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
} else { } else {
None None
}; };
let task_type_hash = if inner.flags.new_task() {
let task_type = inner.get_persistent_task_type().expect(
"It is not possible for a new_task to not have a persistent_task_type. Task \
creation for persistent tasks uses a single ExecutionContextImpl for \
creating the task (which sets new_task) and connect_child (which sets \
persistent_task_type) and take_snapshot waits for all operations to complete \
or suspend before we start snapshotting. So task creation will always set \
the task_type.",
);
Some(compute_task_type_hash(task_type))
} else {
None
};
SnapshotItem { SnapshotItem {
task_id, task_id,
meta, meta,
data, data,
task_type_hash,
} }
}; };
// take_snapshot already filters empty items and empty shards in parallel // take_snapshot already filters empty items and empty shards in parallel
let task_snapshots = self.storage.take_snapshot(&process); let task_snapshots = self.storage.take_snapshot(snapshot_guard, &process);
swap_retain(&mut persisted_task_cache_log, |shard| !shard.is_empty());
drop(snapshot_span); drop(snapshot_span);
let snapshot_duration = start.elapsed(); let snapshot_duration = start.elapsed();
let task_count = task_snapshots.len(); let task_count = task_snapshots.len();
if persisted_task_cache_log.is_empty() && task_snapshots.is_empty() { if task_snapshots.is_empty() {
// This should be impossible — if we got here, modified_count was nonzero, and every
// modification that increments the count also failed during encoding.
std::hint::cold_path();
return Some((snapshot_time, false)); return Some((snapshot_time, false));
} }
let persist_start = Instant::now(); let persist_start = Instant::now();
let _span = tracing::info_span!(parent: parent_span, "persist", reason = reason).entered(); let _span = tracing::info_span!(parent: parent_span, "persist", reason = reason).entered();
{ {
if let Err(err) = self.backing_storage.save_snapshot( if let Err(err) = self
suspended_operations, .backing_storage
persisted_task_cache_log, .save_snapshot(suspended_operations, task_snapshots)
task_snapshots, {
) {
eprintln!("Persisting failed: {err:?}"); eprintln!("Persisting failed: {err:?}");
return None; return None;
} }
...@@ -1551,7 +1553,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> { ...@@ -1551,7 +1553,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
let (task_id, task_type) = match raw_entry(&self.task_cache, &task_type) { let (task_id, task_type) = match raw_entry(&self.task_cache, &task_type) {
RawEntry::Occupied(e) => { RawEntry::Occupied(e) => {
// Another thread beat us to creating this task - use their task_id. // Another thread beat us to creating this task - use their task_id.
// They will handle logging to persisted_task_cache_log. // They will handle logging the new task as modified
let task_id = *e.get(); let task_id = *e.get();
drop(e); drop(e);
self.track_cache_hit(&task_type); self.track_cache_hit(&task_type);
...@@ -1562,12 +1564,12 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> { ...@@ -1562,12 +1564,12 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
let task_type = Arc::new(task_type); let task_type = Arc::new(task_type);
let task_id = self.persisted_task_id_factory.get(); let task_id = self.persisted_task_id_factory.get();
e.insert(task_type.clone(), task_id); e.insert(task_type.clone(), task_id);
// Mark the task as new in storage.
// Do this after e.insert so we aren't holding the task_cache lock
self.storage.initialize_new_task(task_id);
// insert() consumes e, releasing the lock // insert() consumes e, releasing the lock
self.track_cache_miss(&task_type); self.track_cache_miss(&task_type);
is_new = true; is_new = true;
if let Some(log) = &self.persisted_task_cache_log {
log.lock(task_id).push((task_type.clone(), task_id));
}
(task_id, ArcOrOwned::Arc(task_type)) (task_id, ArcOrOwned::Arc(task_type))
} }
}; };
...@@ -1637,6 +1639,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> { ...@@ -1637,6 +1639,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
let task_type = Arc::new(task_type); let task_type = Arc::new(task_type);
let task_id = self.transient_task_id_factory.get(); let task_id = self.transient_task_id_factory.get();
e.insert(task_type.clone(), task_id); e.insert(task_type.clone(), task_id);
self.storage.initialize_new_task(task_id);
self.track_cache_miss(&task_type); self.track_cache_miss(&task_type);
if is_root { if is_root {
......
...@@ -1896,7 +1896,7 @@ impl AggregationUpdateQueue { ...@@ -1896,7 +1896,7 @@ impl AggregationUpdateQueue {
/// See detailed comments in that function, follow the STEP numbers. /// See detailed comments in that function, follow the STEP numbers.
fn inner_of_uppers_has_new_follower<T: TaskIdWithOptionalCount + Clone, const N: usize>( fn inner_of_uppers_has_new_follower<T: TaskIdWithOptionalCount + Clone, const N: usize>(
&mut self, &mut self,
ctx: &mut impl ExecuteContext, ctx: &mut impl ExecuteContext<'_>,
new_follower_id: TaskId, new_follower_id: TaskId,
mut upper_ids: SmallVec<[T; N]>, mut upper_ids: SmallVec<[T; N]>,
) { ) {
...@@ -2618,7 +2618,7 @@ impl AggregationUpdateQueue { ...@@ -2618,7 +2618,7 @@ impl AggregationUpdateQueue {
/// Only used when activeness is tracked. /// Only used when activeness is tracked.
fn increase_active_count( fn increase_active_count(
&mut self, &mut self,
ctx: &mut impl ExecuteContext, ctx: &mut impl ExecuteContext<'_>,
task_id: TaskId, task_id: TaskId,
task_type: Option<Arc<CachedTaskType>>, task_type: Option<Arc<CachedTaskType>>,
) { ) {
...@@ -2637,7 +2637,7 @@ impl AggregationUpdateQueue { ...@@ -2637,7 +2637,7 @@ impl AggregationUpdateQueue {
if let Some(task_type) = task_type if let Some(task_type) = task_type
&& !task.has_persistent_task_type() && !task.has_persistent_task_type()
{ {
let _ = task.set_persistent_task_type(task_type); task.set_persistent_task_type(task_type);
} }
let state = task.get_activeness_mut_or_insert_with(|| ActivenessState::new(task_id)); let state = task.get_activeness_mut_or_insert_with(|| ActivenessState::new(task_id));
let is_new = state.is_empty(); let is_new = state.is_empty();
......
...@@ -165,20 +165,15 @@ impl<'e, B: BackingStorage> ExecuteContextImpl<'e, B> { ...@@ -165,20 +165,15 @@ impl<'e, B: BackingStorage> ExecuteContextImpl<'e, B> {
} }
} }
fn should_check_backing_storage(&self) -> bool {
self.backend.should_restore() && self.backend.local_is_partial
}
fn restore_task_data( fn restore_task_data(
&self, &self,
task_id: TaskId, task_id: TaskId,
category: SpecificTaskDataCategory, category: SpecificTaskDataCategory,
) -> TaskStorage { ) -> TaskStorage {
if !self.should_check_backing_storage() {
// If we don't need to restore, we can just return an empty storage
return TaskStorage::default();
}
let mut storage = TaskStorage::default(); let mut storage = TaskStorage::default();
if !self.backend.should_restore() {
return storage;
}
let result = self let result = self
.backend .backend
.backing_storage .backing_storage
...@@ -204,8 +199,7 @@ impl<'e, B: BackingStorage> ExecuteContextImpl<'e, B> { ...@@ -204,8 +199,7 @@ impl<'e, B: BackingStorage> ExecuteContextImpl<'e, B> {
task_ids.len() > 1, task_ids.len() > 1,
"Use restore_task_data_typed for single task" "Use restore_task_data_typed for single task"
); );
if !self.should_check_backing_storage() { if !self.backend.should_restore() {
// If we don't need to restore, we return None
return None; return None;
} }
let result = self let result = self
...@@ -244,13 +238,10 @@ impl<'e, B: BackingStorage> ExecuteContextImpl<'e, B> { ...@@ -244,13 +238,10 @@ impl<'e, B: BackingStorage> ExecuteContextImpl<'e, B> {
.into_iter() .into_iter()
.filter(|&(id, category)| { .filter(|&(id, category)| {
if id.is_transient() { if id.is_transient() {
// Transient tasks don't need DB restoration (restored flags are
// set at allocation time), so just invoke the callback directly.
if call_prepared_task_callback_for_transient_tasks { if call_prepared_task_callback_for_transient_tasks {
let mut task = self.backend.storage.access_mut(id); let task = self.backend.storage.access_mut(id);
// TODO add is_restoring and avoid concurrent restores and duplicates tasks
// ids in `task_ids`
if !task.flags.is_restored(category) {
task.flags.set_restored(TaskDataCategory::All);
}
prepared_task_callback(self, id, category, task); prepared_task_callback(self, id, category, task);
} }
false false
...@@ -400,40 +391,43 @@ impl<'e, B: BackingStorage> ExecuteContext<'e> for ExecuteContextImpl<'e, B> { ...@@ -400,40 +391,43 @@ impl<'e, B: BackingStorage> ExecuteContext<'e> for ExecuteContextImpl<'e, B> {
let mut task = self.backend.storage.access_mut(task_id); let mut task = self.backend.storage.access_mut(task_id);
if !task.flags.is_restored(category) { if !task.flags.is_restored(category) {
if task_id.is_transient() { // New tasks (transient and persistent) have restored flags set at allocation
task.flags.set_restored(TaskDataCategory::All); // time, so this path is only hit for persistent tasks being restored from DB.
} else { debug_assert!(
// Collect which categories need restoring while we have the lock !task.flags.new_task() && !task_id.is_transient(),
let needs_data = "new or transient task should already be marked restored"
category.includes_data() && !task.flags.is_restored(TaskDataCategory::Data); );
let needs_meta =
category.includes_meta() && !task.flags.is_restored(TaskDataCategory::Meta); // Collect which categories need restoring while we have the lock
let needs_data =
if needs_data || needs_meta { category.includes_data() && !task.flags.is_restored(TaskDataCategory::Data);
// Avoid holding the lock too long since this can also affect other tasks let needs_meta =
// Drop lock once, do all I/O, then re-acquire once category.includes_meta() && !task.flags.is_restored(TaskDataCategory::Meta);
drop(task);
if needs_data || needs_meta {
let storage_data = needs_data // Avoid holding the lock too long since this can also affect other tasks
.then(|| self.restore_task_data(task_id, SpecificTaskDataCategory::Data)); // Drop lock once, do all I/O, then re-acquire once
let storage_meta = needs_meta drop(task);
.then(|| self.restore_task_data(task_id, SpecificTaskDataCategory::Meta));
let storage_data = needs_data
task = self.backend.storage.access_mut(task_id); .then(|| self.restore_task_data(task_id, SpecificTaskDataCategory::Data));
let storage_meta = needs_meta
// Handle race conditions and merge .then(|| self.restore_task_data(task_id, SpecificTaskDataCategory::Meta));
if let Some(storage) = storage_data
&& !task.flags.is_restored(TaskDataCategory::Data) task = self.backend.storage.access_mut(task_id);
{
task.restore_from(storage, TaskDataCategory::Data); // Handle race conditions and merge
task.flags.set_restored(TaskDataCategory::Data); if let Some(storage) = storage_data
} && !task.flags.is_restored(TaskDataCategory::Data)
if let Some(storage) = storage_meta {
&& !task.flags.is_restored(TaskDataCategory::Meta) task.restore_from(storage, TaskDataCategory::Data);
{ task.flags.set_restored(TaskDataCategory::Data);
task.restore_from(storage, TaskDataCategory::Meta); }
task.flags.set_restored(TaskDataCategory::Meta); if let Some(storage) = storage_meta
} && !task.flags.is_restored(TaskDataCategory::Meta)
{
task.restore_from(storage, TaskDataCategory::Meta);
task.flags.set_restored(TaskDataCategory::Meta);
} }
} }
} }
...@@ -600,7 +594,7 @@ impl<'e, B: BackingStorage> ExecuteContext<'e> for ExecuteContextImpl<'e, B> { ...@@ -600,7 +594,7 @@ impl<'e, B: BackingStorage> ExecuteContext<'e> for ExecuteContextImpl<'e, B> {
} }
fn task_by_type(&mut self, task_type: &CachedTaskType) -> Option<TaskId> { fn task_by_type(&mut self, task_type: &CachedTaskType) -> Option<TaskId> {
if !self.should_check_backing_storage() { if !self.backend.should_restore() {
return None; return None;
} }
......
...@@ -187,6 +187,11 @@ struct TaskStorageSchema { ...@@ -187,6 +187,11 @@ struct TaskStorageSchema {
#[field(storage = "flag", category = "transient")] #[field(storage = "flag", category = "transient")]
stateful: bool, stateful: bool,
/// Whether this task is new and needs its type persisted to the task cache.
/// Set when task is created, cleared after persisting.
#[field(storage = "flag", category = "transient")]
pub new_task: bool,
// ========================================================================= // =========================================================================
// CHILDREN & AGGREGATION (meta) // CHILDREN & AGGREGATION (meta)
// ========================================================================= // =========================================================================
...@@ -548,7 +553,8 @@ impl TaskStorage { ...@@ -548,7 +553,8 @@ impl TaskStorage {
task_type: TransientTaskType, task_type: TransientTaskType,
should_track_activeness: bool, should_track_activeness: bool,
) { ) {
// Mark as fully restored since transient tasks don't need restoration from disk // Mark as fully restored since transient tasks don't need restoration from disk,
// and as new since this task was just created.
self.flags.set_restored(TaskDataCategory::All); self.flags.set_restored(TaskDataCategory::All);
// This is a root (or once) task. These tasks use the max aggregation number. // This is a root (or once) task. These tasks use the max aggregation number.
......
...@@ -5,24 +5,50 @@ use either::Either; ...@@ -5,24 +5,50 @@ use either::Either;
use smallvec::SmallVec; use smallvec::SmallVec;
use turbo_bincode::TurboBincodeBuffer; use turbo_bincode::TurboBincodeBuffer;
use turbo_tasks::{TaskId, backend::CachedTaskType}; use turbo_tasks::{TaskId, backend::CachedTaskType};
use turbo_tasks_hash::Xxh3Hash64Hasher;
use crate::{ use crate::backend::{AnyOperation, SpecificTaskDataCategory, storage_schema::TaskStorage};
backend::{AnyOperation, SpecificTaskDataCategory, storage_schema::TaskStorage},
utils::chunked_vec::ChunkedVec,
};
pub type TaskTypeHash = [u8; 8];
/// A single item yielded by the snapshot iterator during persistence.
pub struct SnapshotItem { pub struct SnapshotItem {
pub task_id: TaskId, pub task_id: TaskId,
pub data: Option<TurboBincodeBuffer>, /// Serialized task meta data, if modified
pub meta: Option<TurboBincodeBuffer>, pub meta: Option<TurboBincodeBuffer>,
/// Serialized task data, if modified
pub data: Option<TurboBincodeBuffer>,
/// Task type for new tasks that need to be added to the task cache
pub task_type_hash: Option<TaskTypeHash>,
} }
impl SnapshotItem { impl SnapshotItem {
pub fn is_empty(&self) -> bool { pub fn is_empty(&self) -> bool {
self.meta.is_none() && self.data.is_none() self.meta.is_none() && self.data.is_none() && self.task_type_hash.is_none()
} }
} }
/// Computes a deterministic 64-bit hash of a CachedTaskType for use as a TaskCache key.
///
/// This encodes the task type directly to a hasher, avoiding intermediate buffer allocation.
/// The encoding is deterministic (function IDs from registry, bincode argument encoding).
pub fn compute_task_type_hash(task_type: &CachedTaskType) -> TaskTypeHash {
let mut hasher = Xxh3Hash64Hasher::new();
task_type.hash_encode(&mut hasher);
let hash = hasher.finish();
if cfg!(feature = "verify_serialization") {
hasher = Xxh3Hash64Hasher::new();
task_type.hash_encode(&mut hasher);
let hash2 = hasher.finish();
assert_eq!(
hash, hash2,
"Hashing TaskType twice was non-deterministic: \n{:?}\ngot hashes {} != {}",
task_type, hash, hash2
);
}
hash.to_le_bytes()
}
/// Represents types accepted by [`TurboTasksBackend::new`]. Typically this is the value returned by /// Represents types accepted by [`TurboTasksBackend::new`]. Typically this is the value returned by
/// [`turbo_backing_storage`] or [`noop_backing_storage`]. /// [`turbo_backing_storage`] or [`noop_backing_storage`].
/// ///
...@@ -55,12 +81,7 @@ pub trait BackingStorageSealed: 'static + Send + Sync { ...@@ -55,12 +81,7 @@ pub trait BackingStorageSealed: 'static + Send + Sync {
fn next_free_task_id(&self) -> Result<TaskId>; fn next_free_task_id(&self) -> Result<TaskId>;
fn uncompleted_operations(&self) -> Result<Vec<AnyOperation>>; fn uncompleted_operations(&self) -> Result<Vec<AnyOperation>>;
fn save_snapshot<I>( fn save_snapshot<I>(&self, operations: Vec<Arc<AnyOperation>>, snapshots: Vec<I>) -> Result<()>
&self,
operations: Vec<Arc<AnyOperation>>,
task_cache_updates: Vec<ChunkedVec<(Arc<CachedTaskType>, TaskId)>>,
snapshots: Vec<I>,
) -> Result<()>
where where
I: IntoIterator<Item = SnapshotItem> + Send + Sync; I: IntoIterator<Item = SnapshotItem> + Send + Sync;
/// Returns all task IDs that match the given task type (hash collision candidates). /// Returns all task IDs that match the given task type (hash collision candidates).
...@@ -118,18 +139,12 @@ where ...@@ -118,18 +139,12 @@ where
either::for_both!(self, this => this.uncompleted_operations()) either::for_both!(self, this => this.uncompleted_operations())
} }
fn save_snapshot<I>( fn save_snapshot<I>(&self, operations: Vec<Arc<AnyOperation>>, snapshots: Vec<I>) -> Result<()>
&self,
operations: Vec<Arc<AnyOperation>>,
task_cache_updates: Vec<ChunkedVec<(Arc<CachedTaskType>, TaskId)>>,
snapshots: Vec<I>,
) -> Result<()>
where where
I: IntoIterator<Item = SnapshotItem> + Send + Sync, I: IntoIterator<Item = SnapshotItem> + Send + Sync,
{ {
either::for_both!(self, this => this.save_snapshot( either::for_both!(self, this => this.save_snapshot(
operations, operations,
task_cache_updates,
snapshots, snapshots,
)) ))
} }
......
pub struct ChunkedVec<T> {
chunks: Vec<Vec<T>>,
}
impl<T> Default for ChunkedVec<T> {
fn default() -> Self {
Self::new()
}
}
impl<T> ChunkedVec<T> {
pub fn new() -> Self {
Self { chunks: Vec::new() }
}
pub fn len(&self) -> usize {
for (i, chunk) in self.chunks.iter().enumerate().rev() {
if !chunk.is_empty() {
let free = chunk.capacity() - chunk.len();
return cumulative_chunk_size(i) - free;
}
}
0
}
pub fn push(&mut self, item: T) {
if let Some(chunk) = self.chunks.last_mut()
&& chunk.len() < chunk.capacity()
{
chunk.push(item);
return;
}
let mut chunk = Vec::with_capacity(chunk_size(self.chunks.len()));
chunk.push(item);
self.chunks.push(chunk);
}
pub fn extend<I: IntoIterator<Item = T>>(&mut self, iter: I) {
for item in iter {
self.push(item);
}
}
pub fn iter(&self) -> impl ExactSizeIterator<Item = &T> {
ExactSizeIter {
iter: self.chunks.iter().flat_map(|chunk| chunk.iter()),
len: self.len(),
}
}
pub fn is_empty(&self) -> bool {
self.chunks.first().is_none_or(|chunk| chunk.is_empty())
}
}
impl<T> IntoIterator for ChunkedVec<T> {
type Item = T;
type IntoIter = ExactSizeIter<std::iter::Flatten<std::vec::IntoIter<Vec<T>>>>;
fn into_iter(self) -> Self::IntoIter {
let len = self.len();
ExactSizeIter {
iter: self.chunks.into_iter().flatten(),
len,
}
}
}
impl<T> Extend<T> for ChunkedVec<T> {
fn extend<I: IntoIterator<Item = T>>(&mut self, iter: I) {
for item in iter {
self.push(item);
}
}
}
fn chunk_size(chunk_index: usize) -> usize {
8 << chunk_index
}
fn cumulative_chunk_size(chunk_index: usize) -> usize {
(8 << (chunk_index + 1)) - 8
}
pub struct ExactSizeIter<I: Iterator> {
iter: I,
len: usize,
}
impl<I: Iterator> Iterator for ExactSizeIter<I> {
type Item = I::Item;
fn next(&mut self) -> Option<Self::Item> {
self.iter.next().inspect(|_| self.len -= 1)
}
fn size_hint(&self) -> (usize, Option<usize>) {
(self.len, Some(self.len))
}
}
impl<I: Iterator> ExactSizeIterator for ExactSizeIter<I> {
fn len(&self) -> usize {
self.len
}
}
#[cfg(test)]
mod tests {
use super::ChunkedVec;
#[test]
fn test_chunked_vec() {
for i in 0..1000 {
let mut vec = ChunkedVec::new();
for j in 0..i {
vec.push(j);
}
assert_eq!(vec.len(), i);
assert_eq!(
vec.iter().copied().collect::<Vec<_>>(),
(0..i).collect::<Vec<_>>()
);
assert_eq!(vec.iter().len(), i);
assert_eq!(vec.is_empty(), i == 0);
let iter = vec.into_iter();
assert_eq!(iter.len(), i);
assert_eq!(iter.collect::<Vec<_>>(), (0..i).collect::<Vec<_>>());
}
}
}
pub mod arc_or_owned; pub mod arc_or_owned;
pub mod chunked_vec;
pub mod dash_map_drop_contents; pub mod dash_map_drop_contents;
pub mod dash_map_multi; pub mod dash_map_multi;
pub mod dash_map_raw_entry; pub mod dash_map_raw_entry;
pub mod markdown_table; pub mod markdown_table;
pub mod ptr_eq_arc; pub mod ptr_eq_arc;
pub mod shard_amount; pub mod shard_amount;
pub mod sharded;
pub mod swap_retain; pub mod swap_retain;
pub use swap_retain::swap_retain; pub use swap_retain::swap_retain;
...@@ -11,7 +11,9 @@ use std::thread::available_parallelism; ...@@ -11,7 +11,9 @@ use std::thread::available_parallelism;
/// The returned number is always a power of two as this is often required /// The returned number is always a power of two as this is often required
/// by sharded data structures. The maximum shard amount is capped at 1 << 16 (65536). /// by sharded data structures. The maximum shard amount is capped at 1 << 16 (65536).
pub fn compute_shard_amount(num_workers: Option<usize>, small_preallocation: bool) -> usize { pub fn compute_shard_amount(num_workers: Option<usize>, small_preallocation: bool) -> usize {
let num_workers = num_workers.unwrap_or_else(|| available_parallelism().map_or(4, |v| v.get())); let num_workers = num_workers
.unwrap_or_else(|| available_parallelism().map_or(4, |v| v.get()))
.max(4);
// Once can compute the probability of a shard collision (which leads to false sharing) using // Once can compute the probability of a shard collision (which leads to false sharing) using
// the birthday paradox formula. It's notable that the probability of collisions increases // the birthday paradox formula. It's notable that the probability of collisions increases
......
use std::hash::{BuildHasher, BuildHasherDefault, Hash};
use parking_lot::{Mutex, MutexGuard};
use rustc_hash::FxHasher;
pub struct Sharded<T, H = BuildHasherDefault<FxHasher>> {
data: Box<[Mutex<T>]>,
hasher: H,
bitmask: u16,
}
impl<T, H> Sharded<T, H> {
pub fn new(mut shard_amount: usize) -> Self
where
T: Default,
H: Default,
{
assert!(shard_amount.is_power_of_two());
shard_amount = shard_amount.min(u16::MAX as usize + 1);
let data = (0..shard_amount)
.map(|_| Mutex::new(T::default()))
.collect::<Vec<_>>()
.into_boxed_slice();
Self {
data,
hasher: H::default(),
bitmask: (shard_amount - 1) as u16,
}
}
pub fn lock<K>(&self, key: K) -> MutexGuard<'_, T>
where
K: Hash,
H: BuildHasher,
{
let hash = self.hasher.hash_one(key);
let shard = hash as u16 & self.bitmask;
self.data[shard as usize].lock()
}
pub fn take<R>(&self, map: impl Fn(T) -> R) -> Vec<R>
where
T: Default,
{
let locked = self.data.iter().map(|m| m.lock()).collect::<Vec<_>>();
locked
.into_iter()
.map(|mut m| map(std::mem::take(&mut *m)))
.collect()
}
}
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment