Unverified Commit 2800d2a7 authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

turbo-tasks-backend: prevent duplicate task restores with restoring bits (#92389)

### What?

Adds `data_restoring` and `meta_restoring` transient flag bits to
`TaskStorage`, a shared `restored: Event` on `Storage`, and a
synchronous `EventListener::wait()` primitive. These are used to
coordinate concurrent restore attempts so the same task is never loaded
from the backing store more than once simultaneously.

### Why?

When multiple threads race to access an unrestored task, each thread
would independently call into the backing storage to load the task's
data. This causes redundant I/O and can result in one thread's write
being silently discarded when another thread also applies a restore for
the same task. The fix ensures only one thread performs the actual I/O
while others wait for it to finish.

### How?

**New flag bits** (`storage_schema.rs`):
`meta_restoring` and `data_restoring` are transient (not persisted) bits
that track whether a restore is currently in progress for each category
of a task. `meta_restored` and `data_restored` are also transient bits
that indicate when a category has been fully restored.

**New `restored` event** (`storage.rs`):
A single `Event` on `Storage` that is notified after every restore
attempt (success or failure). Waiting threads subscribe to it, then
re-check the flags under lock after waking.

**`EventListener::wait()`** (`event.rs`):
A synchronous blocking wrapper around `event_listener`'s `.wait()`,
needed because restore logic runs in synchronous backend operation
contexts (not async). Includes both non-`hanging_detection` and
`hanging_detection` builds.

**Restore protocol** (`operation/mod.rs`):
All three restore entry points (`task()`, `task_pair()`,
`prepare_tasks_with_callback()`) follow the same pattern:

1. **Classify under lock** — for each category that needs restoring: if
the `*_restoring` bit is already set, mark the task as "wait";
otherwise, set the bit and claim the I/O.
2. **Drop lock and do I/O** — only the thread that claimed a category
performs the backing-store lookup.
3. **Wait for other threads** — done *after* our own I/O so both
threads' work can overlap. Uses a fast path (check flags under lock
before registering a listener) to skip listener allocation when the
other thread has already finished. Returns the `StorageWriteGuard`
directly to the caller to avoid a second lock acquisition.
4. **Apply results under lock** — merge the loaded data into the task
via `apply_restore_result()` (returns `Result<()>`), set `*_restored`,
clear `*_restoring`.
5. **Notify once after all tasks** — a single `notify(usize::MAX)` on
the `restored` event after all I/O results have been applied, so waiters
are unblocked as late as possible but all at once.

**Batch restores** (`prepare_tasks_with_callback`):
Uses a `TaskRestoreEntry` struct to track per-task state through phases:
- **Phase 1a** — classify each task under lock: already restored →
callback immediately; another thread restoring → mark as wait; unclaimed
→ set `*_restoring` bit and claim.
- **Phase 1b** — batch I/O for all claimed tasks (single-task or batch
backing-store API).
- **Phase 1c** — apply I/O results under lock, clear restoring bits.
Errors are collected (rather than panicking immediately) so all
restoring bits are cleared first—otherwise other threads waiting on
those bits would hang.
- **Phase 2** — callbacks for self-restored tasks (separated from 1c so
the notify happens first and other threads are unblocked as early as
possible). Also inserts task types into the task cache.
- **Phase 3** — per-task loop: for each task being restored by another
thread, wait (using `wait_for_restore_or_panic`), then immediately call
the callback with the returned write guard.

A fast path at the top of `prepare_tasks_with_callback` short-circuits
the entire pipeline when `!should_restore()`, marking tasks as restored
and calling callbacks directly.

**Shared helpers:**
- `apply_restore_result()` — returns `Result<()>`; clears the restoring
bit and merges storage on success, or returns `Err` for the caller to
handle after notifying waiters.
- `wait_for_restoring_task()` — fast path (check under write lock,
return guard if already restored) + slow path (listener loop). Accepts
`TaskDataCategory` to handle waiting for both categories in a single
call with `All`.
- `wait_for_restore_or_panic()` — wrapper that panics on error,
returning the `StorageWriteGuard` on success.

**Other changes:**
- `TurboPersistence` (`turbo-persistence/src/db.rs`): removed the
`is_empty: AtomicBool` field; emptiness is now checked via a read lock
on inner state, avoiding a stale flag that could prevent restores after
writes.

<!-- NEXT_JS_LLM_PR -->

---------
Co-authored-by: 's avatarTobias Koppers <sokra@users.noreply.github.com>
Co-authored-by: 's avatarClaude <noreply@anthropic.com>
parent 71228c49
...@@ -1563,10 +1563,11 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> { ...@@ -1563,10 +1563,11 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
// We're creating a new task. // We're creating a new task.
let task_type = Arc::new(task_type); let task_type = Arc::new(task_type);
let task_id = self.persisted_task_id_factory.get(); let task_id = self.persisted_task_id_factory.get();
e.insert(task_type.clone(), task_id); // Initialize storage BEFORE making task_id visible in the cache.
// Mark the task as new in storage. // This ensures any thread that reads task_id from the cache sees
// Do this after e.insert so we aren't holding the task_cache lock // the storage entry already initialized (restored flags set).
self.storage.initialize_new_task(task_id); self.storage.initialize_new_task(task_id);
e.insert(task_type.clone(), task_id);
// insert() consumes e, releasing the lock // insert() consumes e, releasing the lock
self.track_cache_miss(&task_type); self.track_cache_miss(&task_type);
is_new = true; is_new = true;
...@@ -1638,8 +1639,9 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> { ...@@ -1638,8 +1639,9 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
RawEntry::Vacant(e) => { RawEntry::Vacant(e) => {
let task_type = Arc::new(task_type); let task_type = Arc::new(task_type);
let task_id = self.transient_task_id_factory.get(); let task_id = self.transient_task_id_factory.get();
e.insert(task_type.clone(), task_id); // Initialize storage BEFORE making task_id visible in the cache.
self.storage.initialize_new_task(task_id); self.storage.initialize_new_task(task_id);
e.insert(task_type.clone(), task_id);
self.track_cache_miss(&task_type); self.track_cache_miss(&task_type);
if is_root { if is_root {
......
...@@ -10,7 +10,7 @@ use std::{ ...@@ -10,7 +10,7 @@ use std::{
use thread_local::ThreadLocal; use thread_local::ThreadLocal;
use turbo_bincode::TurboBincodeBuffer; use turbo_bincode::TurboBincodeBuffer;
use turbo_tasks::{FxDashMap, TaskId, parallel}; use turbo_tasks::{FxDashMap, TaskId, event::Event, parallel};
use crate::{ use crate::{
backend::storage_schema::TaskStorage, backend::storage_schema::TaskStorage,
...@@ -53,6 +53,15 @@ pub enum SpecificTaskDataCategory { ...@@ -53,6 +53,15 @@ pub enum SpecificTaskDataCategory {
Data, Data,
} }
impl From<SpecificTaskDataCategory> for TaskDataCategory {
fn from(category: SpecificTaskDataCategory) -> Self {
match category {
SpecificTaskDataCategory::Meta => TaskDataCategory::Meta,
SpecificTaskDataCategory::Data => TaskDataCategory::Data,
}
}
}
impl SpecificTaskDataCategory { impl SpecificTaskDataCategory {
/// Returns the KeySpace for storing data of this category /// Returns the KeySpace for storing data of this category
pub fn key_space(self) -> KeySpace { pub fn key_space(self) -> KeySpace {
...@@ -83,6 +92,11 @@ pub struct Storage { ...@@ -83,6 +92,11 @@ pub struct Storage {
/// be marked as modified at the beginning of the next snapshot cycle. /// be marked as modified at the beginning of the next snapshot cycle.
snapshots: FxDashMap<TaskId, Option<Box<TaskStorage>>>, snapshots: FxDashMap<TaskId, Option<Box<TaskStorage>>>,
map: FxDashMap<TaskId, Box<TaskStorage>>, map: FxDashMap<TaskId, Box<TaskStorage>>,
/// A shared event notified whenever any task finishes restoring (successfully or not).
///
/// Threads waiting for another thread's in-progress restore subscribe to this event,
/// then re-check the specific task's `restoring`/`restored` bits after waking.
pub(crate) restored: Event,
} }
impl Storage { impl Storage {
...@@ -115,6 +129,7 @@ impl Storage { ...@@ -115,6 +129,7 @@ impl Storage {
shard_amount, shard_amount,
), ),
map, map,
restored: Event::new(|| || "Storage::restored".to_string()),
} }
} }
......
...@@ -161,6 +161,14 @@ struct TaskStorageSchema { ...@@ -161,6 +161,14 @@ struct TaskStorageSchema {
#[field(storage = "flag", category = "transient")] #[field(storage = "flag", category = "transient")]
data_restored: bool, data_restored: bool,
/// Whether meta data restoration is currently in progress by another thread.
#[field(storage = "flag", category = "transient")]
meta_restoring: bool,
/// Whether data restoration is currently in progress by another thread.
#[field(storage = "flag", category = "transient")]
data_restoring: bool,
/// Whether meta was modified before snapshot mode was entered. /// Whether meta was modified before snapshot mode was entered.
#[field(storage = "flag", category = "transient")] #[field(storage = "flag", category = "transient")]
meta_modified: bool, meta_modified: bool,
...@@ -348,6 +356,31 @@ impl TaskFlags { ...@@ -348,6 +356,31 @@ impl TaskFlags {
} }
} }
/// Check if the category's restoration is currently in progress by another thread
pub fn is_restoring(&self, category: TaskDataCategory) -> bool {
match category {
TaskDataCategory::Meta => self.meta_restoring(),
TaskDataCategory::Data => self.data_restoring(),
TaskDataCategory::All => self.meta_restoring() || self.data_restoring(),
}
}
/// Set or clear the restoring bits for the given category
pub fn set_restoring(&mut self, category: TaskDataCategory, value: bool) {
match category {
TaskDataCategory::Meta => {
self.set_meta_restoring(value);
}
TaskDataCategory::Data => {
self.set_data_restoring(value);
}
TaskDataCategory::All => {
self.set_meta_restoring(value);
self.set_data_restoring(value);
}
}
}
/// Check if any snapshot flag is set /// Check if any snapshot flag is set
pub fn any_modified_during_snapshot(&self) -> bool { pub fn any_modified_during_snapshot(&self) -> bool {
self.meta_modified_during_snapshot() || self.data_modified_during_snapshot() self.meta_modified_during_snapshot() || self.data_modified_during_snapshot()
......
...@@ -11,6 +11,7 @@ use std::{ ...@@ -11,6 +11,7 @@ use std::{
time::Duration, time::Duration,
}; };
use event_listener::Listener as _;
#[cfg(feature = "hanging_detection")] #[cfg(feature = "hanging_detection")]
use tokio::time::{Timeout, timeout}; use tokio::time::{Timeout, timeout};
...@@ -201,6 +202,17 @@ impl Future for EventListener { ...@@ -201,6 +202,17 @@ impl Future for EventListener {
} }
} }
#[cfg(not(feature = "hanging_detection"))]
impl EventListener {
/// Blocks the current thread until the event is notified.
///
/// This is the synchronous equivalent of `.await`-ing the `EventListener`.
/// Only valid in synchronous contexts (e.g. backend operations).
pub fn wait(self) {
self.listener.wait();
}
}
#[cfg(feature = "hanging_detection")] #[cfg(feature = "hanging_detection")]
pub struct EventListener { pub struct EventListener {
description: Arc<dyn Fn() -> String + Sync + Send>, description: Arc<dyn Fn() -> String + Sync + Send>,
...@@ -273,6 +285,22 @@ impl Future for EventListener { ...@@ -273,6 +285,22 @@ impl Future for EventListener {
} }
} }
#[cfg(feature = "hanging_detection")]
impl EventListener {
/// Blocks the current thread until the event is notified.
///
/// Note: In `hanging_detection` builds, timeout warnings are not emitted
/// for sync waits (only for async `.await` usage).
pub fn wait(mut self) {
if let Some(future) = self.future.take() {
// SAFETY: EventListener is Unpin, so it's safe to move out of the Pin.
unsafe { std::pin::Pin::into_inner_unchecked(future) }
.into_inner()
.wait();
}
}
}
#[cfg(all(test, not(feature = "hanging_detection")))] #[cfg(all(test, not(feature = "hanging_detection")))]
mod tests { mod tests {
use std::hint::black_box; use std::hint::black_box;
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment