Unverified Commit 2521aec5 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

[backport] turbo-tasks-backend: fix strongly consistent read hanging on a canceled task (#98931)

Backports #98312 ("turbo-tasks-backend: fix strongly consistent read
hanging on a canceled task") to `next-16-3`.

### What?

`next build` could hang at shutdown on the 16.3 line: a strongly
consistent read would wait forever on a task that shutdown had already
canceled, which in turn blocked `stop_and_wait`. This brings the
upstream fix, and its regression test, to the release branch.

### Why?

A canceled task has no output, and `has_output()` is used as the proxy
for "not computed yet". Shutdown therefore made the canceled child look
like work still to be scheduled: it was marked dirty and contributed a
dirty container that could never drain, because the (clean) parent never
re-evaluates `all_clean_event`. The strongly consistent reader waited on
that event indefinitely while holding a foreground job that shutdown
blocks on.

### How?

The fix records a terminal error output when a task execution is
canceled, so a canceled task is no longer indistinguishable from an
uncomputed one and the dirty-container accounting drains as it should.
It also makes the `hanging_detection` feature usable again, which
previously panicked on a `data` access check.

The source change is the upstream one, unmodified. One release-branch
adaptation was required: the backported regression test pulls in a
shared `tests/util.rs` helper that does not exist on `next-16-3` — it
was extracted on `canary` by unrelated later commits. A minimal version
is added here, providing only the persistent-backend setup the test
needs and written against this branch's APIs (`BackendOptions` has no
`gc` field here, and `turbo_backing_storage` still takes positional
`is_ci` / `is_short_session` / `skip_compaction` arguments). It follows
the same shape as the existing inline helper in `tests/eviction.rs`.

### Verification

- The regression test was run against an unmodified `next-16-3`
checkout, where it reproduces the bug: it hangs and fails after 30s with
"stop_and_wait hung: a cancelled task stranded a strongly consistent
reader". With this change it passes.
- `cargo fmt --check`, `cargo clippy -p turbo-tasks-backend
--all-targets` and the full `cargo test -p turbo-tasks-backend` suite
are clean.
- `pnpm build-all` succeeds.

<!-- NEXT_JS_LLM -->


<!-- fleet 33c5c205-5580-40f2-bcf8-5f91a945ef4b -->
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarLuke Sandberg <210140+lukesandberg@users.noreply.github.com>
parent a758ffcf
...@@ -22,6 +22,7 @@ verify_serialization = [] ...@@ -22,6 +22,7 @@ verify_serialization = []
verify_aggregation_graph = [] verify_aggregation_graph = []
verify_immutable = [] verify_immutable = []
verify_determinism = ["turbo-tasks/verify_determinism"] verify_determinism = ["turbo-tasks/verify_determinism"]
hanging_detection = ["turbo-tasks/hanging_detection"]
task_dirty_cause = ["turbo-tasks/task_dirty_cause"] task_dirty_cause = ["turbo-tasks/task_dirty_cause"]
trace_aggregation_update_stats = [] trace_aggregation_update_stats = []
trace_aggregation_update_queue = [] trace_aggregation_update_queue = []
......
...@@ -77,7 +77,7 @@ use crate::{ ...@@ -77,7 +77,7 @@ use crate::{
ActivenessState, CellRef, CollectibleRef, CollectiblesRef, Dirtyness, InProgressCellState, ActivenessState, CellRef, CollectibleRef, CollectiblesRef, Dirtyness, InProgressCellState,
InProgressState, InProgressStateInner, OutputValue, TransientTask, InProgressState, InProgressStateInner, OutputValue, TransientTask,
}, },
error::TaskError, error::{TaskError, TaskErrorItem},
kv_backing_storage::TurboBackingStorage, kv_backing_storage::TurboBackingStorage,
utils::{ utils::{
dash_map_raw_entry::{RawEntry, get_shard, raw_entry_in_shard, raw_get_in_shard}, dash_map_raw_entry::{RawEntry, get_shard, raw_entry_in_shard, raw_get_in_shard},
...@@ -1836,6 +1836,19 @@ impl TurboTasksBackend { ...@@ -1836,6 +1836,19 @@ impl TurboTasksBackend {
} }
} }
// Give the task a terminal output. `connect_children` treats a child with no output as
// "not computed yet" and marks it dirty to be scheduled, which for a canceled task means
// it stays a dirty container of its parent forever and any strongly consistent reader
// above it never settles.
task.set_output(OutputValue::Error(Arc::new(TaskError::Error(Box::new(
TaskErrorItem {
message: TurboTasksExecutionErrorMessage::PIISafe(std::borrow::Cow::Borrowed(
"task execution was canceled by shutdown",
)),
source: None,
},
)))));
// Mark the cancelled task as session-dependent dirty so it will be re-executed // Mark the cancelled task as session-dependent dirty so it will be re-executed
// in the next session. Without this, any reader that encounters the cancelled task // in the next session. Without this, any reader that encounters the cancelled task
// records an error in its output. That error is persisted and would poison // records an error in its output. That error is persisted and would poison
......
...@@ -1330,16 +1330,37 @@ pub trait TaskGuard: Debug + TaskStorageAccessors { ...@@ -1330,16 +1330,37 @@ pub trait TaskGuard: Debug + TaskStorageAccessors {
} }
} }
/// A description of this task for diagnostics: `"<id> <task type>"`.
///
/// Is intentionally tolerant of non-resident data.
fn get_task_desc_fn(&self) -> impl Fn() -> String + Send + Sync + 'static { fn get_task_desc_fn(&self) -> impl Fn() -> String + Send + Sync + 'static {
let task_type = self.get_task_type().to_owned(); // Bypass `check_access`!!
// Generally it is a bad idea since accessing a `Data` field like get_persistence_task_type
// without opening the task that way is bug But this is for diagnostics and
// debugging purposes only so we can cheat.
let task_type = self
.typed()
.get_persistent_task_type()
.map(|task_type| TaskTypeRef::Cached(task_type).to_owned())
.or_else(|| {
self.typed()
.get_transient_task_type()
.map(|task_type| TaskTypeRef::Transient(task_type).to_owned())
});
let task_id = self.id(); let task_id = self.id();
move || format!("{task_id:?} {task_type}") move || match &task_type {
Some(task_type) => format!("{task_id:?} {task_type}"),
None => format!("{task_id:?} task-type-not-available"),
}
} }
// Requires the task to have been opened with Data access
fn get_task_description(&self) -> String { fn get_task_description(&self) -> String {
let task_type = self.get_task_type().to_owned(); let task_type = self.get_task_type().to_owned();
let task_id = self.id(); let task_id = self.id();
format!("{task_id:?} {task_type}") format!("{task_id:?} {task_type}")
} }
#[cfg(feature = "trace_task_dirty")] #[cfg(feature = "trace_task_dirty")]
fn get_task_name(&self) -> String { fn get_task_name(&self) -> String {
let task_type = self.get_task_type().to_owned(); let task_type = self.get_task_type().to_owned();
......
...@@ -1714,8 +1714,11 @@ mod tests { ...@@ -1714,8 +1714,11 @@ mod tests {
// Schema Size Tests // Schema Size Tests
// ========================================================================== // ==========================================================================
// `hanging_detection` adds an `Arc<dyn Fn() -> String>` description to every `Event`, which
// grows `LazyField` past the size asserted here. The feature is diagnostic-only, so the sizes
// are not meaningful under it.
#[test] #[test]
#[cfg(target_pointer_width = "64")] #[cfg(all(target_pointer_width = "64", not(feature = "hanging_detection")))]
fn test_schema_size() { fn test_schema_size() {
assert_eq!( assert_eq!(
size_of::<TaskStorage>(), size_of::<TaskStorage>(),
......
#![feature(arbitrary_self_types)]
#![allow(clippy::needless_return)] // tokio macro-generated code doesn't respect this
//! Regression test for a bug where shutdown would cause a strongly consistent read to hang
//! if a task in scope was cancelled.
mod util;
use std::{
sync::{Arc, LazyLock},
time::Duration,
};
use anyhow::Result;
use tokio::sync::Barrier;
use turbo_tasks::{TurboTasks, Vc};
use turbo_tasks_backend::TurboTasksBackend;
use crate::util::create_tt;
/// Rendezvous between the task and the test: the task waits here, the test joins once it is ready
/// to shut down, and both proceed. Two barriers rather than one so the test can shut down
/// *between* them, which is what puts the child's dispatch after `stopped` is set.
static PARKED: LazyLock<Barrier> = LazyLock::new(|| Barrier::new(2));
static RELEASED: LazyLock<Barrier> = LazyLock::new(|| Barrier::new(2));
/// Launched only after shutdown has started, so it is cancelled before it executes.
#[turbo_tasks::function]
fn launched_during_shutdown() -> Vc<u32> {
Vc::cell(42)
}
#[turbo_tasks::function(operation, root)]
async fn pauses_then_launches_a_child() -> Result<Vc<u32>> {
PARKED.wait().await;
RELEASED.wait().await;
// Dispatched after `stopped` is set, so this child is cancelled rather than run. Reading it
// must surface that rather than hang.
Ok(Vc::cell(*launched_during_shutdown().await?))
}
/// A strongly consistent read whose task launches a child during shutdown.
///
/// The read is expected to fail — the child is cancelled. What must not happen is
/// `stop_and_wait` blocking forever on the reader's foreground job.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn shutdown_unblocks_read_of_canceled_child() {
let (tt, _persistence_dir) = create_tt("shutdown_unblocks_read_of_canceled_child");
let tt_reader: Arc<TurboTasks<TurboTasksBackend>> = tt.clone();
let reader = tokio::spawn(async move {
let _ = turbo_tasks::run_once(tt_reader.clone(), async move {
// An error is the expected outcome once the child is cancelled; returning at all is
// the property under test.
let _ = pauses_then_launches_a_child()
.read_strongly_consistent()
.await;
anyhow::Ok(())
})
.await;
});
// Wait for the task to park, so shutdown lands between its two barriers.
PARKED.wait().await;
let tt_stop = tt.clone();
let stop = tokio::spawn(async move { tt_stop.stop_and_wait().await });
// Let shutdown set `stopped` before releasing the task, so its child is dispatched into a
// stopping backend. The task then runs to completion — turbo-tasks does not cancel a task
// that is already executing — and only the cancelled child can strand the reader.
tokio::time::sleep(Duration::from_millis(50)).await;
RELEASED.wait().await;
tokio::time::timeout(Duration::from_secs(30), stop)
.await
.expect("stop_and_wait hung: a cancelled task stranded a strongly consistent reader")
.unwrap();
tokio::time::timeout(Duration::from_secs(30), reader)
.await
.expect("the strongly consistent read never returned after shutdown")
.unwrap();
}
//! Shared setup for tests that need a real on-disk persistent backend and direct access to
//! [`TurboTasksBackend`] test hooks (`snapshot_and_evict_for_testing`, …).
//!
//! These tests can't use `turbo_tasks_testing::register!`: that harness hands back an
//! `Arc<dyn TurboTasksApi>`, which erases the concrete backend type and makes `tt.backend()`
//! unreachable. It also owns the session loop (re-running the body and comparing results), whereas
//! these tests need to drive snapshots — and sometimes a DB reopen — on their own schedule.
use std::{path::Path, sync::Arc};
use turbo_tasks::TurboTasks;
use turbo_tasks_backend::{BackendOptions, EvictionMode, GitVersionInfo, TurboTasksBackend};
/// Opens a backend rooted at `path`.
///
/// Reusing the same `path` (after the previous backend has been stopped) reopens the persisted
/// database, which is how a test can assert that state survives a restart.
fn open_tt_at(path: &Path, num_workers: usize) -> Arc<TurboTasks<TurboTasksBackend>> {
TurboTasks::new(TurboTasksBackend::new(
BackendOptions {
num_workers: Some(num_workers),
small_preallocation: true,
// Avoid racing with the background snapshot loop; these tests drive
// snapshot_and_evict_for_testing manually.
storage_mode: Some(turbo_tasks_backend::StorageMode::ReadWriteOnShutdown),
eviction_mode: EvictionMode::Full,
..Default::default()
},
turbo_tasks_backend::turbo_backing_storage(
path,
&GitVersionInfo {
describe: "test-unversioned",
dirty: false,
},
// is_ci
false,
// is_short_session
true,
// skip_compaction
true,
)
.unwrap()
.0,
))
}
/// A fresh persistent backend in its own temp directory, with `num_workers` workers.
///
/// The directory is rooted under `CARGO_TARGET_TMPDIR/.cache/`, with the test `name` as a prefix
/// so failed runs are easy to find on disk. The returned [`tempfile::TempDir`] cleans up on drop,
/// so callers keep it alive until the backend has finished shutting down.
pub fn create_tt_with_workers(
name: &str,
num_workers: usize,
) -> (Arc<TurboTasks<TurboTasksBackend>>, tempfile::TempDir) {
let parent = Path::new(env!("CARGO_TARGET_TMPDIR")).join(".cache");
std::fs::create_dir_all(&parent).unwrap();
let dir = tempfile::Builder::new()
.prefix(&format!("{name}-"))
.tempdir_in(&parent)
.unwrap();
let tt = open_tt_at(dir.path(), num_workers);
(tt, dir)
}
/// A fresh persistent backend in its own temp directory, with the default worker count.
pub fn create_tt(name: &str) -> (Arc<TurboTasks<TurboTasksBackend>>, tempfile::TempDir) {
create_tt_with_workers(name, 2)
}
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment