Unverified Commit ef6a1004 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

turbo-tasks-backend: treat reads and connects of collected tasks as errors (#99646)

### What?

- Reading a cell or output of a task that no longer exists returns an
ordinary error instead of panicking. That covers a task collected by GC,
whether still soft-deleted in memory or already evicted.
`lock_task_and_optional_reader` opens the target with `AllowMissing` and
returns before any dependency edge is added.
- `connect_task` skips the connect when the task no longer exists. The
output read that follows then fails with the same error.
- `MustExist` stays everywhere in edge maintenance (cleanup,
aggregation, `task_pair`), where a missing task means a GC bug.

### Why?

GC does not keep a cell producer alive for its dependents, so a stale
task can hold a collected task's `Vc` in its **arguments**. Such a task
is in a "dead but doesn't know it yet" state but it still may be
optimistically executed, so these errors are expected in 'eventual
consistency cases'.

`connect_task` had a worse version. An `OperationVc` held without a pin
can name a collected task. Connecting it from outside any task (a
top-level `run`, as NAPI code would) materialized a blank entry,
scheduled it, and its execution panicked outside the per-task panic
boundary, aborting the process. This is considered a bug, the caller
should have added a 'pin' but in the mean time we can regularize the
failure mode.

### How?

New tests in `tests/gc_cross_session.rs`:
- `reading_a_collected_task_is_an_error`: a reader holds the collected
target in its arguments and re-executes.
- `connecting_a_soft_deleted_operation_is_an_error` and
`connecting_an_evicted_collected_operation_is_an_error`: a task holds an
`OperationVc` it never connected, and connects it after the operation is
collected.
- `parentless_connect_of_an_evicted_operation_does_not_abort`: also
asserts no entry is materialized.
parent deaef3cb
......@@ -583,21 +583,24 @@ struct TaskExecutionCompletePrepareResult {
pub is_session_dependent: bool,
}
/// Locks the task being read, and the reader too when a dependency edge may need to be added.
///
/// Returns `None` if the task being read is gone (collected by GC, or missing from storage).
fn lock_task_and_optional_reader<'e>(
ctx: &mut ExecuteContext<'e>,
task_id: TaskId,
reader_id: Option<TaskId>,
) -> (TaskGuard<'e>, Option<TaskGuard<'e>>) {
) -> Option<(TaskGuard<'e>, Option<TaskGuard<'e>>)> {
let task = ctx.try_task(task_id, TaskDataCategory::All)?;
let Some(reader_id) = reader_id else {
return (ctx.task(task_id, TaskDataCategory::All), None);
return Some((task, None));
};
// Immutable tasks never need dependency edges and can never be invalidated. Avoid locking the
// reader too in that common case. When the task is still mutable, drop the speculative lock
// and reacquire both locks together to preserve the invalidation race guarantee.
let task = ctx.task(task_id, TaskDataCategory::All);
if task.immutable() && !cfg!(feature = "verify_immutable") {
(task, None)
Some((task, None))
} else {
drop(task);
......@@ -610,13 +613,21 @@ fn lock_task_and_optional_reader<'e>(
// when dependency edges may still be added.
if task.immutable() && !cfg!(feature = "verify_immutable") {
drop(reader);
(task, None)
Some((task, None))
} else {
(task, Some(reader))
Some((task, Some(reader)))
}
}
}
/// The error a read of a missing task (see [`lock_task_and_optional_reader`]) fails with.
fn collected_task_read_error(task_id: TaskId, operation: &str) -> anyhow::Error {
anyhow::anyhow!(
"{operation}: task {task_id} no longer exists (it was garbage collected). The reading \
task holds a stale reference to it and is expected to be dropped by its parent."
)
}
// Operations
impl TurboTasksBackend {
fn try_read_task_output(
......@@ -635,9 +646,11 @@ impl TurboTasksBackend {
&& reader_id != task_id)
.then_some(reader_id)
});
let (mut task, mut reader_task) =
lock_task_and_optional_reader(&mut ctx, task_id, need_reader_task);
task.assert_not_deleted("read_task_output");
let Some((mut task, mut reader_task)) =
lock_task_and_optional_reader(&mut ctx, task_id, need_reader_task)
else {
return Err(collected_task_read_error(task_id, "read_task_output"));
};
fn listen_to_done_event(
reader_description: Option<EventDescription>,
......@@ -1010,9 +1023,11 @@ impl TurboTasksBackend {
&& reader_id != task_id)
.then_some(reader_id)
});
let (mut task, reader_task) =
lock_task_and_optional_reader(&mut ctx, task_id, need_reader_task);
task.assert_not_deleted("read_task_cell");
let Some((mut task, reader_task)) =
lock_task_and_optional_reader(&mut ctx, task_id, need_reader_task)
else {
return Err(collected_task_read_error(task_id, "read_task_cell"));
};
let content = if final_read_hint {
task.remove_cell_data(&cell, &get_value_type(cell.type_id()).persistence)
......@@ -3426,11 +3441,24 @@ impl TurboTasksBackend {
turbo_tasks: &TurboTasks<TurboTasksBackend>,
) {
self.assert_not_persistent_calling_transient(parent_task, task);
let mut ctx = self.execute_context(turbo_tasks);
// An `OperationVc` held without a pin can name a collected task (soft-deleted or already
// gone). That is a bug in whoever held it, so fail loudly here, in the connecting task,
// rather than letting the dead id spread into this task's output or another task's
// arguments. Panic before `connect_child`, which would materialize and schedule an entry
// for the missing task.
if ctx.try_task(task, TaskDataCategory::Meta).is_none() {
panic!(
"connect_task: task {task} no longer exists (it was garbage collected). An \
`OperationVc` to it was held without a pin; hold it in a `GcRoot` or connect it \
from its parent task."
);
}
connect_child(
parent_task,
task,
/* release_construction_ref */ false,
self.execute_context(turbo_tasks),
ctx,
);
}
......
......@@ -15,11 +15,14 @@ use std::{
};
use anyhow::Result;
use turbo_tasks::{GcRoot, ResolvedVc, TurboTasks, Vc};
use turbo_tasks::{GcRoot, OperationVc, ResolvedVc, TurboTasks, Vc};
use turbo_tasks_backend::TurboTasksBackend;
use crate::{
gc_fixture::{Constant, Selector, create_constant, create_selector, diamond_root_op},
gc_fixture::{
Constant, Selector, create_constant, create_other_constant, create_selector,
diamond_root_op,
},
util::{create_persistence_dir, reopen_tt_with_gc, reopen_tt_with_gc_ttl},
};
......@@ -351,3 +354,222 @@ async fn shared_cell_target_collected_before_its_second_reader() {
tt.stop_and_wait().await;
}
}
/// A task holding a collected task's `Vc` in its arguments fails with an error when it
/// re-executes, rather than panicking.
///
/// This is the stale-mapper shape: reader 2 receives the shared target as an argument and reads
/// it, but only the owning side is the target's parent. Once the owning side is collected, reader 2
/// can never re-execute successfully, and nothing guarantees its parent re-runs (and drops it)
/// first. Here the teardown dirties reader 2, which re-executes right away against the collected
/// target.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn reading_a_collected_task_is_an_error() {
let dir = create_persistence_dir("reading_a_collected_task_is_an_error");
let tt = reopen_tt_with_gc(&dir);
let borrowing_op = turbo_tasks::run_once(tt.clone(), async move {
let selector_op = create_selector(false);
let selector_vc = selector_op.resolve().strongly_consistent().await?;
let selector = selector_op.read_strongly_consistent().await?;
let constant_op = create_constant();
let constant_vc = constant_op.resolve().strongly_consistent().await?;
let owning = select_owning(selector_vc, constant_vc);
let target = owning.resolve().strongly_consistent().await?;
assert_eq!(*target.await?, 41);
let borrowing = borrowing_root(target);
assert_eq!(*borrowing.read_strongly_consistent().await?, 43);
selector.set(true);
assert_eq!(*owning.read_strongly_consistent().await?, 0);
anyhow::Ok(borrowing)
})
.await
.unwrap();
let borrowing_pin = GcRoot::pin(tt.clone(), borrowing_op);
let collected = gc_until_collected(&tt, 3).await;
assert!(
collected >= 3,
"the owning subtree, reader 1 and the shared target should be collected (got {collected})"
);
let result = turbo_tasks::run_once(tt.clone(), async move {
borrowing_op.read_strongly_consistent().await?;
anyhow::Ok(())
})
.await;
let err = result.expect_err("reader 2 reads a collected task, so the root must fail");
assert!(
format!("{err:?}").contains("garbage collected"),
"expected the collected-task read error, got: {err:?}"
);
drop(borrowing_pin);
tt.stop_and_wait().await;
}
/// The operation the connect tests hand out; [`select_owned_op`] is its only parent.
#[turbo_tasks::function(operation)]
async fn owned_op(constant: ResolvedVc<Constant>) -> Result<Vc<u32>> {
Ok(Vc::cell(*constant.await?.get() + 7))
}
#[turbo_tasks::value(transparent)]
struct OptionOp(Option<OperationVc<u32>>);
/// Calls [`owned_op`] (becoming its parent) and hands it out, until the selector drops it.
#[turbo_tasks::function(operation, root)]
async fn select_owned_op(
selector: ResolvedVc<Selector>,
constant: ResolvedVc<Constant>,
) -> Result<Vc<OptionOp>> {
if !*selector.await?.get() {
let op = owned_op(constant);
op.connect().await?;
Ok(Vc::cell(Some(op)))
} else {
Ok(Vc::cell(None))
}
}
/// Holds an `OperationVc` in its arguments but only connects it once `other` is bumped, so it is
/// not the operation's parent when the operation is collected.
#[turbo_tasks::function(operation, root)]
async fn lazy_connector_root(op: OperationVc<u32>, other: ResolvedVc<Constant>) -> Result<Vc<u32>> {
if *other.await?.get() == 0 {
return Ok(Vc::cell(0));
}
Ok(Vc::cell(*op.connect().await?))
}
/// A task that holds a collected operation in its arguments and connects it later. Whether the
/// operation is still soft-deleted or already evicted, the connect fails the connecting task with
/// the missing-pin panic, and nothing is materialized for the collected operation.
/// The error is formatted before the backend stops: formatting resolves task names through it.
async fn connect_collected_operation(name: &str, evict: bool) -> Result<u32, String> {
let dir = create_persistence_dir(name);
let tt = reopen_tt_with_gc(&dir);
let (connector_op, selector) = turbo_tasks::run_once(tt.clone(), async move {
let selector_op = create_selector(false);
let selector_vc = selector_op.resolve().strongly_consistent().await?;
let selector = selector_op.read_strongly_consistent().await?;
let constant_op = create_constant();
let constant_vc = constant_op.resolve().strongly_consistent().await?;
let other_op = create_other_constant();
let other_vc = other_op.resolve().strongly_consistent().await?;
let owner = select_owned_op(selector_vc, constant_vc);
let op = owner.read_strongly_consistent().await?.unwrap();
let connector = lazy_connector_root(op, other_vc);
assert_eq!(*connector.read_strongly_consistent().await?, 0);
anyhow::Ok((connector, selector))
})
.await
.unwrap();
let pin = GcRoot::pin(tt.clone(), connector_op);
// Drop the operation from its only parent and collect it.
turbo_tasks::run_once(tt.clone(), async move {
selector.set(true);
anyhow::Ok(())
})
.await
.unwrap();
let collected = gc_until_collected(&tt, 1).await;
assert!(
collected >= 1,
"owned_op should be collected (got {collected})"
);
if evict {
tt.backend().snapshot_and_evict_for_testing(&tt);
}
let result = turbo_tasks::run_once(tt.clone(), async move {
// Re-read the state: after eviction a `ReadRef` from before would hold a stale copy.
let other = create_other_constant().read_strongly_consistent().await?;
other.set(1);
anyhow::Ok(*connector_op.read_strongly_consistent().await?)
})
.await;
let result = result.map_err(|e| format!("{e:?}"));
drop(pin);
tt.stop_and_wait().await;
result
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn connecting_a_soft_deleted_operation_is_an_error() {
let err = connect_collected_operation("connecting_a_soft_deleted_operation_is_an_error", false)
.await
.expect_err("the operation was collected, so connecting it must fail");
assert!(
err.contains("was held without a pin"),
"expected the missing-pin panic, got: {err}"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn connecting_an_evicted_collected_operation_is_an_error() {
let err = connect_collected_operation(
"connecting_an_evicted_collected_operation_is_an_error",
true,
)
.await
.expect_err("the operation is gone, so connecting it must fail");
assert!(
err.contains("was held without a pin"),
"expected the missing-pin panic, got: {err}"
);
}
/// Connecting a collected operation from outside any task (a top-level `run`, as NAPI code holding
/// an unpinned `OperationVc` would) used to materialize a blank entry for it and schedule it. Its
/// execution then failed outside the per-task panic boundary and aborted the process.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn parentless_connect_of_an_evicted_operation_does_not_abort() {
let dir = create_persistence_dir("parentless_connect_of_an_evicted_operation_does_not_abort");
let tt = reopen_tt_with_gc(&dir);
let (op, selector) = turbo_tasks::run_once(tt.clone(), async move {
let selector_op = create_selector(false);
let selector_vc = selector_op.resolve().strongly_consistent().await?;
let selector = selector_op.read_strongly_consistent().await?;
let constant_op = create_constant();
let constant_vc = constant_op.resolve().strongly_consistent().await?;
let owner = select_owned_op(selector_vc, constant_vc);
let op = owner.read_strongly_consistent().await?.unwrap();
anyhow::Ok((op, selector))
})
.await
.unwrap();
turbo_tasks::run_once(tt.clone(), async move {
selector.set(true);
anyhow::Ok(())
})
.await
.unwrap();
let collected = gc_until_collected(&tt, 1).await;
assert!(
collected >= 1,
"owned_op should be collected (got {collected})"
);
tt.backend().snapshot_and_evict_for_testing(&tt);
let resident_before = tt.backend().resident_task_count_for_testing();
let err = tt
.run(async move { anyhow::Ok(*op.read_strongly_consistent().await?) })
.await
.expect_err("the operation is gone, so connecting it must fail");
assert!(
format!("{err:?}").contains("was held without a pin"),
"expected the missing-pin panic, got: {err:?}"
);
assert_eq!(
tt.backend().resident_task_count_for_testing(),
resident_before,
"no entry may be materialized for the collected operation"
);
tt.stop_and_wait().await;
}
......@@ -26,6 +26,13 @@ pub fn create_constant() -> Vc<Constant> {
Constant(State::new(0)).cell()
}
/// A second, independent [`Constant`] cell, for a test that needs a dependency it can bump without
/// touching the one [`create_constant`] returns.
#[turbo_tasks::function(operation, root)]
pub fn create_other_constant() -> Vc<Constant> {
Constant(State::new(0)).cell()
}
// --- Diamond fixture ---
//
// Reader `A` forward-cell-depends on target `B` without `B` being its child; both are children of
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment