Commit 356d605b authored by Tobias Koppers's avatar Tobias Koppers Committed by Luke Sandberg

turbo-tasks-backend: stability fixes for task cancellation and error handling (#92254)

Bug fixes and a refactoring in `turbo-tasks-backend` targeting stability
issues that surface when filesystem caching is enabled:

1. **Preserve `cell_type_max_index` on task error** — when a task fails
partway through execution, `cell_counters` only reflects the
partially-executed state. Previously, `cell_type_max_index` was updated
from these incomplete counters, which removed entries for cell types not
yet encountered. This caused `"Cell no longer exists"` hard errors for
tasks that still held dependencies on those cells. The fix skips the
`cell_type_max_index` update on error, keeping it consistent with the
preserved cell data (which already wasn't cleared on error).

This bug manifested specifically with `serialization = "hash"` cell
types (e.g. `FileContent`), where cell data is transient and readers
fall back to `cell_type_max_index` to decide whether to schedule
recomputation.

2. **Fix shutdown hang and cache poisoning for cancelled tasks** — three
related fixes for tasks cancelled during shutdown:
- `task_execution_canceled` now drains and notifies all
`InProgressCellState` events, preventing `stop_and_wait` from hanging on
foreground jobs waiting on cells that will never be filled.
- `try_read_task_cell` bails early (before calling `listen_to_cell`)
when a task is in `Canceled` state, avoiding pointless listener
registrations that would never resolve.
- Cancelled tasks are marked as session-dependent dirty, preventing
cache poisoning where `"was canceled"` errors get persisted as task
output and break subsequent builds. The session-dependent dirty flag
causes the task to re-execute in the next session, invalidating stale
dependents.

3. **Extract `update_dirty_state` helper on `TaskGuard`** — the "read
old dirty state → apply new state → propagate via
`ComputeDirtyAndCleanUpdate`" pattern was duplicated between
`task_execution_canceled` and `task_execution_completed_finish`. The new
`update_dirty_state` default method on `TaskGuard` handles both
transitions (to `SessionDependent` or to `None`) and returns the
aggregation job + `ComputeDirtyAndCleanUpdateResult` for callers that
need post-processing (e.g. firing the `all_clean_event`).

These bugs caused observable failures when using Turbopack with
filesystem caching (`--cache` / persistent cache):

- `"Cell no longer exists"` panics/errors on incremental rebuilds after
a task error.
- Hangs on `stop_and_wait` during dev server shutdown.
- Stale `"was canceled"` errors persisted in the cache breaking
subsequent builds until the cache is cleared.

Changes are in `turbopack/crates/turbo-tasks-backend/src/backend/`:

**`mod.rs`:**
- Guard the `cell_type_max_index` update block inside `if
result.is_ok()` to skip it on error, with a cross-reference comment to
`task_execution_completed_cleanup` (which similarly skips cell data
removal on error — the two must stay in sync).
- Move the `is_cancelled` bail in `try_read_task_cell` before the
`listen_to_cell` call to avoid inserting phantom `InProgressCellState`
events that would never be notified.
- In `task_execution_canceled`: switch to `TaskDataCategory::All`
(needed for dirty state metadata access), notify all pending in-progress
cell events, and mark the task as `SessionDependent` dirty via the new
helper.
- In `task_execution_completed_finish`: replace ~77 lines of inline
dirty state logic with a call to
`task.update_dirty_state(new_dirtyness)`, preserving the
`all_clean_event` post-processing and the `dirty_changed` variable under
`#[cfg(feature = "verify_determinism")]`.

**`operation/mod.rs`:**
- Add `update_dirty_state` default method on `TaskGuard` trait (~60
lines), co-located with the existing `dirty_state()` reader. Takes
`Option<Dirtyness>`, applies the transition, builds
`ComputeDirtyAndCleanUpdate`, and returns
`(Option<AggregationUpdateJob>, ComputeDirtyAndCleanUpdateResult)`.
- Add `ComputeDirtyAndCleanUpdateResult` to the public re-exports.

---------
Co-authored-by: 's avatarTobias Koppers <sokra@users.noreply.github.com>
Co-authored-by: 's avatarClaude <noreply@anthropic.com>
parent 3b77a6e2
...@@ -18,6 +18,7 @@ use turbo_tasks::{ ...@@ -18,6 +18,7 @@ use turbo_tasks::{
TurboTasksCallApi, TypedSharedReference, backend::CachedTaskType, TurboTasksCallApi, TypedSharedReference, backend::CachedTaskType,
}; };
use self::aggregation_update::ComputeDirtyAndCleanUpdate;
use crate::{ use crate::{
backend::{ backend::{
EventDescription, OperationGuard, TaskDataCategory, TurboTasksBackend, EventDescription, OperationGuard, TaskDataCategory, TurboTasksBackend,
...@@ -764,6 +765,77 @@ pub trait TaskGuard: Debug + TaskStorageAccessors { ...@@ -764,6 +765,77 @@ pub trait TaskGuard: Debug + TaskStorageAccessors {
Some(Dirtyness::SessionDependent) => (true, self.current_session_clean()), Some(Dirtyness::SessionDependent) => (true, self.current_session_clean()),
} }
} }
/// Update the task's dirty state to `new_dirtyness`, applying the change to stored fields,
/// computing the aggregated propagation update, and firing the `all_clean_event` if the task
/// transitioned to clean.
///
/// Returns an optional `AggregationUpdateJob` that the caller must run via
/// `AggregationUpdateQueue::run` to propagate the change to aggregating ancestors.
fn update_dirty_state(
&mut self,
new_dirtyness: Option<Dirtyness>,
) -> Option<AggregationUpdateJob>
where
Self: Sized,
{
let task_id = self.id();
let old_dirtyness = self.get_dirty().cloned();
let (old_self_dirty, old_current_session_self_clean) = self.dirty_state();
let (new_self_dirty, new_current_session_self_clean) = match new_dirtyness {
None => (false, false),
Some(Dirtyness::Dirty(_)) => (true, false),
Some(Dirtyness::SessionDependent) => (true, true),
};
if old_dirtyness != new_dirtyness {
if let Some(value) = new_dirtyness {
self.set_dirty(value);
} else {
self.take_dirty();
}
}
if old_current_session_self_clean != new_current_session_self_clean {
self.set_current_session_clean(new_current_session_self_clean);
}
if old_self_dirty == new_self_dirty
&& old_current_session_self_clean == new_current_session_self_clean
{
return None;
}
let dirty_container_count = self
.get_aggregated_dirty_container_count()
.cloned()
.unwrap_or_default();
let current_session_clean_container_count = self
.get_aggregated_current_session_clean_container_count()
.copied()
.unwrap_or_default();
let result = ComputeDirtyAndCleanUpdate {
old_dirty_container_count: dirty_container_count,
new_dirty_container_count: dirty_container_count,
old_current_session_clean_container_count: current_session_clean_container_count,
new_current_session_clean_container_count: current_session_clean_container_count,
old_self_dirty,
new_self_dirty,
old_current_session_self_clean,
new_current_session_self_clean,
}
.compute();
// Fire the all_clean_event if the task transitioned to clean
if result.dirty_count_update - result.current_session_clean_update < 0
&& let Some(activeness_state) = self.get_activeness_mut()
{
activeness_state.all_clean_event.notify(usize::MAX);
activeness_state.unset_active_until_clean();
if activeness_state.is_empty() {
self.take_activeness();
}
}
result
.aggregated_update(task_id)
.and_then(|aggregated_update| {
AggregationUpdateJob::data_update(self, aggregated_update)
})
}
fn dirty_containers(&self) -> impl Iterator<Item = TaskId> { fn dirty_containers(&self) -> impl Iterator<Item = TaskId> {
self.dirty_containers_with_count() self.dirty_containers_with_count()
.map(|(task_id, _)| task_id) .map(|(task_id, _)| task_id)
...@@ -1106,8 +1178,8 @@ impl_operation!(LeafDistanceUpdate leaf_distance_update::LeafDistanceUpdateQueue ...@@ -1106,8 +1178,8 @@ impl_operation!(LeafDistanceUpdate leaf_distance_update::LeafDistanceUpdateQueue
pub use self::invalidate::TaskDirtyCause; pub use self::invalidate::TaskDirtyCause;
pub use self::{ pub use self::{
aggregation_update::{ aggregation_update::{
AggregatedDataUpdate, AggregationUpdateJob, ComputeDirtyAndCleanUpdate, AggregatedDataUpdate, AggregationUpdateJob, get_aggregation_number, get_uppers,
get_aggregation_number, get_uppers, is_aggregating_node, is_root_node, is_aggregating_node, is_root_node,
}, },
cleanup_old_edges::OutdatedEdge, cleanup_old_edges::OutdatedEdge,
connect_children::connect_children, connect_children::connect_children,
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment