Unverified Commit 10ed52fb authored by vercel-fleet-prod[bot]'s avatar vercel-fleet-prod[bot] Committed by GitHub

Fail tasks that leak KeyedReadRefs and simplify keyed reads

Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent 4bcaed44
...@@ -16,5 +16,5 @@ These rules apply throughout this directory tree when developing with turbo_task ...@@ -16,5 +16,5 @@ These rules apply throughout this directory tree when developing with turbo_task
- Existing `.cell()` calls or wrapping tasks are not automatically good precedent. Verify their caller and cache-key assumptions before copying the pattern. - Existing `.cell()` calls or wrapping tasks are not automatically good precedent. Verify their caller and cache-key assumptions before copying the pattern.
- Put maps containing information for a large part of the build (for example, a map from modules to analysis results) into separate cells. Use `cell = "keyed"` on their value types, for example `#[turbo_tasks::value(transparent, cell = "keyed")]` for a map wrapper. - Put maps containing information for a large part of the build (for example, a map from modules to analysis results) into separate cells. Use `cell = "keyed"` on their value types, for example `#[turbo_tasks::value(transparent, cell = "keyed")]` for a map wrapper.
- Read keyed cells through keyed access, such as `map.get(&key).await?`, so invalidation depends only on the entries actually read. Awaiting the entire map and then looking up a key creates a dependency on the whole map instead. - Read keyed cells through keyed access, such as `map.get(&key).await?`, so invalidation depends only on the entries actually read. Awaiting the entire map and then looking up a key creates a dependency on the whole map instead.
- When a task reads many keys of the same keyed cell, use `let map = map.keyed().await?;` and then the synchronous `map.get(&key)` / `map.contains_key(&key)`. It reads the cell once and, when the `KeyedReadRef` is dropped within the task, narrows the dependency to the accessed keys in one batch. - When a task reads many keys of the same keyed cell, use `let map = map.keyed().await?;` and then the synchronous `map.get(&key)` / `map.contains_key(&key)`. It reads the cell once and, when the `KeyedReadRef` is dropped, narrows the dependency to the accessed keys in one batch. The `KeyedReadRef` must be dropped within the task execution: don't return, store, leak, or move it to other threads or tasks. Otherwise the task fails with a panic error.
- When changing task boundaries, check the intended invalidation behavior. An unrelated module change should not invalidate consumers of unchanged keyed entries; correct output alone does not demonstrate good cache granularity. - When changing task boundaries, check the intended invalidation behavior. An unrelated module change should not invalidate consumers of unchanged keyed entries; correct output alone does not demonstrate good cache granularity.
...@@ -1245,10 +1245,14 @@ impl TurboTasksBackend { ...@@ -1245,10 +1245,14 @@ impl TurboTasksBackend {
} }
} }
/// Converts all [`CellKey::Keyed`] dependencies of `reader` that are still present (their /// Converts all [`CellKey::Keyed`] dependencies of `reader` that are still present into normal
/// `KeyedReadRef` wasn't dropped within the execution) into normal (full) cell dependencies. /// (full) cell dependencies. Called when an execution of `reader` that made `Vc::keyed()` reads
/// Called when an execution of `reader` that made `Vc::keyed()` reads completes, as the ids are /// completes, as the ids are only meaningful within that execution.
/// only meaningful within that execution. ///
/// They remain when a `KeyedReadRef` wasn't dropped within the execution (the manager then also
/// fails the task with a panic error, but its dependencies must still be kept, so it's
/// re-executed on changes), or when its conversion was refused (see
/// `convert_keyed_cell_dependency`).
/// ///
/// A concurrent `update_cell` that collected the [`CellKey::Keyed`] dependent before the /// A concurrent `update_cell` that collected the [`CellKey::Keyed`] dependent before the
/// conversion still invalidates `reader`, as it also accepts a full dependency on the cell when /// conversion still invalidates `reader`, as it also accepts a full dependency on the cell when
......
...@@ -167,6 +167,12 @@ pub fn update_cell( ...@@ -167,6 +167,12 @@ pub fn update_cell(
// yet). So to avoid that we first remove the cell content, invalidate all dependent // yet). So to avoid that we first remove the cell content, invalidate all dependent
// tasks and after that set the new cell content. When the cell content is unset, // tasks and after that set the new cell content. When the cell content is unset,
// readers will wait for it to be set via InProgressCell. // readers will wait for it to be set via InProgressCell.
//
// `convert_keyed_cell_dependency` also relies on this: it refuses to narrow a
// `CellKey::Keyed` dependency while the content is missing. Removing the content under
// the same task lock as collecting the dependents above means a `Keyed` dependent
// collected here can't be replaced by `CellKey::Hash` dependencies before it is
// checked in `invalidate_cell_dependents`.
let old_content = task.remove_cell_data(&cell, &value_type.persistence); let old_content = task.remove_cell_data(&cell, &value_type.persistence);
......
...@@ -125,7 +125,7 @@ pub use crate::{ ...@@ -125,7 +125,7 @@ pub use crate::{
value_type::{Evictability, TraitMethod, TraitType, ValueType, ValueTypePersistence}, value_type::{Evictability, TraitMethod, TraitType, ValueType, ValueTypePersistence},
vc::{ vc::{
CellId, Dynamic, KeyedReadRef, NonLocalValue, OperationValue, OperationVc, OptionVcExt, CellId, Dynamic, KeyedReadRef, NonLocalValue, OperationValue, OperationVc, OptionVcExt,
OrdResolvedVc, RawVc, RawVcUnpacked, ReadKeyedRefVcFuture, ReadRawVcFuture, ReadVcFuture, OrdResolvedVc, RawVc, RawVcUnpacked, ReadRawVcFuture, ReadVcFuture,
ResolveOperationVcFuture, ResolveRawVcFuture, ResolveVcFuture, ResolvedVc, ResolveOperationVcFuture, ResolveRawVcFuture, ResolveVcFuture, ResolvedVc,
ToResolvedVcFuture, Upcast, UpcastStrict, ValueDefault, Vc, VcCast, VcCellCompareMode, ToResolvedVcFuture, Upcast, UpcastStrict, ValueDefault, Vc, VcCast, VcCellCompareMode,
VcCellHashedCompareMode, VcCellKeyedCompareMode, VcCellNewMode, VcDefaultRead, VcRead, VcCellHashedCompareMode, VcCellKeyedCompareMode, VcCellNewMode, VcDefaultRead, VcRead,
......
use std::{ use std::{
borrow::Borrow, borrow::{Borrow, Cow},
cell::Cell, cell::Cell,
cmp::Reverse, cmp::Reverse,
fmt::{Debug, Display}, fmt::{Debug, Display},
...@@ -36,9 +36,10 @@ use crate::{ ...@@ -36,9 +36,10 @@ use crate::{
SharedReference, TaskId, TraitMethod, ValueTypeId, Vc, VcRead, VcValueTrait, VcValueType, SharedReference, TaskId, TraitMethod, ValueTypeId, Vc, VcRead, VcValueTrait, VcValueType,
backend::{ backend::{
Backend, CellContent, CellHash, TaskCollectiblesMap, TaskExecutionSpec, TransientTaskType, Backend, CellContent, CellHash, TaskCollectiblesMap, TaskExecutionSpec, TransientTaskType,
TurboTasksExecutionError, TypedCellContent, VerificationMode, TurboTasksExecutionError, TurboTasksExecutionErrorMessage, TypedCellContent,
VerificationMode,
}, },
capture_future::CaptureFuture, capture_future::{CaptureFuture, TurboTasksPanic},
dyn_task_inputs::DynTaskInputsStorage, dyn_task_inputs::DynTaskInputsStorage,
event::{Event, EventListener}, event::{Event, EventListener},
id::{ExecutionId, LocalTaskId, TraitTypeId}, id::{ExecutionId, LocalTaskId, TraitTypeId},
...@@ -365,7 +366,8 @@ pub enum CellKey { ...@@ -365,7 +366,8 @@ pub enum CellKey {
/// with [`CellKey::Hash`] dependencies of the accessed keys once it's dropped, without /// with [`CellKey::Hash`] dependencies of the accessed keys once it's dropped, without
/// affecting other (full) dependencies on the same cell. `Keyed` dependencies that are still /// affecting other (full) dependencies on the same cell. `Keyed` dependencies that are still
/// present when the execution completes are converted to normal (full) dependencies, so they /// present when the execution completes are converted to normal (full) dependencies, so they
/// never outlive the execution that created them. /// never outlive the execution that created them. (When that's because a `KeyedReadRef` wasn't
/// dropped within the execution, the task also fails.)
Keyed(u32), Keyed(u32),
} }
...@@ -862,15 +864,26 @@ struct CurrentTaskStateInner { ...@@ -862,15 +864,26 @@ struct CurrentTaskStateInner {
/// Counter of [`Vc::keyed`](crate::Vc::keyed) reads in this execution, used to give every /// Counter of [`Vc::keyed`](crate::Vc::keyed) reads in this execution, used to give every
/// [`CellKey::Keyed`] dependency a unique id. /// [`CellKey::Keyed`] dependency a unique id.
keyed_read_counter: AtomicU32, keyed_read_counter: AtomicU32,
/// Number of tracked [`KeyedReadRef`](crate::KeyedReadRef)s of this execution that haven't
/// been dropped within this execution yet. When it isn't zero at the end of the execution, the
/// task fails (see [`KEYED_READ_REF_LEAKED`]).
alive_keyed_read_refs: AtomicU32,
state: RwLock<CurrentTaskState>, state: RwLock<CurrentTaskState>,
} }
/// The panic message of a task execution that didn't drop all of its
/// [`KeyedReadRef`](crate::KeyedReadRef)s.
const KEYED_READ_REF_LEAKED: &str = "A `KeyedReadRef` was not dropped within the task execution \
that created it. It must be dropped before the task function \
returns, and must not be moved to other threads or tasks.";
impl CurrentTaskStateHandle { impl CurrentTaskStateHandle {
fn new(state: CurrentTaskState) -> Self { fn new(state: CurrentTaskState) -> Self {
Self { Self {
inner: Arc::new(CurrentTaskStateInner { inner: Arc::new(CurrentTaskStateInner {
current_task_id: state.task_id, current_task_id: state.task_id,
keyed_read_counter: AtomicU32::new(0), keyed_read_counter: AtomicU32::new(0),
alive_keyed_read_refs: AtomicU32::new(0),
state: RwLock::new(state), state: RwLock::new(state),
}), }),
} }
...@@ -915,6 +928,28 @@ impl KeyedReadDependency { ...@@ -915,6 +928,28 @@ impl KeyedReadDependency {
.flatten() .flatten()
} }
/// Registers a [`KeyedReadRef`](crate::KeyedReadRef) holding this dependency as alive. It
/// must be released with [`KeyedReadDependency::release_alive_ref`] within the same task
/// execution, otherwise the task fails. Only called within the creating execution.
pub(crate) fn register_alive_ref(&self) {
if let Some(execution) = self.execution.upgrade() {
execution
.alive_keyed_read_refs
.fetch_add(1, Ordering::Relaxed);
}
}
/// Releases a [`KeyedReadRef`](crate::KeyedReadRef) registered with
/// [`KeyedReadDependency::register_alive_ref`]. Only called when it is dropped within the
/// creating execution ([`KeyedReadDependency::is_current_execution`]).
pub(crate) fn release_alive_ref(&self) {
if let Some(execution) = self.execution.upgrade() {
execution
.alive_keyed_read_refs
.fetch_sub(1, Ordering::Relaxed);
}
}
/// Whether this is called within the task execution that created the dependency and that /// Whether this is called within the task execution that created the dependency and that
/// execution is still running (it hasn't reported completion to the backend yet). /// execution is still running (it hasn't reported completion to the backend yet).
pub(crate) fn is_current_execution(&self) -> bool { pub(crate) fn is_current_execution(&self) -> bool {
...@@ -1741,8 +1776,10 @@ impl<B: Backend> Executor<TurboTasks<B>, ScheduledTask, TaskPriority> for TurboT ...@@ -1741,8 +1776,10 @@ impl<B: Backend> Executor<TurboTasks<B>, ScheduledTask, TaskPriority> for TurboT
}; };
let finished_state = this.finish_current_task_state(); let finished_state = this.finish_current_task_state();
let (cell_counters, has_keyed_reads) = let (cell_counters, has_keyed_reads, leaked_keyed_read_refs) =
CURRENT_TASK_STATE.with(|ts| { CURRENT_TASK_STATE.with(|ts| {
// Taking the cell counters ends the execution: later
// drops of its `KeyedReadRef`s are no longer counted.
let cell_counters = let cell_counters =
ts.write().unwrap().cell_counters.take().unwrap(); ts.write().unwrap().cell_counters.take().unwrap();
// All reads of this execution are done (local tasks // All reads of this execution are done (local tasks
...@@ -1751,8 +1788,30 @@ impl<B: Backend> Executor<TurboTasks<B>, ScheduledTask, TaskPriority> for TurboT ...@@ -1751,8 +1788,30 @@ impl<B: Backend> Executor<TurboTasks<B>, ScheduledTask, TaskPriority> for TurboT
// `keyed()` was used. // `keyed()` was used.
let has_keyed_reads = let has_keyed_reads =
ts.inner.keyed_read_counter.load(Ordering::Relaxed) > 0; ts.inner.keyed_read_counter.load(Ordering::Relaxed) > 0;
(cell_counters, has_keyed_reads) let leaked_keyed_read_refs = has_keyed_reads
&& ts
.inner
.alive_keyed_read_refs
.load(Ordering::Relaxed)
> 0;
(cell_counters, has_keyed_reads, leaked_keyed_read_refs)
}); });
// A `KeyedReadRef` that outlives the execution is a bug in the
// task function. Fail the task like a panic, also replacing an
// error of the task, so the misuse is never hidden. The backend
// still converts its dependency to a normal one.
let result = if leaked_keyed_read_refs {
Err(TurboTasksExecutionError::Panic(Arc::new(
TurboTasksPanic {
message: TurboTasksExecutionErrorMessage::PIISafe(
Cow::Borrowed(KEYED_READ_REF_LEAKED),
),
location: None,
},
)))
} else {
result
};
this.backend.task_execution_completed( this.backend.task_execution_completed(
task_id, task_id,
result, result,
......
...@@ -26,9 +26,13 @@ use crate::{ ...@@ -26,9 +26,13 @@ use crate::{
/// change. The hashes of the accessed keys are recorded. When it is dropped within the task /// change. The hashes of the accessed keys are recorded. When it is dropped within the task
/// execution that read it, that dependency is replaced (in a single batch operation) with /// execution that read it, that dependency is replaced (in a single batch operation) with
/// dependencies on only the accessed keys, so the task is only invalidated when one of the /// dependencies on only the accessed keys, so the task is only invalidated when one of the
/// accessed keys changes, like with [`Vc::get`](crate::Vc::get). When it is still alive at the end /// accessed keys changes, like with [`Vc::get`](crate::Vc::get).
/// of that task execution or dropped outside of it (e.g. moved to another thread or task), the ///
/// dependency is converted to a normal dependency on the whole cell when the execution completes. /// It **must** be dropped within the task execution that read it (it may be dropped in local
/// tasks of that execution). When it is still alive at the end of that task execution, or
/// dropped outside of it (e.g. moved to another thread or task), the task fails with a panic error
/// (replacing its result) and its dependency on the cell is converted to a normal dependency on
/// the whole cell.
/// ///
/// Note: Like with [`Vc::contains_key`](crate::Vc::contains_key), a /// Note: Like with [`Vc::contains_key`](crate::Vc::contains_key), a
/// [`contains_key`](KeyedReadRef::contains_key) access is also invalidated when the value of the /// [`contains_key`](KeyedReadRef::contains_key) access is also invalidated when the value of the
...@@ -50,10 +54,14 @@ where ...@@ -50,10 +54,14 @@ where
T: VcValueType, T: VcValueType,
VcReadTarget<T>: KeyedEq, VcReadTarget<T>: KeyedEq,
{ {
/// `dependency` must only be `Some` within the task execution that created it.
pub(crate) fn new( pub(crate) fn new(
read_ref: ReadRef<T>, read_ref: ReadRef<T>,
dependency: Option<(TaskId, CellId, KeyedReadDependency)>, dependency: Option<(TaskId, CellId, KeyedReadDependency)>,
) -> Self { ) -> Self {
if let Some((_, _, dependency)) = &dependency {
dependency.register_alive_ref();
}
Self { Self {
read_ref, read_ref,
dependency, dependency,
...@@ -104,9 +112,11 @@ where ...@@ -104,9 +112,11 @@ where
return; return;
}; };
if !dependency.is_current_execution() { if !dependency.is_current_execution() {
// Keep the conservative dependency on the whole cell. // Keep the conservative dependency on the whole cell. The execution fails as it
// still counts this `KeyedReadRef` as alive.
return; return;
} }
dependency.release_alive_ref();
let key_hashes = std::mem::take(self.accessed_key_hashes.get_mut()); let key_hashes = std::mem::take(self.accessed_key_hashes.get_mut());
with_turbo_tasks(|tt| { with_turbo_tasks(|tt| {
tt.convert_keyed_cell_dependency( tt.convert_keyed_cell_dependency(
......
...@@ -40,10 +40,7 @@ pub use crate::vc::{ ...@@ -40,10 +40,7 @@ pub use crate::vc::{
operation::{OperationValue, OperationVc, ResolveOperationVcFuture}, operation::{OperationValue, OperationVc, ResolveOperationVcFuture},
ord::OrdResolvedVc, ord::OrdResolvedVc,
raw::{CellId, RawVc, RawVcUnpacked, ReadRawVcFuture, ResolveRawVcFuture}, raw::{CellId, RawVc, RawVcUnpacked, ReadRawVcFuture, ResolveRawVcFuture},
read::{ read::{ReadOwnedVcFuture, ReadVcFuture, VcDefaultRead, VcRead, VcTransparentRead},
ReadKeyedRefVcFuture, ReadOwnedVcFuture, ReadVcFuture, VcDefaultRead, VcRead,
VcTransparentRead,
},
resolved::ResolvedVc, resolved::ResolvedVc,
traits::{Dynamic, Upcast, UpcastStrict, VcValueTrait, VcValueType}, traits::{Dynamic, Upcast, UpcastStrict, VcValueTrait, VcValueType},
}; };
...@@ -579,11 +576,12 @@ where ...@@ -579,11 +576,12 @@ where
/// [`KeyedReadRef`] (`vc.keyed().await?.get(key)`). This is more efficient than many /// [`KeyedReadRef`] (`vc.keyed().await?.get(key)`). This is more efficient than many
/// [`Vc::get`] / [`Vc::contains_key`] reads of the same cell within a single task. /// [`Vc::get`] / [`Vc::contains_key`] reads of the same cell within a single task.
/// ///
/// Once the [`KeyedReadRef`] is dropped (within the same task execution), the task only /// Once the [`KeyedReadRef`] is dropped, the task only depends on the accessed keys instead of
/// depends on the accessed keys instead of the full value. See [`KeyedReadRef`] for details. /// the full value. It must be dropped within the same task execution, otherwise the task
pub fn keyed(self) -> ReadKeyedRefVcFuture<T> { /// fails. See [`KeyedReadRef`] for details.
pub async fn keyed(self) -> Result<KeyedReadRef<T>> {
let future: ReadVcFuture<T> = self.node.into_read().into(); let future: ReadVcFuture<T> = self.node.into_read().into();
future.keyed() future.keyed().await
} }
} }
......
use std::{ use std::{
any::Any, any::Any,
future::poll_fn,
hash::{BuildHasher, Hash}, hash::{BuildHasher, Hash},
marker::PhantomData, marker::PhantomData,
mem::ManuallyDrop, mem::ManuallyDrop,
...@@ -229,11 +230,40 @@ where ...@@ -229,11 +230,40 @@ where
/// Read the value once for many synchronous keyed accesses via the returned /// Read the value once for many synchronous keyed accesses via the returned
/// [`KeyedReadRef`]. Once that is dropped, the task only depends on the accessed keys instead /// [`KeyedReadRef`]. Once that is dropped, the task only depends on the accessed keys instead
/// of the full value. See [`KeyedReadRef`] for details. /// of the full value. See [`KeyedReadRef`] for details.
pub fn keyed(self) -> ReadKeyedRefVcFuture<T> { pub async fn keyed(self) -> Result<KeyedReadRef<T>> {
ReadKeyedRefVcFuture { let mut raw = self.raw;
future: self, // Allocated when the future is first polled, as that's the task execution that reads the
dependency: KeyedDependencyState::NotStarted, // cell (not necessarily the one that created the future).
let mut dependency = KeyedReadDependency::new_for_current_task();
if let Some(dependency) = &dependency {
raw.set_track_keyed(dependency.id);
} }
let content = poll_fn(|cx| {
if dependency
.as_ref()
.is_some_and(|dependency| !dependency.is_current_execution())
{
// The future moved to a different task execution while pending. The id belongs to
// the other execution and could collide with an id of this one, so fall back to a
// normal full dependency and never narrow it.
raw.set_track_full();
dependency = None;
}
Pin::new(&mut raw).poll(cx)
})
.await?;
let read_ref = VcValueTypeCast::<T>::cast(content)?;
// Only narrow the dependency later when the read was completed in the execution that
// registered it. Otherwise the `KeyedReadRef` keeps the conservative dependency.
let dependency = dependency
.filter(|dependency| dependency.is_current_execution())
.map(|dependency| {
let Some((task, index)) = raw.resolved_cell() else {
unreachable!("a successful read is always resolved to a cell")
};
(task, index, dependency)
});
Ok(KeyedReadRef::new(read_ref, dependency))
} }
} }
...@@ -344,88 +374,6 @@ where ...@@ -344,88 +374,6 @@ where
} }
} }
/// The [`KeyedReadDependency`] of a [`ReadKeyedRefVcFuture`]. It's allocated when the future is
/// first polled, as that's the task execution that reads the cell (not necessarily the one that
/// created the future).
enum KeyedDependencyState {
NotStarted,
/// `None` if the read isn't a [`CellKey::Keyed`](crate::CellKey::Keyed) read (no current
/// task, or the dependency was already handed to the [`KeyedReadRef`]).
Started(Option<KeyedReadDependency>),
}
pin_project! {
/// A future returned by [`Vc::keyed`](crate::Vc::keyed).
#[must_use]
pub struct ReadKeyedRefVcFuture<T>
where
T: VcValueType,
VcReadTarget<T>: KeyedEq,
{
#[pin]
future: ReadVcFuture<T, VcValueTypeCast<T>>,
dependency: KeyedDependencyState,
}
}
impl<T> Future for ReadKeyedRefVcFuture<T>
where
T: VcValueType,
VcReadTarget<T>: KeyedEq,
{
type Output = Result<KeyedReadRef<T>>;
fn poll(self: Pin<&mut Self>, cx: &mut std::task::Context<'_>) -> Poll<Self::Output> {
let mut this = self.project();
match this.dependency {
KeyedDependencyState::NotStarted => {
let dependency = KeyedReadDependency::new_for_current_task();
if let Some(dependency) = &dependency {
// Safety: Only a field is mutated, nothing is moved.
unsafe { this.future.as_mut().get_unchecked_mut() }
.raw
.set_track_keyed(dependency.id);
}
*this.dependency = KeyedDependencyState::Started(dependency);
}
KeyedDependencyState::Started(dependency) => {
if dependency
.as_ref()
.is_some_and(|dependency| !dependency.is_current_execution())
{
// The future moved to a different task execution while pending. The id
// belongs to the other execution and could collide with an id of this one, so
// fall back to a normal full dependency and never narrow it.
// Safety: Only a field is mutated, nothing is moved.
unsafe { this.future.as_mut().get_unchecked_mut() }
.raw
.set_track_full();
*dependency = None;
}
}
}
let result = std::task::ready!(this.future.as_mut().poll(cx));
Poll::Ready(result.map(|read_ref| {
let KeyedDependencyState::Started(dependency) = this.dependency else {
unreachable!("the dependency state is initialized above")
};
// Only narrow the dependency later when the read was completed in the execution that
// registered it. Otherwise the `KeyedReadRef` keeps the conservative dependency.
let dependency = dependency
.take()
.filter(|dependency| dependency.is_current_execution())
.map(|dependency| {
let Some((task, index)) = this.future.as_ref().get_ref().raw.resolved_cell()
else {
unreachable!("a successful read is always resolved to a cell")
};
(task, index, dependency)
});
KeyedReadRef::new(read_ref, dependency)
}))
}
}
pin_project! { pin_project! {
pub struct ReadContainsKeyedVcFuture<'l, T, Q> pub struct ReadContainsKeyedVcFuture<'l, T, Q>
where where
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment