Unverified Commit 10ed52fb authored by vercel-fleet-prod[bot]'s avatar vercel-fleet-prod[bot] Committed by GitHub

Fail tasks that leak KeyedReadRefs and simplify keyed reads

Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent 4bcaed44
......@@ -16,5 +16,5 @@ These rules apply throughout this directory tree when developing with turbo_task
- Existing `.cell()` calls or wrapping tasks are not automatically good precedent. Verify their caller and cache-key assumptions before copying the pattern.
- Put maps containing information for a large part of the build (for example, a map from modules to analysis results) into separate cells. Use `cell = "keyed"` on their value types, for example `#[turbo_tasks::value(transparent, cell = "keyed")]` for a map wrapper.
- Read keyed cells through keyed access, such as `map.get(&key).await?`, so invalidation depends only on the entries actually read. Awaiting the entire map and then looking up a key creates a dependency on the whole map instead.
- When a task reads many keys of the same keyed cell, use `let map = map.keyed().await?;` and then the synchronous `map.get(&key)` / `map.contains_key(&key)`. It reads the cell once and, when the `KeyedReadRef` is dropped within the task, narrows the dependency to the accessed keys in one batch.
- When a task reads many keys of the same keyed cell, use `let map = map.keyed().await?;` and then the synchronous `map.get(&key)` / `map.contains_key(&key)`. It reads the cell once and, when the `KeyedReadRef` is dropped, narrows the dependency to the accessed keys in one batch. The `KeyedReadRef` must be dropped within the task execution: don't return, store, leak, or move it to other threads or tasks. Otherwise the task fails with a panic error.
- When changing task boundaries, check the intended invalidation behavior. An unrelated module change should not invalidate consumers of unchanged keyed entries; correct output alone does not demonstrate good cache granularity.
......@@ -1245,10 +1245,14 @@ impl TurboTasksBackend {
}
}
/// Converts all [`CellKey::Keyed`] dependencies of `reader` that are still present (their
/// `KeyedReadRef` wasn't dropped within the execution) into normal (full) cell dependencies.
/// Called when an execution of `reader` that made `Vc::keyed()` reads completes, as the ids are
/// only meaningful within that execution.
/// Converts all [`CellKey::Keyed`] dependencies of `reader` that are still present into normal
/// (full) cell dependencies. Called when an execution of `reader` that made `Vc::keyed()` reads
/// completes, as the ids are only meaningful within that execution.
///
/// They remain when a `KeyedReadRef` wasn't dropped within the execution (the manager then also
/// fails the task with a panic error, but its dependencies must still be kept, so it's
/// re-executed on changes), or when its conversion was refused (see
/// `convert_keyed_cell_dependency`).
///
/// A concurrent `update_cell` that collected the [`CellKey::Keyed`] dependent before the
/// conversion still invalidates `reader`, as it also accepts a full dependency on the cell when
......
......@@ -167,6 +167,12 @@ pub fn update_cell(
// yet). So to avoid that we first remove the cell content, invalidate all dependent
// tasks and after that set the new cell content. When the cell content is unset,
// readers will wait for it to be set via InProgressCell.
//
// `convert_keyed_cell_dependency` also relies on this: it refuses to narrow a
// `CellKey::Keyed` dependency while the content is missing. Removing the content under
// the same task lock as collecting the dependents above means a `Keyed` dependent
// collected here can't be replaced by `CellKey::Hash` dependencies before it is
// checked in `invalidate_cell_dependents`.
let old_content = task.remove_cell_data(&cell, &value_type.persistence);
......
......@@ -125,7 +125,7 @@ pub use crate::{
value_type::{Evictability, TraitMethod, TraitType, ValueType, ValueTypePersistence},
vc::{
CellId, Dynamic, KeyedReadRef, NonLocalValue, OperationValue, OperationVc, OptionVcExt,
OrdResolvedVc, RawVc, RawVcUnpacked, ReadKeyedRefVcFuture, ReadRawVcFuture, ReadVcFuture,
OrdResolvedVc, RawVc, RawVcUnpacked, ReadRawVcFuture, ReadVcFuture,
ResolveOperationVcFuture, ResolveRawVcFuture, ResolveVcFuture, ResolvedVc,
ToResolvedVcFuture, Upcast, UpcastStrict, ValueDefault, Vc, VcCast, VcCellCompareMode,
VcCellHashedCompareMode, VcCellKeyedCompareMode, VcCellNewMode, VcDefaultRead, VcRead,
......
use std::{
borrow::Borrow,
borrow::{Borrow, Cow},
cell::Cell,
cmp::Reverse,
fmt::{Debug, Display},
......@@ -36,9 +36,10 @@ use crate::{
SharedReference, TaskId, TraitMethod, ValueTypeId, Vc, VcRead, VcValueTrait, VcValueType,
backend::{
Backend, CellContent, CellHash, TaskCollectiblesMap, TaskExecutionSpec, TransientTaskType,
TurboTasksExecutionError, TypedCellContent, VerificationMode,
TurboTasksExecutionError, TurboTasksExecutionErrorMessage, TypedCellContent,
VerificationMode,
},
capture_future::CaptureFuture,
capture_future::{CaptureFuture, TurboTasksPanic},
dyn_task_inputs::DynTaskInputsStorage,
event::{Event, EventListener},
id::{ExecutionId, LocalTaskId, TraitTypeId},
......@@ -365,7 +366,8 @@ pub enum CellKey {
/// with [`CellKey::Hash`] dependencies of the accessed keys once it's dropped, without
/// affecting other (full) dependencies on the same cell. `Keyed` dependencies that are still
/// present when the execution completes are converted to normal (full) dependencies, so they
/// never outlive the execution that created them.
/// never outlive the execution that created them. (When that's because a `KeyedReadRef` wasn't
/// dropped within the execution, the task also fails.)
Keyed(u32),
}
......@@ -862,15 +864,26 @@ struct CurrentTaskStateInner {
/// Counter of [`Vc::keyed`](crate::Vc::keyed) reads in this execution, used to give every
/// [`CellKey::Keyed`] dependency a unique id.
keyed_read_counter: AtomicU32,
/// Number of tracked [`KeyedReadRef`](crate::KeyedReadRef)s of this execution that haven't
/// been dropped within this execution yet. When it isn't zero at the end of the execution, the
/// task fails (see [`KEYED_READ_REF_LEAKED`]).
alive_keyed_read_refs: AtomicU32,
state: RwLock<CurrentTaskState>,
}
/// The panic message of a task execution that didn't drop all of its
/// [`KeyedReadRef`](crate::KeyedReadRef)s.
const KEYED_READ_REF_LEAKED: &str = "A `KeyedReadRef` was not dropped within the task execution \
that created it. It must be dropped before the task function \
returns, and must not be moved to other threads or tasks.";
impl CurrentTaskStateHandle {
fn new(state: CurrentTaskState) -> Self {
Self {
inner: Arc::new(CurrentTaskStateInner {
current_task_id: state.task_id,
keyed_read_counter: AtomicU32::new(0),
alive_keyed_read_refs: AtomicU32::new(0),
state: RwLock::new(state),
}),
}
......@@ -915,6 +928,28 @@ impl KeyedReadDependency {
.flatten()
}
/// Registers a [`KeyedReadRef`](crate::KeyedReadRef) holding this dependency as alive. It
/// must be released with [`KeyedReadDependency::release_alive_ref`] within the same task
/// execution, otherwise the task fails. Only called within the creating execution.
pub(crate) fn register_alive_ref(&self) {
if let Some(execution) = self.execution.upgrade() {
execution
.alive_keyed_read_refs
.fetch_add(1, Ordering::Relaxed);
}
}
/// Releases a [`KeyedReadRef`](crate::KeyedReadRef) registered with
/// [`KeyedReadDependency::register_alive_ref`]. Only called when it is dropped within the
/// creating execution ([`KeyedReadDependency::is_current_execution`]).
pub(crate) fn release_alive_ref(&self) {
if let Some(execution) = self.execution.upgrade() {
execution
.alive_keyed_read_refs
.fetch_sub(1, Ordering::Relaxed);
}
}
/// Whether this is called within the task execution that created the dependency and that
/// execution is still running (it hasn't reported completion to the backend yet).
pub(crate) fn is_current_execution(&self) -> bool {
......@@ -1741,8 +1776,10 @@ impl<B: Backend> Executor<TurboTasks<B>, ScheduledTask, TaskPriority> for TurboT
};
let finished_state = this.finish_current_task_state();
let (cell_counters, has_keyed_reads) =
let (cell_counters, has_keyed_reads, leaked_keyed_read_refs) =
CURRENT_TASK_STATE.with(|ts| {
// Taking the cell counters ends the execution: later
// drops of its `KeyedReadRef`s are no longer counted.
let cell_counters =
ts.write().unwrap().cell_counters.take().unwrap();
// All reads of this execution are done (local tasks
......@@ -1751,8 +1788,30 @@ impl<B: Backend> Executor<TurboTasks<B>, ScheduledTask, TaskPriority> for TurboT
// `keyed()` was used.
let has_keyed_reads =
ts.inner.keyed_read_counter.load(Ordering::Relaxed) > 0;
(cell_counters, has_keyed_reads)
let leaked_keyed_read_refs = has_keyed_reads
&& ts
.inner
.alive_keyed_read_refs
.load(Ordering::Relaxed)
> 0;
(cell_counters, has_keyed_reads, leaked_keyed_read_refs)
});
// A `KeyedReadRef` that outlives the execution is a bug in the
// task function. Fail the task like a panic, also replacing an
// error of the task, so the misuse is never hidden. The backend
// still converts its dependency to a normal one.
let result = if leaked_keyed_read_refs {
Err(TurboTasksExecutionError::Panic(Arc::new(
TurboTasksPanic {
message: TurboTasksExecutionErrorMessage::PIISafe(
Cow::Borrowed(KEYED_READ_REF_LEAKED),
),
location: None,
},
)))
} else {
result
};
this.backend.task_execution_completed(
task_id,
result,
......
......@@ -26,9 +26,13 @@ use crate::{
/// change. The hashes of the accessed keys are recorded. When it is dropped within the task
/// execution that read it, that dependency is replaced (in a single batch operation) with
/// dependencies on only the accessed keys, so the task is only invalidated when one of the
/// accessed keys changes, like with [`Vc::get`](crate::Vc::get). When it is still alive at the end
/// of that task execution or dropped outside of it (e.g. moved to another thread or task), the
/// dependency is converted to a normal dependency on the whole cell when the execution completes.
/// accessed keys changes, like with [`Vc::get`](crate::Vc::get).
///
/// It **must** be dropped within the task execution that read it (it may be dropped in local
/// tasks of that execution). When it is still alive at the end of that task execution, or
/// dropped outside of it (e.g. moved to another thread or task), the task fails with a panic error
/// (replacing its result) and its dependency on the cell is converted to a normal dependency on
/// the whole cell.
///
/// Note: Like with [`Vc::contains_key`](crate::Vc::contains_key), a
/// [`contains_key`](KeyedReadRef::contains_key) access is also invalidated when the value of the
......@@ -50,10 +54,14 @@ where
T: VcValueType,
VcReadTarget<T>: KeyedEq,
{
/// `dependency` must only be `Some` within the task execution that created it.
pub(crate) fn new(
read_ref: ReadRef<T>,
dependency: Option<(TaskId, CellId, KeyedReadDependency)>,
) -> Self {
if let Some((_, _, dependency)) = &dependency {
dependency.register_alive_ref();
}
Self {
read_ref,
dependency,
......@@ -104,9 +112,11 @@ where
return;
};
if !dependency.is_current_execution() {
// Keep the conservative dependency on the whole cell.
// Keep the conservative dependency on the whole cell. The execution fails as it
// still counts this `KeyedReadRef` as alive.
return;
}
dependency.release_alive_ref();
let key_hashes = std::mem::take(self.accessed_key_hashes.get_mut());
with_turbo_tasks(|tt| {
tt.convert_keyed_cell_dependency(
......
......@@ -40,10 +40,7 @@ pub use crate::vc::{
operation::{OperationValue, OperationVc, ResolveOperationVcFuture},
ord::OrdResolvedVc,
raw::{CellId, RawVc, RawVcUnpacked, ReadRawVcFuture, ResolveRawVcFuture},
read::{
ReadKeyedRefVcFuture, ReadOwnedVcFuture, ReadVcFuture, VcDefaultRead, VcRead,
VcTransparentRead,
},
read::{ReadOwnedVcFuture, ReadVcFuture, VcDefaultRead, VcRead, VcTransparentRead},
resolved::ResolvedVc,
traits::{Dynamic, Upcast, UpcastStrict, VcValueTrait, VcValueType},
};
......@@ -579,11 +576,12 @@ where
/// [`KeyedReadRef`] (`vc.keyed().await?.get(key)`). This is more efficient than many
/// [`Vc::get`] / [`Vc::contains_key`] reads of the same cell within a single task.
///
/// Once the [`KeyedReadRef`] is dropped (within the same task execution), the task only
/// depends on the accessed keys instead of the full value. See [`KeyedReadRef`] for details.
pub fn keyed(self) -> ReadKeyedRefVcFuture<T> {
/// Once the [`KeyedReadRef`] is dropped, the task only depends on the accessed keys instead of
/// the full value. It must be dropped within the same task execution, otherwise the task
/// fails. See [`KeyedReadRef`] for details.
pub async fn keyed(self) -> Result<KeyedReadRef<T>> {
let future: ReadVcFuture<T> = self.node.into_read().into();
future.keyed()
future.keyed().await
}
}
......
use std::{
any::Any,
future::poll_fn,
hash::{BuildHasher, Hash},
marker::PhantomData,
mem::ManuallyDrop,
......@@ -229,11 +230,40 @@ where
/// Read the value once for many synchronous keyed accesses via the returned
/// [`KeyedReadRef`]. Once that is dropped, the task only depends on the accessed keys instead
/// of the full value. See [`KeyedReadRef`] for details.
pub fn keyed(self) -> ReadKeyedRefVcFuture<T> {
ReadKeyedRefVcFuture {
future: self,
dependency: KeyedDependencyState::NotStarted,
pub async fn keyed(self) -> Result<KeyedReadRef<T>> {
let mut raw = self.raw;
// Allocated when the future is first polled, as that's the task execution that reads the
// cell (not necessarily the one that created the future).
let mut dependency = KeyedReadDependency::new_for_current_task();
if let Some(dependency) = &dependency {
raw.set_track_keyed(dependency.id);
}
let content = poll_fn(|cx| {
if dependency
.as_ref()
.is_some_and(|dependency| !dependency.is_current_execution())
{
// The future moved to a different task execution while pending. The id belongs to
// the other execution and could collide with an id of this one, so fall back to a
// normal full dependency and never narrow it.
raw.set_track_full();
dependency = None;
}
Pin::new(&mut raw).poll(cx)
})
.await?;
let read_ref = VcValueTypeCast::<T>::cast(content)?;
// Only narrow the dependency later when the read was completed in the execution that
// registered it. Otherwise the `KeyedReadRef` keeps the conservative dependency.
let dependency = dependency
.filter(|dependency| dependency.is_current_execution())
.map(|dependency| {
let Some((task, index)) = raw.resolved_cell() else {
unreachable!("a successful read is always resolved to a cell")
};
(task, index, dependency)
});
Ok(KeyedReadRef::new(read_ref, dependency))
}
}
......@@ -344,88 +374,6 @@ where
}
}
/// The [`KeyedReadDependency`] of a [`ReadKeyedRefVcFuture`]. It's allocated when the future is
/// first polled, as that's the task execution that reads the cell (not necessarily the one that
/// created the future).
enum KeyedDependencyState {
NotStarted,
/// `None` if the read isn't a [`CellKey::Keyed`](crate::CellKey::Keyed) read (no current
/// task, or the dependency was already handed to the [`KeyedReadRef`]).
Started(Option<KeyedReadDependency>),
}
pin_project! {
/// A future returned by [`Vc::keyed`](crate::Vc::keyed).
#[must_use]
pub struct ReadKeyedRefVcFuture<T>
where
T: VcValueType,
VcReadTarget<T>: KeyedEq,
{
#[pin]
future: ReadVcFuture<T, VcValueTypeCast<T>>,
dependency: KeyedDependencyState,
}
}
impl<T> Future for ReadKeyedRefVcFuture<T>
where
T: VcValueType,
VcReadTarget<T>: KeyedEq,
{
type Output = Result<KeyedReadRef<T>>;
fn poll(self: Pin<&mut Self>, cx: &mut std::task::Context<'_>) -> Poll<Self::Output> {
let mut this = self.project();
match this.dependency {
KeyedDependencyState::NotStarted => {
let dependency = KeyedReadDependency::new_for_current_task();
if let Some(dependency) = &dependency {
// Safety: Only a field is mutated, nothing is moved.
unsafe { this.future.as_mut().get_unchecked_mut() }
.raw
.set_track_keyed(dependency.id);
}
*this.dependency = KeyedDependencyState::Started(dependency);
}
KeyedDependencyState::Started(dependency) => {
if dependency
.as_ref()
.is_some_and(|dependency| !dependency.is_current_execution())
{
// The future moved to a different task execution while pending. The id
// belongs to the other execution and could collide with an id of this one, so
// fall back to a normal full dependency and never narrow it.
// Safety: Only a field is mutated, nothing is moved.
unsafe { this.future.as_mut().get_unchecked_mut() }
.raw
.set_track_full();
*dependency = None;
}
}
}
let result = std::task::ready!(this.future.as_mut().poll(cx));
Poll::Ready(result.map(|read_ref| {
let KeyedDependencyState::Started(dependency) = this.dependency else {
unreachable!("the dependency state is initialized above")
};
// Only narrow the dependency later when the read was completed in the execution that
// registered it. Otherwise the `KeyedReadRef` keeps the conservative dependency.
let dependency = dependency
.take()
.filter(|dependency| dependency.is_current_execution())
.map(|dependency| {
let Some((task, index)) = this.future.as_ref().get_ref().raw.resolved_cell()
else {
unreachable!("a successful read is always resolved to a cell")
};
(task, index, dependency)
});
KeyedReadRef::new(read_ref, dependency)
}))
}
}
pin_project! {
pub struct ReadContainsKeyedVcFuture<'l, T, Q>
where
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment