Unverified Commit 33b7edfc authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

turbo-tasks: execute scheduled tasks inline when they are read (#96808)

### What?

Two changes to how the async turbo-tasks engine handles a read whose
value isn't ready yet:

1. When a read finds its task merely *scheduled* (queued, not started),
it takes the task out of
the scheduler queue and executes it on the reading thread instead of
waiting for a worker to
pick it up. If the execution completes without yielding, the read
returns synchronously; if it
yields, the partially-polled execution is handed to tokio and the read
waits as before.
2. When a read finds its task's execution already **in progress** (a
worker is running it), it now
waits directly on the task's completion event instead of also attempting
to take it over —
avoiding a futile acquisition of the scheduler's queue lock, which is
otherwise the most
   contended lock in the system.

An inline execution is visible in traces: the executed task's span
carries
`inline_execution = "complete"` when the reader's poll finished it, or
`"partial"` when it yielded
and was handed to the runtime. A task a worker executed leaves the field
unset.

Optional diagnostics (`InlineExecutionStats`) count queue pushes, claim
attempts and their outcomes,
and reads that waited for an already-running task. They are behind the
`inline_execution_stats` Cargo
feature, off by default, so a normal build has no counters, no atomics
and no extra fields.

### Why?

A large share of read misses target tasks that have merely been
scheduled, not tasks actually
being computed elsewhere. Parking on those adds two avoidable thread
hops (schedule → worker
pickup → wake) for work the reader could simply do itself. Conversely, a
read of a task a worker
is already executing gains nothing from trying to claim it, so it
shouldn't pay for trying.

### How?

- `PriorityRunner` gained a keyed claim: queued entries are indexed by a
`ScheduleKey` (task or
local task) and stored in a slot store, so one entry can be removed by
key under the existing
queue lock while the priority heap keeps a tombstone that a popping
worker skips. Every scheduled
  execution still runs exactly once.
- The backend already distinguished a queued task from one already
executing
(`InProgressState::Scheduled` vs. `InProgress`) when building a listener
for a read, and
discarded that distinction. It's now threaded through as `ReadOutcome<T>
= Value(T) |
Scheduled(EventListener) | InProgress(EventListener)`; only `Scheduled`
reads attempt a claim.
The state is a hint — a worker pops a task off the queue before it marks
it started, so a read can
see `Scheduled` for a task that is no longer claimable. Acting on a
stale hint costs one failed
  claim, never correctness.
- Inline execution nests (A reads B inline, B reads C inline, ...), so
it's capped at 16 levels per
thread to bound stack growth; at the cap, reads fall back to waiting for
a worker as before. The
cap is a thread-local counter because it guards the *thread* stack, not
the task stack.
- Recording the outcome on the *executed task's* span needs a small
handoff: when
`poll_once_or_spawn` returns, that task's span has already been exited,
so the executor registers
the span it instruments the task body with in a slot that only exists
while a claimed task is
  polled inline.
- Added `turbo-tasks-backend/tests/inline_read_execution.rs` (8 tests),
`inline_execution_span.rs`
and `inline_execution_span_worker.rs`, plus unit tests in
`priority_runner.rs`/`manager.rs`
covering keyed claim/exactly-once, inline completion for global and
local tasks, the
in-progress/scheduled distinction, nested inline executions, restoring
from a persistent cache,
  and a deep dependency chain (stack-depth guard).

### Benchmarks

`bench/nested-deps-app-router-many-pages` (1000 pages, 3020 generated
components, 3010 routes),
release builds of `@next/swc` for this branch and for its merge base,
runs interleaved, first run of
each variant discarded as warm-up. Shared 8-vCPU VM.

**Cold `next build --turbopack`** — 5 measured runs per variant:

| metric | canary (median) | this branch (median) | delta |
| --- | ---: | ---: | ---: |
| total build | 96.70 s | 89.29 s | **−7.7 %** |
| turbopack compile phase | 68 s | 61 s | **−10.3 %** |

Raw totals (ms) — canary: 93566, 95354, 96702, 96717, 97545; branch:
89070, 89140, 89287, 89892,
90462. The distributions don't overlap (the branch is faster in all 25
pairwise comparisons, exact
two-sided Mann-Whitney p ≈ 0.008), and the 7.4 s median gap is about
twice canary's own 4.0 s
run-to-run spread.

**Incremental build** (second build reusing the persistent cache) — 3
measured runs per variant:
36.14 s vs 37.29 s median, compile phase 10.4 s vs 10.5 s. The 1.2 s gap
is *smaller* than canary's
own 1.2 s spread, so this is **no measurable difference** — the
incremental case is dominated by
restore and by static generation in Node workers rather than by the
scheduling this PR changes.

**Cache-hit reads** (`task_overhead/turbo-cached-*`, criterion): no
regression. Measured before the
rebase, on the same read-path code — the hit path never enters this
code, since the inline attempt
only happens on the branch where a read reports a miss.

Caveats: one app and one workload shape, on a shared VM, so only the
relative comparison on the same
machine is meaningful, not the absolute seconds.

<!-- NEXT_JS_LLM -->
Co-authored-by: 's avatarLuke Sandberg <210140+lukesandberg@users.noreply.github.com>

---------
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: 's avatarLuke Sandberg <210140+lukesandberg@users.noreply.github.com>
parent 4b4b878c
......@@ -77,6 +77,9 @@ thiserror = { workspace = true }
turbo-tasks-malloc = { workspace = true , features = ["custom_allocator"]}
rstest = { workspace = true }
turbo-tasks-testing = { workspace = true }
# The read-outcome tests assert on the counters, so they need them compiled in. Only for tests —
# normal builds of this crate get the default (disabled) turbo-tasks.
turbo-tasks = { workspace = true, features = ["inline_execution_stats"] }
[[bench]]
......
......@@ -31,9 +31,9 @@ use tracing::{Span, field::display, trace_span};
use turbo_bincode::{TurboBincodeBuffer, new_turbo_bincode_decoder, new_turbo_bincode_encoder};
use turbo_tasks::{
CellId, DynTaskInputsStorage, RawVc, RawVcUnpacked, ReadCellOptions, ReadCellTracking,
ReadConsistency, ReadOutputOptions, ReadTracking, SharedReference, TRANSIENT_TASK_BIT,
TaskExecutionReason, TaskId, TaskPersistence, TaskPriority, TraitTypeId, TurboTasks,
TurboTasksCallApi, TurboTasksPanic, ValueTypeId,
ReadConsistency, ReadOutcome, ReadOutputOptions, ReadTracking, SharedReference,
TRANSIENT_TASK_BIT, TaskExecutionReason, TaskId, TaskPersistence, TaskPriority, TraitTypeId,
TurboTasks, TurboTasksCallApi, TurboTasksPanic, ValueTypeId,
backend::{
Backend, CachedTaskType, CachedTaskTypeArc, CellContent, CellHash, TaskExecutionSpec,
TransientTaskType, TurboTaskContextError, TurboTaskLocalContextError, TurboTasksError,
......@@ -486,7 +486,7 @@ impl TurboTasksBackend {
reader: Option<TaskId>,
options: ReadOutputOptions,
turbo_tasks: &TurboTasks<TurboTasksBackend>,
) -> Result<Result<RawVc, EventListener>> {
) -> Result<ReadOutcome<RawVc>> {
self.assert_not_persistent_calling_transient(reader, task_id, /* cell_id */ None);
let mut ctx = self.execute_context(turbo_tasks);
......@@ -518,19 +518,25 @@ impl TurboTasksBackend {
})
}
fn check_in_progress(
/// Reports whether the task is already being computed, and — crucially for the reader —
/// whether a worker has actually started it. A task that is only `Scheduled` can be taken
/// over and executed by the reader; one that is `InProgress` can only be waited for.
fn check_in_progress<T>(
task: &impl TaskGuard,
reader_description: Option<EventDescription>,
tracking: ReadTracking,
) -> Option<std::result::Result<std::result::Result<RawVc, EventListener>, anyhow::Error>>
{
) -> Option<Result<ReadOutcome<T>>> {
match task.get_in_progress() {
Some(InProgressState::Scheduled { done_event, .. }) => Some(Ok(Err(
listen_to_done_event(reader_description, tracking, done_event),
))),
Some(InProgressState::Scheduled { done_event, .. }) => {
Some(Ok(ReadOutcome::Scheduled(listen_to_done_event(
reader_description,
tracking,
done_event,
))))
}
Some(InProgressState::InProgress(box InProgressStateInner {
done_event, ..
})) => Some(Ok(Err(listen_to_done_event(
})) => Some(Ok(ReadOutcome::InProgress(listen_to_done_event(
reader_description,
tracking,
done_event,
......@@ -704,7 +710,7 @@ impl TurboTasksBackend {
queue.execute(&mut ctx);
}
return Ok(Err(listener));
return Ok(ReadOutcome::InProgress(listener));
}
}
......@@ -723,8 +729,8 @@ impl TurboTasksBackend {
if let Some(output) = task.get_output() {
let result = match output {
OutputValue::Cell(cell) => Ok(Ok(RawVc::task_cell(cell.task, cell.cell))),
OutputValue::Output(task) => Ok(Ok(RawVc::task_output(*task))),
OutputValue::Cell(cell) => Ok(RawVc::task_cell(cell.task, cell.cell)),
OutputValue::Output(task) => Ok(RawVc::task_output(*task)),
OutputValue::Error(error) => Err(error.clone()),
};
if let Some(mut reader_task) = reader_task.take()
......@@ -770,7 +776,7 @@ impl TurboTasksBackend {
drop(task);
}
return result.map_err(|error| {
return result.map(ReadOutcome::Value).map_err(|error| {
self.task_error_to_turbo_tasks_execution_error(&error, &mut ctx)
.with_task_context(task_id, turbo_tasks.pin())
.into()
......@@ -801,7 +807,7 @@ impl TurboTasksBackend {
debug_assert!(old.is_none(), "InProgress already exists");
ctx.schedule_task(task, TaskPriority::Recomputation);
Ok(Err(listener))
Ok(ReadOutcome::Scheduled(listener))
}
fn try_read_task_cell(
......@@ -811,7 +817,7 @@ impl TurboTasksBackend {
cell: CellId,
options: ReadCellOptions,
turbo_tasks: &TurboTasks<TurboTasksBackend>,
) -> Result<Result<TypedCellContent, EventListener>> {
) -> Result<ReadOutcome<TypedCellContent>> {
self.assert_not_persistent_calling_transient(reader, task_id, Some(cell));
fn add_cell_dependency(
......@@ -876,7 +882,7 @@ impl TurboTasksBackend {
if tracking.should_track(false) {
add_cell_dependency(task_id, task, reader, reader_task, cell, tracking.key());
}
return Ok(Ok(TypedCellContent(
return Ok(ReadOutcome::Value(TypedCellContent(
cell.type_id(),
CellContent(Some(content)),
)));
......@@ -887,9 +893,17 @@ impl TurboTasksBackend {
in_progress,
Some(InProgressState::InProgress(..) | InProgressState::Scheduled { .. })
) {
return Ok(Err(self
// Tell the reader whether the task is merely queued (it may take it over and execute it
// itself) or already being executed by a worker (it can only wait).
let started = matches!(in_progress, Some(InProgressState::InProgress(..)));
let listener = self
.listen_to_cell(&mut task, task_id, need_reader_task, &reader_task, cell)
.0));
.0;
return Ok(if started {
ReadOutcome::InProgress(listener)
} else {
ReadOutcome::Scheduled(listener)
});
}
let is_cancelled = matches!(in_progress, Some(InProgressState::Canceled));
......@@ -926,7 +940,8 @@ impl TurboTasksBackend {
self.listen_to_cell(&mut task, task_id, need_reader_task, &reader_task, cell);
drop(reader_task);
if !new_listener {
return Ok(Err(listener));
// Somebody else is already waiting for this cell, so the task is being taken care of.
return Ok(ReadOutcome::InProgress(listener));
}
let _span = tracing::trace_span!(
......@@ -942,7 +957,7 @@ impl TurboTasksBackend {
);
ctx.schedule_task(task, TaskPriority::Recomputation);
Ok(Err(listener))
Ok(ReadOutcome::Scheduled(listener))
}
fn listen_to_cell(
......@@ -1982,7 +1997,12 @@ impl TurboTasksBackend {
)
}
TaskType::Transient(task_type) => {
let span = tracing::trace_span!("turbo_tasks::root_task");
// `inline_execution` is recorded when a read executed this task on its own thread,
// see `NativeFunction::span`.
let span = tracing::trace_span!(
"turbo_tasks::root_task",
inline_execution = tracing::field::Empty
);
let future = match &*task_type {
TransientTask::Root(f) => f(),
TransientTask::Once(future_mutex) => take(&mut *future_mutex.lock())?,
......@@ -3662,7 +3682,7 @@ impl Backend for TurboTasksBackend {
reader: Option<TaskId>,
options: ReadOutputOptions,
turbo_tasks: &TurboTasks<Self>,
) -> Result<Result<RawVc, EventListener>> {
) -> Result<ReadOutcome<RawVc>> {
self.try_read_task_output(task_id, reader, options, turbo_tasks)
}
......@@ -3673,7 +3693,7 @@ impl Backend for TurboTasksBackend {
reader: Option<TaskId>,
options: ReadCellOptions,
turbo_tasks: &TurboTasks<Self>,
) -> Result<Result<TypedCellContent, EventListener>> {
) -> Result<ReadOutcome<TypedCellContent>> {
self.try_read_task_cell(task_id, reader, cell, options, turbo_tasks)
}
......
......@@ -796,6 +796,22 @@ fn restored_from_disk(result: &Option<Result<Option<TaskStorage>>>) -> bool {
matches!(result, Some(Ok(Some(_))))
}
/// The priority a task is scheduled with: an already computed task is a re-computation of a
/// (possibly deep) dependency, everything else starts at the initial priority.
fn schedule_priority(task: &impl TaskGuard, parent_priority: TaskPriority) -> TaskPriority {
let priority = if task.has_output() {
TaskPriority::invalidation(
task.get_leaf_distance()
.copied()
.unwrap_or_default()
.distance,
)
} else {
TaskPriority::initial()
};
priority.in_parent(parent_priority)
}
/// Combines per-category booleans into a single `TaskDataCategory` for waiting.
fn wait_category(wait_data: bool, wait_meta: bool) -> Option<TaskDataCategory> {
match (wait_data, wait_meta) {
......@@ -1095,18 +1111,8 @@ impl<'e> ExecuteContext<'e> for ExecuteContextImpl<'e> {
}
fn schedule_task(&self, task: Self::TaskGuardImpl, parent_priority: TaskPriority) {
let priority = if task.has_output() {
TaskPriority::invalidation(
task.get_leaf_distance()
.copied()
.unwrap_or_default()
.distance,
)
} else {
TaskPriority::initial()
};
self.turbo_tasks
.schedule(task.id(), priority.in_parent(parent_priority));
let priority = schedule_priority(&task, parent_priority);
self.turbo_tasks.schedule(task.id(), priority);
}
fn get_current_task_priority(&self) -> TaskPriority {
......
......@@ -15,6 +15,14 @@ hanging_detection = []
task_id_details = []
task_dirty_cause = []
verify_determinism = []
# Counts what reads do when they can't get a value straight away: queue pushes, claim attempts and
# their outcomes, and waits for a task a worker is already running. Reported by
# `TurboTasks::inline_execution_stats()`, and printed on shutdown with
# `TURBO_ENGINE_INLINE_STATS=1`.
#
# Off by default: the counters sit on the read-miss path, so a build that doesn't want the numbers
# shouldn't pay for them. With the feature off there are no counters and no atomics at all.
inline_execution_stats = []
# TODO(bgw): This feature is here to unblock turning on local tasks by default. It's only turned on
# in unit tests. This will be removed very soon.
......
......@@ -29,11 +29,10 @@ use turbo_rcstr::RcStr;
use turbo_tasks_hash::DeterministicHasher;
use crate::{
CellId, RawVc, ReadCellOptions, ReadOutputOptions, ReadRef, SharedReference, TaskId, TaskIdSet,
TaskPriority, TraitRef, TraitTypeId, TurboTasksCallApi, TurboTasksPanic, ValueTypeId,
ValueTypePersistence, VcValueTrait, VcValueType,
CellId, RawVc, ReadCellOptions, ReadOutcome, ReadOutputOptions, ReadRef, SharedReference,
TaskId, TaskIdSet, TaskPriority, TraitRef, TraitTypeId, TurboTasksCallApi, TurboTasksPanic,
ValueTypeId, ValueTypePersistence, VcValueTrait, VcValueType,
dyn_task_inputs::{DynTaskInputs, DynTaskInputsStorage},
event::EventListener,
macro_helpers::NativeFunction,
manager::{TaskPersistence, TurboTasks},
registry,
......@@ -654,7 +653,7 @@ pub trait Backend: Sized + Sync + Send {
reader: Option<TaskId>,
options: ReadOutputOptions,
turbo_tasks: &TurboTasks<Self>,
) -> Result<Result<RawVc, EventListener>>;
) -> Result<ReadOutcome<RawVc>>;
/// INVALIDATION: Be careful with this, when reader is None, it will not track dependencies, so
/// using it could break cache invalidation.
......@@ -665,7 +664,7 @@ pub trait Backend: Sized + Sync + Send {
reader: Option<TaskId>,
options: ReadCellOptions,
turbo_tasks: &TurboTasks<Self>,
) -> Result<Result<TypedCellContent, EventListener>>;
) -> Result<ReadOutcome<TypedCellContent>>;
/// INVALIDATION: Be careful with this, it will not track dependencies, so
/// using it could break cache invalidation.
......
......@@ -75,6 +75,8 @@ use rustc_hash::FxHasher;
pub use shrink_to_fit::ShrinkToFit;
pub use turbo_tasks_macros::{DeterministicHash, turbobail, turbofmt};
#[cfg(feature = "inline_execution_stats")]
pub use crate::manager::InlineExecutionStats;
#[cfg(feature = "task_dirty_cause")]
pub use crate::task_dirty_cause::TaskDirtyCause;
pub use crate::{
......@@ -101,15 +103,15 @@ pub use crate::{
join_iter_ext::{JoinIterExt, TryFlatJoinIterExt, TryJoinIterExt},
manager::{
CurrentCellRef, InputResolution, ReadCellTracking, ReadConsistency, ReadTracking,
TaskPersistence, TaskPriority, TurboTasks, TurboTasksApi, TurboTasksCallApi, Unused,
UpdateInfo, dynamic_call, emit, get_serialization_invalidator, mark_finished,
ScheduleKey, TaskPersistence, TaskPriority, TurboTasks, TurboTasksApi, TurboTasksCallApi,
Unused, UpdateInfo, dynamic_call, emit, get_serialization_invalidator, mark_finished,
mark_stateful, mark_top_level_task, prevent_gc, run, run_once, run_once_with_reason,
trait_call, turbo_tasks, turbo_tasks_scope, turbo_tasks_weak,
unmark_top_level_task_may_leak_eventually_consistent_state, with_turbo_tasks,
},
mapped_read_ref::MappedReadRef,
output::OutputContent,
read_options::{ReadCellOptions, ReadOutputOptions},
read_options::{ReadCellOptions, ReadOutcome, ReadOutputOptions},
read_ref::ReadRef,
serialization_invalidation::SerializationInvalidator,
spawn::{JoinHandle, block_for_future, block_in_place, spawn, spawn_blocking, spawn_thread},
......
This diff is collapsed.
......@@ -289,6 +289,9 @@ impl NativeFunction {
TaskPersistence::Persistent => "",
TaskPersistence::Transient => "transient",
};
// `inline_execution` is recorded when a read executed this task on its own thread instead
// of waiting for a worker: "complete" if it finished there, "partial" if it yielded
// and was handed to the runtime. It stays unset for a task a worker executed.
#[cfg(feature = "task_dirty_cause")]
{
tracing::trace_span!(
......@@ -298,6 +301,7 @@ impl NativeFunction {
flags = flags,
reason = reason.as_str(),
cause = cause.map(tracing::field::display),
inline_execution = tracing::field::Empty,
)
}
#[cfg(not(feature = "task_dirty_cause"))]
......@@ -308,11 +312,17 @@ impl NativeFunction {
priority = %priority,
flags = flags,
reason = reason.as_str(),
inline_execution = tracing::field::Empty,
)
}
}
pub fn resolve_span(&'static self, priority: TaskPriority) -> Span {
tracing::trace_span!("turbo_tasks::resolve_call", name = self.ty.name, priority = %priority)
tracing::trace_span!(
"turbo_tasks::resolve_call",
name = self.ty.name,
priority = %priority,
inline_execution = tracing::field::Empty,
)
}
}
use crate::{ReadConsistency, ReadTracking, manager::ReadCellTracking};
use crate::{ReadConsistency, ReadTracking, event::EventListener, manager::ReadCellTracking};
/// What a read of a task's output or cell found.
///
/// The two "not available yet" cases are distinguished because the reader can act on them
/// differently: a task that is only *scheduled* can be taken over and executed by the reader, while
/// one that some worker is already executing can only be waited for — and finding that out without
/// taking the scheduler's queue lock is the point of telling them apart.
///
/// The state is a **hint**: it is a snapshot, and a task can be picked up by a worker right after
/// the read looked (a worker even pops a task off the queue before it marks it as started). Acting
/// on a stale hint costs at most a failed claim, never correctness.
pub enum ReadOutcome<T> {
/// The value is available.
Value(T),
/// The task is queued but no worker has started it, so the reader may take it out of the queue
/// and execute it itself. The listener fires when the task is done.
Scheduled(EventListener),
/// A worker is already executing the task; there is nothing to take over. The listener fires
/// when the task is done.
InProgress(EventListener),
}
#[derive(Clone, Copy, Debug, Default)]
pub struct ReadCellOptions {
......
......@@ -19,8 +19,10 @@ use crate::{
id::{ExecutionId, LocalTaskId, TASK_ID_MAX},
manager::{
ReadCellTracking, ReadTracking, SUPPRESS_EVENTUAL_CONSISTENCY_TOP_LEVEL_TASK_CHECK,
TurboTasksApi, read_local_output, with_turbo_tasks,
ScheduleKey, TurboTasksApi, execute_read_target_inline, read_local_output,
with_turbo_tasks,
},
read_options::ReadOutcome,
registry::get_value_type,
turbo_tasks,
};
......@@ -502,6 +504,15 @@ fn suppress_top_level_task_check<R>(strongly_consistent: bool, f: impl FnOnce()
}
}
/// Executes the task a read is waiting for when it is only scheduled, so the read can continue
/// without waiting for a worker.
///
/// Must be called *outside* [`with_turbo_tasks`]: executing a task enters a task-local scope of its
/// own, which panics while the read borrows `TURBO_TASKS`.
fn execute_inline(key: ScheduleKey) {
execute_read_target_inline(&*turbo_tasks(), key);
}
#[must_use]
pub struct ResolveRawVcFuture {
current: RawVc,
......@@ -551,14 +562,20 @@ impl Future for ResolveRawVcFuture {
// SAFETY: we are not moving self
let this = unsafe { self.get_unchecked_mut() };
let poll_fn = |tt: &Arc<dyn TurboTasksApi>| -> Poll<Self::Output> {
let strongly_consistent = this.strongly_consistent;
// `execute_inline` is the task to execute inline, reported out-of-band so that the value
// path stays exactly as cheap as it was. It is only set for tasks that are merely
// scheduled; one that a worker is already executing cannot be taken over.
let mut poll_fn = |tt: &Arc<dyn TurboTasksApi>,
execute_inline: &mut Option<ScheduleKey>|
-> Poll<Self::Output> {
'outer: loop {
ready!(poll_listener(&mut this.listener, cx));
let listener = match this.current.unpack() {
let (listener, key) = match this.current.unpack() {
RawVcUnpacked::TaskOutput(task) => {
let read_result = tt.try_read_task_output(task, this.read_output_options);
match read_result {
Ok(Ok(vc)) => {
Ok(ReadOutcome::Value(vc)) => {
// turbo-tasks-backend doesn't currently have any sort of
// "transaction" or global lock mechanism to group together chains
// of `TaskOutput`/`TaskCell` reads.
......@@ -571,7 +588,19 @@ impl Future for ResolveRawVcFuture {
this.current = vc;
continue 'outer;
}
Ok(Err(listener)) => listener,
// Nobody has started the task yet, so the caller may take it over.
Ok(ReadOutcome::Scheduled(listener)) => {
(listener, Some(ScheduleKey::Task(task)))
}
Ok(ReadOutcome::InProgress(listener)) => {
// A worker is on it; nothing to take over. Loop back so
// `poll_listener` registers the waker on this listener — returning
// `Pending` here would sleep through the event.
#[cfg(feature = "inline_execution_stats")]
tt.note_waited_for_in_progress_task();
this.listener = Some(listener);
continue 'outer;
}
Err(err) => return Poll::Ready(Err(err)),
}
}
......@@ -587,22 +616,40 @@ impl Future for ResolveRawVcFuture {
this.current = vc;
continue 'outer;
}
Ok(Err(listener)) => listener,
Ok(Err(listener)) => (
listener,
Some(ScheduleKey::LocalTask(execution_id, local_task_id)),
),
Err(err) => return Poll::Ready(Err(err)),
}
}
};
// The task is not done yet, so we have to wait for it — unless it is merely
// scheduled, in which case our caller executes it and we read again.
this.listener = Some(listener);
*execute_inline = key;
return Poll::Pending;
}
};
// HACK: Temporarily suppress top-level task check if doing strongly consistent read.
//
// This masks a bug: There's an unlikely TOCTOU race condition in `poll_fn`. Because the
// strongly consistent read isn't a single atomic operation, any inner `TaskOutput` or
// `TaskCell` could get mutated after the strongly consistent read of the outer
// `TaskOutput`.
suppress_top_level_task_check(this.strongly_consistent, || with_turbo_tasks(poll_fn))
loop {
let mut execute_inline_key = None;
// HACK: Temporarily suppress top-level task check if doing strongly consistent read.
//
// This masks a bug: There's an unlikely TOCTOU race condition in `poll_fn`. Because the
// strongly consistent read isn't a single atomic operation, any inner `TaskOutput` or
// `TaskCell` could get mutated after the strongly consistent read of the outer
// `TaskOutput`.
let result = suppress_top_level_task_check(strongly_consistent, || {
with_turbo_tasks(|tt| poll_fn(tt, &mut execute_inline_key))
});
if let Some(key) = execute_inline_key {
// Not inside `with_turbo_tasks`, see `execute_inline`.
execute_inline(key);
continue;
}
return result;
}
}
}
......@@ -724,23 +771,53 @@ impl Future for ReadRawVcFuture {
let index = *index;
let read_cell_options = this.read_cell_options;
let poll_fn = |tt: &Arc<dyn TurboTasksApi>| -> Poll<Self::Output> {
let strongly_consistent = *strongly_consistent;
let mut poll_fn = |tt: &Arc<dyn TurboTasksApi>,
execute_inline: &mut Option<ScheduleKey>|
-> Poll<Self::Output> {
loop {
ready!(poll_listener(listener, cx));
let new_listener = match tt.try_read_task_cell(task, index, read_cell_options) {
Ok(Ok(content)) => return Poll::Ready(Ok(content)),
Ok(Err(l)) => l,
Err(err) => return Poll::Ready(Err(err)),
};
let (new_listener, key) =
match tt.try_read_task_cell(task, index, read_cell_options) {
Ok(ReadOutcome::Value(content)) => return Poll::Ready(Ok(content)),
Ok(ReadOutcome::Scheduled(l)) => (l, Some(ScheduleKey::Task(task))),
Ok(ReadOutcome::InProgress(l)) => {
// A worker is already filling the cell; nothing to take over. Loop back
// so `poll_listener` registers the waker on this listener — returning
// `Pending` here would sleep through the event.
#[cfg(feature = "inline_execution_stats")]
tt.note_waited_for_in_progress_task();
*listener = Some(l);
continue;
}
Err(err) => return Poll::Ready(Err(err)),
};
// The cell isn't available yet, so we have to wait for the task that fills it —
// unless that task is merely scheduled, in which case our caller executes it and we
// read again.
*listener = Some(new_listener);
*execute_inline = key;
return Poll::Pending;
}
};
// Phase 2 must also suppress the top-level task check when phase 1 was
// strongly-consistent. The suppression from `ResolveRawVcFuture::poll` only lasts for
// the duration of that individual `poll` call and does not carry over to subsequent calls
// or to this phase.
suppress_top_level_task_check(*strongly_consistent, || with_turbo_tasks(poll_fn))
loop {
let mut execute_inline_key = None;
// Phase 2 must also suppress the top-level task check when phase 1 was
// strongly-consistent. The suppression from `ResolveRawVcFuture::poll` only lasts for
// the duration of that individual `poll` call and does not carry over to subsequent
// calls or to this phase.
let result = suppress_top_level_task_check(strongly_consistent, || {
with_turbo_tasks(|tt| poll_fn(tt, &mut execute_inline_key))
});
if let Some(key) = execute_inline_key {
// Not inside `with_turbo_tasks`, see `execute_inline`.
execute_inline(key);
continue;
}
return result;
}
}
}
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment