Unverified Commit 4bcd07f9 authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

turbo-tasks: batch schedule invalidated tasks and cell dependents (#100446)

### What?

Tasks that become dirty together are handed to the turbo-tasks scheduler
(`PriorityRunner`) as one batch instead of one by one, at two sites:

- `invalidate()` (explicit invalidation of a set of tasks)
- `invalidate_cell_dependents()` in `update_cell` (all dependents of a
changed cell — the main fan-out path)

Also adds a `cell_fan_out` criterion benchmark to `turbo-tasks-backend`.

### Why?

`PriorityRunner` keeps all non-lowest-priority work in a
`Mutex<BinaryHeap>`. Scheduling N tasks one by one locks that heap up to
N times and grows it incrementally, contending with the workers that pop
from it. This is a first step towards reducing that contention.

Note: in-tree, `invalidate()` is currently only reached with a single
task (`TurboTasks::invalidate` → `Backend::invalidate_task`);
`invalidate_tasks` / `invalidate_tasks_set` have no in-tree callers. The
real fan-out is a cell change invalidating its dependents, which is why
`update_cell` is batched too.

### How?

- **`PriorityRunner::schedule_many`**: behaves like calling `schedule`
for each item. While there is free worker capacity, items are handed
directly to new workers using the same slot reservation as `schedule`.
The remaining items are queued in bulk: each item is still inserted into
the claimable map before its key is published, lowest-priority keys
still go to the lock-free queue, and all other keys are staged and then
pushed into the heap under a single lock acquisition after one
`reserve`. The staging buffer is only allocated when a batch actually
contains heap items. Each claimable insert finishes before the heap is
locked, preserving the existing heap → claimable lock order.
- The liveness protocol is unchanged: after the bulk push there is one
`SeqCst` fence followed by an `active_workers` re-check. If capacity
opened up concurrently, workers are spawned for queued work as long as
capacity remains. Items replaced by a newer item with the same claim key
are still passed to `Executor::discard`.
- **`TurboTasks::schedule_many`**: batch counterpart of
`TurboTasks::schedule`; begins all foreground jobs and updates the
scheduled-task counter up front; no-op for an empty batch.
- **Backend**: the dirty-marking logic was split into
`make_task_dirty_without_scheduling`, which returns the parent priority
instead of scheduling. `invalidate()` and `invalidate_cell_dependents()`
compute each task's scheduling priority while holding its guard, collect
into a pre-reserved buffer, and schedule once after all tasks are marked
dirty. In `update_cell` the existing order (remove old content → dirty
dependents → set new content → notify waiters → run aggregation updates)
is preserved; only the scheduling is deferred to the end of the dirtying
loop. `make_task_dirty_internal` keeps scheduling immediately for its
other callers (aggregation update, connect child(ren)).

New `priority_runner` unit tests cover mixed priority bands,
interleaving with `schedule`, empty batches, claiming batched items,
duplicate claim keys, a multi-worker run where every item executes
exactly once, and diagnostics counting.

### Benchmark

`cargo bench -p turbo-tasks-backend --bench mod -- cell_fan_out`

Graph: a `State` → one `source` task producing a cell → N `dependent`
tasks reading that cell → a root summing them. The graph is built once
(untimed); each timed iteration changes the state and does a strongly
consistent read of the root, i.e. invalidation fan-out + scheduling +
re-execution of all N dependents. The result is asserted every
iteration.

A/B on one 8-vCPU x86_64 Linux VM (Intel Xeon @ 2.50 GHz), release
build, in-memory backend. Base is the parent commit of this PR with the
same benchmark file added; six alternating base/branch process runs,
values are criterion's central time estimate per run (ms):

| N | base (6 runs) | branch (6 runs) | median base → branch | paired
change (approx. 95% CI) |
|---|---|---|---|---|
| 1,000 | 6.45, 6.14, 5.96, 6.54, 5.69, 6.57 | 7.12, 7.15, 5.53, 5.77,
6.06, 5.29 | 6.29 → 5.91 | −1.7% [−15.5%, +14.4%] |
| 10,000 | 73.49, 75.38, 89.11, 72.21, 69.77, 65.11 | 68.38, 76.65,
64.04, 75.06, 66.85, 68.76 | 72.85 → 68.57 | −5.4% [−18.6%, +9.9%] |
| 100,000 | 864.7, 1011.5, 953.4, 883.8, 856.5, 814.3 | 848.5, 915.0,
791.9, 876.6, 928.1, 848.1 | 874.3 → 862.6 | −3.2% [−12.5%, +7.2%] |

**No detected end-to-end difference**: the branch tends to be slightly
faster, but every interval overlaps zero and run-to-run noise (~5–10%)
is larger than the effect. Re-executing the N dependents (~8–9 µs per
task here) dominates each iteration, so the scheduler's heap lock is a
small share of the measured time. The intervals only reflect repeated
processes on one VM (not independent machines), and runs were always
base-first. The reduced number of heap lock acquisitions follows from
the code structure; it is not separately instrumented.

The benchmark is kept so later steps (batching more scheduling sites,
reducing heap contention further) can be measured against it.

### Verification

- `cargo test -p turbo-tasks --lib priority_runner` (also with
`--features inline_execution_stats`)
- `cargo test -p turbo-tasks-backend`
- `cargo clippy -p turbo-tasks -p turbo-tasks-backend --all-targets --
-D warnings`

<!-- NEXT_JS_LLM -->


<!-- fleet d21ddb86-de88-41df-82c7-aa6a29a6165e -->

---------
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent 2ba33b72
//! Benchmarks invalidation fan-out: one cell that is read by many dependent tasks changes, so all
//! of them are made dirty and scheduled at once (`update_cell` → `invalidate_cell_dependents`).
//!
//! The graph is built once per size. Each iteration changes a `State`, which recomputes the
//! `source` task and updates its cell, which invalidates all `width` `dependent` tasks. The timing
//! covers invalidation, scheduling and re-execution of all dependents until a strongly consistent
//! read of the root returns.
//!
//! ```bash
//! cargo bench -p turbo-tasks-backend --bench mod -- cell_fan_out
//! ```
use std::{
sync::{
Arc,
atomic::{AtomicU32, Ordering},
},
time::{Duration, Instant},
};
use anyhow::Result;
use criterion::{BenchmarkId, Criterion, Throughput};
use turbo_tasks::{OperationVc, ResolvedVc, State, TryJoinIterExt, TurboTasks, Vc};
use turbo_tasks_backend::{BackendOptions, TurboTasksBackend, noop_backing_storage};
#[turbo_tasks::value(transparent)]
struct Input(State<u32>);
#[turbo_tasks::function(operation, root)]
fn create_input() -> Vc<Input> {
Input(State::new(0)).cell()
}
/// The task whose cell fans out to all dependents.
#[turbo_tasks::function]
async fn source(input: ResolvedVc<Input>) -> Result<Vc<u32>> {
Ok(Vc::cell(*input.await?.get()))
}
#[turbo_tasks::function]
async fn dependent(input: ResolvedVc<Input>, index: u32) -> Result<Vc<u32>> {
Ok(Vc::cell(source(*input).await?.wrapping_add(index)))
}
#[turbo_tasks::function(operation, root)]
async fn fan_out_root(input: ResolvedVc<Input>, width: u32) -> Result<Vc<u32>> {
let values = (0..width)
.map(|index| dependent(*input, index))
.try_join()
.await?;
Ok(Vc::cell(
values
.iter()
.fold(0u32, |sum, value| sum.wrapping_add(**value)),
))
}
/// The sum `fan_out_root` computes when the state is `value`.
fn expected_sum(value: u32, width: u32) -> u32 {
(0..width).fold(0u32, |sum, index| {
sum.wrapping_add(value.wrapping_add(index))
})
}
async fn change_and_read(input_op: OperationVc<Input>, width: u32, value: u32) -> Result<Duration> {
let input = input_op.resolve().strongly_consistent().await?;
let state = input_op.read_strongly_consistent().await?;
let start = Instant::now();
state.set(value);
let sum = *fan_out_root(input, width)
.read_strongly_consistent()
.await?;
let elapsed = start.elapsed();
assert_eq!(sum, expected_sum(value, width));
Ok(elapsed)
}
pub fn cell_fan_out(c: &mut Criterion) {
let mut group = c.benchmark_group("turbo_tasks_backend_cell_fan_out");
group.sample_size(20);
group.measurement_time(Duration::from_secs(10));
for width in [1_000u32, 10_000, 100_000] {
group.throughput(Throughput::Elements(width as u64));
group.bench_with_input(BenchmarkId::new("width", width), &width, |b, &width| {
let rt = tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.unwrap();
// `TurboTasks::new` builds a `PriorityRunner` that calls `Handle::current()`, so the
// backend must be constructed inside the runtime context.
let tt = rt.block_on(async {
TurboTasks::new(TurboTasksBackend::new(
BackendOptions {
storage_mode: None,
..Default::default()
},
noop_backing_storage(),
))
});
// Build the graph once (not timed).
let input_op = rt
.block_on(tt.run_once(async move {
let input_op = create_input();
let input = input_op.resolve().strongly_consistent().await?;
let sum = *fan_out_root(input, width)
.read_strongly_consistent()
.await?;
assert_eq!(sum, expected_sum(0, width));
Ok(input_op)
}))
.unwrap();
let next_value = Arc::new(AtomicU32::new(1));
b.to_async(&rt).iter_custom(|iters| {
let tt = tt.clone();
let next_value = next_value.clone();
async move {
tt.run_once(async move {
let mut total = Duration::ZERO;
for _ in 0..iters {
let value = next_value.fetch_add(1, Ordering::Relaxed);
total += change_and_read(input_op, width, value).await?;
}
Ok(total)
})
.await
.unwrap()
}
});
rt.block_on(async { tt.stop_and_wait().await });
});
}
group.finish();
}
......@@ -3,6 +3,7 @@
use criterion::{Criterion, criterion_group, criterion_main};
pub(crate) mod cell_fan_out;
pub(crate) mod gc;
pub(crate) mod overhead;
pub(crate) mod scope_stress;
......@@ -11,6 +12,7 @@ pub(crate) mod stress;
criterion_group!(
name = turbo_tasks_backend_stress;
config = Criterion::default();
targets = stress::fibonacci, scope_stress::scope_stress, overhead::overhead, gc::gc
targets = stress::fibonacci, scope_stress::scope_stress, overhead::overhead, gc::gc,
cell_fan_out::cell_fan_out
);
criterion_main!(turbo_tasks_backend_stress);
......@@ -23,48 +23,62 @@ pub fn invalidate(
mut ctx: ExecuteContext<'_>,
) {
let mut queue = AggregationUpdateQueue::new();
// The tasks to schedule are collected and handed to the scheduler as one batch, which takes
// the scheduler's queue lock once instead of once per task.
let mut to_schedule: SmallVec<[_; 4]> = SmallVec::with_capacity(task_ids.len());
for task_id in task_ids {
try_make_task_dirty(
task_id,
let Some(mut task) = ctx.try_task(task_id, TaskDataCategory::All) else {
// The task no longer exists, nothing to invalidate.
continue;
};
if let Some(parent_priority) = make_task_dirty_without_scheduling(
&mut task,
true,
#[cfg(feature = "task_dirty_cause")]
cause.clone(),
&mut queue,
&mut ctx,
);
) {
to_schedule.push((task_id, ctx.schedule_task_priority(&task, parent_priority)));
}
}
ctx.schedule_tasks(&to_schedule);
queue.execute(&mut ctx);
}
/// Marks a task dirty, doing nothing if it no longer exists.
///
/// Intended for invalidation usecases.
pub fn try_make_task_dirty(
task_id: TaskId,
/// Requires the guard to be allocated with [TaskDataCategory::All]
pub fn make_task_dirty_internal(
task: &mut TaskGuard<'_>,
make_stale: bool,
#[cfg(feature = "task_dirty_cause")] cause: TaskDirtyCause,
queue: &mut AggregationUpdateQueue,
ctx: &mut ExecuteContext<'_>,
) {
let Some(mut task) = ctx.try_task(task_id, TaskDataCategory::All) else {
return;
};
make_task_dirty_internal(
&mut task,
true,
if let Some(parent_priority) = make_task_dirty_without_scheduling(
task,
make_stale,
#[cfg(feature = "task_dirty_cause")]
cause,
queue,
ctx,
);
) {
ctx.schedule_task(task, parent_priority);
}
}
/// Like [`make_task_dirty_internal`], but instead of scheduling the task it returns the parent
/// priority to schedule it with (see [`ExecuteContext::schedule_task`]). The task is already marked
/// as scheduled when this returns `Some`, so the caller must schedule it.
///
/// Requires the guard to be allocated with [TaskDataCategory::All]
pub fn make_task_dirty_internal(
#[must_use]
pub(super) fn make_task_dirty_without_scheduling(
task: &mut TaskGuard<'_>,
make_stale: bool,
#[cfg(feature = "task_dirty_cause")] cause: TaskDirtyCause,
queue: &mut AggregationUpdateQueue,
ctx: &mut ExecuteContext<'_>,
) {
) -> Option<TaskPriority> {
// There must be no way to invalidate immutable tasks. If there would be a way the task is not
// immutable.
#[cfg(any(debug_assertions, feature = "verify_immutable"))]
......@@ -133,7 +147,7 @@ pub fn make_task_dirty_internal(
cause,
});
}
return;
return None;
}
Some(Dirtyness::SessionDependent) => {
task.set_dirty(Dirtyness::Dirty {
......@@ -157,7 +171,7 @@ pub fn make_task_dirty_internal(
)
.entered();
// already dirty
return;
return None;
}
}
None => {
......@@ -213,7 +227,8 @@ pub fn make_task_dirty_internal(
if should_schedule {
let description = EventDescription::new(|| task.get_task_desc_fn());
if task.add_scheduled(TaskExecutionReason::Invalidated, description) {
ctx.schedule_task(&*task, parent_priority);
return Some(parent_priority);
}
}
None
}
......@@ -1049,6 +1049,22 @@ impl<'e> ExecuteContext<'e> {
self.turbo_tasks.schedule(task.id(), priority);
}
/// Returns the priority [`ExecuteContext::schedule_task`] would schedule `task` with, so the
/// scheduling can be deferred and batched with [`ExecuteContext::schedule_tasks`].
pub fn schedule_task_priority(
&self,
task: &TaskGuard<'e>,
parent_priority: TaskPriority,
) -> TaskPriority {
schedule_priority(task, parent_priority)
}
/// Schedules all `tasks` (with priorities from [`ExecuteContext::schedule_task_priority`]) as
/// one batch, which is cheaper for the scheduler than scheduling them one by one.
pub fn schedule_tasks(&self, tasks: &[(TaskId, TaskPriority)]) {
self.turbo_tasks.schedule_many(tasks);
}
pub fn get_current_task_priority(&self) -> TaskPriority {
self.turbo_tasks.get_current_task_priority()
}
......
......@@ -14,7 +14,8 @@ use crate::{
backend::{
TaskDataCategory,
operation::{
AggregationUpdateQueue, ExecuteContext, TaskGuard, invalidate::make_task_dirty_internal,
AggregationUpdateQueue, ExecuteContext, TaskGuard,
invalidate::make_task_dirty_without_scheduling,
},
storage_schema::TaskStorageAccessors,
},
......@@ -238,6 +239,9 @@ fn invalidate_cell_dependents(
ctx: &mut ExecuteContext<'_>,
) {
let mut queue = AggregationUpdateQueue::new();
// The dependents to schedule are collected and handed to the scheduler as one batch, which
// takes the scheduler's queue lock once instead of once per dependent.
let mut to_schedule = Vec::with_capacity(dependent_tasks.len());
while let Some((dependent_task_id, keys)) = dependent_tasks.pop() {
let mut make_stale = false;
let mut dependent = ctx.task(dependent_task_id, TaskDataCategory::All);
......@@ -269,7 +273,7 @@ fn invalidate_cell_dependents(
make_stale = true;
}
}
make_task_dirty_internal(
if let Some(parent_priority) = make_task_dirty_without_scheduling(
&mut dependent,
make_stale,
#[cfg(feature = "task_dirty_cause")]
......@@ -283,8 +287,14 @@ fn invalidate_cell_dependents(
},
&mut queue,
ctx,
);
) {
to_schedule.push((
dependent_task_id,
ctx.schedule_task_priority(&dependent, parent_priority),
));
}
}
ctx.schedule_tasks(&to_schedule);
let mut task = ctx.task(cell_ref.task, TaskDataCategory::Data);
if let Some(content) = content {
......
......@@ -1181,6 +1181,32 @@ impl<B: Backend + 'static> TurboTasks<B> {
self.priority_runner.schedule(&self.pin(), task, priority);
}
/// Schedules all `tasks`, like calling [`TurboTasks::schedule`] for each of them, but hands
/// them to the scheduler as one batch, so its queue is locked once instead of once per task.
#[track_caller]
pub fn schedule_many(&self, tasks: &[(TaskId, TaskPriority)]) {
if tasks.is_empty() {
return;
}
self.begin_foreground_jobs(tasks.len());
self.scheduled_tasks
.fetch_add(tasks.len(), Ordering::AcqRel);
let span = Span::current();
self.priority_runner.schedule_many(
&self.pin(),
tasks.iter().map(|&(task_id, priority)| {
(
ScheduledTask::Task {
task_id,
span: span.clone(),
},
priority,
)
}),
);
}
fn schedule_local_task(
&self,
ty: LocalTaskSpec,
......@@ -1238,9 +1264,15 @@ impl<B: Backend + 'static> TurboTasks<B> {
}
fn begin_foreground_job(&self) {
self.begin_foreground_jobs(1);
}
/// Begins `count` foreground jobs at once. `count` must not be zero.
fn begin_foreground_jobs(&self, count: usize) {
debug_assert!(count > 0);
if self
.currently_scheduled_foreground_jobs
.fetch_add(1, Ordering::AcqRel)
.fetch_add(count, Ordering::AcqRel)
== 0
{
*self.start.lock().unwrap() = Some(Instant::now());
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment