Unverified Commit 44525146 authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

turbo-tasks: lock-free priority queue for lowest-priority tasks, skip heap lock when empty (#99773)

### What?

Removes the global mutex from the turbo-tasks `PriorityRunner` ready
queue for the hot paths, and makes `Queue::pop()` skip the remaining
heap mutex entirely when the heap is empty.

Supersedes #94841: this is that PR rebased onto canary (keeping the
keyed claim / inline execution from #96808), plus the empty-heap fast
path, plus two simplification passes and review fixes.

### Why?

In a from-scratch build, nearly every scheduled task has the lowest
priority (`TaskPriority::Initial`). Before this change, every `schedule`
and every worker `pop` went through one `Mutex<Queue>`, which becomes a
contention point at high worker counts.

### How?

**Queue split by band**
- `Executor<C, T, P>` declares `const LOWEST_PRIORITY: P`
(`TaskPriority::Initial` for turbo-tasks). Items with that priority go
to a lock-free MPMC `ConcurrentQueue` (FIFO; all items share a
priority).
- All other priorities go to a `Mutex<BinaryHeap>` ordered by `P: Ord`.
`pop` drains the heap first, then the lowest band.
- The heap starts with capacity 128 and only shrinks back to 128 when it
becomes empty.
- Lock ordering: `pop` may take a `claimable` shard lock while holding
the heap mutex; nothing takes the heap mutex while holding a shard lock.

**Scheduler / worker handshake**
- `schedule` takes no global lock. Liveness (no task stranded when the
last worker retires) is a store-buffering pattern: the scheduler pushes
then reads `active_workers`; a retiring worker decrements
`active_workers` then re-checks the queue. Both sides put a
`fence(SeqCst)` between their write and their read, so one of them
always observes the other.
- The saturated hot path is push → `fence(SeqCst)` → `load`, so
schedulers no longer write the shared `active_workers` cache line. Every
worker that gives up its count goes through `decrease_active_workers`
(`fetch_sub(AcqRel)` → `fence(SeqCst)` → re-check).

**One queued task per key**
- `Claimable::claim_key()` returns `Self::Key` (no `Option`): every task
has a key.
- Queued tasks are stored by value in `claimable: FxDashMap<T::Key, (P,
T)>`. The heap and the `ConcurrentQueue` only store `T::Key`.
- Pushing a key that is still queued **replaces** the queued task. The
replaced task goes to the new `Executor::discard` hook (default: no-op).
`TurboTasksExecutor::discard` balances the foreground job that
`TurboTasks::schedule` began for it.
- This only applies to queued tasks: a task that `schedule` spawns
directly never enters the queue.
- Claim = `claimable.remove(key)`. Pop = `claimable.remove_if(key,
priority matches the band entry)`. Stale band entries are tombstones
that a pop skips; a stale entry never takes a task queued again with a
different priority. With the *same* priority it may take the newer task
early, which only reorders equal-priority tasks.
- No per-task allocation or per-task mutex, no new `unsafe`. Keyed
pushes/pops take a brief DashMap shard lock.

**Unique local-task keys**
- `ExecutionId` is a wrapping `u16`, so
`ScheduleKey::LocalTask(ExecutionId, LocalTaskId)` was not unique across
concurrent executions; with replacement that hung the build.
- The key is now `ScheduleKey::LocalTask(Option<TaskId>, ExecutionId,
LocalTaskId)`. The remaining exception is two concurrent top-level
executions (no parent task) sharing a wrapped `ExecutionId`.
- Local tasks stay in the runner (not plain tokio tasks) so readers can
keep claiming them for inline execution.

**Empty-heap fast path**
- `heap_non_empty: AtomicBool` is written only under the heap lock.
`pop()` checks it and goes straight to the lowest band when the heap is
empty, so the common cold-build pop never touches the heap mutex.
- A stale `true` costs only an unnecessary lock. A stale `false` can't
hide a push that must be observed, by the same fence handshake.

### Tests

- `priority_runner` tests use plain `u32` priorities. New:
`heap_only_shrinks_when_empty`. The saturated-path regression test is
now `stress_saturated_schedule_vs_retiring_worker`.
- Replacement within and across bands, claims in both bands, stale
entries, concurrent same-key producers/poppers/claimer, claim-vs-pop
exactly-once, band ordering, heap-flag drain/refill, pop not waiting on
the empty heap's lock, three liveness stress tests.
- Stress tests use a timer-only tokio runtime: with `enable_all()` on
`wasm32-wasip1-threads`, newly spawned tasks reproducibly stopped being
polled.
- On canary `b2490553`: `turbo-tasks` 134 passed (two full runs),
`turbo-tasks-backend` 282, wasi lib 110 passed, `priority_runner` 5×
repeated, clippy `-D warnings`, fmt. Cold 32-core `next build` of
`bench/nested-deps-app-router-many-pages` completes (one run, 34.3 s
compiled / 59 s wall; not a benchmark).

### Benchmarks

`bench/nested-deps-app-router-many-pages`, cold `next build`
(Turbopack), `taskset` CPUs = `TURBO_TASKS_AVAILABLE_PARALLELISM`
workers.

Current head (`4d89cde5`) vs its base canary (`b2490553`), both fresh
release builds of the native bindings with the same JS. Per CPU count: 1
warm-up per arm, then 4 interleaved untraced rounds per arm (medians):

| CPUs = workers | canary wall | this PR wall | Δ wall | canary
"Compiled in" | this PR "Compiled in" | Δ compiled | paired per-round
wall Δ |
|---:|---:|---:|---:|---:|---:|---:|---|
| 8 | 93.5 s | 93.0 s | −0.5% | 52.5 s | 51.5 s | −1.9% | −3.3 … +8.1% |
| 16 | 73.6 s | 72.5 s | −1.4% | 42.5 s | 41.5 s | −2.4% | −5.9 … +4.4%
|
| 32 | 69.5 s | 59.2 s | **−14.8%** | 45.5 s | 34.0 s | **−25.2%** |
−25.4 … −5.3% (all 4 faster) |

`whole app module graph` span, from 2 light-traced runs per arm
(`NEXT_TURBOPACK_TRACING=next_api=info`); busy cores are system-wide
`/proc/stat` over the span, as a share of the pinned CPUs:

| CPUs | canary span | this PR span | Δ | busy cores, canary | busy
cores, this PR |
|---:|---:|---:|---:|---:|---:|
| 8 | 9.56, 9.16 s | 8.91, 8.81 s | −5% | 6.6 of 8 | 6.7 of 8 |
| 16 | 7.67, 6.99 s | 5.76, 5.69 s | −22% | ~11 of 16 | ~12 of 16 |
| 32 | 10.36, 12.35 s | 5.60, 5.94 s | **−49%** | ~12 of 32 | ~23 of 32
|

At 8 and 16 CPUs the wall-time differences are within run-to-run noise.
At 32 CPUs, canary keeps only ~12 cores busy during the module graph;
this PR roughly doubles that and halves the phase.

Earlier passes, for history: the first version of this PR measured 77.5
s → 57.9 s wall at 32 CPUs (older base). The first simplification pass
showed no measurable difference to the version before it (8 rounds at 32
CPUs: 63.4 s → 62.2 s median wall, paired per-round median +0.3%).

Review-round microbenchmarks (scratch harnesses, not committed):
- `ConcurrentQueue` vs `crossbeam_queue::SegQueue`: SegQueue was 16–30%
slower at 16/32 threads with split producers/consumers and 3–18% slower
with mixed push/pop at 32 threads (two runs), so `ConcurrentQueue` stays
and no full build benchmark was run.
- `fetch_add(0)` RMW vs fence + `load` on the saturated path: 1.97 s vs
1.94 s median for 4.8M schedules, within noise. No speedup is claimed.
- Layout: `PriorityRunner` is already 128-byte aligned (via
`ConcurrentQueue`'s `CachePadded`), so extra padding isn't needed.

Caveats: single VM, one session, 4 untraced and 2 traced rounds per arm,
no statistical test; the CPU counts ran one after another, so
cross-CPU-count comparisons can include drift. The AtomicBool's own
effect isn't isolated.

<!-- NEXT_JS_LLM -->


<!-- fleet d8fee2f9-18db-4936-b59a-f8417b8f4bbc -->

---------
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarLuke Sandberg <lukesandberg@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: 's avatarVercel Fleet <fleet@vercel.com>
parent d8c1c1b0
...@@ -486,6 +486,9 @@ enum ScheduledTask { ...@@ -486,6 +486,9 @@ enum ScheduledTask {
LocalTask { LocalTask {
ty: LocalTaskSpec, ty: LocalTaskSpec,
persistence: TaskPersistence, persistence: TaskPersistence,
/// The non-local task whose execution created this local task (`None` for a top-level
/// execution), see [`ScheduleKey::LocalTask`].
parent_task: Option<TaskId>,
execution_id: ExecutionId, execution_id: ExecutionId,
local_task_id: LocalTaskId, local_task_id: LocalTaskId,
global_task_state: CurrentTaskStateHandle, global_task_state: CurrentTaskStateHandle,
...@@ -500,22 +503,41 @@ enum ScheduledTask { ...@@ -500,22 +503,41 @@ enum ScheduledTask {
pub enum ScheduleKey { pub enum ScheduleKey {
/// A cached (non-local) task. /// A cached (non-local) task.
Task(TaskId), Task(TaskId),
/// A local task, which is only known within the execution that created it. /// A local task, which is only known within the execution that created it: the non-local task
LocalTask(ExecutionId, LocalTaskId), /// of that execution (`None` for a top-level execution, e.g. [`TurboTasks::run`]), the
/// execution and the local task id within it.
///
/// `ExecutionId` alone isn't unique across concurrent executions (it wraps), but a non-local
/// task only has one execution at a time, and that execution waits for its local tasks. So the
/// key is unique, except for two concurrent top-level executions that share a wrapped
/// `ExecutionId`.
LocalTask(Option<TaskId>, ExecutionId, LocalTaskId),
}
impl ScheduleKey {
/// The key of a local task created by the current execution.
pub(crate) fn current_local_task(
execution_id: ExecutionId,
local_task_id: LocalTaskId,
) -> Self {
let parent_task = CURRENT_TASK_STATE.with(|ts| ts.current_task_id());
Self::LocalTask(parent_task, execution_id, local_task_id)
}
} }
impl Claimable for ScheduledTask { impl Claimable for ScheduledTask {
type Key = ScheduleKey; type Key = ScheduleKey;
fn claim_key(&self) -> Option<ScheduleKey> { fn claim_key(&self) -> ScheduleKey {
Some(match self { match self {
ScheduledTask::Task { task_id, .. } => ScheduleKey::Task(*task_id), ScheduledTask::Task { task_id, .. } => ScheduleKey::Task(*task_id),
ScheduledTask::LocalTask { ScheduledTask::LocalTask {
parent_task,
execution_id, execution_id,
local_task_id, local_task_id,
.. ..
} => ScheduleKey::LocalTask(*execution_id, *local_task_id), } => ScheduleKey::LocalTask(*parent_task, *execution_id, *local_task_id),
}) }
} }
} }
...@@ -1165,12 +1187,13 @@ impl<B: Backend + 'static> TurboTasks<B> { ...@@ -1165,12 +1187,13 @@ impl<B: Backend + 'static> TurboTasks<B> {
persistence: TaskPersistence, persistence: TaskPersistence,
) -> RawVc { ) -> RawVc {
let task_type = ty.task_type; let task_type = ty.task_type;
let (global_task_state, execution_id, priority, local_task_id) = let (global_task_state, parent_task, execution_id, priority, local_task_id) =
CURRENT_TASK_STATE.with(|gts| { CURRENT_TASK_STATE.with(|gts| {
let mut gts_write = gts.write().unwrap(); let mut gts_write = gts.write().unwrap();
let local_task_id = gts_write.local_tasks.create(task_type); let local_task_id = gts_write.local_tasks.create(task_type);
( (
gts.clone(), gts.clone(),
gts_write.task_id,
gts_write.execution_id, gts_write.execution_id,
gts_write.priority, gts_write.priority,
local_task_id, local_task_id,
...@@ -1180,6 +1203,7 @@ impl<B: Backend + 'static> TurboTasks<B> { ...@@ -1180,6 +1203,7 @@ impl<B: Backend + 'static> TurboTasks<B> {
let task = ScheduledTask::LocalTask { let task = ScheduledTask::LocalTask {
ty, ty,
persistence, persistence,
parent_task,
execution_id, execution_id,
local_task_id, local_task_id,
global_task_state, global_task_state,
...@@ -1531,6 +1555,8 @@ async fn abort_on_panic<F: Future>(f: F) -> F::Output { ...@@ -1531,6 +1555,8 @@ async fn abort_on_panic<F: Future>(f: F) -> F::Output {
} }
impl<B: Backend> Executor<TurboTasks<B>, ScheduledTask, TaskPriority> for TurboTasksExecutor { impl<B: Backend> Executor<TurboTasks<B>, ScheduledTask, TaskPriority> for TurboTasksExecutor {
const LOWEST_PRIORITY: TaskPriority = TaskPriority::Initial;
type Future = impl Future<Output = ()> + Send + 'static; type Future = impl Future<Output = ()> + Send + 'static;
fn execute( fn execute(
...@@ -1621,6 +1647,7 @@ impl<B: Backend> Executor<TurboTasks<B>, ScheduledTask, TaskPriority> for TurboT ...@@ -1621,6 +1647,7 @@ impl<B: Backend> Executor<TurboTasks<B>, ScheduledTask, TaskPriority> for TurboT
ScheduledTask::LocalTask { ScheduledTask::LocalTask {
ty, ty,
persistence, persistence,
parent_task: _,
execution_id: _, execution_id: _,
local_task_id, local_task_id,
global_task_state, global_task_state,
...@@ -1691,6 +1718,22 @@ impl<B: Backend> Executor<TurboTasks<B>, ScheduledTask, TaskPriority> for TurboT ...@@ -1691,6 +1718,22 @@ impl<B: Backend> Executor<TurboTasks<B>, ScheduledTask, TaskPriority> for TurboT
} }
} }
} }
fn discard(
&self,
this: &Arc<TurboTasks<B>>,
scheduled_task: ScheduledTask,
_priority: TaskPriority,
) {
match scheduled_task {
// A newer schedule of the same task replaced this one in the queue, so only that one
// executes. Balance the foreground job `TurboTasks::schedule` began for this one.
ScheduledTask::Task { .. } => this.finish_foreground_job(),
// Local tasks have unique keys and are never replaced, and they don't begin a
// foreground job either.
ScheduledTask::LocalTask { .. } => {}
}
}
} }
struct FinishedTaskState { struct FinishedTaskState {
...@@ -2762,7 +2805,7 @@ pub(crate) async fn read_local_output( ...@@ -2762,7 +2805,7 @@ pub(crate) async fn read_local_output(
// instead of waiting for a worker to pick it up. // instead of waiting for a worker to pick it up.
if execute_read_target_inline( if execute_read_target_inline(
this, this,
ScheduleKey::LocalTask(execution_id, local_task_id), ScheduleKey::current_local_task(execution_id, local_task_id),
) { ) {
continue; continue;
} }
......
...@@ -637,7 +637,7 @@ impl Future for ResolveRawVcFuture { ...@@ -637,7 +637,7 @@ impl Future for ResolveRawVcFuture {
} }
Ok(Err(listener)) => ( Ok(Err(listener)) => (
listener, listener,
Some(ScheduleKey::LocalTask(execution_id, local_task_id)), Some(ScheduleKey::current_local_task(execution_id, local_task_id)),
), ),
Err(err) => return Poll::Ready(Err(err)), Err(err) => return Poll::Ready(Err(err)),
} }
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment