Unverified Commit f58c038c authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

turbo-tasks-backend: parallelize decoding of large batch lookups (#100453)

### What?

`TurboBackingStorage::batch_lookup_data` now decodes the `TaskStorage`
entries of a batch in parallel when the batch has at least
`PARALLEL_DECODE_THRESHOLD` (2048) items. Smaller batches still decode
sequentially. This PR also adds an opt-in Criterion benchmark that
measures where parallel decoding starts to pay off.

### Why?

Batched task restores read all entries with a single `batch_get` and
then decoded every entry one after another on the calling thread. With
large restore batches, that single-threaded decode is a bottleneck.

### How?

- `batch_lookup_data` is now split into two steps:
  - a raw fetch (`batch_lookup_raw`), unchanged I/O;
- `decode_task_storage_batch`, which takes the parallel cutoff as a
parameter.
- Both paths share the same decode closure, so ordering, `None` for
missing keys, and failing the batch on a decode error behave exactly as
before.
- The parallel path uses `turbo_tasks::parallel::map_collect`. It
borrows the raw bytes, keeps result order and the turbo-tasks context,
and doesn't add another thread pool.
- `turbo_tasks::scope_bounded` (behind `parallel::*`) now enters the
turbo-tasks context once per helper instead of once per job:
- #100485 fixed re-entering the context while the caller holds a borrow
(restores can run inside `with_turbo_tasks`). Its fix is
`ensure_turbo_tasks_scope`, which compares the current task-local on
every job.
- Now, matching `scope_unbounded`, each helper task enters the captured
context once around its drain loop. Jobs drained inline by the calling
thread run directly in the context that is already there. This drops the
per-job `Arc` clone and task-local check.
  - `ensure_turbo_tasks_scope` is removed, since nothing else uses it.
- The benchmark internals are exposed through a `#[doc(hidden)] pub mod
private` in `turbo-tasks-backend`. No public method is added to
`TurboBackingStorage`.
- New benchmark `benches/batch_lookup.rs`, gated behind an env var like
the GC bench:
- Setup builds and persists a real task graph, then reads the raw
Meta/Data bytes once.
- It times only the decode call, on a Tokio worker thread, for 2–8192
items.
  - It compares sequential, parallel, and the default threshold.

```bash
TURBOPACK_BENCH_BATCH_LOOKUP=1 cargo bench -p turbo-tasks-backend --bench mod -- batch_lookup
```

Results on an 8 vCPU machine (decode time, µs):

| Items | Meta seq | Meta par | Data seq | Data par |
|---:|---:|---:|---:|---:|
| 64 | 7.8 | 25.4 | 10.8 | 29.3 |
| 512 | 62.2 | 81.6 | 78.9 | 73.4 |
| 1024 | 123.9 | 136.4 | 159.2 | 118.5 |
| 2048 | 255.5 | 180.6 | 338.0 | 220.8 |
| 4096 | 524.2 | 331.9 | 711.1 | 401.0 |
| 8192 | 1169.1 | 626.9 | 1410.7 | 728.6 |

- Parallel decoding is 2–8× slower below roughly 256 items.
- Data breaks even around 512–1024 items, while Meta is still slower at
1024.
- At 2048 both win (1.42× for Meta, 1.53× for Data), so the shared
threshold is set conservatively at 2048.

These numbers come from a synthetic graph, so the crossover will vary
with core count, task contents, and how busy the runtime is. They
measure decode time only, not end-to-end build time.

### Tests

- New unit test: sequential and forced-parallel decoding produce
identical results for Meta and Data, including missing entries and empty
input. Both modes fail on malformed bytes.
- New round-trip test: calls `batch_lookup_data` against a real database
at 2047 and 2048 items, with reversed order, duplicate ids, and missing
ids.
- The existing regression test `parallel_inside_with_turbo_tasks` (from
#100485, in `turbo-tasks-backend/tests/invalidate_all.rs`) passes with
the per-helper context. So do the 7 `node_file_trace_persistent` cases
that panicked when large restore batches first started decoding in
parallel.
- Ran `cargo test -p turbo-tasks-backend --lib --tests`, `cargo test -p
turbo-tasks --lib`, the WASI lib tests (`--target wasm32-wasip1-threads`
with the Node WASI host), rustdoc with warnings denied, `cargo clippy -p
turbo-tasks -p turbo-tasks-backend --tests --benches -- -D warnings`,
and `cargo fmt --check`.

<!-- NEXT_JS_LLM -->


<!-- fleet 0b5d5738-9eb7-40fb-953e-c8e74f8635dc -->

---------
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent a04ebd7d
//! Benchmarks sequential vs parallel decoding of a `batch_lookup_data` batch, used to pick
//! `PARALLEL_DECODE_THRESHOLD`.
//!
//! Only decoding is timed: the raw bytes are read from a real persisted database once during setup,
//! so disk I/O does not hide the scheduling overhead of the parallel path.
//!
//! Gated behind `TURBOPACK_BENCH_BATCH_LOOKUP` because the setup builds and persists a task graph.
//! Run with:
//!
//! ```bash
//! TURBOPACK_BENCH_BATCH_LOOKUP=1 cargo bench -p turbo-tasks-backend --bench mod -- batch_lookup
//! ```
use std::{
hint::black_box,
time::{Duration, Instant},
};
use anyhow::Result;
use criterion::{BenchmarkId, Criterion, Throughput};
use turbo_tasks::{TaskId, TurboTasks, Vc};
use turbo_tasks_backend::{
BackendOptions, BackingStorageOptions, GitVersionInfo, StorageMode, TurboTasksBackend,
private::{
ArcBytes, PARALLEL_DECODE_THRESHOLD, SpecificTaskDataCategory, batch_lookup_raw,
decode_task_storage_batch,
},
};
/// Batch sizes to measure. Must not exceed the number of persisted tasks.
const SIZES: &[usize] = &[2, 8, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192];
/// Leaves of the persisted tree. Together with the interior nodes this is more than the largest
/// batch size.
const LEAVES: u32 = 16384;
const BRANCHING: u32 = 4;
fn enabled() -> bool {
!matches!(
std::env::var("TURBOPACK_BENCH_BATCH_LOOKUP")
.ok()
.as_deref(),
None | Some("") | Some("no") | Some("false")
)
}
const VERSION_INFO: GitVersionInfo = GitVersionInfo {
describe: "bench-unversioned",
dirty: false,
};
const STORAGE_OPTIONS: BackingStorageOptions = BackingStorageOptions {
is_ci: false,
is_short_session: true,
skip_compaction: true,
};
#[turbo_tasks::function]
fn leaf(index: u32) -> Vc<u32> {
Vc::cell(index.wrapping_mul(1_000_003))
}
#[turbo_tasks::function]
async fn node(index: u32, span: u32) -> Result<Vc<u32>> {
if span <= 1 {
return Ok(leaf(index));
}
let child_span = span.div_ceil(BRANCHING);
let mut sum = 0u32;
let mut start = index;
while start < index + span {
let this_span = child_span.min(index + span - start);
sum = sum.wrapping_add(*node(start, this_span).await?);
start += this_span;
}
Ok(Vc::cell(sum))
}
#[turbo_tasks::function(operation, root)]
fn root() -> Vc<u32> {
node(0, LEAVES)
}
/// Builds and persists a task graph, then reads the raw bytes of every persisted task for both
/// categories.
fn setup(rt: &tokio::runtime::Runtime) -> (Vec<ArcBytes>, Vec<ArcBytes>, tempfile::TempDir) {
let parent = std::path::PathBuf::from(format!("{}/.cache", env!("CARGO_TARGET_TMPDIR")));
std::fs::create_dir_all(&parent).unwrap();
let dir = tempfile::Builder::new()
.prefix("batch-lookup-bench-")
.tempdir_in(&parent)
.unwrap();
rt.block_on(async {
// `TurboTasks::new` needs the runtime context.
let tt = TurboTasks::new(TurboTasksBackend::new(
BackendOptions {
storage_mode: Some(StorageMode::ReadWriteOnShutdown),
..Default::default()
},
turbo_tasks_backend::turbo_backing_storage(dir.path(), &VERSION_INFO, STORAGE_OPTIONS)
.unwrap()
.0,
));
turbo_tasks::run_once(tt.clone(), async move {
root().read_strongly_consistent().await?;
anyhow::Ok(())
})
.await
.unwrap();
tt.stop_and_wait().await;
});
let (meta, data) = rt.block_on(async {
let (storage, _) =
turbo_tasks_backend::turbo_backing_storage(dir.path(), &VERSION_INFO, STORAGE_OPTIONS)
.unwrap();
// Persistent task ids are allocated densely from 1. Over-fetch and drop the misses.
let task_ids: Vec<TaskId> = (1..=LEAVES * 4)
.map(|id| TaskId::try_from(id).unwrap())
.collect();
let fetch = |category| -> Vec<ArcBytes> {
batch_lookup_raw(&storage, &task_ids, category)
.unwrap()
.into_iter()
.flatten()
.collect()
};
(
fetch(SpecificTaskDataCategory::Meta),
fetch(SpecificTaskDataCategory::Data),
)
});
let max = *SIZES.iter().max().unwrap();
assert!(
meta.len() >= max && data.len() >= max,
"not enough persisted tasks: meta={}, data={}, need {max}",
meta.len(),
data.len()
);
(meta, data, dir)
}
/// Times `iters` decodes of `bytes[..size]` on a runtime worker thread, which is where
/// `batch_lookup_data` runs in practice. Only the decode call is timed, not dropping the decoded
/// storages (which in practice are moved into the in-memory storage).
fn time_decode(
rt: &tokio::runtime::Runtime,
bytes: &[ArcBytes],
size: usize,
category: SpecificTaskDataCategory,
threshold: usize,
iters: u64,
) -> Duration {
let input: Vec<Option<ArcBytes>> = bytes[..size].iter().cloned().map(Some).collect();
rt.block_on(async move {
tokio::spawn(async move {
let mut total = Duration::ZERO;
for _ in 0..iters {
let start = Instant::now();
let decoded =
decode_task_storage_batch(black_box(&input), category, threshold).unwrap();
total += start.elapsed();
drop(black_box(decoded));
}
total
})
.await
.unwrap()
})
}
pub fn batch_lookup(c: &mut Criterion) {
if !enabled() {
return;
}
let rt = tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.unwrap();
let (meta, data, _dir) = setup(&rt);
for (name, category, bytes) in [
("meta", SpecificTaskDataCategory::Meta, &meta),
("data", SpecificTaskDataCategory::Data, &data),
] {
let mut group = c.benchmark_group(format!("turbo_tasks_backend_batch_lookup/{name}"));
group.measurement_time(Duration::from_secs(2));
group.warm_up_time(Duration::from_secs(1));
for &size in SIZES {
group.throughput(Throughput::Elements(size as u64));
for (mode, threshold) in [
("sequential", usize::MAX),
("parallel", 0),
("default", PARALLEL_DECODE_THRESHOLD),
] {
group.bench_with_input(BenchmarkId::new(mode, size), &size, |b, &size| {
b.iter_custom(|iters| time_decode(&rt, bytes, size, category, threshold, iters))
});
}
}
group.finish();
}
}
...@@ -3,6 +3,7 @@ ...@@ -3,6 +3,7 @@
use criterion::{Criterion, criterion_group, criterion_main}; use criterion::{Criterion, criterion_group, criterion_main};
pub(crate) mod batch_lookup;
pub(crate) mod cell_fan_out; pub(crate) mod cell_fan_out;
pub(crate) mod collectibles; pub(crate) mod collectibles;
pub(crate) mod gc; pub(crate) mod gc;
...@@ -15,6 +16,7 @@ criterion_group!( ...@@ -15,6 +16,7 @@ criterion_group!(
name = turbo_tasks_backend_stress; name = turbo_tasks_backend_stress;
config = Criterion::default(); config = Criterion::default();
targets = stress::fibonacci, scope_stress::scope_stress, overhead::overhead, gc::gc, targets = stress::fibonacci, scope_stress::scope_stress, overhead::overhead, gc::gc,
cell_fan_out::cell_fan_out, state_fan_out::state_fan_out, collectibles::collectibles cell_fan_out::cell_fan_out, state_fan_out::state_fan_out, collectibles::collectibles,
batch_lookup::batch_lookup
); );
criterion_main!(turbo_tasks_backend_stress); criterion_main!(turbo_tasks_backend_stress);
...@@ -58,6 +58,8 @@ use turbo_tasks::{FunctionId, TaskDirtyCause}; ...@@ -58,6 +58,8 @@ use turbo_tasks::{FunctionId, TaskDirtyCause};
use turbo_tasks_malloc::TurboMalloc; use turbo_tasks_malloc::TurboMalloc;
use self::eviction::EvictionControl; use self::eviction::EvictionControl;
#[cfg(test)]
pub(crate) use self::storage::TrackOutcome;
pub use self::{ pub use self::{
eviction::EvictionMode, eviction::EvictionMode,
storage::{EvictionCounts, SpecificTaskDataCategory, TaskDataCategory}, storage::{EvictionCounts, SpecificTaskDataCategory, TaskDataCategory},
......
...@@ -103,3 +103,26 @@ pub fn compact_database( ...@@ -103,3 +103,26 @@ pub fn compact_database(
db.shutdown(); db.shutdown();
Ok(()) Ok(())
} }
/// Internals exposed for benchmarks only. Not a stable API.
#[doc(hidden)]
pub mod private {
use anyhow::Result;
pub use turbo_persistence::ArcBytes;
use turbo_tasks::TaskId;
use crate::kv_backing_storage::TurboBackingStorage;
pub use crate::{
backend::{SpecificTaskDataCategory, storage_schema::TaskStorage},
kv_backing_storage::{PARALLEL_DECODE_THRESHOLD, decode_task_storage_batch},
};
/// See `TurboBackingStorage::batch_lookup_raw`.
pub fn batch_lookup_raw(
storage: &TurboBackingStorage,
task_ids: &[TaskId],
category: SpecificTaskDataCategory,
) -> Result<Vec<Option<ArcBytes>>> {
storage.batch_lookup_raw(task_ids, category)
}
}
...@@ -2286,20 +2286,6 @@ pub fn turbo_tasks_scope<T>(tt: Arc<dyn TurboTasksApi>, f: impl FnOnce() -> T) - ...@@ -2286,20 +2286,6 @@ pub fn turbo_tasks_scope<T>(tt: Arc<dyn TurboTasksApi>, f: impl FnOnce() -> T) -
TURBO_TASKS.sync_scope(tt, f) TURBO_TASKS.sync_scope(tt, f)
} }
/// Like [`turbo_tasks_scope`], but runs `f` directly when `tt` is already the current turbo-tasks
/// instance. Entering the scope again would be redundant, and panics while the current one is
/// borrowed (e.g. inside [`with_turbo_tasks`]).
pub(crate) fn ensure_turbo_tasks_scope<T>(tt: Arc<dyn TurboTasksApi>, f: impl FnOnce() -> T) -> T {
let is_current = TURBO_TASKS
.try_with(|current| std::ptr::addr_eq(Arc::as_ptr(current), Arc::as_ptr(&tt)))
.unwrap_or(false);
if is_current {
f()
} else {
TURBO_TASKS.sync_scope(tt, f)
}
}
pub fn turbo_tasks_future_scope<T>( pub fn turbo_tasks_future_scope<T>(
tt: Arc<dyn TurboTasksApi>, tt: Arc<dyn TurboTasksApi>,
f: impl Future<Output = T>, f: impl Future<Output = T>,
......
...@@ -26,10 +26,7 @@ use parking_lot::Mutex; ...@@ -26,10 +26,7 @@ use parking_lot::Mutex;
use tokio::{runtime::Handle, task::block_in_place}; use tokio::{runtime::Handle, task::block_in_place};
use tracing::{Span, info_span}; use tracing::{Span, info_span};
use crate::{ use crate::{TurboTasksApi, manager::try_turbo_tasks, turbo_tasks_scope};
TurboTasksApi,
manager::{ensure_turbo_tasks_scope, try_turbo_tasks},
};
/// A job placed on the work queue: its result-slot index and the closure to run. /// A job placed on the work queue: its result-slot index and the closure to run.
type WorkQueueJob = (usize, Box<dyn FnOnce() + Send + 'static>); type WorkQueueJob = (usize, Box<dyn FnOnce() + Send + 'static>);
...@@ -122,6 +119,8 @@ pub struct Scope<'scope, 'env: 'scope, R: Send + 'env> { ...@@ -122,6 +119,8 @@ pub struct Scope<'scope, 'env: 'scope, R: Send + 'env> {
/// Max number of threads to use, threads are only spawned when needed. The calling thread /// Max number of threads to use, threads are only spawned when needed. The calling thread
/// counts towards this budget, so we spawn at most `worker_tasks - 1` helpers. /// counts towards this budget, so we spawn at most `worker_tasks - 1` helpers.
worker_tasks: NonZeroUsize, worker_tasks: NonZeroUsize,
/// The caller's turbo-tasks context, entered by each helper task so jobs on helper threads see
/// the same context as jobs drained inline by the calling thread.
turbo_tasks: Option<Arc<dyn TurboTasksApi>>, turbo_tasks: Option<Arc<dyn TurboTasksApi>>,
span: Span, span: Span,
/// Invariance over 'env, to make sure 'env cannot shrink, which is necessary for soundness. /// Invariance over 'env, to make sure 'env cannot shrink, which is necessary for soundness.
...@@ -170,19 +169,11 @@ impl<'scope, 'env: 'scope, R: Send + 'env> Scope<'scope, 'env, R> { ...@@ -170,19 +169,11 @@ impl<'scope, 'env: 'scope, R: Send + 'env> Scope<'scope, 'env, R> {
assert!(index < self.results.len(), "Too many tasks spawned"); assert!(index < self.results.len(), "Too many tasks spawned");
let result_cell: &Mutex<Option<R>> = &self.results[index]; let result_cell: &Mutex<Option<R>> = &self.results[index];
let turbo_tasks = self.turbo_tasks.clone(); // The turbo-tasks context is not entered per job: helpers enter it once (below), and the
// calling thread already has it. Re-entering it on the calling thread would panic when the
// caller holds a borrow of the task-local, e.g. inside `with_turbo_tasks`.
let f: Box<dyn FnOnce() + Send + 'scope> = Box::new(|| { let f: Box<dyn FnOnce() + Send + 'scope> = Box::new(|| {
let result = { let result = f();
if let Some(turbo_tasks) = turbo_tasks {
// Ensure that the turbo tasks context is maintained across the job. Jobs run
// on the calling thread already have it (possibly borrowed, e.g. inside
// `with_turbo_tasks`), so it's only entered when needed.
ensure_turbo_tasks_scope(turbo_tasks, f)
} else {
// If no turbo tasks context is available, just run the job.
f()
}
};
*result_cell.lock() = Some(result); *result_cell.lock() = Some(result);
}); });
let f: *mut (dyn FnOnce() + Send + 'scope) = Box::into_raw(f); let f: *mut (dyn FnOnce() + Send + 'scope) = Box::into_raw(f);
...@@ -214,9 +205,15 @@ impl<'scope, 'env: 'scope, R: Send + 'env> Scope<'scope, 'env, R> { ...@@ -214,9 +205,15 @@ impl<'scope, 'env: 'scope, R: Send + 'env> Scope<'scope, 'env, R> {
if index < self.worker_tasks.get() - 1 { if index < self.worker_tasks.get() - 1 {
let inner = self.inner.clone(); let inner = self.inner.clone();
let span = self.span.clone(); let span = self.span.clone();
let turbo_tasks = self.turbo_tasks.clone();
self.handle.spawn(async move { self.handle.spawn(async move {
let _span = span.entered(); let _span = span.entered();
inner.run_jobs(); if let Some(turbo_tasks) = turbo_tasks {
// Ensure that the turbo tasks context is maintained across the jobs.
turbo_tasks_scope(turbo_tasks, || inner.run_jobs());
} else {
inner.run_jobs();
}
}); });
} }
} }
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment