Unverified Commit 88350cfe authored by Benjamin Woodruff's avatar Benjamin Woodruff Committed by GitHub

[ci] Set `TURBO_TASKS_AVAILABLE_PARALLELISM` to 4 in most e2e tests (#93996)

By default, turbo-tasks will try to use every available CPU core.

While we do want it to use as many resources as possible if the CPU is
otherwise idle, this may hurt us when the CPU is busy, as it will lead
to more locking, more thread contention, more memory usage, etc.

**This environment variable does not strictly limit Turbopack to using 4
cores, it just restricts the number of main tokio worker threads and
changes how we shard dashmaps.**

We currently are using 16 core CI machines, but I want to experiment
with 32 core build machines with a higher `TEST_CONCURRENCY`, and the
per-core scaling of Turbopack was a concern that came up, as wall time
improvements are sub-linear starting somewhere around 16 cores.


## Test Plan

Watch CI, compare build times in this commit and the parent one.

Attempt with a value of 2: https://github.com/vercel/next.js/pull/94001
Attempt with a value of 8: https://github.com/vercel/next.js/pull/94002

![Screenshot 2026-05-20 at
4.53.38 PM.png](https://app.graphite.com/user-attachments/assets/2e07db4a-068f-4344-863c-995877b254ce.png)

I think this data is too noisy to draw any conclusions.
parent 90ae0ee0
...@@ -132,6 +132,10 @@ env: ...@@ -132,6 +132,10 @@ env:
# run-tests.js reads `TEST_CONCURRENCY` if no explicit `--concurrency` or `-c` # run-tests.js reads `TEST_CONCURRENCY` if no explicit `--concurrency` or `-c`
# argument is provided # argument is provided
TEST_CONCURRENCY: 8 TEST_CONCURRENCY: 8
# overrides `turbo_tasks::parallel::available_parallelism`. Smaller values are more CPU and memory
# efficient, but won't fully utilize all available CPU cores. A small value here makes sense since
# we're running many tests in parallel.
TURBO_TASKS_AVAILABLE_PARALLELISM: 4
# disable backtrace for test snapshots # disable backtrace for test snapshots
RUST_BACKTRACE: 0 RUST_BACKTRACE: 0
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment