Unverified Commit dcbe37ad authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

turbopack: merge chunk group lists in parallel in compute_merged_modules (#100237)

### What?

Merges the per-chunk `lists_reverse_indices` and
`intra_group_references_rev` maps of `compute_merged_modules` in
parallel, instead of entry by entry on one thread.

This is the second of three stacked PRs; it is based on #100236.
1. intern bitmaps (#100236)
2. parallel merge of the chunk group lists (this PR)
3. reconciliation shortcuts (#100238)

### Why?

After `map chunk groups`, the per-chunk results were folded into one map
with `entry(key).or_default().extend(set)` on a single thread. On the
bench app below that is ~19.1 M list occurrences, merged in a 1.7 s step
on one core.

### How?

New private helper `merge_index_maps_of_sets(parts, shard_count)`:
- **Split:** in parallel, each part is split into shards by the high
bits of the key's FxHash (`(hash as u128 * shard_count as u128) >> 64`).
Each key's position (part index, index in part) is recorded.
- **Merge:** in parallel, each shard merges its buckets in part order.
The first set of a key is moved instead of copied, and later sets extend
it.
- **Order:** the shard results are concatenated and stably sorted by
each key's first position. Each shard's output is already sorted by
position, so the stable sort merges those runs. Positions are unique, so
stability doesn't change the result.
- **One shard:** when `shard_count` is 0 or 1, the parts are merged
serially into the first part instead: it reserves the remaining parts'
lengths, moves sets for new keys and extends existing ones. No
bucketing, regrouping, parallel work or sort.

The result is exactly that of the serial merge:
- keys appear in order of first appearance;
- each key's set is extended part by part, with the same deduplication.

The shard count is `turbo_tasks::parallel::available_parallelism()`.

The gain comes from parallelizing and from moving first sets instead of
copying them; I didn't measure the two separately.

### Results

Same setup as #100236:
- **Bench app:** `bench/nested-deps-app-router-many-pages`.
- **Builds:** cold `next build` at 32 CPUs, with release binaries built
at the base commit with #100236, and with #100236 plus this change.
- **Rounds:** 1 warm-up per arm, then 4 rounds in alternating order.
- **Phase times:** from the `overview` light trace.

These numbers are for the first commit of this PR (`2a3dd855`):

| Metric (s) | #100236 median | this PR median | Median paired change |
Faster in |
|---|---:|---:|---:|---:|
| Wall | 49.62 | 48.43 | −2.9% | 3/4 |
| "Compiled in" | 30.10 | 28.60 | −6.1% | 4/4 |
| `compute merged modules` | 4.12 | 2.42 | −40.0% | 4/4 |
| `map chunk groups` | 0.45 | 0.47 | +3.4% | 2/4 |
| `merging chunk group lists` | 1.71 | 0.11 | −93.8% | 4/4 |
| `reconciliation` | 0.92 | 1.00 | +7.0% | 1/4 |

<details><summary>Per round (#100236 → this PR)</summary>

| Round | Order | Wall | "Compiled in" | `compute merged modules` | `map
chunk groups` | `merging chunk group lists` | `reconciliation` |
|---|---|---:|---:|---:|---:|---:|---:|
| 1 | #100236 first | 49.75 → 50.68 | 30.80 → 28.70 | 4.15 → 2.37 | 0.45
→ 0.45 | 1.89 → 0.10 | 0.94 → 0.99 |
| 2 | this PR first | 50.25 → 48.27 | 30.10 → 28.70 | 4.08 → 2.58 | 0.46
→ 0.49 | 1.69 → 0.11 | 1.08 → 1.06 |
| 3 | #100236 first | 49.22 → 47.18 | 30.00 → 27.80 | 4.17 → 2.36 | 0.58
→ 0.44 | 1.72 → 0.10 | 0.91 → 1.01 |
| 4 | this PR first | 49.49 → 48.60 | 30.10 → 28.50 | 3.91 → 2.46 | 0.45
→ 0.49 | 1.66 → 0.11 | 0.91 → 0.98 |

</details>

- **`reconciliation`:** it came out about 7% slower (0.92 → 1.00 s). It
runs unchanged code, so this may be the different memory layout of the
merged map, or noise. The next PR in the stack reduces it to ~0.25 s.
- **Limits:** one VM, one app, 4 rounds, and no significance test. Peak
memory of the transient buckets wasn't measured.
- **Output:** the `.next` output is identical to #100236's. Only the
per-build random preview-mode and server-action keys differ.

#### Second commit: serial merge for one shard, stable sort

Measured against the first commit (`2a3dd855`) with
`TURBO_TASKS_AVAILABLE_PARALLELISM=1`, so one shard. The process still
ran on CPUs 0–31, and only Turbo Tasks parallelism was limited. 1
warm-up per arm, then 2 rounds in alternating order:

| `merging chunk group lists` (s) | Round 1 | Round 2 |
|---|---:|---:|
| first commit | 0.362 | 0.366 |
| with second commit | 0.344 | 0.373 |

- **One shard:** no measurable difference at this sample size. The
serial path mainly drops the bucketing, sort and runtime work for one
shard; it doesn't show a speedup here.
- **Stable sort:** no separate timing A/B at 32 CPUs.
- **Output:** the 32-CPU `.next` output is identical to the first
commit's. Only the per-build random preview-mode and server-action keys
differ.

### Verification

- **Unit tests:** they compare `merge_index_maps_of_sets` with the
serial merge, including key order and the order within each set.
- **Inputs:** 20 deterministic random inputs with overlapping keys,
values repeated across parts and empty parts.
  - **Shard counts:** 0, 1, 3 and 64, plus empty input for 0, 1 and 4.
- **No-runtime test:** a plain `#[test]` runs the one-shard path without
a Tokio runtime. It failed before the serial path existed.
- `cargo test -p turbopack-core`
- `cargo test -p turbopack-tests --test snapshot --test execution --
scope_hoisting` (2 snapshot and 52 execution fixtures, with unchanged
snapshots)
- `cargo clippy -p turbopack-core --all-targets -- -D warnings`

<!-- NEXT_JS_LLM -->


<!-- fleet d8fee2f9-18db-4936-b59a-f8417b8f4bbc -->

---------
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent c8e811ee
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment