Unverified Commit a5829309 authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

turbopack: skip redundant work in merged modules reconciliation (#100238)

### What?

Removes redundant work from the reconciliation step of
`compute_merged_modules`.

This is the third of three stacked PRs; it is based on #100237.
1. intern bitmaps (#100236)
2. parallel merge of the chunk group lists (#100237)
3. reconciliation shortcuts (this PR)

### Why?

On the bench app below, only 73 modules actually lead to a split. Each
split still repeated work proportional to the common list once per
occurrence: ~18.9 M `swap_remove` calls, plus ~19.1 M items inserted
into the before/common/after hash sets. All of that ran on one thread
for ~1 s.

### How?

- **Occurrence updates:**
- **Before:** each later module of the common list had its occurrences
removed one by one, then one new occurrence was added.
  - **Now:** they are cleared instead.
- **Why it's safe:** every occurrence of such a module lies in the
common sequence. Modules in one list share a merge bitmap, so they are
in the same chunk groups.
- **Check:** `debug_assert!`s check that the set has exactly as many
entries as there are occurrences being replaced, and that all of them
are present.
- **Evidence:** counters across the 54 scope-hoisting fixtures
(including the circular ones) and ~120 generated graphs (cycles, CJS,
2–5 entries) never found any other occurrences.
  - **No release fallback:** this is a deliberate choice.
- **Exposed modules:** when a split has no "before" and no "after" list,
skip computing the modules to expose. Every reference filter requires
membership in one of those lists, so the result was always empty.

### Results

Same setup as #100236 and #100237:
- **Bench app:** `bench/nested-deps-app-router-many-pages`.
- **Builds:** cold `next build` at 32 CPUs, with release binaries built
at the base commit with #100236 and #100237, and with this change on
top.
- **Rounds:** 1 warm-up per arm, then 4 rounds in alternating order.
- **Phase times:** from the `overview` light trace.

| Metric (s) | #100237 median | this PR median | Median paired change |
Faster in |
|---|---:|---:|---:|---:|
| Wall | 50.50 | 49.60 | −1.5% | 3/4 |
| "Compiled in" | 30.00 | 28.80 | −2.0% | 4/4 |
| `compute merged modules` | 2.57 | 1.78 | −30.5% | 4/4 |
| `map chunk groups` | 0.50 | 0.50 | −1.4% | 2/4 |
| `merging chunk group lists` | 0.12 | 0.12 | −0.3% | 2/4 |
| `reconciliation` | 1.00 | 0.25 | −75.3% | 4/4 |

<details><summary>Per round (#100237 → this PR)</summary>

| Round | Order | Wall | "Compiled in" | `compute merged modules` | `map
chunk groups` | `merging chunk group lists` | `reconciliation` |
|---|---|---:|---:|---:|---:|---:|---:|
| 1 | #100237 first | 50.44 → 49.37 | 29.80 → 29.20 | 2.51 → 1.58 | 0.46
→ 0.50 | 0.11 → 0.12 | 1.01 → 0.24 |
| 2 | this PR first | 52.67 → 49.83 | 30.70 → 28.10 | 2.69 → 1.80 | 0.57
→ 0.51 | 0.13 → 0.12 | 0.97 → 0.25 |
| 3 | #100237 first | 47.67 → 48.25 | 29.00 → 28.40 | 2.42 → 1.77 | 0.47
→ 0.48 | 0.11 → 0.11 | 1.00 → 0.26 |
| 4 | this PR first | 50.56 → 50.11 | 30.20 → 30.10 | 2.63 → 1.90 | 0.54
→ 0.51 | 0.12 → 0.11 | 1.02 → 0.25 |

</details>

- **Limits:** one VM, one app, 4 rounds, and no significance test. The
wall-time change is within run-to-run noise.
- **Whole stack:** across all three PRs, `compute merged modules` goes
from ~6.6 s to ~1.8 s on this app. Each PR was measured separately
against its parent, so the percentages don't compound exactly.
- **Output:** the `.next` output is identical to #100237's. Only the
per-build random preview-mode and server-action keys differ.

### Verification

- `cargo test -p turbopack-tests --test snapshot --test execution --
scope_hoisting` (2 snapshot and 52 execution fixtures, with unchanged
snapshots). These ran as debug builds, so the new assertions were
checked.
- `cargo test -p turbopack-core`
- `cargo clippy -p turbopack-core --all-targets -- -D warnings`
- `pnpm build-all` on this branch.

<!-- NEXT_JS_LLM -->


<!-- fleet d8fee2f9-18db-4936-b59a-f8417b8f4bbc -->
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent dcbe37ad
...@@ -695,14 +695,19 @@ pub async fn compute_merged_modules(module_graph: Vc<ModuleGraph>) -> Result<Vc< ...@@ -695,14 +695,19 @@ pub async fn compute_merged_modules(module_graph: Vc<ModuleGraph>) -> Result<Vc<
let occurrences = lists_reverse_indices let occurrences = lists_reverse_indices
.get_mut(&m) .get_mut(&m)
.context("every module should have occurrences")?; .context("every module should have occurrences")?;
for common_occurrence in &common_occurrences { // All occurrences of `m` are the ones in the common sequence, so they are all
let removed = occurrences.swap_remove(&ListOccurrence { // replaced by the single new one. This holds because modules in one list share a
// merge bitmap, so they are contained in the same chunk groups, and a module that
// follows the popped module in every one of its lists has no other occurrences.
debug_assert_eq!(occurrences.len(), common_occurrences.len());
debug_assert!(common_occurrences.iter().all(|common_occurrence| {
occurrences.contains(&ListOccurrence {
chunk_group: common_occurrence.chunk_group, chunk_group: common_occurrence.chunk_group,
list: common_occurrence.list, list: common_occurrence.list,
entry: common_occurrence.entry + i, entry: common_occurrence.entry + i,
}); })
debug_assert!(removed); }));
} occurrences.clear();
occurrences.insert(ListOccurrence { occurrences.insert(ListOccurrence {
chunk_group: LISTS_COMMON_IDX, chunk_group: LISTS_COMMON_IDX,
list: common_list_index, list: common_list_index,
...@@ -722,7 +727,9 @@ pub async fn compute_merged_modules(module_graph: Vc<ModuleGraph>) -> Result<Vc< ...@@ -722,7 +727,9 @@ pub async fn compute_merged_modules(module_graph: Vc<ModuleGraph>) -> Result<Vc<
// Note that due to circular dependencies, there can be // Note that due to circular dependencies, there can be
// references that go against execution order (e.g. from "before" to // references that go against execution order (e.g. from "before" to
// "common"). // "common").
{ // Without a "before" and an "after" list, no reference can cross between them, so
// there is nothing to expose.
if !(before_list.is_empty() && after_list.is_empty()) {
let before_list = let before_list =
FxHashSet::from_iter(before_list.iter().map(|m| ResolvedVc::upcast(*m))); FxHashSet::from_iter(before_list.iter().map(|m| ResolvedVc::upcast(*m)));
let common_list = let common_list =
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment