Unverified Commit 68438424 authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

turbopack: use Bitset in compute_merged_modules (#100568)

Stacked on #100567 (`collect_graph`). Stack: #100566 → #100643 → #100567
→ this → #100630.

## Summary

`compute_merged_modules` now uses `Bitset` for its per-module merge
group sets during the fixed-point traversal, the same change as in the
PRs below. Like #100643, it drops the first-visit `is_zero()` check plus
`clone()`, as `union_with` copies into a set that stores no blocks.

The bitmap interning step that follows (which gives each distinct set a
dense id) also works on `Bitset`. Hashing a `Bitset` hashes its words
directly; hashing a `RoaringBitmapWrapper` serializes the bitmap. The
merged module result types are unchanged.

## Benchmarks

> **Historical, not rerun:** measured in the previous review round,
before this stack was rebased onto canary `502b6b7d` and before the
latest review changes on `b149cf8a` (#100566), `c7ea9063` (#100643),
`e322a1c1` (#100567), `6536eeea` (#100568) and `417a41fd` (#100630).
This round only reran the `turbo-bitset` microbenchmarks in #100566.

`next build --experimental-build-mode compile` with release `next-swc`
(LTO off, 16 codegen units) built from each PR's head plus local-only
stage timers, on a shared VM with 8 vCPUs and 16 GB. Each number is the
median of 5 randomized, interleaved runs per variant, with the min–max
range in parentheses, in ms. All five layers of the stack were measured
in the same interleaved set, so adjacent PRs compare directly.

| stage (ms) | previous PR | this PR |
|---|---|---|
| `bench/nested-deps-app-router-many-pages` (3,000 routes):
merged-modules fixed point | 109 (108–110) | **22.9 (22.4–24.5)** |
| `bench/nested-deps-app-router-many-pages` (3,000 routes): bitmap
interning | 75.9 (73.1–81.5) | **6.72 (6.16–6.87)** |
| generated 300-route app: merged-modules fixed point | 9.14 (8.63–13.7)
| **5.87 (5.79–6.04)** |
| generated 300-route app: bitmap interning | 3.59 (3.49–6.33) | **0.672
(0.631–0.69)** |

Visit counts are identical, and the emitted JS chunks are byte-identical
across all layers of the stack on both apps. Whole-build time is too
noisy on these apps to show a reliable end-to-end difference. The sum
over the whole stack is in #100630.

<details>
<summary>Charts: stage times (linear)</summary>

Median ms per stage; the first series is the previous PR. Exact values
and ranges are in the table above.

```mermaid
xychart-beta
    title "bench/nested-deps-app-router-many-pages, median ms"
    x-axis ["merged-modules fixed point", "bitmap interning"]
    y-axis "ms"
    bar "previous PR" [109, 75.9]
    bar "this PR" [22.9, 6.72]
```
```mermaid
xychart-beta
    title "generated 300-route app, median ms"
    x-axis ["merged-modules fixed point", "bitmap interning"]
    y-axis "ms"
    bar "previous PR" [9.14, 3.59]
    bar "this PR" [5.87, 0.672]
```

</details>

## Verification

- `cargo test -p turbopack-core --lib` (including the interning tests),
all turbopack snapshot tests and all turbopack execution tests pass.
- The real builds above.

<!-- NEXT_JS_LLM -->


<!-- fleet b56269ed-2145-4f30-b0fe-dc9762520521 -->

---------
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarLuke Sandberg <210140+lukesandberg@users.noreply.github.com>
Co-authored-by: 's avatarVercel Fleet <fleet@vercel.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent 18b5024d
......@@ -7,6 +7,7 @@ use std::{
use anyhow::{Context, Result, bail};
use rustc_hash::{FxBuildHasher, FxHashMap, FxHashSet};
use tracing::Instrument;
use turbo_bitset::Bitset;
use turbo_tasks::{FxIndexMap, FxIndexSet, ResolvedVc, TryFlatJoinIterExt, TryJoinIterExt, Vc};
use crate::{
......@@ -15,9 +16,7 @@ use crate::{
MergeableModulesExposed,
},
module::Module,
module_graph::{
GraphTraversalAction, ModuleGraph, RefData, chunk_group_info::RoaringBitmapWrapper,
},
module_graph::{GraphTraversalAction, ModuleGraph, RefData},
resolve::ExportUsage,
};
......@@ -76,13 +75,13 @@ impl MergedModuleInfo {
}
/// Assigns each distinct bitmap a dense id, starting at 0, so that callers can hash and compare a
/// `u32` instead of a whole bitmap (hashing a [`RoaringBitmapWrapper`] serializes it). Equal
/// bitmaps get the same id. Returns the id of every key and the number of distinct bitmaps.
/// `u32` instead of a whole bitmap. Equal bitmaps get the same id. Returns the id of every key and
/// the number of distinct bitmaps.
fn intern_bitmaps<'a, K: Eq + Hash>(
entries: impl IntoIterator<Item = (K, &'a RoaringBitmapWrapper)>,
entries: impl IntoIterator<Item = (K, &'a Bitset)>,
capacity: usize,
) -> (FxHashMap<K, u32>, usize) {
let mut ids: FxHashMap<&'a RoaringBitmapWrapper, u32> = FxHashMap::default();
let mut ids: FxHashMap<&'a Bitset, u32> = FxHashMap::default();
let mut result = FxHashMap::with_capacity_and_hasher(capacity, Default::default());
for (key, bitmap) in entries {
let next_id = ids.len() as u32;
......@@ -220,7 +219,7 @@ pub async fn compute_merged_modules(module_graph: Vc<ModuleGraph>) -> Result<Vc<
// For each module, the indices in the bitmap store which merge group entry modules
// transitively import that module. The bitmap can be treated as an opaque value, merging
// all modules with the same bitmap.
let mut module_merged_groups: FxHashMap<ResolvedVc<Box<dyn Module>>, RoaringBitmapWrapper> =
let mut module_merged_groups: FxHashMap<ResolvedVc<Box<dyn Module>>, Bitset> =
FxHashMap::with_capacity_and_hasher(module_count, Default::default());
// Entries that started a new merge group for some deopt reason
let mut entry_modules =
......@@ -311,13 +310,9 @@ pub async fn compute_merged_modules(module_graph: Vc<ModuleGraph>) -> Result<Vc<
bail!("unreachable except for eventual consistency");
};
if current_merged_groups.is_empty() {
// Initial visit, clone instead of merging
*current_merged_groups = parent_merged_groups.clone();
GraphTraversalAction::Continue
} else if parent_merged_groups.is_proper_superset(current_merged_groups) {
// Add bits from parent, and continue traversal because changed
**current_merged_groups |= &**parent_merged_groups;
// On the initial visit, `union_with` copies the parent's bitmap.
if current_merged_groups.union_with(parent_merged_groups) {
// Added bits from parent, continue traversal because changed
GraphTraversalAction::Continue
} else {
// Unchanged, no need to forward to children
......@@ -907,12 +902,10 @@ pub async fn compute_merged_modules(module_graph: Vc<ModuleGraph>) -> Result<Vc<
#[cfg(test)]
mod tests {
use roaring::RoaringBitmap;
use super::*;
fn bitmap(bits: &[u32]) -> RoaringBitmapWrapper {
RoaringBitmapWrapper(bits.iter().copied().collect::<RoaringBitmap>())
fn bitmap(bits: &[u32]) -> Bitset {
bits.iter().copied().collect()
}
#[test]
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment