Unverified Commit 18b5024d authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

turbopack: use Bitset in collect_graph (#100567)

Stacked on #100643 (`compute_chunk_group_info`), on top of #100566,
which adds `turbo-bitset`. Stack: #100566 → #100643 → this → #100568 →
#100630.

## Summary

`collect_graph` now uses `Bitset` for its per-module entry membership
sets during the fixed-point traversal, the same change as in
`compute_chunk_group_info`. It never stores these sets, so nothing needs
converting back to `RoaringBitmapWrapper`. Like #100643, it drops the
first-visit `is_zero()` check plus `clone()`, as `union_with` copies
into a set that stores no blocks.

When matching collected references to entries, the old code built a
Roaring intersection of the collecting module's and the emitted module's
memberships. It now walks the collecting module's entries and keeps
those that the emitted module's membership `contains`. That allocates no
intermediate set, and the order and the skip-when-empty behavior are
unchanged.

## Benchmarks

> **Historical, not rerun:** measured in the previous review round,
before this stack was rebased onto canary `502b6b7d` and before the
latest review changes on `b149cf8a` (#100566), `c7ea9063` (#100643),
`e322a1c1` (#100567), `6536eeea` (#100568) and `417a41fd` (#100630).
This round only reran the `turbo-bitset` microbenchmarks in #100566.

`next build --experimental-build-mode compile` with release `next-swc`
(LTO off, 16 codegen units) built from each PR's head plus local-only
stage timers, on a shared VM with 8 vCPUs and 16 GB. Each number is the
median of 5 randomized, interleaved runs per variant, with the min–max
range in parentheses, in ms. All five layers of the stack were measured
in the same interleaved set, so adjacent PRs compare directly.

| stage (ms) | previous PR | this PR |
|---|---|---|
| `bench/nested-deps-app-router-many-pages` (3,000 routes):
`collect_graph` fixed point | 189 (186–195) | **16 (15–17.1)** |
| generated 300-route app: `collect_graph` fixed point | 6.38
(6.32–6.68) | **3.37 (3.33–4.02)** |

Visit counts are identical, and the emitted JS chunks are byte-identical
across all layers of the stack on both apps. Whole-build time is too
noisy on these apps to show a reliable end-to-end difference.

<details>
<summary>Charts: stage times (linear)</summary>

Median ms per stage; the first series is the previous PR. Exact values
and ranges are in the table above.

```mermaid
xychart-beta
    title "bench/nested-deps-app-router-many-pages, median ms"
    x-axis ["`collect_graph` fixed point"]
    y-axis "ms"
    bar "previous PR" [189]
    bar "this PR" [16]
```
```mermaid
xychart-beta
    title "generated 300-route app, median ms"
    x-axis ["`collect_graph` fixed point"]
    y-axis "ms"
    bar "previous PR" [6.38]
    bar "this PR" [3.37]
```

</details>

## Verification

- `cargo test -p turbopack-core --lib`, the turbopack snapshot tests and
the collect execution tests pass. All execution tests pass on the top of
the stack.
- The real builds above.

<!-- NEXT_JS_LLM -->


<!-- fleet b56269ed-2145-4f30-b0fe-dc9762520521 -->

---------
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarLuke Sandberg <210140+lukesandberg@users.noreply.github.com>
Co-authored-by: 's avatarVercel Fleet <fleet@vercel.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent d19d3d57
...@@ -2,6 +2,7 @@ use std::{borrow::Cow, cmp::Reverse}; ...@@ -2,6 +2,7 @@ use std::{borrow::Cow, cmp::Reverse};
use anyhow::{Context, Result, bail}; use anyhow::{Context, Result, bail};
use rustc_hash::FxHashMap; use rustc_hash::FxHashMap;
use turbo_bitset::Bitset;
use turbo_rcstr::{RcStr, rcstr}; use turbo_rcstr::{RcStr, rcstr};
use turbo_tasks::{FxIndexMap, FxIndexSet, ResolvedVc, TryJoinIterExt, Vc}; use turbo_tasks::{FxIndexMap, FxIndexSet, ResolvedVc, TryJoinIterExt, Vc};
...@@ -10,10 +11,7 @@ use crate::{ ...@@ -10,10 +11,7 @@ use crate::{
emit_collect::CollectingModule, emit_collect::CollectingModule,
issue::{IssueExt, module::ModuleIssue}, issue::{IssueExt, module::ModuleIssue},
module::Module, module::Module,
module_graph::{ module_graph::{GraphTraversalAction, ModuleGraph, RefData, chunk_group_info::ChunkGroupEntry},
GraphTraversalAction, ModuleGraph, RefData,
chunk_group_info::{ChunkGroupEntry, RoaringBitmapWrapper},
},
}; };
#[turbo_tasks::value(transparent, cell = "keyed")] #[turbo_tasks::value(transparent, cell = "keyed")]
...@@ -60,7 +58,7 @@ pub async fn collect_graph(graph: Vc<ModuleGraph>) -> Result<Vc<CollectedModules ...@@ -60,7 +58,7 @@ pub async fn collect_graph(graph: Vc<ModuleGraph>) -> Result<Vc<CollectedModules
.collect::<Vec<_>>(); .collect::<Vec<_>>();
// Create a mapping of module -> ChunkGroupEntry::Entry that import it // Create a mapping of module -> ChunkGroupEntry::Entry that import it
let mut module_entry_membership: FxHashMap<ResolvedVc<Box<dyn Module>>, RoaringBitmapWrapper> = let mut module_entry_membership: FxHashMap<ResolvedVc<Box<dyn Module>>, Bitset> =
FxHashMap::with_capacity_and_hasher(module_count, Default::default()); FxHashMap::with_capacity_and_hasher(module_count, Default::default());
for (i, entries) in entry_groups.iter().enumerate() { for (i, entries) in entry_groups.iter().enumerate() {
for entry in *entries { for entry in *entries {
...@@ -127,13 +125,9 @@ pub async fn collect_graph(graph: Vc<ModuleGraph>) -> Result<Vc<CollectedModules ...@@ -127,13 +125,9 @@ pub async fn collect_graph(graph: Vc<ModuleGraph>) -> Result<Vc<CollectedModules
bail!("Module entry membership not found"); bail!("Module entry membership not found");
}; };
if current_membership.is_empty() { // On the initial visit, `union_with` copies the parent's membership.
// Initial visit, clone instead of merging if current_membership.union_with(parent_membership) {
*current_membership = parent_membership.clone(); // Added bits from parent, continue traversal because changed
Ok(GraphTraversalAction::Continue)
} else if parent_membership.is_proper_superset(current_membership) {
// Add bits from parent, and continue traversal because changed
**current_membership |= &**parent_membership;
Ok(GraphTraversalAction::Continue) Ok(GraphTraversalAction::Continue)
} else { } else {
// Unchanged, no need to forward to children // Unchanged, no need to forward to children
...@@ -158,7 +152,9 @@ pub async fn collect_graph(graph: Vc<ModuleGraph>) -> Result<Vc<CollectedModules ...@@ -158,7 +152,9 @@ pub async fn collect_graph(graph: Vc<ModuleGraph>) -> Result<Vc<CollectedModules
.get(&collecting_module) .get(&collecting_module)
.context("Module entry membership not found")?; .context("Module entry membership not found")?;
if collecting_membership.len() > 1 { // Reachable from more than one entry. The iterator is lazy, so this stops at the second
// bit instead of counting all of them.
if collecting_membership.iter().nth(1).is_some() {
ModuleIssue::new( ModuleIssue::new(
*collecting_module.ident().to_resolved().await?, *collecting_module.ident().to_resolved().await?,
rcstr!("Invalid use of __turbopack_collect__"), rcstr!("Invalid use of __turbopack_collect__"),
...@@ -219,17 +215,18 @@ pub async fn collect_graph(graph: Vc<ModuleGraph>) -> Result<Vc<CollectedModules ...@@ -219,17 +215,18 @@ pub async fn collect_graph(graph: Vc<ModuleGraph>) -> Result<Vc<CollectedModules
let matching_chunk_groups = if *emit_to_all_entries { let matching_chunk_groups = if *emit_to_all_entries {
// Add to all entry groups the collecting module is in. // Add to all entry groups the collecting module is in.
Cow::Borrowed(&**collecting_membership) Cow::Borrowed(collecting_membership)
} else { } else {
// Add to all entry groups the collecting module is in that also contain the emitted // Add to all entry groups the collecting module is in that also contain the emitted
// module. // module.
Cow::Owned((**collecting_membership).clone() & (&**emitted_membership)) Cow::Owned(collecting_membership & emitted_membership)
}; };
if !matching_chunk_groups.is_empty() { let mut matching_chunk_groups = matching_chunk_groups.iter().peekable();
if matching_chunk_groups.peek().is_some() {
let refs = collected_references let refs = collected_references
.entry(ResolvedVc::upcast(*collecting_module)) .entry(ResolvedVc::upcast(*collecting_module))
.or_default(); .or_default();
for entry in matching_chunk_groups.iter() { for entry in matching_chunk_groups {
refs.entry(entry).or_default().push(( refs.entry(entry).or_default().push((
RefData { RefData {
chunking_type: ChunkingType::Collected { chunking_type: ChunkingType::Collected {
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment