Unverified Commit d19d3d57 authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

turbopack: use Bitset in compute_chunk_group_info (#100643)

Stacked on #100566, which adds `turbo-bitset`. Stack: #100566 → this →
#100567 (`collect_graph`) → #100568 (`compute_merged_modules`) → #100630
(persisted chunk group sets).

## Summary

`compute_chunk_group_info` tracked each module's chunk groups with
`RoaringBitmapWrapper` during its fixed-point traversal. The traversal
checks "would `current |= parent` change `current`?" on every edge and
then does the union; on Roaring's array containers both cost time
proportional to the number of bits, which made this the slowest of the
module graph passes on `bench/nested-deps-app-router-many-pages`.

It now uses `Bitset` for the working per-module sets, and converts them
with `to_roaring()` when building the result. `ModuleToChunkGroups`, the
batch types and production chunking still use `RoaringBitmapWrapper`, so
persisted values are unchanged (#100630 switches those). The old code
called `RoaringBitmapWrapper::is_proper_superset` (which actually
computed `!parent.is_subset(current)`) and then `|=`. Now
`current.union_with(parent)` does both in one pass and returns whether
it added a bit. The separate first-visit `is_zero()` check plus
`clone()` is gone too: `union_with` copies into a set that stores no
blocks. This doesn't change the traversal, as a parent's set is never
empty when its children are visited.

## Benchmarks

> **Historical, not rerun:** measured in the previous review round,
before this stack was rebased onto canary `502b6b7d` and before the
latest review changes on `b149cf8a` (#100566), `c7ea9063` (#100643),
`e322a1c1` (#100567), `6536eeea` (#100568) and `417a41fd` (#100630).
This round only reran the `turbo-bitset` microbenchmarks in #100566.

`next build --experimental-build-mode compile` with release `next-swc`
(LTO off, 16 codegen units) built from each PR's head plus local-only
stage timers, on a shared VM with 8 vCPUs and 16 GB. Each number is the
median of 5 randomized, interleaved runs per variant, with the min–max
range in parentheses, in ms. All five layers of the stack were measured
in the same interleaved set, so adjacent PRs compare directly.

| stage (ms) | previous PR | this PR |
|---|---|---|
| `bench/nested-deps-app-router-many-pages` (3,000 routes): chunk-group
fixed point | 379 (375–384) | **49.9 (45.8–54.2)** |
| `bench/nested-deps-app-router-many-pages` (3,000 routes): building
`ChunkGroupInfo` | 3.73 (3.16–4.08) | **37.4 (35.7–42.2)** |
| generated 300-route app: chunk-group fixed point | 17.3 (16.4–18.1) |
**7.09 (6.93–7.33)** |
| generated 300-route app: building `ChunkGroupInfo` | 0.444
(0.417–0.471) | **3.57 (3.35–3.62)** |

Building `ChunkGroupInfo` now includes the `to_roaring()` conversion,
which #100630 removes. Visit counts are identical, and the emitted JS
chunks are byte-identical across all layers of the stack on both apps
(14,028 and 2,142 files, same digests). Whole-build time is too noisy on
these apps to show a reliable end-to-end difference. The microbenchmarks
are in #100566.

<details>
<summary>Charts: stage times (linear)</summary>

Median ms per stage; the first series is the previous PR. Exact values
and ranges are in the table above.

```mermaid
xychart-beta
    title "bench/nested-deps-app-router-many-pages, median ms"
    x-axis ["chunk-group fixed point", "building `ChunkGroupInfo`"]
    y-axis "ms"
    bar "previous PR" [379, 3.73]
    bar "this PR" [49.9, 37.4]
```
```mermaid
xychart-beta
    title "generated 300-route app, median ms"
    x-axis ["chunk-group fixed point", "building `ChunkGroupInfo`"]
    y-axis "ms"
    bar "previous PR" [17.3, 0.444]
    bar "this PR" [7.09, 3.57]
```

</details>

## Verification

- `cargo test -p turbopack-core --lib` and the turbopack snapshot tests
pass without snapshot changes; the execution tests pass on the top of
the stack.
- The real builds above.

<!-- NEXT_JS_LLM -->


<!-- fleet b56269ed-2145-4f30-b0fe-dc9762520521 -->

---------
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarLuke Sandberg <210140+lukesandberg@users.noreply.github.com>
Co-authored-by: 's avatarVercel Fleet <fleet@vercel.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent 72f5f5fc
......@@ -9657,6 +9657,7 @@ dependencies = [
"tokio",
"tracing",
"turbo-bincode",
"turbo-bitset",
"turbo-esregex",
"turbo-frozenmap",
"turbo-prehash",
......
......@@ -41,6 +41,7 @@ swc_sourcemap = { workspace = true }
swc_core = { workspace = true, features = ["ecma_preset_env", "common"] }
tracing = { workspace = true }
turbo-bincode = { workspace = true }
turbo-bitset = { workspace = true }
turbo-esregex = { workspace = true }
turbo-frozenmap = { workspace = true }
turbo-prehash = { workspace = true }
......
......@@ -12,6 +12,7 @@ use indexmap::map::Entry;
use roaring::RoaringBitmap;
use rustc_hash::FxHashMap;
use tracing::Instrument;
use turbo_bitset::Bitset;
use turbo_rcstr::RcStr;
use turbo_tasks::{
FxIndexMap, FxIndexSet, NonLocalValue, ResolvedVc, TaskInput, TryJoinIterExt, ValueToString,
......@@ -604,6 +605,16 @@ impl Deref for ChunkGroupId {
}
}
/// Converts the `Bitset`s that the traversal works on into the `RoaringBitmapWrapper`s of the
/// result.
fn to_roaring_map(
map: FxHashMap<ResolvedVc<Box<dyn Module>>, Bitset>,
) -> FxHashMap<ResolvedVc<Box<dyn Module>>, RoaringBitmapWrapper> {
map.into_iter()
.map(|(module, chunk_groups)| (module, RoaringBitmapWrapper(chunk_groups.to_roaring())))
.collect()
}
pub async fn compute_chunk_group_info(graph: &ModuleGraph) -> Result<Vc<ChunkGroupInfo>> {
let span_outer = tracing::info_span!(
"compute chunk group info",
......@@ -619,14 +630,13 @@ pub async fn compute_chunk_group_info(graph: &ModuleGraph) -> Result<Vc<ChunkGro
FxIndexSet<ResolvedVc<Box<dyn Module>>>,
> = FxIndexMap::default();
// For each module, the indices in the bitmap store which chunk groups in `chunk_groups_map`
// that module is part of.
let mut module_chunk_groups: FxHashMap<ResolvedVc<Box<dyn Module>>, RoaringBitmapWrapper> =
// For each module, the indices in the bitset store which chunk groups in `chunk_groups_map`
// that module is part of. The traversal works on fast `Bitset`s, which are converted
// into `RoaringBitmapWrapper`s for the result.
let mut module_chunk_groups: FxHashMap<ResolvedVc<Box<dyn Module>>, Bitset> =
FxHashMap::default();
let mut async_loader_chunk_groups: FxHashMap<ResolvedVc<Box<dyn Module>>, Bitset> =
FxHashMap::default();
let mut async_loader_chunk_groups: FxHashMap<
ResolvedVc<Box<dyn Module>>,
RoaringBitmapWrapper,
> = FxHashMap::default();
let module_count = graph
.graphs
......@@ -639,12 +649,12 @@ pub async fn compute_chunk_group_info(graph: &ModuleGraph) -> Result<Vc<ChunkGro
let entries = graph.all_chunk_group_entries().collect::<Vec<_>>();
// First, rank every module in reverse postorder (so that a module is visited after the
// modules importing it), and initialize `module_chunk_groups` with empty bitmaps. The ranks
// modules importing it), and initialize `module_chunk_groups` with empty bitsets. The ranks
// from the entries of the graph are shared with other analyses of this graph.
let module_rank = graph.entry_ranks()?;
module_chunk_groups.reserve(module_rank.len());
for module in module_rank.keys() {
module_chunk_groups.insert(*module, RoaringBitmapWrapper::default());
module_chunk_groups.insert(*module, Bitset::new());
}
// ----
......@@ -724,10 +734,7 @@ pub async fn compute_chunk_group_info(graph: &ModuleGraph) -> Result<Vc<ChunkGro
|parent_info: Option<(ResolvedVc<Box<dyn Module>>, &'_ RefData, _)>,
node: ResolvedVc<Box<dyn Module>>,
_,
module_chunk_groups: &mut FxHashMap<
ResolvedVc<Box<dyn Module>>,
RoaringBitmapWrapper,
>|
module_chunk_groups: &mut FxHashMap<ResolvedVc<Box<dyn Module>>, Bitset>|
-> Result<GraphTraversalAction> {
enum ChunkGroupInheritance<It: Iterator<Item = ChunkGroupKey>> {
Inherit(ResolvedVc<Box<dyn Module>>),
......@@ -743,8 +750,7 @@ pub async fn compute_chunk_group_info(graph: &ModuleGraph) -> Result<Vc<ChunkGro
let parent_groups = module_chunk_groups
.get(&parent)
.context("Module chunk group not found")?;
**async_loader_chunk_groups.entry(node).or_default() |=
&**parent_groups;
*async_loader_chunk_groups.entry(node).or_default() |= parent_groups;
ChunkGroupInheritance::ChunkGroup(Either::Left(std::iter::once(
ChunkGroupKey::Async(node),
)))
......@@ -871,17 +877,14 @@ pub async fn compute_chunk_group_info(graph: &ModuleGraph) -> Result<Vc<ChunkGro
id
});
let chunk_groups =
RoaringBitmapWrapper(RoaringBitmap::from_iter(chunk_group_ids));
let chunk_groups = Bitset::from_iter(chunk_group_ids);
// Assign chunk group to the target node (the entry of the chunk group)
let bitset = module_chunk_groups
.get_mut(&node)
.context("Module chunk group not found")?;
if chunk_groups.is_proper_superset(bitset) {
// Add bits from parent, and continue traversal because changed
**bitset |= chunk_groups.into_inner();
if bitset.union_with(&chunk_groups) {
// Added bits from parent, continue traversal because changed
GraphTraversalAction::Continue
} else {
// Unchanged, no need to forward to children
......@@ -905,13 +908,9 @@ pub async fn compute_chunk_group_info(graph: &ModuleGraph) -> Result<Vc<ChunkGro
bail!("Module chunk groups not found");
};
if current_chunk_groups.is_empty() {
// Initial visit, clone instead of merging
*current_chunk_groups = parent_chunk_groups.clone();
GraphTraversalAction::Continue
} else if parent_chunk_groups.is_proper_superset(current_chunk_groups) {
// Add bits from parent, and continue traversal because changed
**current_chunk_groups |= &**parent_chunk_groups;
// On the initial visit, `union_with` copies the parent's chunk groups.
if current_chunk_groups.union_with(parent_chunk_groups) {
// Added bits from parent, continue traversal because changed
GraphTraversalAction::Continue
} else {
// Unchanged, no need to forward to children
......@@ -950,7 +949,7 @@ pub async fn compute_chunk_group_info(graph: &ModuleGraph) -> Result<Vc<ChunkGro
let mut buckets = BTreeMap::default();
for (module, key) in &module_chunk_groups {
if !key.is_empty() {
if !key.is_zero() {
buckets
.entry(key.iter().collect::<Vec<_>>())
.or_insert(BTreeSet::new())
......@@ -1041,8 +1040,8 @@ pub async fn compute_chunk_group_info(graph: &ModuleGraph) -> Result<Vc<ChunkGro
let chunk_group_priority_routes = RoaringBitmapWrapper(priority_routes);
Ok(ChunkGroupInfo {
module_chunk_groups: ResolvedVc::cell(module_chunk_groups),
async_loader_chunk_groups: ResolvedVc::cell(async_loader_chunk_groups),
module_chunk_groups: ResolvedVc::cell(to_roaring_map(module_chunk_groups)),
async_loader_chunk_groups: ResolvedVc::cell(to_roaring_map(async_loader_chunk_groups)),
chunk_group_keys: chunk_groups_map.keys().cloned().collect(),
chunk_group_indices: ResolvedVc::cell(
chunk_groups_map
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment