Unverified Commit 5f4692ea authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

Turbopack: consume graph-derived chunk groups (#98734)

### What?

Next.js output chunking uses the merged chunk groups derived by
`ChunkGroupInfo`, rather than reconstructing them in output code:

- **Client references** (`app_client_references_chunks.rs`): each layout
segment chunks the isolated merged client reference groups the graph
derived for its references, with one `chunk_group` call per segment and
side (`ssr modules`, `client modules`, using the list form from #99686).
There is no manual availability threading or result concatenation.
- **Server utilities** (`app.rs`, per-page module graph): the entry's
server utilities are chunked as the graph's shared merged group for the
entry and the server utility merge tag, instead of a hand-built
`SharedMerged` group.
- **Ordered traversal**: chunk traversal starts from
`ModuleBatchesGraph::get_ordered_entries` for each group, in list order.

The chunking API still takes `ChunkGroup` values; this PR passes the
groups exactly as the graph registered them. #99750 then switches the
API to keys.

This is PR 3 of a six-PR stack, based on `canary` (#98733, #99686 and
#99749 are already merged):

1. #98823 — Pages Router dev client graph chaining
2. #99910 — regression test for Pages Router `_app` availability
(#99789)
3. **This PR** — use graph-derived merged groups
4. #99750 — chunking takes a `ChunkGroupKey`
5. #98685 — chunk-group availability bitmaps
6. #99050 — keyed chunk-group index lookups

### Why?

The module graph derives merged parents from actual membership bitmaps.
Output code previously guessed a parent and assembled an entry list by
hand, which could disagree with the graph for shared client references
and framework routes.

A segment's client references can derive several merged groups (e.g. one
per page rendering the root layout). Chunking each one separately ran
one chunking task per group; chunking them together runs one per segment
and side.

Graph-derived merged entries have unspecified raw order, so chunk
traversal must use the postorder `ModuleBatchesGraph` already computes,
to preserve CSS evaluation order.

### How?

- `ChunkGroupInfo` exposes exact memberships
(`get_chunk_groups_for_module`) and the shared merged child of a parent
and merge tag (`get_shared_merged_chunk_group`).
- `ClientReference` retains its source module (`parent_module`), so
merge-tagged groups are scoped to their graph parent. Component-owned
references are intersected with their server component's memberships,
which prevents collecting across segments, falling back to the source
module's groups when the intersection is empty.
- The client reference merge tags are shared `pub const`s in
`next_client_reference` (`CLIENT_MERGE_TAG`, used by CSS and ecmascript
client references, and `SSR_MERGE_TAG`), so producers and the chunking
code match on the same values.
- The `server side rendering` / `client side rendering` tracing spans
are kept around the per-segment chunking.

### Verification

- Workspace `cargo clippy --all-targets -D warnings`, `cargo check
--workspace --all-targets`, `cargo fmt --check`
- Turbopack snapshots: 132 passed, no snapshot changes
- `turbo-tasks` + `turbopack-core` + `next-core` unit tests: 364 passed
(incl. the new parent-index and merge-tag tests)
- Production `layout-segment-chunk-sharing`, with a new assertion that
the root layout's client and SSR modules are each chunked by one task.
The suite now waits until the trace server has fully read the trace
before querying spans.
- Turbopack production and dev: `404-page-router`,
`app-dir/actions/app-action`, `app-dir/initial-css-order`; production
`edge-pages-support`

<!-- NEXT_JS_LLM -->


<!-- fleet 4f67c6f7-bbe2-454b-9a17-822a60ef8f7e -->
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent 2a697ac4
use anyhow::{Context, Result}; use anyhow::{Context, Result, ensure};
use bincode::{Decode, Encode}; use bincode::{Decode, Encode};
use next_core::{ use next_core::{
app_structure::{ app_structure::{
...@@ -1950,46 +1950,65 @@ impl AppEndpoint { ...@@ -1950,46 +1950,65 @@ impl AppEndpoint {
// `per_page_module_graph`), so without this the layout segments would share // `per_page_module_graph`), so without this the layout segments would share
// very little across pages. // very little across pages.
// //
// It must not be created in production. Its `parent` is *this* endpoint's // It must not be created in production. The group is the merged child of
// entry chunk group, and `parent` is part of a merged group's identity, so // *this* endpoint's entry chunk group, and that parent is part of a merged
// the group -- and hence the availability info it seeds into the layout // group's identity, so the group -- and hence the availability info it seeds
// segment chain below -- would differ for every endpoint. That makes each // into the layout segment chain below -- would differ for every endpoint.
// shared layout segment re-chunk once per descendant endpoint. In production // That makes each shared layout segment re-chunk once per descendant
// a single whole-app module graph already shares these modules, so they are // endpoint. In production a single whole-app module graph already shares
// chunked as part of the entry instead. // these modules, so they are chunked as part of the entry instead.
if *project.per_page_module_graph().await? { if *project.per_page_module_graph().await? {
let chunk_group_info = module_graph.chunk_group_info(); let chunk_group_info = module_graph.chunk_group_info();
let span = tracing::trace_span!("server utils"); let span = tracing::trace_span!("server utils");
async { async {
let parent_chunk_group = *chunk_group_info let has_server_utils = !client_references.server_utils.is_empty();
.get_index_of(entry_chunk_group.key())
.await?; // The server utilities of this entry are merged into a single group:
// the one merged child of the entry chunk group carrying the server
// utility merge tag.
let derived_group = if !has_server_utils {
None
} else {
let entry_index = *chunk_group_info
.get_index_of(entry_chunk_group.key())
.await?;
let merged = chunk_group_info
.get_shared_merged_chunk_group(
entry_index,
NEXT_SERVER_UTILITY_MERGE_TAG,
)
.await?;
merged.as_ref().map(|group| group.chunk_group.clone())
};
ensure!(
!has_server_utils || derived_group.is_some(),
"could not find a graph-derived shared merged group for server \
utilities"
);
let server_utils = client_references let mut combined_chunk_group = ChunkGroupResult::empty_resolved();
.server_utils if let Some(group) = derived_group {
.iter() combined_chunk_group = chunking_context
.map(async |m| Ok(ResolvedVc::upcast(m.await?.module))) .chunk_group(
.try_join() AssetIdent::from_path(
.await?; this.app_project
let chunk_group = chunking_context .project()
.chunk_group( .project_path()
AssetIdent::from_path( .owned()
this.app_project.project().project_path().owned().await?, .await?,
)
.with_modifier(rcstr!("server-utils"))
.into_vc(),
smallvec![group],
module_graph,
AvailabilityInfo::root(),
) )
.with_modifier(rcstr!("server-utils")) .to_resolved()
.into_vc(), .await?;
smallvec![ChunkGroup::SharedMerged { }
merge_tag: NEXT_SERVER_UTILITY_MERGE_TAG.clone(),
entries: server_utils,
parent: parent_chunk_group,
}],
module_graph,
AvailabilityInfo::root(),
)
.to_resolved()
.await?;
current_chunk_group = chunk_group; current_chunk_group = combined_chunk_group;
anyhow::Ok(()) anyhow::Ok(())
} }
......
...@@ -612,6 +612,7 @@ impl ClientReferencesGraph { ...@@ -612,6 +612,7 @@ impl ClientReferencesGraph {
// Client reference used by the template, not a server component // Client reference used by the template, not a server component
client_references.push(ClientReference { client_references.push(ClientReference {
parent_module: node,
server_component: None, server_component: None,
ty, ty,
}); });
...@@ -665,6 +666,7 @@ impl ClientReferencesGraph { ...@@ -665,6 +666,7 @@ impl ClientReferencesGraph {
}; };
client_references.push(ClientReference { client_references.push(ClientReference {
parent_module: module,
server_component: Some(sc), server_component: Some(sc),
ty, ty,
}); });
......
...@@ -13,6 +13,8 @@ use turbopack_core::{ ...@@ -13,6 +13,8 @@ use turbopack_core::{
}; };
use turbopack_css::chunk::CssChunkPlaceable; use turbopack_css::chunk::CssChunkPlaceable;
use crate::next_client_reference::CLIENT_MERGE_TAG;
/// A [`CssClientReferenceModule`] is a marker module used to indicate which /// A [`CssClientReferenceModule`] is a marker module used to indicate which
/// client reference should appear in the client reference manifest. /// client reference should appear in the client reference manifest.
#[turbo_tasks::value] #[turbo_tasks::value]
...@@ -106,7 +108,7 @@ impl ModuleReference for CssClientReference { ...@@ -106,7 +108,7 @@ impl ModuleReference for CssClientReference {
fn chunking_type(&self) -> Option<ChunkingType> { fn chunking_type(&self) -> Option<ChunkingType> {
Some(ChunkingType::Isolated { Some(ChunkingType::Isolated {
_ty: ChunkGroupType::Evaluated, _ty: ChunkGroupType::Evaluated,
merge_tag: Some(rcstr!("client")), merge_tag: Some(CLIENT_MERGE_TAG),
}) })
} }
} }
...@@ -34,6 +34,8 @@ use turbopack_ecmascript::{ ...@@ -34,6 +34,8 @@ use turbopack_ecmascript::{
utils::StringifyJs, utils::StringifyJs,
}; };
use crate::next_client_reference::{CLIENT_MERGE_TAG, SSR_MERGE_TAG};
/// A [`EcmascriptClientReferenceModule`] is used in RSC to represent /// A [`EcmascriptClientReferenceModule`] is used in RSC to represent
/// a client or SSR asset. /// a client or SSR asset.
#[turbo_tasks::value] #[turbo_tasks::value]
...@@ -186,13 +188,6 @@ impl EcmascriptClientReferenceModule { ...@@ -186,13 +188,6 @@ impl EcmascriptClientReferenceModule {
} }
} }
pub fn ecmascript_client_reference_merge_tag() -> RcStr {
rcstr!("client")
}
pub fn ecmascript_client_reference_merge_tag_ssr() -> RcStr {
rcstr!("ssr")
}
#[turbo_tasks::value_impl] #[turbo_tasks::value_impl]
impl Module for EcmascriptClientReferenceModule { impl Module for EcmascriptClientReferenceModule {
#[turbo_tasks::function] #[turbo_tasks::function]
...@@ -229,7 +224,7 @@ impl Module for EcmascriptClientReferenceModule { ...@@ -229,7 +224,7 @@ impl Module for EcmascriptClientReferenceModule {
EcmascriptClientReference::new( EcmascriptClientReference::new(
*ResolvedVc::upcast(*client_module), *ResolvedVc::upcast(*client_module),
ChunkGroupType::Evaluated, ChunkGroupType::Evaluated,
Some(ecmascript_client_reference_merge_tag()), Some(CLIENT_MERGE_TAG),
rcstr!("ecmascript client reference to client"), rcstr!("ecmascript client reference to client"),
) )
.to_resolved() .to_resolved()
...@@ -239,7 +234,7 @@ impl Module for EcmascriptClientReferenceModule { ...@@ -239,7 +234,7 @@ impl Module for EcmascriptClientReferenceModule {
EcmascriptClientReference::new( EcmascriptClientReference::new(
*ResolvedVc::upcast(*ssr_module), *ResolvedVc::upcast(*ssr_module),
ChunkGroupType::Entry, ChunkGroupType::Entry,
Some(ecmascript_client_reference_merge_tag_ssr()), Some(SSR_MERGE_TAG),
rcstr!("ecmascript client reference to ssr"), rcstr!("ecmascript client reference to ssr"),
) )
.to_resolved() .to_resolved()
......
use turbo_rcstr::{RcStr, rcstr};
pub(crate) mod css_client_reference; pub(crate) mod css_client_reference;
pub(crate) mod ecmascript_client_reference; pub(crate) mod ecmascript_client_reference;
pub(crate) mod visit_client_reference; pub(crate) mod visit_client_reference;
/// Merge tag of the isolated chunk groups for the client side of client references (CSS and
/// ecmascript).
pub const CLIENT_MERGE_TAG: RcStr = rcstr!("client");
/// Merge tag of the isolated chunk groups for the SSR side of ecmascript client references.
pub const SSR_MERGE_TAG: RcStr = rcstr!("ssr");
pub use css_client_reference::{ pub use css_client_reference::{
css_client_reference_module::CssClientReferenceModule, css_client_reference_module::CssClientReferenceModule,
css_client_reference_transition::NextCssClientReferenceTransition, css_client_reference_transition::NextCssClientReferenceTransition,
......
...@@ -27,6 +27,9 @@ use crate::{ ...@@ -27,6 +27,9 @@ use crate::{
Copy, Clone, Eq, PartialEq, Hash, Debug, ValueDebugFormat, NonLocalValue, Encode, Decode, Copy, Clone, Eq, PartialEq, Hash, Debug, ValueDebugFormat, NonLocalValue, Encode, Decode,
)] )]
pub struct ClientReference { pub struct ClientReference {
/// The module whose reference creates the client/SSR chunk group.
pub parent_module: ResolvedVc<Box<dyn Module>>,
/// The server component used to order this reference in the output.
pub server_component: Option<ResolvedVc<NextServerComponentModule>>, pub server_component: Option<ResolvedVc<NextServerComponentModule>>,
pub ty: ClientReferenceType, pub ty: ClientReferenceType,
} }
......
use std::sync::LazyLock;
use turbo_rcstr::{RcStr, rcstr}; use turbo_rcstr::{RcStr, rcstr};
use turbo_tasks::{ResolvedVc, ValueToString, Vc}; use turbo_tasks::{ResolvedVc, ValueToString, Vc};
use turbopack_core::{ use turbopack_core::{
...@@ -24,8 +22,7 @@ impl NextServerUtilityModuleReference { ...@@ -24,8 +22,7 @@ impl NextServerUtilityModuleReference {
} }
} }
pub static NEXT_SERVER_UTILITY_MERGE_TAG: LazyLock<RcStr> = pub const NEXT_SERVER_UTILITY_MERGE_TAG: RcStr = rcstr!("next-server-utility");
LazyLock::new(|| rcstr!("next-server-utility"));
#[turbo_tasks::value_impl] #[turbo_tasks::value_impl]
impl ModuleReference for NextServerUtilityModuleReference { impl ModuleReference for NextServerUtilityModuleReference {
...@@ -37,7 +34,7 @@ impl ModuleReference for NextServerUtilityModuleReference { ...@@ -37,7 +34,7 @@ impl ModuleReference for NextServerUtilityModuleReference {
fn chunking_type(&self) -> Option<ChunkingType> { fn chunking_type(&self) -> Option<ChunkingType> {
Some(ChunkingType::Shared { Some(ChunkingType::Shared {
inherit_async: true, inherit_async: true,
merge_tag: Some(NEXT_SERVER_UTILITY_MERGE_TAG.clone()), merge_tag: Some(NEXT_SERVER_UTILITY_MERGE_TAG),
}) })
} }
......
...@@ -141,6 +141,7 @@ describe('layout-segment-chunk-sharing', () => { ...@@ -141,6 +141,7 @@ describe('layout-segment-chunk-sharing', () => {
) )
mcpPort = await findPort() mcpPort = await findPort()
let traceServerOutput = ''
traceServerProcess = spawn( traceServerProcess = spawn(
'node', 'node',
[ [
...@@ -152,7 +153,37 @@ describe('layout-segment-chunk-sharing', () => { ...@@ -152,7 +153,37 @@ describe('layout-segment-chunk-sharing', () => {
'--mcp-port', '--mcp-port',
String(mcpPort), String(mcpPort),
], ],
{ stdio: 'inherit' } { stdio: 'pipe' }
)
const recordTraceServerOutput = (chunk: Buffer) => {
const text = chunk.toString()
traceServerOutput += text
process.stdout.write(text)
}
traceServerProcess.stdout?.on('data', recordTraceServerOutput)
traceServerProcess.stderr?.on('data', recordTraceServerOutput)
// The MCP server can answer while the trace is still being read, so
// wait for the reader's end-of-file summary before querying spans.
// Large traces report `Initial read completed`; smaller ones report
// only their span count.
await retry(
async () => {
const output = traceServerOutput.replace(
// eslint-disable-next-line no-control-regex
/\x1b\[[0-9;]*[A-Za-z]/g,
''
)
if (
!/Initial read completed|(?:^|[\r\n])\d+ spans(?:[\r\n]|$)/.test(
output
)
) {
throw new Error('Trace file is still being read')
}
},
120_000,
500
) )
// Wait for the MCP HTTP server to be ready. // Wait for the MCP HTTP server to be ready.
...@@ -177,7 +208,7 @@ describe('layout-segment-chunk-sharing', () => { ...@@ -177,7 +208,7 @@ describe('layout-segment-chunk-sharing', () => {
30_000, 30_000,
500 500
) )
}, 120_000) }, 180_000)
afterAll(async () => { afterAll(async () => {
if (traceServerProcess?.pid) { if (traceServerProcess?.pid) {
...@@ -231,6 +262,30 @@ describe('layout-segment-chunk-sharing', () => { ...@@ -231,6 +262,30 @@ describe('layout-segment-chunk-sharing', () => {
1 1
) )
}) })
// The root layout's client references (e.g. the framework's layout
// router) belong to one isolated merged chunk group per page that renders
// the layout. Those groups are chunked together, so the root layout's
// client modules and its SSR modules are each chunked by a single task,
// instead of once per group (`client modules 0`, `client modules 1`, ...).
it('chunks all client reference groups of a segment in one task', async () => {
const spans = await findSpans(
'app/layout.tsx [app-rsc] (ecmascript, Next.js Server Component, '
)
const chunkingOf = (context: string, modifier: string) =>
spans.filter((name) =>
new RegExp(
`^turbopack_${context}::chunking_context chunking \\[project\\]/(?:.*/)?app/layout\\.tsx \\[app-rsc\\] \\(ecmascript, Next\\.js Server Component, ${modifier}(?: \\d+)?\\)$`
).test(name)
)
expect(chunkingOf('browser', 'client modules')).toEqual([
expect.stringMatching(/, client modules\)$/),
])
expect(chunkingOf('nodejs', 'ssr modules')).toEqual([
expect.stringMatching(/, ssr modules\)$/),
])
})
} }
) )
}) })
...@@ -313,16 +313,23 @@ async fn chunk_group_content_operation( ...@@ -313,16 +313,23 @@ async fn chunk_group_content_operation(
collecting_modules: FxIndexSet::default(), collecting_modules: FxIndexSet::default(),
}; };
let chunk_group_info = module_graph.chunk_group_info();
let chunk_group_indices = chunk_groups
.iter()
.map(async |chunk_group| Ok(*chunk_group_info.get_index_of(chunk_group.key()).await?))
.try_join()
.await?;
let chunk_group_info = chunk_group_info.await?;
let available_modules = match availability_info.available_modules() { let available_modules = match availability_info.available_modules() {
Some(available_modules) => Some(available_modules.snapshot().await?), Some(available_modules) => Some(available_modules.snapshot().await?),
None => None, None => None,
}; };
// The entries of all groups, in list order. An entry shared by several groups is traversed // The ordered entries of all groups, in list order. An entry shared by several groups is
// once. // traversed once.
let entries = chunk_groups let entries = chunk_group_indices
.iter() .iter()
.flat_map(|chunk_group| chunk_group.entries()) .flat_map(|&index| module_batches_graph.get_ordered_entries(&chunk_group_info, index))
.collect::<FxIndexSet<_>>() .collect::<FxIndexSet<_>>()
.into_iter() .into_iter()
.map(|entry| module_batches_graph.get_entry_index(entry)) .map(|entry| module_batches_graph.get_entry_index(entry))
......
...@@ -83,6 +83,18 @@ impl Hash for RoaringBitmapWrapper { ...@@ -83,6 +83,18 @@ impl Hash for RoaringBitmapWrapper {
#[turbo_tasks::value(transparent, cell = "keyed")] #[turbo_tasks::value(transparent, cell = "keyed")]
pub struct ModuleToChunkGroups(FxHashMap<ResolvedVc<Box<dyn Module>>, RoaringBitmapWrapper>); pub struct ModuleToChunkGroups(FxHashMap<ResolvedVc<Box<dyn Module>>, RoaringBitmapWrapper>);
#[turbo_tasks::value]
pub struct ChunkGroupWithIndex {
pub index: u32,
pub chunk_group: ChunkGroup,
}
#[turbo_tasks::value(transparent)]
pub struct ChunkGroups(Vec<ChunkGroupWithIndex>);
#[turbo_tasks::value(transparent)]
pub struct OptionChunkGroupWithIndex(Option<ChunkGroupWithIndex>);
#[turbo_tasks::value] #[turbo_tasks::value]
pub struct ChunkGroupInfo { pub struct ChunkGroupInfo {
pub module_chunk_groups: ResolvedVc<ModuleToChunkGroups>, pub module_chunk_groups: ResolvedVc<ModuleToChunkGroups>,
...@@ -143,6 +155,53 @@ impl ChunkGroupInfo { ...@@ -143,6 +155,53 @@ impl ChunkGroupInfo {
} }
} }
} }
/// Returns the exact chunk groups that contain `module`, as derived while traversing the module
/// graph.
#[turbo_tasks::function]
pub async fn get_chunk_groups_for_module(
&self,
module: ResolvedVc<Box<dyn Module>>,
) -> Result<Vc<ChunkGroups>> {
let module_chunk_groups = self.module_chunk_groups.await?;
let Some(groups) = module_chunk_groups.get(&module) else {
return Ok(Vc::cell(Vec::new()));
};
Ok(Vc::cell(
groups
.iter()
.map(|index| ChunkGroupWithIndex {
index,
chunk_group: self.chunk_groups[index as usize].clone(),
})
.collect(),
))
}
/// Returns the shared merged chunk group with `merge_tag` that `parent` is the parent of, if
/// the graph produced one.
///
/// A merge tag collects everything reachable through the tagged shared references of one
/// parent group into a single group, so there is at most one such group per parent and tag.
#[turbo_tasks::function]
pub fn get_shared_merged_chunk_group(
&self,
parent: usize,
merge_tag: RcStr,
) -> Vc<OptionChunkGroupWithIndex> {
let key = ChunkGroupKey::SharedMerged {
parent: ChunkGroupId::from(parent),
merge_tag,
};
Vc::cell(
self.chunk_group_keys
.get_index_of(&key)
.map(|index| ChunkGroupWithIndex {
index: index as u32,
chunk_group: self.chunk_groups[index].clone(),
}),
)
}
} }
/// Per-entry chunking heuristics. /// Per-entry chunking heuristics.
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment