Unverified Commit c15cede5 authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

Turbopack: compute client reference manifest parts once per reference, not per page (#100443)

## Summary

When Turbopack builds an app's client reference manifests, it builds one
per app entry (page). For each client reference a page uses,
`build_manifest` recomputed:

- the reference's server path;
- its client, SSR and RSC module ids (each read goes through the
chunking context's id strategy, the module's ident and a keyed module-id
map read);
- for every JS chunk, its component chunks (the merged chunk's runtime
info plus each component chunk's path and file size).

None of these depend on the page, and pages usually share most of their
client references. So the number of reads grew as pages × references,
and with `experimental.turbopackChunking.generateComponentChunks` as
pages × references × chunks × component chunks. Every read was also
another dependency edge on each page's `build_manifest` task.

This PR moves that work into two turbo-tasks functions that are keyed
only on page-independent inputs:

- `client_reference_ids`: keyed on the client reference module, the
client chunking context and the optional SSR chunking context. It
returns the server path and the client, SSR and RSC module ids.
- `client_chunk_components`: keyed on the chunk and the client-relative
path. It returns the chunk's component chunk paths and sizes.

Each page's manifest now does one cached read per reference and one per
chunk. Values and insertion order are unchanged, so the generated
manifests are the same. After an edit, only the tasks whose inputs
changed rerun.

## Verification

- Compared every `*_client-reference-manifest.js` from full `next build`
runs with and without this change. All were byte-identical:
- a synthetic 300-page app with 40 client references per page, default
config: 302 files;
- a 1000-page app with 80 references per page and `turbopackChunking: {
generateComponentChunks: true, minComponentChunkSize: 0 }`: 1002 files,
including merged-chunk entries.
- `next dev`: after editing a client component, the edit shows up in the
SSR output and the page's client reference manifest still lists the
component.
- `cargo clippy -p next-core --all-targets -- -D warnings`, `cargo fmt
--check`
- Trace counts on the 1000-page app confirm the tasks are shared: 1002
manifests ran `client_reference_ids` 408 times and
`client_chunk_components` 402 times.
- Rough local timings (one 8-core Linux machine, non-LTO release
bindings, `next build --experimental-build-mode=compile`; exploratory,
not a validated benchmark):
- **Default config:** no measurable end-to-end difference; traced
`build_manifest` time drops about 15%.
- **Component chunks forced on (`minComponentChunkSize: 0`):** traced
`build_manifest` time drops from about 63 s to about 6–8 s. Build wall
time drops from about 53.8 s to about 48.5 s (median of 8 interleaved
runs).
- Not run locally: Edge runtime, persistent-cache rebuilds and the e2e
suites (left to CI).

<!-- NEXT_JS_LLM -->


<!-- fleet 27c4bd4e-c7b6-4b23-85a1-32cf20f0a948 -->

---------
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarejc3 <ejc3@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: 's avatarFleet Dev Agent <fleet-dev-agent@users.noreply.github.com>
parent 3a4477cb
...@@ -25,7 +25,9 @@ use turbopack_ecmascript::utils::StringifyJs; ...@@ -25,7 +25,9 @@ use turbopack_ecmascript::utils::StringifyJs;
use crate::{ use crate::{
mode::NextMode, mode::NextMode,
next_app::ClientReferencesChunks, next_app::ClientReferencesChunks,
next_client_reference::{ClientReferenceGraphResult, ClientReferenceType}, next_client_reference::{
ClientReferenceGraphResult, ClientReferenceType, EcmascriptClientReferenceModule,
},
next_config::NextConfig, next_config::NextConfig,
next_manifests::{ModuleId, encode_uri_component::encode_uri_component}, next_manifests::{ModuleId, encode_uri_component::encode_uri_component},
util::NextRuntime, util::NextRuntime,
...@@ -283,11 +285,15 @@ async fn build_manifest( ...@@ -283,11 +285,15 @@ async fn build_manifest(
let app_client_reference_ty = let app_client_reference_ty =
ClientReferenceType::EcmascriptClientReference(client_reference_module); ClientReferenceType::EcmascriptClientReference(client_reference_module);
let server_path = client_reference_module_ref.server_ident.to_string().await?; let ids = client_reference_ids(
*client_reference_module,
**client_chunking_context,
ssr_chunking_context.map(|chunking_context| *chunking_context),
)
.await?;
let server_path = &ids.server_path;
let client_module = client_reference_module_ref.client_module; let client_module = client_reference_module_ref.client_module;
let client_chunk_item_id = client_module let client_chunk_item_id = &ids.client_id;
.chunk_item_id(**client_chunking_context)
.await?;
let (client_chunks_paths, client_is_async) = if let Some(client_assets) = let (client_chunks_paths, client_is_async) = if let Some(client_assets) =
client_component_client_chunks.get(&app_client_reference_ty) client_component_client_chunks.get(&app_client_reference_ty)
...@@ -328,7 +334,9 @@ async fn build_manifest( ...@@ -328,7 +334,9 @@ async fn build_manifest(
// If this is a merged chunk, emit its component chunk paths alongside the // If this is a merged chunk, emit its component chunk paths alongside the
// URL so the browser runtime can split it during navigation. // URL so the browser runtime can split it during navigation.
let components = let components =
client_chunk_components(chunk, &client_relative_path).await?; client_chunk_components(*chunk, client_relative_path.clone())
.owned()
.await?;
Ok(if components.is_empty() { Ok(if components.is_empty() {
ClientChunk::Path(url) ClientChunk::Path(url)
} else { } else {
...@@ -346,13 +354,8 @@ async fn build_manifest( ...@@ -346,13 +354,8 @@ async fn build_manifest(
(Vec::new(), false) (Vec::new(), false)
}; };
if let Some(ssr_chunking_context) = *ssr_chunking_context { if let (Some(ssr_chunk_item_id), Some(rsc_chunk_item_id)) = (&ids.ssr_id, &ids.rsc_id) {
let ssr_module = client_reference_module_ref.ssr_module; let ssr_module = client_reference_module_ref.ssr_module;
let ssr_chunk_item_id = ssr_module.chunk_item_id(*ssr_chunking_context).await?;
let rsc_chunk_item_id = client_reference_module
.chunk_item_id(*ssr_chunking_context)
.await?;
let (ssr_chunks_paths, ssr_is_async) = if *runtime == NextRuntime::Edge { let (ssr_chunks_paths, ssr_is_async) = if *runtime == NextRuntime::Edge {
// the chunks get added to the middleware-manifest.json instead // the chunks get added to the middleware-manifest.json instead
...@@ -394,10 +397,10 @@ async fn build_manifest( ...@@ -394,10 +397,10 @@ async fn build_manifest(
}; };
entry_manifest.client_modules.module_exports.insert( entry_manifest.client_modules.module_exports.insert(
get_client_reference_module_key(&server_path, "*"), get_client_reference_module_key(server_path, "*"),
ManifestNodeEntry { ManifestNodeEntry {
name: rcstr!("*"), name: rcstr!("*"),
id: (&client_chunk_item_id).into(), id: client_chunk_item_id.into(),
chunks: client_chunks_paths, chunks: client_chunks_paths,
// This should of course be client_is_async, but SSR can become // This should of course be client_is_async, but SSR can become
// async due to ESM externals, and // async due to ESM externals, and
...@@ -412,7 +415,7 @@ async fn build_manifest( ...@@ -412,7 +415,7 @@ async fn build_manifest(
rcstr!("*"), rcstr!("*"),
ManifestNodeEntry { ManifestNodeEntry {
name: rcstr!("*"), name: rcstr!("*"),
id: (&ssr_chunk_item_id).into(), id: ssr_chunk_item_id.into(),
chunks: ssr_chunks_paths chunks: ssr_chunks_paths
.into_iter() .into_iter()
.map(ClientChunk::Path) .map(ClientChunk::Path)
...@@ -427,7 +430,7 @@ async fn build_manifest( ...@@ -427,7 +430,7 @@ async fn build_manifest(
rcstr!("*"), rcstr!("*"),
ManifestNodeEntry { ManifestNodeEntry {
name: rcstr!("*"), name: rcstr!("*"),
id: (&rsc_chunk_item_id).into(), id: rsc_chunk_item_id.into(),
chunks: vec![], chunks: vec![],
r#async: rsc_is_async, r#async: rsc_is_async,
}, },
...@@ -437,18 +440,18 @@ async fn build_manifest( ...@@ -437,18 +440,18 @@ async fn build_manifest(
NextRuntime::NodeJs => { NextRuntime::NodeJs => {
entry_manifest entry_manifest
.ssr_module_mapping .ssr_module_mapping
.insert((&client_chunk_item_id).into(), ssr_manifest_node); .insert(client_chunk_item_id.into(), ssr_manifest_node);
entry_manifest entry_manifest
.rsc_module_mapping .rsc_module_mapping
.insert((&client_chunk_item_id).into(), rsc_manifest_node); .insert(client_chunk_item_id.into(), rsc_manifest_node);
} }
NextRuntime::Edge => { NextRuntime::Edge => {
entry_manifest entry_manifest
.edge_ssr_module_mapping .edge_ssr_module_mapping
.insert((&client_chunk_item_id).into(), ssr_manifest_node); .insert(client_chunk_item_id.into(), ssr_manifest_node);
entry_manifest entry_manifest
.edge_rsc_module_mapping .edge_rsc_module_mapping
.insert((&client_chunk_item_id).into(), rsc_manifest_node); .insert(client_chunk_item_id.into(), rsc_manifest_node);
} }
} }
} }
...@@ -578,15 +581,21 @@ impl From<&TurbopackModuleId> for ModuleId { ...@@ -578,15 +581,21 @@ impl From<&TurbopackModuleId> for ModuleId {
} }
} }
/// The paths (relative to the client root) and sizes of a merged chunk's component chunks. A
/// task, so that it runs once per chunk instead of once per page and client reference using it.
#[turbo_tasks::value(transparent)]
struct ClientChunkComponents(Vec<(RcStr, u64)>);
#[turbo_tasks::function]
async fn client_chunk_components( async fn client_chunk_components(
chunk: ResolvedVc<Box<dyn OutputAsset>>, chunk: ResolvedVc<Box<dyn OutputAsset>>,
client_relative_path: &FileSystemPath, client_relative_path: FileSystemPath,
) -> Result<Vec<(RcStr, u64)>> { ) -> Result<Vc<ClientChunkComponents>> {
let Some(output_chunk) = ResolvedVc::try_sidecast::<Box<dyn OutputChunk>>(chunk) else { let Some(output_chunk) = ResolvedVc::try_sidecast::<Box<dyn OutputChunk>>(chunk) else {
return Ok(Vec::new()); return Ok(Vc::cell(Vec::new()));
}; };
let Some(component_chunks) = output_chunk.runtime_info().await?.module_chunks else { let Some(component_chunks) = output_chunk.runtime_info().await?.module_chunks else {
return Ok(Vec::new()); return Ok(Vc::cell(Vec::new()));
}; };
let component_assets = component_chunks.await?; let component_assets = component_chunks.await?;
let mut components = Vec::with_capacity(component_assets.len()); let mut components = Vec::with_capacity(component_assets.len());
...@@ -604,7 +613,58 @@ async fn client_chunk_components( ...@@ -604,7 +613,58 @@ async fn client_chunk_components(
components.push((RcStr::from(rel), size)); components.push((RcStr::from(rel), size));
} }
} }
Ok(components) Ok(Vc::cell(components))
}
/// The parts of a client reference's manifest entries that do not depend on the page: its
/// server path and its module ids for the client, SSR and RSC sides. A task, so that they are
/// computed once per client reference instead of once per page that uses it.
#[turbo_tasks::value(shared)]
struct ClientReferenceIds {
server_path: RcStr,
client_id: TurbopackModuleId,
/// Set when there is an SSR chunking context.
ssr_id: Option<TurbopackModuleId>,
/// Set when there is an SSR chunking context.
rsc_id: Option<TurbopackModuleId>,
}
#[turbo_tasks::function]
async fn client_reference_ids(
client_reference_module: ResolvedVc<EcmascriptClientReferenceModule>,
client_chunking_context: ResolvedVc<Box<dyn ChunkingContext>>,
ssr_chunking_context: Option<ResolvedVc<Box<dyn ChunkingContext>>>,
) -> Result<Vc<ClientReferenceIds>> {
let module = client_reference_module.await?;
let server_path = module.server_ident.to_string().owned().await?;
let client_id = module
.client_module
.chunk_item_id(*client_chunking_context)
.await?;
let (ssr_id, rsc_id) = if let Some(ssr_chunking_context) = ssr_chunking_context {
(
Some(
module
.ssr_module
.chunk_item_id(*ssr_chunking_context)
.await?,
),
Some(
client_reference_module
.chunk_item_id(*ssr_chunking_context)
.await?,
),
)
} else {
(None, None)
};
Ok(ClientReferenceIds {
server_path,
client_id,
ssr_id,
rsc_id,
}
.cell())
} }
/// See next.js/packages/next/src/lib/client-reference.ts /// See next.js/packages/next/src/lib/client-reference.ts
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment