Unverified Commit 9458fb99 authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

[lts-active] Turbopack: backport Pages Router dev module graph chaining (#98823,…

[lts-active] Turbopack: backport Pages Router dev module graph chaining (#98823, #99910, #100091) (#100008)

## Summary

Backports these canary commits to `releases/lts/active`:

| PR | Commit | Change |
| --- | --- | --- |
| #98823 | `6dd4b27b` | Chain the Pages
Router dev client module graphs to fix `_app` availability (#99789). |
| #99910 | `1e1f9e69` | Add Pages Router
availability coverage and restore the import-order assertion. |
| #100091 | `5eec1d9d` | Fix `_app`
`next/dynamic` preload 404s introduced by #98823, and preload those
imports in production too. |

The third commit must ship with #98823: chaining the page and `_app`
graphs made pages preload `_app`'s dynamic imports with the **page's**
availability, unlike `_app`'s own async loader. #100091 chunks those
imports with `_app`'s own graph and availability, then merges them with
the page's imports. The generic collector and App Router are untouched.
The regression was reproduced with
[dcheng1290/next-loadable-404](https://github.com/dcheng1290/next-loadable-404).

All three backports retain cherry-pick provenance. #100091 had an
import-only conflict in `pages.rs` because LTS lacks an unrelated
`smallvec` import; the resolution adds only `FxHashSet`. The
added/removed lines and test files match upstream. Existing branch
history is preserved; the merged GTM test fix (#100011) remains
included.

Fixes #99789 on the LTS line.

## Verification

With a native binary rebuilt from this LTS backport branch:

- `pages-app-next-dynamic-preload`: **5/5 pass** in Turbopack dev and
build/start. The suite checks every referenced static asset returns 200,
the dynamic component renders, and both pages preload the same non-empty
chunk set.
- Before applying #100091, the same suite fails **2/5** in Turbopack dev
on this branch: a non-200 asset and the preload assertion.
- `pages-app-availability-boundary` and `app-document-import-order`: **3
pass, 1 existing skip** in Turbopack dev. The upstream gated
deduplication assertion remains unchanged.
- `pnpm build-all --env-mode=loose`, `cargo check -p next-api`, `cargo
fmt -p next-api --check`, scoped Prettier and ESLint pass. The initial
full bootstrap ran out of sandbox disk during native compilation; after
clearing disposable artifacts and passing the Cargo environment through
Turbo, the full build passed.

Earlier verification, before adding #100091, also covered the #99910
suites in Turbopack start and webpack dev. The new LTS backport has not
been tested locally in deploy mode, on other platforms, or with full
workspace type-checking; CI will provide that coverage.

<!-- NEXT_JS_LLM -->

<!-- fleet 652922cb-dc34-493d-9be5-c241a5164cad -->

---------
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarNiklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
parent eb85307e
...@@ -29,9 +29,12 @@ use next_core::{ ...@@ -29,9 +29,12 @@ use next_core::{
segment_config::ParseSegmentMode, segment_config::ParseSegmentMode,
util::{NextRuntime, get_asset_prefix_from_pathname, pages_function_name}, util::{NextRuntime, get_asset_prefix_from_pathname, pages_function_name},
}; };
use rustc_hash::FxHashSet;
use tracing::Instrument; use tracing::Instrument;
use turbo_rcstr::{RcStr, rcstr}; use turbo_rcstr::{RcStr, rcstr};
use turbo_tasks::{Completion, FxIndexMap, ResolvedVc, ValueToString, Vc, fxindexmap, fxindexset}; use turbo_tasks::{
Completion, FxIndexMap, ReadRef, ResolvedVc, ValueToString, Vc, fxindexmap, fxindexset,
};
use turbo_tasks_fs::{ use turbo_tasks_fs::{
self, File, FileContent, FileSystem, FileSystemPath, FileSystemPathOption, VirtualFileSystem, self, File, FileContent, FileSystem, FileSystemPath, FileSystemPathOption, VirtualFileSystem,
}; };
...@@ -71,7 +74,9 @@ use crate::{ ...@@ -71,7 +74,9 @@ use crate::{
}, },
font::FontManifest, font::FontManifest,
loadable_manifest::create_react_loadable_manifest, loadable_manifest::create_react_loadable_manifest,
module_graph::{NextDynamicGraphs, validate_pages_css_imports}, module_graph::{
DynamicImportEntriesWithImporter, NextDynamicGraphs, validate_pages_css_imports,
},
nft::{EndpointTraceResult, trace_endpoint}, nft::{EndpointTraceResult, trace_endpoint},
nft_json::NftJsonAsset, nft_json::NftJsonAsset,
paths::{ paths::{
...@@ -641,6 +646,87 @@ enum EmitManifests { ...@@ -641,6 +646,87 @@ enum EmitManifests {
Full, Full,
} }
/// The chunk group entry modules of a set of client evaluatable assets.
async fn client_entry_modules(
evaluatable_assets: Vc<EvaluatableAssets>,
) -> Result<Vec<ResolvedVc<Box<dyn Module>>>> {
Ok(evaluatable_assets
.await?
.iter()
.map(|asset| ResolvedVc::upcast(*asset))
.collect())
}
/// The chunks to preload for the `next/dynamic` imports of a page's client graph, including the
/// ones of `/_app` (which is rendered as part of every page).
async fn page_next_dynamic_chunks(
endpoint: Vc<PageEndpoint>,
next_dynamic_imports: ReadRef<DynamicImportEntriesWithImporter>,
client_availability_info: AvailabilityInfo,
) -> Result<ResolvedVc<DynamicImportedChunks>> {
let this = endpoint.await?;
let project = this.pages_project.project();
let client_chunking_context = project.client_chunking_context();
if this.pathname == "/_app" {
return collect_next_dynamic_chunks(
endpoint.client_module_graph(),
client_chunking_context,
next_dynamic_imports,
NextDynamicChunkAvailability::AvailabilityInfo(client_availability_info),
)
.await;
}
// `/_app`'s `next/dynamic` imports are loaded by `/_app`'s async loaders, which are chunked
// with `/_app`'s graph and availability, so preload them with the chunks `/_app` itself
// lists. Chunked with the page's availability, they would name chunks that don't match what
// those loaders request (and that may not be emitted at all).
// With a per-page module graph, a page's client graph is chained after `/_app`'s (see
// `client_module_graph`), so `next_dynamic_imports` also contains `/_app`'s imports: chunk
// only the page's own imports with the page's graph and availability. With a whole-app graph
// they aren't reachable from the page, so `/_app`'s imports are only added here.
let per_page_module_graph = *project.per_page_module_graph().await?;
let app_endpoint = this.pages_project.app_page_endpoint();
let app_module_graph = app_endpoint.client_module_graph();
let app_next_dynamic_imports = NextDynamicGraphs::new(app_module_graph, per_page_module_graph)
.get_next_dynamic_imports_for_endpoint(app_endpoint.client_module())
.await?;
let app_dynamic_entries = app_next_dynamic_imports
.iter()
.map(|(dynamic_entry, _)| *dynamic_entry)
.collect::<FxHashSet<_>>();
let page_next_dynamic_imports = Vc::<DynamicImportEntriesWithImporter>::cell(
next_dynamic_imports
.iter()
.filter(|(dynamic_entry, _)| !app_dynamic_entries.contains(dynamic_entry))
.cloned()
.collect(),
)
.await?;
let page_chunks = collect_next_dynamic_chunks(
endpoint.client_module_graph(),
client_chunking_context,
page_next_dynamic_imports,
NextDynamicChunkAvailability::AvailabilityInfo(client_availability_info),
)
.await?;
let app_chunks = collect_next_dynamic_chunks(
app_module_graph,
client_chunking_context,
app_next_dynamic_imports,
NextDynamicChunkAvailability::AvailabilityInfo(
app_endpoint.client_chunk_group().await?.availability_info,
),
)
.await?;
let mut chunks = (*app_chunks.await?).clone();
chunks.extend(page_chunks.await?.iter().map(|(k, v)| (*k, *v)));
Ok(ResolvedVc::cell(chunks))
}
#[turbo_tasks::value_impl] #[turbo_tasks::value_impl]
impl PageEndpoint { impl PageEndpoint {
#[turbo_tasks::function] #[turbo_tasks::function]
...@@ -729,8 +815,54 @@ impl PageEndpoint { ...@@ -729,8 +815,54 @@ impl PageEndpoint {
async fn client_module_graph(self: Vc<Self>) -> Result<Vc<ModuleGraph>> { async fn client_module_graph(self: Vc<Self>) -> Result<Vc<ModuleGraph>> {
let this = self.await?; let this = self.await?;
let project = this.pages_project.project(); let project = this.pages_project.project();
let evaluatable_assets = self.client_evaluatable_assets();
Ok(project.module_graph_for_modules(evaluatable_assets)) if !*project.per_page_module_graph().await? {
return Ok(project.module_graph_for_modules(self.client_evaluatable_assets()));
}
// With a per-page module graph every endpoint would otherwise get its own graph, and
// chunk group indices are only meaningful within the `ChunkGroupInfo` of a single graph.
// `/_app` is always loaded before the page in the browser and seeds the page's
// availability, so both entry groups have to live in the same graph. Build a graph
// "chain" for app, page -- the same layout segment optimization `ssr_module_graph` uses
// for document, app, page.
let should_trace = *project.should_write_nft_manifests().await?;
let should_read_binding_usage = project.next_mode().await?.is_production();
let mut graphs = vec![];
let mut visited_modules = VisitedModules::empty();
if this.pathname != "/_app" {
let graph = SingleModuleGraph::new_with_entries_visited_intern(
GraphEntries::from_chunk_groups(vec![ChunkGroupEntry::Entry {
modules: client_entry_modules(
this.pages_project
.app_page_endpoint()
.client_evaluatable_assets(),
)
.await?,
heuristics: EntryHeuristics::default(),
}]),
visited_modules,
should_trace,
should_read_binding_usage,
);
graphs.push(graph);
visited_modules = VisitedModules::concatenate(visited_modules, graph);
}
let graph = SingleModuleGraph::new_with_entries_visited_intern(
GraphEntries::from_chunk_groups(vec![ChunkGroupEntry::Entry {
modules: client_entry_modules(self.client_evaluatable_assets()).await?,
heuristics: EntryHeuristics::default(),
}]),
visited_modules,
should_trace,
should_read_binding_usage,
);
graphs.push(graph);
Ok(ModuleGraph::from_graphs(graphs, None).connect())
} }
#[turbo_tasks::function] #[turbo_tasks::function]
...@@ -806,12 +938,7 @@ impl PageEndpoint { ...@@ -806,12 +938,7 @@ impl PageEndpoint {
let module_graph = self.client_module_graph(); let module_graph = self.client_module_graph();
let evaluatable_assets = self let evaluatable_assets = client_entry_modules(self.client_evaluatable_assets()).await?;
.client_evaluatable_assets()
.await?
.iter()
.map(|m| ResolvedVc::upcast(*m))
.collect();
// Like App Router layouts, `/_app` is always loaded before the page. Chunk it first so // Like App Router layouts, `/_app` is always loaded before the page. Chunk it first so
// the page's chunks don't include modules that the browser already downloaded with // the page's chunks don't include modules that the browser already downloaded with
// `/_app`. // `/_app`.
...@@ -1002,13 +1129,8 @@ impl PageEndpoint { ...@@ -1002,13 +1129,8 @@ impl PageEndpoint {
client_availability_info, client_availability_info,
)) = next_dynamic_imports )) = next_dynamic_imports
{ {
collect_next_dynamic_chunks( page_next_dynamic_chunks(self, next_dynamic_imports, client_availability_info)
self.client_module_graph(), .await?
project.client_chunking_context(),
next_dynamic_imports,
NextDynamicChunkAvailability::AvailabilityInfo(client_availability_info),
)
.await?
} else { } else {
DynamicImportedChunks::default().resolved_cell() DynamicImportedChunks::default().resolved_cell()
}; };
......
...@@ -2,7 +2,7 @@ ...@@ -2,7 +2,7 @@
import { nextTestSetup } from 'e2e-utils' import { nextTestSetup } from 'e2e-utils'
describe('Root components import order', () => { describe('Root components import order', () => {
const { next, isTurbopack, isNextDev } = nextTestSetup({ const { next, isTurbopack } = nextTestSetup({
files: __dirname, files: __dirname,
}) })
...@@ -17,38 +17,35 @@ describe('Root components import order', () => { ...@@ -17,38 +17,35 @@ describe('Root components import order', () => {
}) })
}) })
// Only asserted for production builds: in development each entry is chunked from its own // Turbopack dev chunks `_app` from its own module graph, so its batches (and therefore its
// per-page module graph, which can still merge a shared module into per-entry units. // availability info) don't match those in the page's graph, and the shared module is emitted
;(isNextDev ? it.skip : it)( // for both entries.
'loads modules shared by _app and the page only once', // @gate !(turbopack && dev)
async () => { it('loads modules shared by _app and the page only once', async () => {
const requests: Set<string> = new Set() const requests: Set<string> = new Set()
await next.browser('/', { await next.browser('/', {
beforePageLoad(page) { beforePageLoad(page) {
page.on('request', (request) => { page.on('request', (request) => {
const url = new URL(request.url(), next.url) const url = new URL(request.url(), next.url)
if ( if (
url.pathname.startsWith('/_next/static/') && url.pathname.startsWith('/_next/static/') &&
url.pathname.endsWith('.js') url.pathname.endsWith('.js')
) { ) {
requests.add(url.href) requests.add(url.href)
} }
}) })
}, },
}) })
const chunks = await Promise.all( const chunks = await Promise.all(
[...requests].map((url) => [...requests].map((url) => fetch(url).then((response) => response.text()))
fetch(url).then((response) => response.text()) )
) const matchingChunks = chunks.filter((chunk) =>
) chunk.includes('APP_PAGE_SHARED_MODULE_MARKER')
const matchingChunks = chunks.filter((chunk) => )
chunk.includes('APP_PAGE_SHARED_MODULE_MARKER')
)
expect(matchingChunks.length).toBe(1) expect(matchingChunks.length).toBe(1)
} })
)
// Test relies on webpack splitChunks overrides. // Test relies on webpack splitChunks overrides.
;(isTurbopack ? it.skip : it)( ;(isTurbopack ? it.skip : it)(
......
import c from './c-shared'
export default c
import b from './b-boundary'
import d from './d-boundary'
export default b + d
import { nextTestSetup } from 'e2e-utils'
import { retry } from 'next-test-utils'
// Regression test for https://github.com/vercel/next.js/issues/99789: the page's client chunks
// reuse the modules `_app` already loaded. Modules imported only by the page must still be
// chunked, even when they are batched next to a module shared with `_app`.
describe('pages-app-availability-boundary', () => {
const { next } = nextTestSetup({
files: __dirname,
})
it('hydrates a page whose imports surround a module shared with _app', async () => {
const browser = await next.browser('/')
await retry(async () => {
expect(await browser.elementByCss('#hydrated').text()).toBe('yes')
})
expect(await browser.elementByCss('#values').text()).toBe(
'a-page-only,b-boundary,c-shared,d-boundary,e-page'
)
const logs = await browser.log()
expect(
logs.filter(({ message }) =>
message.includes('module factory is not available')
)
).toEqual([])
})
})
import type { AppProps } from 'next/app'
// Also imported by `pages/index.tsx`, so it is available to the page once `_app` has loaded.
import '../lib/c-shared'
import('../lib/app-dyn')
export default function App({ Component, pageProps }: AppProps) {
return <Component {...pageProps} />
}
import { useEffect, useState } from 'react'
// The import order matters (https://github.com/vercel/next.js/issues/99789): the `_app`-shared
// module `c-shared` sits between the boundary modules `b-boundary` and `d-boundary` (also
// imported by the `page-dyn` async chunk), so the page entry's modules get split around it.
import a from '../lib/a-page-only'
import b from '../lib/b-boundary'
import c from '../lib/c-shared'
import d from '../lib/d-boundary'
import e from '../lib/e-page'
import('../lib/page-dyn')
export default function Page() {
const [hydrated, setHydrated] = useState(false)
useEffect(() => {
setHydrated(true)
}, [])
return (
<>
<p id="values">{[a, b, c, d, e].join(',')}</p>
<p id="hydrated">{hydrated ? 'yes' : 'no'}</p>
</>
)
}
export const appShared = (s: string) => `app-shared:${s}`
export const lazyOnly = (s: string) => `lazy-only-marker:${s}`
import { appShared } from './app-shared'
import { lazyOnly } from './lazy-only'
import { pageShared } from './page-shared'
export function Lazy() {
return (
<p id="lazy">
{[appShared('lazy'), pageShared('lazy'), lazyOnly('lazy')].join(',')}
</p>
)
}
export const pageShared = (s: string) => `page-shared:${s}`
import { nextTestSetup } from 'e2e-utils'
import { retry } from 'next-test-utils'
// `_app` renders a `next/dynamic` component that shares a module with `/` (but not with `/other`).
// The component is loaded by `_app`'s own async loader on every page, so every page has to preload
// the same chunks, the ones that loader requests, regardless of what the page already loaded.
describe('pages-app-next-dynamic-preload', () => {
const { next } = nextTestSetup({
files: __dirname,
})
const pages = ['/', '/other']
// The `/_next/static` scripts and stylesheets (including preloads) in the SSR HTML, with their
// query strings (e.g. the deployment id).
async function getReferencedAssets(pathname: string) {
const $ = await next.render$(pathname)
const urls = $('script[src], link[href]')
.toArray()
.map((el) => $(el).attr('src') ?? $(el).attr('href') ?? '')
.filter((url) => /^\/_next\/static\/.+\.(?:js|css)(?:\?|$)/.test(url))
return [...new Set(urls)]
}
it.each(pages)(
'serves every asset referenced in the HTML of %s',
async (pathname) => {
const assets = await getReferencedAssets(pathname)
expect(assets.length).toBeGreaterThan(0)
const failed: string[] = []
for (const asset of assets) {
const res = await next.fetch(asset)
if (res.status !== 200) failed.push(`${res.status} ${asset}`)
}
expect(failed).toEqual([])
}
)
it.each(pages)(
'renders the _app next/dynamic component on %s',
async (pathname) => {
const browser = await next.browser(pathname)
await retry(async () => {
expect(await browser.elementByCss('#lazy').text()).toBe(
'app-shared:lazy,page-shared:lazy,lazy-only-marker:lazy'
)
})
expect(await browser.elementByCss('#app').text()).toBe('app-shared:app')
}
)
it('preloads the same _app next/dynamic chunks on every page', async () => {
const preloadedChunks: Record<string, string[]> = {}
for (const pathname of pages) {
const chunks: string[] = []
for (const asset of await getReferencedAssets(pathname)) {
if (!/\.js(?:\?|$)/.test(asset)) continue
const res = await next.fetch(asset)
expect({ asset, status: res.status }).toEqual({ asset, status: 200 })
if ((await res.text()).includes('lazy-only-marker')) {
chunks.push(asset.split('?')[0])
}
}
preloadedChunks[pathname] = chunks.sort()
}
expect(preloadedChunks['/']).not.toEqual([])
expect(preloadedChunks['/other']).toEqual(preloadedChunks['/'])
})
})
import type { AppProps } from 'next/app'
import dynamic from 'next/dynamic'
import { appShared } from '../components/app-shared'
const Lazy = dynamic(() => import('../components/lazy').then((m) => m.Lazy))
export default function App({ Component, pageProps }: AppProps) {
return (
<>
<p id="app">{appShared('app')}</p>
<Lazy />
<Component {...pageProps} />
</>
)
}
// Shared with `components/lazy`, which `_app` loads with `next/dynamic`: this page has already
// loaded it when `_app`'s dynamic import runs, `/other` hasn't.
import { pageShared } from '../components/page-shared'
export default function Page() {
return <p id="page">{pageShared('index')}</p>
}
export default function Page() {
return <p id="page">other</p>
}
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment