Unverified Commit 4bfd5c71 authored by Hendrik Liebau's avatar Hendrik Liebau Committed by GitHub

Read chunk source maps from disk in the dev validation worker (#96218)

Node.js caches source maps per isolate, so the dev validation worker
only has maps for the chunk files loaded on its own thread. It never
renders server components, and the built entry reaches segment modules
through lazy getters, so it loads whatever `loadComponents` pulls in and
nothing else. A module the bundler splits into a chunk of its own, as
Turbopack does for a dynamic `import()`, is therefore missing from that
cache entirely, and the frames pointing into it are logged without a
source location. In-process validation resolves them, so this is a
regression from moving validation onto a worker.

On the main thread the same miss is covered by asking Turbopack for the
map through the `Project` handle, which cannot cross a thread boundary.
The worker instead reads the `.map` that Turbopack already wrote next to
the chunk, which needs no project handle and, unlike Node's cache, does
not depend on the chunk having been evaluated on this thread. Lookups
are restricted to `distDir` so a stack frame cannot point the reader at
an arbitrary file, and both hits and misses are memoised.

Installing that lookup also required moving
`bundlerFindSourceMapPayload` onto a `globalThis` symbol.
`patch-error-inspect` is bundled into several runtimes that each get
their own copy, and the copy that registers the implementation, here the
worker bundle, is not the copy that symbolicates the frame, which is the
app-page bundle's. The code frame renderer in the same file is already
shared this way for the same reason.

Webpack keeps its dev source maps in the compiler rather than writing
them next to the chunks, so there is nothing for the worker to read and
its frames are unchanged. A follow-up turns the worker off for Webpack
so that validation runs in process, where those frames still resolve.
parent c1617ded
...@@ -7,6 +7,9 @@ import type { ...@@ -7,6 +7,9 @@ import type {
import '../require-hook' import '../require-hook'
import '../node-environment' import '../node-environment'
import { isAbsolute, relative } from 'path'
import { readFileSync, realpathSync } from 'fs'
import { fileURLToPath } from 'url'
import { installBindings } from '../../build/swc/install-bindings' import { installBindings } from '../../build/swc/install-bindings'
import { installCodeFrameSupport } from '../lib/install-code-frame' import { installCodeFrameSupport } from '../lib/install-code-frame'
import { import {
...@@ -20,6 +23,81 @@ import { ...@@ -20,6 +23,81 @@ import {
getServerActionsManifest, getServerActionsManifest,
setManifestsSingleton, setManifestsSingleton,
} from '../app-render/manifests-singleton' } from '../app-render/manifests-singleton'
import { setBundlerFindSourceMapImplementation } from '../patch-error-inspect'
import type { ModernSourceMapPayload } from '../lib/source-maps'
/**
* Resolves a chunk's source map by reading the `.map` file the bundler emitted
* next to it, for chunks inside `distDir`.
*
* The main thread answers the same question through the Turbopack project
* handle, which cannot cross a thread boundary. Reading from disk is
* project-free and, more importantly, does not depend on the chunk having been
* evaluated in this thread: Node.js caches source maps per isolate, and the
* worker never renders server components, so it holds maps only for the chunks
* `loadComponents` pulled in. Frames arriving in the transported payload can
* point at any chunk the main render touched.
*
* This only helps Turbopack, which writes a `.map` beside every chunk. Webpack
* keeps its dev source maps in the compiler rather than on disk, so its frames
* from chunks this thread never evaluated stay unresolved, and its module URLs
* (`webpack-internal://…`) are declined below for the same reason. Frames from
* dependencies that are not bundled never reach this point, because
* `filterStackFrameDEV` drops `node_modules` and `node:` frames; bundled
* dependencies appear as chunks inside `distDir` like any other code.
*/
function createDiskSourceMapLookup(
distDir: string
): (sourceURL: string) => ModernSourceMapPayload | undefined {
// The frames carry resolved paths, so compare against the resolved `distDir`
// to keep the containment check meaningful when the project sits behind a
// symlink.
let canonicalDistDir = distDir
try {
canonicalDistDir = realpathSync(distDir)
} catch {}
const payloads = new Map<string, ModernSourceMapPayload | undefined>()
return function findSourceMapPayloadOnDisk(sourceURL) {
let chunkPath = sourceURL
if (chunkPath.startsWith('file://')) {
try {
chunkPath = fileURLToPath(chunkPath)
} catch {
return undefined
}
}
if (!isAbsolute(chunkPath)) {
// Not an emitted chunk, e.g. `webpack-internal://` or `<anonymous>`.
return undefined
}
const cached = payloads.get(chunkPath)
if (cached !== undefined || payloads.has(chunkPath)) {
return cached
}
let payload: ModernSourceMapPayload | undefined
const relativePath = relative(canonicalDistDir, chunkPath)
// Only chunks emitted into `distDir` have a source map to point at, and
// this keeps the lookup from reading arbitrary paths off disk.
if (!relativePath.startsWith('..') && !isAbsolute(relativePath)) {
try {
payload = JSON.parse(readFileSync(chunkPath + '.map', 'utf8'))
} catch {
payload = undefined
}
}
payloads.set(chunkPath, payload)
return payload
}
}
// Match the main dev server (`next-dev-server.ts`), which raises this so the // Match the main dev server (`next-dev-server.ts`), which raises this so the
// server captures deeper stacks. React's owner-stack capture during the // server captures deeper stacks. React's owner-stack capture during the
...@@ -173,6 +251,9 @@ export async function runDevValidation( ...@@ -173,6 +251,9 @@ export async function runDevValidation(
// worker bundle, the same way the unbundled build worker loads it. // worker bundle, the same way the unbundled build worker loads it.
await installBindings() await installBindings()
installCodeFrameSupport() installCodeFrameSupport()
setBundlerFindSourceMapImplementation(
createDiskSourceMapLookup(message.distDir)
)
setHttpClientAndAgentOptions({ setHttpClientAndAgentOptions({
httpAgentOptions: message.nextConfigSerializable.httpAgentOptions, httpAgentOptions: message.nextConfigSerializable.httpAgentOptions,
}) })
......
...@@ -22,12 +22,31 @@ type FindSourceMapPayload = ( ...@@ -22,12 +22,31 @@ type FindSourceMapPayload = (
// This is only a fallback for when Node.js fails to due to bugs e.g. https://github.com/nodejs/node/issues/52102 // This is only a fallback for when Node.js fails to due to bugs e.g. https://github.com/nodejs/node/issues/52102
// TODO: Remove once all supported Node.js versions are fixed. // TODO: Remove once all supported Node.js versions are fixed.
// TODO(veil): Set from Webpack as well // TODO(veil): Set from Webpack as well
let bundlerFindSourceMapPayload: FindSourceMapPayload = () => undefined //
// Stored on `globalThis` for the same reason as the code frame renderer below:
// this module is bundled into several runtimes that each get their own copy,
// and the copy that installs the implementation is not necessarily the one that
// symbolicates a frame. The dev validation worker depends on that, because it
// installs its implementation from the worker bundle while the frames are
// symbolicated by the app-page bundle's copy.
const BUNDLER_FIND_SOURCE_MAP = Symbol.for('next.dev.bundlerFindSourceMap')
type GlobalWithBundlerFindSourceMap = typeof globalThis & {
[BUNDLER_FIND_SOURCE_MAP]?: FindSourceMapPayload
}
export function setBundlerFindSourceMapImplementation( export function setBundlerFindSourceMapImplementation(
findSourceMapImplementation: FindSourceMapPayload findSourceMapImplementation: FindSourceMapPayload
): void { ): void {
bundlerFindSourceMapPayload = findSourceMapImplementation ;(globalThis as GlobalWithBundlerFindSourceMap)[BUNDLER_FIND_SOURCE_MAP] =
findSourceMapImplementation
}
function bundlerFindSourceMapPayload(
sourceURL: string
): ModernSourceMapPayload | undefined {
return (globalThis as GlobalWithBundlerFindSourceMap)[
BUNDLER_FIND_SOURCE_MAP
]?.(sourceURL)
} }
// Code frame renderer - injected by dev/build to avoid hard dependency on native bindings // Code frame renderer - injected by dev/build to avoid hard dependency on native bindings
......
...@@ -4,13 +4,13 @@ import { getDevCliValidationOutput } from 'e2e-utils/instant-validation' ...@@ -4,13 +4,13 @@ import { getDevCliValidationOutput } from 'e2e-utils/instant-validation'
// The dev validation worker symbolicates the frames it logs on its own thread, // The dev validation worker symbolicates the frames it logs on its own thread,
// where Node.js caches source maps only for the chunk files that thread loaded. // where Node.js caches source maps only for the chunk files that thread loaded.
// The worker never renders server components, and the built entry reaches // The worker never renders server components, and the built entry reaches
// segment modules through lazy getters, so it holds maps only for what // segment modules through lazy getters, so a module the bundler splits into a
// `loadComponents` pulled in. Maps are cached per chunk file rather than per // chunk of its own is never loaded there and has no map in that cache. The
// module, so code that stays in the route's own chunk still resolves, while a // worker resolves those by reading the `.map` the bundler emitted next to the
// module the bundler splits into a chunk of its own does not. Each route below // chunk, which only exists for Turbopack. Each route below owns its copy of the
// owns its copy of the module, because a module shared between routes lands in // module, because a module shared between routes lands in whichever chunk the
// whichever chunk the first compiled route put it in, which would make these // first compiled route put it in, which would make these snapshots depend on
// snapshots depend on the order the routes run in. // the order the routes run in.
describe('dev-validation-worker-source-maps', () => { describe('dev-validation-worker-source-maps', () => {
const { next, isTurbopack } = nextTestSetup({ const { next, isTurbopack } = nextTestSetup({
files: __dirname, files: __dirname,
...@@ -53,16 +53,10 @@ describe('dev-validation-worker-source-maps', () => { ...@@ -53,16 +53,10 @@ describe('dev-validation-worker-source-maps', () => {
() => next.cliOutput () => next.cliOutput
) )
// The module lands in a chunk of its own, which the worker never loads, so // The module lands in a chunk of its own, which the worker never loads,
// it cannot map the frame. In-process validation resolves it, so this is a // so it has no map for it in the worker thread's cache.
// regression from running validation on a worker.
if (isTurbopack) { if (isTurbopack) {
// Turbopack loses the location entirely. // Reading the `.map` next to the chunk resolves it anyway.
//
// TODO(veil): Give the worker a source map lookup that does not depend on
// the chunk having been evaluated on its thread, by reading the `.map`
// Turbopack writes next to the chunk, so that this frame reads like the
// statically imported one above.
expect(output).toMatchInlineSnapshot(` expect(output).toMatchInlineSnapshot(`
"Error: Route "/dynamic": Next.js encountered runtime data during prerendering. "Error: Route "/dynamic": Next.js encountered runtime data during prerendering.
...@@ -73,14 +67,15 @@ describe('dev-validation-worker-source-maps', () => { ...@@ -73,14 +67,15 @@ describe('dev-validation-worker-source-maps', () => {
- [block] Set \`export const instant = false\` to allow a blocking route - [block] Set \`export const instant = false\` to allow a blocking route
Learn more: https://nextjs.org/docs/messages/blocking-prerender-runtime Learn more: https://nextjs.org/docs/messages/blocking-prerender-runtime
at readCookie (<next-dist-dir>) at readCookie (app/dynamic/read-cookie.ts:4:30)
at DynamicPage (app/dynamic/page.tsx:4:31) at DynamicPage (app/dynamic/page.tsx:4:31)
2 | const { readCookie } = await import('./read-cookie') 2 |
3 | 3 | export async function readCookie(): Promise<string> {
> 4 | return <p id="value">{await readCookie()}</p> > 4 | const store = await cookies()
| ^ | ^
5 | } 5 | return store.get('probe')?.value ?? 'none'
6 |" 6 | }
7 |"
`) `)
} else { } else {
// Webpack keeps the module path but reports the position it has in the // Webpack keeps the module path but reports the position it has in the
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment