Unverified Commit e88e6163 authored by vercel-fleet-prod[bot]'s avatar vercel-fleet-prod[bot] Committed by GitHub

Lazily load prerender source maps

Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent caa60a93
...@@ -125,8 +125,8 @@ Generating source maps consumes extra memory during the build process. ...@@ -125,8 +125,8 @@ Generating source maps consumes extra memory during the build process.
You can disable source map generation by adding `productionBrowserSourceMaps: false` and `experimental.serverSourceMaps: false` to your Next.js configuration. You can disable source map generation by adding `productionBrowserSourceMaps: false` and `experimental.serverSourceMaps: false` to your Next.js configuration.
Next.js will use source maps by default during the prerender phase of `next build`. Next.js will use source maps by default for errors during the prerender phase of `next build`, loading them when an error is first logged.
If you consistently encounter memory issues during that phase (after "Generating static pages"), If you consistently encounter memory issues during that phase (after "Generating static pages") while many errors are logged,
you can try disabling source maps in that phase by adding `enablePrerenderSourceMaps: false` to your Next.js configuration. you can try disabling source maps in that phase by adding `enablePrerenderSourceMaps: false` to your Next.js configuration.
> **Good to know**: Some plugins may turn on source maps and may require custom configuration to disable. > **Good to know**: Some plugins may turn on source maps and may require custom configuration to disable.
......
...@@ -7,7 +7,9 @@ related: ...@@ -7,7 +7,9 @@ related:
- app/api-reference/config/next-config-js/productionBrowserSourceMaps - app/api-reference/config/next-config-js/productionBrowserSourceMaps
--- ---
The `enablePrerenderSourceMaps` option lets Node.js use existing source maps during the prerender phase of `next build`. When source maps are available, error stack traces point to the original source code. It defaults to `true`. The `enablePrerenderSourceMaps` option lets Next.js use existing source maps for errors during the prerender phase of `next build`. When source maps are available, error stack traces point to the original source code. It defaults to `true`.
Source maps are read when an error is first logged, so builds without errors don't pay for loading them.
This option does not generate source maps. Webpack does not generate server source maps by default. To generate them when investigating prerender errors, use [`next build --debug-prerender`](/docs/app/api-reference/cli/next#debugging-prerender-errors). This option does not generate source maps. Webpack does not generate server source maps by default. To generate them when investigating prerender errors, use [`next build --debug-prerender`](/docs/app/api-reference/cli/next#debugging-prerender-errors).
...@@ -38,7 +40,7 @@ module.exports = nextConfig ...@@ -38,7 +40,7 @@ module.exports = nextConfig
| --------------------------- | --------- | ------- | | --------------------------- | --------- | ------- |
| `enablePrerenderSourceMaps` | `boolean` | `true` | | `enablePrerenderSourceMaps` | `boolean` | `true` |
Disabling source maps during prerendering can reduce [build memory usage](/docs/app/guides/memory-usage#disable-source-maps), but makes prerendering errors harder to debug. Disabling source maps during prerendering can reduce [build memory usage](/docs/app/guides/memory-usage#disable-source-maps) when many errors are logged, but makes prerendering errors harder to debug.
## Version History ## Version History
......
...@@ -1036,7 +1036,6 @@ export function createStaticWorker( ...@@ -1036,7 +1036,6 @@ export function createStaticWorker(
progress?.clear() progress?.clear()
}, },
debuggerPortOffset, debuggerPortOffset,
enableSourceMaps: config.enablePrerenderSourceMaps,
// remove --max-old-space-size flag as it can cause memory issues. // remove --max-old-space-size flag as it can cause memory issues.
isolatedMemory: true, isolatedMemory: true,
enableWorkerThreads: config.experimental.workerThreads, enableWorkerThreads: config.experimental.workerThreads,
...@@ -1050,6 +1049,10 @@ export function createStaticWorker( ...@@ -1050,6 +1049,10 @@ export function createStaticWorker(
__NEXT_PRIVATE_CPU_PROFILE: 'build-static-worker', __NEXT_PRIVATE_CPU_PROFILE: 'build-static-worker',
} }
: undefined), : undefined),
// worker.ts reads source maps lazily if set
__NEXT_PRERENDER_SOURCE_MAPS: config.enablePrerenderSourceMaps
? '1'
: '',
// worker.ts copies this value into globalThis.NEXT_CLIENT_ASSET_SUFFIX // worker.ts copies this value into globalThis.NEXT_CLIENT_ASSET_SUFFIX
__NEXT_PRERENDER_CLIENT_ASSET_SUFFIX: __NEXT_PRERENDER_CLIENT_ASSET_SUFFIX:
config.supportsImmutableAssets || !config.deploymentId config.supportsImmutableAssets || !config.deploymentId
......
...@@ -5,11 +5,20 @@ import '../server/lib/cpu-profile' ...@@ -5,11 +5,20 @@ import '../server/lib/cpu-profile'
import { installBindings } from './swc/install-bindings' import { installBindings } from './swc/install-bindings'
import { installCodeFrameSupport } from '../server/lib/install-code-frame' import { installCodeFrameSupport } from '../server/lib/install-code-frame'
import { isPageStatic as isPageStaticImpl } from './utils' import { isPageStatic as isPageStaticImpl } from './utils'
import { enableLazySourceMaps } from '../server/lib/source-maps'
// Set the global asset suffix for Turbopack compiled code to use during prerendering // Set the global asset suffix for Turbopack compiled code to use during prerendering
;(globalThis as any).NEXT_CLIENT_ASSET_SUFFIX = ;(globalThis as any).NEXT_CLIENT_ASSET_SUFFIX =
process.env.__NEXT_PRERENDER_CLIENT_ASSET_SUFFIX process.env.__NEXT_PRERENDER_CLIENT_ASSET_SUFFIX
// `enablePrerenderSourceMaps`: source-map errors logged while collecting page
// data and prerendering. The source maps are read when an error is first
// formatted, rather than by Node.js for every loaded module via
// `--enable-source-maps`, which slows down prerendering many pages.
if (process.env.__NEXT_PRERENDER_SOURCE_MAPS === '1') {
enableLazySourceMaps()
}
export { getDefinedNamedExports, hasCustomGetInitialProps } from './utils' export { getDefinedNamedExports, hasCustomGetInitialProps } from './utils'
// Install native bindings and code-frame support before collecting page data so // Install native bindings and code-frame support before collecting page data so
......
...@@ -47,7 +47,6 @@ export class Worker { ...@@ -47,7 +47,6 @@ export class Worker {
* `-1` if not inspectable * `-1` if not inspectable
*/ */
debuggerPortOffset: number debuggerPortOffset: number
enableSourceMaps?: boolean
/** /**
* True if `--max-old-space-size` should not be forwarded to the worker. * True if `--max-old-space-size` should not be forwarded to the worker.
*/ */
...@@ -62,7 +61,6 @@ export class Worker { ...@@ -62,7 +61,6 @@ export class Worker {
} }
) { ) {
let { let {
enableSourceMaps,
timeout, timeout,
onRestart, onRestart,
logger = console, logger = console,
...@@ -110,10 +108,6 @@ export class Worker { ...@@ -110,10 +108,6 @@ export class Worker {
} }
} }
if (enableSourceMaps) {
nodeOptions['enable-source-maps'] = true
}
if (isolatedMemory) { if (isolatedMemory) {
delete nodeOptions['max-old-space-size'] delete nodeOptions['max-old-space-size']
delete nodeOptions['max_old_space_size'] delete nodeOptions['max_old_space_size']
......
...@@ -11,6 +11,16 @@ const findSourceMap = ...@@ -11,6 +11,16 @@ const findSourceMap =
? noSourceMap ? noSourceMap
: (require('module') as typeof import('module')).findSourceMap : (require('module') as typeof import('module')).findSourceMap
// Edge runtime has no file system, so lazy source maps are Node.js only.
const nodeModules =
process.env.NEXT_RUNTIME === 'edge'
? undefined
: {
fs: require('fs') as typeof import('fs'),
path: require('path') as typeof import('path'),
url: require('url') as typeof import('url'),
}
/** /**
* https://tc39.es/source-map/#index-map * https://tc39.es/source-map/#index-map
*/ */
...@@ -56,13 +66,16 @@ const sourceMapPayloads = new WeakMap<SourceMap, ModernSourceMapPayload>() ...@@ -56,13 +66,16 @@ const sourceMapPayloads = new WeakMap<SourceMap, ModernSourceMapPayload>()
* Like `module.findSourceMap`, but returns the source map's payload without * Like `module.findSourceMap`, but returns the source map's payload without
* cloning it on every call. Callers must not mutate the returned payload. * cloning it on every call. Callers must not mutate the returned payload.
* Throws like `module.findSourceMap` does on invalid source maps. * Throws like `module.findSourceMap` does on invalid source maps.
*
* Falls back to reading the source map from disk when lazy source maps are
* enabled (see `enableLazySourceMaps`) and Node.js has none for `sourceURL`.
*/ */
export function findSourceMapPayload( export function findSourceMapPayload(
sourceURL: string sourceURL: string
): ModernSourceMapPayload | undefined { ): ModernSourceMapPayload | undefined {
const sourceMap = findSourceMap(sourceURL) const sourceMap = findSourceMap(sourceURL)
if (sourceMap === undefined) { if (sourceMap === undefined) {
return undefined return findLazySourceMapPayload(sourceURL)
} }
let payload = sourceMapPayloads.get(sourceMap) let payload = sourceMapPayloads.get(sourceMap)
if (payload === undefined) { if (payload === undefined) {
...@@ -72,6 +85,198 @@ export function findSourceMapPayload( ...@@ -72,6 +85,198 @@ export function findSourceMapPayload(
return payload return payload
} }
// Shared via `globalThis` because this module is compiled both into the server
// runtime bundles and into `next/dist/server`, and each copy has its own module
// state. `undefined` means lazy source maps are disabled.
const lazySourceMapPayloadsSymbol = Symbol.for(
'next.server.lazySourceMapPayloads'
)
interface LazySourceMapEntry {
/** `undefined` if the file has no (readable) source map. */
payload: ModernSourceMapPayload | undefined
/** Set if the source map is invalid, rethrown on every lookup. */
error: unknown
/** Size of the source map JSON, to bound the memory the cache holds. */
size: number
}
type LazySourceMapPayloads = LRUCache<LazySourceMapEntry>
type GlobalWithLazySourceMapPayloads = typeof globalThis & {
[lazySourceMapPayloadsSymbol]?: LazySourceMapPayloads
}
/**
* Resolves source maps on demand, by reading the `sourceMappingURL` of the
* generated file from disk, instead of having Node.js load them eagerly for
* every module via `--enable-source-maps`. Some Node.js versions (e.g. v20 and
* v22) read and parse the source map of every module they load when source
* maps are enabled, which costs a lot during prerendering of many pages even
* when no error is ever formatted. Enabling source maps later with
* `process.setSourceMapsEnabled(true)` is no alternative, because Node.js
* doesn't map modules that were loaded before.
*
* Has no effect when Node.js source maps are enabled anyway (e.g. via the
* user's `NODE_OPTIONS`), because Node.js then knows the source maps already.
*/
export function enableLazySourceMaps(): void {
const globalWithPayloads = globalThis as GlobalWithLazySourceMapPayloads
if (globalWithPayloads[lazySourceMapPayloadsSymbol] === undefined) {
globalWithPayloads[lazySourceMapPayloadsSymbol] = new LRUCache(
256 * 1024 * 1024,
// Guestimate a small entry for files without a source map so they don't
// fill the cache.
(entry) => Math.max(entry.size, 1024)
)
}
}
// Same as Node.js: the last `sourceMappingURL` comment wins, so that one in a
// string literal earlier in the file doesn't.
const sourceMappingURLComment =
/\/[*/]#\s+sourceMappingURL=(?<sourceMappingURL>[^\s]+)/g
function findLazySourceMapPayload(
sourceURL: string
): ModernSourceMapPayload | undefined {
const payloads = (globalThis as GlobalWithLazySourceMapPayloads)[
lazySourceMapPayloadsSymbol
]
if (payloads === undefined || nodeModules === undefined) {
return undefined
}
let entry = payloads.get(sourceURL)
if (entry === undefined) {
const sourceMap = readSourceMapJSONFromDisk(sourceURL, nodeModules)
entry = { payload: undefined, error: undefined, size: 0 }
if (sourceMap !== undefined) {
entry.size = sourceMap.json.length
try {
entry.payload = sourcesToAbsolute(
JSON.parse(sourceMap.json),
sourceMap.url,
nodeModules
)
} catch (error) {
entry.error = error
}
}
payloads.set(sourceURL, entry)
}
if (entry.error !== undefined) {
// Like `module.findSourceMap` on invalid source maps, so callers report it.
throw entry.error
}
return entry.payload
}
/**
* Resolves `sources` (with `sourceRoot`) to absolute URLs relative to the
* source map's URL, the same way Node.js does for the payloads it returns from
* `module.findSourceMap`, so both kinds of payloads behave the same, e.g. when
* matching a source against the `ignoreList`.
*/
function sourcesToAbsolute(
payload: ModernSourceMapPayload,
sourceMapURL: string,
modules: NonNullable<typeof nodeModules>
): ModernSourceMapPayload {
if ('sections' in payload) {
for (const section of payload.sections) {
sourcesToAbsolute(section.map, sourceMapURL, modules)
}
return payload
}
const { path, url } = modules
// `sources[number]` can be `null`, see `BasicSourceMapPayload`.
payload.sources = (payload.sources as Array<string | null>).map((source) => {
if (source === null) {
return source
}
source = (payload.sourceRoot || '') + source
if (path.isAbsolute(source)) {
return url.pathToFileURL(source).href
}
return new URL(source, sourceMapURL).href
}) as Array<string>
payload.sourceRoot = ''
return payload
}
function readSourceMapJSONFromDisk(
sourceURL: string,
{ fs, path, url }: NonNullable<typeof nodeModules>
): { json: string; url: string } | undefined {
const { readFileSync } = fs
const { isAbsolute } = path
const { fileURLToPath, pathToFileURL } = url
let generatedFileURL: URL
try {
if (sourceURL.startsWith('file://')) {
generatedFileURL = new URL(sourceURL)
} else if (isAbsolute(sourceURL)) {
generatedFileURL = pathToFileURL(sourceURL)
} else {
// Not a file, e.g. `<anonymous>` or `node:internal/...`.
return undefined
}
} catch {
return undefined
}
let sourceMappingURL: string | undefined
try {
const content = readFileSync(
/* turbopackIgnore: true */ generatedFileURL,
'utf8'
)
for (const match of content.matchAll(sourceMappingURLComment)) {
sourceMappingURL = match.groups!.sourceMappingURL
}
} catch {
// e.g. the file was deleted since it was loaded.
return undefined
}
if (sourceMappingURL === undefined) {
return undefined
}
try {
const sourceMapURL = new URL(sourceMappingURL, generatedFileURL)
if (sourceMapURL.protocol === 'data:') {
// e.g. `data:application/json;charset=utf-8;base64,eyJ2...`
const commaIndex = sourceMapURL.pathname.indexOf(',')
if (commaIndex === -1) {
return undefined
}
const metadata = sourceMapURL.pathname.slice(0, commaIndex)
const data = sourceMapURL.pathname.slice(commaIndex + 1)
return {
json: metadata.endsWith(';base64')
? Buffer.from(data, 'base64').toString('utf8')
: decodeURIComponent(data),
// Same as Node.js: inline source maps resolve against the file.
url: generatedFileURL.href,
}
} else if (sourceMapURL.protocol === 'file:') {
return {
json: readFileSync(
/* turbopackIgnore: true */ fileURLToPath(sourceMapURL),
'utf8'
),
url: sourceMapURL.href,
}
} else {
return undefined
}
} catch {
// Missing source map file or malformed URL. Same as Node.js, which treats
// these as having no source map.
return undefined
}
}
export function sourceMapIgnoreListsEverything( export function sourceMapIgnoreListsEverything(
sourceMap: BasicSourceMapPayload sourceMap: BasicSourceMapPayload
): boolean { ): boolean {
...@@ -147,7 +352,8 @@ export function filterStackFrameDEV( ...@@ -147,7 +352,8 @@ export function filterStackFrameDEV(
return false return false
} }
try { try {
// Node.js loads source maps eagerly so this call is cheap. // Cheap after the first lookup, since Node.js or the lazy source maps (see
// `enableLazySourceMaps`) cache the source map.
// TODO: ESM sourcemaps are O(1) but CommonJS sourcemaps are O(Number of CJS modules). // TODO: ESM sourcemaps are O(1) but CommonJS sourcemaps are O(Number of CJS modules).
// Make sure this doesn't adversely affect performance when CJS is used by Next.js. // Make sure this doesn't adversely affect performance when CJS is used by Next.js.
const payload = findSourceMapPayload(sourceURL) const payload = findSourceMapPayload(sourceURL)
......
function logError() {
// Reports whether Node.js source maps are enabled in the prerender worker.
console.error(
new Error(
`logged prerender error (sourceMapsEnabled: ${process.sourceMapsEnabled})`
)
)
}
function throwError(): never {
throw new Error('thrown prerender error')
}
function Inner() {
// The first error formatted in the worker, then a second one.
logError()
throwError()
return null
}
export default function Page() {
return <Inner />
}
import type { ReactNode } from 'react'
export default function Root({ children }: { children: ReactNode }) {
return (
<html>
<body>{children}</body>
</html>
)
}
export default function Page() {
return <p>hello world</p>
}
import type { NextConfig } from 'next'
const nextConfig: NextConfig = {
experimental: {
// Webpack doesn't emit server source maps by default.
serverSourceMaps: true,
},
}
export default nextConfig
import { nextTestSetup } from 'e2e-utils'
import stripAnsi from 'strip-ansi'
function normalizeCliOutput(output: string) {
// Windows paths use backslashes.
return stripAnsi(output).replaceAll('\\', '/')
}
describe('prerender-lazy-source-maps', () => {
const { next } = nextTestSetup({
files: __dirname,
skipStart: true,
})
it('source-maps prerender errors without enabling Node.js source maps', async () => {
const { exitCode, cliOutput } = await next.build()
const output = normalizeCliOutput(cliOutput)
expect(exitCode).toBe(1)
// The source maps are read lazily, so the prerender workers run without
// `--enable-source-maps`, which loads them eagerly for every module.
expect(output).toContain(
'Error: logged prerender error (sourceMapsEnabled: false)'
)
// Already the first error formatted in the worker is source-mapped and has
// a code frame. Columns are flaky in Webpack.
expect(output).toMatch(
/at \S+ \((webpack:\/\/\S*\/)?app\/first\/page\.tsx:4:\d+\)/
)
expect(output).toContain('> 4 | new Error(\n')
// And so are later ones, which reuse the source maps read before.
expect(output).toContain('Error: thrown prerender error')
expect(output).toMatch(
/at \S+ \((webpack:\/\/\S*\/)?app\/first\/page\.tsx:11:\d+\)/
)
})
it('does not source-map prerender errors with enablePrerenderSourceMaps: false', async () => {
await next.patchFile('next.config.ts', (config) =>
config.replace(
'experimental: {',
'enablePrerenderSourceMaps: false,\n experimental: {'
)
)
try {
const { exitCode, cliOutput } = await next.build()
const output = normalizeCliOutput(cliOutput)
expect(exitCode).toBe(1)
expect(output).toContain(
'Error: logged prerender error (sourceMapsEnabled: false)'
)
expect(output).not.toMatch(/app\/first\/page\.tsx:\d+:\d+/)
expect(output).toMatch(/at \S+ \(\.next\/server\/\S+\.js:\d+:\d+\)/)
} finally {
await next.patchFile('next.config.ts', (config) =>
config.replace('enablePrerenderSourceMaps: false,\n ', '')
)
}
})
})
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment