Unverified Commit e953bdaa authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

Fix dev route discovery startup race (#97920)

### What?

Fix a dev-server startup race that could temporarily publish an empty
Pages Router route set when Watchpack's first filesystem aggregation
arrived before its route-directory scan completed.

The dev bundler now snapshots the initial `pages/` and `app/` files
before starting the watcher. During startup it ignores Watchpack
aggregations that are missing an existing snapshot page file, so it does
not publish route definitions or errors from a partial filesystem view.
Empty route directories are detected from the same snapshot rather than
from a separate callback that could race or mistake a read error for an
empty directory.

The `next-head` e2e fixture now delays Watchpack's initial `pages/` scan
in development mode, deterministically covering the failing ordering
while preserving every existing head-tag assertion and non-dev test
mode.

### Why?

PR #97905's build-and-test run 32911367895, job 98006962799 (`test cache
components dev (6/6) / build`) exposed this as a flaky
`test/e2e/next-head/index.test.ts` failure. In the raw job log around
lines 3135–3305, retry 0 reported `Ready in 420ms` and then served `/`
as a 404 four times. The returned document contained the `_error` page
title (`404: This page could not be found`) and `pages__error` chunks,
so four head-tag assertions failed downstream. Retry 1 passed about nine
seconds later.

The first Watchpack aggregation could run route discovery before
`pages/index.js` appeared in `getTimeInfoEntries()`. That published no
page route definitions and released the request handler, causing `/` to
fall through to `_error` until a later aggregation repaired the route
table. This change waits to publish any initial route state until
Watchpack has a complete view, rather than adding request retries,
sleeps, or broader timeouts.

### How?

- Snapshot route directories once before watcher startup, tolerating a
directory removed during boot so Watchpack can report the removal
normally.
- Ignore incomplete startup aggregations until Watchpack observes every
snapshot page that still exists, with a 30-second safety deadline so an
unwatchable file cannot stall startup.
- Drive the empty-directory startup path from the snapshot, removing the
error-swallowing `fs.readdir` race.
- Delay the first Watchpack `pages/` scan in a separate dev-only
route-discovery case and assert that the injection fired. The focused
test reproduced `GET / 404` before the fix and returns `GET / 200` with
the fix.
- Preserve the original five `next/head` test cases unchanged; validate
the app/pages conflict path separately.

### Verification

- Exact CI environment: `__NEXT_CACHE_COMPONENTS=true
__NEXT_EXPERIMENTAL_CACHED_NAVIGATIONS=true
__NEXT_EXPERIMENTAL_SERVER_COMPONENTS_HMR_CANCELLATION=true
NEXT_EXTERNAL_TESTS_FILTERS=test/cache-components-tests-manifest.json
NEXT_TEST_MODE=dev IS_TURBOPACK_TEST=1 TURBOPACK_DEV=1 pnpm
test-dev-turbo test/e2e/next-head/index.test.ts` — 5/5 passed
- `IS_WEBPACK_TEST=1 NEXT_TEST_MODE=dev pnpm test-dev-webpack
test/e2e/next-head/index.test.ts` — 5/5 passed
- `pnpm --filter=next types`
- `pnpm --filter=next build`
- Prettier, ESLint, and `git diff --check`

<!-- NEXT_JS_LLM -->


<!-- fleet 4ac891b6-2d24-4103-95bf-9daf04359e3f -->

---------
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent 6cb63960
......@@ -88,6 +88,7 @@ import {
import { getDefineEnv } from '../../../build/define-env'
import { TurbopackInternalError } from '../../../shared/lib/turbopack/internal-error'
import { normalizePath } from '../../../lib/normalize-path'
import { recursiveReadDir } from '../../../lib/recursive-readdir'
import {
JSON_CONTENT_TYPE_HEADER,
MIDDLEWARE_FILENAME,
......@@ -384,24 +385,37 @@ async function startWatcher(
let prevSortedRoutes: string[] = []
let hasComputedSortedRoutes = false
await new Promise<void>(async (resolve, reject) => {
if (pagesDir) {
// Watchpack doesn't emit an event for an empty directory
fs.readdir(pagesDir, (_, files) => {
if (files?.length) {
return
}
if (!resolved) {
resolve()
resolved = true
const pages = pagesDir ? [pagesDir] : []
const app = appDir ? [appDir] : []
const directories = [...pages, ...app]
// Snapshot the route directories before watching starts. Watchpack can report
// its first aggregation while a directory scan is still in flight, so route
// discovery needs this snapshot to avoid observing a partial filesystem.
const initialFiles = (
await Promise.all(
directories.map(async (directory) => {
try {
return await recursiveReadDir(directory, {
relativePathnames: false,
})
} catch (err: any) {
// The directory can be removed while the dev server boots. Watchpack
// reports that as a removal later on.
if (err.code !== 'ENOENT') throw err
return []
}
})
}
)
).flat()
const initialPageFiles = initialFiles.filter(validFileMatcher.isPageFile)
const pages = pagesDir ? [pagesDir] : []
const app = appDir ? [appDir] : []
const directories = [...pages, ...app]
await new Promise<void>(async (resolve, reject) => {
if (initialFiles.length === 0) {
// Watchpack doesn't emit an event when all route directories are empty.
resolve()
resolved = true
}
const rootDir = pagesDir || appDir
const files = [
......@@ -455,14 +469,32 @@ async function startWatcher(
const validatorFilePath = path.join(distDir, 'types', 'validator.ts')
let initialWatchTime = performance.now() + performance.timeOrigin
// Startup only waits this long for Watchpack to observe the files found by
// the snapshot above, so a file it never reports cannot stall the server.
const initialScanDeadline = initialWatchTime + 30_000
wp.on('aggregated', async () => {
const isInitialScan = !hadInitialScan
hadInitialScan = true
let writeEnvDefinitions = false
let typescriptStatusFromLastAggregation = enabledTypeScript
let middlewareMatchers: ProxyMatcher[] | undefined
const routedPages: string[] = []
const knownFiles = wp.getTimeInfoEntries()
// Watchpack can emit an aggregation while its initial directory scan is
// still in flight. Ignore partial startup passes so route definitions and
// errors are not published until every initial page file is observable.
// Bounded by a deadline so an unwatchable file can never block startup.
if (
!resolved &&
performance.now() + performance.timeOrigin < initialScanDeadline &&
initialPageFiles.some(
(file) => fs.existsSync(file) && !knownFiles.has(file)
)
) {
return
}
hadInitialScan = true
const appPaths: Record<string, string[]> = {}
const defaultAppPaths = new Set<string>()
const pageNameSet = new Set<string>()
......
const fs = require('node:fs')
const path = require('node:path')
const pagesDir = path.join(process.cwd(), 'pages')
const originalReaddir = fs.readdir
let injected = false
fs.readdir = function readdir(directory, ...args) {
const stack = new Error().stack
if (!injected && directory === pagesDir && stack.includes('watchpack')) {
injected = true
console.log('[next-head] delaying initial Watchpack pages scan')
return setTimeout(
() => originalReaddir.call(this, directory, ...args),
1000
)
}
return originalReaddir.call(this, directory, ...args)
}
......@@ -61,3 +61,30 @@ describe('next/head', () => {
)
})
})
describe('next/head dev route discovery', () => {
if (!(global as any).isNextDev) {
return
}
const { next } = nextTestSetup({
files: {
pages: new FileRef(join(__dirname, 'app/pages')),
components: new FileRef(join(__dirname, 'app/components')),
'inject-readdir-delay.cjs': new FileRef(
join(__dirname, 'app/inject-readdir-delay.cjs')
),
},
env: { NODE_OPTIONS: '--require ./inject-readdir-delay.cjs' },
})
it('waits for the initial route scan before serving pages', async () => {
const response = await next.fetch('/')
expect(response.status).toBe(200)
expect(await response.text()).toContain('index page')
expect(next.cliOutput).toContain(
'[next-head] delaying initial Watchpack pages scan'
)
})
})
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment