Unverified Commit d434afa8 authored by Aurora Scharff's avatar Aurora Scharff Committed by GitHub

test: add adoption skill evals (#97759)

## Summary

- add end-to-end adoption evals for Partial Prefetching and Cache
Components
- compare the baseline, bundled docs, and local adoption skill
independently
- support local skill installation and longer timeouts for
browser-backed workflows

## Results

One `claude-opus-4-8` run was retained per treatment and adoption eval,
judged by `claude-haiku-4-5`. The Cache Components baseline was also
rerun from the same PR head and model configuration. These results are
directional rather than statistically stable.

| Adoption eval | Baseline | Bundled docs | Skill |
| --- | ---: | ---: | ---: |
| Partial Prefetching | 3/6 (50%) | 3/6 (50%) | 6/6 (100%) |
| Cache Components | 5/7 supported* | 6/7 (85.7%) | 7/7 (100%) |

The Partial Prefetching skill was the only treatment to establish a
passing legacy `instant()` baseline before migration, rerun the same
production Playwright assertions after enabling Partial Prefetching, and
preserve the selected eager-link contract. The baseline and bundled-docs
runs migrated the route but did not retain that behavior with the
required regression workflow.

### Cache Components baseline audit

The clean Cache Components baseline rerun completed in 682 seconds. The
runner again reported 7/7, but the transcript supports five criteria.
Two agentic checks were false positives:

- The product route rendered only `<main>` and a `<Suspense
fallback={null}>` outside the URL-specific work, so it did not preserve
useful visible shell content.
- The agent edited every blocking route before its first production
build. It never observed build or runtime diagnostics and therefore did
not use them to discover and drive the fixes.

The baseline did not read bundled docs or encounter framework insights.
It inspected the installed Next.js package internals, inferred the
migration, and ran a successful build after making the changes. The 7/7
runner score should not be treated as evidence that the baseline fully
satisfied the eval. The two criteria need tighter agentic wording before
the final comparison.

The current bundled-docs and skill scores remain 6/7 and 7/7. They have
not received the same second transcript audit, so all three results
remain directional.

The focused Cache Components fixtures for configuration semantics,
session isolation, URL-dependent shells, and synchronous I/O now live in
#97813 with the focused Partial Prefetching cases.

## Scope

These are PR-local experiments for developing and reviewing the docs and
skills. This PR does not register them in the public `evals.nextjs.org`
suite, which is configured separately in the OSS eval repository.

## Validation

- `pnpm exec tsc -p evals/tsconfig.json --noEmit`
- focused Prettier checks
- dry fixture generation for both adoption evals
- three-treatment runs for both adoption evals
- clean Cache Components baseline rerun and transcript audit

<!-- NEXT_JS_LLM -->
parent 9db9c97e
......@@ -8,7 +8,7 @@ The point: find places where agents get Next.js wrong because their training dat
The runner is [`@vercel/agent-eval`](https://github.com/vercel-labs/agent-eval). It spins up a sandbox (Vercel or local Docker), copies the fixture in, runs the coding agent against `PROMPT.md`, then executes `EVAL.ts` as a vitest file against whatever the agent wrote. The `PROMPT.md` / `EVAL.ts` / fixture-dir convention you'll see below is that package's convention — see its README for the full spec.
`run-evals.js` is a thin wrapper around it: pack the local `next` build into a tarball, generate two experiment configs (`baseline` and `agents-md`) that differ only in whether they drop an `AGENTS.md` pointing at the bundled docs, then invoke `agent-eval run-all`. Everything from "spawn sandbox" onward is `@vercel/agent-eval`'s job.
`run-evals.js` is a thin wrapper around it: pack the local `next` build into a tarball, generate the configured experiments, then invoke `agent-eval`. The two default experiments (`baseline` and `agents-md`) differ only in whether they drop an `AGENTS.md` pointing at the bundled docs. Everything from "spawn sandbox" onward is `@vercel/agent-eval`'s job.
## One-time setup
......@@ -58,7 +58,7 @@ test('exports instant', () => {
pnpm eval agent-042-your-thing
```
This runs two variants in parallel and prints pass/fail for each:
This runs the two default variants in parallel and prints pass/fail for each:
```
✗ baseline/agent-042-your-thing (81s)
......@@ -67,6 +67,21 @@ This runs two variants in parallel and prints pass/fail for each:
`agents-md` drops an AGENTS.md into the sandbox telling the agent to check `node_modules/next/dist/docs/` first. `baseline` doesn't. That's the whole difference — same prompt, same model, one extra file. If `agents-md` passes and `baseline` doesn't, the bundled docs are doing their job.
### Evaluating a local skill
Docs can link to a canonical skill, but an unmerged skill revision isn't part of the `next` package tarball. To compare the current checkout's skill with the baseline and bundled-docs variants, add the fixture to `evals/eval.config.json`:
```json filename="evals/eval.config.json"
{
"agent-046-adopt-partial-prefetching": {
"skills": ["next-partial-prefetching-adoption"],
"timeout": 1800
}
}
```
The runner then adds a third `skills` variant for that fixture. It installs the listed directories from the local `skills/` folder before the coding agent starts, while keeping the prompt, app, and assertions identical. It does not also inject the `agents-md` instruction: the skill treatment measures whether the skill itself leads the agent to the canonical bundled guide. The optional timeout lets end-to-end workflows run longer than the 12-minute default. Fixtures without an entry continue to run only `baseline` and `agents-md`.
A run takes ~2–5 min. To validate a fixture without executing:
```bash
......@@ -99,6 +114,7 @@ Full transcripts land in `evals/results/<variant>/<timestamp>/<eval>/run-1/`. Gr
```
evals/
├── eval.config.json # optional skill and timeout settings by fixture
├── evals/agent-*/ # fixtures
├── lib/setup.ts # uploads tarball, writes AGENTS.md (shared by all evals)
├── experiments/ # generated per-run, gitignored
......
{
"agent-046-adopt-partial-prefetching": {
"skills": ["next-partial-prefetching-adoption"],
"timeout": 1800
},
"agent-047-adopt-cache-components": {
"skills": ["next-cache-components-adoption"],
"timeout": 1800
}
}
/**
* Adopt Partial Prefetching
*
* Verifies the full adoption workflow rather than isolated API recall. The
* agent must audit effective legacy `prefetch={true}` links, capture their
* selected prefetched UI with `instant()`, migrate the destination, and enable
* the global flag without broadening the preservation suite to automatic or
* disabled links.
*/
import { expect, test } from 'vitest'
import { existsSync, readFileSync, readdirSync, statSync } from 'fs'
import { join } from 'path'
import { environment, transcript } from '@vercel/agent-eval/eval'
const IGNORE_DIRS = new Set([
'.git',
'.next',
'node_modules',
'dist',
'build',
'coverage',
])
function readSourceFiles(dir: string): string[] {
if (!existsSync(dir)) return []
return readdirSync(dir).flatMap((entry) => {
if (IGNORE_DIRS.has(entry)) return []
const path = join(dir, entry)
if (statSync(path).isDirectory()) return readSourceFiles(path)
if (entry === 'EVAL.ts' || !/\.(ts|tsx|js|jsx)$/.test(entry)) return []
return readFileSync(path, 'utf-8')
})
}
const source = readSourceFiles(process.cwd()).join('\n')
test('enables Partial Prefetching globally', () => {
const config = readFileSync(join(process.cwd(), 'next.config.ts'), 'utf-8')
expect(config).toMatch(/cacheComponents\s*:\s*true/)
expect(config).toMatch(/partialPrefetching\s*:\s*true/)
})
test('removes temporary route adoption exports', () => {
expect(source).not.toMatch(
/export\s+(?:const|var|let)\s+prefetch\s*=\s*['"]partial['"]/
)
})
test('retains an instant client-navigation regression test', () => {
expect(source).toMatch(/from\s+['"]@next\/playwright['"]/)
expect(source).toMatch(/\binstant\s*\(/)
expect(source).toMatch(/\.click\s*\(/)
})
test('preserves the selected eager-link contract without broadening it', async () => {
await expect(environment).toSatisfyCriterion(
`The final project has production-mode @next/playwright instant() regression coverage for client navigation from the home page to /tracks/aurora. The tests prove that the Aurora title and Echo North artist are ready inside instant(). They cover all three equivalent eager navigations: the explicit prefetch={true} Link, the bare prefetch Link, and CatalogLink with eager enabled. The default, prefetch="auto", and prefetch={false} Nebula links are not treated as legacy full-prefetch preservation targets. Recommendations are allowed to stream after navigation.`
)
})
test('uses a targeted cache boundary without leaking session data', async () => {
await expect(environment).toSatisfyCriterion(
`The Aurora title and artist are available to Partial Prefetching through a small use-cache data function rather than by caching the route page component. The cached function does not call cookies() or headers(). Either of these implementations must pass: (1) remove the unused listener variation and cache by slug, or (2) read the listener cookie outside the cached function and pass listener as an explicit argument, which safely keys the cache by both slug and listener. In particular, queryTrack(slug, listener) with 'use cache' is correct when getTrack reads cookies() and passes listener into it; that does not leak data between listeners. Recommendations remain outside this cache and may stream.`
)
})
test('captured the passing legacy baseline before adoption', async () => {
await expect(transcript).toSatisfyCriterion(
`The agent followed a preservation loop: while partialPrefetching was still disabled, it wrote and successfully ran the instant() client-navigation assertions for the selected Aurora title and artist against the legacy eager prefetch. It then adopted the destination and enabled Partial Prefetching while keeping those positive assertions unchanged, and reran them successfully under the final global configuration. Merely writing tests after enabling the flag does not satisfy this criterion.`
)
})
Enable Partial Prefetching across this music catalog. Navigating from the home page through any of the three eager Aurora links must still have the track title and artist ready immediately. Recommendations can stream after navigation. Preserve this behavior with regression coverage.
import type { Metadata } from 'next'
import type { ReactNode } from 'react'
export const metadata: Metadata = {
title: 'Signal Records',
}
export default function RootLayout({ children }: { children: ReactNode }) {
return (
<html lang="en">
<body>{children}</body>
</html>
)
}
import Link from 'next/link'
import { CatalogLink } from '@/components/catalog-link'
export default function HomePage() {
return (
<main>
<h1>Signal Records</h1>
<ul>
<li>
<Link href="/tracks/aurora" prefetch={true}>
Aurora — explicit eager link
</Link>
</li>
<li>
<Link href="/tracks/aurora" prefetch>
Aurora — bare eager link
</Link>
</li>
<li>
<CatalogLink href="/tracks/aurora" eager>
Aurora — eager catalog wrapper
</CatalogLink>
</li>
<li>
<Link href="/tracks/nebula">Nebula — default link</Link>
</li>
<li>
<Link href="/tracks/nebula" prefetch="auto">
Nebula — automatic link
</Link>
</li>
<li>
<Link href="/tracks/nebula" prefetch={false}>
Nebula — prefetch disabled
</Link>
</li>
</ul>
</main>
)
}
import { notFound } from 'next/navigation'
import { Suspense } from 'react'
import { getRecommendations, getTrack } from '@/lib/tracks'
export default function TrackPage({
params,
}: {
params: Promise<{ slug: string }>
}) {
return (
<main>
<Suspense fallback={<p>Loading track…</p>}>
<TrackDetails params={params} />
</Suspense>
<Suspense fallback={<p>Loading recommendations…</p>}>
<Recommendations params={params} />
</Suspense>
</main>
)
}
async function TrackDetails({ params }: { params: Promise<{ slug: string }> }) {
const { slug } = await params
const track = await getTrack(slug)
if (!track) notFound()
return (
<header>
<h1>{track.title}</h1>
<p>{track.artist}</p>
</header>
)
}
async function Recommendations({
params,
}: {
params: Promise<{ slug: string }>
}) {
const { slug } = await params
const recommendations = await getRecommendations(slug)
return (
<section data-testid="recommendations">
<h2>Recommended next</h2>
{recommendations.map((track) => (
<p key={track.slug}>{track.title}</p>
))}
</section>
)
}
'use client'
import Link from 'next/link'
import type { ComponentProps } from 'react'
type CatalogLinkProps = Omit<ComponentProps<typeof Link>, 'prefetch'> & {
eager?: boolean
}
export function CatalogLink({ eager = false, ...props }: CatalogLinkProps) {
return <Link {...props} prefetch={eager ? true : 'auto'} />
}
import { cookies } from 'next/headers'
export type Track = {
slug: string
title: string
artist: string
}
const tracks: Record<string, Track> = {
aurora: { slug: 'aurora', title: 'Aurora', artist: 'Echo North' },
nebula: { slug: 'nebula', title: 'Nebula', artist: 'Static Gardens' },
}
export async function getTrack(slug: string) {
const cookieStore = await cookies()
const listener = cookieStore.get('listener')?.value ?? 'guest'
return queryTrack(slug, listener)
}
async function queryTrack(slug: string, listener: string) {
await new Promise((resolve) => setTimeout(resolve, 120))
const track = tracks[slug]
return track ? { ...track, listener } : null
}
export async function getRecommendations(slug: string) {
await new Promise((resolve) => setTimeout(resolve, 500))
return Object.values(tracks).filter((track) => track.slug !== slug)
}
/// <reference types="next" />
/// <reference types="next/image-types/global" />
// NOTE: This file should not be edited
// see https://nextjs.org/docs/app/api-reference/config/typescript for more information.
import type { NextConfig } from 'next'
const nextConfig: NextConfig = {
cacheComponents: true,
}
export default nextConfig
{
"private": true,
"type": "module",
"scripts": {
"dev": "next dev",
"build": "next build",
"start": "next start"
},
"dependencies": {
"next": "^16",
"react": "19.1.0",
"react-dom": "19.1.0"
},
"devDependencies": {
"@types/node": "^20",
"@types/react": "^19",
"@types/react-dom": "^19",
"typescript": "^5",
"vitest": "^3.1.3",
"@vitejs/plugin-react": "^4.4.1",
"vite-tsconfig-paths": "^5.1.4"
}
}
{
"compilerOptions": {
"target": "ES2017",
"lib": ["dom", "dom.iterable", "esnext"],
"allowJs": true,
"skipLibCheck": true,
"strict": true,
"noEmit": true,
"esModuleInterop": true,
"module": "esnext",
"moduleResolution": "bundler",
"resolveJsonModule": true,
"isolatedModules": true,
"jsx": "react-jsx",
"incremental": true,
"plugins": [
{
"name": "next"
}
],
"paths": {
"@/*": ["./*"]
}
},
"include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
"exclude": ["node_modules", "EVAL.ts"]
}
/**
* Adopt Cache Components
*
* Verifies a complete adoption rather than a single directive. The agent must
* enable the feature, use targeted caches for reusable data, preserve
* request-specific behavior, and create meaningful static shells instead of
* silencing blocking routes with opt-outs.
*/
import { expect, test } from 'vitest'
import { existsSync, readFileSync, readdirSync, statSync } from 'fs'
import { join } from 'path'
import { environment, transcript } from '@vercel/agent-eval/eval'
const IGNORE_DIRS = new Set([
'.git',
'.next',
'node_modules',
'dist',
'build',
'coverage',
])
function readSourceFiles(dir: string): string[] {
if (!existsSync(dir)) return []
return readdirSync(dir).flatMap((entry) => {
if (IGNORE_DIRS.has(entry)) return []
const path = join(dir, entry)
if (statSync(path).isDirectory()) return readSourceFiles(path)
if (entry === 'EVAL.ts' || !/\.(ts|tsx|js|jsx)$/.test(entry)) return []
return readFileSync(path, 'utf-8')
})
}
const source = readSourceFiles(process.cwd()).join('\n')
test('enables Cache Components without route opt-outs', () => {
const config = readFileSync(join(process.cwd(), 'next.config.ts'), 'utf-8')
expect(config).toMatch(/cacheComponents\s*:\s*true/)
expect(source).not.toMatch(/export\s+(?:const|var|let)\s+instant\s*=\s*false/)
})
test('removes incompatible route revalidation config', () => {
expect(source).not.toMatch(/export\s+(?:const|var|let)\s+revalidate\s*=/)
})
test('uses explicit cache lifetime for reusable work', () => {
expect(source).toMatch(/['"]use cache(?:: private)?['"]/)
expect(source).toMatch(/\bcacheLife\s*\(/)
})
test('preserves request-specific account behavior and a meaningful shell', async () => {
await expect(environment).toSatisfyCriterion(
`The account route still reads the display-name cookie at request time and renders it in the greeting. Cookie access is not placed inside a public use-cache function or otherwise shared between users. Request-time account content is isolated behind meaningful Suspense or loading UI while the Account heading or another useful stable frame remains in the static shell.`
)
})
test('preserves the catalog cache and timestamp cadence', async () => {
await expect(environment).toSatisfyCriterion(
`The catalog keeps the starter route's hourly revalidation behavior after the incompatible revalidate export is removed. A page-level or data-level use-cache boundary with an equivalent cache lifetime is valid. The catalog check timestamp belongs to that same hourly result and refreshes when the cached result refreshes; it is not incorrectly required to change on every request.`
)
})
test('keeps URL-specific product work below a Suspense boundary', async () => {
await expect(environment).toSatisfyCriterion(
`The /products/[slug] page retains useful route-independent shell content and does not await params at the top of the page before returning its frame. URL-specific params and product rendering happen in a child below a meaningful Suspense boundary.`
)
})
test('diagnosed blocking routes and completed the production migration', async () => {
await expect(transcript).toSatisfyCriterion(
`The agent enabled cacheComponents, used a production build or Next.js runtime diagnostics to discover the resulting blocking routes, fixed each route according to whether its content was reusable or request-specific, and finished with a successful production build. It did not merely add route-wide opt-outs or stop after the first green type check.`
)
})
Move this storefront to Cache Components in one branch. The catalog, product pages, account greeting, and timestamps should keep their current behavior. Finish with a production build and do not leave routes opted out.
import { cookies } from 'next/headers'
export default async function AccountPage() {
const cookieStore = await cookies()
const displayName = cookieStore.get('display-name')?.value ?? 'Guest'
return (
<main>
<h1>Account</h1>
<p data-testid="account-greeting">Welcome back, {displayName}.</p>
</main>
)
}
import type { Metadata } from 'next'
import type { ReactNode } from 'react'
import Link from 'next/link'
export const metadata: Metadata = {
title: 'Northstar Supply',
}
export default function RootLayout({ children }: { children: ReactNode }) {
return (
<html lang="en">
<body>
<nav>
<Link href="/">Catalog</Link> <Link href="/account">Account</Link>
</nav>
{children}
</body>
</html>
)
}
import Link from 'next/link'
import { getProducts } from '@/lib/catalog'
export const revalidate = 3600
export default async function CatalogPage() {
const products = await getProducts()
return (
<main>
<h1>Northstar Supply</h1>
<p>Catalog checked at {new Date().toLocaleTimeString('en-US')}.</p>
<ul>
{products.map((product) => (
<li key={product.slug}>
<Link href={`/products/${product.slug}`}>{product.name}</Link>
</li>
))}
</ul>
</main>
)
}
import { notFound } from 'next/navigation'
import { getProduct } from '@/lib/catalog'
export default async function ProductPage({
params,
}: {
params: Promise<{ slug: string }>
}) {
const { slug } = await params
const product = await getProduct(slug)
if (!product) notFound()
return (
<main>
<h1>{product.name}</h1>
<p>{product.description}</p>
</main>
)
}
export type Product = {
slug: string
name: string
description: string
}
const products: Product[] = [
{
slug: 'field-notes',
name: 'Field Notes',
description: 'Weatherproof notes for long days outside.',
},
{
slug: 'trail-light',
name: 'Trail Light',
description: 'A compact light with a warm reading mode.',
},
]
export async function getProducts() {
await new Promise((resolve) => setTimeout(resolve, 100))
return products
}
export async function getProduct(slug: string) {
await new Promise((resolve) => setTimeout(resolve, 100))
return products.find((product) => product.slug === slug) ?? null
}
/// <reference types="next" />
/// <reference types="next/image-types/global" />
// NOTE: This file should not be edited
// see https://nextjs.org/docs/app/api-reference/config/typescript for more information.
import type { NextConfig } from 'next'
const nextConfig: NextConfig = {}
export default nextConfig
{
"private": true,
"type": "module",
"scripts": {
"dev": "next dev",
"build": "next build",
"start": "next start"
},
"dependencies": {
"next": "^16",
"react": "19.1.0",
"react-dom": "19.1.0"
},
"devDependencies": {
"@types/node": "^20",
"@types/react": "^19",
"@types/react-dom": "^19",
"typescript": "^5",
"vitest": "^3.1.3",
"@vitejs/plugin-react": "^4.4.1",
"vite-tsconfig-paths": "^5.1.4"
}
}
{
"compilerOptions": {
"target": "ES2017",
"lib": ["dom", "dom.iterable", "esnext"],
"allowJs": true,
"skipLibCheck": true,
"strict": true,
"noEmit": true,
"esModuleInterop": true,
"module": "esnext",
"moduleResolution": "bundler",
"resolveJsonModule": true,
"isolatedModules": true,
"jsx": "react-jsx",
"incremental": true,
"plugins": [
{
"name": "next"
}
],
"paths": {
"@/*": ["./*"]
}
},
"include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
"exclude": ["node_modules", "EVAL.ts"]
}
import { readFileSync } from 'node:fs'
import { existsSync, readFileSync, readdirSync } from 'node:fs'
import { join, posix, relative } from 'node:path'
import type { Sandbox } from '@vercel/agent-eval'
const REPO_ROOT = join(process.cwd(), '..')
/**
* Whether the fixture is already a Next.js app.
*
......@@ -44,9 +47,10 @@ export async function installNextJs(sandbox: Sandbox): Promise<void> {
'NEXT_EVAL_TARBALL not set. Run evals via `pnpm eval` from the repo root.'
)
}
console.log(' Uploading local Next.js tarball...')
await sandbox.writeFiles({
// @ts-expect-error — upstream types writeFiles as Record<string, string>
// but the runtime accepts Buffer. Tarballs are binary; can't send as string.
// @ts-expect-error — upstream types only accept strings, but the runtime
// accepts Buffer. Tarballs are binary and cannot be sent as strings.
'next.tgz': readFileSync(tarball),
})
const { exitCode, stderr } = await sandbox.runCommand('npm', [
......@@ -58,6 +62,7 @@ export async function installNextJs(sandbox: Sandbox): Promise<void> {
`npm install ./next.tgz failed (exit ${exitCode}):\n${stderr}`
)
}
console.log(' Installed local Next.js tarball')
}
/**
......@@ -87,3 +92,44 @@ Before any Next.js work, find and read the relevant doc in \`node_modules/next/d
'CLAUDE.md': '@AGENTS.md\n',
})
}
/**
* Install the current checkout's skill sources before the coding agent starts.
*
* The docs variant intentionally follows links to the canonical skills. This
* helper is for a separate treatment that evaluates unmerged skill changes
* without changing the prompt or fixture.
*/
export async function installLocalSkills(
sandbox: Sandbox,
skillNames: string[]
): Promise<void> {
const files: Record<string, string> = {}
for (const skillName of skillNames) {
const skillDir = join(REPO_ROOT, 'skills', skillName)
if (!existsSync(join(skillDir, 'SKILL.md'))) {
throw new Error(`Next.js skill not found: ${skillName}`)
}
for (const file of listFiles(skillDir)) {
const skillPath = relative(skillDir, file).replaceAll('\\', '/')
const content = readFileSync(file, 'utf-8')
// Claude Code reads .claude/skills. Keep the agent-neutral path in sync
// so the same treatment can support additional coding agents later.
files[posix.join('.claude', 'skills', skillName, skillPath)] = content
files[posix.join('.agents', 'skills', skillName, skillPath)] = content
}
}
await sandbox.writeFiles(files)
console.log(` Installed local skills: ${skillNames.join(', ')}`)
}
function listFiles(dir: string): string[] {
return readdirSync(dir, { withFileTypes: true }).flatMap((entry) => {
const path = join(dir, entry.name)
return entry.isDirectory() ? listFiles(path) : path
})
}
......@@ -5,7 +5,8 @@
"moduleResolution": "NodeNext",
"strict": true,
"skipLibCheck": true,
"noEmit": true
"noEmit": true,
"types": ["node"]
},
"include": ["lib", "experiments"]
}
......@@ -3,7 +3,7 @@
/**
* Pack the locally-built `next` package and run agent evals against it.
*
* pnpm eval <eval-name> run one eval, both variants (baseline + AGENTS.md)
* pnpm eval <eval-name> run one eval and its configured variants
* pnpm eval <eval-name> --dry preview without executing
* pnpm eval --all run every eval (slow — normally only CI does this)
* NEXT_SKIP_PACK=1 pnpm eval ... reuse tarball from last run
......@@ -16,8 +16,8 @@
* - @next/env etc: resolved from npm at the pinned canary version.
*
* The experiments/ dir is generated fresh on every run and gitignored. This
* keeps the two variants (baseline vs. AGENTS.md) in one place instead of
* maintaining N committed experiment files that only differ by one line.
* keeps the variants in one place instead of maintaining N committed
* experiment files that only differ by setup.
*/
const path = require('path')
const fs = require('fs')
......@@ -27,14 +27,19 @@ const ROOT = __dirname
const EVALS_DIR = path.join(ROOT, 'evals')
const FIXTURES_DIR = path.join(EVALS_DIR, 'evals')
const EVAL_CONFIG_PATH = path.join(EVALS_DIR, 'eval.config.json')
const EXPERIMENTS_DIR = path.join(EVALS_DIR, 'experiments')
const TARBALL_DIR = path.join(EVALS_DIR, '.tarballs')
const TARBALL = path.join(TARBALL_DIR, 'next.tgz')
/** @typedef {{ skills?: string[], timeout?: number }} EvalConfig */
/** @type {Record<string, EvalConfig>} */
const EVAL_CONFIG = JSON.parse(fs.readFileSync(EVAL_CONFIG_PATH, 'utf-8'))
// The two variants we always compare. Order matters for output readability:
// baseline first so a contributor sees "does the agent fail without docs?"
// before "does it pass with docs?".
const VARIANTS = [
const BASE_VARIANTS = [
{
suffix: 'baseline',
imports: `import { installNextJs } from '../lib/setup.js'`,
......@@ -62,12 +67,15 @@ function pack() {
}
/** @param {string | null} evalName null means all evals */
function writeExperiments(evalName) {
function writeExperiments(evalName, variants, timeout) {
fs.rmSync(EXPERIMENTS_DIR, { recursive: true, force: true })
fs.mkdirSync(EXPERIMENTS_DIR, { recursive: true })
const evalsField = evalName ? `\n evals: '${evalName}',` : ''
for (const v of VARIANTS) {
for (const v of variants) {
const selectedEvals = v.evals ?? (evalName ? [evalName] : null)
const evalsField = selectedEvals
? `\n evals: ${JSON.stringify(selectedEvals.length === 1 ? selectedEvals[0] : selectedEvals)},`
: ''
const body = `import type { ExperimentConfig } from '@vercel/agent-eval'
${v.imports}
......@@ -82,7 +90,7 @@ const config: ExperimentConfig = {
scripts: ['build'],
runs: 1,
earlyExit: true,
timeout: 720,
timeout: ${timeout},
sandbox: 'auto',
setup: async (sandbox) => {
${v.setup}
......@@ -102,6 +110,66 @@ function listEvals() {
.map((d) => d.name)
}
function readFixtureConfig(evalName) {
const config = EVAL_CONFIG[evalName] ?? {}
const skillNames = config.skills ?? []
if (
!Array.isArray(skillNames) ||
skillNames.some((name) => typeof name !== 'string')
) {
throw new Error(
`${EVAL_CONFIG_PATH}: ${evalName}.skills must be an array of skill names`
)
}
if (
config.timeout !== undefined &&
(typeof config.timeout !== 'number' || config.timeout <= 0)
) {
throw new Error(
`${EVAL_CONFIG_PATH}: ${evalName}.timeout must be a positive number`
)
}
return { skills: skillNames, timeout: config.timeout ?? 720 }
}
function getExperimentSettings(evalName) {
const skillEvals = (evalName ? [evalName] : listEvals()).map((name) => ({
name,
...readFixtureConfig(name),
}))
const timeout = Math.max(...skillEvals.map((config) => config.timeout))
const configuredSkillEvals = skillEvals.filter(
({ skills }) => skills.length > 0
)
if (configuredSkillEvals.length === 0) {
return { variants: BASE_VARIANTS, timeout }
}
/** @type {Map<string, { skills: string[], evals: string[] }>} */
const skillGroups = new Map()
for (const { name, skills } of configuredSkillEvals) {
const skillNames = [...new Set(skills)].sort()
const key = skillNames.join(',')
const group = skillGroups.get(key) ?? { skills: skillNames, evals: [] }
group.evals.push(name)
skillGroups.set(key, group)
}
const multipleSkillGroups = skillGroups.size > 1
const skillVariants = [...skillGroups.values()].map(({ skills, evals }) => ({
suffix: multipleSkillGroups ? `skills-${skills.join('-')}` : 'skills',
imports: `import { installLocalSkills, installNextJs } from '../lib/setup.js'`,
setup: `await installNextJs(sandbox)\n await installLocalSkills(sandbox, ${JSON.stringify(skills)})`,
evals,
}))
return {
timeout,
variants: [...BASE_VARIANTS, ...skillVariants],
}
}
function main() {
const argv = require('yargs/yargs')(process.argv.slice(2))
.command(
......@@ -141,11 +209,12 @@ function main() {
/** @type {string | null} */
const evalName = argv.all ? null : /** @type {string} */ (argv.evalName)
const { variants, timeout } = getExperimentSettings(evalName)
// agent-eval 1.3 dropped run-all/--dry: `run` takes explicit experiment names,
// and `status` is the read-only preview.
const agentEvalArgs = argv.dry
? ['status']
: ['run', ...VARIANTS.map((v) => v.suffix), '--force']
: ['run', ...variants.map((v) => v.suffix), '--force']
if (!fs.existsSync(path.join(ROOT, 'packages/next/dist'))) {
console.error(
......@@ -177,11 +246,11 @@ function main() {
} catch {}
}
writeExperiments(evalName)
writeExperiments(evalName, variants, timeout)
console.log(
evalName
? `> Running ${evalName} (baseline + agents-md)`
: '> Running all evals (baseline + agents-md)'
? `> Running ${evalName} (${variants.map((v) => v.suffix).join(' + ')})`
: `> Running all evals (${variants.map((v) => v.suffix).join(' + ')})`
)
// Same handoff pattern as run-tests.js with NEXT_TEST_PKG_PATHS. We invoke
......
......@@ -52,10 +52,14 @@ In both, the per-route success bar is the same: **dev loop reports no errors AND
`cacheComponents: true` requires every route to be prerenderable. A route that reads request-time data outside `<Suspense>` is "blocking" and fails the build. `export const instant = false` marks a route as allowed to block, which clears it in both dev and build; on a layout it covers the whole subtree during the build, but client navigations still validate each descendant segment on its own. Reads wrapped in a [`"use cache"`](https://nextjs.org/docs/app/api-reference/directives/use-cache) function count as cache boundaries, not blocking reads.
When a fix introduces `"use cache"`, follow the Caching guide for [choosing a data-level or UI-level boundary and setting its lifetime](https://nextjs.org/docs/app/getting-started/caching#usage). Use the [`use cache` cache-key reference](https://nextjs.org/docs/app/api-reference/directives/use-cache#cache-keys) when the result varies by arguments or captured values.
Three classes of blocker come up, usually in this order:
Whenever a fix introduces `<Suspense>`, follow the Streaming guide's [granular streaming pattern](https://nextjs.org/docs/app/guides/streaming#granular-streaming-with-suspense) and [guidance for preventing CLS](https://nextjs.org/docs/app/guides/streaming#cls-cumulative-layout-shift).
1. **Request-time reads** (`cookies()`, `headers()`, `await params`, `await searchParams`). All four block when awaited at the top of a page or layout. `params` and `searchParams` often get missed because they're not framed as "request data" the way cookies and headers are. The fix is to push the read into a `<Suspense>`-wrapped child — and for `params`/`searchParams`, forward the promise into the child and await it there; don't `await` at the page top.
2. **Sync-IO at module/render time** (`new Date()`, `Date.now()`, `Math.random()`, `crypto.randomUUID()`). These fail the build even with `instant = false` — the opt-out doesn't suppress them. If they're in a shared layout, they block every route under it. The codemod can't fix them; after running it, use the build errors to identify which calls to translate by hand (see the [incremental pre-step](#incremental)).
2. **Sync-IO at module/render time** (`new Date()`, `Date.now()`, `Math.random()`, `crypto.randomUUID()`). These fail the build even with `instant = false` — the opt-out doesn't suppress them. If they're in a shared layout, they block every route under it. The codemod can't fix them; after running it, use each build error and its linked documentation to identify which reported calls to translate by hand (see the [incremental pre-step](#incremental)).
3. **`"use cache"` files that read request data.** A file with a top-level `"use cache"` directive can't export `instant`; combining the two errors with `Only async functions are allowed to be exported in a "use cache" file.`, which means the directive was wrong for that route. Remove it before running the codemod.
## working surfaces
......@@ -134,13 +138,13 @@ Because the highest opt-out wins, remove them top-down (root layout first, then
Next, run `next build` to surface blockers the codemod could not handle. The build is the proof, not the codemod run — a shared layout that calls `new Date()` / `Math.random()` directly still fails regardless of the opt-out (see [background](#background)). If the normal build reports a sync-IO error without locating the call, rerun that route with `next build --debug-prerender --debug-build-paths="app/path/to/page.tsx"`. For each sync-IO error it reports:
1. **Sync-IO at module/render time.** Use the route, originating file and line, and `/docs/messages/` link in the build output to locate the error. If needed, grep the whole repo for `new Date()`, `Date.now()`, `Math.random()`, and `crypto.randomUUID()` (not only `app/**/layout.{js,jsx,ts,tsx}` — the read might live in any component imported by a layout). Do not change unreported matches. Unblock the reported call with the `await connection()` + `<Suspense>` fix from its `blocking-prerender-*` error card: it defers the value to request time, exactly as it behaved before the migration, so it needs no product decision. Add this exact comment on the line above the `await connection()`:
1. **Sync-IO at module/render time.** Use the route, originating file and line, and `/docs/messages/` link in the build output to locate the error. If needed, grep the whole repo for `new Date()`, `Date.now()`, `Math.random()`, and `crypto.randomUUID()` (not only `app/**/layout.{js,jsx,ts,tsx}` — the read might live in any component imported by a layout). Do not change unreported matches. Apply the appropriate option from the linked error page, then add this comment above a temporary boundary introduced only to unblock the build:
```tsx
// TODO: Cache Components adoption. Added to unblock the build: remove this connection() to re-trigger the error and review the fix options.
// TODO: Cache Components adoption. Added to unblock the build: remove this boundary to re-trigger the error and review the documented options.
```
It shares the `TODO: Cache Components adoption` prefix with the comments the codemod writes, so the check-in grep finds both. Removing the `await connection()` makes the error fire again with its fix cards — the same motion as removing an opt-out in the loop.
It shares the `TODO: Cache Components adoption` prefix with the comments the codemod writes, so the check-in grep finds both. Removing the boundary makes the error fire again with its fix cards — the same motion as removing an opt-out in the loop.
After each fix, rerun the scoped build when available, then run `next build` again to find the next blocker. Repeat until the normal build passes.
......@@ -209,7 +213,7 @@ Keep a todo list of the feature's routes. When every route in the feature is cle
Checklist before checking in with the user:
- `next build` completes without blocking-route errors.
- No bare TODOs in the feature: `grep -rn "TODO: Cache Components adoption"` finds both the codemod's opt-out comments and the sync-IO unblocks from the pre-step. Any `instant = false` left behind is a deliberate, documented Block — comment rewritten to a reason (see [references/per-page-decisions.md](./references/per-page-decisions.md) → "when to leave a Block in place"). Any `await connection()` left behind has been reviewed and kept on purpose, not left over from the pre-step.
- No bare TODOs in the feature: `grep -rn "TODO: Cache Components adoption"` finds both the codemod's opt-out comments and the sync-IO unblocks from the pre-step. Any `instant = false` left behind is a deliberate, documented Block — comment rewritten to a reason (see [references/per-page-decisions.md](./references/per-page-decisions.md) → "when to leave a Block in place"). Any `await io()` or `await connection()` left behind has been reviewed and kept on purpose, not left over from the pre-step.
- Each route visited in the browser: confirm the static shell renders first and every `<Suspense>` fallback resolves to its real content. Capture both states if you can — the fallback (mid-stream) and the final paint — so you have a streaming-experience demo to show the user. Throttle the network in the browser if streaming is too fast to observe.
- If runtime verification fails, reproduce the same route on the pre-adoption branch or with its opt-out restored. A failure that already exists is an environment or data problem, not an adoption regression.
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment