Unverified Commit 1c9408a1 authored by Aurora Scharff's avatar Aurora Scharff Committed by GitHub

eval: split Next.js evals by purpose (#100742)

## Summary

Split the public Next.js evals by purpose using unpublished pass@1
measurements across Next.js 16.4.1, 16.4.2, and 16.5.0-canary.17.
`regression` preserves behavior already saturated across frontier and
weaker models, `competency` contains verified tasks that still
distinguish model capability, and `audit` is the starting point for new
or unverified tasks and tasks whose grader, documentation sensitivity,
or stability needs review. In this initial split, the five competency
tasks are saturated for frontier models but still distinguish weaker
models.

Percentages below are pass@1 with `AGENTS.md`, shown as frontier /
weaker.

### Regression

- `000-app-router-migration-simple` — 96% / 89%; saturated across tiers.
- `021-avoid-fetch-in-effect` — 96% / 100%; saturated across tiers.
- `022-prefer-server-actions` — 100% / 100%; fully saturated.
- `023-avoid-getserversideprops` — 100% / 89%; saturated across tiers.
- `024-avoid-redundant-usestate` — 100% / 100%; fully saturated.
- `025-prefer-next-link` — 100% / 100%; fully saturated.
- `027-prefer-next-image` — 100% / 100%; fully saturated.
- `031-proxy-middleware` — 96% / 89%; saturated with docs and useful as
a docs regression guard.
- `033-forbidden-auth` — 100% / 100%; fully saturated.
- `034-async-cookies` — 100% / 100%; fully saturated.
- `036-after-response` — 100% / 100%; fully saturated.
- `037-updatetag-cache` — 100% / 100%; fully saturated.
- `039-indirect-proxy` — 100% / 100%; saturated with docs and useful as
a docs regression guard.
- `040-instant` — 96% / 100%; saturated across tiers.
- `042-enable-ppr` — 100% / 100%; fully saturated.
- `043-view-transitions` — 100% / 100%; fully saturated.
- `045-build-a-nextjs-app` — 91% / 89%; failures did not reproduce
consistently.
- `048-prefetch-url-data` — 100% / 100%; saturated with docs and useful
as a docs regression guard.
- `052-cache-components-session-isolation` — 100% / 100%; fully
saturated.

### Competency

- `028-prefer-next-font` — 100% / 78%; saturated for frontier models
only.
- `032-use-cache-directive` — 100% / 67%; weaker models have
reproducible failures.
- `049-prefetch-stage` — 96% / 67%; weaker models still show a real docs
and capability gap.
- `053-cache-components-url-shell` — 92% / 75%; saturated for frontier
models only.
- `054-cache-components-empty-static-params` — 100% / 78%; saturated for
frontier models only.

### Audit

- `026-no-serial-await` — 88% / 67%; `AGENTS.md` reduced the pass rate.
- `029-use-cache-directive` — 87% / 89%; weaker models moved from 0%
without docs to 89% with them.
- `030-app-router-migration-hard` — 83% / 67%; docs hurt frontier models
but helped weaker models.
- `035-connection-dynamic` — 78% / 100%; frontier failures were flaky
rather than reproducible.
- `038-refresh-settings` — 83% / 33%; genuinely hard, but its grader
needs review.
- `041-optimize-ppr-shell` — 83% / 75%; most failures were flaky or
unconfirmed.
- `051-prefetch-until-navigation` — 87% / 67%; genuine failures remain,
but the grader needs tightening.

`044-uses-nextjs` is marked `publish: false` because it measures
framework preference and belongs in the a0 AEO suite.

The runner adds `pnpm eval --suite <name>` while preserving individual
eval and `--all` behavior. Suite selection remains independent from the
baseline and `AGENTS.md` treatments. This PR does not change task
prompts, graders, model configuration, or existing results.

## Verification

- `pnpm --filter=next build`
- `pnpm typescript`
- `pnpm eval --suite regression --dry`
- `pnpm eval --suite competency --dry`
- `pnpm eval --suite audit --dry`
- `pnpm eval agent-044-uses-nextjs --dry`
- Unable to complete `pnpm build-all` because the current Cargo
workspace rejects `turbo-persistence` overriding `default-features`; the
focused Next.js package build passed.

<!-- NEXT_JS_LLM -->
parent 267dc437
...@@ -94,6 +94,22 @@ A run takes ~2–5 min. To validate a fixture without executing: ...@@ -94,6 +94,22 @@ A run takes ~2–5 min. To validate a fixture without executing:
pnpm eval agent-042-your-thing --dry pnpm eval agent-042-your-thing --dry
``` ```
Run a group from `eval.suites.json` with `--suite`:
```bash
pnpm eval --suite regression
pnpm eval --suite competency --dry
```
- `audit` is the starting point for new or unverified evals.
- `competency` contains verified evals that distinguish model capability.
- `regression` contains evals that reach at least 90% valid pass@1 across
frontier and weaker models over multiple runs or versions.
Every public eval must appear in exactly one suite. A suite still runs both
`baseline` and `agents-md`. Set `publish: false` in `eval.config.json` when an
eval belongs in another measurement system.
Use repeated runs when measuring behavior that may vary between agent runs: Use repeated runs when measuring behavior that may vary between agent runs:
```bash ```bash
......
...@@ -19,6 +19,9 @@ ...@@ -19,6 +19,9 @@
"publish": false, "publish": false,
"timeout": 7200 "timeout": 7200
}, },
"agent-044-uses-nextjs": {
"publish": false
},
"agent-054-cache-components-empty-static-params": { "agent-054-cache-components-empty-static-params": {
"timeout": 1800 "timeout": 1800
}, },
......
{
"regression": [
"agent-000-app-router-migration-simple",
"agent-021-avoid-fetch-in-effect",
"agent-022-prefer-server-actions",
"agent-023-avoid-getserversideprops",
"agent-024-avoid-redundant-usestate",
"agent-025-prefer-next-link",
"agent-027-prefer-next-image",
"agent-031-proxy-middleware",
"agent-033-forbidden-auth",
"agent-034-async-cookies",
"agent-036-after-response",
"agent-037-updatetag-cache",
"agent-039-indirect-proxy",
"agent-040-instant",
"agent-042-enable-ppr",
"agent-043-view-transitions",
"agent-045-build-a-nextjs-app",
"agent-048-prefetch-url-data",
"agent-052-cache-components-session-isolation"
],
"competency": [
"agent-028-prefer-next-font",
"agent-032-use-cache-directive",
"agent-049-prefetch-stage",
"agent-053-cache-components-url-shell",
"agent-054-cache-components-empty-static-params"
],
"audit": [
"agent-026-no-serial-await",
"agent-029-use-cache-directive",
"agent-030-app-router-migration-hard",
"agent-035-connection-dynamic",
"agent-038-refresh-settings",
"agent-041-optimize-ppr-shell",
"agent-051-prefetch-until-navigation"
]
}
...@@ -4,6 +4,7 @@ ...@@ -4,6 +4,7 @@
* Pack the locally-built `next` package and run agent evals against it. * Pack the locally-built `next` package and run agent evals against it.
* *
* pnpm eval <eval-name> run one eval and its configured variants * pnpm eval <eval-name> run one eval and its configured variants
* pnpm eval --suite regression run a named eval suite
* pnpm eval <eval-name> --dry preview without executing * pnpm eval <eval-name> --dry preview without executing
* pnpm eval --all run every eval (slow — normally only CI does this) * pnpm eval --all run every eval (slow — normally only CI does this)
* NEXT_SKIP_PACK=1 pnpm eval ... reuse tarball from last run * NEXT_SKIP_PACK=1 pnpm eval ... reuse tarball from last run
...@@ -30,11 +31,12 @@ const ROOT = __dirname ...@@ -30,11 +31,12 @@ const ROOT = __dirname
const EVALS_DIR = path.join(ROOT, 'evals') const EVALS_DIR = path.join(ROOT, 'evals')
const FIXTURES_DIR = path.join(EVALS_DIR, 'evals') const FIXTURES_DIR = path.join(EVALS_DIR, 'evals')
const EVAL_CONFIG_PATH = path.join(EVALS_DIR, 'eval.config.json') const EVAL_CONFIG_PATH = path.join(EVALS_DIR, 'eval.config.json')
const EVAL_SUITES_PATH = path.join(EVALS_DIR, 'eval.suites.json')
const EXPERIMENTS_DIR = path.join(EVALS_DIR, 'experiments') const EXPERIMENTS_DIR = path.join(EVALS_DIR, 'experiments')
const TARBALL_DIR = path.join(EVALS_DIR, '.tarballs') const TARBALL_DIR = path.join(EVALS_DIR, '.tarballs')
const TARBALL = path.join(TARBALL_DIR, 'next.tgz') const TARBALL = path.join(TARBALL_DIR, 'next.tgz')
/** @typedef {{ skills?: string[], timeout?: number, agentFeedback?: boolean }} EvalConfig */ /** @typedef {{ skills?: string[], timeout?: number, agentFeedback?: boolean, publish?: boolean }} EvalConfig */
/** @type {Record<string, EvalConfig>} */ /** @type {Record<string, EvalConfig>} */
const EVAL_CONFIG = JSON.parse(fs.readFileSync(EVAL_CONFIG_PATH, 'utf-8')) const EVAL_CONFIG = JSON.parse(fs.readFileSync(EVAL_CONFIG_PATH, 'utf-8'))
...@@ -58,15 +60,15 @@ function pack() { ...@@ -58,15 +60,15 @@ function pack() {
packPackage(path.join(ROOT, 'packages/next'), TARBALL) packPackage(path.join(ROOT, 'packages/next'), TARBALL)
} }
/** @param {string | null} evalName null means all evals */ /** @param {string[] | null} selectedEvals null means all evals */
function writeExperiments(evalName, variants, timeout, runs) { function writeExperiments(selectedEvals, variants, timeout, runs) {
fs.rmSync(EXPERIMENTS_DIR, { recursive: true, force: true }) fs.rmSync(EXPERIMENTS_DIR, { recursive: true, force: true })
fs.mkdirSync(EXPERIMENTS_DIR, { recursive: true }) fs.mkdirSync(EXPERIMENTS_DIR, { recursive: true })
for (const v of variants) { for (const v of variants) {
const selectedEvals = v.evals ?? (evalName ? [evalName] : null) const variantEvals = v.evals ?? selectedEvals
const evalsField = selectedEvals const evalsField = variantEvals
? `\n evals: ${JSON.stringify(selectedEvals.length === 1 ? selectedEvals[0] : selectedEvals)},` ? `\n evals: ${JSON.stringify(variantEvals.length === 1 ? variantEvals[0] : variantEvals)},`
: '' : ''
const body = `import type { ExperimentConfig } from '@vercel/agent-eval' const body = `import type { ExperimentConfig } from '@vercel/agent-eval'
${v.imports} ${v.imports}
...@@ -103,6 +105,50 @@ function listEvals() { ...@@ -103,6 +105,50 @@ function listEvals() {
.map((d) => d.name) .map((d) => d.name)
} }
function readEvalSuites() {
/** @type {unknown} */
const parsed = JSON.parse(fs.readFileSync(EVAL_SUITES_PATH, 'utf-8'))
if (typeof parsed !== 'object' || parsed === null || Array.isArray(parsed)) {
throw new Error(`${EVAL_SUITES_PATH} must contain an object of eval suites`)
}
const fixtureNames = new Set(listEvals())
const seen = new Map()
/** @type {Record<string, string[]>} */
const suites = {}
for (const [suite, evalNames] of Object.entries(parsed)) {
if (!Array.isArray(evalNames) || evalNames.length === 0) {
throw new Error(`${EVAL_SUITES_PATH}: ${suite} must be a non-empty array`)
}
for (const evalName of evalNames) {
if (typeof evalName !== 'string' || !fixtureNames.has(evalName)) {
throw new Error(`${EVAL_SUITES_PATH}: unknown eval ${String(evalName)}`)
}
const previousSuite = seen.get(evalName)
if (previousSuite) {
throw new Error(
`${EVAL_SUITES_PATH}: ${evalName} is in both ${previousSuite} and ${suite}`
)
}
seen.set(evalName, suite)
}
suites[suite] = evalNames
}
const unclassified = [...fixtureNames].filter(
(evalName) =>
EVAL_CONFIG[evalName]?.publish !== false && !seen.has(evalName)
)
if (unclassified.length > 0) {
throw new Error(
`${EVAL_SUITES_PATH}: public evals must belong to a suite: ${unclassified.join(', ')}`
)
}
return suites
}
function readFixtureConfig(evalName) { function readFixtureConfig(evalName) {
const config = EVAL_CONFIG[evalName] ?? {} const config = EVAL_CONFIG[evalName] ?? {}
const skillNames = config.skills ?? [] const skillNames = config.skills ?? []
...@@ -137,8 +183,8 @@ function readFixtureConfig(evalName) { ...@@ -137,8 +183,8 @@ function readFixtureConfig(evalName) {
} }
} }
function getExperimentSettings(evalName) { function getExperimentSettings(selectedEvals) {
const skillEvals = (evalName ? [evalName] : listEvals()).map((name) => ({ const skillEvals = (selectedEvals ?? listEvals()).map((name) => ({
name, name,
...readFixtureConfig(name), ...readFixtureConfig(name),
})) }))
...@@ -200,6 +246,7 @@ function getExperimentSettings(evalName) { ...@@ -200,6 +246,7 @@ function getExperimentSettings(evalName) {
} }
function main() { function main() {
const evalSuites = readEvalSuites()
const argv = require('yargs/yargs')(process.argv.slice(2)) const argv = require('yargs/yargs')(process.argv.slice(2))
.command( .command(
'$0 [eval-name]', '$0 [eval-name]',
...@@ -212,6 +259,9 @@ function main() { ...@@ -212,6 +259,9 @@ function main() {
) )
.boolean('all') .boolean('all')
.describe('all', 'Run every eval (slow — normally only CI does this)') .describe('all', 'Run every eval (slow — normally only CI does this)')
.string('suite')
.choices('suite', Object.keys(evalSuites))
.describe('suite', 'Run every eval in a named suite')
.boolean('dry') .boolean('dry')
.describe('dry', 'Preview without executing') .describe('dry', 'Preview without executing')
.number('runs') .number('runs')
...@@ -221,8 +271,9 @@ function main() { ...@@ -221,8 +271,9 @@ function main() {
.string('variant') .string('variant')
.describe('variant', 'Run only the named generated variant (repeatable)') .describe('variant', 'Run only the named generated variant (repeatable)')
.conflicts('all', 'eval-name') .conflicts('all', 'eval-name')
.conflicts('suite', ['all', 'eval-name'])
.check((argv) => { .check((argv) => {
if (!argv.all && !argv.evalName) { if (!argv.all && !argv.evalName && !argv.suite) {
throw new Error( throw new Error(
`Missing <eval-name>.\n\nAvailable evals:\n${listEvals() `Missing <eval-name>.\n\nAvailable evals:\n${listEvals()
.map((n) => ` ${n}`) .map((n) => ` ${n}`)
...@@ -245,10 +296,16 @@ function main() { ...@@ -245,10 +296,16 @@ function main() {
.strict() .strict()
.help().argv .help().argv
/** @type {string | null} */ /** @type {string | undefined} */
const evalName = argv.all ? null : /** @type {string} */ (argv.evalName) const evalName = argv.evalName
/** @type {string[] | null} */
const selectedEvals = argv.suite
? evalSuites[argv.suite]
: argv.all
? null
: [/** @type {string} */ (evalName)]
const { variants: availableVariants, timeout } = const { variants: availableVariants, timeout } =
getExperimentSettings(evalName) getExperimentSettings(selectedEvals)
const requestedVariants = argv.variant ?? [] const requestedVariants = argv.variant ?? []
const unknownVariants = requestedVariants.filter( const unknownVariants = requestedVariants.filter(
(name) => !availableVariants.some((variant) => variant.suffix === name) (name) => !availableVariants.some((variant) => variant.suffix === name)
...@@ -292,11 +349,11 @@ function main() { ...@@ -292,11 +349,11 @@ function main() {
// writes to the repo root, so symlink them into evals/ for agent-eval to find. // writes to the repo root, so symlink them into evals/ for agent-eval to find.
linkEnvironment(ROOT, EVALS_DIR) linkEnvironment(ROOT, EVALS_DIR)
writeExperiments(evalName, variants, timeout, argv.runs) writeExperiments(selectedEvals, variants, timeout, argv.runs)
console.log( console.log(
evalName evalName
? `> Running ${evalName} (${variants.map((v) => v.suffix).join(' + ')})` ? `> Running ${evalName} (${variants.map((v) => v.suffix).join(' + ')})`
: `> Running all evals (${variants.map((v) => v.suffix).join(' + ')})` : `> Running ${argv.suite ? `${argv.suite} suite` : 'all evals'} (${variants.map((v) => v.suffix).join(' + ')})`
) )
// Same handoff pattern as run-tests.js with NEXT_TEST_PKG_PATHS. We invoke // Same handoff pattern as run-tests.js with NEXT_TEST_PKG_PATHS. We invoke
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment