Unverified Commit 433fe0ed authored by Aurora Scharff's avatar Aurora Scharff Committed by GitHub

eval: resolve and classify the audit suite (#100748)

The October pass@1 study left seven tasks in `audit` because their
graders, framework contracts, or failures were not trustworthy enough to
classify. This resolves every task: three repaired tasks move to
regression, two valid competency tasks remain discriminatory, and two
invalid public measurements are unpublished.

## Classification

| Task | Before | Change and evidence | Final |
| --- | --- | --- | --- |
| `026-no-serial-await` | Generic JavaScript task with solution leakage
and source-shape grading | Current score cannot support a Next.js
competency claim | Unpublished; removed from runnable suites |
| `029-use-cache-directive` | Missing Cache Components setup and
incomplete flow verification | Requires Cache Components and verifies
the cached read, form-bound action, shared tag, and two-argument
revalidation as one flow. Gemini failed both scored treatments;
confirmations were mixed | Competency |
| `030-app-router-migration-hard` | Broad migration with incomplete
coverage and unstable judges | Current score cannot distinguish
migration ability from grader gaps | Unpublished; held out until split
or substantially repaired |
| `035-connection-dynamic` | Accepted `io()` for a real-request
requirement and used three variable judge checks | Requires
`connection()`, preserves the stable frame, and consolidates the
request-specific behavior check. All 8 valid treatments passed |
Regression |
| `038-refresh-settings` | Grader did not connect the mutation, UI, and
`refresh()` | Ties the complete Server Action flow together and rejects
redirects. All 9 valid calibration results passed | Regression |
| `041-optimize-ppr-shell` | Vague prompt and one
implementation-oriented judge | Names the visible shell and
independently revealing sections, keeps Cache Components enabled, and
grades the rendered component tree. All 8 valid treatments passed |
Regression |
| `051-prefetch-until-navigation` | Broad source checks around
navigation-stage prefetching | Scopes the title prefetch and deferred
details to the destination. Frontier passed; Luna Base failed its scored
attempt and both confirmations, while Luna AGENTS.md passed | Competency
|

The final suite contains 22 regression tasks and 7 competency tasks. No
task remains in `audit`. `044-uses-nextjs` remains outside these suites
because it measures framework preference for a0.

## Repaired-task results

Pass@1 against the exact `16.5.0-canary.27` package:

| Model | Treatment | `029` | `035` | `041` |
| --- | --- | ---: | ---: | ---: |
| Claude Opus 5.5 | Base | 1/1 | 1/1 | 1/1 |
| Claude Opus 5.5 | AGENTS.md | 1/1 | 1/1 | 1/1 |
| GPT-6 Sol | Base | 1/1 | 1/1 | 1/1 |
| GPT-6 Sol | AGENTS.md | 1/1 | 1/1 | 1/1 |
| Gemini 4 Argon | Base | 0/1 | 1/1 | 1/1 |
| Gemini 4 Argon | AGENTS.md | 0/1 | 1/1 | 1/1 |
| GPT-6 Luna | Base | 1/1 | 1/1 | 1/1 |
| GPT-6 Luna | AGENTS.md | discarded | 1/1 | 1/1 |

The discarded Luna `029` attempt timed out without a transcript and is
not a model result. Diagnostic reruns do not change pass@1: Gemini Base
passed one of two confirmations and repeated the same substantive
failure once; Gemini AGENTS.md passed both confirmations. The two scored
Gemini cells remain failed and should be reported as unstable.

## Validation

- `pnpm prettier ... --check` on all changed files
- `npx eslint --config eslint.config.mjs` on all changed graders
- `pnpm eval --suite competency --dry`
- `pnpm eval --suite regression --dry`
- `git diff --check`

Stacked on #100742, which adds suite selection and the classification
protocol.

<!-- NEXT_JS_LLM -->
parent a2fb5e3b
......@@ -106,9 +106,10 @@ pnpm eval --suite competency --dry
- `regression` contains evals that reach at least 90% valid pass@1 across
frontier and weaker models over multiple runs or versions.
Every public eval must appear in exactly one suite. A suite still runs both
`baseline` and `agents-md`. Set `publish: false` in `eval.config.json` when an
eval belongs in another measurement system.
Every eval under active measurement must appear in exactly one suite. A suite
still runs both `baseline` and `agents-md`. Set `publish: false` in
`eval.config.json` while an eval is being audited, when its results should stay
internal, or when it belongs in another measurement system.
Use repeated runs when measuring behavior that may vary between agent runs:
......
{
"agent-026-no-serial-await": {
"publish": false
},
"agent-030-app-router-migration-hard": {
"publish": false
},
"agent-046-adopt-partial-prefetching": {
"skills": ["next-partial-prefetching-adoption"],
"publish": false,
......
......@@ -10,10 +10,13 @@
"agent-031-proxy-middleware",
"agent-033-forbidden-auth",
"agent-034-async-cookies",
"agent-035-connection-dynamic",
"agent-036-after-response",
"agent-037-updatetag-cache",
"agent-038-refresh-settings",
"agent-039-indirect-proxy",
"agent-040-instant",
"agent-041-optimize-ppr-shell",
"agent-042-enable-ppr",
"agent-043-view-transitions",
"agent-045-build-a-nextjs-app",
......@@ -22,18 +25,11 @@
],
"competency": [
"agent-028-prefer-next-font",
"agent-029-use-cache-directive",
"agent-032-use-cache-directive",
"agent-049-prefetch-stage",
"agent-051-prefetch-until-navigation",
"agent-053-cache-components-url-shell",
"agent-054-cache-components-empty-static-params"
],
"audit": [
"agent-026-no-serial-await",
"agent-029-use-cache-directive",
"agent-030-app-router-migration-hard",
"agent-035-connection-dynamic",
"agent-038-refresh-settings",
"agent-041-optimize-ppr-shell",
"agent-051-prefetch-until-navigation"
]
}
......@@ -72,6 +72,11 @@ function readSourceFiles(dir: string): SourceFile[] {
const sourceFiles = readSourceFiles(process.cwd())
const source = sourceFiles.map((file) => file.content).join('\n')
const config = readFileSync(join(process.cwd(), 'next.config.ts'), 'utf-8')
test('enables Cache Components', () => {
expect(config).toMatch(/cacheComponents\s*:\s*true/)
})
test('Catalog is cached, tagged, and revalidated by the sync action', async () => {
// Cheap and style-independent: the directive itself must be present somewhere.
......@@ -81,7 +86,7 @@ test('Catalog is cached, tagged, and revalidated by the sync action', async () =
// 60s or vitest's worker RPC times out and fails the run even when every
// assertion passed. Two passes measured 76s; one measures well under.
await expect(environment).toSatisfyCriterion(
`Product catalog reads are cached and tagged, so that one tagged revalidation refreshes every view built on that data, and the "Sync latest catalog" Server Action invalidates them by revalidating that same tag with a revalidation profile. The catalog itself comes from the getAllProducts() helper the project already ships in lib/db rather than a reimplemented query.
`Product catalog reads are cached and tagged, so that one tagged revalidation refreshes every view built on that data. The actual inline Server Action bound to the "Sync latest catalog" form must invalidate the same tag with a revalidation profile. The catalog itself comes from the getAllProducts() helper the project already ships in lib/db rather than a reimplemented query.
Correct:
......@@ -120,7 +125,7 @@ Incorrect:
Judge the code that runs. Naming updateTag in a comment to explain why it was not chosen is correct reasoning, not a violation.
Judge whether the caching, tagging and invalidation are wired to each other, not how they are spelled or which files they live in.
Judge whether the caching, tagging, form action, and invalidation are wired to each other, not how they are spelled or which files they live in. Reject an unused Server Action, a form bound to a different function, or a revalidation call that is not reached by submitting that form.
These APIs are newer than most training data. Docs for the exact Next.js version installed here ship at node_modules/next/dist/docs — see 01-app/03-api-reference/01-directives/use-cache.md, 01-app/03-api-reference/04-functions/cacheTag.md and 01-app/03-api-reference/04-functions/revalidateTag.md.`
)
......
You are building an admin product catalog page for an e-commerce team.
The catalog is read-heavy and should feel fast for day-to-day browsing, so avoid re-querying product data on every request.
The app should use Next.js Cache Components. The catalog is read-heavy and should feel fast for day-to-day browsing, so avoid re-querying product data on every request. Configure the project for this behavior.
Admins can trigger a "Sync latest catalog" action from the page when upstream ERP/PIM data changes (pricing, inventory, availability). After submitting, they should be able to continue working immediately, even if the product list is briefly stale.
......
......@@ -27,24 +27,12 @@ test('enables Cache Components without opt-outs', () => {
})
test('uses an explicit synchronous I/O boundary', () => {
expect(source).toMatch(/\b(?:io|connection)\s*\(/)
expect(source).toMatch(/\bconnection\s*\(/)
expect(source).toMatch(/<Suspense[\s>]/)
})
test('keeps the operations frame useful while metadata loads', async () => {
test('keeps the frame useful and metadata request-specific', async () => {
await expect(environment).toSatisfyCriterion(
`The stable Operations navigation and page frame must remain outside the Suspense boundary used for request metadata. Accept equivalent component and file organization.`
)
})
test('keeps request metadata fresh', async () => {
await expect(environment).toSatisfyCriterion(
`The request identifier, current time, and sampling value must remain request-specific. Render them in a focused async child that awaits io() or connection() before calling randomUUID, new Date or Date.now, and Math.random. Prefer io() unless the implementation specifically needs to wait for a real user request. Accept equivalent component and file organization.`
)
})
test('does not reuse request metadata', async () => {
await expect(environment).toSatisfyCriterion(
`The request identifier, current time, and sampling value must not be cached, evaluated at module scope, or replaced with static values. Reject any solution that can reuse those values across requests. Accept equivalent component and file organization.`
`The stable Operations navigation and page frame remain outside the Suspense boundary used for request metadata. Inside that focused async boundary, the implementation awaits connection() before generating the request identifier, current time, and sampling value with randomUUID, new Date or Date.now, and Math.random. None of those values are cached, evaluated at module scope, or replaced with static values, so they are generated for each real request rather than during prerendering or prefetching. Accept equivalent component and file organization.`
)
})
Enable Cache Components for this operations console. The navigation must stay immediately useful, while the request identifier, current time, and sampling value must still be generated per request. Finish with a production build and do not leave routes opted out.
Enable Cache Components for this operations console. The navigation must stay immediately useful, while the request identifier, current time, and sampling value must still be generated for each real request rather than during prerendering or prefetching. Finish with a production build and do not leave routes opted out.
/**
* Refresh Page via revalidatePath
*
* Tests whether the agent uses revalidatePath() from next/cache inside a
* Server Action to refresh the current page after a mutation.
*
* Tricky because agents use redirect() to the same page (loses scroll/state),
* return data for manual refresh, or use client-side router.refresh().
*/
/** Refresh the current page after a Server Action mutation. */
import { expect, test } from 'vitest'
import { readFileSync, readdirSync } from 'fs'
import { join } from 'path'
import { environment } from '@vercel/agent-eval/eval'
// Helper to find all .ts and .tsx files
function findAllTsFiles(dir: string): string[] {
const files: string[] = []
try {
const items = readdirSync(dir, { withFileTypes: true })
for (const item of items) {
const fullPath = join(dir, item.name)
if (
item.isDirectory() &&
item.name !== 'node_modules' &&
item.name !== '.next'
) {
files.push(...findAllTsFiles(fullPath))
} else if (
item.isFile() &&
(item.name.endsWith('.ts') || item.name.endsWith('.tsx'))
) {
files.push(fullPath)
}
}
} catch {
// Ignore directories that can't be read
}
return files
}
test('toggles the preference and refreshes from the Server Action', async () => {
await expect(environment).toSatisfyCriterion(
`The settings page displays the current notification preference and provides a form or control wired to a Server Action that toggles that same stored preference. The mutation must change the value read by the page so the refreshed UI shows the new state.
test('Server Action imports refresh from next/cache', () => {
const allFiles = findAllTsFiles(process.cwd())
let foundRefreshImport = false
for (const file of allFiles) {
const content = readFileSync(file, 'utf-8')
if (content.match(/import.*refresh.*from\s+['"]next\/cache['"]/)) {
foundRefreshImport = true
break
}
}
expect(foundRefreshImport).toBe(true)
})
test('Server Action uses "use server" directive', () => {
const allFiles = findAllTsFiles(process.cwd())
let foundServerAction = false
for (const file of allFiles) {
const content = readFileSync(file, 'utf-8')
if (content.includes("'use server'") || content.includes('"use server"')) {
foundServerAction = true
break
}
}
expect(foundServerAction).toBe(true)
})
test('Server Action calls refresh() for page refresh', () => {
const allFiles = findAllTsFiles(process.cwd())
let foundRefreshCall = false
for (const file of allFiles) {
const content = readFileSync(file, 'utf-8')
// Must have "use server" and refresh() call
if (
(content.includes("'use server'") || content.includes('"use server"')) &&
content.match(/refresh\s*\(\s*\)/)
) {
foundRefreshCall = true
break
}
}
expect(foundRefreshCall).toBe(true)
})
test('Does NOT use redirect for same-page update', () => {
const allFiles = findAllTsFiles(process.cwd())
for (const file of allFiles) {
const content = readFileSync(file, 'utf-8')
if (content.includes("'use server'") || content.includes('"use server"')) {
// If it has refresh(), good - that's what we want
if (content.match(/refresh\s*\(\s*\)/)) {
// Should use refresh, not redirect for same-page updates
expect(content).toMatch(/refresh\s*\(\s*\)/)
}
}
}
})
test('Does NOT use router.refresh() from client (should use server-side refresh)', () => {
const allFiles = findAllTsFiles(process.cwd())
let usesServerRefresh = false
for (const file of allFiles) {
const content = readFileSync(file, 'utf-8')
// Check for server-side refresh import
if (content.match(/import.*refresh.*from\s+['"]next\/cache['"]/)) {
usesServerRefresh = true
}
}
// Should use server-side refresh from next/cache
expect(usesServerRefresh).toBe(true)
})
test('Server Action has notification/toggle logic', () => {
const allFiles = findAllTsFiles(process.cwd())
let hasToggleLogic = false
for (const file of allFiles) {
const content = readFileSync(file, 'utf-8')
if (
(content.includes("'use server'") || content.includes('"use server"')) &&
content.match(/notification|toggle|preference|setting/i)
) {
hasToggleLogic = true
break
}
}
expect(hasToggleLogic).toBe(true)
After the mutation succeeds, the Server Action must call refresh() imported from next/cache. The call must execute inside a real Server Action marked with the use server directive and happen after the mutation. Fail placeholder actions, no-op mutations, client-only state, or unrelated read and write values. Also fail redirect(), revalidatePath(), returning a flag for a Client Component, or client-side router.refresh() when used instead of the Server Action refresh(). The interaction must stay on the current page. Accept equivalent storage, component, and file organization.`
)
})
export default function RootLayout({
children,
}: Readonly<{
children: React.ReactNode
}>) {
return (
<html lang="en">
<body>{children}</body>
</html>
)
}
......@@ -23,5 +23,5 @@
}
},
"include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
"exclude": ["node_modules"]
"exclude": ["node_modules", "EVAL.ts"]
}
......@@ -33,10 +33,18 @@
import { expect, test } from 'vitest'
import { environment } from '@vercel/agent-eval/eval'
import { readFileSync } from 'fs'
import { join } from 'path'
test('Page does not await all data before rendering', async () => {
const config = readFileSync(join(process.cwd(), 'next.config.ts'), 'utf8')
test('keeps Cache Components enabled', () => {
expect(config).toMatch(/cacheComponents\s*:\s*true/)
})
test('renders the dashboard shell while sections stream independently', async () => {
await expect(environment).toSatisfyCriterion(
`The dashboard page must produce a static PPR shell: the default-exported Page component in app/page.tsx returns its JSX frame without blocking on dashboard data, and the data-driven sections suspend independently of one another rather than collapsing into a single all-or-nothing loading state. Judge where the boundaries sit in the rendered tree, not how many there are or which file the <Suspense> tag is written in. Docs for the exact Next.js version installed here ship at node_modules/next/dist/docs — see 01-app/03-api-reference/05-config/01-next-config-js/cacheComponents.md and 01-app/03-api-reference/03-file-conventions/loading.md.
`The dashboard produces a useful static shell: the Dashboard heading renders without waiting for dashboard data, while the summary cards, Revenue section, and Latest Invoices section suspend and reveal independently with section-appropriate fallbacks rather than collapsing into one all-or-nothing loading state. Judge the rendered component tree, not how many <Suspense> strings appear in one file. Docs for the exact Next.js version installed here ship at node_modules/next/dist/docs — see 01-app/03-api-reference/05-config/01-next-config-js/cacheComponents.md and 01-app/03-api-reference/03-file-conventions/loading.md.
For reference, one correct solution keeps Page synchronous and moves each await into a Suspense-wrapped child:
......
Optimize the partial pre-rendering shell for this app.
Make the dashboard useful immediately while its data loads. The Dashboard heading should appear in the initial shell, and the summary cards, revenue chart, and latest invoices should reveal independently as each section becomes ready instead of hiding the entire page behind one loading screen.
Keep Cache Components enabled and finish with a production build.
/**
* Prefetch until navigation
*
* Verifies that the agent uses Partial Prefetching to include a selected
* project's title in a per-link prefetch while deferring the rest of the
* project page until navigation.
*/
/** Include project titles in selected prefetches while deferring details. */
import { expect, test } from 'vitest'
import { existsSync, readFileSync, readdirSync, statSync } from 'fs'
import { join } from 'path'
import { environment } from '@vercel/agent-eval/eval'
const IGNORE_DIRS = new Set([
'.git',
'.next',
'node_modules',
'dist',
'build',
'coverage',
])
const IGNORE_FILES = new Set(['EVAL.ts', 'PROMPT.md'])
function stripComments(source: string): string {
return source.replace(/\/\*[\s\S]*?\*\//g, '').replace(/\/\/.*$/gm, '')
}
function readSourceFiles(dir: string): string[] {
if (!existsSync(dir)) return []
const files: string[] = []
for (const entry of readdirSync(dir)) {
if (IGNORE_DIRS.has(entry)) continue
const fullPath = join(dir, entry)
const stats = statSync(fullPath)
if (stats.isDirectory()) {
files.push(...readSourceFiles(fullPath))
} else if (!IGNORE_FILES.has(entry) && /\.(ts|tsx|js|jsx)$/.test(entry)) {
files.push(stripComments(readFileSync(fullPath, 'utf-8')))
}
}
return files
}
const source = readSourceFiles(process.cwd()).join('\n---FILE---\n')
const config = stripComments(
readFileSync(join(process.cwd(), 'next.config.ts'), 'utf-8')
)
test('enables Partial Prefetching globally or for the project route', () => {
expect(
/partialPrefetching\s*:\s*true/.test(config) ||
/export\s+const\s+prefetch\s*=\s*['"]partial['"]/.test(source)
).toBe(true)
})
test('opts the project link into per-link prefetching', () => {
expect(source).toMatch(/<Link\b[^>]*\bprefetch(?:\s*=\s*\{true\})?(?:\s|>)/)
})
test('uses the navigation cache stage', () => {
const navigationImport = source.match(
/import\s*\{[^}]*\bnavigation\b(?:\s+as\s+([A-Za-z_$][\w$]*))?[^}]*\}\s*from\s*['"]next\/cache['"]/
)
expect(navigationImport).not.toBeNull()
const localName = navigationImport?.[1] ?? 'navigation'
expect(source).toMatch(new RegExp(`await\\s+${localName}\\s*\\(`))
})
test('includes the project title in the selected prefetch', async () => {
test('includes selected project titles in their prefetches', async () => {
await expect(environment).toSatisfyCriterion(
`The selected project's title must be available before the click from the dashboard's per-link prefetch. Keep the params-dependent title inside Suspense so the reusable App Shell can show a fallback while the selected prefetch resolves the title. The title must render before the nested boundary that waits for navigation.
`Partial Prefetching must be enabled globally or for the /projects/[slug] route. The dashboard Links that navigate to project pages must explicitly opt into per-link prefetching with prefetch={true} or the equivalent bare prefetch prop.
Accept equivalent component and file organization. Reject solutions that leave the title behind the reusable shell fallback, put it behind navigation(), disable prefetching, or prefetch the entire page.`
The selected project's cached title must be available from that per-link prefetch and render before the project details. Keep the params-dependent title inside a meaningful Suspense boundary so the reusable App Shell can show a fallback while the selected prefetch resolves it. Fail an unrelated prefetched Link, disabling prefetch, changing the default for unrelated navigation, leaving the title behind navigation(), fetching every project's details during prefetch, or moving the whole page into the reusable shell. Accept equivalent component and file organization.`
)
})
test('waits until navigation for project details', async () => {
await expect(environment).toSatisfyCriterion(
`Project activity, deployments, description, and other detail content must not join the prefetch. Render those details inside a nested Suspense boundary through an async component that awaits navigation() before loading or rendering them.
`Project description, activity, deployments, and other detail content must stay outside the prefetch. Render the details through an async component inside a nested Suspense boundary. That component must await navigation() from next/cache before calling the existing cached project-details function or rendering its result.
The navigation() call must remain outside every use cache scope; cached data functions may be called after it. Reject solutions that fetch project details before reaching navigation(), replace it with connection() or an uncached fetch, or prefetch the entire page. Accept equivalent component and file organization.`
The navigation() call must stay outside every use cache scope. Fail solutions that load project details before navigation(), replace navigation() with connection() or an uncached fetch, remove the details cache, or prefetch the entire page. Accept equivalent component and file organization.`
)
})
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment