Unverified Commit f5f4ab6a authored by Jude Gao's avatar Jude Gao Committed by GitHub

evals: judge behavior instead of matching source text (#97826)

Makes a few of the evals judge what the code actually does rather than
how it happens to be written, so correct solutions stop failing on
style. One of them was already being vetoed by leftover assertions the
judge was meant to replace.

---------
Co-authored-by: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
parent fcf66c37
...@@ -2,14 +2,30 @@ ...@@ -2,14 +2,30 @@
* Use Cache Directive * Use Cache Directive
* *
* Generic behavior checks for this scenario: * Generic behavior checks for this scenario:
* - product reads use cache + cacheTag("products") * - product reads use `use cache` and are tagged with the products key
* - getAllProducts() from lib/db is used * - the catalog is sourced from getAllProducts() in lib/db
* - an inline Server Action flow exists and is form-triggered * - an inline Server Action flow exists and is form-triggered
* - revalidateTag("products", profile) is used * - the action revalidates that tag with a profile argument
* - updateTag is not used * - updateTag is not used
*
* The updateTag point is deliberate but is judged, not grepped: a source-text
* ban fails a correct solution that merely names the API in a comment
* explaining why it was rejected, which is exactly the reasoning we want. The prompt asks for the
* admin to keep working while the list is briefly stale and refreshes in the
* background, which is revalidateTag's stale-while-revalidate semantics.
* updateTag is the read-your-own-writes API and answers a different question;
* agent-037-updatetag-cache covers that one.
*
* The tag and data-source checks are judged rather than matched literally.
* They used to require the exact strings cacheTag('products') and
* revalidateTag('products', ...), so hoisting the key into a named constant
* failed, and they looked for the substring "lib/db" in an import, so a caching
* wrapper placed inside lib/ importing './db.js' relatively failed. Both are
* correct, and arguably better, implementations of the same requirement.
*/ */
import { expect, test } from 'vitest' import { expect, test } from 'vitest'
import { environment } from '@vercel/agent-eval/eval'
import { existsSync, readdirSync, readFileSync, statSync } from 'fs' import { existsSync, readdirSync, readFileSync, statSync } from 'fs'
import { join } from 'path' import { join } from 'path'
...@@ -57,22 +73,57 @@ function readSourceFiles(dir: string): SourceFile[] { ...@@ -57,22 +73,57 @@ function readSourceFiles(dir: string): SourceFile[] {
const sourceFiles = readSourceFiles(process.cwd()) const sourceFiles = readSourceFiles(process.cwd())
const source = sourceFiles.map((file) => file.content).join('\n') const source = sourceFiles.map((file) => file.content).join('\n')
function fileWith(pattern: RegExp): SourceFile | undefined { test('Catalog is cached, tagged, and revalidated by the sync action', async () => {
return sourceFiles.find((file) => pattern.test(file.content)) // Cheap and style-independent: the directive itself must be present somewhere.
}
test('Catalog reads use use-cache directive and products cache tag', () => {
// Allow caching logic to live in app or lib helper modules.
expect(source).toMatch(/['"]use cache['"];?/) expect(source).toMatch(/['"]use cache['"];?/)
// Tagged invalidation should target the required products key. // One judge call rather than two: the whole EVAL.ts run must finish inside
expect(source).toMatch(/cacheTag\s*\(\s*['"]products['"]\s*\)/) // 60s or vitest's worker RPC times out and fails the run even when every
}) // assertion passed. Two passes measured 76s; one measures well under.
await expect(environment).toSatisfyCriterion(
`Product catalog reads are cached and tagged, so that one tagged revalidation refreshes every view built on that data, and the "Sync latest catalog" Server Action invalidates them by revalidating that same tag with a revalidation profile. The catalog itself comes from the getAllProducts() helper the project already ships in lib/db rather than a reimplemented query.
Correct:
// lib/products.ts
export const PRODUCTS_TAG = 'products'
export async function getProducts() {
'use cache'
cacheTag(PRODUCTS_TAG)
return getAllProducts()
}
// the Server Action
async function syncCatalog() {
'use server'
await refreshFromErp()
revalidateTag(PRODUCTS_TAG, 'max')
}
Incorrect:
export async function getProducts() {
return getAllProducts() // nothing cached, nothing to revalidate
}
test('Page fetches products via lib/db', () => { export async function getProducts() {
// Keep data source expectation explicit without location assumptions. 'use cache' // cached but untagged, so the action has
expect(source).toMatch(/import.*getAllProducts.*lib\/db|from.*lib\/db/) return getAllProducts() // no way to invalidate it
expect(source).toMatch(/await\s+getAllProducts\s*\(|getAllProducts\s*\(/) }
revalidateTag(PRODUCTS_TAG) // no profile argument
revalidateTag('catalog', 'max') // not the tag the catalog read carries
updateTag(PRODUCTS_TAG) // read-your-own-writes; wrong API for this
// scenario, which wants the admin to carry
// on while the catalog refreshes behind them
Judge the code that runs. Naming updateTag in a comment to explain why it was not chosen is correct reasoning, not a violation.
Judge whether the caching, tagging and invalidation are wired to each other, not how they are spelled or which files they live in.
These APIs are newer than most training data. Docs for the exact Next.js version installed here ship at node_modules/next/dist/docs — see 01-app/03-api-reference/01-directives/use-cache.md, 01-app/03-api-reference/04-functions/cacheTag.md and 01-app/03-api-reference/04-functions/revalidateTag.md.`
)
}) })
test('Inline form-triggered Server Action flow exists', () => { test('Inline form-triggered Server Action flow exists', () => {
...@@ -90,22 +141,3 @@ test('Inline form-triggered Server Action flow exists', () => { ...@@ -90,22 +141,3 @@ test('Inline form-triggered Server Action flow exists', () => {
'Expected one file to contain form action={...} and inline Server Action markers' 'Expected one file to contain form action={...} and inline Server Action markers'
).toBeDefined() ).toBeDefined()
}) })
test('Server Action revalidates products using revalidateTag profile', () => {
const revalidateFile = fileWith(/revalidateTag\s*\(/)
expect(revalidateFile, 'Expected source to call revalidateTag').toBeDefined()
// The chosen API should be revalidateTag in this workflow.
expect(revalidateFile?.content ?? '').toMatch(
/import.*revalidateTag.*from\s+['"]next\/cache['"]/
)
expect(revalidateFile?.content ?? '').toMatch(/revalidateTag\s*\(/)
// Require the same explicit products tag and a profile/second argument.
expect(revalidateFile?.content ?? '').toMatch(
/revalidateTag\s*\(\s*['"]products['"]\s*,/
)
// Avoid read-your-own-writes invalidation API in this scenario.
expect(source).not.toMatch(/\bupdateTag\s*\(/)
})
...@@ -23,5 +23,5 @@ ...@@ -23,5 +23,5 @@
} }
}, },
"include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"], "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
"exclude": ["node_modules"] "exclude": ["node_modules", "EVAL.ts"]
} }
/** /**
* Proxy (formerly Middleware) * Proxy (formerly Middleware)
* *
* Tests whether the agent creates proxy.ts with a proxy() function (Next.js * Tests one thing: that the agent creates proxy.ts with a proxy() handler, the
* 16+ convention) instead of the deprecated middleware.ts/middleware(). * Next.js 16+ convention, rather than the deprecated middleware.ts/middleware().
* *
* Tricky because agents trained on pre-16 data create middleware.ts with a * Tricky because agents trained on pre-16 data reach for middleware.ts with a
* middleware() function — the file and function were both renamed in Next.js 16. * middleware() function. The file and the function were both renamed.
*
* Deliberately scoped to the rename. What the handler does with the request is
* not what this eval is measuring, so it is not asserted; adding checks for
* that only creates ways for a correct solution to fail on style.
*/ */
import { expect, test } from 'vitest' import { expect, test } from 'vitest'
import { readFileSync, existsSync } from 'fs' import { readFileSync, existsSync } from 'fs'
import { join } from 'path' import { join } from 'path'
test('proxy.ts file exists in root (Next.js 16+ convention)', () => { // The docs place the file at the project root or in src/, in .ts or .js.
const proxyPath = join(process.cwd(), 'proxy.ts') const ROOTS = [process.cwd(), join(process.cwd(), 'src')]
const middlewarePath = join(process.cwd(), 'middleware.ts') const EXTS = ['ts', 'js']
// In Next.js 16+, the file should be named proxy.ts, not middleware.ts
// middleware.ts is deprecated
const hasProxy = existsSync(proxyPath)
const hasMiddleware = existsSync(middlewarePath)
// Must have proxy.ts
expect(hasProxy).toBe(true)
// Should NOT have middleware.ts (deprecated)
expect(hasMiddleware).toBe(false)
})
test('Proxy function uses correct name (not middleware)', () => {
const proxyPath = join(process.cwd(), 'proxy.ts')
if (existsSync(proxyPath)) {
const content = readFileSync(proxyPath, 'utf-8')
// In Next.js 16+, the function should be named 'proxy', not 'middleware'
// Should export proxy function
expect(content).toMatch(
/export\s+(async\s+)?(default\s+)?function\s+proxy|export\s+default\s+async\s+function\s+proxy/
)
// Should NOT have a function named 'middleware' function locate(base: string): string | undefined {
expect(content).not.toMatch(/export\s+(default\s+)?function\s+middleware/) for (const dir of ROOTS) {
for (const ext of EXTS) {
const candidate = join(dir, `${base}.${ext}`)
if (existsSync(candidate)) return candidate
}
} }
}
/** Any export form binding `name`: declaration, const/let/var, default, or list. */
function exportsBinding(source: string, name: string): boolean {
return (
new RegExp(
`export\\s+(default\\s+)?(async\\s+)?function\\s+${name}\\b`
).test(source) ||
new RegExp(`export\\s+(const|let|var)\\s+${name}\\b`).test(source) ||
new RegExp(`export\\s+default\\s+${name}\\b`).test(source) ||
new RegExp(`export\\s*\\{[^}]*\\b${name}\\b[^}]*\\}`).test(source)
)
}
test('creates proxy, not the deprecated middleware', () => {
expect(locate('proxy'), 'expected a proxy file (Next.js 16+)').toBeDefined()
expect(
locate('middleware'),
'middleware is deprecated in Next.js 16+; expected proxy instead'
).toBeUndefined()
}) })
test('Proxy imports NextResponse from next/server', () => { test('the handler is named proxy, not middleware', () => {
const proxyPath = join(process.cwd(), 'proxy.ts') const path = locate('proxy')
if (existsSync(proxyPath)) { expect(path, 'no proxy file to inspect').toBeDefined()
const content = readFileSync(proxyPath, 'utf-8') const source = readFileSync(path!, 'utf-8')
// Should import NextResponse from next/server // Any export form is fine — a typed arrow const is as valid as a declaration.
expect(content).toMatch(/import.*NextResponse.*from\s+['"]next\/server['"]/) // This is what stops an empty proxy file from satisfying the eval.
} expect(
}) exportsBinding(source, 'proxy'),
'expected the proxy file to export a handler named `proxy`'
test('Proxy adds custom header X-Request-Id', () => { ).toBe(true)
const proxyPath = join(process.cwd(), 'proxy.ts') expect(
if (existsSync(proxyPath)) { exportsBinding(source, 'middleware'),
const content = readFileSync(proxyPath, 'utf-8') 'the Next.js 16+ handler is named `proxy`, not `middleware`'
).toBe(false)
// Should use NextResponse.next()
expect(content).toMatch(/NextResponse\.next\(\)/)
// Should set X-Request-Id header
expect(content).toMatch(/['"]X-Request-Id['"]/i)
// Should return response
expect(content).toMatch(/return\s+/)
}
})
test('Proxy logs request pathname', () => {
const proxyPath = join(process.cwd(), 'proxy.ts')
if (existsSync(proxyPath)) {
const content = readFileSync(proxyPath, 'utf-8')
// Should log pathname
expect(content).toMatch(/console\.log.*pathname|pathname.*console\.log/)
}
}) })
...@@ -12,6 +12,16 @@ ...@@ -12,6 +12,16 @@
* the PPR shell requires replacing it with per-section Suspense boundaries * the PPR shell requires replacing it with per-section Suspense boundaries
* so each section can stream independently. * so each section can stream independently.
* *
* This eval is judged semantically end to end. It previously also grepped
* app/page.tsx for >=3 literal <Suspense> tags and for each section sitting in
* its own block in that file. Those two assertions contradicted the judge: a
* model that co-locates each boundary inside the section component builds
* fine, yields a correctly partially prerendered route, and was passed by the
* judge, yet failed the greps purely because the tags were not typed in
* page.tsx. They vetoed the judge they were meant to be replaced by, so they
* are gone; the granularity requirement they encoded now lives in the
* criterion below.
*
* The does-Page-block-on-data check is semantic, so it uses the agentic LLM * The does-Page-block-on-data check is semantic, so it uses the agentic LLM
* judge rather than regex. The old /getDashboardData\s*\(/ whole-file ban * judge rather than regex. The old /getDashboardData\s*\(/ whole-file ban
* rejected functionally correct streaming — e.g. async section components * rejected functionally correct streaming — e.g. async section components
...@@ -22,43 +32,11 @@ ...@@ -22,43 +32,11 @@
*/ */
import { expect, test } from 'vitest' import { expect, test } from 'vitest'
import { readFileSync } from 'fs'
import { join } from 'path'
import { environment } from '@vercel/agent-eval/eval' import { environment } from '@vercel/agent-eval/eval'
const appDir = join(process.cwd(), 'app')
function readFile(name: string): string {
return readFileSync(join(appDir, name), 'utf-8')
}
test('Page has at least 3 Suspense boundaries', () => {
const page = readFile('page.tsx')
const suspenseCount = (page.match(/<Suspense[\s>]/g) || []).length
expect(suspenseCount).toBeGreaterThanOrEqual(3)
})
test('Each dashboard section has its own Suspense boundary in page.tsx', () => {
const page = readFile('page.tsx')
// Split page into Suspense blocks: text between each <Suspense and </Suspense>
const suspenseBlocks = page.split(/<Suspense[\s>]/).slice(1)
const components = ['CardStats', 'RevenueChart', 'LatestInvoices']
for (const component of components) {
const inOwnBlock = suspenseBlocks.some(
(block) => block.includes(component) && block.includes('</Suspense>')
)
expect(inOwnBlock, `${component} should be inside its own <Suspense>`).toBe(
true
)
}
})
test('Page does not await all data before rendering', async () => { test('Page does not await all data before rendering', async () => {
await expect(environment).toSatisfyCriterion( await expect(environment).toSatisfyCriterion(
`The dashboard page must produce a static PPR shell: the default-exported Page component in app/page.tsx returns its JSX frame without blocking on dashboard data, and the data-driven sections stream in under <Suspense> boundaries. `The dashboard page must produce a static PPR shell: the default-exported Page component in app/page.tsx returns its JSX frame without blocking on dashboard data, and the data-driven sections suspend independently of one another rather than collapsing into a single all-or-nothing loading state. Judge where the boundaries sit in the rendered tree, not how many there are or which file the <Suspense> tag is written in. Docs for the exact Next.js version installed here ship at node_modules/next/dist/docs — see 01-app/03-api-reference/05-config/01-next-config-js/cacheComponents.md and 01-app/03-api-reference/03-file-conventions/loading.md.
For reference, one correct solution keeps Page synchronous and moves each await into a Suspense-wrapped child: For reference, one correct solution keeps Page synchronous and moves each await into a Suspense-wrapped child:
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment