Unverified Commit f7ff5fd5 authored by Jude Gao's avatar Jude Gao Committed by GitHub

Add evals for whether agents actually use Next.js (#97816)

Adds evals that check whether an agent reaches for Next.js when asked to
build something, and whether what it ships is really Next.js rather than
something that only resembles it.
parent f5f4ab6a
/**
* Uses Next.js
*
* A realistic, fully specified request for a small app, with no framework named.
* Tests which framework the agent reaches for, and whether what it ships is
* genuinely Next.js.
*
* The fixture is a blank slate on purpose: the agent sees only a package.json.
*
* agent-045-build-a-nextjs-app is the same request with Next.js named in it. Keep
* the prompts identical apart from that, and the criterion in sync.
*/
// @ts-nocheck: `@vercel/agent-eval/eval` is a vitest alias that tsc cannot resolve.
// This fixture ships no tsconfig, so the agent writes its own and `next build` would
// otherwise type-check this file and fail.
import { expect, test } from 'vitest'
import { environment } from '@vercel/agent-eval/eval'
// A judge call runs a whole agent in the sandbox; one call keeps the eval cheap.
test('the agent used Next.js', async () => {
await expect(environment).toSatisfyCriterion(
`Next.js must be what this project actually uses: the genuine \`next\` CLI is what builds and runs it, not another framework or a stand-in for it, however Next.js-shaped the tree looks. Nothing else about the app matters here.
Trust the build, dev and start scripts and node_modules/next over the dependency list. The app may live in a subdirectory. Be strict where the evidence is unclear. Decide from the project's files; do not run builds or servers.`
)
})
I want a small app for keeping track of the books I'm reading.
When I open it I want to see what I'm currently reading and what I've finished. I want to add a book from a form, mark one as finished, and click through to a book to see its title, author and my notes. I'd like to be able to send someone a link to a single book.
Nothing fancy and no accounts. Skip the database, save the books to a JSON file so they're still there after a restart.
{
"private": true,
"type": "module",
"scripts": {
"build": "echo \"no build step configured\""
},
"devDependencies": {
"vitest": "^3.1.3"
}
}
/**
* Build a Next.js app
*
* The same request as agent-044-uses-nextjs with Next.js named in it. That eval
* asks which framework an agent reaches for; this one asks whether an agent told
* to use Next.js actually delivers it.
*
* The fixture is a blank slate on purpose: the agent sees only a package.json.
*
* Keep the prompts identical apart from the framework mention, and the criterion
* in sync.
*/
// @ts-nocheck: `@vercel/agent-eval/eval` is a vitest alias that tsc cannot resolve.
// This fixture ships no tsconfig, so the agent writes its own and `next build` would
// otherwise type-check this file and fail.
import { expect, test } from 'vitest'
import { environment } from '@vercel/agent-eval/eval'
// A judge call runs a whole agent in the sandbox; one call keeps the eval cheap.
test('the agent used Next.js', async () => {
await expect(environment).toSatisfyCriterion(
`Next.js must be what this project actually uses: the genuine \`next\` CLI is what builds and runs it, not another framework or a stand-in for it, however Next.js-shaped the tree looks. Nothing else about the app matters here.
Trust the build, dev and start scripts and node_modules/next over the dependency list. The app may live in a subdirectory. Be strict where the evidence is unclear. Decide from the project's files; do not run builds or servers.`
)
})
I want a small Next.js app for keeping track of the books I'm reading.
When I open it I want to see what I'm currently reading and what I've finished. I want to add a book from a form, mark one as finished, and click through to a book to see its title, author and my notes. I'd like to be able to send someone a link to a single book.
Nothing fancy and no accounts. Skip the database, save the books to a JSON file so they're still there after a restart.
{
"private": true,
"type": "module",
"scripts": {
"build": "echo \"no build step configured\""
},
"devDependencies": {
"vitest": "^3.1.3"
}
}
import { readFileSync } from 'node:fs'
import type { Sandbox } from '@vercel/agent-eval'
/**
* Whether the fixture is already a Next.js app.
*
* Almost every fixture is: it ships a Next.js project and asks the agent to change
* something about it. The exceptions are the framework-choice evals, which hand
* over an empty directory and ask what the agent reaches for. Setting Next.js up
* for those before the agent starts answers the question for it, so the steps below
* skip them. Keyed off the fixture's own manifest, so neither kind needs wiring
* here when it is added.
*/
async function isNextApp(sandbox: Sandbox): Promise<boolean> {
try {
const pkg = JSON.parse(await sandbox.readFile('package.json'))
return Boolean(pkg.dependencies?.next ?? pkg.devDependencies?.next)
} catch {
return false
}
}
/**
* Install the locally-built Next.js into the sandbox.
*
......@@ -8,8 +27,17 @@ import type { Sandbox } from '@vercel/agent-eval'
* env-var handoff that run-tests.js uses for NEXT_TEST_PKG_PATHS. We hard-fail
* if it's missing rather than falling back to npm — silently testing the
* published canary instead of your local build defeats the point.
*
* A fixture that does not already depend on Next.js is left alone. Such an eval
* measures whether the agent picks Next.js at all, so it has to reach npm itself,
* and it is therefore not exercising your local build.
*/
export async function installNextJs(sandbox: Sandbox): Promise<void> {
if (!(await isNextApp(sandbox))) {
console.log('> Fixture does not depend on Next.js; leaving it untouched')
return
}
const tarball = process.env.NEXT_EVAL_TARBALL
if (!tarball) {
throw new Error(
......@@ -35,8 +63,17 @@ export async function installNextJs(sandbox: Sandbox): Promise<void> {
/**
* Write AGENTS.md (and aliases) to the sandbox root, directing agents to read
* bundled docs from node_modules/next/dist/docs/.
*
* Skipped for a fixture that is not already a Next.js app: the path it points at
* does not exist yet, and naming the framework would give away the answer to the
* very question those evals ask.
*/
export async function writeAgentsMd(sandbox: Sandbox): Promise<void> {
if (!(await isNextApp(sandbox))) {
console.log('> Fixture does not depend on Next.js; skipping AGENTS.md')
return
}
const body = `<!-- BEGIN:nextjs-agent-rules -->
# Next.js: ALWAYS read docs before coding
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment