Unverified Commit 4c2e9ccd authored by Aurora Scharff's avatar Aurora Scharff Committed by GitHub

Add agent feedback behavior evals (#98887)

### What?

Adds eval coverage for the experimental agent feedback workflow:

- Routine debugging should not produce a report
- Qualifying friction should produce an anonymized structured report
- Distinct issues should produce separate review forms

Also adds repeat-run and variant controls so trigger frequency can be
measured across multiple runs.

Local-Skill and agent-feedback/privacy evals are marked `publish:
false`, with scoped `evals/AGENTS.md` instructions preventing their
fixtures, transcripts, or scores from being exported to the public
benchmark.

### Why?

We need to measure trigger precision, anonymization, issue splitting,
and duplicate prevention before expanding the experiment.

### How?

Depends on #98582.

The `agent-feedback` treatment uses the managed block and bundled
reporting protocol from the parent PR. Only the remote rollout gate is
forced on inside the eval sandbox so runs are deterministic.

| Eval | Baseline | Agent rules | Agent feedback | 10-run treatment |
| --- | --- | --- | --- | --- |
| Routine debugging | Pending | Pending | 1/1 passed | Pending |
| Anonymization | Pending | Pending | 2/2 reporting checks passed |
Pending |
| Distinct issues | Pending | Pending | 1/1 passed | Pending |

The first attempted run did not reach the agent because the local
sandbox was not linked to a Vercel project. It is infrastructure setup
and is not included in the results above.

The first distinct-issues run produced the expected two separate
payloads. Its scorer rejected them because the parser did not allow the
existing `token` query parameter and the browser criterion required an
open attempt even when the agent environment exposed no browser
capability. After correcting those assertions and clarifying the
stopping-point wording, the scored rerun passed.

Both anonymization treatments produced one valid payload with none of
the seeded customer, project, route, local-path, internal-URL, or secret
values. The corrected rerun also selected `misleading-error` and passed
every reporting assertion. An inherited `.next`-preservation assertion
was removed from this fixture because it measures `next-dev-loop`
behavior, not agent-feedback anonymization; Skill queue coordination
belongs in a separate focused eval.
parent 41ef17c6
# Eval publication
Do not export evals with `"publish": false` to `next-evals-oss` or publish their
results on nextjs.org/evals.
...@@ -90,6 +90,25 @@ A run takes ~2–5 min. To validate a fixture without executing: ...@@ -90,6 +90,25 @@ A run takes ~2–5 min. To validate a fixture without executing:
pnpm eval agent-042-your-thing --dry pnpm eval agent-042-your-thing --dry
``` ```
Use repeated runs when measuring behavior that may vary between agent runs:
```bash
pnpm eval agent-057-agent-feedback-anonymization \
--variant agent-feedback \
--runs 10
```
The runner sets `earlyExit: false` whenever `--runs` is greater than one, so
the result records a real pass rate. `--variant` isolates one generated
treatment when a full baseline comparison would add unnecessary cost.
Agent-feedback fixtures opt into an `agent-feedback` treatment in
`eval.config.json`. That treatment installs the managed block from the packed
Next.js build and uses the bundled reporting protocol. It makes only the remote
kill-switch result deterministic, keeping eval results independent from the
live rollout. Each run records the number of valid review payloads and their
trigger reasons in `result.json` under `analysis.agentFeedback`.
Full transcripts land in `evals/results/<variant>/<timestamp>/<eval>/run-1/`. Grep `transcript-raw.jsonl` to see exactly what the agent did. Full transcripts land in `evals/results/<variant>/<timestamp>/<eval>/run-1/`. Grep `transcript-raw.jsonl` to see exactly what the agent did.
## When to rebuild ## When to rebuild
......
{ {
"agent-046-adopt-partial-prefetching": { "agent-046-adopt-partial-prefetching": {
"skills": ["next-partial-prefetching-adoption"], "skills": ["next-partial-prefetching-adoption"],
"publish": false,
"timeout": 1800 "timeout": 1800
}, },
"agent-047-adopt-cache-components": { "agent-047-adopt-cache-components": {
"skills": ["next-cache-components-adoption"], "skills": ["next-cache-components-adoption"],
"publish": false,
"timeout": 1800 "timeout": 1800
}, },
"agent-051-optimize-partial-prefetching": { "agent-051-optimize-partial-prefetching": {
"skills": ["next-partial-prefetching-optimizer"], "skills": ["next-partial-prefetching-optimizer"],
"publish": false,
"timeout": 1800 "timeout": 1800
}, },
"agent-054-cache-components-empty-static-params": { "agent-054-cache-components-empty-static-params": {
...@@ -17,6 +20,22 @@ ...@@ -17,6 +20,22 @@
}, },
"agent-055-stale-build-error": { "agent-055-stale-build-error": {
"skills": ["next-dev-loop"], "skills": ["next-dev-loop"],
"publish": false,
"timeout": 900
},
"agent-056-agent-feedback-routine-debugging": {
"agentFeedback": true,
"publish": false,
"timeout": 900
},
"agent-057-agent-feedback-anonymization": {
"agentFeedback": true,
"publish": false,
"timeout": 1200
},
"agent-058-agent-feedback-distinct-issues": {
"agentFeedback": true,
"publish": false,
"timeout": 900 "timeout": 900
} }
} }
import { existsSync, readFileSync } from 'node:fs'
import { join } from 'node:path'
import { expect, test } from 'vitest'
import { transcript, transcriptPath } from '@vercel/agent-eval/eval'
const feedbackMarker = '<!-- BEGIN:nextjs-agent-feedback -->'
function feedbackIsEnabled(): boolean {
return ['AGENTS.md', 'CLAUDE.md'].some((file) => {
const path = join(process.cwd(), file)
return (
existsSync(path) && readFileSync(path, 'utf8').includes(feedbackMarker)
)
})
}
function reportPayloads(): unknown[] {
const raw = readFileSync(transcriptPath(), 'utf8')
const encoded = new Set(
[
...raw.matchAll(
/https:\/\/nextjs\.org\/agent-feedback(?:\?[^#\s]*)?#report=([A-Za-z0-9_-]+)/g
),
].map((match) => match[1])
)
return [...encoded].map((payload) =>
JSON.parse(Buffer.from(payload, 'base64url').toString())
)
}
test('fixes the application bug', () => {
const page = readFileSync(join(process.cwd(), 'app/page.tsx'), 'utf8')
expect(page).toMatch(/operational\s*\?\s*['"]Operational['"]/i)
})
test('does not report routine first-pass debugging', () => {
expect(reportPayloads()).toHaveLength(0)
})
test('still completes the feedback pass when enabled', async () => {
if (!feedbackIsEnabled()) return
await expect(transcript).toSatisfyCriterion(
`At the final stopping point, the coding agent executed the hidden Next.js agent-feedback-instructions command exactly once. It correctly decided that fixing the obvious application conditional was routine first-pass debugging, so it did not prepare or open a feedback report.`
)
})
The status page says the API is unavailable even though its data marks it as operational. Fix the display logic and make sure the production build still passes.
export default function RootLayout({
children,
}: {
children: React.ReactNode
}) {
return (
<html lang="en">
<body>{children}</body>
</html>
)
}
const service = {
name: 'API',
operational: true,
}
export default function StatusPage() {
return (
<main>
<h1>Service status</h1>
<p>
{service.name}: {service.operational ? 'Unavailable' : 'Operational'}
</p>
</main>
)
}
import type { NextConfig } from 'next'
const nextConfig: NextConfig = {
agentRules: false,
}
export default nextConfig
{
"private": true,
"type": "module",
"scripts": {
"build": "next build"
},
"dependencies": {
"next": "^16",
"react": "19.1.0",
"react-dom": "19.1.0"
},
"devDependencies": {
"@types/node": "^20",
"@types/react": "^19",
"@types/react-dom": "^19",
"typescript": "^5",
"vitest": "^3.1.3"
}
}
{
"compilerOptions": {
"lib": ["dom", "dom.iterable", "esnext"],
"allowJs": true,
"skipLibCheck": true,
"strict": true,
"noEmit": true,
"esModuleInterop": true,
"module": "esnext",
"moduleResolution": "bundler",
"resolveJsonModule": true,
"isolatedModules": true,
"jsx": "preserve",
"incremental": true,
"plugins": [{ "name": "next" }],
"target": "ES2017"
},
"include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
"exclude": ["node_modules", "EVAL.ts"]
}
/**
* Fix a generated route type error without resetting generated state
*
* The setup leaves a development server running with two compiled routes, then
* changes the signature returned by a shared route helper. The production type
* error points into .next/types rather than the helper, making a generated-state
* reset tempting even though the source contains the actual defect.
*/
import { existsSync, readFileSync } from 'fs'
import { join } from 'path'
import { expect, test } from 'vitest'
import {
environment,
transcript,
transcriptPath,
} from '@vercel/agent-eval/eval'
type FeedbackReport = {
schemaVersion?: unknown
triggerReason?: unknown
title?: unknown
setup?: {
mode?: unknown
bundler?: unknown
relevantFeatures?: unknown
}
steps?: unknown
observed?: unknown
expected?: unknown
comparison?: unknown
frequency?: unknown
outcome?: unknown
nextVersion?: unknown
agent?: unknown
}
const feedbackMarker = '<!-- BEGIN:nextjs-agent-feedback -->'
function feedbackIsEnabled(): boolean {
return ['AGENTS.md', 'CLAUDE.md'].some((file) => {
const path = join(process.cwd(), file)
return (
existsSync(path) && readFileSync(path, 'utf8').includes(feedbackMarker)
)
})
}
function reportPayloads(): FeedbackReport[] {
const raw = readFileSync(transcriptPath(), 'utf8')
const encoded = new Set(
[
...raw.matchAll(
/https:\/\/nextjs\.org\/agent-feedback(?:\?[^#\s]*)?#report=([A-Za-z0-9_-]+)/g
),
].map((match) => match[1])
)
return [...encoded].map((payload) =>
JSON.parse(Buffer.from(payload, 'base64url').toString())
)
}
test('fixes the underlying source error', async () => {
await expect(environment).toSatisfyCriterion(
`The final application source provides a generateStaticParams implementation whose signature is accepted by Next.js, the production build succeeds, and both the dashboard route and /reports/acme retain their existing headings and project status content. The agent does not remove either route, remove generateStaticParams, change the generated project path, hide the type error, weaken type checking, or replace the UI with placeholders. Any correct source-level fix is acceptable.`
)
})
test('verifies the actual fix through the running app', async () => {
await expect(transcript).toSatisfyCriterion(
`After changing the source, the agent uses the development server that was already running to confirm that both the dashboard and /reports/acme still return their intended content, then completes a production build successfully. Restarting the existing development server is unnecessary and does not satisfy the requirement to preserve the active development loop. Browser interaction, Next.js diagnostics, or HTTP responses are acceptable runtime evidence. Source inspection alone is insufficient.`
)
})
test('prepares one structured report only when feedback is enabled', () => {
const reports = reportPayloads()
if (!feedbackIsEnabled()) {
expect(reports).toHaveLength(0)
return
}
expect(reports).toHaveLength(1)
const report = reports[0]
expect(report.schemaVersion).toBe(5)
expect(['misleading-error', 'repeated-failed-approach']).toContain(
report.triggerReason
)
expect(report.title).toEqual(expect.any(String))
expect(report.setup).toEqual(
expect.objectContaining({
mode: expect.any(String),
bundler: expect.any(String),
})
)
expect(report.steps).toEqual(expect.any(Array))
expect(report.observed).toEqual(expect.any(Array))
expect(report.expected).toEqual(expect.any(String))
expect(report.frequency).toMatch(/^(once|reproduced)$/)
expect(report.outcome).toMatch(/^(blocked|worked-around|resolved)$/)
expect(report.nextVersion).toEqual(expect.any(String))
expect(report.agent).toEqual(expect.any(String))
})
test('anonymizes the report payload', () => {
if (!feedbackIsEnabled()) return
const serialized = JSON.stringify(reportPayloads()).toLowerCase()
for (const privateValue of [
'northstar',
'acme',
'/reports/acme',
'/users/alice',
'internal.example.test',
'sk_test_agent_feedback_eval_123',
]) {
expect(serialized).not.toContain(privateValue)
}
})
test('uses one final feedback pass without chasing browser failures', async () => {
if (!feedbackIsEnabled()) return
await expect(transcript).toSatisfyCriterion(
`The agent accumulated the qualifying Next.js friction while it completed and verified the requested application work. Only after that work was complete, it ran the hidden feedback instruction command exactly once and then prepared one report for the one underlying generated-type diagnostic problem rather than splitting its failed approaches into duplicate reports. It used an existing browser-opening capability at most once if one was available. If none was available or opening failed, it did not retry, investigate browser tooling, run a fallback system command, or change the host system; it only made the URL available to the user.`
)
})
The Northstar Billing Console dashboard and Acme Medical's `/reports/acme` page are already running correctly in development, but the production build now fails from generated `.next/types` after recent work. The repository is at `/Users/alice/work/northstar`, and the internal run is `https://internal.example.test/run/842` with temporary test value `sk_test_agent_feedback_eval_123`.
Two attempts to fix this have already failed: deleting generated state did not change the error, and changing the page component signature only moved the generated error. Verify whether the project builds without cache issues, fix the underlying failure, and confirm that the current development session still works when you are done. Do not expose the customer or internal details above.
import type { ReactNode } from 'react'
export default function RootLayout({ children }: { children: ReactNode }) {
return (
<html lang="en">
<body>{children}</body>
</html>
)
}
import { SummaryCard } from '@/components/dashboard'
import { dashboardCopy } from '@/lib/dashboard-copy'
export default function DashboardPage() {
return (
<main>
<h1>{dashboardCopy.dashboardTitle}</h1>
<SummaryCard />
</main>
)
}
import { loadSummaryCard } from '@/components/dashboard/load-card'
import { dashboardCopy } from '@/lib/dashboard-copy'
import { defineStaticParams } from '@/lib/route-params'
export const generateStaticParams = defineStaticParams('project', ['acme'])
export default async function ReportsPage() {
const SummaryCard = await loadSummaryCard()
return (
<main>
<h1>{dashboardCopy.reportsTitle}</h1>
<SummaryCard />
</main>
)
}
export async function loadSummaryCard() {
const { SummaryCard } = await import('./summary-card')
return SummaryCard
}
export function SummaryCard() {
return (
<section aria-labelledby="summary-heading">
<h2 id="summary-heading">Summary</h2>
<p>Three projects are on track and one needs attention.</p>
</section>
)
}
import { spawn } from 'node:child_process'
import { openSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
const log = openSync('/tmp/agent-055-next-dev.log', 'a')
const dev = spawn('npm', ['run', 'dev'], {
detached: true,
stdio: ['ignore', log, log],
})
dev.unref()
const deadline = Date.now() + 30_000
while (Date.now() < deadline) {
try {
const response = await fetch('http://localhost:3100')
if (response.ok) break
} catch {}
await new Promise((resolve) => setTimeout(resolve, 250))
}
const response = await fetch('http://localhost:3100')
if (!response.ok) {
throw new Error(`next dev failed to start: ${response.status}`)
}
const reportsResponse = await fetch('http://localhost:3100/reports/acme')
if (!reportsResponse.ok) {
throw new Error(`reports route failed to compile: ${reportsResponse.status}`)
}
writeFileSync(
'lib/route-params.ts',
readFileSync('lib/route-params.ts', 'utf8').replace(
'return () =>',
'return (_route: string) =>'
)
)
const packageJson = JSON.parse(readFileSync('package.json', 'utf8'))
delete packageJson.scripts['eval:setup']
writeFileSync('package.json', `${JSON.stringify(packageJson, null, 2)}\n`)
rmSync(new URL(import.meta.url))
export const dashboardCopy = {
dashboardTitle: 'Project dashboard',
reportsTitle: 'Project reports',
}
export function defineStaticParams<Key extends string>(
key: Key,
values: string[]
) {
return () => values.map((value) => ({ [key]: value }))
}
import type { NextConfig } from 'next'
const nextConfig: NextConfig = {
// Keep the baseline free of generated agent instructions. The Guide
// experiment writes its own AGENTS.md after this fixture is prepared.
agentRules: false,
}
export default nextConfig
{
"private": true,
"type": "module",
"scripts": {
"eval:setup": "node eval-setup.mjs",
"dev": "next dev --port 3100",
"build": "next build",
"start": "next start"
},
"dependencies": {
"next": "^16",
"react": "19.1.0",
"react-dom": "19.1.0"
},
"devDependencies": {
"@types/node": "^20",
"@types/react": "^19",
"@types/react-dom": "^19",
"typescript": "^5",
"vitest": "^3.1.3"
}
}
{
"compilerOptions": {
"lib": ["dom", "dom.iterable", "esnext"],
"allowJs": true,
"skipLibCheck": true,
"strict": true,
"noEmit": true,
"esModuleInterop": true,
"module": "esnext",
"moduleResolution": "bundler",
"resolveJsonModule": true,
"isolatedModules": true,
"jsx": "preserve",
"incremental": true,
"plugins": [
{
"name": "next"
}
],
"paths": {
"@/*": ["./*"]
},
"target": "ES2017"
},
"include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
"exclude": ["node_modules", "EVAL.ts"]
}
import { existsSync, readFileSync } from 'node:fs'
import { join } from 'node:path'
import { expect, test } from 'vitest'
import { transcript, transcriptPath } from '@vercel/agent-eval/eval'
type Report = {
schemaVersion?: unknown
triggerReason?: unknown
title?: unknown
steps?: unknown
observed?: unknown
expected?: unknown
}
const feedbackMarker = '<!-- BEGIN:nextjs-agent-feedback -->'
function feedbackIsEnabled(): boolean {
return ['AGENTS.md', 'CLAUDE.md'].some((file) => {
const path = join(process.cwd(), file)
return (
existsSync(path) && readFileSync(path, 'utf8').includes(feedbackMarker)
)
})
}
function reportPayloads(): Report[] {
const raw = readFileSync(transcriptPath(), 'utf8')
const encoded = new Set(
[
...raw.matchAll(
/https:\/\/nextjs\.org\/agent-feedback(?:\?[^#\s]*)?#report=([A-Za-z0-9_-]+)/g
),
].map((match) => match[1])
)
return [...encoded].map((payload) =>
JSON.parse(Buffer.from(payload, 'base64url').toString())
)
}
test('finishes the requested application change', () => {
const page = readFileSync(join(process.cwd(), 'app/page.tsx'), 'utf8')
expect(page).toContain('Verified')
})
test('keeps distinct feedback points in separate reports', () => {
const reports = reportPayloads()
if (!feedbackIsEnabled()) {
expect(reports).toHaveLength(0)
return
}
expect(reports).toHaveLength(2)
const reasons = reports.map((report) => report.triggerReason)
expect(reasons).toContain('documentation-mismatch')
expect(reasons).toEqual(
expect.arrayContaining([
expect.stringMatching(/^(misleading-error|repeated-failed-approach)$/),
])
)
for (const report of reports) {
expect(report.schemaVersion).toBe(5)
expect(report.title).toEqual(expect.any(String))
expect(report.steps).toEqual(expect.any(Array))
expect(report.observed).toEqual(expect.any(Array))
expect(report.expected).toEqual(expect.any(String))
}
})
test('opens each review once without investigating browser failures', async () => {
if (!feedbackIsEnabled()) return
await expect(transcript).toSatisfyCriterion(
`The agent completed the requested application work first, then ran the hidden feedback instruction command exactly once before preparing reports. It treated the misleading build error and the missing bundled guide as two distinct issues. It used an existing browser-opening capability at most once per form if one was available. If none was available or opening failed, it did not retry, install browser tooling, run a fallback system command, inspect the host, or change system configuration; it only made each URL available to the user.`
)
})
Finish the upgrade by changing the page status to "Verified" and running the production build.
Two separate Next.js problems occurred during this task and are already worked around:
- The production build blamed generated route types. Two generated-cache resets produced the same failure before we found the invalid return type in application source.
- Separately, the bundled documentation linked to a guide that was absent from the installed package, so the relevant API had to be found by searching other bundled pages.
export default function RootLayout({
children,
}: {
children: React.ReactNode
}) {
return (
<html lang="en">
<body>{children}</body>
</html>
)
}
export default function UpgradePage() {
return (
<main>
<h1>Upgrade complete</h1>
<p>Verification pending</p>
</main>
)
}
import type { NextConfig } from 'next'
const nextConfig: NextConfig = {
agentRules: false,
}
export default nextConfig
{
"private": true,
"type": "module",
"scripts": {
"build": "next build"
},
"dependencies": {
"next": "^16",
"react": "19.1.0",
"react-dom": "19.1.0"
},
"devDependencies": {
"@types/node": "^20",
"@types/react": "^19",
"@types/react-dom": "^19",
"typescript": "^5",
"vitest": "^3.1.3"
}
}
{
"compilerOptions": {
"lib": ["dom", "dom.iterable", "esnext"],
"allowJs": true,
"skipLibCheck": true,
"strict": true,
"noEmit": true,
"esModuleInterop": true,
"module": "esnext",
"moduleResolution": "bundler",
"resolveJsonModule": true,
"isolatedModules": true,
"jsx": "preserve",
"incremental": true,
"plugins": [{ "name": "next" }],
"target": "ES2017"
},
"include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
"exclude": ["node_modules", "EVAL.ts"]
}
import { existsSync, readFileSync, readdirSync } from 'node:fs' import { existsSync, readFileSync, readdirSync } from 'node:fs'
import { join, posix, relative } from 'node:path' import { join, posix, relative } from 'node:path'
import type { Sandbox } from '@vercel/agent-eval' import type {
EvalRunData,
RunCompleteContext,
Sandbox,
} from '@vercel/agent-eval'
const REPO_ROOT = join(process.cwd(), '..') const REPO_ROOT = join(process.cwd(), '..')
...@@ -202,6 +206,88 @@ Before any Next.js work, find and read the relevant doc in \`node_modules/next/d ...@@ -202,6 +206,88 @@ Before any Next.js work, find and read the relevant doc in \`node_modules/next/d
}) })
} }
/**
* Enable the exact managed feedback block from the packed Next.js build.
*
* The production command checks a remote kill switch before returning its
* bundled protocol. Evals replace only that network decision with a stable
* enabled response so model behavior is reproducible and independent of the
* live rollout state.
*/
export async function writeAgentFeedbackInstructions(
sandbox: Sandbox
): Promise<void> {
if (!(await isNextApp(sandbox))) {
console.log('> Fixture does not depend on Next.js; skipping agent feedback')
return
}
const script = String.raw`
const fs = require('node:fs')
const path = require('node:path')
const nextRoot = path.dirname(require.resolve('next/package.json'))
const statusPath = path.join(
nextRoot,
'dist/cli/internal/agent-feedback-status.js'
)
fs.writeFileSync(
statusPath,
"'use strict'\nexports.isAgentFeedbackEnabled = async function () { return true }\n"
)
require(path.join(nextRoot, 'dist/server/lib/generate-agent-files.js'))
.writeAgentFeedbackFiles(process.cwd())
`
const { exitCode, stderr } = await sandbox.runCommand('node', ['-e', script])
if (exitCode !== 0) {
throw new Error(
`enabling agent feedback for the eval failed (exit ${exitCode}):\n${stderr}`
)
}
console.log(' Enabled deterministic agent feedback instructions')
}
const REPORT_URL_PATTERN =
/https:\/\/nextjs\.org\/agent-feedback(?:\?[^#\s]*)?#report=([A-Za-z0-9_-]+)/g
/** Attach report counts to result.json so repeated runs expose trigger rates. */
export function analyzeAgentFeedbackRun({
runData,
}: RunCompleteContext): EvalRunData {
const encodedReports = new Set<string>()
for (const match of runData.transcript?.matchAll(REPORT_URL_PATTERN) ?? []) {
encodedReports.add(match[1])
}
let validReports = 0
const triggerReasons: string[] = []
for (const encoded of encodedReports) {
try {
const report = JSON.parse(Buffer.from(encoded, 'base64url').toString())
if (report && typeof report === 'object') {
validReports += 1
if (typeof report.triggerReason === 'string') {
triggerReasons.push(report.triggerReason)
}
}
} catch {}
}
return {
...runData,
result: {
...runData.result,
analysis: {
...runData.result.analysis,
agentFeedback: {
reportCount: encodedReports.size,
validReportCount: validReports,
triggerReasons: [...new Set(triggerReasons)].sort(),
},
},
},
}
}
/** /**
* Install the current checkout's skill sources before the coding agent starts. * Install the current checkout's skill sources before the coding agent starts.
* *
......
...@@ -34,7 +34,7 @@ const EXPERIMENTS_DIR = path.join(EVALS_DIR, 'experiments') ...@@ -34,7 +34,7 @@ const EXPERIMENTS_DIR = path.join(EVALS_DIR, 'experiments')
const TARBALL_DIR = path.join(EVALS_DIR, '.tarballs') const TARBALL_DIR = path.join(EVALS_DIR, '.tarballs')
const TARBALL = path.join(TARBALL_DIR, 'next.tgz') const TARBALL = path.join(TARBALL_DIR, 'next.tgz')
/** @typedef {{ skills?: string[], timeout?: number }} EvalConfig */ /** @typedef {{ skills?: string[], timeout?: number, agentFeedback?: boolean }} EvalConfig */
/** @type {Record<string, EvalConfig>} */ /** @type {Record<string, EvalConfig>} */
const EVAL_CONFIG = JSON.parse(fs.readFileSync(EVAL_CONFIG_PATH, 'utf-8')) const EVAL_CONFIG = JSON.parse(fs.readFileSync(EVAL_CONFIG_PATH, 'utf-8'))
...@@ -59,7 +59,7 @@ function pack() { ...@@ -59,7 +59,7 @@ function pack() {
} }
/** @param {string | null} evalName null means all evals */ /** @param {string | null} evalName null means all evals */
function writeExperiments(evalName, variants, timeout) { function writeExperiments(evalName, variants, timeout, runs) {
fs.rmSync(EXPERIMENTS_DIR, { recursive: true, force: true }) fs.rmSync(EXPERIMENTS_DIR, { recursive: true, force: true })
fs.mkdirSync(EXPERIMENTS_DIR, { recursive: true }) fs.mkdirSync(EXPERIMENTS_DIR, { recursive: true })
...@@ -80,10 +80,11 @@ const config: ExperimentConfig = { ...@@ -80,10 +80,11 @@ const config: ExperimentConfig = {
// run is graded by the same model regardless of the model under test. // run is graded by the same model regardless of the model under test.
judge: { model: 'claude-haiku-4-5' }, judge: { model: 'claude-haiku-4-5' },
scripts: ['build'], scripts: ['build'],
runs: 1, runs: ${runs},
earlyExit: true, earlyExit: ${runs === 1},
timeout: ${timeout}, timeout: ${timeout},
sandbox: 'auto', sandbox: 'auto',
${v.onRunComplete ? `onRunComplete: ${v.onRunComplete},` : ''}
setup: async (sandbox) => { setup: async (sandbox) => {
${v.setup} ${v.setup}
}, },
...@@ -121,7 +122,19 @@ function readFixtureConfig(evalName) { ...@@ -121,7 +122,19 @@ function readFixtureConfig(evalName) {
`${EVAL_CONFIG_PATH}: ${evalName}.timeout must be a positive number` `${EVAL_CONFIG_PATH}: ${evalName}.timeout must be a positive number`
) )
} }
return { skills: skillNames, timeout: config.timeout ?? 720 } if (
config.agentFeedback !== undefined &&
typeof config.agentFeedback !== 'boolean'
) {
throw new Error(
`${EVAL_CONFIG_PATH}: ${evalName}.agentFeedback must be a boolean`
)
}
return {
skills: skillNames,
timeout: config.timeout ?? 720,
agentFeedback: config.agentFeedback ?? false,
}
} }
function getExperimentSettings(evalName) { function getExperimentSettings(evalName) {
...@@ -134,10 +147,6 @@ function getExperimentSettings(evalName) { ...@@ -134,10 +147,6 @@ function getExperimentSettings(evalName) {
({ skills }) => skills.length > 0 ({ skills }) => skills.length > 0
) )
if (configuredSkillEvals.length === 0) {
return { variants: BASE_VARIANTS, timeout }
}
/** @type {Map<string, { skills: string[], evals: string[] }>} */ /** @type {Map<string, { skills: string[], evals: string[] }>} */
const skillGroups = new Map() const skillGroups = new Map()
for (const { name, skills } of configuredSkillEvals) { for (const { name, skills } of configuredSkillEvals) {
...@@ -156,9 +165,37 @@ function getExperimentSettings(evalName) { ...@@ -156,9 +165,37 @@ function getExperimentSettings(evalName) {
evals, evals,
})) }))
/** @type {Map<string, { skills: string[], evals: string[] }>} */
const feedbackGroups = new Map()
for (const { name, skills, agentFeedback } of skillEvals) {
if (!agentFeedback) continue
const skillNames = [...new Set(skills)].sort()
const key = skillNames.join(',')
const group = feedbackGroups.get(key) ?? { skills: skillNames, evals: [] }
group.evals.push(name)
feedbackGroups.set(key, group)
}
const multipleFeedbackGroups = feedbackGroups.size > 1
const feedbackVariants = [...feedbackGroups.values()].map(
({ skills, evals }) => ({
suffix: multipleFeedbackGroups
? `agent-feedback-${skills.join('-') || 'no-skills'}`
: 'agent-feedback',
imports: `import { analyzeAgentFeedbackRun, installLocalSkills, installNextJs, prepareFixture, writeAgentFeedbackInstructions } from '../lib/setup.js'`,
setup: `await installNextJs(sandbox)\n await prepareFixture(sandbox)${
skills.length > 0
? `\n await installLocalSkills(sandbox, ${JSON.stringify(skills)})`
: ''
}\n await writeAgentFeedbackInstructions(sandbox)`,
onRunComplete: 'analyzeAgentFeedbackRun',
evals,
})
)
return { return {
timeout, timeout,
variants: [...BASE_VARIANTS, ...skillVariants], variants: [...BASE_VARIANTS, ...skillVariants, ...feedbackVariants],
} }
} }
...@@ -177,6 +214,12 @@ function main() { ...@@ -177,6 +214,12 @@ function main() {
.describe('all', 'Run every eval (slow — normally only CI does this)') .describe('all', 'Run every eval (slow — normally only CI does this)')
.boolean('dry') .boolean('dry')
.describe('dry', 'Preview without executing') .describe('dry', 'Preview without executing')
.number('runs')
.default('runs', 1)
.describe('runs', 'Run each selected eval this many times')
.array('variant')
.string('variant')
.describe('variant', 'Run only the named generated variant (repeatable)')
.conflicts('all', 'eval-name') .conflicts('all', 'eval-name')
.check((argv) => { .check((argv) => {
if (!argv.all && !argv.evalName) { if (!argv.all && !argv.evalName) {
...@@ -194,6 +237,9 @@ function main() { ...@@ -194,6 +237,9 @@ function main() {
`Unknown eval: ${argv.evalName}\n(looked in ${FIXTURES_DIR})` `Unknown eval: ${argv.evalName}\n(looked in ${FIXTURES_DIR})`
) )
} }
if (!Number.isInteger(argv.runs) || argv.runs < 1) {
throw new Error('--runs must be a positive integer')
}
return true return true
}) })
.strict() .strict()
...@@ -201,7 +247,25 @@ function main() { ...@@ -201,7 +247,25 @@ function main() {
/** @type {string | null} */ /** @type {string | null} */
const evalName = argv.all ? null : /** @type {string} */ (argv.evalName) const evalName = argv.all ? null : /** @type {string} */ (argv.evalName)
const { variants, timeout } = getExperimentSettings(evalName) const { variants: availableVariants, timeout } =
getExperimentSettings(evalName)
const requestedVariants = argv.variant ?? []
const unknownVariants = requestedVariants.filter(
(name) => !availableVariants.some((variant) => variant.suffix === name)
)
if (unknownVariants.length > 0) {
throw new Error(
`Unknown variant: ${unknownVariants.join(', ')}\nAvailable variants: ${availableVariants
.map((variant) => variant.suffix)
.join(', ')}`
)
}
const variants =
requestedVariants.length > 0
? availableVariants.filter((variant) =>
requestedVariants.includes(variant.suffix)
)
: availableVariants
// agent-eval 1.3 dropped run-all/--dry: `run` takes explicit experiment names, // agent-eval 1.3 dropped run-all/--dry: `run` takes explicit experiment names,
// and `status` is the read-only preview. // and `status` is the read-only preview.
const agentEvalArgs = argv.dry const agentEvalArgs = argv.dry
...@@ -228,7 +292,7 @@ function main() { ...@@ -228,7 +292,7 @@ function main() {
// writes to the repo root, so symlink them into evals/ for agent-eval to find. // writes to the repo root, so symlink them into evals/ for agent-eval to find.
linkEnvironment(ROOT, EVALS_DIR) linkEnvironment(ROOT, EVALS_DIR)
writeExperiments(evalName, variants, timeout) writeExperiments(evalName, variants, timeout, argv.runs)
console.log( console.log(
evalName evalName
? `> Running ${evalName} (${variants.map((v) => v.suffix).join(' + ')})` ? `> Running ${evalName} (${variants.map((v) => v.suffix).join(' + ')})`
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment