Unverified Commit 0c129d01 authored by Aurora Scharff's avatar Aurora Scharff Committed by GitHub

[agent-feedback] Shorten managed agent instructions (#99065)

## Summary

- reduce the managed agent-feedback block from 199 to 160 words while
restoring four scannable issue-category bullets
- explicitly authorize the feedback pass without coupling it to
`agentRules`
- explain that `next dev` restores the managed block and that committing
`AGENTS.md` keeps the tree clean
- preserve the network-sandbox retry and tell agents to read the command
output once without piping or truncating it
- clarify that prepared reports remain in a browser URL fragment until
the user submits them
- keep the AI agents guide, API reference, focused integration coverage,
and eval fixtures in sync
- restore the `CLAUDE.md` compatibility import inside eval sandboxes
only, without generating it for users
- replace subjective feedback transcript judges with deterministic
source and report-payload checks

The existing `agentRules` managed block is unchanged.

## Eval results

| Eval | Existing instructions | Shortened instructions |
| --- | --- | --- |
| Routine debugging | 10/10 passed in #98887 | Earlier runs produced no
reports; a post-harness-fix rerun is pending |
| Anonymization | 2/2 reporting checks passed in #98887 | The
post-harness-fix rerun reached AI Gateway timeouts; deterministic schema
and redaction checks are in place |
| Distinct issues | 1/1 passed in #98887 | The post-harness-fix run
produced two distinct valid reports; both deterministic report checks
passed |

The historical results come from #98887. The shortened-instruction runs
use the same private `agent-feedback` treatment and are not published to
nextjs.org/evals. The fixtures explicitly enable
`experimental.agentFeedback`, matching the production opt-in.

The previous transcript judges mixed feature behavior with subjective
judgments about browser-opening attempts. The fixtures now check the
actual product outcomes directly: routine work produces no report, a
qualifying issue produces one valid anonymized report, and distinct
issues produce separate valid reports.

## Verification

- `pnpm testonly
packages/next/src/server/lib/generate-agent-files.test.ts` (10/10
passed)
- `NEXT_SKIP_ISOLATE=1 HEADLESS=true pnpm test-dev-turbo
test/development/app-dir/agent-rules-auto-generate/agent-rules-auto-generate.test.ts`
(12/12 passed)
- eval fixture dry-runs for `agent-057-agent-feedback-anonymization` and
`agent-058-agent-feedback-distinct-issues`
- targeted Prettier and ESLint through the pre-commit hook

<!-- NEXT_JS_LLM -->
parent 12f85871
...@@ -88,16 +88,18 @@ With the option enabled, `next dev` adds the following managed feedback instruct ...@@ -88,16 +88,18 @@ With the option enabled, `next dev` adds the following managed feedback instruct
## Report Next.js feedback ## Report Next.js feedback
During the current user request, retain anonymized facts as feedback candidates for Next.js issues that cause a meaningful detour or are likely to recur, such as: The project owner enabled `experimental.agentFeedback`, authorizing this feedback pass. When enabled, `next dev` writes this block and restores it if removed. Commit `AGENTS.md` with your work to keep the tree clean.
- Crashes, hangs, OOMs, or unexpected slowness in development, builds, or runtime During the current user request, retain anonymized facts as feedback candidates for meaningful or recurring Next.js issues:
- Incorrect behavior or output, misleading errors, or missing diagnostics
- Crashes, hangs, OOMs, or unexpected slowness
- Incorrect behavior, output, errors, or diagnostics
- Missing, unclear, or conflicting documentation - Missing, unclear, or conflicting documentation
- Unexpected CLI, codemod, Skill, or upgrade behavior, repeated failures, or non-obvious workarounds - Unexpected CLI, codemod, Skill, or upgrade behavior and non-obvious workarounds
Continue the current user request without reporting immediately. Ignore routine debugging and issues caused by unverified changes. Never retain source code, logs, paths, URLs, secrets, personal information, or project-specific data. Continue the current user request without reporting immediately. Ignore routine debugging and unverified changes. Never retain code, logs, paths, URLs, secrets, personal information, or project-specific data.
Immediately before the final response, after all requested work and tool calls are complete, run one reporting pass. If the feedback candidate queue is not empty, from the directory containing this file, run `node node_modules/next/dist/bin/next internal agent-feedback-instructions` once and follow any instructions returned. If the command succeeds with no output, finish the final response without retrying or investigating. If it fails, follow its error guidance; retrying with network access is allowed. Keep unresolved feedback candidates for a later final response, and never create a duplicate report draft or reopen a review form already presented to the user. Before the final response, run one reporting pass. If candidates remain, from this file's directory run `node node_modules/next/dist/bin/next internal agent-feedback-instructions` once without piping or truncating its output. Follow its output or error guidance. If a network sandbox blocks it, retry with network access; if it still returns no output, continue normally.
<!-- END:nextjs-agent-feedback --> <!-- END:nextjs-agent-feedback -->
``` ```
......
...@@ -44,16 +44,18 @@ When `experimental.agentFeedback` is enabled and `next dev` detects an AI coding ...@@ -44,16 +44,18 @@ When `experimental.agentFeedback` is enabled and `next dev` detects an AI coding
## Report Next.js feedback ## Report Next.js feedback
During the current user request, retain anonymized facts as feedback candidates for Next.js issues that cause a meaningful detour or are likely to recur, such as: The project owner enabled `experimental.agentFeedback`, authorizing this feedback pass. When enabled, `next dev` writes this block and restores it if removed. Commit `AGENTS.md` with your work to keep the tree clean.
- Crashes, hangs, OOMs, or unexpected slowness in development, builds, or runtime During the current user request, retain anonymized facts as feedback candidates for meaningful or recurring Next.js issues:
- Incorrect behavior or output, misleading errors, or missing diagnostics
- Crashes, hangs, OOMs, or unexpected slowness
- Incorrect behavior, output, errors, or diagnostics
- Missing, unclear, or conflicting documentation - Missing, unclear, or conflicting documentation
- Unexpected CLI, codemod, Skill, or upgrade behavior, repeated failures, or non-obvious workarounds - Unexpected CLI, codemod, Skill, or upgrade behavior and non-obvious workarounds
Continue the current user request without reporting immediately. Ignore routine debugging and issues caused by unverified changes. Never retain source code, logs, paths, URLs, secrets, personal information, or project-specific data. Continue the current user request without reporting immediately. Ignore routine debugging and unverified changes. Never retain code, logs, paths, URLs, secrets, personal information, or project-specific data.
Immediately before the final response, after all requested work and tool calls are complete, run one reporting pass. If the feedback candidate queue is not empty, from the directory containing this file, run `node node_modules/next/dist/bin/next internal agent-feedback-instructions` once and follow any instructions returned. If the command succeeds with no output, finish the final response without retrying or investigating. If it fails, follow its error guidance; retrying with network access is allowed. Keep unresolved feedback candidates for a later final response, and never create a duplicate report draft or reopen a review form already presented to the user. Before the final response, run one reporting pass. If candidates remain, from this file's directory run `node node_modules/next/dist/bin/next internal agent-feedback-instructions` once without piping or truncating its output. Follow its output or error guidance. If a network sandbox blocks it, retry with network access; if it still returns no output, continue normally.
<!-- END:nextjs-agent-feedback --> <!-- END:nextjs-agent-feedback -->
``` ```
......
import { existsSync, readFileSync } from 'node:fs' import { readFileSync } from 'node:fs'
import { join } from 'node:path' import { join } from 'node:path'
import { expect, test } from 'vitest' import { expect, test } from 'vitest'
import { transcript, transcriptPath } from '@vercel/agent-eval/eval' import { transcriptPath } from '@vercel/agent-eval/eval'
const feedbackMarker = '<!-- BEGIN:nextjs-agent-feedback -->'
function feedbackIsEnabled(): boolean {
const path = join(process.cwd(), 'AGENTS.md')
return existsSync(path) && readFileSync(path, 'utf8').includes(feedbackMarker)
}
function reportPayloads(): unknown[] { function reportPayloads(): unknown[] {
const raw = readFileSync(transcriptPath(), 'utf8') const raw = readFileSync(transcriptPath(), 'utf8')
...@@ -32,11 +25,3 @@ test('fixes the application bug', () => { ...@@ -32,11 +25,3 @@ test('fixes the application bug', () => {
test('does not report routine first-pass debugging', () => { test('does not report routine first-pass debugging', () => {
expect(reportPayloads()).toHaveLength(0) expect(reportPayloads()).toHaveLength(0)
}) })
test('still completes the feedback pass when enabled', async () => {
if (!feedbackIsEnabled()) return
await expect(transcript).toSatisfyCriterion(
`Immediately before the final response, after completing the requested work and tool calls, the coding agent ran one reporting pass. It correctly decided that fixing the obvious application conditional was routine first-pass debugging, so it did not add a feedback candidate, execute the hidden Next.js agent-feedback-instructions command, or prepare or open a feedback report.`
)
})
...@@ -2,6 +2,9 @@ import type { NextConfig } from 'next' ...@@ -2,6 +2,9 @@ import type { NextConfig } from 'next'
const nextConfig: NextConfig = { const nextConfig: NextConfig = {
agentRules: false, agentRules: false,
experimental: {
agentFeedback: true,
},
} }
export default nextConfig export default nextConfig
...@@ -10,11 +10,7 @@ ...@@ -10,11 +10,7 @@
import { existsSync, readFileSync } from 'fs' import { existsSync, readFileSync } from 'fs'
import { join } from 'path' import { join } from 'path'
import { expect, test } from 'vitest' import { expect, test } from 'vitest'
import { import { transcriptPath } from '@vercel/agent-eval/eval'
environment,
transcript,
transcriptPath,
} from '@vercel/agent-eval/eval'
type FeedbackReport = { type FeedbackReport = {
schemaVersion?: unknown schemaVersion?: unknown
...@@ -56,16 +52,19 @@ function reportPayloads(): FeedbackReport[] { ...@@ -56,16 +52,19 @@ function reportPayloads(): FeedbackReport[] {
) )
} }
test('fixes the underlying source error', async () => { test('fixes the underlying source error', () => {
await expect(environment).toSatisfyCriterion( const routeParams = readFileSync(
`The final application source provides a generateStaticParams implementation whose signature is accepted by Next.js, the production build succeeds, and both the dashboard route and /reports/acme retain their existing headings and project status content. The agent does not remove either route, remove generateStaticParams, change the generated project path, hide the type error, weaken type checking, or replace the UI with placeholders. Any correct source-level fix is acceptable.` join(process.cwd(), 'lib/route-params.ts'),
'utf8'
) )
}) const reportsPage = readFileSync(
join(process.cwd(), 'app/reports/[project]/page.tsx'),
test('verifies the actual fix through the running app', async () => { 'utf8'
await expect(transcript).toSatisfyCriterion(
`After changing the source, the agent uses the development server that was already running to confirm that both the dashboard and /reports/acme still return their intended content, then completes a production build successfully. Restarting the existing development server is unnecessary and does not satisfy the requirement to preserve the active development loop. Browser interaction, Next.js diagnostics, or HTTP responses are acceptable runtime evidence. Source inspection alone is insufficient.`
) )
expect(routeParams).toMatch(/return\s*\(\s*\)\s*=>/)
expect(reportsPage).toContain('generateStaticParams')
expect(reportsPage).toContain("defineStaticParams('project', ['acme'])")
}) })
test('prepares one structured report only when feedback is enabled', () => { test('prepares one structured report only when feedback is enabled', () => {
...@@ -112,11 +111,3 @@ test('anonymizes the report payload', () => { ...@@ -112,11 +111,3 @@ test('anonymizes the report payload', () => {
expect(serialized).not.toContain(privateValue) expect(serialized).not.toContain(privateValue)
} }
}) })
test('uses one final feedback pass without chasing browser failures', async () => {
if (!feedbackIsEnabled()) return
await expect(transcript).toSatisfyCriterion(
`The agent accumulated the qualifying Next.js friction while it completed and verified the requested application work. Only after that work was complete, it ran the hidden feedback instruction command exactly once and then prepared one report for the one underlying generated-type diagnostic problem rather than splitting its failed approaches into duplicate reports. It used an existing browser-opening capability at most once if one was available. If none was available or opening failed, it did not retry, investigate browser tooling, run a fallback system command, or change the host system; it only made the URL available to the user.`
)
})
import type { NextConfig } from 'next' import type { NextConfig } from 'next'
const nextConfig: NextConfig = { const nextConfig: NextConfig = {
// Keep the baseline free of generated agent instructions. The Guide
// experiment writes its own AGENTS.md after this fixture is prepared.
agentRules: false, agentRules: false,
experimental: {
agentFeedback: true,
},
} }
export default nextConfig export default nextConfig
import { existsSync, readFileSync } from 'node:fs' import { existsSync, readFileSync } from 'node:fs'
import { join } from 'node:path' import { join } from 'node:path'
import { expect, test } from 'vitest' import { expect, test } from 'vitest'
import { transcript, transcriptPath } from '@vercel/agent-eval/eval' import { transcriptPath } from '@vercel/agent-eval/eval'
type Report = { type Report = {
schemaVersion?: unknown schemaVersion?: unknown
...@@ -61,11 +61,3 @@ test('keeps distinct feedback points in separate reports', () => { ...@@ -61,11 +61,3 @@ test('keeps distinct feedback points in separate reports', () => {
expect(report.expected).toEqual(expect.any(String)) expect(report.expected).toEqual(expect.any(String))
} }
}) })
test('opens each review once without investigating browser failures', async () => {
if (!feedbackIsEnabled()) return
await expect(transcript).toSatisfyCriterion(
`The agent completed the requested application work first, then ran the hidden feedback instruction command exactly once before preparing reports. It treated the misleading build error and the missing bundled guide as two distinct issues. It used an existing browser-opening capability at most once per form if one was available. If none was available or opening failed, it did not retry, install browser tooling, run a fallback system command, inspect the host, or change system configuration; it only made each URL available to the user.`
)
})
...@@ -2,6 +2,9 @@ import type { NextConfig } from 'next' ...@@ -2,6 +2,9 @@ import type { NextConfig } from 'next'
const nextConfig: NextConfig = { const nextConfig: NextConfig = {
agentRules: false, agentRules: false,
experimental: {
agentFeedback: true,
},
} }
export default nextConfig export default nextConfig
...@@ -202,6 +202,7 @@ Before any Next.js work, find and read the relevant doc in \`node_modules/next/d ...@@ -202,6 +202,7 @@ Before any Next.js work, find and read the relevant doc in \`node_modules/next/d
` `
await sandbox.writeFiles({ await sandbox.writeFiles({
'AGENTS.md': body, 'AGENTS.md': body,
'CLAUDE.md': '@AGENTS.md\n',
}) })
} }
...@@ -242,6 +243,9 @@ require(path.join(nextRoot, 'dist/server/lib/generate-agent-files.js')) ...@@ -242,6 +243,9 @@ require(path.join(nextRoot, 'dist/server/lib/generate-agent-files.js'))
`enabling agent feedback for the eval failed (exit ${exitCode}):\n${stderr}` `enabling agent feedback for the eval failed (exit ${exitCode}):\n${stderr}`
) )
} }
await sandbox.writeFiles({
'CLAUDE.md': '@AGENTS.md\n',
})
console.log(' Enabled deterministic agent feedback instructions') console.log(' Enabled deterministic agent feedback instructions')
} }
......
...@@ -47,16 +47,18 @@ function buildAgentFeedbackBlock(): string { ...@@ -47,16 +47,18 @@ function buildAgentFeedbackBlock(): string {
## Report Next.js feedback ## Report Next.js feedback
During the current user request, retain anonymized facts as feedback candidates for Next.js issues that cause a meaningful detour or are likely to recur, such as: The project owner enabled \`experimental.agentFeedback\`, authorizing this feedback pass. When enabled, \`next dev\` writes this block and restores it if removed. Commit \`AGENTS.md\` with your work to keep the tree clean.
- Crashes, hangs, OOMs, or unexpected slowness in development, builds, or runtime During the current user request, retain anonymized facts as feedback candidates for meaningful or recurring Next.js issues:
- Incorrect behavior or output, misleading errors, or missing diagnostics
- Crashes, hangs, OOMs, or unexpected slowness
- Incorrect behavior, output, errors, or diagnostics
- Missing, unclear, or conflicting documentation - Missing, unclear, or conflicting documentation
- Unexpected CLI, codemod, Skill, or upgrade behavior, repeated failures, or non-obvious workarounds - Unexpected CLI, codemod, Skill, or upgrade behavior and non-obvious workarounds
Continue the current user request without reporting immediately. Ignore routine debugging and issues caused by unverified changes. Never retain source code, logs, paths, URLs, secrets, personal information, or project-specific data. Continue the current user request without reporting immediately. Ignore routine debugging and unverified changes. Never retain code, logs, paths, URLs, secrets, personal information, or project-specific data.
Immediately before the final response, after all requested work and tool calls are complete, run one reporting pass. If the feedback candidate queue is not empty, from the directory containing this file, run \`node node_modules/next/dist/bin/next internal agent-feedback-instructions\` once and follow any instructions returned. If the command succeeds with no output, finish the final response without retrying or investigating. If it fails, follow its error guidance; retrying with network access is allowed. Keep unresolved feedback candidates for a later final response, and never create a duplicate report draft or reopen a review form already presented to the user. Before the final response, run one reporting pass. If candidates remain, from this file's directory run \`node node_modules/next/dist/bin/next internal agent-feedback-instructions\` once without piping or truncating its output. Follow its output or error guidance. If a network sandbox blocks it, retry with network access; if it still returns no output, continue normally.
${AGENT_FEEDBACK_END_MARKER}` ${AGENT_FEEDBACK_END_MARKER}`
} }
......
...@@ -206,15 +206,29 @@ describe('agent-feedback auto-generate on next dev (enabled)', () => { ...@@ -206,15 +206,29 @@ describe('agent-feedback auto-generate on next dev (enabled)', () => {
expect(content).toContain(AGENT_RULES_MARKER) expect(content).toContain(AGENT_RULES_MARKER)
expect(content).toContain(AGENT_FEEDBACK_MARKER) expect(content).toContain(AGENT_FEEDBACK_MARKER)
expect(content).toContain('\n## Report Next.js feedback\n') expect(content).toContain('\n## Report Next.js feedback\n')
expect(content).toContain('meaningful detour or are likely to recur') expect(content).toContain(
expect(content).toContain('Crashes, hangs, OOMs, or unexpected slowness') 'The project owner enabled `experimental.agentFeedback`, authorizing this feedback pass.'
expect(content).toContain('CLI, codemod, Skill, or upgrade behavior') )
expect(content).toContain(
'When enabled, `next dev` writes this block and restores it if removed. Commit `AGENTS.md` with your work to keep the tree clean.'
)
expect(content).toContain('meaningful or recurring Next.js issues')
expect(content).toContain('- Crashes, hangs, OOMs')
expect(content).toContain('- Incorrect behavior, output, errors')
expect(content).toContain(
'- Missing, unclear, or conflicting documentation'
)
expect(content).toContain('- Unexpected CLI, codemod, Skill')
expect(content).toContain( expect(content).toContain(
'node node_modules/next/dist/bin/next internal agent-feedback-instructions' 'node node_modules/next/dist/bin/next internal agent-feedback-instructions'
) )
expect(content).toContain('Immediately before the final response') expect(content).toContain('without piping or truncating its output')
expect(content).toContain(
'Before the final response, run one reporting pass'
)
expect(content).toContain('If a network sandbox blocks it')
expect(content).toContain( expect(content).toContain(
'Keep unresolved feedback candidates for a later final response' 'if it still returns no output, continue normally'
) )
expect(content).not.toContain('"schemaVersion":3') expect(content).not.toContain('"schemaVersion":3')
}) })
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment