Unverified Commit a2470dbd authored by Aurora Scharff's avatar Aurora Scharff Committed by GitHub

[agent-feedback] Respect CI and telemetry opt-out (#99062)

### What?

Make automatic agent feedback respect CI, the user's Next.js Telemetry
preference, and the existing remote kill switch. The managed
instructions are omitted in CI or when telemetry is disabled, and the
internal reporting command fails closed before returning the protocol.

### Why?

Agent feedback should never open from CI and should share the user's
telemetry opt-out. The existing Playground gate remains the remote
emergency switch; this change makes the local entry points consistently
honor the other safety conditions.

### How?

- Gate managed feedback instructions on non-CI execution and enabled
telemetry.
- Gate the internal feedback command locally before checking the remote
switch.
- Preserve explicit `--dry-run` diagnostics without opening a browser.
- Document the CI and telemetry behavior and cover it with focused
tests.

<!-- NEXT_JS_LLM -->
parent fdcb7386
...@@ -79,7 +79,7 @@ The index at [`/docs/llms.txt`](https://nextjs.org/docs/llms.txt) and the single ...@@ -79,7 +79,7 @@ The index at [`/docs/llms.txt`](https://nextjs.org/docs/llms.txt) and the single
Automatic agent feedback lets coding agents keep track of potential Next.js issues without interrupting the current user request. Before the final response, the agent evaluates those issues and opens each qualifying report in a separate review form in your default browser. Automatic agent feedback lets coding agents keep track of potential Next.js issues without interrupting the current user request. Before the final response, the agent evaluates those issues and opens each qualifying report in a separate review form in your default browser.
Before opening a form, the agent attempts to anonymize the report by replacing customer and project details with generic descriptions and omitting source code, logs, paths, URLs, secrets, and personal information. Review the draft and remove anything you don't want to include. Nothing is sent until you select **Send feedback**, and agent feedback is separate from [Next.js Telemetry](/telemetry). Before opening a form, the agent attempts to anonymize the report by replacing customer and project details with generic descriptions and omitting source code, logs, paths, URLs, secrets, and personal information. Review the draft and remove anything you don't want to include. Nothing is sent until you select **Send feedback**.
With the option enabled, `next dev` adds the following managed feedback instructions to the agent file when it detects an AI coding agent: With the option enabled, `next dev` adds the following managed feedback instructions to the agent file when it detects an AI coding agent:
...@@ -104,7 +104,7 @@ Immediately before the final response, after all requested work and tool calls a ...@@ -104,7 +104,7 @@ Immediately before the final response, after all requested work and tool calls a
Next.js updates only the content between the agent feedback markers and preserves the rest of the file. Next.js updates only the content between the agent feedback markers and preserves the rest of the file.
Automatic agent feedback is experimental and disabled by default. See the [`agentFeedback` API reference](/docs/app/api-reference/config/next-config-js/agentFeedback) to enable it. Automatic agent feedback is experimental, disabled by default, and available only when [Next.js Telemetry](/telemetry) is enabled. It does not run in CI. See the [`agentFeedback` API reference](/docs/app/api-reference/config/next-config-js/agentFeedback) to enable it.
<span id="step-2-give-agents-runtime-visibility" /> <span id="step-2-give-agents-runtime-visibility" />
......
...@@ -8,7 +8,7 @@ related: ...@@ -8,7 +8,7 @@ related:
- app/api-reference/config/next-config-js/agentRules - app/api-reference/config/next-config-js/agentRules
--- ---
The `experimental.agentFeedback` option controls whether `next dev` creates and updates instructions that let detected AI coding agents prepare Next.js issue reports for your review. It is disabled by default and works independently from [Next.js Telemetry](/telemetry). The `experimental.agentFeedback` option controls whether `next dev` creates and updates instructions that let detected AI coding agents prepare Next.js issue reports for your review. It is disabled by default and requires [Next.js Telemetry](/telemetry) to be enabled. Agent feedback does not run in CI.
Set `experimental.agentFeedback` to `true` to enable this behavior: Set `experimental.agentFeedback` to `true` to enable this behavior:
...@@ -78,6 +78,8 @@ Submitting the form sends the content you reviewed, the agent name, and the Next ...@@ -78,6 +78,8 @@ Submitting the form sends the content you reviewed, the agent name, and the Next
Set `experimental.agentFeedback` to `false` or remove the option to stop `next dev` from creating or updating the managed block. The next `next dev` run removes the existing agent feedback block and preserves all other content. If the file contained only the managed block, Next.js deletes the file instead of leaving it empty. Set `experimental.agentFeedback` to `false` or remove the option to stop `next dev` from creating or updating the managed block. The next `next dev` run removes the existing agent feedback block and preserves all other content. If the file contained only the managed block, Next.js deletes the file instead of leaving it empty.
Disabling Next.js Telemetry also removes the managed feedback block the next time you run `next dev`. The feedback command does not return reporting instructions while telemetry is disabled.
`experimental.agentFeedback` and top-level [`agentRules`](/docs/app/api-reference/config/next-config-js/agentRules) work independently. Changing one option does not affect the other. `experimental.agentFeedback` and top-level [`agentRules`](/docs/app/api-reference/config/next-config-js/agentRules) work independently. Changing one option does not affect the other.
## Version History ## Version History
......
...@@ -9,7 +9,8 @@ describe('loadAgentFeedbackInstructions', () => { ...@@ -9,7 +9,8 @@ describe('loadAgentFeedbackInstructions', () => {
loadAgentFeedbackInstructions( loadAgentFeedbackInstructions(
{}, {},
async () => true, async () => true,
async () => '# Agent feedback protocol\n' async () => '# Agent feedback protocol\n',
() => true
) )
).resolves.toBe('# Agent feedback protocol\n') ).resolves.toBe('# Agent feedback protocol\n')
}) })
...@@ -23,19 +24,33 @@ describe('loadAgentFeedbackInstructions', () => { ...@@ -23,19 +24,33 @@ describe('loadAgentFeedbackInstructions', () => {
expect(readProtocol).not.toHaveBeenCalled() expect(readProtocol).not.toHaveBeenCalled()
}) })
it('does not check the remote gate when feedback is locally disabled', async () => {
const isEnabled = jest.fn(async () => true)
const readProtocol = jest.fn(async () => '# Agent feedback protocol\n')
await expect(
loadAgentFeedbackInstructions({}, isEnabled, readProtocol, () => false)
).resolves.toBeNull()
expect(isEnabled).not.toHaveBeenCalled()
expect(readProtocol).not.toHaveBeenCalled()
})
it('returns dry-run instructions without checking the remote gate', async () => { it('returns dry-run instructions without checking the remote gate', async () => {
const isEnabled = jest.fn(async () => false) const isEnabled = jest.fn(async () => false)
const isLocallyEnabled = jest.fn(() => false)
await expect( await expect(
loadAgentFeedbackInstructions( loadAgentFeedbackInstructions(
{ dryRun: true }, { dryRun: true },
isEnabled, isEnabled,
async () => '# Agent feedback protocol\n' async () => '# Agent feedback protocol\n',
isLocallyEnabled
) )
).resolves.toBe( ).resolves.toBe(
'# Dry run\n\nUse the protocol below to prepare each qualifying report draft and encode its review URL, but do not open a browser tab. Print each review URL for inspection instead. Do not clear the feedback candidate queue or mark the reporting pass complete.\n\n# Agent feedback protocol\n' '# Dry run\n\nUse the protocol below to prepare each qualifying report draft and encode its review URL, but do not open a browser tab. Print each review URL for inspection instead. Do not clear the feedback candidate queue or mark the reporting pass complete.\n\n# Agent feedback protocol\n'
) )
expect(isEnabled).not.toHaveBeenCalled() expect(isEnabled).not.toHaveBeenCalled()
expect(isLocallyEnabled).not.toHaveBeenCalled()
}) })
it('fails closed when the protocol cannot be read', async () => { it('fails closed when the protocol cannot be read', async () => {
...@@ -45,16 +60,22 @@ describe('loadAgentFeedbackInstructions', () => { ...@@ -45,16 +60,22 @@ describe('loadAgentFeedbackInstructions', () => {
async () => true, async () => true,
async () => { async () => {
throw new Error('protocol unavailable') throw new Error('protocol unavailable')
} },
() => true
) )
).resolves.toBeNull() ).resolves.toBeNull()
}) })
it('propagates feedback status errors', async () => { it('propagates feedback status errors', async () => {
await expect( await expect(
loadAgentFeedbackInstructions({}, async () => { loadAgentFeedbackInstructions(
throw new Error('network unavailable') {},
}) async () => {
throw new Error('network unavailable')
},
undefined,
() => true
)
).rejects.toThrow('network unavailable') ).rejects.toThrow('network unavailable')
}) })
}) })
......
import { readFile } from 'fs/promises' import { readFile } from 'fs/promises'
import path from 'path' import path from 'path'
import { isAgentFeedbackEnabled } from './agent-feedback-status' import { isAgentFeedbackEnabled } from './agent-feedback-status'
import { isCI } from '../../server/ci-info'
import { Telemetry } from '../../telemetry/storage'
const AGENT_FEEDBACK_PROTOCOL_PATH = path.join( const AGENT_FEEDBACK_PROTOCOL_PATH = path.join(
__dirname, __dirname,
...@@ -8,6 +10,7 @@ const AGENT_FEEDBACK_PROTOCOL_PATH = path.join( ...@@ -8,6 +10,7 @@ const AGENT_FEEDBACK_PROTOCOL_PATH = path.join(
) )
type IsEnabled = () => Promise<boolean> type IsEnabled = () => Promise<boolean>
type IsLocallyEnabled = () => boolean
type ReadProtocol = () => Promise<string> type ReadProtocol = () => Promise<string>
export interface AgentFeedbackInstructionsOptions { export interface AgentFeedbackInstructionsOptions {
dryRun?: boolean dryRun?: boolean
...@@ -27,9 +30,10 @@ export async function loadAgentFeedbackInstructions( ...@@ -27,9 +30,10 @@ export async function loadAgentFeedbackInstructions(
options: AgentFeedbackInstructionsOptions = {}, options: AgentFeedbackInstructionsOptions = {},
isEnabled: IsEnabled = isAgentFeedbackEnabled, isEnabled: IsEnabled = isAgentFeedbackEnabled,
readProtocol: ReadProtocol = () => readProtocol: ReadProtocol = () =>
readFile(AGENT_FEEDBACK_PROTOCOL_PATH, 'utf8') readFile(AGENT_FEEDBACK_PROTOCOL_PATH, 'utf8'),
isLocallyEnabled: IsLocallyEnabled = isAgentFeedbackLocallyEnabled
): Promise<string | null> { ): Promise<string | null> {
if (!options.dryRun && !(await isEnabled())) { if (!options.dryRun && (!isLocallyEnabled() || !(await isEnabled()))) {
return null return null
} }
...@@ -41,6 +45,15 @@ export async function loadAgentFeedbackInstructions( ...@@ -41,6 +45,15 @@ export async function loadAgentFeedbackInstructions(
} }
} }
function isAgentFeedbackLocallyEnabled(): boolean {
if (isCI) return false
return new Telemetry({
distDir: path.join(process.cwd(), '.next'),
skipNotify: true,
}).isEnabled
}
export async function agentFeedbackInstructionsCli( export async function agentFeedbackInstructionsCli(
options: AgentFeedbackInstructionsOptions = {}, options: AgentFeedbackInstructionsOptions = {},
loadInstructions: LoadInstructions = loadAgentFeedbackInstructions loadInstructions: LoadInstructions = loadAgentFeedbackInstructions
......
...@@ -45,6 +45,7 @@ import { isIPv6 } from './is-ipv6' ...@@ -45,6 +45,7 @@ import { isIPv6 } from './is-ipv6'
import { AsyncCallbackSet } from './async-callback-set' import { AsyncCallbackSet } from './async-callback-set'
import type { NextServer } from '../next' import type { NextServer } from '../next'
import { durationToString } from '../../build/duration-to-string' import { durationToString } from '../../build/duration-to-string'
import { isCI } from '../ci-info'
const debug = setupDebug('next:start-server') const debug = setupDebug('next:start-server')
let startServerSpan: Span | undefined let startServerSpan: Span | undefined
...@@ -518,16 +519,28 @@ export async function startServer( ...@@ -518,16 +519,28 @@ export async function startServer(
`Removed agent rules from ${files} because \`agentRules\` is disabled.` `Removed agent rules from ${files} because \`agentRules\` is disabled.`
) )
logAgentFileSync( if (!isCI) {
await syncAgentFeedbackForDev( const { traceGlobals } =
dir, require('../../trace/shared') as typeof import('../../trace/shared')
initResult.agentFeedback === true const telemetry = traceGlobals.get('telemetry') as
), | InstanceType<typeof import('../../telemetry/storage').Telemetry>
(files) => | undefined
`Generated agent feedback instructions in ${files}. Set \`experimental.agentFeedback: false\` in next.config to disable.`, const agentFeedbackConfigured = initResult.agentFeedback === true
(files) => const telemetryEnabled = telemetry?.isEnabled === true
`Removed agent feedback instructions from ${files} because \`experimental.agentFeedback\` is disabled.`
) logAgentFileSync(
await syncAgentFeedbackForDev(
dir,
agentFeedbackConfigured && telemetryEnabled
),
(files) =>
`Generated agent feedback instructions in ${files}. Set \`experimental.agentFeedback: false\` in next.config to disable.`,
(files) =>
agentFeedbackConfigured
? `Removed agent feedback instructions from ${files} because Next.js Telemetry is disabled.`
: `Removed agent feedback instructions from ${files} because \`experimental.agentFeedback\` is disabled.`
)
}
} }
handlersReady() handlersReady()
......
...@@ -59,7 +59,13 @@ export class Telemetry { ...@@ -59,7 +59,13 @@ export class Telemetry {
private queue: Set<Promise<RecordObject>> private queue: Set<Promise<RecordObject>>
constructor({ distDir }: { distDir: string }) { constructor({
distDir,
skipNotify = false,
}: {
distDir: string
skipNotify?: boolean
}) {
// Read in the constructor so that .env can be loaded before reading // Read in the constructor so that .env can be loaded before reading
const { NEXT_TELEMETRY_DISABLED, NEXT_TELEMETRY_DEBUG } = process.env const { NEXT_TELEMETRY_DISABLED, NEXT_TELEMETRY_DEBUG } = process.env
this.NEXT_TELEMETRY_DISABLED = NEXT_TELEMETRY_DISABLED this.NEXT_TELEMETRY_DISABLED = NEXT_TELEMETRY_DISABLED
...@@ -78,7 +84,9 @@ export class Telemetry { ...@@ -78,7 +84,9 @@ export class Telemetry {
this.sessionId = randomBytes(32).toString('hex') this.sessionId = randomBytes(32).toString('hex')
this.queue = new Set() this.queue = new Set()
this.notify() if (!skipNotify) {
this.notify()
}
} }
private notify = () => { private notify = () => {
......
...@@ -27,6 +27,17 @@ const NO_AGENT_ENV = { ...@@ -27,6 +27,17 @@ const NO_AGENT_ENV = {
COPILOT_GITHUB_TOKEN: '', COPILOT_GITHUB_TOKEN: '',
} }
const NON_CI_ENV = {
CI: '',
CONTINUOUS_INTEGRATION: '',
BUILD_NUMBER: '',
RUN_ID: '',
GITHUB_ACTIONS: '',
NOW_BUILDER: '',
STACK: '',
NEXT_TELEMETRY_DISABLED: '',
}
/** /**
* The canonical block as the version under test generates it, * The canonical block as the version under test generates it,
* obtained by running the real generator into a temp dir — the test * obtained by running the real generator into a temp dir — the test
...@@ -178,7 +189,7 @@ describe('agent-rules auto-generate on next dev (AGENTS.md exists without marker ...@@ -178,7 +189,7 @@ describe('agent-rules auto-generate on next dev (AGENTS.md exists without marker
describe('agent-feedback auto-generate on next dev (enabled)', () => { describe('agent-feedback auto-generate on next dev (enabled)', () => {
const { next } = nextTestSetup({ const { next } = nextTestSetup({
files: __dirname, files: __dirname,
env: { CLAUDECODE: '1' }, env: { ...NON_CI_ENV, CLAUDECODE: '1' },
nextConfig: { nextConfig: {
experimental: { experimental: {
agentFeedback: true, agentFeedback: true,
...@@ -226,7 +237,7 @@ describe('agent-feedback auto-generate on next dev (enabled)', () => { ...@@ -226,7 +237,7 @@ describe('agent-feedback auto-generate on next dev (enabled)', () => {
describe('agent-feedback auto-generate on next dev (agentRules: false)', () => { describe('agent-feedback auto-generate on next dev (agentRules: false)', () => {
const { next } = nextTestSetup({ const { next } = nextTestSetup({
files: __dirname, files: __dirname,
env: { CLAUDECODE: '1' }, env: { ...NON_CI_ENV, CLAUDECODE: '1' },
nextConfig: { nextConfig: {
agentRules: false, agentRules: false,
experimental: { experimental: {
...@@ -249,7 +260,7 @@ describe('agent-feedback auto-generate on next dev (agentRules: false)', () => { ...@@ -249,7 +260,7 @@ describe('agent-feedback auto-generate on next dev (agentRules: false)', () => {
describe('agent-feedback auto-generate on next dev (no agent)', () => { describe('agent-feedback auto-generate on next dev (no agent)', () => {
const { next } = nextTestSetup({ const { next } = nextTestSetup({
files: __dirname, files: __dirname,
env: NO_AGENT_ENV, env: { ...NO_AGENT_ENV, ...NON_CI_ENV },
nextConfig: { nextConfig: {
agentRules: false, agentRules: false,
experimental: { experimental: {
...@@ -267,7 +278,7 @@ describe('agent-feedback auto-generate on next dev (no agent)', () => { ...@@ -267,7 +278,7 @@ describe('agent-feedback auto-generate on next dev (no agent)', () => {
describe('agent-feedback auto-generate on next dev (disabled)', () => { describe('agent-feedback auto-generate on next dev (disabled)', () => {
const { next } = nextTestSetup({ const { next } = nextTestSetup({
files: __dirname, files: __dirname,
env: { CLAUDECODE: '1' }, env: { ...NON_CI_ENV, CLAUDECODE: '1' },
skipStart: true, skipStart: true,
}) })
...@@ -294,7 +305,7 @@ describe('agent-feedback auto-generate on next dev (disabled)', () => { ...@@ -294,7 +305,7 @@ describe('agent-feedback auto-generate on next dev (disabled)', () => {
describe('agent-rules auto-generate on next dev (disabled with existing blocks)', () => { describe('agent-rules auto-generate on next dev (disabled with existing blocks)', () => {
const { next } = nextTestSetup({ const { next } = nextTestSetup({
files: __dirname, files: __dirname,
env: NO_AGENT_ENV, env: { ...NO_AGENT_ENV, ...NON_CI_ENV },
nextConfig: { nextConfig: {
agentRules: false, agentRules: false,
experimental: { experimental: {
...@@ -323,3 +334,43 @@ describe('agent-rules auto-generate on next dev (disabled with existing blocks)' ...@@ -323,3 +334,43 @@ describe('agent-rules auto-generate on next dev (disabled with existing blocks)'
expect(content).toContain(AGENT_FEEDBACK_MARKER) expect(content).toContain(AGENT_FEEDBACK_MARKER)
}) })
}) })
describe('agent-feedback auto-generate on next dev (telemetry disabled)', () => {
const { next } = nextTestSetup({
files: __dirname,
env: {
...NON_CI_ENV,
CLAUDECODE: '1',
NEXT_TELEMETRY_DISABLED: '1',
},
nextConfig: {
agentRules: false,
experimental: {
agentFeedback: true,
},
},
})
it('does not create feedback instructions', async () => {
await next.fetch('/')
expect(fs.existsSync(path.join(next.testDir, 'AGENTS.md'))).toBe(false)
})
})
describe('agent-feedback auto-generate on next dev (CI)', () => {
const { next } = nextTestSetup({
files: __dirname,
env: { CLAUDECODE: '1', CI: '1' },
nextConfig: {
agentRules: false,
experimental: {
agentFeedback: true,
},
},
})
it('does not create feedback instructions', async () => {
await next.fetch('/')
expect(fs.existsSync(path.join(next.testDir, 'AGENTS.md'))).toBe(false)
})
})
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment