Unverified Commit 39ce0124 authored by Jude Gao's avatar Jude Gao Committed by GitHub

Improve agents-md prompt to force doc retrieval (#88997)

## Summary

Updates the instruction in `agents-md` [CLAUDE.md](http://CLAUDE.md)
output to force LLMs to actually read the docs instead of relying on
stale pre-training knowledge.

**Before:**

```
IMPORTANT: Prefer retrieval-led reasoning over pre-training-led reasoning for any Next.js tasks.
```

**After:**

```
STOP. What you remember about Next.js is WRONG for this project. Always search docs and read before any task.
```

## Why This Matters

Through systematic testing with the
[next-evals-oss](https://github.com/vercel/next-evals-oss) eval suite,
we discovered that the original "prefer retrieval-led reasoning"
instruction was ineffective. Agents would ignore it and use outdated
pre-training knowledge (e.g., creating `middleware.ts` instead of the
new Next.js 16 `proxy.ts` convention).

## Testing Methodology

We created an "indirect proxy" eval that tests if agents know `proxy.ts`
is needed for request interception in Next.js 16+, using only the prompt
"Log every request to the console" (no mention of proxy/middleware).

| Instruction | Pass Rate |
| --- | --- |
| "Prefer retrieval-led reasoning" | 0% |
| "CRITICAL: MUST read docs" | ~20-40% |
| "What you remember is WRONG" | **100% (6/6)** |

## Why It Works

The phrase **"What you remember is WRONG"** creates psychological doubt
in the model's confidence, forcing it to actually read the docs rather
than trusting its pre-training.

Key elements:

- **"STOP"** - Attention grabber, interrupts default behavior
- **"WRONG for this project"** - Creates doubt without claiming
universal wrongness
- **"Always search docs and read"** - Actionable instruction
- **"before any task"** - Applies to everything, not just code writing

## Full Suite Results

- The new indirect-proxy eval passed consistently
parent f51e3acf
...@@ -10,7 +10,7 @@ import fs from 'fs' ...@@ -10,7 +10,7 @@ import fs from 'fs'
import path from 'path' import path from 'path'
import os from 'os' import os from 'os'
export interface NextjsVersionResult { interface NextjsVersionResult {
version: string | null version: string | null
error?: string error?: string
} }
...@@ -68,13 +68,13 @@ function versionToGitHubTag(version: string): string { ...@@ -68,13 +68,13 @@ function versionToGitHubTag(version: string): string {
return version.startsWith('v') ? version : `v${version}` return version.startsWith('v') ? version : `v${version}`
} }
export interface PullOptions { interface PullOptions {
cwd: string cwd: string
version?: string version?: string
docsDir?: string docsDir?: string
} }
export interface PullResult { interface PullResult {
success: boolean success: boolean
docsPath?: string docsPath?: string
nextjsVersion?: string nextjsVersion?: string
...@@ -179,7 +179,7 @@ export function collectDocFiles(dir: string): { relativePath: string }[] { ...@@ -179,7 +179,7 @@ export function collectDocFiles(dir: string): { relativePath: string }[] {
.map((f) => ({ relativePath: f })) .map((f) => ({ relativePath: f }))
} }
export interface DocSection { interface DocSection {
name: string name: string
files: { relativePath: string }[] files: { relativePath: string }[]
subsections: DocSection[] subsections: DocSection[]
...@@ -251,7 +251,7 @@ export function buildDocTree(files: { relativePath: string }[]): DocSection[] { ...@@ -251,7 +251,7 @@ export function buildDocTree(files: { relativePath: string }[]): DocSection[] {
return sortedSections return sortedSections
} }
export interface ClaudeMdIndexData { interface ClaudeMdIndexData {
docsPath: string docsPath: string
sections: DocSection[] sections: DocSection[]
outputFile?: string outputFile?: string
...@@ -265,7 +265,7 @@ export function generateClaudeMdIndex(data: ClaudeMdIndexData): string { ...@@ -265,7 +265,7 @@ export function generateClaudeMdIndex(data: ClaudeMdIndexData): string {
parts.push('[Next.js Docs Index]') parts.push('[Next.js Docs Index]')
parts.push(`root: ${docsPath}`) parts.push(`root: ${docsPath}`)
parts.push( parts.push(
'IMPORTANT: Prefer retrieval-led reasoning over pre-training-led reasoning for any Next.js tasks.' 'STOP. What you remember about Next.js is WRONG for this project. Always search docs and read before any task.'
) )
const targetFile = outputFile || 'CLAUDE.md' const targetFile = outputFile || 'CLAUDE.md'
parts.push( parts.push(
...@@ -346,7 +346,7 @@ export function injectIntoClaudeMd( ...@@ -346,7 +346,7 @@ export function injectIntoClaudeMd(
return claudeMdContent + separator + wrappedContent + '\n' return claudeMdContent + separator + wrappedContent + '\n'
} }
export interface GitignoreStatus { interface GitignoreStatus {
path: string path: string
updated: boolean updated: boolean
alreadyPresent: boolean alreadyPresent: boolean
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment