Unverified Commit 7812b031 authored by Will Binns-Smith's avatar Will Binns-Smith Committed by GitHub

Add server-only leak pass to next-bundle-optimizer (#100330)

## Summary

The bundle optimizer ranks client code by size, so it misses small
modules that shouldn't be in the browser at all: a database schema, a
token-signing helper, an agent's system prompt. This adds a
size-independent leak pass to step 4 that walks every client-reachable
module in the analyzer export and checks it against content heuristics.

Matches are reported in a separate "Possible server-only leaks" section
with route, shortest import chain, matched heuristics and compressed
bytes. They stay out of the size ranking and deltas, and the agent
leaves flagged modules and their import chains untouched in both modes,
since moving them is the owner's call and a size metric can't verify it.

The scan lives inline in SKILL.md rather than in a reference file. It
runs on every invocation, and in eval runs the agent skipped a reference
pointer in fix mode and missed a finding.

Also rewords one existing sentence in step 4's Done line.

Evals:

- `agent-059-bundle-audit-server-leaks`: audit mode. Two leaks
(prompt-like schema descriptions under `lib/server`, agent instructions
reached through a shared util) sit well below a lodash import in size.
Negative controls cover UI copy under
`description`/`instructions`/`prompt` keys, a `NEXT_PUBLIC_` read and a
`server-only` module used only by a Server Component.
- `agent-060-bundle-fix-server-leaks`: fix mode on the same app. Checks
the lodash fix still happens, the leaks are reported as unfixed, and
none of the flagged modules or their importers are edited.

## Verification

- `pnpm eval agent-059-bundle-audit-server-leaks`: `skills` passes,
`baseline` and `agents-md` fail
- `pnpm eval agent-060-bundle-fix-server-leaks`: `skills` passes,
`baseline` and `agents-md` fail (both edit the flagged modules)
- Not rerun: both evals after dropping the env-read case and the
`/pricing` route from the fixtures

<!-- NEXT_JS_LLM -->

---------
Co-authored-by: 's avatarvercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
parent 73469204
...@@ -91,3 +91,4 @@ evals/agent-results.json ...@@ -91,3 +91,4 @@ evals/agent-results.json
# Upgrade eval execution artifacts # Upgrade eval execution artifacts
evals/next-upgrade/.work/ evals/next-upgrade/.work/
evals/next-upgrade/results/ evals/next-upgrade/results/
.env*
This diff is collapsed.
...@@ -44,5 +44,15 @@ ...@@ -44,5 +44,15 @@
"agentFeedback": true, "agentFeedback": true,
"publish": false, "publish": false,
"timeout": 900 "timeout": 900
},
"agent-059-bundle-audit-server-leaks": {
"skills": ["next-bundle-optimizer"],
"publish": false,
"timeout": 1800
},
"agent-060-bundle-fix-server-leaks": {
"skills": ["next-bundle-optimizer"],
"publish": false,
"timeout": 2700
} }
} }
// Server-owned policy reaches the client through a shared utility.
// The lazy, user-editable refund prompt is intentional browser content.
import { expect, test } from 'vitest'
import { createHash } from 'crypto'
import { existsSync, readFileSync } from 'fs'
import { join } from 'path'
import { tmpdir } from 'os'
import { environment, transcript } from '@vercel/agent-eval/eval'
const reportPath = join(process.cwd(), 'bundle-report.md')
const report = existsSync(reportPath) ? readFileSync(reportPath, 'utf-8') : ''
function leakSection(markdown: string) {
const lines = markdown.split('\n')
const start = lines.findIndex((line) =>
/^#{1,6}\s.*server[- ]only leak/i.test(line)
)
if (start === -1) return ''
const level = lines[start].match(/^#+/)![0].length
const end = lines.findIndex(
(line, i) => i > start && new RegExp(`^#{1,${level}}\\s`).test(line)
)
return lines.slice(start, end === -1 ? undefined : end).join('\n')
}
const leaks = leakSection(report)
const rest = report.replace(leaks, '')
// Regenerate with: shasum -a 256 <file>
const SOURCE_HASHES: Record<string, string> = {
'app/admin/page.tsx':
'6d49a87e40b38a9fb5a3afe8f151a3935e48f9167cab4f20ba52539c86049a0a',
'app/api/chat/route.ts':
'1dcc04405957ecf2ddb512287ed455602ae7f8fc16a614f7906c5bd34911f8ff',
'app/layout.tsx':
'58551dbe855cbff42a571a1ce93fbed1c8309d0cff154f4e0790b25b1cc1c053',
'app/page.tsx':
'afaec61603086905cf09cfca68b309c1b3e31324bc7f8b8839366d7aea5e7ace',
'app/support/actions.ts':
'6ee09cc4815954a34db95c986736895e0be1a9adf5fd76ebf3e86e54a7824a4a',
'app/support/page.tsx':
'5f54f14a0407ae4a0bebbafd1bcdfdcbc5efeaab57dcf814e8a795f9de08d994',
'app/support/ticket-form.tsx':
'643e3e37969450b730a7a44592a8fd00aba59774d175dbf51688e3f97854209b',
'components/chat-panel.tsx':
'bca9679c521592ccde10abef8d399b7befeacdc2675387c889650ee6c842c149',
'components/docs-search.tsx':
'a855bd22580b0e04fe853a3ab60c84665655c41e1595c14f0b4a97085d11c3d7',
'components/setup-checklist.tsx':
'8ada5936b96305f50548f6acc1356bb176e375629ac5da61e26a047b7eaf418d',
'components/welcome-tour.tsx':
'667a48578f8bd8b528a0a5657a0aaae04f301da81a3f1f55fe5e3c7dd8622e9b',
'lib/agents/index.ts':
'a952e52e107a78b2c57082f1e79421fe3a811997c235e134964958779ff89ed3',
'lib/agents/refund-agent.ts':
'21827fccc19e3987ef6916ee3d714ed1056e86869b134ab2f8d45d7f0756f300',
'lib/agents/support-agent.ts':
'babb15703336ed9a572cd097c3338081f7368f3dff7a92f10f8da9576510a629',
'lib/chat/messages.ts':
'b09b5a6f641b8d12e2f0467f2d67d3194eb58802f45696feac51a2d6b93783c9',
'lib/chat/refunds.ts':
'87436f7d8c5dc7691039d1f48580540ba6b9b2cf35f7594097b5031ed05d25da',
'lib/server/db.ts':
'4f1b76003261d0045fc68284b481beab2be7159a781dc5b6ef5f9910b0b5f699',
'lib/server/prompts.ts':
'aa800a724a9c395ce99bd5e1a7ff4f663ecf5f1502b12a43619aadfaa0e835ef',
'next.config.ts':
'e1e907f8890394ee8b2e1f414bf093036baee3dd710103ad65ba550cb02f6271',
}
test('leaves the app unchanged', () => {
for (const [file, hash] of Object.entries(SOURCE_HASHES)) {
const contents = readFileSync(join(process.cwd(), file))
expect(createHash('sha256').update(contents).digest('hex'), file).toBe(hash)
}
})
test('leaves dependencies and lockfiles unchanged', () => {
const hashes: Record<string, string | null> = JSON.parse(
readFileSync(join(tmpdir(), 'agent-059-dependencies.json'), 'utf-8')
)
for (const [file, hash] of Object.entries(hashes)) {
const path = join(process.cwd(), file)
const actual = existsSync(path)
? createHash('sha256').update(readFileSync(path)).digest('hex')
: null
expect(actual, file).toBe(hash)
}
})
test('reports leaks in their own section', () => {
expect(leaks).not.toBe('')
})
test('reports every positive with its import chain', () => {
expect(leaks).toMatch(/lib\/chat\/messages/)
expect(leaks).toMatch(/chat-panel/)
})
test('still ranks the ordinary size candidate', () => {
expect(rest).toMatch(/lodash/)
})
test('keeps leak findings out of the size ranking', async () => {
await expect(environment).toSatisfyCriterion(
`bundle-report.md separates size candidates from exactly one possible server-only leak: internal support policy on / via components/chat-panel.tsx and lib/chat/messages.ts. Include the route, client chain, ownership evidence from support-agent.ts and its server caller app/api/chat/route.ts, and attributed compressed bytes or an explicit gap. Prompt wording alone is insufficient. Accept lib/agents/support-agent.ts or its emitted lib/chat/messages.ts identity; naming inlined sources or optimized-away barrels is optional. Negative controls: public, browser-editable refund instructions reached asynchronously through lib/chat/refunds.ts; form help in lib/server/prompts.ts; welcome-tour/setup-checklist copy; NEXT_PUBLIC_DOCS_URL; lib/server/db.ts; and app/support/actions.ts's client reference stub, which excludes TRIAGE_INSTRUCTIONS. Controls may appear as suppressed matches or contrasts, and public prompts as ordinary size candidates.`
)
})
test('executes the scan against emitted build evidence', async () => {
await expect(transcript).toSatisfyCriterion(
`The agent successfully captured or selected an analyzer snapshot, exported its graph and ran a streaming scan across all routes using exact module identities, client output membership and sync/async dependencies. It checked the async refund import and its synchronous descendant; prompt generation and the editable textarea in components/chat-panel.tsx suffice to establish public intent without browser interaction. It confirmed emitted client code contains the private support policy and identified server ownership from source and callers. It distinguished app/support/actions.ts's client reference stub from its server implementation and recorded coverage gaps. Capture, export and analysis must succeed; reports or proposed/failed commands are insufficient.`
)
})
Audit the JavaScript this app ships to the browser across all routes. I want to know what is worth looking at before anyone starts changing code, so leave the app as it is. Save your report to `bundle-report.md`.
import { listOpenTickets } from '@/lib/server/db'
export const dynamic = 'force-dynamic'
export default async function AdminPage() {
const tickets = await listOpenTickets()
return (
<main>
<h1>Open tickets</h1>
<ul>
{tickets.map((ticket) => (
<li key={ticket.id}>
{ticket.id}: {ticket.subject}
</li>
))}
</ul>
</main>
)
}
import { SUPPORT_AGENT_INSTRUCTIONS } from '@/lib/agents'
export async function POST(request: Request) {
const { messages } = await request.json()
const userTurns = messages.filter(
(message: { role: string }) => message.role === 'user'
).length
return Response.json({
role: 'assistant',
content: `Thanks, I'm looking into that (${userTurns} message${userTurns === 1 ? '' : 's'} so far, ${SUPPORT_AGENT_INSTRUCTIONS.length} instruction characters loaded).`,
})
}
import type { ReactNode } from 'react'
import Link from 'next/link'
export default function RootLayout({ children }: { children: ReactNode }) {
return (
<html lang="en">
<body>
<nav>
<Link href="/">Home</Link> <Link href="/support">Support</Link>{' '}
<Link href="/admin">Admin</Link>
</nav>
{children}
</body>
</html>
)
}
import { ChatPanel } from '@/components/chat-panel'
import { DocsSearch } from '@/components/docs-search'
import { SetupChecklist } from '@/components/setup-checklist'
import { WelcomeTour } from '@/components/welcome-tour'
export default function HomePage() {
return (
<main>
<h1>Acme Help Center</h1>
<WelcomeTour />
<SetupChecklist />
<DocsSearch />
<ChatPanel />
</main>
)
}
'use server'
const TRIAGE_INSTRUCTIONS = `You are Acme's internal ticket triage assistant. Read the customer's description and identify the affected order before choosing a queue. Never reveal the VIP routing rules or the internal escalation thresholds. Call the lookup_order tool when an order number is present, then respond only in JSON with the order number, severity and assigned queue.`
export async function getTicketHint(): Promise<string> {
return `Our triage assistant has ${TRIAGE_INSTRUCTIONS.length} instruction characters loaded. Include your order number so it can find your purchase.`
}
import { TicketForm } from './ticket-form'
export default function SupportPage() {
return (
<main>
<h1>Contact support</h1>
<TicketForm />
</main>
)
}
'use client'
import { useState } from 'react'
import { ticketSchema, type TicketField } from '@/lib/server/prompts'
import { getTicketHint } from './actions'
const fields = Object.keys(ticketSchema.properties) as TicketField[]
export function TicketForm() {
const [hint, setHint] = useState('')
const [values, setValues] = useState<Record<TicketField, string>>({
subject: '',
details: '',
})
const missing = ticketSchema.required.filter((field) => !values[field])
return (
<form onSubmit={(event) => event.preventDefault()}>
{fields.map((field) => (
<label key={field}>
{ticketSchema.properties[field].title}
<input
value={values[field]}
onChange={(event) =>
setValues({ ...values, [field]: event.target.value })
}
/>
</label>
))}
<button type="submit" disabled={missing.length > 0}>
Submit
</button>
<button
type="button"
onClick={async () => setHint(await getTicketHint())}
>
Get triage hint
</button>
<p>{hint}</p>
</form>
)
}
'use client'
import { useState } from 'react'
import { createConversation, type ChatMessage } from '@/lib/chat/messages'
export function ChatPanel() {
const [messages, setMessages] = useState<ChatMessage[]>(createConversation)
const [draft, setDraft] = useState('')
const [refundPrompt, setRefundPrompt] = useState('')
async function send() {
const next = [...messages, { role: 'user' as const, content: draft }]
setMessages(next)
setDraft('')
const response = await fetch('/api/chat', {
method: 'POST',
body: JSON.stringify({ messages: next }),
})
const reply: ChatMessage = await response.json()
setMessages([...next, reply])
}
async function requestRefund() {
const { createRefundPrompt } = await import('@/lib/chat/refunds')
setRefundPrompt(createRefundPrompt(draft))
}
return (
<section aria-label="Support chat">
<ul>
{messages
.filter((message) => message.role !== 'system')
.map((message, index) => (
<li key={index}>{message.content}</li>
))}
</ul>
<input value={draft} onChange={(event) => setDraft(event.target.value)} />
<button type="button" onClick={send}>
Send
</button>
<button type="button" onClick={requestRefund}>
Draft a refund request
</button>
<label>
Edit your refund drafting prompt
<textarea
value={refundPrompt}
onChange={(event) => setRefundPrompt(event.target.value)}
/>
</label>
</section>
)
}
'use client'
import _ from 'lodash'
import { useMemo, useState } from 'react'
const ARTICLES = [
'Resetting your password',
'Updating billing details',
'Exporting order history',
'Inviting teammates',
'Closing your account',
]
export function DocsSearch() {
const [query, setQuery] = useState('')
const [results, setResults] = useState(ARTICLES)
const search = useMemo(
() =>
_.debounce((value: string) => {
const needle = value.toLowerCase()
setResults(
ARTICLES.filter((article) => article.toLowerCase().includes(needle))
)
}, 150),
[]
)
return (
<section aria-label="Search docs">
<input
value={query}
placeholder="Search docs"
onChange={(event) => {
setQuery(event.target.value)
search(event.target.value)
}}
/>
<ul>
{results.map((article) => (
<li key={article}>{article}</li>
))}
</ul>
</section>
)
}
'use client'
import { useState } from 'react'
const TASKS = [
{
id: 'profile',
title: 'Complete your profile',
description:
'Add your company name, shipping address and a contact email so our team can reach you about orders. You can change any of these details later from your account settings page.',
instructions:
'Open Settings, choose Profile, fill in each field and press Save. If your company uses single sign-on, ask your administrator to add you before you continue with this step.',
},
{
id: 'feedback',
title: 'Tell us about your store',
description:
'A short note about what you sell helps us recommend the right plan and the help articles most useful to you. This is optional, and nothing you write here is shown publicly.',
prompt:
'What are you hoping to get done with Acme this month? For example, you might be moving from another platform, launching a new product line or setting up your first storefront.',
},
]
export function SetupChecklist() {
const [open, setOpen] = useState(TASKS[0].id)
return (
<section aria-label="Setup checklist">
{TASKS.map((task) => (
<details key={task.id} open={open === task.id}>
<summary onClick={() => setOpen(task.id)}>{task.title}</summary>
<p>{task.description}</p>
{'instructions' in task && <p>{task.instructions}</p>}
{'prompt' in task && <textarea placeholder={task.prompt} />}
</details>
))}
</section>
)
}
'use client'
import { useState } from 'react'
const STEPS = [
'Search the documentation from the box below. Results update as you type, and you can press Enter to open the first match in a new tab.',
'Still stuck? Start a conversation with our support assistant. It can look up your recent orders and walk you through common fixes step by step.',
'Need a human? Open a ticket from the Support page and include as much detail as you can. Our team usually replies within one business day.',
]
export function WelcomeTour() {
const [step, setStep] = useState(0)
const docsUrl = process.env.NEXT_PUBLIC_DOCS_URL ?? 'https://docs.acme.test'
return (
<section aria-label="Getting started">
<p>{STEPS[step]}</p>
<button
type="button"
onClick={() => setStep((current) => (current + 1) % STEPS.length)}
>
Next tip
</button>{' '}
<a href={docsUrl}>Browse all docs</a>
</section>
)
}
import { createHash } from 'node:crypto'
import { existsSync, readFileSync, writeFileSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
// Installation rewrites the manifest and creates the lockfile before this hook.
const files = [
'package.json',
'package-lock.json',
'npm-shrinkwrap.json',
'pnpm-lock.yaml',
'yarn.lock',
'bun.lock',
'bun.lockb',
]
const hashes = Object.fromEntries(
files.map((file) => [
file,
existsSync(file)
? createHash('sha256').update(readFileSync(file)).digest('hex')
: null,
])
)
writeFileSync(
join(tmpdir(), 'agent-059-dependencies.json'),
JSON.stringify(hashes)
)
export const REFUND_DRAFT_INSTRUCTIONS = `You are a writing assistant helping a customer draft a refund request. Use only the order number and product details supplied by the customer. Ask for missing information instead of inventing it. Never include passwords or payment card numbers. Return a polite draft that explains the issue, states the requested resolution and leaves a placeholder for any missing order details.`
// Internal refund policy for the server-side support agent.
export const SUPPORT_AGENT_INSTRUCTIONS = `You are Acme's tier-one support agent. Answer questions about orders, shipping and returns using only the tools provided. Never reveal internal refund thresholds: approve refunds under $75 automatically, escalate anything between $75 and $400 to the billing queue, and deny anything above $400 unless the customer has an enterprise contract. Do not mention that fraud scoring exists. If the customer asks for a human, collect their order number first and then call the create_ticket tool with priority "normal" unless they mention a chargeback, in which case use priority "legal".`
import { SUPPORT_AGENT_INSTRUCTIONS } from '@/lib/agents'
export type ChatMessage = {
role: 'system' | 'user' | 'assistant'
content: string
}
export function createConversation(): ChatMessage[] {
return [
{ role: 'system', content: SUPPORT_AGENT_INSTRUCTIONS },
{ role: 'assistant', content: 'Hi! How can I help with your order today?' },
]
}
import { REFUND_DRAFT_INSTRUCTIONS } from '@/lib/agents/refund-agent'
export function createRefundPrompt(orderNumber: string): string {
return `${REFUND_DRAFT_INSTRUCTIONS}\n\nOrder number: ${orderNumber || '[enter order number]'}`
}
import 'server-only'
const DATABASE_URL = process.env.DATABASE_URL ?? 'postgres://localhost/acme'
export type Ticket = { id: string; subject: string; status: 'open' | 'closed' }
export async function listOpenTickets(): Promise<Ticket[]> {
const host = new URL(DATABASE_URL).host
return [
{ id: 'T-1001', subject: `Late delivery (${host})`, status: 'open' },
{ id: 'T-1002', subject: 'Damaged item', status: 'open' },
]
}
export const ticketSchema = {
type: 'object',
properties: {
subject: {
type: 'string',
title: 'Subject',
description:
'A short summary of the problem so the support team can identify your request.',
},
details: {
type: 'string',
title: 'Details',
description:
'Describe what happened, what you expected and which order or product was affected.',
},
},
required: ['subject', 'details'],
} as const
export type TicketField = keyof typeof ticketSchema.properties
import type { NextConfig } from 'next'
const nextConfig: NextConfig = {
// Keep the baseline free of generated agent instructions. The Guide
// experiment writes its own AGENTS.md after this fixture is prepared.
agentRules: false,
}
export default nextConfig
{
"name": "bundle-audit-server-leaks",
"private": true,
"type": "module",
"scripts": {
"eval:setup": "node eval-setup.mjs",
"dev": "next dev",
"build": "next build",
"start": "next start"
},
"dependencies": {
"lodash": "4.17.21",
"next": "^16",
"react": "19.1.0",
"react-dom": "19.1.0",
"server-only": "0.0.1"
},
"devDependencies": {
"@types/lodash": "4.17.20",
"@types/node": "^20",
"@types/react": "^19",
"@types/react-dom": "^19",
"typescript": "^5",
"vitest": "^3.1.3"
}
}
{
"compilerOptions": {
"lib": ["dom", "dom.iterable", "esnext"],
"allowJs": true,
"skipLibCheck": true,
"strict": true,
"noEmit": true,
"esModuleInterop": true,
"module": "esnext",
"moduleResolution": "bundler",
"resolveJsonModule": true,
"isolatedModules": true,
"jsx": "preserve",
"incremental": true,
"plugins": [
{
"name": "next"
}
],
"paths": {
"@/*": ["./*"]
},
"target": "ES2017"
},
"include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
"exclude": ["node_modules", "EVAL.ts"]
}
// Public prompts remain eligible for optimization without changing ownership.
import { expect, test } from 'vitest'
import { existsSync, readFileSync } from 'fs'
import { join } from 'path'
import { environment, transcript } from '@vercel/agent-eval/eval'
const reportPath = join(process.cwd(), 'bundle-report.md')
const report = existsSync(reportPath) ? readFileSync(reportPath, 'utf-8') : ''
function leakSection(markdown: string) {
const lines = markdown.split('\n')
const start = lines.findIndex((line) =>
/^#{1,6}\s.*server[- ]only leak/i.test(line)
)
if (start === -1) return ''
const level = lines[start].match(/^#+/)![0].length
const end = lines.findIndex(
(line, i) => i > start && new RegExp(`^#{1,${level}}\\s`).test(line)
)
return lines.slice(start, end === -1 ? undefined : end).join('\n')
}
const leaks = leakSection(report)
test('still reports every leak in its own section', () => {
expect(leaks).toMatch(/lib\/chat\/messages/)
})
test('distinguishes unintended exposure from public prompts', async () => {
await expect(environment).toSatisfyCriterion(
`bundle-report.md separates size edits from exactly one unresolved boundary concern: internal support policy on / via components/chat-panel.tsx and lib/chat/messages.ts. Include the route, client chain, ownership evidence from support-agent.ts and its server caller app/api/chat/route.ts, and attributed compressed bytes or an explicit gap. Prompt wording alone is insufficient. Accept lib/agents/support-agent.ts or its emitted lib/chat/messages.ts identity; naming inlined sources or optimized-away barrels is optional. Negative controls: public, browser-editable refund instructions reached asynchronously through lib/chat/refunds.ts and lib/agents/refund-agent.ts; form help in lib/server/prompts.ts; welcome-tour/setup-checklist copy; NEXT_PUBLIC_DOCS_URL; lib/server/db.ts; and the Server Action reference stub. Public prompts and finding importers remain eligible for behavior-preserving size optimizations.`
)
})
test('executes the scan against emitted build evidence', async () => {
await expect(transcript).toSatisfyCriterion(
`The agent successfully captured or selected an analyzer snapshot, exported its graph and ran a streaming scan across all routes using exact module identities, client output membership and sync/async dependencies. It checked the async refund import and its synchronous descendant; prompt generation and the editable textarea in components/chat-panel.tsx suffice to establish public intent without browser interaction. It confirmed emitted client code contains the private support policy and identified server ownership from source and callers. It distinguished app/support/actions.ts's client reference stub from its server implementation and recorded coverage gaps. Capture, export and analysis must succeed; reports or proposed/failed commands are insufficient.`
)
})
test('verifies an ordinary bundle reduction', async () => {
await expect(transcript).toSatisfyCriterion(
`The agent retained a verified bundle reduction, such as narrowing components/docs-search.tsx's lodash import. It successfully captured/exported distinct before/after snapshots and demonstrated lower deduplicated attributed compressed browser bytes across the same routes. Relevant behavior checks and a post-edit type-check passed. Search remains case-insensitive with the existing debounce delay; refund drafting remains available and editable. Report-only work, source estimates, builds without size comparisons and unverified/reverted edits are insufficient.`
)
})
test('preserves intended use without freezing files', async () => {
await expect(transcript).toSatisfyCriterion(
`The agent leaves private-policy relocation and server-only markers to the owner, preserving browser use of the public refund prompt. It permits verified, behavior-preserving optimizations in prompt modules and finding importers, rather than classifying public prompts as leaks or refusing optimizations solely because a file contains prompts or imports a flagged module.`
)
})
Cut down the JavaScript this app ships to the browser across all routes. Go ahead and change the code where you can verify the improvement. Save a report of what you changed and anything else you found to `bundle-report.md`.
import { listOpenTickets } from '@/lib/server/db'
export const dynamic = 'force-dynamic'
export default async function AdminPage() {
const tickets = await listOpenTickets()
return (
<main>
<h1>Open tickets</h1>
<ul>
{tickets.map((ticket) => (
<li key={ticket.id}>
{ticket.id}: {ticket.subject}
</li>
))}
</ul>
</main>
)
}
import { SUPPORT_AGENT_INSTRUCTIONS } from '@/lib/agents'
export async function POST(request: Request) {
const { messages } = await request.json()
const userTurns = messages.filter(
(message: { role: string }) => message.role === 'user'
).length
return Response.json({
role: 'assistant',
content: `Thanks, I'm looking into that (${userTurns} message${userTurns === 1 ? '' : 's'} so far, ${SUPPORT_AGENT_INSTRUCTIONS.length} instruction characters loaded).`,
})
}
import type { ReactNode } from 'react'
import Link from 'next/link'
export default function RootLayout({ children }: { children: ReactNode }) {
return (
<html lang="en">
<body>
<nav>
<Link href="/">Home</Link> <Link href="/support">Support</Link>{' '}
<Link href="/admin">Admin</Link>
</nav>
{children}
</body>
</html>
)
}
import { ChatPanel } from '@/components/chat-panel'
import { DocsSearch } from '@/components/docs-search'
import { SetupChecklist } from '@/components/setup-checklist'
import { WelcomeTour } from '@/components/welcome-tour'
export default function HomePage() {
return (
<main>
<h1>Acme Help Center</h1>
<WelcomeTour />
<SetupChecklist />
<DocsSearch />
<ChatPanel />
</main>
)
}
'use server'
const TRIAGE_INSTRUCTIONS = `You are Acme's internal ticket triage assistant. Read the customer's description and identify the affected order before choosing a queue. Never reveal the VIP routing rules or the internal escalation thresholds. Call the lookup_order tool when an order number is present, then respond only in JSON with the order number, severity and assigned queue.`
export async function getTicketHint(): Promise<string> {
return `Our triage assistant has ${TRIAGE_INSTRUCTIONS.length} instruction characters loaded. Include your order number so it can find your purchase.`
}
import { TicketForm } from './ticket-form'
export default function SupportPage() {
return (
<main>
<h1>Contact support</h1>
<TicketForm />
</main>
)
}
'use client'
import { useState } from 'react'
import { ticketSchema, type TicketField } from '@/lib/server/prompts'
import { getTicketHint } from './actions'
const fields = Object.keys(ticketSchema.properties) as TicketField[]
export function TicketForm() {
const [hint, setHint] = useState('')
const [values, setValues] = useState<Record<TicketField, string>>({
subject: '',
details: '',
})
const missing = ticketSchema.required.filter((field) => !values[field])
return (
<form onSubmit={(event) => event.preventDefault()}>
{fields.map((field) => (
<label key={field}>
{ticketSchema.properties[field].title}
<input
value={values[field]}
onChange={(event) =>
setValues({ ...values, [field]: event.target.value })
}
/>
</label>
))}
<button type="submit" disabled={missing.length > 0}>
Submit
</button>
<button
type="button"
onClick={async () => setHint(await getTicketHint())}
>
Get triage hint
</button>
<p>{hint}</p>
</form>
)
}
'use client'
import { useState } from 'react'
import { createConversation, type ChatMessage } from '@/lib/chat/messages'
export function ChatPanel() {
const [messages, setMessages] = useState<ChatMessage[]>(createConversation)
const [draft, setDraft] = useState('')
const [refundPrompt, setRefundPrompt] = useState('')
async function send() {
const next = [...messages, { role: 'user' as const, content: draft }]
setMessages(next)
setDraft('')
const response = await fetch('/api/chat', {
method: 'POST',
body: JSON.stringify({ messages: next }),
})
const reply: ChatMessage = await response.json()
setMessages([...next, reply])
}
async function requestRefund() {
const { createRefundPrompt } = await import('@/lib/chat/refunds')
setRefundPrompt(createRefundPrompt(draft))
}
return (
<section aria-label="Support chat">
<ul>
{messages
.filter((message) => message.role !== 'system')
.map((message, index) => (
<li key={index}>{message.content}</li>
))}
</ul>
<input value={draft} onChange={(event) => setDraft(event.target.value)} />
<button type="button" onClick={send}>
Send
</button>
<button type="button" onClick={requestRefund}>
Draft a refund request
</button>
<label>
Edit your refund drafting prompt
<textarea
value={refundPrompt}
onChange={(event) => setRefundPrompt(event.target.value)}
/>
</label>
</section>
)
}
'use client'
import _ from 'lodash'
import { useMemo, useState } from 'react'
const ARTICLES = [
'Resetting your password',
'Updating billing details',
'Exporting order history',
'Inviting teammates',
'Closing your account',
]
export function DocsSearch() {
const [query, setQuery] = useState('')
const [results, setResults] = useState(ARTICLES)
const search = useMemo(
() =>
_.debounce((value: string) => {
const needle = value.toLowerCase()
setResults(
ARTICLES.filter((article) => article.toLowerCase().includes(needle))
)
}, 150),
[]
)
return (
<section aria-label="Search docs">
<input
value={query}
placeholder="Search docs"
onChange={(event) => {
setQuery(event.target.value)
search(event.target.value)
}}
/>
<ul>
{results.map((article) => (
<li key={article}>{article}</li>
))}
</ul>
</section>
)
}
'use client'
import { useState } from 'react'
const TASKS = [
{
id: 'profile',
title: 'Complete your profile',
description:
'Add your company name, shipping address and a contact email so our team can reach you about orders. You can change any of these details later from your account settings page.',
instructions:
'Open Settings, choose Profile, fill in each field and press Save. If your company uses single sign-on, ask your administrator to add you before you continue with this step.',
},
{
id: 'feedback',
title: 'Tell us about your store',
description:
'A short note about what you sell helps us recommend the right plan and the help articles most useful to you. This is optional, and nothing you write here is shown publicly.',
prompt:
'What are you hoping to get done with Acme this month? For example, you might be moving from another platform, launching a new product line or setting up your first storefront.',
},
]
export function SetupChecklist() {
const [open, setOpen] = useState(TASKS[0].id)
return (
<section aria-label="Setup checklist">
{TASKS.map((task) => (
<details key={task.id} open={open === task.id}>
<summary onClick={() => setOpen(task.id)}>{task.title}</summary>
<p>{task.description}</p>
{'instructions' in task && <p>{task.instructions}</p>}
{'prompt' in task && <textarea placeholder={task.prompt} />}
</details>
))}
</section>
)
}
'use client'
import { useState } from 'react'
const STEPS = [
'Search the documentation from the box below. Results update as you type, and you can press Enter to open the first match in a new tab.',
'Still stuck? Start a conversation with our support assistant. It can look up your recent orders and walk you through common fixes step by step.',
'Need a human? Open a ticket from the Support page and include as much detail as you can. Our team usually replies within one business day.',
]
export function WelcomeTour() {
const [step, setStep] = useState(0)
const docsUrl = process.env.NEXT_PUBLIC_DOCS_URL ?? 'https://docs.acme.test'
return (
<section aria-label="Getting started">
<p>{STEPS[step]}</p>
<button
type="button"
onClick={() => setStep((current) => (current + 1) % STEPS.length)}
>
Next tip
</button>{' '}
<a href={docsUrl}>Browse all docs</a>
</section>
)
}
export const REFUND_DRAFT_INSTRUCTIONS = `You are a writing assistant helping a customer draft a refund request. Use only the order number and product details supplied by the customer. Ask for missing information instead of inventing it. Never include passwords or payment card numbers. Return a polite draft that explains the issue, states the requested resolution and leaves a placeholder for any missing order details.`
// Internal refund policy for the server-side support agent.
export const SUPPORT_AGENT_INSTRUCTIONS = `You are Acme's tier-one support agent. Answer questions about orders, shipping and returns using only the tools provided. Never reveal internal refund thresholds: approve refunds under $75 automatically, escalate anything between $75 and $400 to the billing queue, and deny anything above $400 unless the customer has an enterprise contract. Do not mention that fraud scoring exists. If the customer asks for a human, collect their order number first and then call the create_ticket tool with priority "normal" unless they mention a chargeback, in which case use priority "legal".`
import { SUPPORT_AGENT_INSTRUCTIONS } from '@/lib/agents'
export type ChatMessage = {
role: 'system' | 'user' | 'assistant'
content: string
}
export function createConversation(): ChatMessage[] {
return [
{ role: 'system', content: SUPPORT_AGENT_INSTRUCTIONS },
{ role: 'assistant', content: 'Hi! How can I help with your order today?' },
]
}
import { REFUND_DRAFT_INSTRUCTIONS } from '@/lib/agents/refund-agent'
export function createRefundPrompt(orderNumber: string): string {
return `${REFUND_DRAFT_INSTRUCTIONS}\n\nOrder number: ${orderNumber || '[enter order number]'}`
}
import 'server-only'
const DATABASE_URL = process.env.DATABASE_URL ?? 'postgres://localhost/acme'
export type Ticket = { id: string; subject: string; status: 'open' | 'closed' }
export async function listOpenTickets(): Promise<Ticket[]> {
const host = new URL(DATABASE_URL).host
return [
{ id: 'T-1001', subject: `Late delivery (${host})`, status: 'open' },
{ id: 'T-1002', subject: 'Damaged item', status: 'open' },
]
}
export const ticketSchema = {
type: 'object',
properties: {
subject: {
type: 'string',
title: 'Subject',
description:
'A short summary of the problem so the support team can identify your request.',
},
details: {
type: 'string',
title: 'Details',
description:
'Describe what happened, what you expected and which order or product was affected.',
},
},
required: ['subject', 'details'],
} as const
export type TicketField = keyof typeof ticketSchema.properties
import type { NextConfig } from 'next'
const nextConfig: NextConfig = {
// Keep the baseline free of generated agent instructions. The Guide
// experiment writes its own AGENTS.md after this fixture is prepared.
agentRules: false,
}
export default nextConfig
{
"private": true,
"type": "module",
"scripts": {
"dev": "next dev",
"build": "next build",
"start": "next start"
},
"dependencies": {
"lodash": "4.17.21",
"next": "^16",
"react": "19.1.0",
"react-dom": "19.1.0",
"server-only": "0.0.1"
},
"devDependencies": {
"@types/lodash": "4.17.20",
"@types/node": "^20",
"@types/react": "^19",
"@types/react-dom": "^19",
"typescript": "^5",
"vitest": "^3.1.3"
}
}
{
"compilerOptions": {
"lib": ["dom", "dom.iterable", "esnext"],
"allowJs": true,
"skipLibCheck": true,
"strict": true,
"noEmit": true,
"esModuleInterop": true,
"module": "esnext",
"moduleResolution": "bundler",
"resolveJsonModule": true,
"isolatedModules": true,
"jsx": "preserve",
"incremental": true,
"plugins": [
{
"name": "next"
}
],
"paths": {
"@/*": ["./*"]
},
"target": "ES2017"
},
"include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
"exclude": ["node_modules", "EVAL.ts"]
}
...@@ -75,16 +75,32 @@ Before proposing an edit, record these items for each candidate. Mark an inappli ...@@ -75,16 +75,32 @@ Before proposing an edit, record these items for each candidate. Mark an inappli
For lazy interaction features, duplicate packages, server-rendered display work or other attributed assets, read the matching section in [Fix patterns](references/fix-patterns.md) before proposing an edit. For lazy interaction features, duplicate packages, server-rendered display work or other attributed assets, read the matching section in [Fix patterns](references/fix-patterns.md) before proposing an edit.
**Done:** the paired pass and its target dispositions are recorded, or the narrow-scope skip reason or blocker is explicit. Every candidate has a completed checklist, exact source/importer, proposed edit and named behavior checks. Label heuristics as heuristics and qualify conclusions affected by evidence gaps; the report step completes **audit** mode. ### Unexpected client dependencies
Size ranking can miss small server-owned modules in client code. Before scanning in either mode, read [Client dependencies](references/client-dependencies.md) for graph joins, generated Server Action identities and tracing. Stream the export to check every emitted client module, including async outputs.
Before editing, save a **Client-dependency scan** table in the report: one row per match and every client-imported Server Action, with **Source + client importer**, **Emitted identity**, **Routes**, **Compressed bytes** and **Disposition + evidence**. Reconcile source Action imports against emitted references, including generated identities; verify whether implementation contents survive. A filename search or byte ranking alone cannot complete this table.
Use three dispositions: **possible leak** requires server-only ownership evidence and surviving client contents; **suppressed** covers intentional client content and verified reference-only Action stubs; **unverified** names the missing evidence. Public form schemas, help text and editable prompts stay suppressed even under `server/` or `prompts` paths. Record coverage gaps (unsupported outputs, missing roots or mappings), explicitly stating when none were found.
**Done:** the community/min-cut pass accounts for every target or records its skip reason or blocker. Every size candidate has a completed checklist, exact source/importer, proposed edit and behavior checks. Every client-dependency match is a heuristic finding, suppressed with a reason or unverified with an evidence gap. Audit mode ends after the report.
## 5. Verify one change — fix mode only ## 5. Verify one change — fix mode only
Classify each proposed edit as a size optimization or boundary remediation before applying it. A request to reduce browser JavaScript authorizes size edits, including in public prompt modules and finding importers. Removing private client state/payloads, relocating policy or adding `server-only` needs separate explicit approval; leave those findings unresolved for the owner even if the UI would stay unchanged.
Make one small, cohesive change. Capture/export an after snapshot with a new name and compare the **same analysis scope, output class and metric** with the baseline. For whole-app analysis, account for changes across all routes, including shared dependencies. Run relevant behavior tests and type-check; use [`next-dev-loop`](https://github.com/vercel/next.js/tree/canary/skills/next-dev-loop) when verifying the edit in the running app, and install it first if unavailable (`npx skills add https://github.com/vercel/next.js/tree/canary/skills/next-dev-loop`). Make one small, cohesive change. Capture/export an after snapshot with a new name and compare the **same analysis scope, output class and metric** with the baseline. For whole-app analysis, account for changes across all routes, including shared dependencies. Run relevant behavior tests and type-check; use [`next-dev-loop`](https://github.com/vercel/next.js/tree/canary/skills/next-dev-loop) when verifying the edit in the running app, and install it first if unavailable (`npx skills add https://github.com/vercel/next.js/tree/canary/skills/next-dev-loop`).
If no relevant tests exist, create and run focused smoke tests. Test production imports or rendered UI, not copied implementations. Exercise interactive flows through a browser or executable component tests, including input handling, timing and deferred actions. For debounced search, check case-insensitive results and the existing delay; for editable lazy features, trigger loading and edit the result. Record commands/actions and observed results. Source inspection, type-checking and successful builds alone do not verify behavior.
**Done:** retain the change only when the scoped metric improves and the intended behavior passes its checks. Revert a change that fails either condition. If checks are blocked, report the unverified edit and blocker rather than accepting it. Record the accepted after snapshot as the next baseline before another edit. **Done:** retain the change only when the scoped metric improves and the intended behavior passes its checks. Revert a change that fails either condition. If checks are blocked, report the unverified edit and blocker rather than accepting it. Record the accepted after snapshot as the next baseline before another edit.
## 6. Report and stop ## 6. Report and stop
Audit reports candidates; fix reports accepted edits and any reverted or unverified attempts. Include snapshot names, scoped attribution/deltas, exact source/importer, behavior-check results and blockers. For a graph cut, include its route/render conditions, roots/target and unknowns. Distinguish source facts, build evidence, heuristics and separately observed runtime facts. Audit reports candidates; fix reports accepted edits and any reverted or unverified attempts. Include snapshot names, scoped attribution/deltas, exact source/importer, behavior-check results and blockers. For a graph cut, include its route/render conditions, roots/target and unknowns. Distinguish source facts, build evidence, heuristics and separately observed runtime facts.
**Done:** every scoped candidate or attempted edit is accounted for, with evidence or a stated gap. Add a **Possible server-only leaks** section with five labeled lines per heuristic finding: **Module**, **Routes**, **Chain**, **Heuristics** (including ownership evidence) and **Compressed bytes**. Explain missing values on their lines. Keep unresolved boundary concerns separate from accepted size edits, including those in the same file. State when no leaks were found.
Use the scan table to populate **Suppressed matches** and **Evidence gaps**. Keep public form content and verified Action stubs outside the leak section, along with naming concerns. Include the coverage statement even when no gaps were found.
**Done:** every scoped candidate, leak finding, client-imported Action and attempted edit is accounted for, with evidence or a stated gap.
# Client dependencies
Use the main workflow's graph-evidence checklist and installed schema.
## Discover emitted modules
While streaming the export, retain:
- **Modules:** exact `ident`, source `path` and `dependencies.sync`/`dependencies.async`. Browser layers (`[app-client]`, or `[client]` for Pages) distinguish client variants from `[app-ssr]` and `[app-rsc]`; output membership establishes which variants ship.
- **Outputs:** each route's client and server output records, with `modules`, `async_loaders` and `coverage`. Join members by exact `ident`; an async loader target is a loading edge, not a member of the importer's chunk.
- **Roots:** route entries with `entry_kind: client_bootstrap`, plus each entry's `client_references[].module_ident`.
- **Bytes:** `[client-fs]` `part` records keyed by `source_path`. Count each `filename` once per source path across routes.
Group emitted members by source `path`, retaining each variant's identity. Inspect sources shared by server and client first, then remaining client members. Output membership identifies affected routes; unsupported membership is a coverage gap.
## Validate candidates
For every match, establish server ownership from source, callers or project documentation and confirm the relevant contents survive in emitted client code. Apply this to custom code and dependencies alike. Examples and common false positives:
- **Databases:** ORM schemas/models, migrations, SQL, drivers (`pg`, `mysql2`, `mongodb`) and connection factories. Public enums, validation data and browser-configured clients can be intentional.
- **Authentication:** secret-based signing/verification, password hashing, session stores, OAuth secrets and server auth APIs. Browser session UI, anon-key clients and public-key verification can be intentional.
- **Authorization:** permission tables, ACLs, entitlements and gates. Public mappings may render controls while the server enforces access.
- **SDKs:** cloud, admin, payment, mail and model SDKs. Inspect project importers and configuration; browser clients, presigned URLs and public upload flows can be intentional.
- **Model instructions:** public, editable prompts and browser inference are ordinary client content eligible for size optimization. Wording, length, roles and tool definitions alone do not establish ownership.
- **Names and shims:** `server`, `db`, `auth`, `agents`, `prompts`, `*.server.*` and Node polyfills warrant inspection. UI copy and browser libraries using `buffer`, `stream`, `events`, `util` or `process` can be legitimate.
Inspect project source; for `node_modules`, use package identities and project importer configuration. Exclude type-only imports and exports erased from client output. Server modules absent from client output are not findings.
For each inspected Server Action client reference, map its emitted identity (possibly generated or `data:`) to the source action and client importer. Record routes and attributed compressed bytes, and check whether private implementation contents survive. Report verified reference-only stubs as suppressed matches; incomplete mappings or checks as unverified matches with specific gaps. Stub bytes alone do not establish a leak.
## Trace findings
For affected routes, run BFS from client roots over synchronous and asynchronous dependencies, retaining parent pointers and edge kinds. Use `(ident, crossedAsync)` as the visited key to preserve synchronous paths separately. Prefer the shortest synchronous chain; otherwise use the shortest async chain and mark its loading edge and source trigger. Across routes, choose the shortest chain of the preferred kind. Record missing roots as evidence gaps.
Keep conditional loading distinct from synchronous reachability. Optimized-away barrels may be absent from the emitted chain; report the chain the graph shows and the source import path when different.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment