Unverified Commit 5d9ab72c authored by Jiwon Choi's avatar Jiwon Choi Committed by GitHub

Add `next upgrade --ai` and security vulnerability coverage (#98562)

> [!TIP]
> Recommended to review commit by commit.

This PR adds `next upgrade --experimental-ai="security"` flag (alias
`--ai`), which is targeted to help users leverage agents to upgrade
their app to the safe major version when their app's Next.js version has
any security advisories.

Once the command is ran from the user, Next.js will detect the installed
agent harness in user's device, currently limited to Codex and Claude,
and will proceed with starting an agent session once approved. If it is
called within an agent session, the work will continue off within that
agent.

`next upgrade --ai` simply does two things:

- prepare the relevant context to temporary dir
- print hand off prompt, guiding to read those context

The context will guide the agent to run relevant codemods and migration
checklist to proceed. This PR is a base core of the workflow, and will
have wrappers of entry point around this. Also, will add "latest" and
"future" as follow up, which will cover the app to be always latest, and
adopt the future defaults like Cache Components.

This PR also sets up the evals infra and adds evals.
parent 1ae49102
...@@ -301,6 +301,8 @@ fn next_owned_ignores( ...@@ -301,6 +301,8 @@ fn next_owned_ignores(
rcstr!("**/next/dist/compiled/webpack/*"), rcstr!("**/next/dist/compiled/webpack/*"),
rcstr!("**/node_modules/webpack5/**/*"), rcstr!("**/node_modules/webpack5/**/*"),
rcstr!("**/next/dist/server/lib/route-resolver*"), rcstr!("**/next/dist/server/lib/route-resolver*"),
// Upgrade workflows are CLI-only and are not needed by production servers.
rcstr!("**/next/dist/lib/upgrade/**/*"),
// The testmode interceptors bundle reads its HTTP parser WASM with a // The testmode interceptors bundle reads its HTTP parser WASM with a
// dynamic path, making the tracer include the bundle's whole // dynamic path, making the tracer include the bundle's whole
// directory. Test proxying is not supported in standalone output, so // directory. Test proxying is not supported in standalone output, so
......
---
title: Complete an agentic Next.js upgrade
description: Agent workflow for repository preflight, codemods, repairs, verification, and delivery of an agentic Next.js upgrade.
# Experimental agent workflow. Not ready for indexing; keep this document marked as draft.
version: draft
---
Each section has a checklist of items to complete. Read line by line and complete
each item before moving on to the next section.
## 1. Check for duplicates
Before reading another guide or changing files, complete every item:
- [ ] Identify the repository provider from `git remote -v`, then use its CLI or
API to list every open pull request. For GitHub, run
`gh pr list --state open --limit 100 --json number,title,body,url,headRefName`.
- [ ] Inspect the body and diff of likely matches. For GitHub, run
`gh pr diff <number>`.
- [ ] Check local and remote branches and commits for an equivalent upgrade.
Look for unmarked upgrades and these markers:
```md
<!-- next-upgrade: <type>; path="." -->
<!-- next-upgrade: <type>; path="apps/web" -->
```
Do not substitute Git history for the provider lookup. If any check is unavailable,
fails, or finds existing work, stop before changing files and report it. Continue
to step 2 only after every check completes and finds no equivalent work.
## 2. Make a checklist
- [ ] Read `./codemods.md` and the applicable `./version-<major>.md` files for
every crossed major, or the target major for a same-major upgrade.
- [ ] For version 14 and later, include the **Review migration checklist**
section. For version 13 and below, derive the checklist from the Pages
Router guide under `docs/02-pages/02-guides/upgrading/`.
- [ ] Stop if a required guide is missing.
## 3. Upgrade and repair
- [ ] Run the exact command prepared for this upgrade:
```text
<codemod-command>
```
- [ ] Run the codemod once, directly to the exact target. Do not install
intermediate Next.js versions or rerun the codemod for each crossed
major.
- [ ] If a required option is unavailable, correct the tool version or stop.
- [ ] Complete the codemod's manual steps, resolve its review markers, and
complete every applicable migration item. Keep only changes required by
the final target.
## 4. Verify
**This is the most important step.**
- [ ] Review the checklist created in step 2 and verify that every item has been
completed. Iterate until every item is addressed as applied or blocked
with a reason.
- [ ] Run the repository's checks, build when supported, and test affected
runtime behavior.
## 5. Commit and deliver
- [ ] Commit the final diff in ascending major order, with one commit for each
crossed major. Each commit contains that major's surviving final-target
changes. Do not add transitional changes solely to make an intermediate
version work. Each commit message must include its source and target
versions and explain the change as if it were a PR description, so the
commit can later be split into its own PR.
- [ ] With permission, recheck open pull requests for duplicates and publish one
draft PR with the app marker:
```md
<!-- next-upgrade: <type>; path="." -->
<!-- next-upgrade: <type>; path="apps/web" -->
```
- [ ] If the upgrade is incomplete, report completed work, the blocker, and how
to resume.
...@@ -35,3 +35,12 @@ bun add next@next-14 react@18 react-dom@18 && bun add eslint-config-next@next-14 ...@@ -35,3 +35,12 @@ bun add next@next-14 react@18 react-dom@18 && bun add eslint-config-next@next-14
- The `next/server` import for `ImageResponse` was renamed to `next/og`. A [codemod is available](/docs/app/guides/upgrading/codemods#next-og-import) to safely and automatically rename your imports. - The `next/server` import for `ImageResponse` was renamed to `next/og`. A [codemod is available](/docs/app/guides/upgrading/codemods#next-og-import) to safely and automatically rename your imports.
- The `@next/font` package has been fully removed in favor of the built-in `next/font`. A [codemod is available](/docs/app/guides/upgrading/codemods#built-in-next-font) to safely and automatically rename your imports. - The `@next/font` package has been fully removed in favor of the built-in `next/font`. A [codemod is available](/docs/app/guides/upgrading/codemods#built-in-next-font) to safely and automatically rename your imports.
- The WASM target for `next-swc` has been removed. - The WASM target for `next-swc` has been removed.
## Review migration checklist
- [ ] The app uses Node.js 18.17 or later and compatible React 18 dependencies and types.
- [ ] Static exports use `output: 'export'` and `next build` instead of `next export`.
- [ ] `ImageResponse` imports use `next/og` instead of `next/server`.
- [ ] Font imports use `next/font` instead of `@next/font`.
- [ ] The app does not rely on the removed `next-swc` WASM target.
- [ ] The app's supported checks and affected runtime behavior pass.
...@@ -625,3 +625,17 @@ export function middleware(request: NextRequest) { ...@@ -625,3 +625,17 @@ export function middleware(request: NextRequest) {
// ... // ...
} }
``` ```
## Review migration checklist
After applying the changes relevant to your app, review these action items before
you consider the migration complete:
- [ ] React dependencies and types meet the React 19 requirements for the app's router.
- [ ] Async Request API access, including affected helpers and callers, is migrated. Resolve codemod error comments and `UnsafeUnwrapped` casts instead of treating them as completed repairs.
- [ ] Route segment runtime settings no longer use `experimental-edge`.
- [ ] Font imports use `next/font` instead of `@next/font`.
- [ ] Existing `bundlePagesExternals` and `serverComponentsExternalPackages` settings use their documented stable names while preserving package configuration.
- [ ] `NextRequest` `geo` and `ip` consumers use an appropriate source for the actual hosting provider.
- [ ] Apps that relied on automatic Speed Insights instrumentation have an explicit integration if they still need it.
- [ ] The app's supported checks and affected runtime behavior pass. Remaining required failures keep the migration incomplete.
...@@ -801,7 +801,9 @@ const nextConfig = { ...@@ -801,7 +801,9 @@ const nextConfig = {
module.exports = nextConfig module.exports = nextConfig
``` ```
If you specify a `quality` prop not included in the `image.qualities` array, the quality will be coerced to the closest value in `images.qualities`. For example, given the configuration above, a `quality` prop of 80, is coerced to 75. If you specify a `quality` prop not included in the `images.qualities` array, the quality will be coerced to the closest value in `images.qualities`. For example, given the configuration above, a `quality` prop of 80, is coerced to 75.
Review custom image qualities. Add required values to `images.qualities`, and test direct `/_next/image` requests because unsupported `q` values return `400`.
### Local IP Restriction (Breaking change) ### Local IP Restriction (Breaking change)
...@@ -1246,3 +1248,20 @@ For the full migration path, see [Migrating to Cache Components](/docs/app/guide ...@@ -1246,3 +1248,20 @@ For the full migration path, see [Migrating to Cache Components](/docs/app/guide
### `unstable_rootParams` ### `unstable_rootParams`
The `unstable_rootParams` function has been removed. Use [`next/root-params`](/docs/app/api-reference/functions/next-root-params) instead. The `unstable_rootParams` function has been removed. Use [`next/root-params`](/docs/app/api-reference/functions/next-root-params) instead.
## Review migration checklist
After applying the changes relevant to your app, review these action items before
you consider the migration complete:
- [ ] The app meets the Node.js, TypeScript, and React requirements in this guide.
- [ ] Async Request API access and affected callers are migrated. Resolve codemod error comments and `UnsafeUnwrapped` casts; use React `use` for synchronous Client Components where appropriate.
- [ ] Dynamic metadata image functions and sitemap functions handle their asynchronous parameters, and affected generated URLs work.
- [ ] Development and build scripts agree with the effective bundler configuration, including plugins, loaders, aliases, and affected Sass imports.
- [ ] Middleware and proxy changes preserve the required runtime and routing behavior. Edge middleware is not blindly renamed to proxy.
- [ ] Parallel route slots have appropriate defaults, and direct navigation and reloads retain the intended fallback behavior.
- [ ] Affected image requests succeed under the documented restrictions and defaults while preserving the intended quality.
- [ ] Existing cache API calls follow the documented signatures and semantics. React Compiler and Cache Components are not enabled merely to complete the upgrade.
- [ ] Removed APIs and configuration have no remaining consumers. Runtime configuration replacements preserve server and client visibility and the required evaluation time.
- [ ] Scripts and CI no longer invoke `next lint`; linting runs separately where the app requires it.
- [ ] The app's supported checks and affected runtime behavior pass. A green build alone does not close runtime findings.
...@@ -19,6 +19,9 @@ export default defineConfig([ ...@@ -19,6 +19,9 @@ export default defineConfig([
// tsconfig, not repo code — EVAL.ts files may import modules that only // tsconfig, not repo code — EVAL.ts files may import modules that only
// resolve inside the sandbox (e.g. @vercel/agent-eval/eval). // resolve inside the sandbox (e.g. @vercel/agent-eval/eval).
'evals/evals/**/*', 'evals/evals/**/*',
'evals/next-upgrade/evals/**/*',
'evals/next-upgrade/results/**/*',
'evals/next-upgrade/shared/**/*',
'examples/**/*', 'examples/**/*',
'test/**/*', 'test/**/*',
'**/*.d.ts', '**/*.d.ts',
......
...@@ -410,6 +410,9 @@ export default defineConfig([ ...@@ -410,6 +410,9 @@ export default defineConfig([
// tsconfig, not repo code — EVAL.ts files may import modules that only // tsconfig, not repo code — EVAL.ts files may import modules that only
// resolve inside the sandbox (e.g. @vercel/agent-eval/eval). // resolve inside the sandbox (e.g. @vercel/agent-eval/eval).
'evals/evals/**/*', 'evals/evals/**/*',
'evals/next-upgrade/evals/**/*',
'evals/next-upgrade/results/**/*',
'evals/next-upgrade/shared/**/*',
'examples/**/*', 'examples/**/*',
'test/**/*', 'test/**/*',
'**/*.d.ts', '**/*.d.ts',
......
const fs = require('node:fs')
const path = require('node:path')
/** Make the repo's existing vc env pull output available to agent-eval. */
function linkEnvironment(root, directory) {
for (const name of ['.env', '.env.local']) {
const source = path.join(root, name)
const destination = path.join(directory, name)
try {
fs.rmSync(destination, { force: true })
if (fs.existsSync(source)) fs.symlinkSync(source, destination)
} catch {}
}
}
module.exports = { linkEnvironment }
const fs = require('node:fs')
const path = require('node:path')
const { execFileSync } = require('node:child_process')
/** Pack one built workspace package for upload by an eval runner. */
function packPackage(packageDirectory, destination) {
const directory = path.dirname(destination)
fs.mkdirSync(directory, { recursive: true })
const output = execFileSync(
'pnpm',
['pack', '--pack-destination', directory],
{
cwd: packageDirectory,
encoding: 'utf8',
}
)
const produced = output.trim().split('\n').pop()
const source = path.isAbsolute(produced)
? produced
: path.join(directory, produced)
if (source !== destination) fs.renameSync(source, destination)
return destination
}
module.exports = { packPackage }
.tarballs/
results/
.env
.env.local
# Next.js upgrade evals
This suite extends the existing `@vercel/agent-eval` setup. Fixtures use exact old
Next.js versions, while the separately packed candidate provides the global
`next upgrade` command.
## Run
Use the existing [eval credential setup](../README.md#one-time-setup): `vc link`
and `vc env pull` at the repo root. Both runners share environment-file linking
and package packing. Authentication, sandbox selection, native agents, withheld
assertions, judging and result storage belong to `@vercel/agent-eval`.
```sh
pnpm build-all
pnpm eval:upgrade <fixture-name> --dry
NEXT_UPGRADE_EVAL_EXPERIMENT=codex pnpm eval:upgrade <fixture-name>
```
Omit the experiment filter to run Codex and Claude. `--list` lists fixtures without
packing or making model calls. Run one named fixture at a time. Results use the
framework's normal `results/` layout. Fixtures are added by the feature PRs stacked
above this infrastructure.
## Lifecycle
1. Create one temporary Vercel Sandbox snapshot with the agent CLIs.
2. Upload the fixture and establish its git baseline.
3. Install candidate Next.js and codemod packages separately, route npm and npx
upgrade commands to the candidate CLI, then install app dependencies.
4. Snapshot the prepared fixture and fork each selected agent from it.
5. Run each native agent and judge independently. Agent-eval withholds `EVAL.ts`
and captures transcripts and results as usual.
Package archives use the same fixed, overwritten paths as existing evals. Invalid
fixtures fail before execution, and infrastructure failures are retained in the results.
## Adding feature coverage
Feature PRs add ordinary npm app fixtures with exact dependency versions,
`PROMPT.md`, and `EVAL.ts`. Prompts invoke `npx next@canary upgrade --ai`. Feature
PRs own browser setup, repository remotes, advisory responses, codemod routing,
grading, and reference or negative controls. Keep graders and reference solutions
withheld, and retain sandbox or authentication failures as failures.
.next/
*.tsbuildinfo
__agent_eval__/
eval-evidence/
node_modules/
next-env.d.ts
You may edit this application and create local commits. Do not push or create pull requests. Use npm. Preserve the behavior described in README.md.
import { expect, test } from 'vitest'
import { readFileSync } from 'node:fs'
import { join } from 'node:path'
import { securityChecks } from './checks/EVAL'
const { target } = JSON.parse(
readFileSync('/tmp/next-upgrade-eval/security/assessment.json', 'utf8')
) as { target: string }
securityChecks(
'13.5.11',
target,
(app) => {
test('completes the async request API migration', () => {
const source = readFileSync(join(app.cwd, 'lib/viewer.ts'), 'utf8')
expect(source).not.toContain('@next-codemod-')
expect(source).not.toContain('UnsafeUnwrapped')
})
test('preserves request identity between visitors', async () => {
await Promise.all(
[
['Alice', 'fr'],
['Bob', 'de'],
['Guest', 'en'],
].map(async ([name, language]) => {
const headers: Record<string, string> = {
'accept-language': language,
}
if (name !== 'Guest') headers.cookie = `member=${name}`
const [page, api] = await Promise.all([
fetch(app.url, { headers }),
fetch(`${app.url}/api/viewer`, { headers }),
])
expect(page.status).toBe(200)
expect(api.status).toBe(200)
const html = await page.text()
expect(html).toContain(`id="member">${name}<`)
expect(html).toContain(`id="language">${language}<`)
expect(await api.json()).toEqual({ name, language })
})
)
})
},
{
changedFiles: [
'app/page.tsx',
'eslint.config.mjs',
'lib/viewer.ts',
'package.json',
],
migrationGuides: [14, 15],
}
)
Run `npx next@canary upgrade --ai` for this app and follow the instructions.
# Next.js 13 member dashboard
The home page greets the member identified by the `member` cookie and displays
the request's `accept-language` header. Missing cookies show Guest. These values
must stay isolated between visitors. `/api/viewer` returns the same request
identity.
Use `npm run lint`, `npm run typecheck`, `npm run build`, and `npm start` to
check the app.
import { viewer } from '../../../lib/viewer'
export async function GET() {
return Response.json(await viewer())
}
export default function Layout({ children }: { children: React.ReactNode }) {
return (
<html>
<body>{children}</body>
</html>
)
}
import { viewer } from '../lib/viewer'
export default function Page() {
const member = viewer()
return (
<main>
<h1>Member dashboard</h1>
<p id="member">{member.name}</p>
<p id="language">{member.language}</p>
</main>
)
}
../../../shared/security-checks.ts
\ No newline at end of file
import { cookies, headers } from 'next/headers'
export function viewer() {
const cookieStore = cookies()
const headerStore = headers()
return {
name: cookieStore.get('member')?.value || 'Guest',
language: headerStore.get('accept-language') || 'en',
}
}
{
"name": "security-cross-major",
"private": true,
"type": "module",
"scripts": {
"dev": "next dev",
"build": "next build",
"start": "next start",
"lint": "next lint",
"typecheck": "tsc --noEmit"
},
"dependencies": {
"next": "13.5.11",
"react": "18.2.0",
"react-dom": "18.2.0"
},
"devDependencies": {
"@types/node": "20.17.7",
"@types/react": "18.2.79",
"@types/react-dom": "18.2.25",
"eslint": "8.57.1",
"eslint-config-next": "13.5.11",
"typescript": "5.8.3",
"vitest": "3.1.3",
"@vitejs/plugin-react": "4.4.1",
"vite-tsconfig-paths": "5.1.4"
}
}
{
"compilerOptions": {
"target": "ES2017",
"lib": ["dom", "dom.iterable", "esnext"],
"allowJs": true,
"skipLibCheck": true,
"strict": true,
"noEmit": true,
"esModuleInterop": true,
"module": "esnext",
"moduleResolution": "bundler",
"resolveJsonModule": true,
"isolatedModules": true,
"jsx": "preserve",
"incremental": true,
"plugins": [{ "name": "next" }]
},
"include": [
"next-env.d.ts",
"app/**/*.ts",
"app/**/*.tsx",
"lib/**/*.ts",
".next/types/**/*.ts"
],
"exclude": ["node_modules"]
}
.next/
*.tsbuildinfo
__agent_eval__/
eval-evidence/
node_modules/
next-env.d.ts
You may edit this application and create local commits. Do not push or create pull requests. Use npm. Preserve the behavior described in README.md.
import { duplicateSecurityChecks } from './checks/EVAL'
duplicateSecurityChecks('15.5.23', '15.5.24')
Run `npx next@canary upgrade --ai` for this app and follow the instructions.
# Member dashboard
The home page greets the member identified by the `member` cookie and displays
the request's `accept-language` header. Missing cookies show Guest. These values
must stay isolated between visitors. `/api/viewer` returns the same request
identity.
Use `npm run typecheck`, `npm run build`, and `npm start` to check the app.
import { viewer } from '../../../lib/viewer'
export async function GET() {
return Response.json(await viewer())
}
export default function Layout({ children }: { children: React.ReactNode }) {
return (
<html>
<body>{children}</body>
</html>
)
}
import { viewer } from '../lib/viewer'
export default async function Page() {
const member = await viewer()
return (
<main>
<h1>Member dashboard</h1>
<p id="member">{member.name}</p>
<p id="language">{member.language}</p>
</main>
)
}
../../../shared/security-checks.ts
\ No newline at end of file
import { cookies, headers } from 'next/headers'
export async function viewer() {
const [cookieStore, headerStore] = await Promise.all([cookies(), headers()])
return {
name: cookieStore.get('member')?.value || 'Guest',
language: headerStore.get('accept-language') || 'en',
}
}
{
"name": "security-same-major",
"private": true,
"type": "module",
"scripts": {
"dev": "next dev",
"build": "next build",
"start": "next start",
"typecheck": "tsc --noEmit"
},
"dependencies": {
"next": "15.5.23",
"react": "19.1.0",
"react-dom": "19.1.0"
},
"devDependencies": {
"@types/node": "20.17.7",
"@types/react": "19.1.2",
"@types/react-dom": "19.1.2",
"typescript": "5.8.3",
"vitest": "3.1.3",
"@vitejs/plugin-react": "4.4.1",
"vite-tsconfig-paths": "5.1.4"
}
}
{
"compilerOptions": {
"target": "ES2017",
"lib": ["dom", "dom.iterable", "esnext"],
"allowJs": true,
"skipLibCheck": true,
"strict": true,
"noEmit": true,
"esModuleInterop": true,
"module": "esnext",
"moduleResolution": "bundler",
"resolveJsonModule": true,
"isolatedModules": true,
"jsx": "preserve",
"incremental": true,
"plugins": [{ "name": "next" }]
},
"include": [
"next-env.d.ts",
"app/**/*.ts",
"app/**/*.tsx",
"lib/**/*.ts",
".next/types/**/*.ts"
],
"exclude": ["node_modules"]
}
.next/
*.tsbuildinfo
__agent_eval__/
eval-evidence/
node_modules/
next-env.d.ts
You may edit this application and create local commits. Do not push or create pull requests. Use npm. Preserve the behavior described in README.md.
import { expect, test } from 'vitest'
import { securityChecks } from './checks/EVAL'
securityChecks('15.5.23', '15.5.24', (app) => {
test('preserves request identity between visitors', async () => {
await Promise.all(
[
['Alice', 'fr'],
['Bob', 'de'],
['Guest', 'en'],
].map(async ([name, language]) => {
const headers: Record<string, string> = {
'accept-language': language,
}
if (name !== 'Guest') headers.cookie = `member=${name}`
const [page, api] = await Promise.all([
fetch(app.url, { headers }),
fetch(`${app.url}/api/viewer`, { headers }),
])
expect(page.status).toBe(200)
expect(api.status).toBe(200)
const html = await page.text()
expect(html).toContain(`id="member">${name}<`)
expect(html).toContain(`id="language">${language}<`)
expect(await api.json()).toEqual({ name, language })
})
)
})
})
Run `npx next@canary upgrade --ai` for this app and follow the instructions.
# Member dashboard
The home page greets the member identified by the `member` cookie and displays
the request's `accept-language` header. Missing cookies show Guest. These values
must stay isolated between visitors. `/api/viewer` returns the same request
identity.
Use `npm run typecheck`, `npm run build`, and `npm start` to check the app.
import { viewer } from '../../../lib/viewer'
export async function GET() {
return Response.json(await viewer())
}
export default function Layout({ children }: { children: React.ReactNode }) {
return (
<html>
<body>{children}</body>
</html>
)
}
import { viewer } from '../lib/viewer'
export default async function Page() {
const member = await viewer()
return (
<main>
<h1>Member dashboard</h1>
<p id="member">{member.name}</p>
<p id="language">{member.language}</p>
</main>
)
}
../../../shared/security-checks.ts
\ No newline at end of file
import { cookies, headers } from 'next/headers'
export async function viewer() {
const [cookieStore, headerStore] = await Promise.all([cookies(), headers()])
return {
name: cookieStore.get('member')?.value || 'Guest',
language: headerStore.get('accept-language') || 'en',
}
}
{
"name": "security-same-major",
"private": true,
"type": "module",
"scripts": {
"dev": "next dev",
"build": "next build",
"start": "next start",
"typecheck": "tsc --noEmit"
},
"dependencies": {
"next": "15.5.23",
"react": "19.1.0",
"react-dom": "19.1.0"
},
"devDependencies": {
"@types/node": "20.17.7",
"@types/react": "19.1.2",
"@types/react-dom": "19.1.2",
"typescript": "5.8.3",
"vitest": "3.1.3",
"@vitejs/plugin-react": "4.4.1",
"vite-tsconfig-paths": "5.1.4"
}
}
{
"compilerOptions": {
"target": "ES2017",
"lib": ["dom", "dom.iterable", "esnext"],
"allowJs": true,
"skipLibCheck": true,
"strict": true,
"noEmit": true,
"esModuleInterop": true,
"module": "esnext",
"moduleResolution": "bundler",
"resolveJsonModule": true,
"isolatedModules": true,
"jsx": "preserve",
"incremental": true,
"plugins": [{ "name": "next" }]
},
"include": [
"next-env.d.ts",
"app/**/*.ts",
"app/**/*.tsx",
"lib/**/*.ts",
".next/types/**/*.ts"
],
"exclude": ["node_modules"]
}
import { upgradeExperiment } from '../lib/experiment'
export default upgradeExperiment('claude-code')
import { upgradeExperiment } from '../lib/experiment'
export default upgradeExperiment('codex')
#!/usr/bin/env node
import { appendFileSync, existsSync, realpathSync } from 'node:fs'
import { dirname, join } from 'node:path'
import { fileURLToPath, pathToFileURL } from 'node:url'
const tools = dirname(fileURLToPath(import.meta.url))
const args = process.argv.slice(2)
const executable = join(tools, 'next/node_modules/next/dist/bin/next')
if (args[0] === 'upgrade') {
const assessment = join(tools, 'security/assessment.mjs')
if (existsSync(assessment)) await import(pathToFileURL(assessment).href)
process.env.__NEXT_UPGRADE_USE_CURRENT_CLI = '1'
appendFileSync(
join(tools, 'invocations.jsonl'),
JSON.stringify({
args,
executable: realpathSync(executable),
cwd: process.cwd(),
packageRunner: process.env.NEXT_UPGRADE_EVAL_PACKAGE_RUNNER,
requestedPackage: process.env.NEXT_UPGRADE_EVAL_REQUESTED_PACKAGE,
}) + '\n'
)
}
process.argv = [process.execPath, executable, ...args]
await import(pathToFileURL(executable).href)
import type { ExperimentConfig } from '@vercel/agent-eval'
import { setupUpgrade } from './fixture'
import { setupSecurity } from '../security/setup'
export function upgradeExperiment(
harness: 'codex' | 'claude-code'
): ExperimentConfig {
const fixture = process.env.NEXT_UPGRADE_EVAL_CASE
if (!fixture) throw new Error('Select one upgrade eval case')
const security = fixture.startsWith('security-')
return {
agent: `vercel-ai-gateway/${harness}`,
model: harness === 'codex' ? 'openai/gpt-5.6-terra' : 'claude-sonnet-4-6',
judge: {
agent: 'vercel-ai-gateway/claude-code',
model: 'claude-haiku-4-5',
},
evals: fixture,
earlyExit: false,
timeout: 1800,
copyFiles: 'changed',
setup: async (sandbox) => {
const setup = await setupUpgrade(sandbox)
if (security) await setupSecurity(sandbox)
return setup
},
}
}
import { existsSync, readFileSync } from 'node:fs'
import { join } from 'node:path'
import type { Sandbox } from '@vercel/agent-eval'
export const toolsDirectory = '/tmp/next-upgrade-eval'
export async function setupUpgrade(sandbox: Sandbox) {
const run = async (command: string, args: string[]) => {
const result = await sandbox.runCommand(command, args)
if (result.exitCode !== 0)
throw new Error(
`${command} failed during upgrade setup:\n${result.stderr}`
)
return result.stdout.trim()
}
const nextTarball = process.env.NEXT_UPGRADE_EVAL_NEXT_TARBALL
const codemodTarball = process.env.NEXT_UPGRADE_EVAL_CODEMOD_TARBALL
if (!nextTarball || !codemodTarball)
throw new Error(
'Run through pnpm eval:upgrade to provide the candidate packages'
)
const fixture = process.env.NEXT_UPGRADE_EVAL_CASE
if (!fixture) throw new Error('Select one upgrade eval case')
const fixtureDirectory = join(__dirname, '../evals', fixture)
const baselineFiles = Object.fromEntries(
['package-lock.json'].flatMap((name) => {
const file = join(fixtureDirectory, name)
return existsSync(file) ? [[name, readFileSync(file, 'utf8')]] : []
})
)
await sandbox.writeFiles(baselineFiles)
await run('git', ['add', '--force', ...Object.keys(baselineFiles)])
await run('git', ['commit', '--amend', '--no-edit'])
await run('mkdir', ['-p', toolsDirectory])
await sandbox.writeFiles({
// @ts-expect-error agent-eval accepts binary upload at runtime
[`${toolsDirectory}/next.tgz`]: readFileSync(nextTarball),
// @ts-expect-error agent-eval accepts binary upload at runtime
[`${toolsDirectory}/codemod.tgz`]: readFileSync(codemodTarball),
[`${toolsDirectory}/entry.mjs`]: readFileSync(
join(__dirname, 'entry.mjs'),
'utf8'
),
[`${toolsDirectory}/package-runner.mjs`]: readFileSync(
join(__dirname, 'package-runner.mjs'),
'utf8'
),
})
await run('npm', [
'install',
'--prefix',
`${toolsDirectory}/next`,
`${toolsDirectory}/next.tgz`,
])
await run('npm', [
'install',
'--prefix',
`${toolsDirectory}/codemod`,
`${toolsDirectory}/codemod.tgz`,
])
await run('chmod', ['+x', `${toolsDirectory}/entry.mjs`])
const path = await run('sh', ['-c', 'printf %s "$PATH"'])
const bin = `${toolsDirectory}/bin`
const npm = await run('sh', ['-c', 'command -v npm'])
const npx = await run('sh', ['-c', 'command -v npx'])
const nextVersion = await run('node', [
'-p',
`require('${toolsDirectory}/next/node_modules/next/package.json').version`,
])
await run('mkdir', ['-p', bin])
await sandbox.writeFiles({
[`${toolsDirectory}/package-runner.json`]: JSON.stringify({
nextVersion,
npm,
npx,
}),
[join(bin, 'npm')]:
`#!/bin/sh\nexec node ${toolsDirectory}/package-runner.mjs npm "$@"\n`,
[join(bin, 'npx')]:
`#!/bin/sh\nexec node ${toolsDirectory}/package-runner.mjs npx "$@"\n`,
})
await run('chmod', ['+x', join(bin, 'npm'), join(bin, 'npx')])
await run('ln', ['-sf', `${toolsDirectory}/entry.mjs`, join(bin, 'next')])
return { env: { PATH: `${bin}:${path}` } }
}
#!/usr/bin/env node
import { spawnSync } from 'node:child_process'
import { readFileSync } from 'node:fs'
import { dirname, join } from 'node:path'
import { fileURLToPath } from 'node:url'
const tools = dirname(fileURLToPath(import.meta.url))
const config = JSON.parse(
readFileSync(join(tools, 'package-runner.json'), 'utf8')
)
const [runner, ...args] = process.argv.slice(2)
const command = config[runner]
if (!command) throw new Error(`Unsupported package runner: ${runner}`)
function packageInvocation() {
let packageIndex = -1
if (runner === 'npx') {
packageIndex = 0
} else if (runner === 'npm' && ['exec', 'x'].includes(args[0])) {
packageIndex = 1
} else {
return
}
const optionIndex = args.findIndex(
(arg) => arg === '--package' || arg.startsWith('--package=')
)
if (optionIndex !== -1) {
const requestedPackage = args[optionIndex].startsWith('--package=')
? args[optionIndex].slice('--package='.length)
: args[optionIndex + 1]
const separator = args.indexOf('--')
if (separator === -1) return
const [executable, ...invocationArgs] = args.slice(separator + 1)
return { requestedPackage, executable, args: invocationArgs }
}
while (['--', '--yes', '-y'].includes(args[packageIndex])) packageIndex++
const requestedPackage = args[packageIndex]
const invocationArgs = args.slice(packageIndex + 1)
if (invocationArgs[0] === '--') invocationArgs.shift()
return { requestedPackage, args: invocationArgs }
}
const invocation = packageInvocation()
if (
invocation &&
['next', 'next@canary', `next@${config.nextVersion}`].includes(
invocation.requestedPackage
) &&
(!invocation.executable || invocation.executable === 'next') &&
['upgrade', '--help', '-h', 'help'].includes(invocation.args[0])
) {
const result = spawnSync(
process.execPath,
[join(tools, 'entry.mjs'), ...invocation.args],
{
stdio: 'inherit',
env: {
...process.env,
NEXT_UPGRADE_EVAL_PACKAGE_RUNNER: runner,
NEXT_UPGRADE_EVAL_REQUESTED_PACKAGE: invocation.requestedPackage,
},
}
)
if (result.error) throw result.error
process.exit(result.status ?? 1)
}
const result = spawnSync(command, args, {
stdio: 'inherit',
env: process.env,
})
if (result.error) throw result.error
process.exit(result.status ?? 1)
#!/usr/bin/env node
const fs = require('node:fs')
const path = require('node:path')
const { spawnSync } = require('node:child_process')
const { Sandbox } = require('@vercel/sandbox')
const { config: loadEnvironment } = require('dotenv')
const { packPackage } = require('../lib/pack')
const { linkEnvironment } = require('../lib/environment')
const snapshotEnvironmentVariable = 'AGENT_EVAL_SANDBOX_SNAPSHOT_ID'
async function createToolchainSnapshot() {
console.log('Preparing shared eval toolchain...')
const token = process.env.VERCEL_TOKEN
const credentials = token
? {
token,
teamId: process.env.VERCEL_TEAM_ID,
projectId: process.env.VERCEL_PROJECT_ID,
}
: {}
const sandbox = await Sandbox.create({
runtime: 'node24',
timeout: 600_000,
...credentials,
})
try {
const install = await sandbox.runCommand({
cmd: 'npm',
args: [
'install',
'--global',
'--no-audit',
'--no-fund',
'@anthropic-ai/claude-code',
'@openai/codex',
],
})
if (install.exitCode !== 0) {
const output =
`${await install.stdout()}\n${await install.stderr()}`.trim()
throw new Error(`Preparing shared eval toolchain failed:\n${output}`)
}
return await sandbox.snapshot({ expiration: 24 * 60 * 60 * 1000 })
} catch (error) {
await sandbox.stop()
throw error
}
}
async function main() {
const root = path.resolve(__dirname, '../..')
linkEnvironment(root, __dirname)
loadEnvironment({ path: path.join(__dirname, '.env.local'), override: true })
loadEnvironment({ path: path.join(__dirname, '.env'), override: true })
const args = process.argv.slice(2)
const fixturesDirectory = path.join(__dirname, 'evals')
const { discoverFixtures, loadFixture } = await import('@vercel/agent-eval')
const cases = fs.existsSync(fixturesDirectory)
? discoverFixtures(fixturesDirectory)
: []
if (args.includes('--list')) {
if (args.length !== 1) throw new Error('Use --list by itself')
console.log(cases.join('\n'))
return
}
const selected = args.filter((arg) => !arg.startsWith('--'))
if (selected.length !== 1)
throw new Error(
'Select one upgrade eval fixture; use --list to list fixtures'
)
const fixture = selected[0]
const harness = process.env.NEXT_UPGRADE_EVAL_EXPERIMENT
if (harness && !['codex', 'claude'].includes(harness))
throw new Error('Select codex or claude')
if (!cases.includes(fixture))
throw new Error(`Available cases: ${cases.join(', ')}`)
if (args.some((arg) => arg.startsWith('--') && arg !== '--dry'))
throw new Error('Supported flags: --dry, --list')
// Validate using the framework's own fixture rules. Its run command otherwise
// falls back to all fixtures when an explicit filter matches no valid fixture.
loadFixture(fixturesDirectory, fixture)
if (args.includes('--dry')) return console.log(fixture)
for (const [name, entry] of [
['next', 'dist/bin/next'],
['next-codemod', 'bin/next-codemod.js'],
]) {
if (!fs.existsSync(path.join(root, 'packages', name, entry)))
throw new Error(`Build packages/${name} before running upgrade evals`)
}
const tarballs = path.join(__dirname, '.tarballs')
fs.mkdirSync(tarballs, { recursive: true })
const env = {
...process.env,
NEXT_UPGRADE_EVAL_NEXT_TARBALL: packPackage(
path.join(root, 'packages/next'),
path.join(tarballs, 'next.tgz')
),
NEXT_UPGRADE_EVAL_CODEMOD_TARBALL: packPackage(
path.join(root, 'packages/next-codemod'),
path.join(tarballs, 'codemod.tgz')
),
}
fs.mkdirSync(path.join(__dirname, 'results'), { recursive: true })
let snapshot
try {
if (!env[snapshotEnvironmentVariable]) {
snapshot = await createToolchainSnapshot()
env[snapshotEnvironmentVariable] = snapshot.snapshotId
}
const experiments = harness ? [harness] : ['codex', 'claude']
const result = spawnSync(
path.join(root, 'node_modules/.bin/agent-eval'),
['run', ...experiments, '--force', '--ack-failures'],
{
cwd: __dirname,
env: {
...env,
NEXT_UPGRADE_EVAL_CASE: fixture,
...(experiments.length > 1
? {
AGENT_EVAL_PREPARE_FIXTURE_ONCE: '1',
AGENT_EVAL_PREPARED_FIXTURE_CONSUMERS: String(
experiments.length
),
}
: {}),
},
stdio: 'inherit',
}
)
if (result.error) throw result.error
if (result.status !== 0) process.exitCode = 1
} finally {
if (snapshot) {
try {
await snapshot.delete()
} catch (error) {
console.error('Failed to delete eval toolchain snapshot:', error)
process.exitCode = 1
}
}
}
}
main().catch((error) => {
console.error(error)
process.exitCode = 1
})
import { appendFileSync, readFileSync } from 'node:fs'
import { dirname, join } from 'node:path'
import { fileURLToPath } from 'node:url'
const tools = dirname(dirname(fileURLToPath(import.meta.url)))
const realFetch = globalThis.fetch
const { range, versions } = JSON.parse(
readFileSync(
join(dirname(fileURLToPath(import.meta.url)), 'assessment.json'),
'utf8'
)
)
globalThis.fetch = async (input, init) => {
const url = String(input)
let value
if (url.startsWith('https://api.github.com/advisories?')) {
value = [
{
withdrawn_at: null,
vulnerabilities: [
{
package: { ecosystem: 'npm', name: 'next' },
vulnerable_version_range: range,
},
],
},
]
} else if (url === 'https://registry.npmjs.org/next') {
value = {
versions: Object.fromEntries(
versions.map((version) => [version, { version }])
),
}
} else if (
url === 'https://registry.npmjs.org/-/npm/v1/security/advisories/bulk'
) {
value = { next: [{ vulnerable_versions: range }] }
} else {
return realFetch(input, init)
}
appendFileSync(
join(tools, 'assessment.jsonl'),
JSON.stringify({ url, value }) + '\n'
)
return Response.json(value)
}
#!/usr/bin/env node
import { spawnSync } from 'node:child_process'
import { appendFileSync, readFileSync } from 'node:fs'
import { dirname, join } from 'node:path'
import { fileURLToPath } from 'node:url'
const security = dirname(fileURLToPath(import.meta.url))
const tools = dirname(security)
const config = JSON.parse(
readFileSync(join(security, 'package-runner.json'), 'utf8')
)
const [runner, ...args] = process.argv.slice(2)
if (!['git', 'npm', 'npx'].includes(runner))
throw new Error(`Unsupported package runner: ${runner}`)
const record = (event) =>
appendFileSync(
join(tools, 'codemod-runs.jsonl'),
JSON.stringify(event) + '\n'
)
if (runner === 'git' && args[0] === 'remote' && args.includes('-v')) {
process.stdout.write(
`origin\t${config.repository} (fetch)\norigin\t${config.repository} (push)\n`
)
process.exit(0)
}
if (
runner === 'git' &&
args[0] === 'remote' &&
args[1] === 'get-url' &&
args[2] === 'origin'
) {
process.stdout.write(`${config.repository}\n`)
process.exit(0)
}
if (runner === 'git' && ['fetch', 'push', 'ls-remote'].includes(args[0])) {
for (let index = 1; index < args.length; index++) {
if (args[index] === 'origin' || args[index] === config.repository) {
args[index] = config.remote
break
}
}
}
if (runner === 'git' && args[0] === 'ls-remote') {
appendFileSync(
join(tools, 'provider.jsonl'),
JSON.stringify({ runner, args }) + '\n'
)
}
if (
runner === 'npm' &&
['info', 'show', 'view'].includes(args[0]) &&
args[1] === '@next/codemod@canary'
) {
record({
kind: 'resolve',
requestedArgs: args,
resolvedVersion: config.codemodVersion,
cwd: process.cwd(),
})
const value =
args[2] === 'version'
? config.codemodVersion
: { version: config.codemodVersion }
process.stdout.write(
(args.includes('--json') ? JSON.stringify(value) : String(value)) + '\n'
)
process.exit(0)
}
function packageInvocation() {
let packageIndex = -1
if (runner === 'npx') {
packageIndex = 0
} else if (runner === 'npm' && ['exec', 'x'].includes(args[0])) {
packageIndex = 1
} else {
return
}
const optionIndex = args.findIndex(
(arg) => arg === '--package' || arg.startsWith('--package=')
)
if (optionIndex !== -1) {
const requestedPackage = args[optionIndex].startsWith('--package=')
? args[optionIndex].slice('--package='.length)
: args[optionIndex + 1]
const separator = args.indexOf('--')
if (separator === -1) return
const [executable, ...invocationArgs] = args.slice(separator + 1)
return { requestedPackage, executable, args: invocationArgs }
}
while (['--', '--yes', '-y'].includes(args[packageIndex])) packageIndex++
const requestedPackage = args[packageIndex]
const invocationArgs = args.slice(packageIndex + 1)
if (invocationArgs[0] === '--') invocationArgs.shift()
return { requestedPackage, args: invocationArgs }
}
const invocation = packageInvocation()
const requestedPackage = invocation?.requestedPackage
const executable = invocation?.executable
const invocationArgs = invocation?.args
if (
invocationArgs &&
(requestedPackage === '@next/codemod@canary' ||
requestedPackage === `@next/codemod@${config.codemodVersion}`) &&
(!executable || ['codemod', 'next-codemod'].includes(executable))
) {
record({
kind: 'run',
requestedPackage,
requestedArgs: args,
resolvedVersion: config.codemodVersion,
args: invocationArgs,
cwd: process.cwd(),
})
const result = spawnSync(
process.execPath,
[
join(tools, 'codemod/node_modules/@next/codemod/bin/next-codemod.js'),
...invocationArgs,
],
{ stdio: 'inherit', env: process.env }
)
if (result.error) throw result.error
process.exit(result.status ?? 1)
}
const result = spawnSync(
runner === 'git' ? config.git : process.execPath,
runner === 'git' ? args : [config.baseRunner, runner, ...args],
{
stdio: 'inherit',
env: process.env,
}
)
if (result.error) throw result.error
process.exit(result.status ?? 1)
#!/usr/bin/env node
import { appendFileSync, readFileSync } from 'node:fs'
import { dirname, join } from 'node:path'
import { fileURLToPath } from 'node:url'
const tools = dirname(dirname(fileURLToPath(import.meta.url)))
const { duplicate, target } = JSON.parse(
readFileSync(
join(dirname(fileURLToPath(import.meta.url)), 'assessment.json'),
'utf8'
)
)
const args = process.argv.slice(2)
appendFileSync(join(tools, 'provider.jsonl'), JSON.stringify({ args }) + '\n')
const repo = {
name: 'fixture',
nameWithOwner: 'next-upgrade-eval/fixture',
full_name: 'next-upgrade-eval/fixture',
url: 'https://github.com/next-upgrade-eval/fixture',
html_url: 'https://github.com/next-upgrade-eval/fixture',
defaultBranchRef: { name: 'main' },
default_branch: 'main',
owner: { login: 'next-upgrade-eval' },
}
const pullRequest = {
number: 42,
title: `Upgrade Next.js to ${target}`,
body: '<!-- next-upgrade: security; path="." -->',
url: 'https://github.com/next-upgrade-eval/fixture/pull/42',
html_url: 'https://github.com/next-upgrade-eval/fixture/pull/42',
state: 'OPEN',
headRefName: 'security-upgrade',
baseRefName: 'main',
head: { ref: 'security-upgrade' },
base: { ref: 'main' },
}
const pullRequestFiles = [
{
filename: 'package.json',
status: 'modified',
patch: `@@ -9,1 +9,1 @@\n- "next": "15.5.23"\n+ "next": "${target}"`,
},
]
let result
if (args[0] === 'auth' && args[1] === 'status') {
console.log('github.com: authenticated as upgrade-eval')
process.exit(0)
} else if (args[0] === 'repo' && args[1] === 'view') {
result = repo
} else if (
(args[0] === 'pr' && args[1] === 'list') ||
(args[0] === 'search' && args[1] === 'prs')
) {
result = duplicate ? [pullRequest] : []
} else if (args[0] === 'pr' && args[1] === 'view' && duplicate) {
result = pullRequest
} else if (args[0] === 'pr' && args[1] === 'diff' && duplicate) {
process.stdout.write(`${pullRequestFiles[0].patch}\n`)
process.exit(0)
} else if (args[0] === 'api') {
const endpoint = args
.find((arg) => /^(\/)?repos\//.test(arg))
?.replace(/^\//, '')
if (endpoint === 'repos/next-upgrade-eval/fixture') result = repo
else if (
endpoint?.startsWith('repos/next-upgrade-eval/fixture/pulls/42/files')
)
result = duplicate ? pullRequestFiles : []
else if (endpoint?.startsWith('repos/next-upgrade-eval/fixture/pulls/42'))
result = duplicate ? pullRequest : undefined
else if (endpoint?.startsWith('repos/next-upgrade-eval/fixture/pulls'))
result = duplicate ? [pullRequest] : []
else if (args.includes('user') || args.includes('/user'))
result = { login: 'upgrade-eval' }
}
if (result === undefined) {
console.error(`Unsupported evaluation provider read: ${JSON.stringify(args)}`)
process.exit(1)
}
const json = args.indexOf('--json')
if (json !== -1 && !Array.isArray(result)) {
result = Object.fromEntries(
args[json + 1].split(',').map((field) => [field, result[field]])
)
}
console.log(JSON.stringify(result))
import { readFileSync } from 'node:fs'
import { join } from 'node:path'
import type { Sandbox } from '@vercel/agent-eval'
import { toolsDirectory } from '../lib/fixture'
export async function setupSecurity(sandbox: Sandbox) {
const run = async (command: string, args: string[]) => {
const result = await sandbox.runCommand(command, args)
if (result.exitCode !== 0)
throw new Error(
`${command} failed during security setup:\n${result.stderr}`
)
return result.stdout.trim()
}
const fixture = process.env.NEXT_UPGRADE_EVAL_CASE
if (!fixture?.startsWith('security-'))
throw new Error('Select a security upgrade eval case')
const sameMajorTarget = '15.5.24'
const crossMajorTarget = '15.5.24'
const scenarios: Record<
string,
{ target: string; range: string; versions: string[]; duplicate: boolean }
> = {
'security-cross-major': {
target: crossMajorTarget,
range: `>=13.0.0 <${crossMajorTarget}`,
versions: ['13.5.11', '14.2.35', crossMajorTarget, '16.0.0'],
duplicate: false,
},
'security-duplicate': {
target: sameMajorTarget,
range: `>=15.0.0 <${sameMajorTarget}`,
versions: ['15.5.23', sameMajorTarget, '16.0.0'],
duplicate: true,
},
'security-same-major': {
target: sameMajorTarget,
range: `>=15.0.0 <${sameMajorTarget}`,
versions: ['15.5.23', sameMajorTarget, '16.0.0'],
duplicate: false,
},
}
const scenario = scenarios[fixture]
if (!scenario) throw new Error('Unknown security upgrade eval case')
const security = `${toolsDirectory}/security`
const bin = `${toolsDirectory}/bin`
const repository = 'https://github.com/next-upgrade-eval/fixture.git'
const remote = `${toolsDirectory}/origin.git`
const config = `${toolsDirectory}/gitconfig`
const baseline = await run('git', ['rev-parse', 'HEAD'])
await run('mkdir', ['-p', security])
await sandbox.writeFiles({
[`${security}/assessment.mjs`]: readFileSync(
join(__dirname, 'assessment.mjs'),
'utf8'
),
[`${security}/assessment.json`]: JSON.stringify(scenario),
[`${security}/provider.mjs`]: readFileSync(
join(__dirname, 'provider.mjs'),
'utf8'
),
[`${security}/package-runner.mjs`]: readFileSync(
join(__dirname, 'package-runner.mjs'),
'utf8'
),
[config]: `[url "file://${remote}"]\n\tinsteadOf = ${repository}\n`,
[`${toolsDirectory}/baseline.json`]: JSON.stringify({ head: baseline }),
})
await run('git', ['branch', '-M', 'main'])
await run('git', ['clone', '--bare', '.', remote])
await run('git', ['config', 'include.path', config])
await run('git', ['remote', 'add', 'origin', repository])
await run('git', ['fetch', 'origin'])
await run('git', ['remote', 'set-head', 'origin', 'main'])
await run('git', ['config', '--unset', 'include.path'])
const git = await run('sh', ['-c', 'command -v git'])
const codemodVersion = await run('node', [
'-p',
`require('${toolsDirectory}/codemod/node_modules/@next/codemod/package.json').version`,
])
await sandbox.writeFiles({
[`${security}/package-runner.json`]: JSON.stringify({
baseRunner: `${toolsDirectory}/package-runner.mjs`,
codemodVersion,
git,
remote,
repository,
}),
[join(bin, 'npm')]:
`#!/bin/sh\nexec node ${security}/package-runner.mjs npm "$@"\n`,
[join(bin, 'npx')]:
`#!/bin/sh\nexec node ${security}/package-runner.mjs npx "$@"\n`,
[join(bin, 'git')]:
`#!/bin/sh\nexec node ${security}/package-runner.mjs git "$@"\n`,
})
await run('chmod', [
'+x',
join(bin, 'git'),
join(bin, 'npm'),
join(bin, 'npx'),
])
await run('chmod', ['+x', `${security}/provider.mjs`])
await run('ln', ['-sf', `${security}/provider.mjs`, join(bin, 'gh')])
}
import { afterAll, beforeAll, expect, test, vi } from 'vitest'
import { execFileSync, spawn, type ChildProcess } from 'node:child_process'
import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'
import { join } from 'node:path'
type SecurityCheckOptions = {
changedFiles: string[]
migrationGuides: number[]
}
export function securityChecks(
source: string,
target: string,
behavior: (app: { url: string; cwd: string }) => void,
options: SecurityCheckOptions | undefined = undefined
) {
void options
const tools = '/tmp/next-upgrade-eval'
const evidence = join(process.cwd(), 'eval-evidence')
const git = (...args: string[]) =>
execFileSync('git', args, { encoding: 'utf8' }).trim()
let baseline: string
let cwd: string
let server: ChildProcess | undefined
let serverOutput = ''
let url = ''
beforeAll(async () => {
baseline = JSON.parse(
readFileSync(join(tools, 'baseline.json'), 'utf8')
).head
expect(
JSON.parse(git('show', `${baseline}:package.json`)).dependencies.next
).toBe(source)
const manifest = JSON.parse(readFileSync('package.json', 'utf8'))
expect(manifest.dependencies.next).toBe(target)
cwd = process.cwd()
server = spawn('npm', ['run', 'dev', '--', '--port', '0'], {
cwd,
stdio: ['ignore', 'pipe', 'pipe'],
env: { ...process.env, NEXT_TELEMETRY_DISABLED: '1' },
})
server.stdout!.on('data', (chunk) => {
serverOutput += chunk.toString()
})
server.stderr!.on('data', (chunk) => {
serverOutput += chunk.toString()
})
await vi.waitFor(
() => {
url = serverOutput.match(/http:\/\/localhost:\d+/)?.[0] ?? ''
expect(url).not.toBe('')
},
{ timeout: 30000 }
)
await vi.waitFor(
async () => {
expect((await fetch(url)).status).toBe(200)
},
{ timeout: 30000 }
)
}, 600000)
afterAll(() => {
server?.kill('SIGTERM')
mkdirSync(evidence, { recursive: true })
writeFileSync(join(evidence, 'committed-server.log'), serverOutput)
for (const name of [
'invocations.jsonl',
'provider.jsonl',
'assessment.jsonl',
'codemod-runs.jsonl',
]) {
if (existsSync(join(tools, name)))
writeFileSync(join(evidence, name), readFileSync(join(tools, name)))
}
})
test('does not publish the migration', () => {
expect(
execFileSync(
'git',
['--git-dir', join(tools, 'origin.git'), 'rev-parse', 'main'],
{ encoding: 'utf8' }
).trim()
).toBe(baseline)
})
behavior({
get url() {
return url
},
get cwd() {
return cwd
},
})
}
export function duplicateSecurityChecks(source: string, target: string) {
const tools = '/tmp/next-upgrade-eval'
const git = (...args: string[]) =>
execFileSync('git', args, { encoding: 'utf8' }).trim()
const records = (name: string) => {
const file = join(tools, name)
if (!existsSync(file)) return []
const content = readFileSync(file, 'utf8').trim()
return content ? content.split('\n').map((line) => JSON.parse(line)) : []
}
const harnessFiles = new Set([
'EVAL.ts',
'PROMPT.md',
'checks/EVAL.ts',
'package-lock.json',
'vitest.config.ts',
])
const appStatus = () =>
git('status', '--short', '--untracked-files=all')
.split('\n')
.filter((line) => line && !harnessFiles.has(line.slice(3)))
let baseline: string
beforeAll(() => {
baseline = JSON.parse(
readFileSync(join(tools, 'baseline.json'), 'utf8')
).head
expect(
JSON.parse(git('show', `${baseline}:package.json`)).dependencies.next
).toBe(source)
})
test('stops without changing or publishing the app', () => {
expect(git('rev-parse', 'HEAD')).toBe(baseline)
expect(appStatus()).toEqual([])
expect(
execFileSync(
'git',
[
'--git-dir',
join(tools, 'origin.git'),
'for-each-ref',
'--format=%(refname:short) %(objectname)',
'refs/heads',
],
{ encoding: 'utf8' }
).trim()
).toBe(`main ${baseline}`)
expect(
records('codemod-runs.jsonl').some(({ kind }) => kind === 'run')
).toBe(false)
})
test('recognizes the existing security upgrade', async () => {
expect(records('invocations.jsonl')).toContainEqual(
expect.objectContaining({
args: ['upgrade', '--ai'],
packageRunner: 'npx',
requestedPackage: 'next@canary',
})
)
expect(
records('provider.jsonl').some(({ args }) =>
args.some((arg) =>
/^(pr|search)$|refs\/(pull|merge-requests)|\/pulls(?:[/?]|$)/.test(
arg
)
)
)
).toBe(true)
expect(readFileSync(join(tools, 'assessment.jsonl'), 'utf8')).toContain(
target
)
await expect(transcript).toSatisfyCriterion(
`The agent found the existing equivalent open security upgrade pull request and stopped. It did not run the codemod, change files, commit, push, or create another pull request.`
)
})
}
...@@ -8,5 +8,11 @@ ...@@ -8,5 +8,11 @@
"noEmit": true, "noEmit": true,
"types": ["node"] "types": ["node"]
}, },
"include": ["lib", "experiments"] "include": [
"lib",
"experiments",
"next-upgrade/experiments",
"next-upgrade/lib",
"next-upgrade/security/setup.ts"
]
} }
...@@ -20,6 +20,7 @@ ...@@ -20,6 +20,7 @@
"bench:dev-validation": "tsx bench/dev-validation/benchmark.ts", "bench:dev-validation": "tsx bench/dev-validation/benchmark.ts",
"pack-next": "tsx scripts/pack-next.ts", "pack-next": "tsx scripts/pack-next.ts",
"eval": "node run-evals.js", "eval": "node run-evals.js",
"eval:upgrade": "node evals/next-upgrade/run.js",
"test-types": "tsc", "test-types": "tsc",
"test-unit": "jest test/unit/ packages/next/ packages/font", "test-unit": "jest test/unit/ packages/next/ packages/font",
"test-dev": "scripts/run-jest.sh --mode=dev --bundler=webpack --headless --", "test-dev": "scripts/run-jest.sh --mode=dev --bundler=webpack --headless --",
...@@ -170,6 +171,7 @@ ...@@ -170,6 +171,7 @@
"@vercel/devlow-bench": "workspace:*", "@vercel/devlow-bench": "workspace:*",
"@vercel/kv": "3.0.0", "@vercel/kv": "3.0.0",
"@vercel/og": "1.0.1", "@vercel/og": "1.0.1",
"@vercel/sandbox": "1.8.0",
"alex": "9.1.0", "alex": "9.1.0",
"async-sema": "3.0.1", "async-sema": "3.0.1",
"babel-plugin-react-compiler": "0.0.0-experimental-1371fcb-20260227", "babel-plugin-react-compiler": "0.0.0-experimental-1371fcb-20260227",
...@@ -184,6 +186,7 @@ ...@@ -184,6 +186,7 @@
"cross-env": "6.0.3", "cross-env": "6.0.3",
"cross-spawn": "6.0.5", "cross-spawn": "6.0.5",
"dd-trace": "4.12.0", "dd-trace": "4.12.0",
"dotenv": "16.4.7",
"es5-ext": "0.10.53", "es5-ext": "0.10.53",
"escape-string-regexp": "2.0.0", "escape-string-regexp": "2.0.0",
"eslint": "9.37.0", "eslint": "9.37.0",
...@@ -338,7 +341,8 @@ ...@@ -338,7 +341,8 @@
"@vercel/blob": "patches/@vercel__blob.patch", "@vercel/blob": "patches/@vercel__blob.patch",
"postcss-scss": "patches/postcss-scss.patch", "postcss-scss": "patches/postcss-scss.patch",
"playwright-core@1.61.0": "patches/playwright-core@1.61.0.patch", "playwright-core@1.61.0": "patches/playwright-core@1.61.0.patch",
"@types/node@20.17.7": "patches/@types__node@20.17.7.patch" "@types/node@20.17.7": "patches/@types__node@20.17.7.patch",
"@vercel/agent-eval@2.2.1": "patches/@vercel__agent-eval@2.2.1.patch"
} }
} }
} }
...@@ -109,8 +109,16 @@ class NextRootCommand extends Command { ...@@ -109,8 +109,16 @@ class NextRootCommand extends Command {
} }
} }
// The upgrade harness may run both dev and production checks. Preserve
// its caller's environment instead of forcing all child commands into
// production mode merely because they were launched through this CLI.
if (
commandName !== 'upgrade' ||
!event.getOptionValue('experimentalAi')
) {
;(process.env as any).NODE_ENV = process.env.NODE_ENV || defaultEnv ;(process.env as any).NODE_ENV = process.env.NODE_ENV || defaultEnv
;(process.env as any).NEXT_RUNTIME = 'nodejs' ;(process.env as any).NEXT_RUNTIME = 'nodejs'
}
if ( if (
process.platform === 'darwin' && process.platform === 'darwin' &&
...@@ -554,6 +562,7 @@ program ...@@ -554,6 +562,7 @@ program
const nextVersion = process.env.__NEXT_VERSION || 'unknown' const nextVersion = process.env.__NEXT_VERSION || 'unknown'
program program
.command('upgrade') .command('upgrade')
.aliases(['update', 'up'])
.description( .description(
'Upgrade Next.js apps to desired versions with a single command.' 'Upgrade Next.js apps to desired versions with a single command.'
) )
...@@ -576,9 +585,18 @@ program ...@@ -576,9 +585,18 @@ program
: 'latest' : 'latest'
) )
.option('--verbose', 'Verbose output', false) .option('--verbose', 'Verbose output', false)
.addOption(
new Option(
'--ai, --experimental-ai [type]',
'Upgrade with AI. Defaults to security.'
).conflicts('revision')
)
.action(async (directory, options) => { .action(async (directory, options) => {
const mod = await import('../cli/next-upgrade.js') const mod = await import('../cli/next-upgrade.js')
mod.spawnNextUpgrade(directory, options) await mod.spawnNextUpgrade(directory, {
...options,
ai: options.experimentalAi,
})
}) })
program program
......
...@@ -226,6 +226,8 @@ export async function collectBuildTraces({ ...@@ -226,6 +226,8 @@ export async function collectBuildTraces({
'**/next/dist/compiled/webpack/*', '**/next/dist/compiled/webpack/*',
'**/node_modules/webpack5/**/*', '**/node_modules/webpack5/**/*',
'**/next/dist/server/lib/route-resolver*', '**/next/dist/server/lib/route-resolver*',
// Upgrade workflows are CLI-only and are not needed by production servers.
'**/next/dist/lib/upgrade/**/*',
// The testmode interceptors bundle reads its HTTP parser WASM with a // The testmode interceptors bundle reads its HTTP parser WASM with a
// dynamic path, making nft trace the bundle's whole directory. Test // dynamic path, making nft trace the bundle's whole directory. Test
// proxying is not supported in standalone output, so keep the parser // proxying is not supported in standalone output, so keep the parser
......
import { spawn } from 'child_process' import { spawn } from 'child_process'
import { cp, mkdtemp, readFile, rm, writeFile } from 'fs/promises'
import { tmpdir } from 'os'
import { join } from 'path'
import * as Log from '../build/output/log'
import createSpinner from '../build/spinner'
import { findDir } from '../lib/find-pages-dir'
import { getProjectDir } from '../lib/get-project-dir' import { getProjectDir } from '../lib/get-project-dir'
import { getNpxCommand } from '../lib/helpers/get-npx-command' import { getNpxCommand } from '../lib/helpers/get-npx-command'
import { dim } from '../lib/picocolors'
import { runChildProcess } from '../lib/upgrade/run-child-process'
interface NextUpgradeOptions { type NextUpgradeOptions = {
revision: string revision: string
verbose: boolean verbose: boolean
ai: boolean | string | undefined
} }
export function spawnNextUpgrade( const CODEMOD_COMMAND_PLACEHOLDER = '<codemod-command>'
export async function spawnNextUpgrade(
directory: string | undefined, directory: string | undefined,
options: NextUpgradeOptions options: NextUpgradeOptions
) { ) {
const baseDir = getProjectDir(directory) const baseDir = getProjectDir(directory)
if (options.ai) {
try {
// A delegated canary uses itself. Local runs and evals use their invoked build.
const useCurrentCli = process.env.__NEXT_UPGRADE_USE_CURRENT_CLI === '1'
delete process.env.__NEXT_UPGRADE_USE_CURRENT_CLI
if (!useCurrentCli) {
Log.info(dim('Preparing upgrade...'))
const [command, ...runnerArgs] = getNpxCommand(baseDir).split(' ')
const aiArgument =
typeof options.ai === 'string' ? `--ai=${options.ai}` : '--ai'
const args = [
...runnerArgs,
'next@canary',
'upgrade',
baseDir,
aiArgument,
]
if (options.verbose) {
args.push('--verbose')
}
process.exitCode = await runChildProcess(command, args, {
cwd: baseDir,
stdio: 'inherit',
env: { ...process.env, __NEXT_UPGRADE_USE_CURRENT_CLI: '1' },
})
return
}
// A workspace root must not launch an upgrade for an unspecified app.
if (!findDir(baseDir, 'app') && !findDir(baseDir, 'pages')) {
throw new Error(
'No Next.js app found in this directory. Run the command from an app directory or pass its path.'
)
}
// TODO: Once `agenticAutoUpgrade` can be read without validating a
// legacy app's config against the current Next.js version, use it for
// bare `--ai` before falling back to security.
const upgradeType =
typeof options.ai === 'string' ? options.ai : 'security'
if (upgradeType !== 'security') {
throw new Error(
`Unsupported AI upgrade type ${JSON.stringify(upgradeType)}. Expected "security".`
)
}
// Resolve the requested target before preparing an agent session.
const { prepareUpgrade } =
require('../lib/upgrade/prepare-upgrade') as typeof import('../lib/upgrade/prepare-upgrade')
const assessmentSpinner = createSpinner('Checking for security updates')
const result = await prepareUpgrade(baseDir).finally(() =>
assessmentSpinner?.stop()
)
if (result.status !== 'ready') {
Log.info(result.reason)
return
}
Log.info(
`Security update: Next.js ${result.installedVersion} → ${result.targetVersion}`
)
// Use the invoking CLI's guides, even when the app runs an older Next.js.
// Retain them outside the app so dependency changes cannot remove them.
const bundledDocs = join(__dirname, '../docs')
const runDirectory = await mkdtemp(join(tmpdir(), 'next-upgrade-'))
const guidePath = join(
runDirectory,
'docs/01-app/02-guides/upgrading/agentic-upgrade.md'
)
const guidesSpinner = createSpinner('Preparing upgrade')
try {
for (const router of ['01-app', '02-pages']) {
await cp(
join(bundledDocs, router, '02-guides/upgrading'),
join(runDirectory, 'docs', router, '02-guides/upgrading'),
{ recursive: true }
)
}
const codemodVersion = process.env.__NEXT_VERSION
if (!codemodVersion) {
throw new Error('Could not determine the @next/codemod version.')
}
const codemodCommand = `${getNpxCommand(baseDir)} @next/codemod@${codemodVersion} upgrade ${result.targetVersion} --yes --skip-adoption${options.verbose ? ' --verbose' : ''}`
const guide = await readFile(guidePath, 'utf8')
if (!guide.includes(CODEMOD_COMMAND_PLACEHOLDER)) {
throw new Error('Could not prepare the upgrade guide.')
}
await writeFile(
guidePath,
guide.replace(CODEMOD_COMMAND_PLACEHOLDER, codemodCommand)
)
} catch (error) {
await rm(runDirectory, { recursive: true, force: true })
throw error
} finally {
guidesSpinner?.stop()
}
// TODO: Once every eligible security target supports
// `experimental.agenticAutoUpgrade`, ask the agent to enable it after
// verification so future upgrade reminders can use the same policy.
const references = result.references
.map((reference) => `- ${reference}`)
.join('\n')
// Pass resolved inputs directly; the agent owns repairs and verification.
const prompt = `Read and follow every applicable instruction in ${JSON.stringify(guidePath)} before proceeding.
We're upgrading the app in ${JSON.stringify(baseDir)} from Next.js ${result.installedVersion} to ${result.targetVersion} because the installed version is affected by a published security advisory.
References:
${references}`
const { handoffUpgrade } =
require('../lib/upgrade/harness') as typeof import('../lib/upgrade/harness')
await handoffUpgrade(prompt, baseDir)
} catch (error) {
Log.error(
'Could not prepare the security upgrade:',
error instanceof Error ? error.message : error
)
process.exitCode = 1
}
return
}
const [upgradeProcessCommand, ...upgradeProcessDefaultArgs] = const [upgradeProcessCommand, ...upgradeProcessDefaultArgs] =
getNpxCommand(baseDir).split(' ') getNpxCommand(baseDir).split(' ')
...@@ -22,6 +170,7 @@ export function spawnNextUpgrade( ...@@ -22,6 +170,7 @@ export function spawnNextUpgrade(
'upgrade', 'upgrade',
options.revision, options.revision,
] ]
if (options.verbose) { if (options.verbose) {
upgradeProcessCommandArgs.push('--verbose') upgradeProcessCommandArgs.push('--verbose')
} }
......
import { constants } from 'fs'
import { access, stat } from 'fs/promises'
import { delimiter, resolve } from 'path'
import cliSelect from 'next/dist/compiled/cli-select'
import spawn from 'next/dist/compiled/cross-spawn'
import * as Log from '../../build/output/log'
import { getAgentName } from '../../telemetry/agent-name'
import { bold, cyan, dim } from '../picocolors'
import { runChildProcess } from './run-child-process'
// Model defaults for newly launched sessions; existing agents keep their model.
const UPGRADE_MODELS = {
codex: 'gpt-5.6-terra',
claude: 'sonnet',
} as const
type UpgradeHarness = {
name: keyof typeof UPGRADE_MODELS
path: string
}
function getHarnessDisplayName(name: UpgradeHarness['name']): string {
return name === 'codex' ? 'Codex' : 'Claude Code'
}
async function findHarnesses(): Promise<UpgradeHarness[]> {
const names: UpgradeHarness['name'][] = ['codex', 'claude']
const directories = (process.env.PATH ?? '').split(delimiter).filter(Boolean)
const extensions =
process.platform === 'win32'
? (process.env.PATHEXT || '.EXE;.CMD;.BAT;.COM')
.split(';')
.filter(Boolean)
: ['']
// Probe agents independently, preserving menu order and each detected path.
const installed = await Promise.all(
names.map(async (name): Promise<UpgradeHarness | null> => {
for (const directory of directories) {
for (const extension of extensions) {
const file = resolve(directory, `${name}${extension}`)
try {
await access(
file,
process.platform === 'win32' ? constants.F_OK : constants.X_OK
)
if ((await stat(file)).isFile()) {
return { name, path: file }
}
} catch {}
}
}
return null
})
)
return installed.filter(
(harness): harness is UpgradeHarness => harness !== null
)
}
async function chooseHarness(
harnesses: UpgradeHarness[]
): Promise<UpgradeHarness | 'copy' | undefined> {
const question =
harnesses.length === 1
? `${getHarnessDisplayName(harnesses[0].name)} detected. Would you like to proceed?`
: 'Multiple coding agents detected. Which one would you like to use?'
Log.bootstrap('')
Log.bootstrap(` ${question}`)
Log.bootstrap(` ${dim('Use ↑/↓ to choose, then press Enter.')}\n`)
try {
const { id } = await cliSelect({
values: {
...Object.fromEntries(
harnesses.map(({ name }) => [
name,
`Continue with ${getHarnessDisplayName(name)}`,
])
),
copy: 'Copy prompt for another coding agent',
cancel: 'Cancel',
},
// cli-select indexes rows numerically, even when values is an object.
defaultValue: 0,
selected: cyan('❯'),
unselected: ' ',
indentation: 2,
valueRenderer: (value: string, selected: boolean) =>
selected ? cyan(bold(value)) : value,
})
return id === 'copy' ? 'copy' : harnesses.find(({ name }) => name === id)
} catch (error) {
// cli-select rejects without an error when Escape or Ctrl+C cancels the menu.
if (error) {
throw error
}
return undefined
}
}
function copyUpgradePrompt(prompt: string, noHarness = false): void {
const commands =
process.platform === 'darwin'
? [['pbcopy']]
: process.platform === 'win32'
? [['clip.exe']]
: [
['wl-copy'],
['xclip', '-selection', 'clipboard'],
['xsel', '--clipboard', '--input'],
]
for (const [command, ...args] of commands) {
const result = spawn.sync(command, args, {
input:
process.platform === 'win32' ? Buffer.from(prompt, 'utf16le') : prompt,
stdio: ['pipe', 'ignore', 'ignore'],
timeout: 1000,
windowsHide: true,
})
if (!result.error && result.status === 0) {
Log.info(
noHarness
? 'No supported coding agent found. The upgrade prompt was copied to your clipboard.'
: 'Upgrade prompt copied. Paste it into your coding agent.'
)
return
}
}
Log.info(
noHarness
? 'No supported coding agent found. Copy this upgrade prompt:'
: 'Could not access the clipboard. Copy this upgrade prompt:'
)
Log.bootstrap(prompt)
}
function launchHarness(
harness: UpgradeHarness,
prompt: string,
directory: string
): Promise<number> {
// Windows shell shims cannot carry literal line breaks in an argument.
if (process.platform === 'win32' && /\.(cmd|bat)$/i.test(harness.path)) {
prompt = prompt.replace(/[\r\n]+/g, ' ')
}
return runChildProcess(
harness.path,
['--model', UPGRADE_MODELS[harness.name], prompt],
{ cwd: directory, stdio: 'inherit' }
)
}
export async function handoffUpgrade(
prompt: string,
directory: string
): Promise<void> {
// Existing agents keep their session, model and permissions.
if (await getAgentName()) {
Log.bootstrap(prompt)
return
}
if (!process.stdin.isTTY || !process.stdout.isTTY) {
Log.info('Copy this upgrade prompt into your coding agent:')
Log.bootstrap(prompt)
return
}
Log.info(dim('Looking for coding agents...'))
const installed = await findHarnesses()
if (installed.length === 0) {
copyUpgradePrompt(prompt, true)
return
}
// Let the selected agent take over the terminal with its existing permissions.
const harness = await chooseHarness(installed)
if (harness === 'copy') {
copyUpgradePrompt(prompt)
return
}
if (!harness) {
Log.bootstrap(` ${dim('Upgrade cancelled.')}\n`)
process.exitCode = 1
return
}
Log.bootstrap(
` Continuing with ${cyan(bold(getHarnessDisplayName(harness.name)))}...\n`
)
try {
process.exitCode = await launchHarness(harness, prompt, directory)
} catch {
Log.error(`Could not start ${getHarnessDisplayName(harness.name)}.`)
process.exitCode = 1
}
}
This diff is collapsed.
import type { SpawnOptions } from 'child_process'
import { constants as osConstants } from 'os'
import spawn from 'next/dist/compiled/cross-spawn'
export function runChildProcess(
command: string,
args: string[],
options: SpawnOptions
): Promise<number> {
const child = spawn(command, args, options)
return new Promise((resolve, reject) => {
const onInterrupt = () => child.kill('SIGINT')
const onTerminate = () => child.kill('SIGTERM')
process.on('SIGINT', onInterrupt)
process.on('SIGTERM', onTerminate)
const cleanup = () => {
process.removeListener('SIGINT', onInterrupt)
process.removeListener('SIGTERM', onTerminate)
}
child.once('error', (error: Error) => {
cleanup()
reject(error)
})
child.once(
'close',
(code: number | null, signal: NodeJS.Signals | null) => {
cleanup()
resolve(code ?? (signal ? 128 + (osConstants.signals[signal] ?? 1) : 1))
}
)
})
}
This diff is collapsed.
...@@ -35,6 +35,9 @@ patchedDependencies: ...@@ -35,6 +35,9 @@ patchedDependencies:
'@types/node@20.17.7': '@types/node@20.17.7':
hash: fe6957869a9b616ec526ededb72bbe846b863bd2349204218772df5c1e62d6ab hash: fe6957869a9b616ec526ededb72bbe846b863bd2349204218772df5c1e62d6ab
path: patches/@types__node@20.17.7.patch path: patches/@types__node@20.17.7.patch
'@vercel/agent-eval@2.2.1':
hash: 8c207275ddbf77bc5afacec008ba773f3e8d562c83f84ed6d80bf16719b4dd70
path: patches/@vercel__agent-eval@2.2.1.patch
'@vercel/blob': '@vercel/blob':
hash: cb53bfa5effb15b3a361e176003df667cadf757b27b7c9b458c2f0fce86ea893 hash: cb53bfa5effb15b3a361e176003df667cadf757b27b7c9b458c2f0fce86ea893
path: patches/@vercel__blob.patch path: patches/@vercel__blob.patch
...@@ -237,7 +240,7 @@ importers: ...@@ -237,7 +240,7 @@ importers:
version: 16.0.11 version: 16.0.11
'@vercel/agent-eval': '@vercel/agent-eval':
specifier: 2.2.1 specifier: 2.2.1
version: 2.2.1 version: 2.2.1(patch_hash=8c207275ddbf77bc5afacec008ba773f3e8d562c83f84ed6d80bf16719b4dd70)
'@vercel/blob': '@vercel/blob':
specifier: 2.3.2 specifier: 2.3.2
version: 2.3.2(patch_hash=cb53bfa5effb15b3a361e176003df667cadf757b27b7c9b458c2f0fce86ea893) version: 2.3.2(patch_hash=cb53bfa5effb15b3a361e176003df667cadf757b27b7c9b458c2f0fce86ea893)
...@@ -250,6 +253,9 @@ importers: ...@@ -250,6 +253,9 @@ importers:
'@vercel/og': '@vercel/og':
specifier: 1.0.1 specifier: 1.0.1
version: 1.0.1(@types/node@20.17.7(patch_hash=fe6957869a9b616ec526ededb72bbe846b863bd2349204218772df5c1e62d6ab)) version: 1.0.1(@types/node@20.17.7(patch_hash=fe6957869a9b616ec526ededb72bbe846b863bd2349204218772df5c1e62d6ab))
'@vercel/sandbox':
specifier: 1.8.0
version: 1.8.0
alex: alex:
specifier: 9.1.0 specifier: 9.1.0
version: 9.1.0 version: 9.1.0
...@@ -292,6 +298,9 @@ importers: ...@@ -292,6 +298,9 @@ importers:
dd-trace: dd-trace:
specifier: 4.12.0 specifier: 4.12.0
version: 4.12.0 version: 4.12.0
dotenv:
specifier: 16.4.7
version: 16.4.7
es5-ext: es5-ext:
specifier: 0.10.53 specifier: 0.10.53
version: 0.10.53 version: 0.10.53
...@@ -24971,7 +24980,7 @@ snapshots: ...@@ -24971,7 +24980,7 @@ snapshots:
dependencies: dependencies:
uncrypto: 0.1.3 uncrypto: 0.1.3
'@vercel/agent-eval@2.2.1': '@vercel/agent-eval@2.2.1(patch_hash=8c207275ddbf77bc5afacec008ba773f3e8d562c83f84ed6d80bf16719b4dd70)':
dependencies: dependencies:
'@ai-sdk/anthropic': 1.2.12(zod@3.25.76) '@ai-sdk/anthropic': 1.2.12(zod@3.25.76)
'@vercel/sandbox': 1.8.0 '@vercel/sandbox': 1.8.0
...@@ -21,7 +21,9 @@ ...@@ -21,7 +21,9 @@
*/ */
const path = require('path') const path = require('path')
const fs = require('fs') const fs = require('fs')
const { execFileSync, spawnSync } = require('child_process') const { spawnSync } = require('child_process')
const { packPackage } = require('./evals/lib/pack')
const { linkEnvironment } = require('./evals/lib/environment')
const ROOT = __dirname const ROOT = __dirname
...@@ -53,17 +55,7 @@ const BASE_VARIANTS = [ ...@@ -53,17 +55,7 @@ const BASE_VARIANTS = [
] ]
function pack() { function pack() {
fs.mkdirSync(TARBALL_DIR, { recursive: true }) packPackage(path.join(ROOT, 'packages/next'), TARBALL)
const out = execFileSync(
'pnpm',
['pack', '--pack-destination', TARBALL_DIR],
{ cwd: path.join(ROOT, 'packages/next'), encoding: 'utf8' }
)
const produced = out.trim().split('\n').pop()
const src = path.isAbsolute(produced)
? produced
: path.join(TARBALL_DIR, produced)
fs.renameSync(src, TARBALL)
} }
/** @param {string | null} evalName null means all evals */ /** @param {string | null} evalName null means all evals */
...@@ -234,17 +226,7 @@ function main() { ...@@ -234,17 +226,7 @@ function main() {
// agent-eval loads .env / .env.local from its own cwd (evals/). `vc env pull` // agent-eval loads .env / .env.local from its own cwd (evals/). `vc env pull`
// writes to the repo root, so symlink them into evals/ for agent-eval to find. // writes to the repo root, so symlink them into evals/ for agent-eval to find.
for (const envFile of ['.env', '.env.local']) { linkEnvironment(ROOT, EVALS_DIR)
const src = path.join(ROOT, envFile)
const dest = path.join(EVALS_DIR, envFile)
try {
// Remove stale symlink or file before creating a fresh one.
fs.rmSync(dest, { force: true })
if (fs.existsSync(src)) {
fs.symlinkSync(src, dest)
}
} catch {}
}
writeExperiments(evalName, variants, timeout) writeExperiments(evalName, variants, timeout)
console.log( console.log(
......
import { access, cp, mkdtemp, readFile, rm, stat, writeFile } from 'fs/promises'
import * as Log from 'next/dist/build/output/log'
import cliSelect from 'next/dist/compiled/cli-select'
import { spawnNextUpgrade } from 'next/dist/cli/next-upgrade'
import { findDir } from 'next/dist/lib/find-pages-dir'
import { getProjectDir } from 'next/dist/lib/get-project-dir'
import { handoffUpgrade } from 'next/dist/lib/upgrade/harness'
import { prepareUpgrade } from 'next/dist/lib/upgrade/prepare-upgrade'
import { getAgentName } from 'next/dist/telemetry/agent-name'
jest.mock('fs/promises', () => ({
access: jest.fn(),
cp: jest.fn(),
mkdtemp: jest.fn(),
readFile: jest.fn(),
rm: jest.fn(),
stat: jest.fn(),
writeFile: jest.fn(),
}))
jest.mock('next/dist/build/spinner', () => ({
__esModule: true,
default: jest.fn(),
}))
jest.mock('next/dist/build/output/log', () => ({
bootstrap: jest.fn(),
error: jest.fn(),
info: jest.fn(),
}))
jest.mock('next/dist/compiled/cli-select', () => ({
__esModule: true,
default: jest.fn(),
}))
jest.mock('next/dist/lib/find-pages-dir', () => ({
findDir: jest.fn(),
}))
jest.mock('next/dist/lib/get-project-dir', () => ({
getProjectDir: jest.fn(),
}))
jest.mock('next/dist/lib/helpers/get-npx-command', () => ({
getNpxCommand: () => 'npx',
}))
jest.mock('next/dist/lib/picocolors', () => ({
bold: (text: string) => text,
cyan: (text: string) => text,
dim: (text: string) => text,
}))
jest.mock('next/dist/lib/upgrade/prepare-upgrade', () => ({
prepareUpgrade: jest.fn(),
}))
jest.mock('next/dist/telemetry/agent-name', () => ({
getAgentName: jest.fn(),
}))
const createSpinner = require('next/dist/build/spinner').default as jest.Mock
const restoreDescriptors: Array<() => void> = []
function normalizedBootstrapCalls(): string[][] {
return jest
.mocked(Log.bootstrap)
.mock.calls.map(([message]) => [String(message).replace(/\\+/g, '/')])
}
function overrideTTY(target: NodeJS.ReadStream | NodeJS.WriteStream): void {
const descriptor = Object.getOwnPropertyDescriptor(target, 'isTTY')
restoreDescriptors.push(() => {
if (descriptor) {
Object.defineProperty(target, 'isTTY', descriptor)
} else {
delete target.isTTY
}
})
Object.defineProperty(target, 'isTTY', {
configurable: true,
value: true,
})
}
describe('agentic upgrade prompts', () => {
const originalPath = process.env.PATH
const originalUseCurrentCli = process.env.__NEXT_UPGRADE_USE_CURRENT_CLI
const originalExitCode = process.exitCode
beforeEach(() => {
jest.resetAllMocks()
process.env.__NEXT_UPGRADE_USE_CURRENT_CLI = '1'
process.exitCode = undefined
jest.mocked(getProjectDir).mockReturnValue('/workspace/app')
jest.mocked(findDir).mockReturnValue('/workspace/app/app')
jest.mocked(createSpinner).mockReturnValue({
stop: jest.fn(),
} as never)
jest.mocked(prepareUpgrade).mockResolvedValue({
status: 'ready',
installedVersion: '14.1.1',
targetVersion: '16.3.5',
references: [
'https://api.github.com/advisories?affects=next',
'https://registry.npmjs.org/next',
],
})
jest.mocked(mkdtemp).mockResolvedValue('/tmp/next-upgrade-test')
jest.mocked(cp).mockResolvedValue(undefined)
jest.mocked(readFile).mockResolvedValue('Run <codemod-command>')
jest.mocked(rm).mockResolvedValue(undefined)
jest.mocked(writeFile).mockResolvedValue(undefined)
jest.mocked(getAgentName).mockResolvedValue('codex')
})
afterEach(() => {
if (originalPath === undefined) {
delete process.env.PATH
} else {
process.env.PATH = originalPath
}
if (originalUseCurrentCli === undefined) {
delete process.env.__NEXT_UPGRADE_USE_CURRENT_CLI
} else {
process.env.__NEXT_UPGRADE_USE_CURRENT_CLI = originalUseCurrentCli
}
process.exitCode = originalExitCode
while (restoreDescriptors.length > 0) {
restoreDescriptors.pop()?.()
}
})
it('uses the planned multiple-agent question and choices', async () => {
delete process.env.__NEXT_UPGRADE_USE_CURRENT_CLI
process.env.PATH = '/agents'
overrideTTY(process.stdin)
overrideTTY(process.stdout)
jest.mocked(getAgentName).mockResolvedValue(null)
jest.mocked(access).mockResolvedValue(undefined)
jest.mocked(stat).mockResolvedValue({ isFile: () => true } as never)
jest.mocked(cliSelect).mockResolvedValue({ id: 'cancel' } as never)
await handoffUpgrade('Prepared upgrade prompt.', '/workspace/app')
const selectOptions = jest.mocked(cliSelect).mock.calls[0][0]
expect({
progress: jest.mocked(Log.info).mock.calls,
prompt: jest.mocked(Log.bootstrap).mock.calls,
menu: {
values: Object.entries(selectOptions.values),
defaultValue: selectOptions.defaultValue,
selected: selectOptions.selected,
unselected: selectOptions.unselected,
indentation: selectOptions.indentation,
},
}).toMatchInlineSnapshot(`
{
"menu": {
"defaultValue": 0,
"indentation": 2,
"selected": "❯",
"unselected": " ",
"values": [
[
"codex",
"Continue with Codex",
],
[
"claude",
"Continue with Claude Code",
],
[
"copy",
"Copy prompt for another coding agent",
],
[
"cancel",
"Cancel",
],
],
},
"progress": [
[
"Looking for coding agents...",
],
],
"prompt": [
[
"",
],
[
" Multiple coding agents detected. Which one would you like to use?",
],
[
" Use ↑/↓ to choose, then press Enter.
",
],
[
" Upgrade cancelled.
",
],
],
}
`)
})
it('uses the planned single-agent question', async () => {
delete process.env.__NEXT_UPGRADE_USE_CURRENT_CLI
process.env.PATH = '/agents'
overrideTTY(process.stdin)
overrideTTY(process.stdout)
jest.mocked(getAgentName).mockResolvedValue(null)
jest.mocked(access).mockImplementation(async (file) => {
if (/[/\\]codex(?:\.(?:exe|cmd|bat|com))?$/i.test(String(file))) return
throw new Error('not found')
})
jest.mocked(stat).mockResolvedValue({ isFile: () => true } as never)
jest.mocked(cliSelect).mockResolvedValue({ id: 'cancel' } as never)
await handoffUpgrade('Prepared upgrade prompt.', '/workspace/app')
expect(Log.bootstrap).toHaveBeenCalledWith(
' Codex detected. Would you like to proceed?'
)
})
it('passes the complete migration prompt to an existing agent', async () => {
await spawnNextUpgrade('/workspace/app', {
revision: 'latest',
verbose: false,
ai: 'security',
})
expect(prepareUpgrade).toHaveBeenCalledWith('/workspace/app')
const [guidePath, guide] = jest.mocked(writeFile).mock.calls[0]
expect(String(guidePath).replace(/\\+/g, '/')).toBe(
'/tmp/next-upgrade-test/docs/01-app/02-guides/upgrading/agentic-upgrade.md'
)
expect(String(guide)).toMatch(
/^Run npx @next\/codemod@\S+ upgrade 16\.3\.5 --yes --skip-adoption$/
)
expect(normalizedBootstrapCalls()).toMatchInlineSnapshot(`
[
[
"Read and follow every applicable instruction in "/tmp/next-upgrade-test/docs/01-app/02-guides/upgrading/agentic-upgrade.md" before proceeding.
We're upgrading the app in "/workspace/app" from Next.js 14.1.1 to 16.3.5 because the installed version is affected by a published security advisory.
References:
- https://api.github.com/advisories?affects=next
- https://registry.npmjs.org/next",
],
]
`)
})
it('renders verbose codemod instructions in the guide', async () => {
await spawnNextUpgrade('/workspace/app', {
revision: 'latest',
verbose: true,
ai: 'security',
})
const [guidePath, guide] = jest.mocked(writeFile).mock.calls[0]
expect(String(guidePath).replace(/\\+/g, '/')).toBe(
'/tmp/next-upgrade-test/docs/01-app/02-guides/upgrading/agentic-upgrade.md'
)
expect(String(guide)).toMatch(/--skip-adoption --verbose$/)
})
it('defaults a bare AI upgrade to security', async () => {
await spawnNextUpgrade('/workspace/app', {
revision: 'latest',
verbose: false,
ai: true,
})
expect(prepareUpgrade).toHaveBeenCalledWith('/workspace/app')
})
})
...@@ -24,6 +24,10 @@ ...@@ -24,6 +24,10 @@
}, },
"include": [ "include": [
"run-tests.js", "run-tests.js",
"evals/lib/**/*",
"evals/next-upgrade/experiments/**/*.ts",
"evals/next-upgrade/lib/**/*.ts",
"evals/next-upgrade/security/setup.ts",
"test/**/*.test.ts", "test/**/*.test.ts",
"test/**/*.test.tsx", "test/**/*.test.tsx",
"test/**/*.util.ts", "test/**/*.util.ts",
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment