Rebuild upgrade evals around delivered apps (#99880)
### Why? Upgrade evals need to verify the app an agent delivers and whether it communicates security/latest notices correctly. ### How? - Run six cases with Codex and Claude: same-major and cross-major upgrades for security and latest, plus one nudge per policy. Exercise the built candidate against frozen upstream metadata. - Capture committed, dirty and untracked changes from delivered worktrees. Require successful candidate invocation and verify manifests, lockfiles, installed versions, TypeScript, build and HTTP/browser behavior in a clean Sandbox. - Require nudge emission and a policy-specific notification or question about proceeding in the final response. Fail any user-facing prompt-injection discussion or language discrediting Next.js or its nudge. Require completed boolean judge verdicts and distinguish failed checks from invalid execution or grading evidence. - Reuse the existing agent-eval runner and tools. Patch its lifecycle for delivery/judge evidence and shared fixture preparation; stop the preparation VM when its last consumer cancels and delete snapshots that arrive after cancellation. - Remove obsolete Future cases and duplicate fixtures/helpers. Generate starting-app lockfiles during setup and share credential-free tooling and app preparation. <!-- NEXT_JS_LLM -->
Showing
File moved
File moved
File moved
File moved
File moved
File moved
File moved
File moved
File moved
File moved
File moved
File moved
File moved
File moved
File moved
File moved
File moved
File moved
File moved
File moved
File moved
File moved
File moved
File moved
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
Please
register
or
sign in
to comment