Add CI for upgrade evals with shared preparation (#99908)
Stacked on #99880. ### Why? Upgrade evals need CI feedback with reusable preparation and failure evidence that reviewers can read directly in the job log. ### How? - Run all twelve Codex/Claude trials in parallel through `pnpm eval:upgrade --all` on relevant same-repository PR changes and manual dispatch. Check out the exact PR head to evaluate the reviewed candidate. - Prepare candidate packages once per batch. Reuse pnpm/Turbo caches and expiring credential-free Sandbox tooling snapshots, then share app preparation between agent forks. Save replacement snapshot IDs even after a cache hit or eval failure. - Use repository credentials to pull the existing Vercel project's OIDC token. Run a pinned Vercel CLI only for authentication, without a root CLI dependency or GitHub deployment record. - Fail CI for failed or invalid trials and retain transcripts, delivered apps and grading evidence for 14 days. Print brief case results, then collect failed/invalid verdicts and redacted native case logs at the end with workflow-command parsing disabled. <!-- NEXT_JS_LLM -->
Showing
This diff is collapsed.
Please
register
or
sign in
to comment