eval: resolve and classify the audit suite (#100748)
The October pass@1 study left seven tasks in `audit` because their graders, framework contracts, or failures were not trustworthy enough to classify. This resolves every task: three repaired tasks move to regression, two valid competency tasks remain discriminatory, and two invalid public measurements are unpublished. ## Classification | Task | Before | Change and evidence | Final | | --- | --- | --- | --- | | `026-no-serial-await` | Generic JavaScript task with solution leakage and source-shape grading | Current score cannot support a Next.js competency claim | Unpublished; removed from runnable suites | | `029-use-cache-directive` | Missing Cache Components setup and incomplete flow verification | Requires Cache Components and verifies the cached read, form-bound action, shared tag, and two-argument revalidation as one flow. Gemini failed both scored treatments; confirmations were mixed | Competency | | `030-app-router-migration-hard` | Broad migration with incomplete coverage and unstable judges | Current score cannot distinguish migration ability from grader gaps | Unpublished; held out until split or substantially repaired | | `035-connection-dynamic` | Accepted `io()` for a real-request requirement and used three variable judge checks | Requires `connection()`, preserves the stable frame, and consolidates the request-specific behavior check. All 8 valid treatments passed | Regression | | `038-refresh-settings` | Grader did not connect the mutation, UI, and `refresh()` | Ties the complete Server Action flow together and rejects redirects. All 9 valid calibration results passed | Regression | | `041-optimize-ppr-shell` | Vague prompt and one implementation-oriented judge | Names the visible shell and independently revealing sections, keeps Cache Components enabled, and grades the rendered component tree. All 8 valid treatments passed | Regression | | `051-prefetch-until-navigation` | Broad source checks around navigation-stage prefetching | Scopes the title prefetch and deferred details to the destination. Frontier passed; Luna Base failed its scored attempt and both confirmations, while Luna AGENTS.md passed | Competency | The final suite contains 22 regression tasks and 7 competency tasks. No task remains in `audit`. `044-uses-nextjs` remains outside these suites because it measures framework preference for a0. ## Repaired-task results Pass@1 against the exact `16.5.0-canary.27` package: | Model | Treatment | `029` | `035` | `041` | | --- | --- | ---: | ---: | ---: | | Claude Opus 5.5 | Base | 1/1 | 1/1 | 1/1 | | Claude Opus 5.5 | AGENTS.md | 1/1 | 1/1 | 1/1 | | GPT-6 Sol | Base | 1/1 | 1/1 | 1/1 | | GPT-6 Sol | AGENTS.md | 1/1 | 1/1 | 1/1 | | Gemini 4 Argon | Base | 0/1 | 1/1 | 1/1 | | Gemini 4 Argon | AGENTS.md | 0/1 | 1/1 | 1/1 | | GPT-6 Luna | Base | 1/1 | 1/1 | 1/1 | | GPT-6 Luna | AGENTS.md | discarded | 1/1 | 1/1 | The discarded Luna `029` attempt timed out without a transcript and is not a model result. Diagnostic reruns do not change pass@1: Gemini Base passed one of two confirmations and repeated the same substantive failure once; Gemini AGENTS.md passed both confirmations. The two scored Gemini cells remain failed and should be reported as unstable. ## Validation - `pnpm prettier ... --check` on all changed files - `npx eslint --config eslint.config.mjs` on all changed graders - `pnpm eval --suite competency --dry` - `pnpm eval --suite regression --dry` - `git diff --check` Stacked on #100742, which adds suite selection and the classification protocol. <!-- NEXT_JS_LLM -->
Showing
Please
register
or
sign in
to comment