Unverified Commit aa6086d0 authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

/pr-status (former /ci-failures): fetch PR reviews too (#89082)

### What?

* allow to pass PR number
* fetch PR reviews
* Rename to /pr-status update agent instructions
parent 4a73f528
# Check CI Failures # PR Status
Analyze failing tests from PR CI runs. Analyze PR status including CI failures and review comments.
## Usage ## Usage
``` ```
/ci-failures /pr-status
``` ```
Automatically detects PR from current branch. Automatically detects PR from current branch.
## Instructions ## Instructions
1. Run the script to fetch CI failure data: 1. Run the script to fetch PR status data:
```bash ```bash
node scripts/ci-failures.js node scripts/pr-status.js
``` ```
This fetches workflow runs, failed jobs, and logs, then generates markdown files. This fetches workflow runs, failed jobs, logs, and PR review comments, then generates markdown files.
2. Read the generated index file for a summary: 2. Read the generated index file for a summary:
```bash ```bash
# Read scripts/ci-failures/index.md # Read scripts/pr-status/index.md
``` ```
The index shows all failed jobs with links to details. The index shows failed jobs, PR reviews, and inline review comments with links to details.
3. Spawn parallel haiku subagents to analyze the failing jobs (limit to 3-4 to avoid rate limits) 3. Spawn parallel haiku subagents to analyze the failing jobs (limit to 3-4 to avoid rate limits)
- **Agent prompt template** (copy-paste for each agent): - **Agent prompt template** (copy-paste for each agent):
``` ```
Analyze CI results for these jobs: scripts/ci-failures/job-{id1}.md scripts/ci-failures/job-{id2}.md Analyze CI results for these jobs: scripts/pr-status/job-{id1}.md scripts/pr-status/job-{id2}.md
For each failing test, extract: For each failing test, extract:
1. TEST FILE: (full path, e.g., test/production/required-server-files-ssr-404/test/index.test.ts) 1. TEST FILE: (full path, e.g., test/production/required-server-files-ssr-404/test/index.test.ts)
2. TEST NAME: (the specific test case name)1 2. TEST NAME: (the specific test case name)
3. JOB TYPE: (the kind of the job, e.g. turbopack production, webpack dev, rust check) 3. JOB TYPE: (the kind of the job, e.g. turbopack production, webpack dev, rust check)
4. EXPECTED: (exact expected value from assertion) 4. EXPECTED: (exact expected value from assertion)
5. RECEIVED: (exact received value from assertion) 5. RECEIVED: (exact received value from assertion)
...@@ -49,27 +49,53 @@ Automatically detects PR from current branch. ...@@ -49,27 +49,53 @@ Automatically detects PR from current branch.
``` ```
4. **Deduplicate by test file** before summarizing: 4. Spawn parallel haiku subagents to analyze review comments (if any review threads exist):
- **Agent prompt template**:
```
Analyze PR review comments from these files: scripts/pr-status/thread-{i}.md scripts/pr-status/review-{id}.md
For each review thread/comment, extract:
1. FILE: (the file path being reviewed)
2. REVIEWER: (who left the comment)
3. STATUS: (Open/Resolved for threads, APPROVED/CHANGES_REQUESTED/COMMENTED for reviews)
4. TYPE: (code-style|bug|design|question|suggestion|nitpick|blocker)
5. SUMMARY: (one sentence summary of the feedback)
6. ACTION REQUIRED: (yes/no - does this require changes?)
7. PRIORITY: (high if CHANGES_REQUESTED or blocker, medium if open suggestion, low if resolved or nitpick)
Return findings grouped by file path.
```
5. **Deduplicate by test file** before summarizing:
- Group all failures by TEST FILE path, not by CI job name - Group all failures by TEST FILE path, not by CI job name
- If multiple jobs fail the same test file, count them but report once - If multiple jobs fail the same test file, count them but report once
- Identify systemic issues (same test failing across many jobs) - Identify systemic issues (same test failing across many jobs)
5. Analyze failures and create a summary **grouped by test file**: 6. Analyze failures and review comments, create a summary:
**CI Failures (grouped by test file):**
| Test File | Type. | Issue (Expected vs Received) | Jobs | Priority | | Test File | Type | Issue (Expected vs Received) | Jobs | Priority |
| --------------------------------------------------- | -------------- | ----------------------------------- | ---- | -------- | | --------------------------------------------------- | -------------- | ----------------------------------- | ---- | -------- |
| `test/production/required-server-files-ssr-404/...` | Turbopack prod | `"second"` vs `"[slug]"` (routing) | 3 | HIGH | | `test/production/required-server-files-ssr-404/...` | Turbopack prod | `"second"` vs `"[slug]"` (routing) | 3 | HIGH |
| `test/integration/server-side-dev-errors/...` | webpack dev | source map paths wrong | 5 | HIGH | | `test/integration/server-side-dev-errors/...` | webpack dev | source map paths wrong | 5 | HIGH |
| `test/e2e/app-dir/disable-logging-route/...` | prod | "Compiling" appearing when disabled | 2 | MEDIUM | | `test/e2e/app-dir/disable-logging-route/...` | prod | "Compiling" appearing when disabled | 2 | MEDIUM |
| N/A | rust check | Formatting incorrect | 2 | MEDIUM | | N/A | rust check | Formatting incorrect | 2 | MEDIUM |
6. Recommend fixes: **Review Feedback (grouped by file):**
- **HIGH priority**: Show specific expected vs actual values, include test file path
- **MEDIUM priority**: Identify root cause pattern
- **LOW priority**: Mark as likely flaky/transient
- Do not try to fix these failures. | File | Reviewer | Type | Summary | Action Required | Priority |
- If failures would require complex analysis and there are multiple problems, only do some basic analysis and point out that further investigation is needed and could be performant when requested. | ---------------- | -------- | ---------- | -------------------------- | --------------- | -------- |
| `src/server/...` | reviewer | suggestion | Consider using async/await | yes | MEDIUM |
| `test/e2e/...` | reviewer | nitpick | Typo in comment | no | LOW |
| N/A | reviewer | blocker | Missing error handling | yes | HIGH |
7. Recommend fixes:
- **HIGH priority**: Show specific expected vs actual values, include test file path, address blocker review comments
- **MEDIUM priority**: Identify root cause pattern, address open suggestions
- **LOW priority**: Mark as likely flaky/transient, note resolved/nitpick comments
- Do not try to fix these failures or address review comments without user confirmation.
- If failures would require complex analysis and there are multiple problems, only do some basic analysis and point out that further investigation is needed and could be performed when requested.
## Failure Categories ## Failure Categories
...@@ -81,3 +107,13 @@ Automatically detects PR from current branch. ...@@ -81,3 +107,13 @@ Automatically detects PR from current branch.
- **Routing/SSR**: Dynamic params not resolved, wrong status codes, JSON parse errors - **Routing/SSR**: Dynamic params not resolved, wrong status codes, JSON parse errors
- **Source Maps**: `webpack-internal://` paths, wrong line numbers, missing code frames - **Source Maps**: `webpack-internal://` paths, wrong line numbers, missing code frames
- **CLI Output**: Missing warnings, wrong log order, "Ready" printed before errors - **CLI Output**: Missing warnings, wrong log order, "Ready" printed before errors
## Review Comment Categories
- **code-style**: Formatting, naming conventions, code organization
- **bug**: Potential bugs or logic errors
- **design**: Architectural or design concerns
- **question**: Questions about implementation or intent
- **suggestion**: Non-blocking improvements
- **nitpick**: Minor issues that don't require changes
- **blocker**: Must be addressed before merge
...@@ -181,28 +181,23 @@ pnpm prettier-fix # Fix formatting only ...@@ -181,28 +181,23 @@ pnpm prettier-fix # Fix formatting only
pnpm types # TypeScript type checking pnpm types # TypeScript type checking
``` ```
## Investigating CI Test Failures ## PR Status (CI Failures and Reviews)
**Use `/ci-failures` for automated analysis** - analyzes failing jobs in parallel and groups by test file. When the user asks about CI failures, PR reviews, or the status of a PR, run the pr-status script:
**CI Analysis Tips:** ```bash
node scripts/pr-status.js # Auto-detects PR from current branch
node scripts/pr-status.js <number> # Analyze specific PR by number
```
- Don't spawn too many parallel agents hitting GitHub API (causes rate limits) This fetches CI workflow runs, failed jobs, logs, and PR review comments, generating markdown files in `scripts/pr-status/`.
- Prioritize blocking jobs first: lint, types, then test jobs
- Use `gh api` for logs (works on in-progress runs), not `gh run view --log`
**Quick triage:** **Use `/pr-status` for automated analysis** - analyzes failing jobs and review comments in parallel, groups failures by test file.
```bash **CI Analysis Tips:**
# List failed jobs for a PR
gh pr checks <pr-number> | grep fail
# Get failed job names
gh run view <run-id> --json jobs --jq '.jobs[] | select(.conclusion == "failure") | .name'
# Search job logs for errors (completed runs only - use gh api for in-progress) - Prioritize blocking jobs first: build, lint, types, then test jobs
gh run view <run-id> --job <job-id> --log 2>&1 | grep -E "FAIL|Error|error:" | head -30 - Prioritize CI failures over review comments
```
**Common failure patterns:** **Common failure patterns:**
......
ci-failures/* pr-status/*
\ No newline at end of file \ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment