Unverified Commit 1ae49102 authored by Benjamin Woodruff ‮'s avatar Benjamin Woodruff Committed by GitHub

agents: Add a `--comments-only` flag to `pr-status` script, trim down context…

agents: Add a `--comments-only` flag to `pr-status` script, trim down context bloat in AGENTS.md (#98781)

- The `pr-status` can be very slow if you have a lot of test failures on
a PR. If the user asks a targeted question about PR review comments,
just fetch that information, it's much faster.
- Context bloat: Rely on the agent to pull in the `pr-status-triage`
skill if they need it, don't bloat the AGENTS.md, except to mention the
skill.
parent a34c3030
...@@ -16,20 +16,47 @@ Use this skill when the user asks about PR status, CI failures, or review commen ...@@ -16,20 +16,47 @@ Use this skill when the user asks about PR status, CI failures, or review commen
## Workflow ## Workflow
1. Run `node scripts/pr-status.js --wait` in the background (timeout 1 min), then read `scripts/pr-status/results/index.md`. Start by fetching data to answer the user's query:
2. Analyze each `job-{id}.md` and `thread-{N}.md` file in `scripts/pr-status/results/` for failures and review feedback.
- For a targeted question about specific review comments, run `node scripts/pr-status.js [PR] --comments-only`. This skips all Actions run, job, log, and flaky-test requests.
- For CI status or general triage, run `node scripts/pr-status.js [PR] --wait` in the background (timeout 1 min).
Then:
1. Read `scripts/pr-status/results/index.md`.
2. For a targeted review question, inspect the relevant `thread-{N}.md`, `review-{id}.md`, or `comment-{id}.md` files. For full triage, also analyze each `job-{id}.md` and review file for failures and feedback.
3. Prioritize blocking jobs first: build, lint, types, then test jobs. 3. Prioritize blocking jobs first: build, lint, types, then test jobs.
4. Treat failures as real until disproven; check the "Known Flaky Tests" section before calling anything flaky. 4. Treat failures as real until disproven; check the "Known Flaky Tests" section before calling anything flaky.
5. Reproduce locally with the same mode and env vars as CI. 5. Reproduce test failures locally with the same mode and environment as CI (e.g. dev or start, webpack or turbopack).
6. After addressing review comments, reply to the thread describing what was done, then resolve it. Use `reply-and-resolve-thread` to do both in one step, or use `reply-thread` + `resolve-thread` separately. See `scripts/pr-status/results/thread-N.md` files for ready-to-use commands. 6. After addressing review comments, reply to the thread describing what was done, then resolve it. Use `reply-and-resolve-thread` to do both in one step, or use `reply-thread` + `resolve-thread` separately. See `scripts/pr-status/results/thread-N.md` files for ready-to-use commands.
7. When the only remaining failures are known flaky tests and no code changes are needed, retrigger the failing CI jobs with `gh run rerun <run-id> --failed`. Then wait 5 minutes and go back to step 1. Repeat this loop up to 5 times. 7. When the only remaining failures are known flaky tests and no code changes are needed, retrigger the failing CI jobs with `gh run rerun <run-id> --failed`. Then wait 5 minutes and go back to step 1. Repeat this loop up to 5 times.
## CI Analysis Tips
- Prioritize CI failures over review comments.
- Prioritize blocking jobs first: build, lint, types, then test jobs.
- Common fast checks:
- `rust check / build` → Run `cargo fmt -- --check`, then `cargo fmt`
- `lint / build` → Run `pnpm prettier --write <file>` for prettier errors
- test failures → Run the specific failing test path locally
Run tests in the mode (e.g.):
```bash
# Development mode with Turbopack
pnpm test-dev-turbo test/path/to/test.ts
# Production build and start with Webpack
pnpm test-start-webpack test/path/to/test.ts
```
## Quick Commands ## Quick Commands
```bash ```bash
node scripts/pr-status.js # current branch PR node scripts/pr-status.js # current branch PR
node scripts/pr-status.js <number> # specific PR node scripts/pr-status.js <number> # specific PR
node scripts/pr-status.js [PR] --wait # background mode, waits for CI to finish node scripts/pr-status.js [PR] --wait # background mode, waits for CI to finish
node scripts/pr-status.js [PR] --comments-only # reviews/comments only; skips CI jobs
node scripts/pr-status.js --skip-flaky-check # skip flaky test detection node scripts/pr-status.js --skip-flaky-check # skip flaky test detection
``` ```
......
...@@ -258,46 +258,10 @@ That symlink does not bring in per-package `node_modules` or a built `packages/n ...@@ -258,46 +258,10 @@ That symlink does not bring in per-package `node_modules` or a built `packages/n
## PR Status (CI Failures and Reviews) ## PR Status (CI Failures and Reviews)
When the user asks about CI failures, PR reviews, or the status of a PR, run the pr-status script: Use `$pr-status-triage` whenever the user asks about CI failures, PR reviews, or
PR status. It contains the `scripts/pr-status.js` commands, prioritization,
```bash review-thread workflow, and CI reproduction guidance. See
node scripts/pr-status.js # Auto-detects PR from current branch `.agents/skills/pr-status-triage/SKILL.md`.
node scripts/pr-status.js <number> # Analyze specific PR by number
```
This generates analysis files in `scripts/pr-status/`.
General triage rules (always apply; `$pr-status-triage` skill expands on these):
- Prioritize blocking failures first: build, lint, types, then tests.
- Assume failures are real until disproven; use "Known Flaky Tests" as context, not auto-dismissal.
- Reproduce with the same CI mode/env vars (especially `IS_WEBPACK_TEST=1` when present).
- For module-resolution/build-graph fixes, use the normal mode-specific test command so package resolution is exercised.
For full triage workflow (failure prioritization, mode selection, CI env reproduction, and common failure patterns), use the `$pr-status-triage` skill:
- Skill file: `.agents/skills/pr-status-triage/SKILL.md`
**Use `$pr-status-triage` for automated analysis** - see `.agents/skills/pr-status-triage/SKILL.md` for the full step-by-step workflow.
**CI Analysis Tips:**
- Prioritize CI failures over review comments
- Prioritize blocking jobs first: build, lint, types, then test jobs
- Common fast checks:
- `rust check / build` → Run `cargo fmt -- --check`, then `cargo fmt`
- `lint / build` → Run `pnpm prettier --write <file>` for prettier errors
- test failures → Run the specific failing test path locally
**Run tests in the right mode:**
```bash
# Dev mode (Turbopack)
pnpm test-dev-turbo test/path/to/test.ts
# Prod mode
pnpm test-start-turbo test/path/to/test.ts
```
## GitHub Pull Requests ## GitHub Pull Requests
......
...@@ -895,8 +895,14 @@ function generateIndexMd( ...@@ -895,8 +895,14 @@ function generateIndexMd(
lines.push('') lines.push('')
} }
// Add PR reviews section if we have review data appendReviewData(lines, reviewData)
if (reviewData) {
return lines.join('\n')
}
function appendReviewData(lines, reviewData) {
if (!reviewData) return
const { reviews, reviewThreads, prComments } = reviewData const { reviews, reviewThreads, prComments } = reviewData
// Filter reviews to only include meaningful ones // Filter reviews to only include meaningful ones
...@@ -989,11 +995,62 @@ function generateIndexMd( ...@@ -989,11 +995,62 @@ function generateIndexMd(
) )
} }
} }
}
function generateCommentsIndexMd(branchInfo, reviewData) {
const lines = ['# PR Review Comments Report', '']
if (branchInfo.branchName) {
lines.push(`Branch: ${branchInfo.branchName}`)
} }
lines.push(`PR: #${branchInfo.prNumber}`, '')
appendReviewData(lines, reviewData)
return lines.join('\n') return lines.join('\n')
} }
async function prepareOutputDirectory() {
console.log('Cleaning output directory...')
await fs.rm(OUTPUT_ROOT, { recursive: true, force: true })
await fs.mkdir(RESULTS_DIR, { recursive: true })
await fs.mkdir(INTERMEDIATE_DIR, { recursive: true })
}
function fetchReviewData(prNumber) {
console.log('Fetching PR reviews and comments...')
const reviews = getPRReviews(prNumber)
const reviewThreads = getPRReviewThreads(prNumber)
const prComments = getPRComments(prNumber)
console.log(
`Found ${reviews.length} reviews, ${reviewThreads.length} review threads, ${prComments.length} general comments`
)
return { reviews, reviewThreads, prComments }
}
async function writeReviewFiles(reviewData) {
for (let i = 0; i < reviewData.reviewThreads.length; i++) {
const thread = reviewData.reviewThreads[i]
await fs.writeFile(
resultPath(`thread-${i + 1}.md`),
generateThreadMd(thread, i)
)
}
for (const review of reviewData.reviews) {
if (review.body?.trim()) {
await fs.writeFile(
resultPath(`review-${review.id}.md`),
generateReviewMd(review)
)
}
}
for (const comment of reviewData.prComments) {
await fs.writeFile(
resultPath(`comment-${comment.id}.md`),
generateCommentMd(comment)
)
}
}
function generateJobMd(jobMetadata, testResults, testFiles, sections) { function generateJobMd(jobMetadata, testResults, testFiles, sections) {
const duration = formatDuration( const duration = formatDuration(
jobMetadata.started_at, jobMetadata.started_at,
...@@ -1400,10 +1457,7 @@ async function getFlakyTests(currentBranch, runsToCheck = 5) { ...@@ -1400,10 +1457,7 @@ async function getFlakyTests(currentBranch, runsToCheck = 5) {
*/ */
async function runAnalysis(prNumberArg, skipFlakyCheck) { async function runAnalysis(prNumberArg, skipFlakyCheck) {
// Step 1: Delete and recreate output directory // Step 1: Delete and recreate output directory
console.log('Cleaning output directory...') await prepareOutputDirectory()
await fs.rm(OUTPUT_ROOT, { recursive: true, force: true })
await fs.mkdir(RESULTS_DIR, { recursive: true })
await fs.mkdir(INTERMEDIATE_DIR, { recursive: true })
// Step 2: Get branch info // Step 2: Get branch info
console.log('Getting branch info...') console.log('Getting branch info...')
...@@ -1464,14 +1518,7 @@ async function runAnalysis(prNumberArg, skipFlakyCheck) { ...@@ -1464,14 +1518,7 @@ async function runAnalysis(prNumberArg, skipFlakyCheck) {
// Fetch PR reviews if we have a PR number // Fetch PR reviews if we have a PR number
let reviewData = null let reviewData = null
if (branchInfo.prNumber) { if (branchInfo.prNumber) {
console.log('Fetching PR reviews and comments...') reviewData = fetchReviewData(branchInfo.prNumber)
const reviews = getPRReviews(branchInfo.prNumber)
const reviewThreads = getPRReviewThreads(branchInfo.prNumber)
const prComments = getPRComments(branchInfo.prNumber)
reviewData = { reviews, reviewThreads, prComments }
console.log(
`Found ${reviews.length} reviews, ${reviewThreads.length} review threads, ${prComments.length} general comments`
)
} }
// Check if we should write an early report (no failed jobs yet) // Check if we should write an early report (no failed jobs yet)
...@@ -1485,30 +1532,7 @@ async function runAnalysis(prNumberArg, skipFlakyCheck) { ...@@ -1485,30 +1532,7 @@ async function runAnalysis(prNumberArg, skipFlakyCheck) {
// Write review files if we have PR data // Write review files if we have PR data
if (reviewData) { if (reviewData) {
// Write individual thread files await writeReviewFiles(reviewData)
for (let i = 0; i < reviewData.reviewThreads.length; i++) {
const thread = reviewData.reviewThreads[i]
await fs.writeFile(
resultPath(`thread-${i + 1}.md`),
generateThreadMd(thread, i)
)
}
// Write individual review files for reviews with comments
for (const review of reviewData.reviews) {
if (review.body && review.body.trim()) {
await fs.writeFile(
resultPath(`review-${review.id}.md`),
generateReviewMd(review)
)
}
}
// Write individual comment files
for (const comment of reviewData.prComments) {
await fs.writeFile(
resultPath(`comment-${comment.id}.md`),
generateCommentMd(comment)
)
}
} }
const emptyCategorizedJobs = { const emptyCategorizedJobs = {
...@@ -1605,30 +1629,7 @@ async function runAnalysis(prNumberArg, skipFlakyCheck) { ...@@ -1605,30 +1629,7 @@ async function runAnalysis(prNumberArg, skipFlakyCheck) {
// Step 7: Write PR review files if we have PR data // Step 7: Write PR review files if we have PR data
if (reviewData) { if (reviewData) {
console.log('Generating review files...') console.log('Generating review files...')
// Write individual thread files await writeReviewFiles(reviewData)
for (let i = 0; i < reviewData.reviewThreads.length; i++) {
const thread = reviewData.reviewThreads[i]
await fs.writeFile(
resultPath(`thread-${i + 1}.md`),
generateThreadMd(thread, i)
)
}
// Write individual review files for reviews with comments
for (const review of reviewData.reviews) {
if (review.body?.trim()) {
await fs.writeFile(
resultPath(`review-${review.id}.md`),
generateReviewMd(review)
)
}
}
// Write individual comment files
for (const comment of reviewData.prComments) {
await fs.writeFile(
resultPath(`comment-${comment.id}.md`),
generateCommentMd(comment)
)
}
} }
// Step 8: Check for known flaky tests across branches (skip with --skip-flaky-check) // Step 8: Check for known flaky tests across branches (skip with --skip-flaky-check)
...@@ -1666,6 +1667,28 @@ async function runAnalysis(prNumberArg, skipFlakyCheck) { ...@@ -1666,6 +1667,28 @@ async function runAnalysis(prNumberArg, skipFlakyCheck) {
return { runId: latestRun.id, isRunInProgress } return { runId: latestRun.id, isRunInProgress }
} }
async function runCommentsAnalysis(prNumberArg) {
await prepareOutputDirectory()
console.log('Getting pull request info...')
const branchInfo = getBranchInfo(prNumberArg)
if (!branchInfo.prNumber) {
throw new Error(
'No pull request found. Pass a PR number or run this command from a PR branch.'
)
}
console.log(`Branch: ${branchInfo.branchName}, PR: ${branchInfo.prNumber}`)
const reviewData = fetchReviewData(branchInfo.prNumber)
await writeReviewFiles(reviewData)
await fs.writeFile(
resultPath('index.md'),
generateCommentsIndexMd(branchInfo, reviewData)
)
console.log(`\nDone! Output written to ${RESULTS_DIR}/index.md`)
}
async function main() { async function main() {
// Dispatch subcommands // Dispatch subcommands
const subcommand = process.argv[2] const subcommand = process.argv[2]
...@@ -1713,8 +1736,14 @@ async function main() { ...@@ -1713,8 +1736,14 @@ async function main() {
const args = process.argv.slice(2) const args = process.argv.slice(2)
const waitFlag = args.includes('--wait') const waitFlag = args.includes('--wait')
const skipFlakyCheck = args.includes('--skip-flaky-check') const skipFlakyCheck = args.includes('--skip-flaky-check')
const commentsOnly = args.includes('--comments-only')
const prNumberArg = args.find((a) => !a.startsWith('--')) const prNumberArg = args.find((a) => !a.startsWith('--'))
if (commentsOnly) {
await runCommentsAnalysis(prNumberArg)
return
}
// Run the initial analysis // Run the initial analysis
const { runId, isRunInProgress } = await runAnalysis( const { runId, isRunInProgress } = await runAnalysis(
prNumberArg, prNumberArg,
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment