Unverified Commit 683180bf authored by JJ Kasper's avatar JJ Kasper Committed by GitHub

Centralize test timings to fix shard divergence in CI (#90672)

## Summary

For context we discovered jobs were not reliably always running in
https://github.com/vercel/next.js/pull/90668 as addapters + turbopack
group ran more tests than just turbopack group.

- Each sharded test sub-job independently fetched test timings from KV
via a turbo task. Because turbo cache keys can vary across job groups
and KV can return slightly different data at different times, the
sharding algorithm produced different test-to-shard assignments across
groups — causing some tests to be missing entirely and others to run in
multiple shards.

- Adds a `--require-timings` flag to `run-tests.js` that fails loudly if
`test-timings.json` can't be loaded from disk, preventing silent
fallback to KV or round-robin.

- Adds a `testTimingsArtifact` input to `build_reusable.yml` that
downloads a pre-built timings artifact instead of fetching via turbo.
When unset (default), existing behavior is preserved.

- In both `test_e2e_deploy_release.yml` and `build_and_test.yml`,
timings are now fetched once in a setup job, uploaded as a GitHub
artifact, and downloaded by all sub-jobs — guaranteeing identical shard
assignments across all job groups.

## Test plan

- [ ] Verify `build_and_test.yml` jobs pick up the shared `test-timings`
artifact
- [ ] Verify `test_e2e_deploy_release.yml` deploy jobs pick up the
shared `test-timings` artifact
- [ ] Verify workflows that don't pass `testTimingsArtifact` (e.g.
`integration_tests_reusable.yml`, `pull_request_stats.yml`) still use
the turbo fallback path unchanged
parent 39e8fb2a
This diff is collapsed.
...@@ -80,6 +80,12 @@ on: ...@@ -80,6 +80,12 @@ on:
type: string type: string
default: '' default: ''
testTimingsArtifact:
description: 'Name of an uploaded artifact containing test-timings.json. When set, download it instead of fetching via turbo.'
required: false
type: string
default: ''
browser: browser:
description: 'Browser to use for tests' description: 'Browser to use for tests'
required: false required: false
...@@ -307,7 +313,24 @@ jobs: ...@@ -307,7 +313,24 @@ jobs:
- run: pnpm playwright install --with-deps ${{ inputs.browser }} - run: pnpm playwright install --with-deps ${{ inputs.browser }}
if: ${{ inputs.skipInstallBuild != 'yes' }} if: ${{ inputs.skipInstallBuild != 'yes' }}
- run: pnpm dlx turbo@${TURBO_VERSION} run get-test-timings -- --build ${{ github.sha }} - name: Download pre-built test timings
if: ${{ inputs.testTimingsArtifact != '' }}
uses: actions/download-artifact@v4
with:
name: ${{ inputs.testTimingsArtifact }}
- name: Verify test timings
if: ${{ inputs.testTimingsArtifact != '' }}
run: |
if [ ! -f test-timings.json ]; then
echo "::error::test-timings.json not found"
exit 1
fi
echo "Test timings loaded ($(wc -c < test-timings.json) bytes)"
- name: Fetch test timings via turbo
if: ${{ inputs.testTimingsArtifact == '' }}
run: pnpm dlx turbo@${TURBO_VERSION} run get-test-timings -- --build ${{ github.sha }}
- run: ${{ inputs.afterBuild }} - run: ${{ inputs.afterBuild }}
# defaults.run.shell sets a stronger options (`-leo pipefail`) # defaults.run.shell sets a stronger options (`-leo pipefail`)
......
...@@ -76,6 +76,31 @@ jobs: ...@@ -76,6 +76,31 @@ jobs:
name: Extract `next` version name: Extract `next` version
run: echo 'value=${{ fromJson(steps.nextPackageInfo.outputs.value).version }}' >> "$GITHUB_OUTPUT" run: echo 'value=${{ fromJson(steps.nextPackageInfo.outputs.value).version }}' >> "$GITHUB_OUTPUT"
- name: Install dependencies
run: pnpm install
- name: Fetch test timings
run: node run-tests.js --timings --write-timings -g 1/1
continue-on-error: true
env:
KV_REST_API_URL: ${{ secrets.KV_REST_API_URL }}
KV_REST_API_TOKEN: ${{ secrets.KV_REST_API_TOKEN }}
- name: Ensure test timings file exists
run: |
if [ ! -f test-timings.json ]; then
echo "No timings fetched, creating empty timings file"
echo '{}' > test-timings.json
fi
- name: Upload test timings
uses: actions/upload-artifact@v4
with:
name: test-timings
path: test-timings.json
retention-days: 1
if-no-files-found: error
test-deploy-webpack: test-deploy-webpack:
name: Run Deploy Tests (Webpack) name: Run Deploy Tests (Webpack)
needs: setup needs: setup
...@@ -97,7 +122,8 @@ jobs: ...@@ -97,7 +122,8 @@ jobs:
VERCEL_CLI_VERSION="${{ github.event.inputs.vercelCliVersion || 'vercel@latest' }}" \ VERCEL_CLI_VERSION="${{ github.event.inputs.vercelCliVersion || 'vercel@latest' }}" \
NEXT_TEST_DEPLOY_SCRIPT_PATH="${{ github.event.inputs.deployScriptPath || '' }}" \ NEXT_TEST_DEPLOY_SCRIPT_PATH="${{ github.event.inputs.deployScriptPath || '' }}" \
NEXT_TEST_DEPLOY_LOGS_SCRIPT_PATH="${{ github.event.inputs.deployLogsScriptPath || '' }}" \ NEXT_TEST_DEPLOY_LOGS_SCRIPT_PATH="${{ github.event.inputs.deployLogsScriptPath || '' }}" \
node run-tests.js --timings -g ${{ matrix.group }} -c 2 --type e2e node run-tests.js --timings --require-timings -g ${{ matrix.group }} -c 2 --type e2e
testTimingsArtifact: 'test-timings'
skipNativeBuild: 'yes' skipNativeBuild: 'yes'
skipNativeInstall: 'no' skipNativeInstall: 'no'
stepName: 'test-deploy-webpack-${{ matrix.group }}' stepName: 'test-deploy-webpack-${{ matrix.group }}'
...@@ -140,7 +166,8 @@ jobs: ...@@ -140,7 +166,8 @@ jobs:
VERCEL_CLI_VERSION="${{ github.event.inputs.vercelCliVersion || 'vercel@latest' }}" \ VERCEL_CLI_VERSION="${{ github.event.inputs.vercelCliVersion || 'vercel@latest' }}" \
NEXT_TEST_DEPLOY_SCRIPT_PATH="${{ github.event.inputs.deployScriptPath || '' }}" \ NEXT_TEST_DEPLOY_SCRIPT_PATH="${{ github.event.inputs.deployScriptPath || '' }}" \
NEXT_TEST_DEPLOY_LOGS_SCRIPT_PATH="${{ github.event.inputs.deployLogsScriptPath || '' }}" \ NEXT_TEST_DEPLOY_LOGS_SCRIPT_PATH="${{ github.event.inputs.deployLogsScriptPath || '' }}" \
node run-tests.js --timings -g ${{ matrix.group }} -c 2 --type e2e node run-tests.js --timings --require-timings -g ${{ matrix.group }} -c 2 --type e2e
testTimingsArtifact: 'test-timings'
skipNativeBuild: 'yes' skipNativeBuild: 'yes'
skipNativeInstall: 'no' skipNativeInstall: 'no'
stepName: 'test-deploy-turbopack-${{ matrix.group }}' stepName: 'test-deploy-turbopack-${{ matrix.group }}'
...@@ -182,7 +209,8 @@ jobs: ...@@ -182,7 +209,8 @@ jobs:
NEXT_EXTERNAL_TESTS_FILTERS="test/deploy-tests-manifest.json" \ NEXT_EXTERNAL_TESTS_FILTERS="test/deploy-tests-manifest.json" \
NEXT_TEST_VERSION="${{ github.event.inputs.nextVersion || needs.setup.outputs.next-version || 'canary' }}" \ NEXT_TEST_VERSION="${{ github.event.inputs.nextVersion || needs.setup.outputs.next-version || 'canary' }}" \
VERCEL_CLI_VERSION="${{ github.event.inputs.vercelCliVersion || 'vercel@latest' }}" \ VERCEL_CLI_VERSION="${{ github.event.inputs.vercelCliVersion || 'vercel@latest' }}" \
node run-tests.js --timings -g ${{ matrix.group }} -c 2 --type e2e node run-tests.js --timings --require-timings -g ${{ matrix.group }} -c 2 --type e2e
testTimingsArtifact: 'test-timings'
skipNativeBuild: 'yes' skipNativeBuild: 'yes'
skipNativeInstall: 'no' skipNativeInstall: 'no'
stepName: 'test-deploy-deploy-${{ matrix.group }}' stepName: 'test-deploy-deploy-${{ matrix.group }}'
......
...@@ -32,6 +32,11 @@ let argv = require('yargs/yargs')(process.argv.slice(2)) ...@@ -32,6 +32,11 @@ let argv = require('yargs/yargs')(process.argv.slice(2))
.boolean('dry') .boolean('dry')
.boolean('print-tests') .boolean('print-tests')
.describe('print-tests', 'Prints the test files that will be run') .describe('print-tests', 'Prints the test files that will be run')
.boolean('require-timings')
.describe(
'require-timings',
'Require test-timings.json from disk; fail if missing instead of falling back to KV or round-robin'
)
.boolean('local') .boolean('local')
.alias('c', 'concurrency').argv .alias('c', 'concurrency').argv
...@@ -233,6 +238,7 @@ async function main() { ...@@ -233,6 +238,7 @@ async function main() {
testPattern: argv.testPattern ?? false, testPattern: argv.testPattern ?? false,
type: argv.type ?? false, type: argv.type ?? false,
retries: argv.retries ?? DEFAULT_NUM_RETRIES, retries: argv.retries ?? DEFAULT_NUM_RETRIES,
requireTimings: argv.requireTimings ?? false,
dry: argv.dry ?? false, dry: argv.dry ?? false,
local: argv.local ?? false, local: argv.local ?? false,
printTests: argv.printTests ?? false, printTests: argv.printTests ?? false,
...@@ -322,13 +328,21 @@ async function main() { ...@@ -322,13 +328,21 @@ async function main() {
prevTimings = JSON.parse(await fsp.readFile(timingsFile, 'utf8')) prevTimings = JSON.parse(await fsp.readFile(timingsFile, 'utf8'))
console.log('Loaded test timings from disk successfully') console.log('Loaded test timings from disk successfully')
} catch (_) { } catch (_) {
if (options.requireTimings) {
console.error(
'ERROR: --require-timings is set but test-timings.json could not be loaded from disk:',
_
)
await cleanUpAndExit(1)
return
}
console.error( console.error(
'Failed to load test timings from disk. Proceeding to fetch from KV store. Original error: ', 'Failed to load test timings from disk. Proceeding to fetch from KV store. Original error: ',
_ _
) )
} }
if (!prevTimings) { if (!prevTimings && !options.requireTimings) {
try { try {
prevTimings = await getTestTimings() prevTimings = await getTestTimings()
if (prevTimings) { if (prevTimings) {
...@@ -412,6 +426,15 @@ async function main() { ...@@ -412,6 +426,15 @@ async function main() {
Math.round(groupTimes[curGroupIdx]) + 's' Math.round(groupTimes[curGroupIdx]) + 's'
) )
} else { } else {
if (options.requireTimings) {
console.error(
'ERROR: --require-timings is set but no timing data is available for sharding. ' +
'Ensure test-timings.json is provided.'
)
await cleanUpAndExit(1)
return
}
// assign every nth test "round-robin" to the group, so that similar slow // assign every nth test "round-robin" to the group, so that similar slow
// tests tend not to get clustered together // tests tend not to get clustered together
tests = tests.filter((_value, idx) => idx % groupTotal === groupPos - 1) tests = tests.filter((_value, idx) => idx % groupTotal === groupPos - 1)
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment