Unverified Commit 13fcfabf authored by Benjamin Woodruff's avatar Benjamin Woodruff Committed by GitHub

refactor(CI): Convert daily turbopack (areweturboyet) integration tests into…

refactor(CI): Convert daily turbopack (areweturboyet) integration tests into reusable workflows (#76251)

This builds on top of work that @timneutkens did in the
`wbinnssmith/try-ci-test` branch to rebuild the integration test CI on
top of `build_reusable.yml` and work around issues with the
`next-integration-stat`, but takes it a few steps further:

- As much of the integration test logic as possible is moved into a
single re-usable workflow so that we won't need to duplicate logic
between 6 workflows going forward (turbopack dev, turbopack build,
turbopack examples, rspack dev, rspack build, rspack examples). Right
now this just ports turbopack dev and build.

- Cleaned up a ton of legacy dead/broken code in
`next-integration-stat`. We might want to revive the comment-on-PR
functionality, so I didn't completely remove it, just the very obviously
broken parts (e.g. references to a >2-year-old stale branch of the
turborepo repository).

---

Manually triggered the workflows:

Dev tests: https://github.com/vercel/next.js/actions/runs/13466611352
Prod tests: https://github.com/vercel/next.js/actions/runs/13466609857

Tested by patching the `test/build-turbopack-*-tests-manifest.js`
scripts like so:

```diff
diff --git a/test/build-turbopack-dev-tests-manifest.js b/test/build-turbopack-dev-tests-manifest.js
index bc1928cf2d..e6546e0018 100644
--- a/test/build-turbopack-dev-tests-manifest.js
+++ b/test/build-turbopack-dev-tests-manifest.js
@@ -91,7 +91,11 @@ async function fetchLatestTestArtifact() {
   const res = JSON.parse(stdout)
 
   for (const artifact of res.artifacts) {
-    if (artifact.expired || artifact.workflow_run.head_branch !== 'canary') {
+    if (
+      artifact.expired ||
+      artifact.workflow_run.head_branch !==
+        'bgw/reusable-integration-test-workflow'
+    ) {
       continue
     }
```

And then diffing the generated json manifests:
https://gist.github.com/bgw/08778b5f63861811fbe88e3f466a8679

Note that some differences are expected: These jobs have been broken for
at least 2-3 weeks.
parent 319e3058
.github/actions/*/dist/** -text linguist-vendored
packages/next/bundles/** -text linguist-vendored packages/next/bundles/** -text linguist-vendored
packages/next/compiled/** -text linguist-vendored packages/next/compiled/** -text linguist-vendored
......
...@@ -4,10 +4,7 @@ interface JobResult { ...@@ -4,10 +4,7 @@ interface JobResult {
} }
interface TestResultManifest { interface TestResultManifest {
nextjsVersion: string
ref: string ref: string
buildTime?: string
buildSize?: string
result: Array<JobResult> result: Array<JobResult>
flakyMonitorJobResults: Array<JobResult> flakyMonitorJobResults: Array<JobResult>
} }
......
...@@ -226,7 +226,13 @@ jobs: ...@@ -226,7 +226,13 @@ jobs:
- run: turbo run get-test-timings -- --build ${{ github.sha }} - run: turbo run get-test-timings -- --build ${{ github.sha }}
- run: /bin/bash -c "${{ inputs.afterBuild }}" - run: ${{ inputs.afterBuild }}
# defaults.run.shell sets a stronger options (`-leo pipefail`)
# Set this back to github action's weaker defaults:
# https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsshell
#
# We must use a login shell: fnm installation may modify the `.profile`
shell: bash -le {0}
timeout-minutes: ${{ inputs.timeout_minutes }} timeout-minutes: ${{ inputs.timeout_minutes }}
- name: Upload artifact - name: Upload artifact
......
name: Integration Tests Reusable
on:
workflow_call:
inputs:
name:
description: A unique identifer used for uploaded assets
type: string
test_type:
description: '"development" or "production"'
required: true
type: string
run_before_test:
description: >
Bash code to run before executing the test (e.g. setting environment
variables). Runs in the same step as the test.
type: string
default: ''
e2e_groups:
description: >
Size of the matrix used for running e2e tests (controls parallelism)
type: number
default: 6
integration_groups:
description: >
Size of the matrix used for running legacy integration tests (controls
parallelism)
type: number
default: 6
e2e_timeout_minutes:
type: number
default: 30
integration_timeout_minutes:
type: number
default: 30
jobs:
# First, build Next.js to execute across tests.
build-next:
name: build-next
uses: ./.github/workflows/build_reusable.yml
with:
skipNativeBuild: yes
stepName: build-next
secrets: inherit
build-native:
name: build-native
uses: ./.github/workflows/build_reusable.yml
with:
skipInstallBuild: yes
stepName: build-native
secrets: inherit
generate-matrices:
runs-on: [self-hosted, linux, x64, metal]
steps:
- id: out
run: |
printf 'e2e=[%s]\n' \
"$(seq -s, 1 ${{ inputs.e2e_groups }})" | \
tee -a "$GITHUB_OUTPUT"
printf 'integration=[%s]\n' \
"$(seq -s, 1 ${{ inputs.integration_groups }})" | \
tee -a "$GITHUB_OUTPUT"
outputs:
e2e: ${{ steps.out.outputs.e2e }}
integration: ${{ steps.out.outputs.integration }}
# Actual test scheduling. These jobs mimic the normal test jobs.
# Refer build_and_test.yml for more details.
#
# We run tests in two parts. Legacy integration tests are run separately:
# https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#test-types-in-nextjs
test-e2e:
# Name must match `integrationTestJobs` in
# `./.github/actions/next-integration-stat`
name: >-
Next.js integration test (E2E and ${{ inputs.test_type }})
(${{ matrix.group }}/${{ inputs.e2e_groups }})
needs: [build-next, build-native, generate-matrices]
strategy:
fail-fast: false
matrix:
group: ${{ fromJSON(needs.generate-matrices.outputs.e2e) }}
uses: ./.github/workflows/build_reusable.yml
with:
afterBuild: |
# e2e and ${{ inputs.test_type }} tests with `node run-tests.js`
export NEXT_TEST_CONTINUE_ON_ERROR=TRUE
export NEXT_E2E_TEST_TIMEOUT=240000
export NEXT_TEST_MODE=${{
inputs.test_type == 'development' && 'dev' || 'start'
}}
${{ inputs.run_before_test }}
node run-tests.js \
-g ${{ matrix.group }}/${{ inputs.e2e_groups }} \
-c $TEST_CONCURRENCY \
--type ${{ inputs.test_type }}
stepName: test-${{ inputs.name }}-${{ matrix.group }}
timeout_minutes: ${{ inputs.e2e_timeout_minutes }}
secrets: inherit
test-integration:
# Name must match `integrationTestJobs` in
# `./.github/actions/next-integration-stat`
name: >-
Next.js integration test (Integration)
(${{ matrix.group }}/${{ inputs.e2e_groups }})
needs: [build-next, build-native, generate-matrices]
strategy:
fail-fast: false
matrix:
group: ${{ fromJSON(needs.generate-matrices.outputs.integration) }}
uses: ./.github/workflows/build_reusable.yml
with:
nodeVersion: 18.18.2
afterBuild: |
# legacy integration tests with `node run-tests.js`
export NEXT_TEST_CONTINUE_ON_ERROR=TRUE
export NEXT_E2E_TEST_TIMEOUT=240000
# HACK: Despite the name, these environment variables are just used to
# gate tests, so they're applicable to both turbopack and rspack tests
export ${{
inputs.test_type == 'development' &&
'TURBOPACK_DEV=1' ||
'TURBOPACK_BUILD=1'
}}
${{ inputs.run_before_test }}
node run-tests.js \
-g ${{ matrix.group }}/${{ inputs.integration_groups }} \
-c $TEST_CONCURRENCY \
--type integration
stepName: test-${{ inputs.name }}-integration-${{ matrix.group }}
timeout_minutes: ${{ inputs.integration_timeout_minutes }}
secrets: inherit
# Collect integration test results from execute_tests,
# Store it as github artifact for next step to consume.
collect_nextjs_development_integration_stat:
needs: [test-e2e, test-integration]
name: Next.js integration test development status report
runs-on: [self-hosted, linux, x64, metal]
if: always()
permissions:
pull-requests: write
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Collect integration test stat
uses: ./.github/actions/next-integration-stat
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Store artifacts
uses: actions/upload-artifact@v4
with:
name: test-results-${{ inputs.name }}
path: |
nextjs-test-results.json
failed-test-path-list.json
passed-test-path-list.json
# Reusable workflow to execute certain version of Next.js integration tests
# with turbopack.
#
# Refer test.yml for how this workflow is being initialized
# - Workflow can specify `inputs.version` to specify which version of next.js to use, otherwise will use latest release version.
name: Turbopack Next.js production integration tests name: Turbopack Next.js production integration tests
on: on:
schedule: schedule:
- cron: '0 6 * * *' - cron: '0 6 * * *'
workflow_dispatch: workflow_dispatch: {}
inputs:
# Allow to specify Next.js version to run integration test against.
# If not specified, will use latest release version including canary.
version:
description: Next.js version, sha, branch to test
type: string
default: 'canary'
# The base of the test results to compare against. If not specified, will try to compare with latest main branch's test results.
diff_base:
type: string
default: 'none'
# Workflow-common env variables
env:
# Enabling backtrace will makes snapshot tests fail
RUST_BACKTRACE: 0
NEXT_TELEMETRY_DISABLED: 1
TEST_CONCURRENCY: 6
NEXT_JUNIT_TEST_REPORT: 'true'
__INTERNAL_CUSTOM_TURBOPACK_BINDINGS: ${{ github.workspace }}/packages/next-swc/native/next-swc.linux-x64-gnu.node
NEXT_TEST_SKIP_RETRY_MANIFEST: ${{ github.workspace }}/integration-test-data/test-results/main/failed-test-path-list.json
NEXT_TEST_CONTINUE_ON_ERROR: TRUE
NEXT_E2E_TEST_TIMEOUT: 240000
NEXT_TEST_JOB: 1
jobs: jobs:
# First, build Next.js to execute across tests. test-dev:
setup_nextjs: name: Next.js integration tests
name: Setup Next.js build uses: ./.github/workflows/integration_tests_reusable.yml
uses: ./.github/workflows/setup-nextjs-build.yml
with: with:
nodeVersion: 18.18.2 name: turbopack-production
version: ${{ inputs.version || 'canary' }} test_type: production
run_before_test: |
# Actual test scheduling. These jobs mimic the same jobs in Next.js repo, export TURBOPACK=1 TURBOPACK_BUILD=1
# which we do allow some of duplications to make it easier to update if upstream changes. # Failing tests take longer (due to timeouts and retries). Since we have
# Refer build_and_test.yml in the Next.js repo for more details. # many failing tests, we need smaller groups and longer timeouts, in case
test-production: # a group gets stuck with a cluster of failing tests.
# This job name is being used in github action to collect test results. Do not change it, or should update e2e_groups: 12
# ./.github/actions/next-integration-stat to match the new name. integration_groups: 12
name: Next.js integration test (Production) e2e_timeout_minutes: 60
# Currently it is possible test grouping puts large number of failing tests suites in a single group, integration_timeout_minutes: 60
# which ends up job timeouts. Temporarily relieve the timeout until we make progresses on the failing suites. secrets: inherit
# ref: https://github.com/vercel/turbo/pull/5668
# timeout-minutes: 180
runs-on:
- 'self-hosted'
- 'linux'
- 'x64'
- 'metal'
needs: [setup_nextjs]
strategy:
fail-fast: false
matrix:
group: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
steps:
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 18.18.2
check-latest: true
- uses: actions/cache/restore@v3
id: restore-build
with:
path: ./*
key: ${{ inputs.version || 'canary' }}-${{ github.sha }}-${{ github.run_id }}-${{ github.run_attempt}}-${{ github.run_number }}
fail-on-cache-miss: true
- name: Enable corepack and install yarn
run: |
corepack enable
corepack prepare --activate yarn@1.22.19
- name: Setup playwright
run: |
pnpm playwright install
- name: Run test/production
run: |
NEXT_TEST_MODE=start TURBOPACK=1 TURBOPACK_BUILD=1 NEXT_E2E_TEST_TIMEOUT=240000 node run-tests.js -g ${{ matrix.group }}/12 -c ${TEST_CONCURRENCY} --type production
# It is currently expected to fail some of next.js integration test, do not fail CI check.
continue-on-error: true
- name: Upload test report artifacts
uses: actions/upload-artifact@v4
with:
name: test-reports-start-${{ matrix.group }}
if-no-files-found: 'error'
path: |
test/turbopack-test-junit-report
test-integration-production:
name: Next.js integration test (Integration)
needs: [setup_nextjs]
runs-on:
- 'self-hosted'
- 'linux'
- 'x64'
- 'metal'
timeout-minutes: 180
strategy:
fail-fast: false
matrix:
group: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
steps:
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 18.18.2
check-latest: true
- uses: actions/cache/restore@v3
id: restore-build
with:
path: ./*
key: ${{ inputs.version || 'canary' }}-${{ github.sha }}
fail-on-cache-miss: true
- name: Enable corepack and install yarn
run: |
corepack enable
corepack prepare --activate yarn@1.22.19
- name: Setup playwright
run: |
pnpm playwright install
- name: Run test/integration
run: |
TURBOPACK=1 TURBOPACK_BUILD=1 NEXT_E2E_TEST_TIMEOUT=240000 node run-tests.js -g ${{ matrix.group }}/12 -c ${TEST_CONCURRENCY} --type integration
continue-on-error: true
- name: Upload test report artifacts
uses: actions/upload-artifact@v4
with:
name: test-reports-build-integration-${{ matrix.group }}
if-no-files-found: 'error'
path: |
test/turbopack-test-junit-report
# Collect integration test results from execute_tests,
# Store it as github artifact for next step to consume.
collect_nextjs_production_integration_stat:
needs: [test-production, test-integration-production]
name: Next.js integration test production status report
runs-on:
- 'self-hosted'
- 'linux'
- 'x64'
- 'metal'
if: always()
permissions:
pull-requests: write
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Collect integration test stat
uses: ./.github/actions/next-integration-stat
with:
diff_base: ${{ inputs.diff_base || 'none' }}
- name: Store artifacts
uses: actions/upload-artifact@v4
with:
name: test-results-turbopack-production
path: |
nextjs-test-results.json
failed-test-path-list.json
passed-test-path-list.json
upload_test_report:
needs: [test-production, test-integration-production]
name: Upload test report to datadog
runs-on:
- 'self-hosted'
- 'linux'
- 'x64'
- 'metal'
if: ${{ !cancelled() }}
steps:
- name: Download test report artifacts
id: download-test-reports
uses: actions/download-artifact@v4
with:
pattern: test-reports-*
path: test/reports
merge-multiple: true
- name: Upload to datadog
env:
DATADOG_API_KEY: ${{ secrets.DATA_DOG_API_KEY }}
DD_ENV: 'ci'
run: |
# We'll tag this to the "Turbopack" datadog service, not "nextjs"
npx @datadog/datadog-ci@2.23.1 junit upload --tags test.type:turbopack-build.daily --service Turbopack-build ./test/reports
# Reusable workflow to execute certain version of Next.js integration tests
# with turbopack.
#
# Refer test.yml for how this workflow is being initialized
# - Workflow can specify `inputs.version` to specify which version of next.js to use, otherwise will use latest release version.
name: Turbopack Next.js development integration tests name: Turbopack Next.js development integration tests
on: on:
schedule: schedule:
- cron: '0 6 * * *' - cron: '0 6 * * *'
workflow_dispatch: workflow_dispatch: {}
inputs:
# Allow to specify Next.js version to run integration test against.
# If not specified, will use latest release version including canary.
version:
description: Next.js version, sha, branch to test
type: string
default: 'canary'
# The base of the test results to compare against. If not specified, will try to compare with latest main branch's test results.
diff_base:
type: string
default: 'none'
# Workflow-common env variables
env:
# Enabling backtrace will makes snapshot tests fail
RUST_BACKTRACE: 0
NEXT_TELEMETRY_DISABLED: 1
TEST_CONCURRENCY: 6
NEXT_JUNIT_TEST_REPORT: 'true'
__INTERNAL_CUSTOM_TURBOPACK_BINDINGS: ${{ github.workspace }}/packages/next-swc/native/next-swc.linux-x64-gnu.node
NEXT_TEST_SKIP_RETRY_MANIFEST: ${{ github.workspace }}/integration-test-data/test-results/main/failed-test-path-list.json
NEXT_TEST_CONTINUE_ON_ERROR: TRUE
NEXT_E2E_TEST_TIMEOUT: 240000
NEXT_TEST_JOB: 1
jobs: jobs:
# First, build next-dev and Next.js both to execute across tests.
setup_nextjs:
name: Setup Next.js build
uses: ./.github/workflows/setup-nextjs-build.yml
with:
nodeVersion: 18.18.2
version: ${{ inputs.version || 'canary' }}
# Actual test scheduling. These jobs mimic the same jobs in Next.js repo,
# which we do allow some of duplications to make it easier to update if upstream changes.
# Refer build_and_test.yml in the Next.js repo for more details.
test-dev: test-dev:
# This job name is being used in github action to collect test results. Do not change it, or should update name: Next.js integration tests
# ./.github/actions/next-integration-stat to match the new name. uses: ./.github/workflows/integration_tests_reusable.yml
name: Next.js integration test (Development) with:
# Currently it is possible test grouping puts large number of failing tests suites in a single group, name: turbopack-development
# which ends up job timeouts. Temporarily relieve the timeout until we make progresses on the failing suites. test_type: development
# ref: https://github.com/vercel/turbo/pull/5668 run_before_test: |
# timeout-minutes: 180 export TURBOPACK=1
runs-on: secrets: inherit
- 'self-hosted'
- 'linux'
- 'x64'
- 'metal'
needs: [setup_nextjs]
strategy:
fail-fast: false
matrix:
group: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
steps:
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 18.18.2
check-latest: true
- uses: actions/cache/restore@v3
id: restore-build
with:
path: ./*
key: ${{ inputs.version || 'canary' }}-${{ github.sha }}-${{ github.run_id }}-${{ github.run_attempt}}-${{ github.run_number }}
fail-on-cache-miss: true
- name: Enable corepack and install yarn
run: |
corepack enable
corepack prepare --activate yarn@1.22.19
- name: Setup playwright
run: |
pnpm playwright install
- name: Run test/development
run: |
NEXT_TEST_MODE=dev TURBOPACK=1 TURBOPACK_DEV=1 NEXT_E2E_TEST_TIMEOUT=240000 node run-tests.js -g ${{ matrix.group }}/12 -c ${TEST_CONCURRENCY} --type development
# It is currently expected to fail some of next.js integration test, do not fail CI check.
continue-on-error: true
- name: Upload test report artifacts
uses: actions/upload-artifact@v4
with:
name: test-reports-dev-${{ matrix.group }}
if-no-files-found: 'error'
path: |
test/turbopack-test-junit-report
test-integration-development:
name: Next.js integration test (Integration)
needs: [setup_nextjs]
runs-on:
- 'self-hosted'
- 'linux'
- 'x64'
- 'metal'
timeout-minutes: 180
strategy:
fail-fast: false
matrix:
group: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
steps:
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 18.18.2
check-latest: true
- uses: actions/cache/restore@v3
id: restore-build
with:
path: ./*
key: ${{ inputs.version || 'canary' }}-${{ github.sha }}
fail-on-cache-miss: true
- name: Enable corepack and install yarn
run: |
corepack enable
corepack prepare --activate yarn@1.22.19
- name: Setup playwright
run: |
pnpm playwright install
- name: Run test/integration
run: |
TURBOPACK=1 TURBOPACK_DEV=1 NEXT_E2E_TEST_TIMEOUT=240000 node run-tests.js -g ${{ matrix.group }}/12 -c ${TEST_CONCURRENCY} --type integration
continue-on-error: true
- name: Upload test report artifacts
uses: actions/upload-artifact@v4
with:
name: test-reports-dev-integration-${{ matrix.group }}
if-no-files-found: 'error'
path: |
test/turbopack-test-junit-report
# Collect integration test results from execute_tests,
# Store it as github artifact for next step to consume.
collect_nextjs_development_integration_stat:
needs: [test-dev, test-integration-development]
name: Next.js integration test development status report
runs-on:
- 'self-hosted'
- 'linux'
- 'x64'
- 'metal'
if: always()
permissions:
pull-requests: write
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Collect integration test stat
uses: ./.github/actions/next-integration-stat
with:
diff_base: ${{ inputs.diff_base || 'none' }}
- name: Store artifacts
uses: actions/upload-artifact@v4
with:
name: test-results-turbopack-development
path: |
nextjs-test-results.json
failed-test-path-list.json
passed-test-path-list.json
upload_test_report:
needs: [test-dev, test-integration-development]
name: Upload test report to datadog
runs-on:
- 'self-hosted'
- 'linux'
- 'x64'
- 'metal'
if: ${{ !cancelled() }}
steps:
- name: Download test report artifacts
id: download-test-reports
uses: actions/download-artifact@v4
with:
pattern: test-reports-*
path: test/reports
merge-multiple: true
- name: Upload to datadog
env:
DATADOG_API_KEY: ${{ secrets.DATA_DOG_API_KEY }}
DD_ENV: 'ci'
run: |
# We'll tag this to the "Turbopack" datadog service, not "nextjs"
npx @datadog/datadog-ci@2.23.1 junit upload --tags test.type:turbopack.daily --service Turbopack ./test/reports
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment