Unverified Commit 299c2f48 authored by Hendrik Liebau's avatar Hendrik Liebau Committed by GitHub

[ci] Fix test axis fixtures never running their off state in CI (#100912)

parent fcd3fdeb
...@@ -144,7 +144,7 @@ start passing. Working example: `test/e2e/app-dir/concurrent-router-queue/` ...@@ -144,7 +144,7 @@ start passing. Working example: `test/e2e/app-dir/concurrent-router-queue/`
pairs with a lazy `@force-gate` when the off state proves nothing — pairs with a lazy `@force-gate` when the off state proves nothing —
`test/e2e/app-dir/use-offline/` — which skips the redundant axis run (build `test/e2e/app-dir/use-offline/` — which skips the redundant axis run (build
included) instead of covering it. Axis `A` aliases `__NEXT_CACHE_COMPONENTS` included) instead of covering it. Axis `A` aliases `__NEXT_CACHE_COMPONENTS`
for now (see `scripts/run-jest.sh`) — fine, because these fixtures pin for now (see `test/jest-global-setup.ts`) — fine, because these fixtures pin
`cacheComponents` explicitly, so that run's env default is a no-op for them. `cacheComponents` explicitly, so that run's env default is a no-op for them.
**Keep exactly one flag varying per fixture.** A red shard must attribute to a **Keep exactly one flag varying per fixture.** A red shard must attribute to a
......
...@@ -8,6 +8,7 @@ const createJestConfig = nextJest() ...@@ -8,6 +8,7 @@ const createJestConfig = nextJest()
const customJestConfig = { const customJestConfig = {
displayName: process.env.IS_WEBPACK_TEST ? 'webpack' : 'Turbopack', displayName: process.env.IS_WEBPACK_TEST ? 'webpack' : 'Turbopack',
testMatch: ['**/*.test.js', '**/*.test.ts', '**/*.test.jsx', '**/*.test.tsx'], testMatch: ['**/*.test.js', '**/*.test.ts', '**/*.test.jsx', '**/*.test.tsx'],
globalSetup: '<rootDir>/jest-global-setup.ts',
setupFilesAfterEnv: ['<rootDir>/jest-setup-after-env.ts'], setupFilesAfterEnv: ['<rootDir>/jest-setup-after-env.ts'],
verbose: true, verbose: true,
rootDir: 'test', rootDir: 'test',
......
...@@ -56,35 +56,6 @@ while [ $# -gt 0 ]; do ...@@ -56,35 +56,6 @@ while [ $# -gt 0 ]; do
shift shift
done done
# `__NEXT_TEST_AXIS` names the alternate flag configurations of the test
# matrix. Axes are lettered (`A`, `B`, …) — a fixed enumeration a fixture opts
# into, not a boolean, and not one of the buckets test *sharding* splits a run
# into. CI runs the suites once plainly and once per axis, and a fixture keys
# an experimental flag on an axis to cover both states of the flag — enabled
# by default, disabled on that axis:
#
# experimental: {
# concurrentRouterQueue: process.env.__NEXT_TEST_AXIS !== 'A',
# }
#
# paired with a `// @gate concurrentRouterQueue` on the affected tests: a
# plain run — including a local run with no special env — exercises the
# feature, and the axis run covers the off state (see
# test/lib/gate/README.md).
#
# For now there is a single axis, `A`, and it is an alias for
# `__NEXT_CACHE_COMPONENTS` (the `--experimental` run) rather than a CI
# dimension of its own. That works because most experiments hard-code
# `cacheComponents: true` in their fixture anyway — the cache-components env
# default only applies to fixtures that don't set it themselves, so for these
# fixtures that run is free to double as the axis run. Setting either name
# implies the other.
if [ -n "${__NEXT_TEST_AXIS:-}" ]; then
export __NEXT_CACHE_COMPONENTS=true
elif [ "${__NEXT_CACHE_COMPONENTS:-}" = "true" ]; then
export __NEXT_TEST_AXIS=A
fi
# Resolves to `node_modules/.bin/jest` via `$PATH`. This relies on being # Resolves to `node_modules/.bin/jest` via `$PATH`. This relies on being
# invoked through pnpm (or another package runner), which prepends the # invoked through pnpm (or another package runner), which prepends the
# workspace's `node_modules/.bin/` to `$PATH` before running the script. # workspace's `node_modules/.bin/` to `$PATH` before running the script.
......
export default function globalSetup() {
// `__NEXT_TEST_AXIS` names an alternate flag configuration of the test
// matrix. Axes are letters (`A`, `B`, …). A fixture can key an experimental
// flag on an axis to cover both states of the flag:
//
// ```js
// experimental: {
// concurrentRouterQueue: process.env.__NEXT_TEST_AXIS !== 'A',
// }
// ```
//
// A plain run enables the flag, and the axis run disables it. See
// `test/lib/gate/README.md`.
//
// The only axis today is `A`. It is an alias for the Cache Components run
// (`__NEXT_CACHE_COMPONENTS=true`), so CI has no separate job for it. Most
// fixtures that use the axis set `cacheComponents: true` themselves, so the
// env default of that run does not change them. Setting either variable sets
// the other.
if (process.env.__NEXT_TEST_AXIS) {
process.env.__NEXT_CACHE_COMPONENTS = 'true'
} else if (process.env.__NEXT_CACHE_COMPONENTS === 'true') {
process.env.__NEXT_TEST_AXIS = 'A'
}
}
...@@ -2,11 +2,16 @@ const deploymentTestFlags = [ ...@@ -2,11 +2,16 @@ const deploymentTestFlags = [
'__NEXT_CACHE_COMPONENTS', '__NEXT_CACHE_COMPONENTS',
'__NEXT_PARTIAL_PREFETCHING', '__NEXT_PARTIAL_PREFETCHING',
'__NEXT_EXPERIMENTAL_CACHED_NAVIGATIONS', '__NEXT_EXPERIMENTAL_CACHED_NAVIGATIONS',
'__NEXT_TEST_AXIS',
] ]
/** /**
* Capture test flags in the deployed config because Vercel environment variable * Capture test flags in the deployed config because Vercel environment variable
* names cannot start with an underscore. This runs in the local test harness. * names cannot start with an underscore. This runs in the local test harness.
*
* Callers insert the assignments before the config content. A fixture can read
* a flag while its config module loads, for example
* `process.env.__NEXT_TEST_AXIS !== 'A'`.
*/ */
export function getDeploymentTestEnvAssignments(): string { export function getDeploymentTestEnvAssignments(): string {
let assignments = '' let assignments = ''
......
...@@ -235,10 +235,10 @@ export class PatchedFileRef { ...@@ -235,10 +235,10 @@ export class PatchedFileRef {
} }
/** /**
* Return a PatchedFileRef that appends captured test env assignments when the * Return a PatchedFileRef that prepends captured test env assignments when the
* harness copies the file in deploy mode. Other modes copy the content unchanged. * harness copies the file in deploy mode. Other modes copy the content
* Nested app configs need this because the deploy harness only patches the * unchanged. Nested app configs need this because the deploy harness only
* fixture-root config. * patches the fixture-root config.
* *
* TODO: Patch nested configs automatically in the deploy harness so fixtures no * TODO: Patch nested configs automatically in the deploy harness so fixtures no
* longer need these explicit overrides. * longer need these explicit overrides.
...@@ -248,7 +248,7 @@ export function patchFileWithDeployEnvAssignments( ...@@ -248,7 +248,7 @@ export function patchFileWithDeployEnvAssignments(
path: string path: string
): PatchedFileRef { ): PatchedFileRef {
return new PatchedFileRef(path, (content) => return new PatchedFileRef(path, (content) =>
isNextDeploy ? `${content}\n` + getDeploymentTestEnvAssignments() : content isNextDeploy ? getDeploymentTestEnvAssignments() + content : content
) )
} }
......
...@@ -119,8 +119,8 @@ On top of the dimensions the test matrix already has (mode, bundler, React ...@@ -119,8 +119,8 @@ On top of the dimensions the test matrix already has (mode, bundler, React
version), the suites run once plainly and once per _test axis_ — a fixed, version), the suites run once plainly and once per _test axis_ — a fixed,
lettered set of alternate flag configurations marked by `__NEXT_TEST_AXIS`. lettered set of alternate flag configurations marked by `__NEXT_TEST_AXIS`.
(Today there is one axis, `A`, an alias for the `--experimental` / (Today there is one axis, `A`, an alias for the `--experimental` /
`__NEXT_CACHE_COMPONENTS` run; see `scripts/run-jest.sh`.) A fixture can key `__NEXT_CACHE_COMPONENTS` run; see `test/jest-global-setup.ts`.) A fixture can
an experimental flag on an axis instead of pinning it — enabled by default, key an experimental flag on an axis instead of pinning it — enabled by default,
disabled on that axis: disabled on that axis:
```js ```js
...@@ -149,8 +149,8 @@ dimension. The gate itself keeps working either way — lazy conditions read the ...@@ -149,8 +149,8 @@ dimension. The gate itself keeps working either way — lazy conditions read the
_resolved_ config, so they observe whatever the fixture decided, not how it _resolved_ config, so they observe whatever the fixture decided, not how it
decided it. decided it.
To reproduce the disabled (axis) state locally, set the marker the same way To reproduce the disabled (axis) state locally, set the marker. CI sets
CI does: `__NEXT_CACHE_COMPONENTS=true` instead, which has the same effect:
```sh ```sh
__NEXT_TEST_AXIS=A NEXT_SKIP_ISOLATE=1 pnpm test-start-webpack test/e2e/app-dir/concurrent-router-queue/concurrent-router-queue.test.ts __NEXT_TEST_AXIS=A NEXT_SKIP_ISOLATE=1 pnpm test-start-webpack test/e2e/app-dir/concurrent-router-queue/concurrent-router-queue.test.ts
......
...@@ -551,8 +551,8 @@ export class NextInstance { ...@@ -551,8 +551,8 @@ export class NextInstance {
await fs.writeFile( await fs.writeFile(
fileName, fileName,
`${content}\n` + getDeploymentTestEnvAssignments() +
getDeploymentTestEnvAssignments() + `${content}\n` +
` `
// alias __NEXT_TEST_MODE for next-deploy as "_" is not a valid // alias __NEXT_TEST_MODE for next-deploy as "_" is not a valid
// env variable during deploy // env variable during deploy
......
...@@ -5,6 +5,7 @@ describe.each([ ...@@ -5,6 +5,7 @@ describe.each([
'__NEXT_CACHE_COMPONENTS', '__NEXT_CACHE_COMPONENTS',
'__NEXT_PARTIAL_PREFETCHING', '__NEXT_PARTIAL_PREFETCHING',
'__NEXT_EXPERIMENTAL_CACHED_NAVIGATIONS', '__NEXT_EXPERIMENTAL_CACHED_NAVIGATIONS',
'__NEXT_TEST_AXIS',
])('deployment capture of %s', (flag) => { ])('deployment capture of %s', (flag) => {
beforeEach(() => { beforeEach(() => {
jest.replaceProperty(process, 'env', { jest.replaceProperty(process, 'env', {
...@@ -12,6 +13,7 @@ describe.each([ ...@@ -12,6 +13,7 @@ describe.each([
__NEXT_CACHE_COMPONENTS: '', __NEXT_CACHE_COMPONENTS: '',
__NEXT_PARTIAL_PREFETCHING: '', __NEXT_PARTIAL_PREFETCHING: '',
__NEXT_EXPERIMENTAL_CACHED_NAVIGATIONS: '', __NEXT_EXPERIMENTAL_CACHED_NAVIGATIONS: '',
__NEXT_TEST_AXIS: '',
}) })
}) })
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment