Unverified Commit 4cb54721 authored by Josh Story's avatar Josh Story Committed by GitHub

test: support deploy exclusions with force gates (#98149)

## Summary

- add static gate conditions for the deployment CI variants used by
existing test suites
- support `@force-gate` on parameterized `describe.each` suites
- document and test the force-gate infrastructure needed by the
migration

This establishes the gate support used by the deployment-exclusion
migration PRs above it.

## Verification

- `pnpm typescript`
- `pnpm jest test/unit/gate/pragma-transform.test.ts
test/unit/gate/runtime.test.ts --runInBand`

<!-- NEXT_JS_LLM -->
parent 2b11d01a
...@@ -92,9 +92,11 @@ Every name in a pragma must be declared in `test/lib/gate/conditions.ts` ...@@ -92,9 +92,11 @@ Every name in a pragma must be declared in `test/lib/gate/conditions.ts`
(typos fail the suite at collection). Two tiers: (typos fail the suite at collection). Two tiers:
- **static** — the run's shape: `dev`, `start`, `deploy`, `mode`, `turbopack`, - **static** — the run's shape: `dev`, `start`, `deploy`, `mode`, `turbopack`,
`rspack`, `webpack`, `bundler`, `react18`, `wasm`, `ci`, plus the `rspack`, `webpack`, `bundler`, `react18`, `wasm`, `ci`; specialized CI
always-false `FIXME`/`TODO`. `prod` and `prefetching` are semantic aliases variants `adapter`, `standaloneOutput`, `turbopackDev`, and `turbopackBuild`;
for `!dev` — prefer the name that states _why_ the suite cannot run. plus the always-false `FIXME`/`TODO`.
`prod` and `prefetching` are semantic aliases for `!dev` — prefer the name
that states _why_ the suite cannot run.
- **lazy** — a predicate over the fixture's _resolved_ `next.config` - **lazy** — a predicate over the fixture's _resolved_ `next.config`
(`cacheComponents`, `ppr`, `useOffline`, `output`, …). (`cacheComponents`, `ppr`, `useOffline`, `output`, …).
......
...@@ -85,7 +85,9 @@ There are two tiers: ...@@ -85,7 +85,9 @@ There are two tiers:
- **static** — the run's own shape (`dev`, `start`, `deploy`, `mode`, - **static** — the run's own shape (`dev`, `start`, `deploy`, `mode`,
`turbopack`, `rspack`, `webpack`, `bundler`, `react18`, `wasm`, `ci`), `turbopack`, `rspack`, `webpack`, `bundler`, `react18`, `wasm`, `ci`),
semantic aliases for `!dev` that state the reason rather than the mode semantic aliases for `!dev` that state the reason rather than the mode
(`prod`, `prefetching`), plus `FIXME` / `TODO`, which are always false. (`prod`, `prefetching`), specialized CI variants (`adapter`,
`standaloneOutput`, `turbopackDev`, `turbopackBuild`), plus `FIXME` / `TODO`,
which are always false.
- **lazy** — a predicate over the fixture's *resolved* `next.config` - **lazy** — a predicate over the fixture's *resolved* `next.config`
(`cacheComponents`, `ppr`, `prefetchInlining`, `output`, …), read the first (`cacheComponents`, `ppr`, `prefetchInlining`, `output`, …), read the first
time a gate asks for it. time a gate asks for it.
...@@ -190,9 +192,11 @@ var is not what the fixture actually resolved. ...@@ -190,9 +192,11 @@ var is not what the fixture actually resolved.
## How it works ## How it works
1. `pragma-transform.js` rewrites the pragma into 1. `pragma-transform.js` rewrites the pragma into
`_test_gate([{force,source}], 'it')(...)`. It is a line-oriented regex, not an `_test_gate([{force,source}], 'it')(...)`. `describe.each` uses the analogous
AST transform, so **only the `it(` line changes** and every other line keeps `_test_gate_describe_each([{force,source}], table)(...)` helper. It is a
its byte offsets — `toMatchInlineSnapshot()` is written back by line/column. line-oriented regex, not an AST transform, so **only the call line changes**
and every other line keeps its byte offsets — `toMatchInlineSnapshot()` is
written back by line/column.
2. `jest-transformer.js` chains that rewrite in front of the SWC transformer 2. `jest-transformer.js` chains that rewrite in front of the SWC transformer
`next/jest` configures. `jest.config.js` wires it up with `next/jest` configures. `jest.config.js` wires it up with
`withGateTransformer()`. `withGateTransformer()`.
......
...@@ -134,6 +134,21 @@ export const conditions: Record<string, Condition> = { ...@@ -134,6 +134,21 @@ export const conditions: Record<string, Condition> = {
ci: staticCondition('running in CI (`NEXT_TEST_CI`)', () => ci: staticCondition('running in CI (`NEXT_TEST_CI`)', () =>
Boolean(process.env.NEXT_TEST_CI) Boolean(process.env.NEXT_TEST_CI)
), ),
adapter: staticCondition('running the adapter test variant', () =>
Boolean(process.env.NEXT_ENABLE_ADAPTER === '1')
),
standaloneOutput: staticCondition(
'running the standalone-output test variant (`TEST_OUTPUT_STANDALONE`)',
() => process.env.TEST_OUTPUT_STANDALONE === 'true'
),
turbopackDev: staticCondition(
'the test run uses the Turbopack development build (`TURBOPACK_DEV`)',
() => Boolean(process.env.TURBOPACK_DEV)
),
turbopackBuild: staticCondition(
'the test run uses the Turbopack production build (`TURBOPACK_BUILD`)',
() => Boolean(process.env.TURBOPACK_BUILD)
),
// Always false, so `// @gate FIXME` marks a test as a known failure without // Always false, so `// @gate FIXME` marks a test as a known failure without
// inventing a condition for it. Mirrors the same convention in React's // inventing a condition for it. Mirrors the same convention in React's
......
...@@ -28,13 +28,14 @@ ...@@ -28,13 +28,14 @@
/** /**
* A pragma block is one or more consecutive `// @gate` lines *immediately* * A pragma block is one or more consecutive `// @gate` lines *immediately*
* followed by an `it` / `test` / `fit` / `describe` call. Anything else (a * followed by an `it` / `test` / `fit` / `describe` call. `describe.each` has
* pragma in a JSDoc block, a pragma with a blank line under it, a pragma over * a dedicated runtime helper because its table is bound before the suite name.
* `it.each`) is not matched, and is reported as an error by `rewrite()` rather * Anything else (a pragma in a JSDoc block, a pragma with a blank line under
* than silently ignored. * it, a pragma over `it.each`) is not matched, and is reported as an error by
* `rewrite()` rather than silently ignored.
*/ */
const PRAGMA_BLOCK = const PRAGMA_BLOCK =
/((?:^[ \t]*\/\/[ \t]*@(?:force-)?gate\b[^\n]*\n)+)([ \t]*)(it|test|fit|describe)((?:\.only)?)([ \t]*\()/gm /((?:^[ \t]*\/\/[ \t]*@(?:force-)?gate\b[^\n]*\n)+)([ \t]*)(?:(it|test|fit|describe)((?:\.only)?)([ \t]*\()|(describe)\.each[ \t]*\()/gm
/** Matches a single pragma line and captures `force` + the condition source. */ /** Matches a single pragma line and captures `force` + the condition source. */
const PRAGMA_LINE = /^[ \t]*\/\/[ \t]*@(force-)?gate\b[ \t]*([^\n]*)$/ const PRAGMA_LINE = /^[ \t]*\/\/[ \t]*@(force-)?gate\b[ \t]*([^\n]*)$/
...@@ -108,9 +109,19 @@ function rewrite(src, filename) { ...@@ -108,9 +109,19 @@ function rewrite(src, filename) {
* @param {string} callee * @param {string} callee
* @param {string} only * @param {string} only
* @param {string} openParen * @param {string} openParen
* @param {string | undefined} describeEach
* @param {number} offset * @param {number} offset
*/ */
(_all, pragmaLines, indent, callee, only, openParen, offset) => { (
_all,
pragmaLines,
indent,
callee,
only,
openParen,
describeEach,
offset
) => {
const firstLine = getLineNumber(lineStarts, offset) const firstLine = getLineNumber(lineStarts, offset)
const gates = [] const gates = []
const lines = pragmaLines.split('\n') const lines = pragmaLines.split('\n')
...@@ -133,14 +144,17 @@ function rewrite(src, filename) { ...@@ -133,14 +144,17 @@ function rewrite(src, filename) {
gates.push({ force: Boolean(match[1]), source }) gates.push({ force: Boolean(match[1]), source })
} }
return ( if (describeEach) {
pragmaLines + return (
indent + pragmaLines +
`_test_gate(${JSON.stringify(gates)},${JSON.stringify( indent +
callee + only `_test_gate_describe_each(${JSON.stringify(gates)},`
)})` + )
openParen }
)
return `${pragmaLines}${indent}_test_gate(${JSON.stringify(
gates
)},${JSON.stringify(callee + only)})${openParen}`
} }
) )
...@@ -181,9 +195,10 @@ function assertNoInertPragmas(src, consumed, filename) { ...@@ -181,9 +195,10 @@ function assertNoInertPragmas(src, consumed, filename) {
`effect.\n\n` + `effect.\n\n` +
` ${lines[i].trim()}\n\n` + ` ${lines[i].trim()}\n\n` +
`A pragma must sit on the line(s) immediately above an ` + `A pragma must sit on the line(s) immediately above an ` +
`\`it(\`, \`test(\`, \`fit(\`, \`describe(\`, \`it.only(\`, ` + `\`it(\`, \`test(\`, \`fit(\`, \`describe(\`, \`describe.each(\`, ` +
`\`test.only(\` or \`describe.only(\` call — with no blank line in ` + `\`it.only(\`, \`test.only(\` or \`describe.only(\` call — with no ` +
`between. \`it.each\` and \`it.failing\` are not supported. If this ` + `blank line in between. \`it.each\` and \`it.failing\` are not ` +
`supported. If this ` +
`comment is prose rather than a pragma, reword it so it does not ` + `comment is prose rather than a pragma, reword it so it does not ` +
`start with \`@gate\`.` `start with \`@gate\`.`
) )
......
...@@ -431,6 +431,35 @@ export function _test_gate(pragmas: GatePragma[], kind: string) { ...@@ -431,6 +431,35 @@ export function _test_gate(pragmas: GatePragma[], kind: string) {
// Parsing and validation happen while the test file is being collected, so a // Parsing and validation happen while the test file is being collected, so a
// typo'd condition fails the whole suite instead of one test. // typo'd condition fails the whole suite instead of one test.
const allGates = pragmas.map(parseGate) const allGates = pragmas.map(parseGate)
return createGatedTest(
allGates,
kind,
() => resolveTestFn(kind),
() => resolveSkipFn(kind)
)
}
/** `describe.each(table)` binds the table before receiving the suite call. */
export function _test_gate_describe_each(
pragmas: GatePragma[],
table: readonly unknown[]
) {
const g = global as any
const allGates = pragmas.map(parseGate)
return createGatedTest(
allGates,
'describe.each',
() => g.describe.each(table),
() => g.describe.skip.each(table)
)
}
function createGatedTest(
allGates: Gate[],
kind: string,
getTestFn: () => TestFn,
getSkipFn: () => TestFn
) {
// A static `@force-gate` is decided at collection time (a real Jest skip). // A static `@force-gate` is decided at collection time (a real Jest skip).
// Everything else — `@gate`, and *lazy* `@force-gate` — is resolved at // Everything else — `@gate`, and *lazy* `@force-gate` — is resolved at
// runtime, so it inherits down into the tests via the describe stack. // runtime, so it inherits down into the tests via the describe stack.
...@@ -441,7 +470,6 @@ export function _test_gate(pragmas: GatePragma[], kind: string) { ...@@ -441,7 +470,6 @@ export function _test_gate(pragmas: GatePragma[], kind: string) {
(gate) => !gate.force || gate.needsResolvedConfig (gate) => !gate.force || gate.needsResolvedConfig
) )
const isDescribe = kind.startsWith('describe') const isDescribe = kind.startsWith('describe')
const testFn = resolveTestFn(kind)
return function gated(name: string, callback: Function, timeout?: number) { return function gated(name: string, callback: Function, timeout?: number) {
// A false static `@force-gate` is a real Jest skip, decided right here. // A false static `@force-gate` is a real Jest skip, decided right here.
...@@ -449,21 +477,18 @@ export function _test_gate(pragmas: GatePragma[], kind: string) { ...@@ -449,21 +477,18 @@ export function _test_gate(pragmas: GatePragma[], kind: string) {
(gate) => !evaluate(gate.node, (condition) => readCondition(condition)) (gate) => !evaluate(gate.node, (condition) => readCondition(condition))
) )
if (forcedOff) { if (forcedOff) {
return resolveSkipFn(kind)( return getSkipFn()(name, callback as jest.ProvidesCallback, timeout)
name,
callback as jest.ProvidesCallback,
timeout
)
} }
const testFn = getTestFn()
if (isDescribe) { if (isDescribe) {
// Register the `describe` normally, but make its runtime gates (including // Register the `describe` normally, but make its runtime gates (including
// a lazy `@force-gate`) visible while its body is collected so nested // a lazy `@force-gate`) visible while its body is collected so nested
// tests inherit them and `nextTestSetup` can gate the build. // tests inherit them and `nextTestSetup` can gate the build.
return testFn(name, function (this: unknown) { return testFn(name, function (this: unknown, ...args: unknown[]) {
describeGateStack.push(...runtimeGates) describeGateStack.push(...runtimeGates)
try { try {
return callback.call(this) return callback.apply(this, args)
} finally { } finally {
describeGateStack.length -= runtimeGates.length describeGateStack.length -= runtimeGates.length
} }
...@@ -614,6 +639,7 @@ function wrapHookGlobals(): void { ...@@ -614,6 +639,7 @@ function wrapHookGlobals(): void {
/** Called from `test/jest-setup-after-env.ts`. */ /** Called from `test/jest-setup-after-env.ts`. */
export function installGate(): void { export function installGate(): void {
;(global as any)._test_gate = _test_gate ;(global as any)._test_gate = _test_gate
;(global as any)._test_gate_describe_each = _test_gate_describe_each
wrapTestGlobals() wrapTestGlobals()
wrapHookGlobals() wrapHookGlobals()
} }
......
...@@ -89,6 +89,27 @@ describe('@gate pragma transform', () => { ...@@ -89,6 +89,27 @@ describe('@gate pragma transform', () => {
) )
}) })
it('supports describe.each without changing the table or line count', () => {
const input = src(
'// @force-gate !deploy',
'describe.each([',
" { name: 'one' },",
" { name: 'two' },",
'])("suite: $name", ({ name }) => {})'
)
const out = rewrite(input, 'x.ts')
expect(out.split('\n')).toHaveLength(input.split('\n').length)
expect(out).toBe(
src(
'// @force-gate !deploy',
`_test_gate_describe_each([{"force":true,"source":"!deploy"}],[`,
" { name: 'one' },",
" { name: 'two' },",
'])("suite: $name", ({ name }) => {})'
)
)
})
it('keeps the condition source verbatim, including quotes', () => { it('keeps the condition source verbatim, including quotes', () => {
const out = rewrite( const out = rewrite(
src("// @gate mode === 'start' && !turbopack", "it('a', () => {})"), src("// @gate mode === 'start' && !turbopack", "it('a', () => {})"),
......
// Imported through the same specifier tests use, so this also covers the // Imported through the same specifier tests use, so this also covers the
// re-export from next-test-utils. // re-export from next-test-utils.
import { gate } from 'next-test-utils' import { gate } from 'next-test-utils'
import { __testing, _test_gate, expectTestToFail } from '../../lib/gate/runtime' import {
__testing,
_test_gate,
_test_gate_describe_each,
expectTestToFail,
} from '../../lib/gate/runtime'
import { import {
clearGateTestContext, clearGateTestContext,
setGateTestContext, setGateTestContext,
...@@ -32,17 +37,21 @@ const runGated = (sources: string[], body: () => unknown) => { ...@@ -32,17 +37,21 @@ const runGated = (sources: string[], body: () => unknown) => {
} }
type FakeTestFn = jest.Mock & { type FakeTestFn = jest.Mock & {
skip: jest.Mock skip: jest.Mock & { each: jest.Mock }
only: jest.Mock & { failing: jest.Mock } only: jest.Mock & { failing: jest.Mock }
failing: jest.Mock failing: jest.Mock
each: jest.Mock
} }
const makeFakeTestFn = (): FakeTestFn => const makeFakeTestFn = (): FakeTestFn => {
Object.assign(jest.fn(), { const skip = Object.assign(jest.fn(), { each: jest.fn(() => jest.fn()) })
skip: jest.fn(), return Object.assign(jest.fn(), {
skip,
only: Object.assign(jest.fn(), { failing: jest.fn() }), only: Object.assign(jest.fn(), { failing: jest.fn() }),
failing: jest.fn(), failing: jest.fn(),
each: jest.fn(() => jest.fn()),
}) as FakeTestFn }) as FakeTestFn
}
/** /**
* Swaps `global.it` / `global.test` / `global.describe` for spies while `fn` * Swaps `global.it` / `global.test` / `global.describe` for spies while `fn`
...@@ -292,6 +301,45 @@ describe('@gate runtime', () => { ...@@ -292,6 +301,45 @@ describe('@gate runtime', () => {
expect(fakes.describe).not.toHaveBeenCalled() expect(fakes.describe).not.toHaveBeenCalled()
}) })
it('skips every row of describe.each with describe.skip.each', () => {
const table = [{ name: 'one' }, { name: 'two' }]
const fakes = withFakeTestGlobals(() => {
_test_gate_describe_each([{ force: true, source: 'dev' }], table)(
'suite: $name',
() => {}
)
})
expect(fakes.describe.skip.each).toHaveBeenCalledWith(table)
expect(fakes.describe.each).not.toHaveBeenCalled()
})
it('registers describe.each normally when a force-gate holds', () => {
const table = [{ name: 'one' }]
const fakes = withFakeTestGlobals(() => {
_test_gate_describe_each([{ force: true, source: '!dev' }], table)(
'suite: $name',
() => {}
)
})
expect(fakes.describe.each).toHaveBeenCalledWith(table)
expect(fakes.describe.skip.each).not.toHaveBeenCalled()
})
it('forwards describe.each row arguments to the suite callback', () => {
const table = [{ name: 'one' }]
const callback = jest.fn()
const fakes = withFakeTestGlobals(() => {
_test_gate_describe_each([{ force: true, source: '!dev' }], table)(
'suite: $name',
callback
)
})
const boundDescribe = fakes.describe.each.mock.results[0].value
const registeredCallback = boundDescribe.mock.calls[0][1]
registeredCallback(table[0])
expect(callback).toHaveBeenCalledWith(table[0])
})
it('leaves a `@gate` on the same test in charge when it holds', async () => { it('leaves a `@gate` on the same test in charge when it holds', async () => {
const fakes = withFakeTestGlobals(() => { const fakes = withFakeTestGlobals(() => {
_test_gate( _test_gate(
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment