Unverified Commit c6345b09 authored by Hendrik Liebau's avatar Hendrik Liebau Committed by GitHub

[test] Expose first-run e2e test failures in CI (#99528)

PR #89929 added automatic retries of individual e2e tests to reduce CI
time. These retries reuse the fixture without rerunning `beforeAll`. A
failed attempt can write an ISR cache or change other state. The retry
can then pass against a different starting state, potentially hiding a
real issue.

Each CI test job runs a shard containing multiple test files.
Previously, rerunning a failed job also reran the files that had already
passed. PR #92832 caches passing results so a job rerun can skip those
files and rerun only the failed files. This reduces both retry cost and
time, allowing us to remove the individual-test retries.

This change preserves the timeout improvements from #89929. It also
removes the redundant retry opt-out and updates stale comments and
test-gate documentation.

The retry removal also exposed test assumptions that required these
changes:

- Both concurrent root-param cache tests now exclude deploy mode.
Cross-request deduplication requires requests to share a server process,
but deployed requests can reach different function instances. The tests
remain enabled in dev and `next start`.
- Both affected App Shell tests now refresh and repeat only the
preparatory prefetch until root-specific shell segments arrive. Deploy
can initially serve generic fallback segments while ISR generates a
shell for unprerendered root params; `next start` generates that shell
blockingly. The cache-reuse and navigation assertions remain outside the
retry. The static-shell test also loads a fresh home page to prevent
BFCache from restoring visible accordions, then primes both language
variants. The TODOs leave the direction for consolidating the serving
behavior open.

The [deploy run and
rerun](https://github.com/vercel/next.js/actions/runs/36878368304/attempts/2)
exposed the failures addressed by these test corrections.
parent 6c2ab2c2
...@@ -165,9 +165,6 @@ single dimension. ...@@ -165,9 +165,6 @@ single dimension.
- `afterEach` failures (e.g. redbox matchers) are not gated, only the body is. - `afterEach` failures (e.g. redbox matchers) are not gated, only the body is.
Hooks under a false lazy `@force-gate` are the exception: they are skipped Hooks under a false lazy `@force-gate` are the exception: they are skipped
with the suite instead of running against a fixture that was never booted. with the suite instead of running against a fixture that was never booted.
- `jest.retryTimes(1)` on non-dev CI means a _flaky_ gated-false test passes
whenever it happens to fail; the tripwire is only deterministic for
deterministic tests.
- Gated titles are unchanged in the Jest output; the - Gated titles are unchanged in the Jest output; the
`⚠ gated test failed as expected` log line is the only signal. `⚠ gated test failed as expected` log line is the only signal.
- `pragma-transform.js` bails out early on files containing neither `@gate` - `pragma-transform.js` bails out early on files containing neither `@gate`
......
...@@ -395,10 +395,10 @@ describe('app-root-param-getters - cache dedup with root params', () => { ...@@ -395,10 +395,10 @@ describe('app-root-param-getters - cache dedup with root params', () => {
expect(french('#second').text()).toBe('fr') expect(french('#second').text()).toBe('fr')
}) })
// Cross-request deduplication requires a shared server process. Deployed
// requests can reach different function instances.
// @force-gate !deploy
it('should dedupe same root params and isolate different root params', async () => { it('should dedupe same root params and isolate different root params', async () => {
// In deploy mode, this can flake if Fluid routes concurrent requests
// to different function instances: each instance may independently
// compute a value before the cache is populated.
// Three concurrent requests: ca/en, ca/fr, ca/fr. // Three concurrent requests: ca/en, ca/fr, ca/fr.
const [$en, $fr1, $fr2] = await Promise.all([ const [$en, $fr1, $fr2] = await Promise.all([
next.render$('/ca/en'), next.render$('/ca/en'),
...@@ -420,6 +420,9 @@ describe('app-root-param-getters - cache dedup with root params', () => { ...@@ -420,6 +420,9 @@ describe('app-root-param-getters - cache dedup with root params', () => {
expect(randomFr1).toBe(randomFr2) expect(randomFr1).toBe(randomFr2)
}) })
// These cross-request cache checks require a shared server process. Deployed
// requests can reach different function instances.
// @force-gate !deploy
it('should dedupe same root params and isolate different root params for private caches', async () => { it('should dedupe same root params and isolate different root params for private caches', async () => {
// Three concurrent requests: ca/en, ca/fr, ca/fr. // Three concurrent requests: ca/en, ca/fr, ca/fr.
const [$en, $fr1, $fr2] = await Promise.all([ const [$en, $fr1, $fr2] = await Promise.all([
......
import { nextTestSetup } from 'e2e-utils' import { nextTestSetup } from 'e2e-utils'
import type * as Playwright from 'playwright' import type * as Playwright from 'playwright'
import { createRouterAct } from 'router-act' import { createRouterAct } from 'router-act'
import { retry } from 'next-test-utils'
// The App Shell is keyed by root params so that a locale-aware (etc.) shell can // The App Shell is keyed by root params so that a locale-aware (etc.) shell can
// still be reused across navigations. But a shell should only vary on the root // still be reused across navigations. But a shell should only vary on the root
...@@ -18,7 +19,7 @@ import { createRouterAct } from 'router-act' ...@@ -18,7 +19,7 @@ import { createRouterAct } from 'router-act'
// asserts on App Shell requests (next-router-prefetch: '3') instead of ignoring // asserts on App Shell requests (next-router-prefetch: '3') instead of ignoring
// them. // them.
describe('App Shell varies only on the root params it reads', () => { describe('App Shell varies only on the root params it reads', () => {
const { next, isNextDev } = nextTestSetup({ const { next, isNextDev, isNextDeploy } = nextTestSetup({
files: __dirname, files: __dirname,
}) })
...@@ -39,14 +40,37 @@ describe('App Shell varies only on the root params it reads', () => { ...@@ -39,14 +40,37 @@ describe('App Shell varies only on the root params it reads', () => {
// Prime the shared shell for /en/uk/posts. The shell render reads `lang` // Prime the shared shell for /en/uk/posts. The shell render reads `lang`
// ('en') but not `region`. // ('en') but not `region`.
await act( const prefetchEnglishShell = () =>
async () => { act(
await browser async () => {
.elementByCss('input[data-link-accordion="/en/uk/posts/1"]') await browser
.click() .elementByCss('input[data-link-accordion="/en/uk/posts/1"]')
}, .click()
{ includes: 'App shell for posts' } },
) { includes: 'Shell lang: en' }
)
if (isNextDeploy) {
// The test refreshes before each prefetch to clear the client's cached
// fallback. Deploy can serve generic segments while ISR generates a shell
// for unprerendered root params. Next start generates that shell
// blockingly.
//
// TODO: Consolidate on-demand shell generation between next start and
// deploy. Decide whether both modes should block on the root-specific
// shell or initially serve fallback segments.
await retry(
async () => {
await browser.refresh()
await prefetchEnglishShell()
},
10_000,
1_000,
'wait for the English root-specific shell'
)
} else {
await prefetchEnglishShell()
}
// Reveal /en/gb/posts/1 — SAME lang, DIFFERENT region. The shell does not // Reveal /en/gb/posts/1 — SAME lang, DIFFERENT region. The shell does not
// read `region`, so the shell cached for /en/uk is reusable: this reveal // read `region`, so the shell cached for /en/uk is reusable: this reveal
......
import { nextTestSetup } from 'e2e-utils' import { nextTestSetup } from 'e2e-utils'
import type * as Playwright from 'playwright' import type * as Playwright from 'playwright'
import { createRouterAct } from 'router-act' import { createRouterAct } from 'router-act'
import { retry } from 'next-test-utils'
// This suite asserts directly on App Shell prefetch responses, so every // This suite asserts directly on App Shell prefetch responses, so every
// `createRouterAct` call passes `{ includeAppShellRequests: true }`. By default // `createRouterAct` call passes `{ includeAppShellRequests: true }`. By default
...@@ -8,7 +9,7 @@ import { createRouterAct } from 'router-act' ...@@ -8,7 +9,7 @@ import { createRouterAct } from 'router-act'
// for assertion purposes, since the App Shell is conceptually part of the route // for assertion purposes, since the App Shell is conceptually part of the route
// rather than prefetch data. These tests are the exception that opts back in. // rather than prefetch data. These tests are the exception that opts back in.
describe('App Shell prefetching', () => { describe('App Shell prefetching', () => {
const { next, isNextDev } = nextTestSetup({ const { next, isNextDev, isNextDeploy } = nextTestSetup({
files: __dirname, files: __dirname,
}) })
if (isNextDev) { if (isNextDev) {
...@@ -1110,25 +1111,28 @@ describe('App Shell prefetching', () => { ...@@ -1110,25 +1111,28 @@ describe('App Shell prefetching', () => {
page = p page = p
}, },
}) })
const act = createRouterAct(page, { includeAppShellRequests: true }) let act = createRouterAct(page, { includeAppShellRequests: true })
// Reveal the LinkAccordion for /with-root-param/en/static-posts/1. The // Reveal the LinkAccordion for /with-root-param/en/static-posts/1. The
// route is fully static and opts into Partial Prefetching, so a single // route is fully static and opts into Partial Prefetching, so a single
// per-segment static prefetch fires, carrying the resolved page content // per-segment static prefetch fires, carrying the resolved page content
// plus the shell prefix above the params boundary (which the client // plus the shell prefix above the params boundary (which the client
// extracts and caches at the Fallback vary path). // extracts and caches at the Fallback vary path).
await act(async () => { const prefetchEnglishShell = () =>
await browser act(async () => {
.elementByCss( await browser
'input[data-link-accordion="/with-root-param/en/static-posts/1"]' .elementByCss(
) 'input[data-link-accordion="/with-root-param/en/static-posts/1"]'
.click() )
}, [ .click()
{ }, [
includes: 'App shell for static posts with root param: en', {
kind: 'static', includes: 'App shell for static posts with root param: en',
}, kind: 'static',
]) },
])
await prefetchEnglishShell()
await act(async () => { await act(async () => {
const startingUrl = await browser.url() const startingUrl = await browser.url()
...@@ -1159,26 +1163,54 @@ describe('App Shell prefetching', () => { ...@@ -1159,26 +1163,54 @@ describe('App Shell prefetching', () => {
'Static post 125 with root param: pl' 'Static post 125 with root param: pl'
) )
// Go back to the home page, then repeat the "prefetch post / navigate to // The test loads a fresh home page to prevent BFCache from restoring
// an unprefetched post that shares its shell" flow (from the "includes // visible accordions and triggering uncontrolled prefetches. It primes
// root params" tests) with a THIRD root param ("fr"). This proves the // English again before French to keep both root-param entries in the
// freshly prefetched "fr" shell is actually used — not the earlier "en" // client cache.
// shell. await browser.loadPage(next.url + '/with-root-param/en', {
await browser.back() beforePageLoad(p: Playwright.Page) {
page = p
},
})
act = createRouterAct(page, { includeAppShellRequests: true })
await act( const prefetchFrenchShell = async () => {
async () => { await prefetchEnglishShell()
await act(async () => {
await browser await browser
.elementByCss( .elementByCss(
'input[data-link-accordion="/with-root-param/fr/static-posts/1"]' 'input[data-link-accordion="/with-root-param/fr/static-posts/1"]'
) )
.click() .click()
}, }, [
[ {
// TODO(app-shells): why aren't there requests here? includes: 'App shell for static posts with root param: fr',
// { includes: 'App shell for static posts with root param: fr' }, kind: 'static',
] },
) ])
}
if (isNextDeploy) {
// The test refreshes before each prefetch to clear the client's cached
// fallback. Deploy can serve generic segments while ISR generates a
// shell for unprerendered root params. Next start generates that shell
// blockingly.
//
// TODO: Consolidate on-demand shell generation between next start and
// deploy. Decide whether both modes should block on the root-specific
// shell or initially serve fallback segments.
await retry(
async () => {
await browser.refresh()
await prefetchFrenchShell()
},
10_000,
1_000,
'wait for the French root-specific shell'
)
} else {
await prefetchFrenchShell()
}
await act(async () => { await act(async () => {
// Navigate to an unprefetched post that shares the "fr" shell. The // Navigate to an unprefetched post that shares the "fr" shell. The
......
...@@ -2,8 +2,6 @@ ...@@ -2,8 +2,6 @@
import { nextTestSetup, isNextDev } from 'e2e-utils' import { nextTestSetup, isNextDev } from 'e2e-utils'
import { waitForRedbox, getRedboxHeader, retry } from 'next-test-utils' import { waitForRedbox, getRedboxHeader, retry } from 'next-test-utils'
jest.retryTimes(0)
describe('Invalid hrefs', () => { describe('Invalid hrefs', () => {
const { next } = nextTestSetup({ const { next } = nextTestSetup({
files: __dirname, files: __dirname,
......
...@@ -94,12 +94,9 @@ describe('Telemetry CLI', () => { ...@@ -94,12 +94,9 @@ describe('Telemetry CLI', () => {
expect(stdout).toMatch(/Status: Disabled/) expect(stdout).toMatch(/Status: Disabled/)
}) })
;(isNextStart ? describe : describe.skip)('production mode', () => { ;(isNextStart ? describe : describe.skip)('production mode', () => {
// Tests in this block run a full `next build` per test. With a custom // Allow time for a full `next build` in each test. Webpack disables SWC
// `.babelrc` webpack switches off SWC and the build can take 60s+, // with a custom `.babelrc`, so the build can exceed the default 60s
// exceeding Jest's default 60s timeout. When the test times out, the // per-test timeout.
// build process keeps running, so the auto-retry (jest.retryTimes(1) in
// CI start mode) and subsequent tests see `this.childProcess` still set
// and throw "can not run export while server is running".
jest.setTimeout(180 * 1000) jest.setTimeout(180 * 1000)
it('emits event when swc fails to load', async () => { it('emits event when swc fails to load', async () => {
......
...@@ -138,10 +138,6 @@ if (!e2eGlobal.__NEXT_E2E_TEST_CONFIG_PATCHED__) { ...@@ -138,10 +138,6 @@ if (!e2eGlobal.__NEXT_E2E_TEST_CONFIG_PATCHED__) {
if (typeof global.test === 'function') { if (typeof global.test === 'function') {
global.test = wrapJestTestFn(global.test) as jest.It global.test = wrapJestTestFn(global.test) as jest.It
} }
if (process.env.NEXT_TEST_CI && !process.env.NEXT_FLAKE_DETECTION) {
jest.retryTimes(1)
}
} }
e2eGlobal.__NEXT_E2E_TEST_CONFIG_PATCHED__ = true e2eGlobal.__NEXT_E2E_TEST_CONFIG_PATCHED__ = true
......
...@@ -235,9 +235,6 @@ A suite with no lazy gate never resolves a config, so the cost is zero. ...@@ -235,9 +235,6 @@ A suite with no lazy gate never resolves a config, so the cost is zero.
matchers) still fails the test. Hooks registered inside a `describe` whose matchers) still fails the test. Hooks registered inside a `describe` whose
lazy `@force-gate` is false are the one exception: the suite's build and lazy `@force-gate` is false are the one exception: the suite's build and
tests are skipped, so its hooks are skipped too. tests are skipped, so its hooks are skipped too.
- `jest.retryTimes(1)` is on for non-dev CI. A stale gate fails deterministically
on both attempts, but a *flaky* gated-false test now "passes" whenever it
happens to fail.
- A gated test's title is unchanged (React renames its to - A gated test's title is unchanged (React renames its to
`[GATED, SHOULD FAIL] …`; we can't, because a lazy gate is not decided when `[GATED, SHOULD FAIL] …`; we can't, because a lazy gate is not decided when
titles are fixed). The `⚠ gated test failed as expected` line is the only titles are fixed). The `⚠ gated test failed as expected` line is the only
......
...@@ -76,12 +76,9 @@ export class NextStartInstance extends NextInstance { ...@@ -76,12 +76,9 @@ export class NextStartInstance extends NextInstance {
}) })
} }
// When a previous test attempt was interrupted (typically by exceeding the // Stop a build left running after a test exceeded its timeout so subsequent
// per-test timeout) while `next build` was still running, the build process // tests can start or build the fixture. Throw `serverRunningError` if the
// is still tracked here. Since `jest.retryTimes` re-runs the test body in the // process is serving requests instead.
// same process, stop the orphaned build so the caller can continue instead of
// failing the retry. If a server is genuinely running, throw
// `serverRunningError` instead.
private async stopLeftoverBuildOrThrow(serverRunningError: string) { private async stopLeftoverBuildOrThrow(serverRunningError: string) {
if (!this.childProcess) { if (!this.childProcess) {
return return
......
...@@ -21,9 +21,7 @@ describe('app dir - metadata static routes cache', () => { ...@@ -21,9 +21,7 @@ describe('app dir - metadata static routes cache', () => {
// runners (most often while collecting build traces). The first build and // runners (most often while collecting build traces). The first build and
// start (with the original metadata files) therefore runs in `beforeAll`, // start (with the original metadata files) therefore runs in `beforeAll`,
// which gets the longer setup-hook budget, leaving the test body responsible // which gets the longer setup-hook budget, leaving the test body responsible
// for only the second build. Keeping the build out of the test body also // for only the second build.
// makes it idempotent, so a `jest.retryTimes` retry recomputes the new hashes
// against a stable baseline.
beforeAll(async () => { beforeAll(async () => {
await next.start() await next.start()
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment