Unverified Commit 390eff3b authored by Benjamin Woodruff's avatar Benjamin Woodruff Committed by GitHub

[ci] Share a single browser instance across all test suites in a single job (#95589)

The idea here is that we waste a bunch of CPU setting up each test suite
by spawning an entirely new Chrome process, but playwright can share the
same browser across many tests. Tests are still isolated at the browser
level, just not at the OS process level.

Hoping to see some performance improvement in e2e tests. Compare to:
https://github.com/vercel/next.js/pull/95617

Somewhat unscientific (single sample) results:

```
Comparing the head-commit build-and-test runs — PR #95589 run [28981702219 ](https://github.com/vercel/next.js/actions/runs/28981702219)vs PR #95617 run [28981686692](https://github.com/vercel/next.js/actions/runs/28981686692). Both ran 101 jobs and both succeeded.

  ┌─────────────────────────┬────────────────┬───────────────────────┬───────────────────┐
  │         Metric          │ #95617 control │ #95589 shared browser │       Delta       │
  ├─────────────────────────┼────────────────┼───────────────────────┼───────────────────┤
  │ Raw wall-time sum       │ 616.2 min      │ 571.2 min             │ −45.0 min (−7.3%) │
  ├─────────────────────────┼────────────────┼───────────────────────┼───────────────────┤
  │ Billable (ceil per job) │ 658 min        │ 617 min               │ −41 min (−6.2%)   │
  └─────────────────────────┴────────────────┴───────────────────────┴───────────────────┘

  The shared-browser experiment is cheaper, and the savings land almost exactly where you'd expect — the browser-driven test suites — while non-browser jobs (rust, lint, unit, windows) are flat within noise:

  ┌─────────────────────────────────┬─────────┬────────┬───────┐
  │          Job category           │ control │ shared │ delta │
  ├─────────────────────────────────┼─────────┼────────┼───────┤
  │ test prod                       │   130.7 │  110.5 │ −20.2 │
  ├─────────────────────────────────┼─────────┼────────┼───────┤
  │ cache components dev            │    51.5 │   42.9 │  −8.7 │
  ├─────────────────────────────────┼─────────┼────────┼───────┤
  │ turbopack dev                   │    75.1 │   69.5 │  −5.5 │
  ├─────────────────────────────────┼─────────┼────────┼───────┤
  │ turbopack production            │    86.3 │   81.1 │  −5.2 │
  ├─────────────────────────────────┼─────────┼────────┼───────┤
  │ test dev                        │   110.4 │  105.5 │  −4.9 │
  ├─────────────────────────────────┼─────────┼────────┼───────┤
  │ cache components prod           │    48.7 │   49.0 │  +0.3 │
  ├─────────────────────────────────┼─────────┼────────┼───────┤
  │ firefox and safari              │     5.3 │    7.3 │  +2.0 │
  ├─────────────────────────────────┼─────────┼────────┼───────┤
  │ flake-detection jobs (combined) │     ~15 │    ~21 │   +~6 │
  └─────────────────────────────────┴─────────┴────────┴───────┘

  Takeaways:
  - Net savings of ~41 billable minutes per run (~6%), concentrated in the Chromium-driven test prod/dev, turbopack, and cache components suites — consistent with reusing one browser process instead of spawning per-suite.
  - The small regressions are in firefox and safari and the "new/changed tests for flakes" jobs (+~8 min combined). Worth a glance, though they may just be run-to-run variance.

  Caveat: this is a single run per PR, so there's real variance run-to-run. That said, the fact that the deltas track the browser-heavy jobs specifically — and not the Rust/lint/unit jobs — is a good signal the effect is genuine rather than noise. If you want a firmer number, re-running each PR 2–3× and averaging would tighten it up.
```
parent e90189b9
...@@ -9,6 +9,7 @@ const { promisify } = require('util') ...@@ -9,6 +9,7 @@ const { promisify } = require('util')
const { Sema } = require('async-sema') const { Sema } = require('async-sema')
const { spawn, exec: execOrig } = require('child_process') const { spawn, exec: execOrig } = require('child_process')
const { createNextInstall } = require('./test/lib/create-next-install') const { createNextInstall } = require('./test/lib/create-next-install')
const { getBrowserLaunch } = require('./test/lib/browsers/launch')
const glob = promisify(_glob) const glob = promisify(_glob)
const exec = promisify(execOrig) const exec = promisify(execOrig)
const core = require('@actions/core') const core = require('@actions/core')
...@@ -39,6 +40,7 @@ class TestProfile { ...@@ -39,6 +40,7 @@ class TestProfile {
'NEXT_E2E_TEST_TIMEOUT', 'NEXT_E2E_TEST_TIMEOUT',
'NEXT_TURBOPACK_IO_CONCURRENCY', 'NEXT_TURBOPACK_IO_CONCURRENCY',
'NEXT_TEST_PASSED_FILE', 'NEXT_TEST_PASSED_FILE',
'NEXT_TEST_BROWSER_WS_ENDPOINT',
'TURBO_TASKS_AVAILABLE_PARALLELISM', 'TURBO_TASKS_AVAILABLE_PARALLELISM',
]) ])
...@@ -297,6 +299,14 @@ ${output} ...@@ -297,6 +299,14 @@ ${output}
let exiting = false let exiting = false
/**
* Browser server shared across all test suites. Suites connect to it via
* `test/lib/browsers/playwright.ts` instead of each launching their own
* browser process.
* @type {import('playwright').BrowserServer | undefined}
*/
let sharedBrowserServer
const cleanUpAndExit = async (code) => { const cleanUpAndExit = async (code) => {
if (exiting) { if (exiting) {
return return
...@@ -304,6 +314,11 @@ const cleanUpAndExit = async (code) => { ...@@ -304,6 +314,11 @@ const cleanUpAndExit = async (code) => {
exiting = true exiting = true
console.log(`exiting with code ${code}`) console.log(`exiting with code ${code}`)
if (sharedBrowserServer) {
await sharedBrowserServer.close().catch((err) => {
console.error('Failed to close shared browser server:', err)
})
}
if (process.env.NEXT_TEST_STARTER) { if (process.env.NEXT_TEST_STARTER) {
await fsp.rm(process.env.NEXT_TEST_STARTER, { await fsp.rm(process.env.NEXT_TEST_STARTER, {
recursive: true, recursive: true,
...@@ -624,6 +639,24 @@ ${ENDGROUP}`) ...@@ -624,6 +639,24 @@ ${ENDGROUP}`)
console.log(`${ENDGROUP}`) console.log(`${ENDGROUP}`)
} }
// best-effort, don't spawn a browser for unit tests, if we don't spawn a
// browser but should've, that's okay, `next-webdriver` will still set it up
if (
!options.dry &&
((options.type && options.type !== 'unit') ||
tests.some((test) => !testFilters.unit.test(test.file)))
) {
// Launch a single browser server shared by all test suites, instead of
// each suite launching its own browser process. Suites connect to it via
// the ws endpoint env var in `test/lib/browsers/playwright.ts`.
const { browserType, launchOptions } = getBrowserLaunch(
process.env.BROWSER_NAME || 'chrome',
{ headless: true } // matches per-test env below
)
sharedBrowserServer = await browserType.launchServer(launchOptions)
process.env.NEXT_TEST_BROWSER_WS_ENDPOINT = sharedBrowserServer.wsEndpoint()
}
const sema = new Sema(options.concurrency, { capacity: tests.length }) const sema = new Sema(options.concurrency, { capacity: tests.length })
const outputSema = new Sema(1, { capacity: tests.length }) const outputSema = new Sema(1, { capacity: tests.length })
const children = new Set() const children = new Set()
......
...@@ -91,8 +91,9 @@ describe('app dir - prefetching', () => { ...@@ -91,8 +91,9 @@ describe('app dir - prefetching', () => {
}) })
itHeaded('should not suppress prefetches after navigating back', async () => { itHeaded('should not suppress prefetches after navigating back', async () => {
// Force headed mode, as bfcache is not available in headless mode. // Requires headed mode, as bfcache is not available in headless mode
const browser = await next.browser('/', { headless: false }) // (`itHeaded` skips this test when the HEADLESS env var is set).
const browser = await next.browser('/')
// Trigger a hard navigation. // Trigger a hard navigation.
await browser.elementById('to-static-page-hard').click() await browser.elementById('to-static-page-hard').click()
......
...@@ -10,7 +10,9 @@ describeHeaded('fetch-abort-on-refresh', () => { ...@@ -10,7 +10,9 @@ describeHeaded('fetch-abort-on-refresh', () => {
it('should not show abort error in global error boundary when restoring from bfcache', async () => { it('should not show abort error in global error boundary when restoring from bfcache', async () => {
// This test ensures that when restoring a page from the browser bfcache that was pending RSC data, // This test ensures that when restoring a page from the browser bfcache that was pending RSC data,
// that the abort does not propagate to a user's error boundary. // that the abort does not propagate to a user's error boundary.
const browser = await next.browser('/', { headless: false }) // Requires headed mode, as bfcache is not available in headless mode
// (`describeHeaded` skips this suite when the HEADLESS env var is set).
const browser = await next.browser('/')
await browser.elementById('trigger-navigation').click() await browser.elementById('trigger-navigation').click()
......
//@ts-check
const { chromium, firefox, webkit } = require('playwright')
/**
* Maps a test browser name (`BROWSER_NAME`) to a Playwright `BrowserType` and
* the launch options we always use for it.
*
* This is a plain CommonJS module so it can be shared between `run-tests.js`
* (which launches a browser server that's reused across all test suites) and
* `test/lib/browsers/playwright.ts` (which launches its own browser when no
* shared server is available, e.g. when running a test directly via the jest
* CLI). Both must launch the browser with identical options.
*
* @param {string} browserName
* @param {{ headless: boolean }} options
* @returns {{
* browserType: import('playwright').BrowserType,
* launchOptions: import('playwright').LaunchOptions,
* }}
*/
function getBrowserLaunch(browserName, { headless }) {
if (browserName === 'safari') {
return { browserType: webkit, launchOptions: { headless } }
} else if (browserName === 'firefox') {
return {
browserType: firefox,
launchOptions: {
headless,
firefoxUserPrefs: {
// The "fission.webContentIsolationStrategy" pref must be
// set to 1 on Firefox due to the bug where a new history
// state is pushed on a page reload.
// See https://github.com/microsoft/playwright/issues/22640
// See https://bugzilla.mozilla.org/show_bug.cgi?id=1832341
'fission.webContentIsolationStrategy': 1,
},
},
}
} else {
return {
browserType: chromium,
launchOptions: {
headless,
args: headless ? [] : ['--auto-open-devtools-for-tabs'],
ignoreDefaultArgs: ['--disable-back-forward-cache'],
},
}
}
}
module.exports = { getBrowserLaunch }
import fs from 'fs-extra' import fs from 'fs-extra'
import { debugPrint } from 'next-test-utils' import { debugPrint } from 'next-test-utils'
import { import {
chromium,
webkit,
firefox,
Browser, Browser,
BrowserContext, BrowserContext,
Page, Page,
...@@ -15,6 +12,7 @@ import { ...@@ -15,6 +12,7 @@ import {
BrowserContextOptions, BrowserContextOptions,
} from 'playwright' } from 'playwright'
import path from 'path' import path from 'path'
import { getBrowserLaunch } from './launch'
type EventType = 'request' | 'response' type EventType = 'request' | 'response'
...@@ -170,7 +168,6 @@ export class Playwright<TCurrent = undefined> { ...@@ -170,7 +168,6 @@ export class Playwright<TCurrent = undefined> {
locale: string, locale: string,
javaScriptEnabled: boolean, javaScriptEnabled: boolean,
ignoreHTTPSErrors: boolean, ignoreHTTPSErrors: boolean,
headless: boolean,
userAgent: string | undefined, userAgent: string | undefined,
permissions: Permissions permissions: Permissions
) { ) {
...@@ -211,7 +208,7 @@ export class Playwright<TCurrent = undefined> { ...@@ -211,7 +208,7 @@ export class Playwright<TCurrent = undefined> {
return return
} }
browser = await this.launchBrowser(browserName, { headless }) browser = await this.launchBrowser(browserName)
context = await browser.newContext({ context = await browser.newContext({
locale, locale,
javaScriptEnabled, javaScriptEnabled,
...@@ -228,35 +225,25 @@ export class Playwright<TCurrent = undefined> { ...@@ -228,35 +225,25 @@ export class Playwright<TCurrent = undefined> {
await page?.close() await page?.close()
} }
async launchBrowser( async launchBrowser(browserName: string) {
browserName: string, // Headless mode is controlled exclusively by the HEADLESS env var. It
launchOptions: { headless: boolean } // can't be a per-test option because the browser is shared: either
) { // launched once per process, or via the shared browser server below.
if (browserName === 'safari') { const headless = !!process.env.HEADLESS
return await webkit.launch(launchOptions) const { browserType, launchOptions } = getBrowserLaunch(browserName, {
} else if (browserName === 'firefox') { headless,
return await firefox.launch({ })
...launchOptions,
firefoxUserPrefs: { // `run-tests.js` launches a browser server that's shared across all test
// The "fission.webContentIsolationStrategy" pref must be // suites. Connect to it if it's available, otherwise (e.g. when running a
// set to 1 on Firefox due to the bug where a new history // test directly via the jest CLI) launch a browser owned by this process.
// state is pushed on a page reload. // The shared server is always headless (`run-tests.js` sets HEADLESS=true
// See https://github.com/microsoft/playwright/issues/22640 // for all test suites), so only connect when we're headless too.
// See https://bugzilla.mozilla.org/show_bug.cgi?id=1832341 const wsEndpoint = process.env.NEXT_TEST_BROWSER_WS_ENDPOINT
'fission.webContentIsolationStrategy': 1, if (wsEndpoint && headless) {
}, return await browserType.connect(wsEndpoint)
})
} else {
let launchArgs: string[] = []
if (!launchOptions.headless) {
launchArgs.push('--auto-open-devtools-for-tabs')
}
return await chromium.launch({
...launchOptions,
args: launchArgs,
ignoreDefaultArgs: ['--disable-back-forward-cache'],
})
} }
return await browserType.launch(launchOptions)
} }
async get(url: string): Promise<void> { async get(url: string): Promise<void> {
......
import { debugPrint, getFullUrl } from 'next-test-utils' import { debugPrint, getFullUrl } from 'next-test-utils'
import os from 'os'
import { import {
Permissions, Permissions,
Playwright, Playwright,
...@@ -13,26 +12,8 @@ if (!process.env.TEST_FILE_PATH) { ...@@ -13,26 +12,8 @@ if (!process.env.TEST_FILE_PATH) {
process.env.TEST_FILE_PATH = module.parent!.filename process.env.TEST_FILE_PATH = module.parent!.filename
} }
let deviceIP: string
const isBrowserStack = !!process.env.BROWSERSTACK
;(global as any).browserName = process.env.BROWSER_NAME || 'chrome' ;(global as any).browserName = process.env.BROWSER_NAME || 'chrome'
if (isBrowserStack) {
const nets = os.networkInterfaces()
for (const key of Object.keys(nets)) {
let done = false
for (const item of nets[key]!) {
if (item.family === 'IPv4' && !item.internal) {
deviceIP = item.address
done = true
break
}
}
if (done) break
}
}
let browserTeardown: (() => Promise<void>)[] = [] let browserTeardown: (() => Promise<void>)[] = []
let browserQuit: (() => Promise<void>) | undefined let browserQuit: (() => Promise<void>) | undefined
...@@ -84,7 +65,6 @@ export interface WebdriverOptions { ...@@ -84,7 +65,6 @@ export interface WebdriverOptions {
* disable javascript * disable javascript
*/ */
disableJavaScript?: boolean disableJavaScript?: boolean
headless?: boolean
/** /**
* ignore https errors * ignore https errors
*/ */
...@@ -138,7 +118,6 @@ export default async function webdriver( ...@@ -138,7 +118,6 @@ export default async function webdriver(
disableJavaScript, disableJavaScript,
permissions, permissions,
ignoreHTTPSErrors, ignoreHTTPSErrors,
headless,
cpuThrottleRate, cpuThrottleRate,
pushErrorAsConsoleLog, pushErrorAsConsoleLog,
disableBrowserLog, disableBrowserLog,
...@@ -160,18 +139,12 @@ export default async function webdriver( ...@@ -160,18 +139,12 @@ export default async function webdriver(
locale!, locale!,
!disableJavaScript, !disableJavaScript,
Boolean(ignoreHTTPSErrors), Boolean(ignoreHTTPSErrors),
// allow headless to be overwritten for a particular test
typeof headless !== 'undefined' ? headless : !!process.env.HEADLESS,
userAgent, userAgent,
permissions permissions
) )
;(global as any).browserName = browserName ;(global as any).browserName = browserName
const fullUrl = getFullUrl( const fullUrl = getFullUrl(appPortOrUrl, url, 'localhost')
appPortOrUrl,
url,
isBrowserStack ? deviceIP : 'localhost'
)
debugPrint(`Loading browser with ${fullUrl}`) debugPrint(`Loading browser with ${fullUrl}`)
......
...@@ -31,12 +31,12 @@ describe('bfcache-routing', () => { ...@@ -31,12 +31,12 @@ describe('bfcache-routing', () => {
itHeaded( itHeaded(
'should not suspend indefinitely when page is restored from bfcache after an mpa navigation', 'should not suspend indefinitely when page is restored from bfcache after an mpa navigation',
async () => { async () => {
// bfcache is not currently supported by CDP, so we need to run this particular test in headed mode // bfcache is not currently supported by CDP, so this test only runs in
// headed mode (`itHeaded` skips it when the HEADLESS env var is set)
// https://bugs.chromium.org/p/chromium/issues/detail?id=1317959 // https://bugs.chromium.org/p/chromium/issues/detail?id=1317959
const browser = await next.browser('/index.html', { const browser = await next.browser('/index.html', {
baseUrl: port, baseUrl: port,
headless: false,
}) })
// we overwrite the typical waitUntil: 'load' option here as the event is never being triggered if we hit the bfcache // we overwrite the typical waitUntil: 'load' option here as the event is never being triggered if we hit the bfcache
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment