Unverified Commit 5ada2ffa authored by vercel-fleet[bot]'s avatar vercel-fleet[bot] Committed by GitHub

[bench skill] Bench Edge builds of React too (#96234)

Extends the bench to run the edge versions of React SSR fixture.
Co-authored-by: 's avatarClaude Fable 5 <noreply@anthropic.com>
Co-authored-by: 's avatardan <dan.abramov@gmail.com>
parent 7f50ebac
...@@ -5,7 +5,9 @@ description: > ...@@ -5,7 +5,9 @@ description: >
A/B statistics: react PR/commit vs base, or Next.js PR/commit vs base, A/B statistics: react PR/commit vs base, or Next.js PR/commit vs base,
measured end-to-end through the bench/render-pipeline app (rps, measured end-to-end through the bench/render-pipeline app (rps,
latency, p95; TTFB, RSS and document/Flight bytes when the Next latency, p95; TTFB, RSS and document/Flight bytes when the Next
side captures them). Use whenever the user asks to bench, perf side captures them) and, for React changes, through the react
repo's flight-ssr-bench fixture (Node AND Edge web-streams paths,
Fizz and Flight+Fizz). Use whenever the user asks to bench, perf
test, or A/B a React PR, a react-server-dom / Flight / vendored React test, or A/B a React PR, a react-server-dom / Flight / vendored React
change, or a Next.js PR ("is this PR faster", "does this regress change, or a Next.js PR ("is this PR faster", "does this regress
RSC?", "measure the perf impact of <commit>"), even if they don't say RSC?", "measure the perf impact of <commit>"), even if they don't say
...@@ -105,6 +107,26 @@ bash -c 'node scripts/sandbox-e2e.mjs --pr <url> --label <slug> \ ...@@ -105,6 +107,26 @@ bash -c 'node scripts/sandbox-e2e.mjs --pr <url> --label <slug> \
2>&1 | grep --line-buffered -v "^live "; exit ${PIPESTATUS[0]}' 2>&1 | grep --line-buffered -v "^live "; exit ${PIPESTATUS[0]}'
``` ```
For React PRs, launch BOTH suites (separate background tasks; they
share arm builds and caches):
```sh
bash -c 'node scripts/sandbox-ssr.mjs --pr <url> --label <slug>-ssr \
2>&1 | grep --line-buffered -v "^live "; exit ${PIPESTATUS[0]}'
```
The e2e suite measures the Node path through a real Next.js app; the
ssr suite measures the react repo's flight-ssr-bench fixture — 8
variants (Fizz and Flight+Fizz, Node and Edge web streams, sync and
async), each sequentially with Flight script injection and behind an
HTTP server at c=1/c=10. Edge cells are the ssr suite's headline (the
e2e suite cannot see that path); its Node and Fizz-only cells
attribute an effect to the Flight layer, the Fizz layer, or the
stream plumbing. The fixture (the workload) is pinned to one ref for
both arms — react main by default — so only the React builds differ;
if the PR itself edits the fixture, the launcher says so and the run
does not measure those edits. Next PRs run the e2e suite only.
- Run it as a background task and proceed on its completion - Run it as a background task and proceed on its completion
notification. Never hold a foreground wait; never poll in a loop — notification. Never hold a foreground wait; never poll in a loop —
the rule is about control flow, not status relay: reading the the rule is about control flow, not status relay: reading the
......
...@@ -54,21 +54,31 @@ with `--isolate-routes`, which restarts the server between routes. ...@@ -54,21 +54,31 @@ with `--isolate-routes`, which restarts the server between routes.
## A/A validation ## A/A validation
Before claiming anything on a new team/config, run an A/A cell (same Before claiming anything on a new team/config/suite, run an A/A cell
ref as both arms) at default allocation: (same ref as both arms) at default allocation:
```sh ```sh
node scripts/sandbox-e2e.mjs --arms base=<ref>,cand=<ref> --label aa node scripts/sandbox-e2e.mjs --arms base=<ref>,cand=<ref> --label aa
node scripts/sandbox-ssr.mjs --arms base=<ref>,cand=<ref> --label ssr-aa
``` ```
The A/A validates the inference itself: with identical arms, no The A/A validates the inference itself. With identical arms, expect
route/phase/metric cell should reach p < 0.01 (beyond the nominal about (number of metric-cells x 0.01) cells at p < 0.01 by chance —
false-positive rate). Cells that do mean boots are sharing conditions under 1 for the e2e suite's ~24 cells, roughly 1 for the ssr suite's
and the CIs are too narrow — fix the allocation (more spread-out ~88. The pass criterion is therefore two runs: a suite fails its A/A
boots) before claiming; a magnitude gate is not a substitute. The A/A when any cell reaches p < 0.01 with the same sign in both (a bias
CIs also tell you what effect sizes this config can resolve. Re-run tied to the arm label recurs; chance does not), or when any single
the A/A when the platform changes (VM hardware generation, node cell lands beyond the familywise bar (p < 0.01 / number of
version, bench app changes). independent variant-x-phase cells). A failing cell means boots are
sharing label-correlated conditions and the CIs are too narrow — find
the mechanism before claiming; a magnitude gate is not a substitute.
The A/A CIs also tell you what effect sizes this config can resolve.
Re-run the A/A when the platform changes (VM hardware generation,
node version, bench app or fixture changes).
Validated so far: e2e (twice, 0/24 cells), ssr (two-run criterion:
one chance cell at p=0.0024 in run 1 dissolved to p=0.86 in run 2,
nothing recurred; runs ssr-aa2/ssr-aa3 in the results db).
## Interim progress display ## Interim progress display
......
...@@ -66,6 +66,16 @@ for (const a of meta?.arms ?? []) ...@@ -66,6 +66,16 @@ for (const a of meta?.arms ?? [])
if (rows[0].route === '') { if (rows[0].route === '') {
// Micro (sandbox-ab) samples: phase holds the payload name. // Micro (sandbox-ab) samples: phase holds the payload name.
analyzeE2eRows(rows, base, cand, ['mean', 'p50', 'p95', 'p99', 'gcMs']) analyzeE2eRows(rows, base, cand, ['mean', 'p50', 'p95', 'p99', 'gcMs'])
} else if (meta?.suite === 'ssr') {
analyzeE2eRows(rows, base, cand, [
'rps',
'mean',
'median',
'p95',
'p99',
'gcMs',
'heapMb',
])
} else { } else {
// No p99 for e2e: the load phases are far too small for it. // No p99 for e2e: the load phases are far too small for it.
analyzeE2eRows(rows, base, cand, [ analyzeE2eRows(rows, base, cand, [
......
This diff is collapsed.
...@@ -96,6 +96,7 @@ export const METRICS = { ...@@ -96,6 +96,7 @@ export const METRICS = {
gcMs: { unit: 'ms', better: 'lower' }, gcMs: { unit: 'ms', better: 'lower' },
p50: { unit: 'ms', better: 'lower' }, p50: { unit: 'ms', better: 'lower' },
rss: { unit: 'MB', better: 'lower' }, rss: { unit: 'MB', better: 'lower' },
heapMb: { unit: 'MB', better: 'lower' },
rssHw: { unit: 'MB peak', better: 'lower' }, rssHw: { unit: 'MB peak', better: 'lower' },
} }
......
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment