Unverified Commit 5ada2ffa authored by vercel-fleet[bot]'s avatar vercel-fleet[bot] Committed by GitHub

[bench skill] Bench Edge builds of React too (#96234)

Extends the bench to run the edge versions of React SSR fixture.
Co-authored-by: 's avatarClaude Fable 5 <noreply@anthropic.com>
Co-authored-by: 's avatardan <dan.abramov@gmail.com>
parent 7f50ebac
......@@ -5,7 +5,9 @@ description: >
A/B statistics: react PR/commit vs base, or Next.js PR/commit vs base,
measured end-to-end through the bench/render-pipeline app (rps,
latency, p95; TTFB, RSS and document/Flight bytes when the Next
side captures them). Use whenever the user asks to bench, perf
side captures them) and, for React changes, through the react
repo's flight-ssr-bench fixture (Node AND Edge web-streams paths,
Fizz and Flight+Fizz). Use whenever the user asks to bench, perf
test, or A/B a React PR, a react-server-dom / Flight / vendored React
change, or a Next.js PR ("is this PR faster", "does this regress
RSC?", "measure the perf impact of <commit>"), even if they don't say
......@@ -105,6 +107,26 @@ bash -c 'node scripts/sandbox-e2e.mjs --pr <url> --label <slug> \
2>&1 | grep --line-buffered -v "^live "; exit ${PIPESTATUS[0]}'
```
For React PRs, launch BOTH suites (separate background tasks; they
share arm builds and caches):
```sh
bash -c 'node scripts/sandbox-ssr.mjs --pr <url> --label <slug>-ssr \
2>&1 | grep --line-buffered -v "^live "; exit ${PIPESTATUS[0]}'
```
The e2e suite measures the Node path through a real Next.js app; the
ssr suite measures the react repo's flight-ssr-bench fixture — 8
variants (Fizz and Flight+Fizz, Node and Edge web streams, sync and
async), each sequentially with Flight script injection and behind an
HTTP server at c=1/c=10. Edge cells are the ssr suite's headline (the
e2e suite cannot see that path); its Node and Fizz-only cells
attribute an effect to the Flight layer, the Fizz layer, or the
stream plumbing. The fixture (the workload) is pinned to one ref for
both arms — react main by default — so only the React builds differ;
if the PR itself edits the fixture, the launcher says so and the run
does not measure those edits. Next PRs run the e2e suite only.
- Run it as a background task and proceed on its completion
notification. Never hold a foreground wait; never poll in a loop —
the rule is about control flow, not status relay: reading the
......
......@@ -54,21 +54,31 @@ with `--isolate-routes`, which restarts the server between routes.
## A/A validation
Before claiming anything on a new team/config, run an A/A cell (same
ref as both arms) at default allocation:
Before claiming anything on a new team/config/suite, run an A/A cell
(same ref as both arms) at default allocation:
```sh
node scripts/sandbox-e2e.mjs --arms base=<ref>,cand=<ref> --label aa
node scripts/sandbox-ssr.mjs --arms base=<ref>,cand=<ref> --label ssr-aa
```
The A/A validates the inference itself: with identical arms, no
route/phase/metric cell should reach p < 0.01 (beyond the nominal
false-positive rate). Cells that do mean boots are sharing conditions
and the CIs are too narrow — fix the allocation (more spread-out
boots) before claiming; a magnitude gate is not a substitute. The A/A
CIs also tell you what effect sizes this config can resolve. Re-run
the A/A when the platform changes (VM hardware generation, node
version, bench app changes).
The A/A validates the inference itself. With identical arms, expect
about (number of metric-cells x 0.01) cells at p < 0.01 by chance —
under 1 for the e2e suite's ~24 cells, roughly 1 for the ssr suite's
~88. The pass criterion is therefore two runs: a suite fails its A/A
when any cell reaches p < 0.01 with the same sign in both (a bias
tied to the arm label recurs; chance does not), or when any single
cell lands beyond the familywise bar (p < 0.01 / number of
independent variant-x-phase cells). A failing cell means boots are
sharing label-correlated conditions and the CIs are too narrow — find
the mechanism before claiming; a magnitude gate is not a substitute.
The A/A CIs also tell you what effect sizes this config can resolve.
Re-run the A/A when the platform changes (VM hardware generation,
node version, bench app or fixture changes).
Validated so far: e2e (twice, 0/24 cells), ssr (two-run criterion:
one chance cell at p=0.0024 in run 1 dissolved to p=0.86 in run 2,
nothing recurred; runs ssr-aa2/ssr-aa3 in the results db).
## Interim progress display
......
......@@ -66,6 +66,16 @@ for (const a of meta?.arms ?? [])
if (rows[0].route === '') {
// Micro (sandbox-ab) samples: phase holds the payload name.
analyzeE2eRows(rows, base, cand, ['mean', 'p50', 'p95', 'p99', 'gcMs'])
} else if (meta?.suite === 'ssr') {
analyzeE2eRows(rows, base, cand, [
'rps',
'mean',
'median',
'p95',
'p99',
'gcMs',
'heapMb',
])
} else {
// No p99 for e2e: the load phases are far too small for it.
analyzeE2eRows(rows, base, cand, [
......
This diff is collapsed.
......@@ -96,6 +96,7 @@ export const METRICS = {
gcMs: { unit: 'ms', better: 'lower' },
p50: { unit: 'ms', better: 'lower' },
rss: { unit: 'MB', better: 'lower' },
heapMb: { unit: 'MB', better: 'lower' },
rssHw: { unit: 'MB peak', better: 'lower' },
}
......
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment