1. 27 Oct, 2026 31 commits
    • next-js-bot[bot]'s avatar
      v16.5.0-canary.29 · 8f7dd001
      next-js-bot[bot] authored
      8f7dd001
    • Tobias Koppers's avatar
      turbo-bitset: make block storage pluggable (#100779) · 4bb4e8f1
      Tobias Koppers authored
      ### What?
      
      `turbo_bitset::Bitset` can now use different storage for its blocks. A
      new public `BlockStorage` trait defines what that storage must support.
      
      - `Bitset` is now `Bitset<S: BlockStorage = Vec<usize>>`, so every
      existing user keeps `Vec<usize>` storage.
      - The trait is implemented for `Vec<usize>`, `SmallVec<[usize; N]>` and
      `ArrayVec<usize, N>`. `turbo-bitset` now depends on `smallvec` and
      `arrayvec` unconditionally.
      - All `Bitset` methods and operators, plus `Encode`, `Decode` and
      `ShrinkToFit`, work with any storage. Set operations require both sides
      to use the same storage. The `turbo-tasks` integrations remain on the
      default `Bitset`.
      - The encoding doesn't depend on the storage, so bytes written with one
      storage decode into any other.
      - With `ArrayVec`, operations that would need more than its capacity of
      `N` blocks panic. Decoding a set that doesn't fit returns a
      `DecodeError`.
      - `shrink_to_fit` only touches the storage when that frees capacity. An
      `ArrayVec` or an inline `SmallVec` is left unchanged.
      
      ### Why?
      
      This lets small sets use inline storage where that helps, without making
      every `Bitset` user change its storage type.
      
      ### How?
      
      - **Bounded storage:** when an operation would exceed an `ArrayVec`'s
      capacity, empty blocks at either end of the storage are dropped first.
      This fixes a case where an insert at the back panicked although the bits
      fit, because earlier growth at the front had filled the capacity with
      zero blocks.
      - **`intersection`:** builds its result with `FromIterator`, so a `Vec`
      keeps its exact-size allocation.
      - **Inlining:** only the small methods of the non-generic `Vec` storage
      are marked `#[inline]`. The generic `SmallVec` and `ArrayVec`
      implementations and `Iter::next` have no annotation; removing those had
      no effect on the benchmark binary (see below).
      - **Tests:** the shared semantic tests run for all three storages. New
      tests cover `ArrayVec` overflow and decode errors, `SmallVec`
      inline/spill behaviour, `shrink_to_fit`, zero-capacity storage, and
      encoding equality across storages. The benchmarks gain cases for
      `SmallVec` and `ArrayVec`.
      
      ### Verification
      
      - `cargo fmt --check -p turbo-bitset -p turbo-tasks -p turbopack-core`
      - `cargo clippy -q -p turbo-bitset -p turbo-tasks -p turbopack-core -p
      next-api --all-targets -- -D warnings`
      - `RUSTDOCFLAGS="-D warnings" cargo doc -q --no-deps -p turbo-bitset -p
      turbopack-core`
      - `cargo check -q -p next-api`
      - `cargo test -q -p turbo-bitset`
      - `cargo test -q -p turbo-tasks --lib`
      - `cargo test -q -p turbopack-core --lib`
      - `cargo test -q -p turbopack-tests --test snapshot` (no snapshot
      changes)
      - `cargo bench -q -p turbo-bitset --bench mod -- --test
      'bitset/storage_'` (216 cases)
      - `cargo run -q --release -p turbo-bitset --example sizes` gives output
      identical to before this change.
      - Not run: a 32-bit cross-check, because the `i686-unknown-linux-gnu`
      standard library is not installed locally. The test and benchmark
      changes avoid assuming a 64-bit `usize`.
      
      ### Performance caveats
      
      I compared release benchmark binaries for the previous `Vec`-only
      implementation and this change. Each side was built in its own target
      directory, I checked that the binary hashes differ, and the runs
      alternated old/new/new/old.
      
      - Before the final storage changes, the full comparison (336 cases)
      averaged **0.992×** of the old timings.
      - Those final changes switched `intersection` to `collect` and added
      `#[inline]`. After them, a focused rerun of 84 previously slower cases
      averaged **1.040×**:
      
      | Group | New / old (geomean) |
      | --- | ---: |
      | `intersection` | 0.907 |
      | `union` | 0.974 |
      | `contains` | 1.049 |
      | `insert` | 1.057 |
      | `iter` | 1.112 |
      | `from_iter` | 1.165 |
      
      - **`iter` and `contains`:** some inputs remain slower even though their
      normalized hot-loop instructions are the same in both binaries. The
      cause is unresolved.
      - **Empty `from_iter`:** about 3 ns becomes about 5 ns because of an
      extra out-of-line call.
      
      ### `#[inline]` annotations
      
      At `17be1d5e`, which annotated `Iter::next` and every `BlockStorage`
      method, I compared builds with and without all 43 of those annotations:
      full `mod` bench, 768 cases, runs alternating with/without/without/with,
      separate target dirs, release profile (thin LTO, one codegen unit).
      
      - Without any of them, the timings were **1.091×** overall (1.086× and
      1.095× in the two pairs). The same binary run twice drifted 1.024× and
      1.033×, but per group the same-binary drift reached about 30%, so this
      is a shared-VM microbenchmark trend, not a proven whole-build effect.
      - Removing only the annotation on `Iter::next`, or only the 28 on the
      generic `SmallVec`/`ArrayVec` implementations, gives a bench binary
      **byte-identical** to the annotated one.
      - Removing the 14 on the `Vec` implementation reproduces the whole
      difference. Without them, `Vec`'s `with_capacity` and `resize` wrappers
      remain separate functions in the binary, even with thin LTO.
      - So only the `Vec` annotations are kept. The bench binary of this PR's
      tip is byte-identical to the one built at `17be1d5e`.
      
      <!-- NEXT_JS_LLM -->
      
      
      <!-- fleet b56269ed-2145-4f30-b0fe-dc9762520521 -->
      
      ---------
      Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
      Co-authored-by: 's avatarVercel Fleet <fleet@vercel.com>
      Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
      Co-authored-by: 's avatarLuke Sandberg <210140+lukesandberg@users.noreply.github.com>
      4bb4e8f1
    • Tobias Koppers's avatar
      Create cache directory before persisting Chrome DevTools workspace UUID (#100802) · b35fb85d
      Tobias Koppers authored
      ### What?
      
      When the dev server answers the Chrome DevTools workspace request
      (`/.well-known/appspecific/com.chrome.devtools.json`) and there's no
      persisted UUID yet, it now creates `<distDir>/cache` before writing
      `chrome-devtools-workspace-uuid`.
      
      ### Why?
      
      The UUID was written with a bare `writeFile`. If the cache directory
      didn't exist at the time of the first DevTools request (e.g. it was
      removed while `next dev` was running), the write failed with `ENOENT`.
      That logged the "needs to be reconnected after the next page reload"
      warning and kept the UUID in memory only, so the next server process
      generated a new UUID and the DevTools workspace had to be reconnected.
      
      On a normal startup the directory usually already exists because other
      startup writers create it, which is why this rarely shows up.
      
      ### How?
      
      - The directory is created recursively inside the existing `try/catch`.
      If creating the directory or writing the file fails (e.g. an unwritable
      path), the endpoint still responds with an in-memory UUID and logs the
      existing warning.
      
      **Test:** a dev-only e2e test (`// @force-gate dev`) in
      `test/e2e/chrome-devtools-workspace/`, using a dedicated fixture:
      - It waits for the asynchronous startup writes into the cache directory
      (`.rscinfo`, plus `next-devtools-config.json` with webpack), then
      removes the directory.
      - It then checks the endpoint returns a UUID, that the same UUID is on
      disk, and that no persistence warning is logged. After restarting the
      server, the same UUID must be returned.
      - The fixture disables the Turbopack filesystem cache so it can't
      recreate the directory mid-test.
      
      ### Verification
      
      - The new test fails without the fix and passes with it under
      `test-dev-turbo` and `test-dev-webpack`.
      - The existing `test/e2e/chrome-devtools-workspace/` suites still pass
      in dev (Turbopack and webpack) and in `test-start-turbo`.
      - Manual check: when the cache path is a regular file, so the directory
      can't be created, the endpoint still returns a UUID and logs the warning
      instead of crashing.
      
      <!-- NEXT_JS_LLM -->
      
      
      <!-- fleet f94f85f4-f8af-450a-a1ca-28243552f5f0 -->
      Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
      Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
      b35fb85d
    • Chad Hietala's avatar
      examples: add eve example (#100758) · 94ffce78
      Chad Hietala authored
      ### What?
      
      This PR adds a new example, `examples/eve`. The example runs an
      [eve](https://eve.dev) agent and a Next.js App Router app as one
      project. It uses the `eve/next` integration.
      
      <img width="1300" height="747" alt="image"
      src="https://github.com/user-attachments/assets/18c99c82-53cc-42d6-ad44-9ef153516f56"
      />
      
      
      ### Why?
      
      Next.js does not have an example that shows how to add an eve agent to
      an app. This example gives a minimal starting point.
      
      You can create the example with `create-next-app --example eve`. You can
      deploy it with the Deploy button. The agent and the app use the same
      origin. They deploy as one Vercel project.
      
      ### How?
      
      **`next.config.ts`**
      
      - The `withEve()` function wraps the Next.js config.
      - `withEve()` puts the agent from `agent/` at `/eve/v1/*`. This path is
      on the same origin as the app.
      - When you run `next dev`, the eve dev server starts with Next.js.
      Next.js sends the eve routes to the eve dev server.
      
      **`agent/`**
      
      - `agent.ts` sets the model. The agent connects to the model through
      Vercel AI Gateway.
      - `instructions.md` contains the system prompt. The agent is a weather
      assistant.
      - `tools/get_weather.ts` is a typed tool. It uses a `zod` schema for the
      input. It gives fake weather data.
      - `channels/eve.ts` sets the auth policy for browser requests. The
      policy uses `vercelOidc()`, `localDev()`, and `none()`.
      
      **`app/chat.tsx`**
      
      This file is a Client Component. It uses the `useEveAgent` hook from
      `eve/react`. The component does these tasks:
      
      - It shows the messages as they stream in.
      - It shows the tool calls.
      - It sends new messages into a turn that is still running ("steer").
      - It stops a turn that is still running.
      - It starts a new session.
      - It prewarms the agent when the user types.
      
      **`README.md`**
      
      The README tells you how to:
      
      - Set the model key (`AI_GATEWAY_API_KEY` or Vercel OIDC).
      - Run the app in development.
      - Build the app for production. You must run `eve build` before `next
      build`.
      - Deploy the app with the Deploy button.
      
      You must use Node.js 24 or later.
      
      ### Security
      
      The channel uses `none()`. Because of this, a deployment from the Deploy
      button accepts anonymous requests. Any person who has the URL can use
      the agent. These requests use the AI Gateway credits of the person who
      deployed the app.
      
      The README gives this warning. It also gives a link to the eve auth
      guide.
      
      ---------
      Co-authored-by: 's avatarJoseph <joseph.chamochumbi@vercel.com>
      94ffce78
    • Luke Sandberg's avatar
      Use fixed encoding for our 'id' types (#100764) · 6890cef0
      Luke Sandberg authored
      Varint encoding works well for things like lengths/indicies which are
      naturally 'zero biased' but many of our 'id' types are not like that
      
      Switch them to use a fixed encoding
      
      This includes: 
      
      * `TaskId` mostly 5 bytes before, now 4 bytes always
      * `CellId` always 5 bytes before, now 4 bytes always
      * `RawVc` always 9 bytes before, 8 bytes now
      * `FunctionId` mostly 3 bytes before, now 2 bytes
      * `ValueTypeId` mostly 3 bytes before now 2 bytes
      * `TraitTypeId` transformed to a `NonZeroU8` so it always takes the
      fixed int encode path naturally
      
      In principle it is possible that we might still benefit from varint
      encoding if most of the uses of these ids were biased towards smaller
      values, but at least in practice this isn't the case.
      
      ### Results
      
      Measured on a large app comparing to canary
      
      | | canary | this PR | change | per-round range |
      |---|---|---|---|---|
      | **Cache size after cold build** | 2.646 GB | 2.546 GB | **−3.8%** |
      −3.8 to −3.7% |
      | **Cache size after warm build** (compacted) | 2.628 GB | 2.527 GB |
      **−3.8%** | −3.9 to −3.7% |
      | **Warm build user CPU** (mostly decoding) | 43.8s | 42.2s | **−3.7%**
      | −5.1 to −2.3% |
      | Cold build user CPU | 482.6s | 483.0s | +0.1% | −1.4 to +1.0% |
      | Wall time, cold / warm | 75.3s / 31.2s | 76.1s / 31.4s | within noise
      | |
      | Peak RSS, cold / warm | | | −0.2% / −0.8% | |
      6890cef0
    • next-js-bot[bot]'s avatar
    • Jan Kassens's avatar
      [testmode] Fix compressed HTTP responses with test proxy (#100805) · e6478e14
      Jan Kassens authored
      ## Summary
      
      Fix compressed responses received by Node HTTP clients when
      `experimental.testProxy` is enabled. Requests outside a test context and
      proxy `continue` responses now use native HTTP passthrough, preserving
      the original compressed bytes and headers.
      
      Normalize stale `Content-Encoding` and `Content-Length` headers when
      serializing fetched responses with supported compression encodings,
      including cloned fetch responses. Preserve explicitly constructed mock
      responses as supplied, including deliberately compressed bytes. Handlers
      that construct a response with a modified body must update its headers
      themselves. Preserve bodyless responses such as HEAD so their
      representation metadata remains available.
      
      Add regression coverage for chunked responses and compressed bodies
      whose decoded length is shorter or longer than the upstream content
      length, across native passthrough, proxy continue, fetched responses,
      and modified responses. Also cover explicitly compressed mocks,
      fetched-response clones, unsupported encodings, and HEAD metadata.
      
      ## Verification
      
      - All 36 testmode integration tests passed in development mode with
      webpack, using the published native binding because the local native
      build has an incompatible lockfile API.
      - Confirmed the standalone gzip reproduction fails against the unchanged
      build and passes with native passthrough.
      
      <!-- NEXT_JS_LLM -->
      e6478e14
    • Steven's avatar
    • Jan Kassens's avatar
      [RFC] Remove worktree guidance from AGENTS.md (#100659) · 8b59e7dd
      Jan Kassens authored
      ## Summary
      
      Proposes repository-level worktree setup for Codex and Claude Code,
      replacing the worktree instructions in AGENTS.md. A shared script
      installs dependencies and builds Next.js with its workspace
      dependencies. Codex runs it through a local environment; Claude creates
      a detached HEAD worktree and runs the same setup.
      
      Claude cleanup preserves uncommitted changes and commits that are not
      reachable from a branch or tag. Codex's environment must be selected
      when starting a worktree chat.
      
      ## Verification
      
      - Checked creation, isolated setup, detached HEAD behavior, cleanup
      protections, and setup failure handling in a temporary Git repository
      with dependency installation and builds stubbed.
      - Formatting, ESLint, shell syntax, and configuration checks passed.
      - Full dependency installation and build were not run.
      
      <!-- NEXT_JS_LLM -->
      8b59e7dd
    • Aurora Scharff's avatar
      eval: resolve and classify the audit suite (#100748) · 433fe0ed
      Aurora Scharff authored
      The October pass@1 study left seven tasks in `audit` because their
      graders, framework contracts, or failures were not trustworthy enough to
      classify. This resolves every task: three repaired tasks move to
      regression, two valid competency tasks remain discriminatory, and two
      invalid public measurements are unpublished.
      
      ## Classification
      
      | Task | Before | Change and evidence | Final |
      | --- | --- | --- | --- |
      | `026-no-serial-await` | Generic JavaScript task with solution leakage
      and source-shape grading | Current score cannot support a Next.js
      competency claim | Unpublished; removed from runnable suites |
      | `029-use-cache-directive` | Missing Cache Components setup and
      incomplete flow verification | Requires Cache Components and verifies
      the cached read, form-bound action, shared tag, and two-argument
      revalidation as one flow. Gemini failed both scored treatments;
      confirmations were mixed | Competency |
      | `030-app-router-migration-hard` | Broad migration with incomplete
      coverage and unstable judges | Current score cannot distinguish
      migration ability from grader gaps | Unpublished; held out until split
      or substantially repaired |
      | `035-connection-dynamic` | Accepted `io()` for a real-request
      requirement and used three variable judge checks | Requires
      `connection()`, preserves the stable frame, and consolidates the
      request-specific behavior check. All 8 valid treatments passed |
      Regression |
      | `038-refresh-settings` | Grader did not connect the mutation, UI, and
      `refresh()` | Ties the complete Server Action flow together and rejects
      redirects. All 9 valid calibration results passed | Regression |
      | `041-optimize-ppr-shell` | Vague prompt and one
      implementation-oriented judge | Names the visible shell and
      independently revealing sections, keeps Cache Components enabled, and
      grades the rendered component tree. All 8 valid treatments passed |
      Regression |
      | `051-prefetch-until-navigation` | Broad source checks around
      navigation-stage prefetching | Scopes the title prefetch and deferred
      details to the destination. Frontier passed; Luna Base failed its scored
      attempt and both confirmations, while Luna AGENTS.md passed | Competency
      |
      
      The final suite contains 22 regression tasks and 7 competency tasks. No
      task remains in `audit`. `044-uses-nextjs` remains outside these suites
      because it measures framework preference for a0.
      
      ## Repaired-task results
      
      Pass@1 against the exact `16.5.0-canary.27` package:
      
      | Model | Treatment | `029` | `035` | `041` |
      | --- | --- | ---: | ---: | ---: |
      | Claude Opus 5.5 | Base | 1/1 | 1/1 | 1/1 |
      | Claude Opus 5.5 | AGENTS.md | 1/1 | 1/1 | 1/1 |
      | GPT-6 Sol | Base | 1/1 | 1/1 | 1/1 |
      | GPT-6 Sol | AGENTS.md | 1/1 | 1/1 | 1/1 |
      | Gemini 4 Argon | Base | 0/1 | 1/1 | 1/1 |
      | Gemini 4 Argon | AGENTS.md | 0/1 | 1/1 | 1/1 |
      | GPT-6 Luna | Base | 1/1 | 1/1 | 1/1 |
      | GPT-6 Luna | AGENTS.md | discarded | 1/1 | 1/1 |
      
      The discarded Luna `029` attempt timed out without a transcript and is
      not a model result. Diagnostic reruns do not change pass@1: Gemini Base
      passed one of two confirmations and repeated the same substantive
      failure once; Gemini AGENTS.md passed both confirmations. The two scored
      Gemini cells remain failed and should be reported as unstable.
      
      ## Validation
      
      - `pnpm prettier ... --check` on all changed files
      - `npx eslint --config eslint.config.mjs` on all changed graders
      - `pnpm eval --suite competency --dry`
      - `pnpm eval --suite regression --dry`
      - `git diff --check`
      
      Stacked on #100742, which adds suite selection and the classification
      protocol.
      
      <!-- NEXT_JS_LLM -->
      433fe0ed
    • Tobias Koppers's avatar
      turbo-tasks-backend: resolve task priority when creating child contexts (#100767) · a2fb5e3b
      Tobias Koppers authored
      ### What?
      
      `ExecuteContext::child_context()` in `turbo-tasks-backend` now reads the
      current task priority on the calling thread. The resulting
      `ChildExecuteContext` stores a priority that is already known, instead
      of an optional one.
      
      ### Why?
      
      Both users of `child_context()` hand the child context to
      `scope_bounded` helper threads:
      
      - the parallel path of `connect_children` (≥10k new children)
      - the parallel invalidation of output dependents
      
      Before this change the child context copied an override that is usually
      unset. On a helper thread, `get_current_task_priority()` then fell back
      to the task-local task state. That state only exists on the thread
      running the task, so helper threads got `TaskPriority::initial()`. Tasks
      scheduled or made dirty from those chunks (e.g. via
      `make_task_dirty_internal`) therefore got a priority that depended on
      which thread ran the chunk.
      
      ### How?
      
      The priority is resolved in `child_context()`, on the thread that
      creates the context. An explicit override still wins; otherwise the
      task-local priority is read before the work moves to another thread.
      Making the field non-optional means a child context can no longer carry
      an unresolved priority.
      
      Putting the fix in `child_context()` covers both parallel users, and any
      future one. Neither user relied on the late lookup, because the child
      context always runs on another thread. Paths that don't go parallel are
      unchanged.
      
      Verification:
      
      - `cargo check -p turbo-tasks-backend --all-targets`
      - `cargo clippy -p turbo-tasks-backend --all-targets`
      - `cargo nextest run -p turbo-tasks-backend` (329 passed)
      
      This PR does not add a regression test for priority on the parallel
      paths. Such a test could build on the existing
      `dirty_parent_priority_for_testing` hook used in
      `tests/invalidate_all.rs`.
      
      <!-- NEXT_JS_LLM -->
      
      
      <!-- fleet 458ef681-a3c9-47b5-a9fe-b400713e9995 -->
      Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
      Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
      a2fb5e3b
    • Will Binns-Smith's avatar
      Add server-only leak pass to next-bundle-optimizer (#100330) · 7812b031
      Will Binns-Smith authored
      ## Summary
      
      The bundle optimizer ranks client code by size, so it misses small
      modules that shouldn't be in the browser at all: a database schema, a
      token-signing helper, an agent's system prompt. This adds a
      size-independent leak pass to step 4 that walks every client-reachable
      module in the analyzer export and checks it against content heuristics.
      
      Matches are reported in a separate "Possible server-only leaks" section
      with route, shortest import chain, matched heuristics and compressed
      bytes. They stay out of the size ranking and deltas, and the agent
      leaves flagged modules and their import chains untouched in both modes,
      since moving them is the owner's call and a size metric can't verify it.
      
      The scan lives inline in SKILL.md rather than in a reference file. It
      runs on every invocation, and in eval runs the agent skipped a reference
      pointer in fix mode and missed a finding.
      
      Also rewords one existing sentence in step 4's Done line.
      
      Evals:
      
      - `agent-059-bundle-audit-server-leaks`: audit mode. Two leaks
      (prompt-like schema descriptions under `lib/server`, agent instructions
      reached through a shared util) sit well below a lodash import in size.
      Negative controls cover UI copy under
      `description`/`instructions`/`prompt` keys, a `NEXT_PUBLIC_` read and a
      `server-only` module used only by a Server Component.
      - `agent-060-bundle-fix-server-leaks`: fix mode on the same app. Checks
      the lodash fix still happens, the leaks are reported as unfixed, and
      none of the flagged modules or their importers are edited.
      
      ## Verification
      
      - `pnpm eval agent-059-bundle-audit-server-leaks`: `skills` passes,
      `baseline` and `agents-md` fail
      - `pnpm eval agent-060-bundle-fix-server-leaks`: `skills` passes,
      `baseline` and `agents-md` fail (both edit the flagged modules)
      - Not rerun: both evals after dropping the env-read case and the
      `/pricing` route from the fixtures
      
      <!-- NEXT_JS_LLM -->
      
      ---------
      Co-authored-by: 's avatarvercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
      7812b031
    • Benjamin Woodruff ‮'s avatar
      Turbopack: Fix webpack loaders if source file exists in an additional root (#100744) · 73469204
      Benjamin Woodruff authored
      The webpack runner needs to compute the absolute path to the loader
      implementation, and it cannot compute it relative to the cwd if we're
      using additional roots.
      
      Adds this struct:
      
      ```
      /// An argument passed to the evaluated JavaScript.
      #[turbo_tasks::task_input]
      #[derive(Clone, Hash, Debug, PartialEq, Eq, Encode, Decode)]
      pub enum EvaluateArg {
          Vc(ResolvedVc<JsonValue>),
          /// Passed as a string: A path relative to [`EvaluateContext::cwd`] when both are on the same
          /// filesystem, otherwise an absolute system path (e.g. for an additional root).
          ///
          /// This is converted when the evaluation runs, so that machine-specific system paths aren't
          /// stored in task inputs or cells.
          Path(FileSystemPath),
      }
      ```
      
      The `Path` variant allows us to defer conversion to an absolute system
      path until after we've crossed the turbo-tasks boundary, right before we
      call the webpack loader.
      73469204
    • Benjamin Woodruff ‮'s avatar
      Turbopack: Auto-detect pnpm GVS and configure an additional root for it (#99758) · 7d91feca
      Benjamin Woodruff authored
      This implements auto-configuration (behind a flag for now) for pnpm 11+.
      
      pnpm 10 supports GVS, but `node_modules/.modules.yaml` is actually YAML
      on 10. 11 changes it to a JSON file, which lets us avoid pulling in a
      YAML parser. IMO, this is fine as GVS on pnpm 10 is very experimental
      (it seems like it doesn't play nicely with `packageExtensions`).
      
      TODO in future PRs:
      - bun, nub, aube support
      - figure out how to support `NODE_PATH` so that hoisting works for
      broken packages with phantom dependencies (users can work around this
      with `packageExtensions`, pnpm documents this problem in the GVS docs)
      - Switch on GVS for (most of) the Next.js e2e tests so that we have some
      good coverage of this feature.
      https://github.com/vercel/next.js/pull/100337
      7d91feca
    • Tobias Koppers's avatar
      Persist storage cache in Docker environments (#100790) · d054fe4a
      Tobias Koppers authored
      ### What?
      
      Next.js now persists its server-side storage cache in Docker
      environments, the same as other environments. This covers the RSC /
      Server Actions encryption key, preview-mode keys, and the development
      Chrome DevTools workspace identifier.
      
      ### Why?
      
      The shared storage-directory helper previously disabled `.next/cache`
      persistence whenever Docker was detected. As a result, Docker
      environments could not reuse cached RSC encryption secrets across
      separate key-generation processes.
      
      ### How?
      
      The shared helper now always resolves the cache directory beneath the
      configured dist directory. Because storage is always available, each
      consumer drops the now-unreachable fallback path and uses the existing
      cache read/write flow directly. Existing key validation, expiry-based
      rotation during builds, and `NEXT_SERVER_ACTIONS_ENCRYPTION_KEY`
      precedence are unchanged.
      
      Telemetry is intentionally unchanged: it has its own storage-location
      logic for CI and Docker, which already stores telemetry configuration in
      `.next/cache` there.
      
      ### Verification
      
      - Ran a focused probe against the updated source with Docker detection
      mocked. It confirmed RSC encryption-key cache write and reuse,
      expired-key handling for runtime and builds, explicit encryption-key
      precedence, and preview-key cache write and reuse.
      - `pnpm --filter=next types`
      - Not run: full Next.js integration tests.
      
      <!-- NEXT_JS_LLM -->
      
      
      <!-- fleet 3a3e1bf2-4e32-499c-8f71-ae9dd2d64f16 -->
      Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
      Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
      d054fe4a
    • Tobias Koppers's avatar
    • Tobias Koppers's avatar
      ci: run changed deploy tests with both builders (#100591) · 324b2972
      Tobias Koppers authored
      ### What?
      
      Run new and changed deploy tests in PR CI for both deploy builders, and
      fold the separate cache-components job into the same matrix:
      
      - `builder`: `adapter` (`NEXT_ENABLE_ADAPTER=1`, current behavior) and
      `non-adapter` (`NEXT_ENABLE_ADAPTER=0`)
      - `cache-components`: `false` / `true` (replaces the separate
      `test-new-tests-deploy-cache-components` job)
      
      `test-new-tests-deploy` now runs group (5) × builder (2) × cache
      components (2) = 20 entries. Per entry, the env and command are the same
      as before:
      
      - cache components off sets
      `__NEXT_EXPERIMENTAL_STRICT_ROUTE_TYPES=true`
      - cache components on sets `__NEXT_CACHE_COMPONENTS`,
      `__NEXT_EXPERIMENTAL_CACHED_NAVIGATIONS` and the combined
      `NEXT_EXTERNAL_TESTS_FILTERS`
      
      Adapter entries keep their existing `stepName`s, so their test-report
      and Datadog history continues. Non-adapter entries get a `-non-adapter`
      suffix. Job names include the variant, builder and group.
      
      ### Why?
      
      The release deploy tests run both builders, but PR CI only checked new
      and changed deploy tests against the adapter. Changes that behave
      differently with the non-adapter builder were only caught after release.
      A recent example is a `resume-data-cache.partial-prefetching` failure
      that only showed up in the non-adapter Turbopack release shards.
      
      ### How?
      
      - Parsed the workflow and expanded the matrix to 20 entries. Each
      rendered `afterBuild` script was run in an empty environment with `node`
      stubbed out. Every entry exports the same env and runs the same command,
      `stepName` and display name as the matching entry from the previous two
      jobs. The rest of the workflow is unchanged.
      - `prettier --check .github/workflows/build_and_test.yml`
      
      <!-- NEXT_JS_LLM -->
      
      
      <!-- fleet 115592cd-38e9-4ec3-904b-f1c778e1e1d9 -->
      
      ---------
      Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
      Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
      324b2972
    • nt's avatar
      Persist fully static PPR payloads in filesystem cache (#100759) · 4a31419c
      nt authored
      Problem: Complete PPR pages were saved without the .rsc file that
      disk-cache reads require. After memory eviction or restart, those reads
      missed and pages rendered again.
      Change: Save the complete RSC payload for non-postponed PPR pages.
      Continue excluding fallbacks and postponed pages.
      Regression evidence: Without the fix, the complete-PPR unit test fails;
      the other 21 pass. With the fix, all 22 pass.
      HTTP validation: Disk-only repeat requests and cache persistence across
      restart pass under both Turbopack and Webpack.
      
      Fixes #100670
      Fixes #98306
      Closes #98307
      Co-authored-by: 's avatarShtokarev Yuriy <shtokarev@gmail.com>
      4a31419c
    • Tobias Koppers's avatar
      Turbopack: compute client reference manifest parts once per reference, not per page (#100443) · c15cede5
      Tobias Koppers authored
      ## Summary
      
      When Turbopack builds an app's client reference manifests, it builds one
      per app entry (page). For each client reference a page uses,
      `build_manifest` recomputed:
      
      - the reference's server path;
      - its client, SSR and RSC module ids (each read goes through the
      chunking context's id strategy, the module's ident and a keyed module-id
      map read);
      - for every JS chunk, its component chunks (the merged chunk's runtime
      info plus each component chunk's path and file size).
      
      None of these depend on the page, and pages usually share most of their
      client references. So the number of reads grew as pages × references,
      and with `experimental.turbopackChunking.generateComponentChunks` as
      pages × references × chunks × component chunks. Every read was also
      another dependency edge on each page's `build_manifest` task.
      
      This PR moves that work into two turbo-tasks functions that are keyed
      only on page-independent inputs:
      
      - `client_reference_ids`: keyed on the client reference module, the
      client chunking context and the optional SSR chunking context. It
      returns the server path and the client, SSR and RSC module ids.
      - `client_chunk_components`: keyed on the chunk and the client-relative
      path. It returns the chunk's component chunk paths and sizes.
      
      Each page's manifest now does one cached read per reference and one per
      chunk. Values and insertion order are unchanged, so the generated
      manifests are the same. After an edit, only the tasks whose inputs
      changed rerun.
      
      ## Verification
      
      - Compared every `*_client-reference-manifest.js` from full `next build`
      runs with and without this change. All were byte-identical:
      - a synthetic 300-page app with 40 client references per page, default
      config: 302 files;
      - a 1000-page app with 80 references per page and `turbopackChunking: {
      generateComponentChunks: true, minComponentChunkSize: 0 }`: 1002 files,
      including merged-chunk entries.
      - `next dev`: after editing a client component, the edit shows up in the
      SSR output and the page's client reference manifest still lists the
      component.
      - `cargo clippy -p next-core --all-targets -- -D warnings`, `cargo fmt
      --check`
      - Trace counts on the 1000-page app confirm the tasks are shared: 1002
      manifests ran `client_reference_ids` 408 times and
      `client_chunk_components` 402 times.
      - Rough local timings (one 8-core Linux machine, non-LTO release
      bindings, `next build --experimental-build-mode=compile`; exploratory,
      not a validated benchmark):
      - **Default config:** no measurable end-to-end difference; traced
      `build_manifest` time drops about 15%.
      - **Component chunks forced on (`minComponentChunkSize: 0`):** traced
      `build_manifest` time drops from about 63 s to about 6–8 s. Build wall
      time drops from about 53.8 s to about 48.5 s (median of 8 interleaved
      runs).
      - Not run locally: Edge runtime, persistent-cache rebuilds and the e2e
      suites (left to CI).
      
      <!-- NEXT_JS_LLM -->
      
      
      <!-- fleet 27c4bd4e-c7b6-4b23-85a1-32cf20f0a948 -->
      
      ---------
      Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
      Co-authored-by: 's avatarejc3 <ejc3@users.noreply.github.com>
      Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
      Co-authored-by: 's avatarFleet Dev Agent <fleet-dev-agent@users.noreply.github.com>
      c15cede5
    • Batrić Perović's avatar
      Stamp FileSystemCache revalidations and entries with the monotonic clock (#100357) · 3a4477cb
      Batrić Perović authored
      ### What?
      `FileSystemCache` dates tag revalidations and cache entries with the
      wall clock, but since #94457 `areTagsExpired` compares them against the
      monotonic clock (`performance.timeOrigin + performance.now()`).
      When the wall clock runs ahead, a revalidation is dated in the future,
      so it's ignored and the old page keeps being served.
      
      ### Why?
      I ran into this with `next start`, when our E2E tests kept getting stale
      pages after `revalidateTag`. In a fresh 16.4.2 app with the wall clock
      5ms ahead, 30 out of 30 reads right after a revalidation returned the
      old page. With this change, it's 0 of 30.
      
      The two clocks drift apart in ordinary setups, for example when NTP, a
      VM or WSL2 corrects the wall clock while the server runs.
      
      ### How?
      Use the monotonic clock everywhere that `FileSystemCache` dates
      something, as `incremental-cache/index.ts` already does:
      
      - `revalidateTag` and `set()` stamp with `performance.timeOrigin +
      performance.now()` instead of `Date.now()`.
      - `set()` also stores that timestamp on disk, as
      `routeCacheLastModified` in the `.meta` file and as `lastModified` in
      the fetch cache entry. Reads used to take the file's mtime(wall-clock
      time).
      Files without a stored time, from `next build` or an older Next.js
      version, still fall back to the mtime. They were written before the
      server started, so their mtime is earlier than any revalidation it
      records.
      This matters when entries are read from disk: with `cacheMaxMemorySize:
      0`, or after an eviction from the memory cache. With only the in-memory
      stamps fixed, a page written just before a revalidation comes back with
      an mtime that's ahead, so it isn't even marked stale. Running `next
      start` with the memory cache off and the wall clock stepped 1s ahead, 20
      of 40 revalidations left the old page in place that way. With both
      changes, it's 0 of 40.
      
      Notes:
      - `routeCacheLastModified` was only written when a build seed was
      promoted. Runtime writes set it now, so the "preserves seed age" test
      checks for it instead of its absence.
      - Fetch cache files are kept across deploys, and they gain a
      `lastModified` field.
      - `test/production/app-dir/file-system-cache-clock` offsets
      `performance.timeOrigin` at startup and checks the first request after
      `revalidateTag` for an ISR page, an ISR route handler and a dynamic
      page, with the default memory cache and `cacheMaxMemorySize: 0`. It also
      restarts on a fetch cache file without `lastModified`. It fails on
      canary, and with only the in-memory fix the `cacheMaxMemorySize: 0`
      cases fail.
      - Ran the new test with `test-start-webpack` and `test-start-turbo`, and
      four existing revalidation and ISR suites with `test-start-webpack`. Not
      run in dev.
      3a4477cb
    • Tobias Koppers's avatar
      turbo-bitset: remove FixedBitSet and RoaringBitmap comparisons (#100724) · 94e2c32d
      Tobias Koppers authored
      ### What?
      Removes the `FixedBitSet` and `RoaringBitmap` comparisons from the
      `turbo-bitset` benchmarks and the `sizes` example.
      
      - Every benchmark group now measures `Bitset` only. The persistence-path
      `serialize` / `deserialize` and `hash_encode` benchmarks are unchanged.
      - `sizes` reports `Bitset`'s in-memory and serialized sizes only.
      - The `fixedbitset` and `roaring` dev-dependencies are removed from
      `turbo-bitset`. `roaring` has no other user, so it is also removed from
      the workspace and `Cargo.lock`. `fixedbitset` stays in the workspace for
      its other users.
      - The benchmark validation now checks `Bitset` against `BTreeSet`
      reference sets, outside the timed code.
      
      ### Why?
      The comparisons were for choosing the data structure. The design-choice
      data stays in #100566. Keeping them in the regular benchmark suite
      triples its runtime and keeps two dev-only dependencies around.
      
      ### How?
      `Bitset`'s implementation is unchanged, so there are no new benchmark
      numbers. The benchmarks still use the same inputs and persistence writer
      and reader.
      
      Stacked on #100630.
      
      
      <!-- fleet b56269ed-2145-4f30-b0fe-dc9762520521 -->
      Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
      Co-authored-by: 's avatarLuke Sandberg <210140+lukesandberg@users.noreply.github.com>
      94e2c32d
    • Tobias Koppers's avatar
      turbopack: use Bitset for persisted chunk group sets (#100630) · bba6e762
      Tobias Koppers authored
      Stacked on #100568, the top of the `Bitset` stack: #100566 (crate) →
      #100643 → #100567 → #100568 → this.
      
      ## Summary
      
      The PRs below move the module-graph traversals onto `Bitset`, but they
      still convert to `RoaringBitmapWrapper` at the boundary, and persisted
      values and production chunking stay Roaring. This PR finishes that,
      using the bincode encoding and set operations that #100566 adds:
      
      - **turbo-tasks** depends on `turbo-bitset` and implements `TaskInput`,
      `NonLocalValue` and `ValueDebugFormat` for `Bitset`, next to its other
      foreign-type impls.
      - **turbopack-core** drops `RoaringBitmapWrapper` and the `roaring`
      dependency:
      - `ModuleToChunkGroups`, `ChunkingHeuristicsInfo::priority_routes`, the
      module and chunk-item batch types, and production chunking use `Bitset`
      directly.
      - So do the local sets: `inherits_from`, cluster propagation,
      `cluster_groups` and `pairs_with`.
      - `Bitset::to_roaring()` (and its tests and benchmark) is removed, and
      `roaring` becomes a dev-dependency of `turbo-bitset` (for the benchmarks
      and the `sizes` example).
      - `next-api` drops its unused `roaring` dependency, which the
      workspace's unused-dependency lint reports once `turbopack-core` no
      longer uses it.
      
      **Cache compatibility:** the persisted encoding and value hashes of
      these types change (task inputs are hashed with fixed-width integers,
      for which the `Bitset` encoding always uses the bitmap form). The
      persistent cache is versioned by `git describe`
      (`turbo-tasks-backend/src/database/db_versioning.rs`), so normal builds
      won't read old entries. With `TURBO_ENGINE_VERSION`,
      `TURBO_ENGINE_DISABLE_VERSIONING` or `TURBO_ENGINE_IGNORE_DIRTY` in a
      dirty checkout, clear the cache when switching between binaries with and
      without this change.
      
      ## Results
      
      > **Historical, not rerun:** measured in the previous review round,
      before this stack was rebased onto canary `502b6b7d` and before the
      latest review changes on `b149cf8a` (#100566), `c7ea9063` (#100643),
      `e322a1c1` (#100567), `6536eeea` (#100568) and `417a41fd` (#100630).
      This round only reran the `turbo-bitset` microbenchmarks in #100566.
      
      Base is #100568's head, so the encoding of #100566 is the same on both
      sides; only what this PR persists as `Bitset` instead of Roaring
      differs.
      
      ### Full builds
      
      Full `next build` (including prerendering) with uninstrumented
      `next-swc` built with the repository's `release` profile (thin LTO), on
      a shared VM with 8 vCPUs and 16 GB. 5 randomized, interleaved runs per
      variant and app, median with min–max. Each run:
      
      1. **cold**: removes `.next` and builds;
      2. **cache**: measures `.next/cache` (the Turbopack build cache) with
      `du -sb`;
      3. **warm**: removes everything in `.next` except that variant's own
      cache and builds again.
      
      Caches are never shared between variants. Both binaries were built from
      a checkout with uncommitted changes, so every build ran with
      `TURBO_ENGINE_IGNORE_DIRTY=1` to keep the filesystem cache enabled (each
      build logs that it is).
      
      | full `next build` | base (#100568) | this PR |
      |---|---|---|
      | `bench/nested-deps-app-router-many-pages` (3,000 routes): cold build |
      53.1 s (51.7–54.1) | 52.2 s (51–56.5) |
      | `bench/nested-deps-app-router-many-pages` (3,000 routes):
      `.next/cache` size | 762.95 MB (762.85–763.19) | 761.49 MB
      (761.24–761.84) |
      | `bench/nested-deps-app-router-many-pages` (3,000 routes): warm build
      (own cache) | 35.8 s (34.8–36.7) | 35.5 s (34.8–36.6) |
      | generated 300-route app: cold build | 11.3 s (10.8–11.7) | 11.1 s
      (10.9–11.7) |
      | generated 300-route app: `.next/cache` size | 213.84 MB
      (213.45–214.45) | 213.85 MB (213.76–213.94) |
      | generated 300-route app: warm build (own cache) | 5.42 s (5.14–5.5) |
      5.46 s (5.18–5.57) |
      
      All 40 builds succeeded with the same routes per app. The cache is
      reused: warm builds take 48–68% of the cold time for both variants and
      rewrite part of the cache. On `many-pages` the cache is 1.46 MB (0.19%)
      smaller with non-overlapping ranges; on the generated app the difference
      is within the run-to-run variation (±0.5 MB). Cold and warm times
      overlap, so this makes no end-to-end speed claim.
      
      <details>
      <summary>Charts: full builds (linear)</summary>
      
      Median wall time in seconds. Ranges overlap; exact values are in the
      table above.
      
      ```mermaid
      xychart-beta
          title "bench/nested-deps-app-router-many-pages: full build, median s"
          x-axis ["cold", "warm"]
          y-axis "s"
          bar "base (#100568)" [53.1, 35.8]
          bar "this PR" [52.2, 35.5]
      ```
      ```mermaid
      xychart-beta
          title "generated 300-route app: full build, median s"
          x-axis ["cold", "warm"]
          y-axis "s"
          bar "base (#100568)" [11.3, 5.42]
          bar "this PR" [11.1, 5.46]
      ```
      
      </details>
      
      ### Encoded sizes
      
      Logical set payloads from local-only size probes (bytes of the encoded
      values, not the cache database, so keys, metadata and compression aren't
      included), same compile-mode builds as below. They are identical in
      every run:
      
      | logical payload | base (#100568) | this PR |
      |---|---|---|
      | `bench/nested-deps-app-router-many-pages` (3,000 routes): encoded
      `ModuleToChunkGroups` values (33,779 sets) | 40,212,209 B | **9,711,974
      B** (-76%) |
      | `bench/nested-deps-app-router-many-pages` (3,000 routes): encoded
      `ModuleBatch` chunk groups (4,132 sets) | 721,522 B | **97,208 B**
      (-87%) |
      | `bench/nested-deps-app-router-many-pages` (3,000 routes): encoded
      `ModuleBatchGroup` chunk groups (3,021 sets) | 138,211 B | **27,960 B**
      (-80%) |
      | `bench/nested-deps-app-router-many-pages` (3,000 routes): per-module
      set p50 / p99 / max | 19 / 6,023 / 8,211 B | **5 / 1,006 / 2,501 B** |
      | generated 300-route app: encoded `ModuleToChunkGroups` values (12,240
      sets) | 1,971,644 B | **434,660 B** (-78%) |
      | generated 300-route app: encoded `ModuleBatch` chunk groups (2,101
      sets) | 220,751 B | **42,221 B** (-81%) |
      | generated 300-route app: encoded `ModuleBatchGroup` chunk groups (629
      sets) | 30,165 B | **5,578 B** (-82%) |
      | generated 300-route app: per-module set p50 / p99 / max | 19 / 1,241 /
      3,687 B | **3 / 235 / 339 B** |
      
      ### Stage times
      
      `next build --experimental-build-mode compile` with release `next-swc`
      (LTO off, 16 codegen units) built from each PR's head plus local-only
      stage timers. Median of 5 randomized, interleaved runs per variant,
      min–max in parentheses, in ms. All five layers of the stack were
      measured in the same interleaved set.
      
      | stage (ms) | previous PR | this PR |
      |---|---|---|
      | `bench/nested-deps-app-router-many-pages` (3,000 routes): heuristics +
      building the result | 37.7 (37.1–39.4) | **5.17 (4.66–5.39)** |
      | `bench/nested-deps-app-router-many-pages` (3,000 routes): production
      chunk merging (summed) | 317 (313–326) | **249 (240–253)** |
      | generated 300-route app: heuristics + building the result | 3.71
      (3.57–3.78) | **0.62 (0.574–0.657)** |
      | generated 300-route app: production chunk merging (summed) | 19.3
      (18.7–20.5) | **14.2 (13.6–14.8)** |
      
      <details>
      <summary>Charts: stage times (linear)</summary>
      
      Median ms per stage; the first series is the previous PR. Exact values
      and ranges are in the table above.
      
      ```mermaid
      xychart-beta
          title "bench/nested-deps-app-router-many-pages, median ms"
          x-axis ["heuristics + building the result", "production chunk merging (summed)"]
          y-axis "ms"
          bar "previous PR" [37.7, 317]
          bar "this PR" [5.17, 249]
      ```
      ```mermaid
      xychart-beta
          title "generated 300-route app, median ms"
          x-axis ["heuristics + building the result", "production chunk merging (summed)"]
          y-axis "ms"
          bar "previous PR" [3.71, 19.3]
          bar "this PR" [0.62, 14.2]
      ```
      
      </details>
      
      Sum of the measured module-graph stages per run (chunk group info and
      building its result, collect, merged modules and interning), whole
      stack:
      
      | sum of module-graph stages (ms) | #100566 (before the usage PRs) | top
      of stack (#100630) |
      |---|---|---|
      | `bench/nested-deps-app-router-many-pages` (3,000 routes) | 750
      (746–767) | **102 (99.7–113)** |
      | generated 300-route app | 36.3 (35.1–38) | **17.8 (17.5–18.1)** |
      
      <details>
      <summary>Charts: stage sum per layer (linear)</summary>
      
      Median ms of the summed module-graph stages, one bar per layer of the
      stack. Exact values for the ends are in the table above.
      
      ```mermaid
      xychart-beta
          title "bench/nested-deps-app-router-many-pages: stage sum per layer, median ms"
          x-axis ["#100566", "#100643", "#100567", "#100568", "#100630"]
          y-axis "ms"
          bar "stage sum" [750, 455, 288, 128, 102]
      ```
      ```mermaid
      xychart-beta
          title "generated 300-route app: stage sum per layer, median ms"
          x-axis ["#100566", "#100643", "#100567", "#100568", "#100630"]
          y-axis "ms"
          bar "stage sum" [36.3, 28.8, 27.1, 21, 17.8]
      ```
      
      </details>
      
      Visit counts and the emitted output are identical across all layers on
      both apps (14,028 and 2,142 files, same digests).
      
      ## Verification
      
      - `cargo test -p turbo-bitset`, `-p turbo-tasks --lib`, `-p
      turbopack-core --lib`, the turbopack snapshot tests (no snapshot
      changes) and execution tests.
      - `cargo fmt --check`, `cargo clippy` (touched crates plus `next-api`,
      `next-core`, all targets) and `cargo doc` with `-D warnings` (no
      unused-dependency warning).
      - `cargo tree -p turbopack-core -e normal` no longer contains `roaring`.
      - The builds above, with timers and probes kept local only.
      
      <!-- NEXT_JS_LLM -->
      
      
      <!-- fleet b56269ed-2145-4f30-b0fe-dc9762520521 -->
      
      ---------
      Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
      Co-authored-by: 's avatarLuke Sandberg <210140+lukesandberg@users.noreply.github.com>
      Co-authored-by: 's avatarVercel Fleet <fleet@vercel.com>
      Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
      bba6e762
    • Tobias Koppers's avatar
      turbopack: use Bitset in compute_merged_modules (#100568) · 68438424
      Tobias Koppers authored
      Stacked on #100567 (`collect_graph`). Stack: #100566 → #100643 → #100567
      → this → #100630.
      
      ## Summary
      
      `compute_merged_modules` now uses `Bitset` for its per-module merge
      group sets during the fixed-point traversal, the same change as in the
      PRs below. Like #100643, it drops the first-visit `is_zero()` check plus
      `clone()`, as `union_with` copies into a set that stores no blocks.
      
      The bitmap interning step that follows (which gives each distinct set a
      dense id) also works on `Bitset`. Hashing a `Bitset` hashes its words
      directly; hashing a `RoaringBitmapWrapper` serializes the bitmap. The
      merged module result types are unchanged.
      
      ## Benchmarks
      
      > **Historical, not rerun:** measured in the previous review round,
      before this stack was rebased onto canary `502b6b7d` and before the
      latest review changes on `b149cf8a` (#100566), `c7ea9063` (#100643),
      `e322a1c1` (#100567), `6536eeea` (#100568) and `417a41fd` (#100630).
      This round only reran the `turbo-bitset` microbenchmarks in #100566.
      
      `next build --experimental-build-mode compile` with release `next-swc`
      (LTO off, 16 codegen units) built from each PR's head plus local-only
      stage timers, on a shared VM with 8 vCPUs and 16 GB. Each number is the
      median of 5 randomized, interleaved runs per variant, with the min–max
      range in parentheses, in ms. All five layers of the stack were measured
      in the same interleaved set, so adjacent PRs compare directly.
      
      | stage (ms) | previous PR | this PR |
      |---|---|---|
      | `bench/nested-deps-app-router-many-pages` (3,000 routes):
      merged-modules fixed point | 109 (108–110) | **22.9 (22.4–24.5)** |
      | `bench/nested-deps-app-router-many-pages` (3,000 routes): bitmap
      interning | 75.9 (73.1–81.5) | **6.72 (6.16–6.87)** |
      | generated 300-route app: merged-modules fixed point | 9.14 (8.63–13.7)
      | **5.87 (5.79–6.04)** |
      | generated 300-route app: bitmap interning | 3.59 (3.49–6.33) | **0.672
      (0.631–0.69)** |
      
      Visit counts are identical, and the emitted JS chunks are byte-identical
      across all layers of the stack on both apps. Whole-build time is too
      noisy on these apps to show a reliable end-to-end difference. The sum
      over the whole stack is in #100630.
      
      <details>
      <summary>Charts: stage times (linear)</summary>
      
      Median ms per stage; the first series is the previous PR. Exact values
      and ranges are in the table above.
      
      ```mermaid
      xychart-beta
          title "bench/nested-deps-app-router-many-pages, median ms"
          x-axis ["merged-modules fixed point", "bitmap interning"]
          y-axis "ms"
          bar "previous PR" [109, 75.9]
          bar "this PR" [22.9, 6.72]
      ```
      ```mermaid
      xychart-beta
          title "generated 300-route app, median ms"
          x-axis ["merged-modules fixed point", "bitmap interning"]
          y-axis "ms"
          bar "previous PR" [9.14, 3.59]
          bar "this PR" [5.87, 0.672]
      ```
      
      </details>
      
      ## Verification
      
      - `cargo test -p turbopack-core --lib` (including the interning tests),
      all turbopack snapshot tests and all turbopack execution tests pass.
      - The real builds above.
      
      <!-- NEXT_JS_LLM -->
      
      
      <!-- fleet b56269ed-2145-4f30-b0fe-dc9762520521 -->
      
      ---------
      Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
      Co-authored-by: 's avatarLuke Sandberg <210140+lukesandberg@users.noreply.github.com>
      Co-authored-by: 's avatarVercel Fleet <fleet@vercel.com>
      Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
      68438424
    • Tobias Koppers's avatar
      turbopack: use Bitset in collect_graph (#100567) · 18b5024d
      Tobias Koppers authored
      Stacked on #100643 (`compute_chunk_group_info`), on top of #100566,
      which adds `turbo-bitset`. Stack: #100566 → #100643 → this → #100568 →
      #100630.
      
      ## Summary
      
      `collect_graph` now uses `Bitset` for its per-module entry membership
      sets during the fixed-point traversal, the same change as in
      `compute_chunk_group_info`. It never stores these sets, so nothing needs
      converting back to `RoaringBitmapWrapper`. Like #100643, it drops the
      first-visit `is_zero()` check plus `clone()`, as `union_with` copies
      into a set that stores no blocks.
      
      When matching collected references to entries, the old code built a
      Roaring intersection of the collecting module's and the emitted module's
      memberships. It now walks the collecting module's entries and keeps
      those that the emitted module's membership `contains`. That allocates no
      intermediate set, and the order and the skip-when-empty behavior are
      unchanged.
      
      ## Benchmarks
      
      > **Historical, not rerun:** measured in the previous review round,
      before this stack was rebased onto canary `502b6b7d` and before the
      latest review changes on `b149cf8a` (#100566), `c7ea9063` (#100643),
      `e322a1c1` (#100567), `6536eeea` (#100568) and `417a41fd` (#100630).
      This round only reran the `turbo-bitset` microbenchmarks in #100566.
      
      `next build --experimental-build-mode compile` with release `next-swc`
      (LTO off, 16 codegen units) built from each PR's head plus local-only
      stage timers, on a shared VM with 8 vCPUs and 16 GB. Each number is the
      median of 5 randomized, interleaved runs per variant, with the min–max
      range in parentheses, in ms. All five layers of the stack were measured
      in the same interleaved set, so adjacent PRs compare directly.
      
      | stage (ms) | previous PR | this PR |
      |---|---|---|
      | `bench/nested-deps-app-router-many-pages` (3,000 routes):
      `collect_graph` fixed point | 189 (186–195) | **16 (15–17.1)** |
      | generated 300-route app: `collect_graph` fixed point | 6.38
      (6.32–6.68) | **3.37 (3.33–4.02)** |
      
      Visit counts are identical, and the emitted JS chunks are byte-identical
      across all layers of the stack on both apps. Whole-build time is too
      noisy on these apps to show a reliable end-to-end difference.
      
      <details>
      <summary>Charts: stage times (linear)</summary>
      
      Median ms per stage; the first series is the previous PR. Exact values
      and ranges are in the table above.
      
      ```mermaid
      xychart-beta
          title "bench/nested-deps-app-router-many-pages, median ms"
          x-axis ["`collect_graph` fixed point"]
          y-axis "ms"
          bar "previous PR" [189]
          bar "this PR" [16]
      ```
      ```mermaid
      xychart-beta
          title "generated 300-route app, median ms"
          x-axis ["`collect_graph` fixed point"]
          y-axis "ms"
          bar "previous PR" [6.38]
          bar "this PR" [3.37]
      ```
      
      </details>
      
      ## Verification
      
      - `cargo test -p turbopack-core --lib`, the turbopack snapshot tests and
      the collect execution tests pass. All execution tests pass on the top of
      the stack.
      - The real builds above.
      
      <!-- NEXT_JS_LLM -->
      
      
      <!-- fleet b56269ed-2145-4f30-b0fe-dc9762520521 -->
      
      ---------
      Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
      Co-authored-by: 's avatarLuke Sandberg <210140+lukesandberg@users.noreply.github.com>
      Co-authored-by: 's avatarVercel Fleet <fleet@vercel.com>
      Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
      18b5024d
    • Tobias Koppers's avatar
      turbopack: use Bitset in compute_chunk_group_info (#100643) · d19d3d57
      Tobias Koppers authored
      Stacked on #100566, which adds `turbo-bitset`. Stack: #100566 → this →
      #100567 (`collect_graph`) → #100568 (`compute_merged_modules`) → #100630
      (persisted chunk group sets).
      
      ## Summary
      
      `compute_chunk_group_info` tracked each module's chunk groups with
      `RoaringBitmapWrapper` during its fixed-point traversal. The traversal
      checks "would `current |= parent` change `current`?" on every edge and
      then does the union; on Roaring's array containers both cost time
      proportional to the number of bits, which made this the slowest of the
      module graph passes on `bench/nested-deps-app-router-many-pages`.
      
      It now uses `Bitset` for the working per-module sets, and converts them
      with `to_roaring()` when building the result. `ModuleToChunkGroups`, the
      batch types and production chunking still use `RoaringBitmapWrapper`, so
      persisted values are unchanged (#100630 switches those). The old code
      called `RoaringBitmapWrapper::is_proper_superset` (which actually
      computed `!parent.is_subset(current)`) and then `|=`. Now
      `current.union_with(parent)` does both in one pass and returns whether
      it added a bit. The separate first-visit `is_zero()` check plus
      `clone()` is gone too: `union_with` copies into a set that stores no
      blocks. This doesn't change the traversal, as a parent's set is never
      empty when its children are visited.
      
      ## Benchmarks
      
      > **Historical, not rerun:** measured in the previous review round,
      before this stack was rebased onto canary `502b6b7d` and before the
      latest review changes on `b149cf8a` (#100566), `c7ea9063` (#100643),
      `e322a1c1` (#100567), `6536eeea` (#100568) and `417a41fd` (#100630).
      This round only reran the `turbo-bitset` microbenchmarks in #100566.
      
      `next build --experimental-build-mode compile` with release `next-swc`
      (LTO off, 16 codegen units) built from each PR's head plus local-only
      stage timers, on a shared VM with 8 vCPUs and 16 GB. Each number is the
      median of 5 randomized, interleaved runs per variant, with the min–max
      range in parentheses, in ms. All five layers of the stack were measured
      in the same interleaved set, so adjacent PRs compare directly.
      
      | stage (ms) | previous PR | this PR |
      |---|---|---|
      | `bench/nested-deps-app-router-many-pages` (3,000 routes): chunk-group
      fixed point | 379 (375–384) | **49.9 (45.8–54.2)** |
      | `bench/nested-deps-app-router-many-pages` (3,000 routes): building
      `ChunkGroupInfo` | 3.73 (3.16–4.08) | **37.4 (35.7–42.2)** |
      | generated 300-route app: chunk-group fixed point | 17.3 (16.4–18.1) |
      **7.09 (6.93–7.33)** |
      | generated 300-route app: building `ChunkGroupInfo` | 0.444
      (0.417–0.471) | **3.57 (3.35–3.62)** |
      
      Building `ChunkGroupInfo` now includes the `to_roaring()` conversion,
      which #100630 removes. Visit counts are identical, and the emitted JS
      chunks are byte-identical across all layers of the stack on both apps
      (14,028 and 2,142 files, same digests). Whole-build time is too noisy on
      these apps to show a reliable end-to-end difference. The microbenchmarks
      are in #100566.
      
      <details>
      <summary>Charts: stage times (linear)</summary>
      
      Median ms per stage; the first series is the previous PR. Exact values
      and ranges are in the table above.
      
      ```mermaid
      xychart-beta
          title "bench/nested-deps-app-router-many-pages, median ms"
          x-axis ["chunk-group fixed point", "building `ChunkGroupInfo`"]
          y-axis "ms"
          bar "previous PR" [379, 3.73]
          bar "this PR" [49.9, 37.4]
      ```
      ```mermaid
      xychart-beta
          title "generated 300-route app, median ms"
          x-axis ["chunk-group fixed point", "building `ChunkGroupInfo`"]
          y-axis "ms"
          bar "previous PR" [17.3, 0.444]
          bar "this PR" [7.09, 3.57]
      ```
      
      </details>
      
      ## Verification
      
      - `cargo test -p turbopack-core --lib` and the turbopack snapshot tests
      pass without snapshot changes; the execution tests pass on the top of
      the stack.
      - The real builds above.
      
      <!-- NEXT_JS_LLM -->
      
      
      <!-- fleet b56269ed-2145-4f30-b0fe-dc9762520521 -->
      
      ---------
      Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
      Co-authored-by: 's avatarLuke Sandberg <210140+lukesandberg@users.noreply.github.com>
      Co-authored-by: 's avatarVercel Fleet <fleet@vercel.com>
      Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
      d19d3d57
    • Tobias Koppers's avatar
      turbo-bitset: add Bitset, a fast bitset over a bounded bit range (#100566) · 72f5f5fc
      Tobias Koppers authored
      Stacked on #100452. This PR only adds the crate; the PRs above use it:
      #100643 (`compute_chunk_group_info`) → #100567 (`collect_graph`) →
      #100568 (`compute_merged_modules`) → #100630 (persisted chunk group
      sets) → #100724 (removes the FixedBitSet/Roaring comparisons from the
      benchmarks).
      
      ## Summary
      
      The module graph fixed-point traversals track set membership with
      `RoaringBitmap`, which is built for compact storage, not speed. They
      check "would `current |= parent` change `current`?" on every edge and
      then do the union; on array containers both cost time proportional to
      the number of bits.
      
      This PR adds a new `turbo-bitset` crate with `Bitset`: a `Vec<usize>` of
      blocks plus an offset counted in whole blocks.
      
      - Bit `i` is stored at `i - offset * usize::BITS`, so a set only stores
      the word range between its lowest and highest bit. Sets whose bits are
      clustered at high indices stay small.
      - Unions, intersections, differences, subset checks, `is_disjoint` and
      `intersection_len` are word loops over the overlapping range, so their
      cost is proportional to the range, not the number of bits. The subset
      check reduces chunks of 4 words before branching, so it vectorizes.
      - `union_with` returns whether it added a bit. It ORs the words and
      collects the added bits in the same loop, so a traversal checks for a
      change and applies the union in one pass instead of a subset check plus
      `|=`. If the set stores no blocks (a module's first visit), it copies
      `other` without its zero edge blocks, reusing a reservation or
      allocating exactly the copied blocks, so the traversals need no separate
      `is_zero()` check and `clone()`.
      - `contains`, `insert` and `remove` check the stored range with a single
      comparison. Growth is amortized at both ends: growing at the back uses
      the `Vec`'s growth, and growing at the front moves the blocks once to
      the end of the (reused or doubled) capacity, which leaves headroom for
      further front growth. So inserting in any order, including descending,
      moves each block O(log n) times. `extend` inserts the items directly.
      - `Bitset::with_len(bits: u32)` stores exactly the zeroed blocks for the
      bits `0..bits` at offset 0, in an allocation without spare capacity, so
      inserting any of them doesn't grow the storage.
      `Bitset::with_capacity(bits: u32)` reserves capacity for that many bits
      without storing any block; the first insert still picks the offset.
      - Equality and hashing depend only on the bits, not on the stored range.
      `count_ones()` counts over the stored range (the name says it isn't
      free), `is_zero()` checks for no bits. `Bitset` implements
      `ShrinkToFit`.
      - The storage is a plain `Vec`. A `SmallVec<[usize; INLINE]>` variant
      was tried and reverted: it adds an inline-or-heap branch to every block
      access, and no user needs inline blocks. Against it, `Vec` measured
      0.967× overall in the microbenchmarks; unions right after a clone are
      slower, because `Vec::clone` allocates exactly the stored blocks
      (details in the review thread). Clones keep the default exact capacity,
      to avoid extra memory per module set in the traversals.
      - `to_roaring()` converts back container by container (2^16 bits at a
      time) straight from the words, with the same container types (and
      serialization) as inserting the bits one by one. The PRs above use it at
      the boundary until #100630 removes it.
      - No `#[inline]` hints: the crate relies on thin LTO. Removing the 8
      hints it had (one went away with the hand-written varint writer) changed
      41 of 480 benchmark cases by ≥ 5% consistently (24 faster, 17 slower;
      geometric mean 0.99×), and restoring the decoder's most likely candidate
      made 3 cases faster and 3 slower, so none is kept.
      
      ### Bincode encoding
      
      `Encode`/`Decode` (in `src/encode.rs`) start with a signed length
      header:
      
      - `0`: the empty set.
      - `len > 0`, the **bitmap form**: a `u32` byte offset, then `len` raw
      bytes, with leading and trailing zero bytes trimmed.
      - `len = -n < 0`, the **list form**: the payload length `len` the bitmap
      form would have, then the first bit and `n - 1` differences (each `≥
      1`), all `u32` (varints under the standard config). The bits must cover
      exactly that payload range.
      
      Which form is written:
      
      - With varint integers (how Turbopack persists), the list form is used
      iff `5 · n < 4 · len`, for `n` bits and a bitmap payload of `len` bytes.
      That only needs the bit count (counting stops once it reaches the
      threshold), not the sizes of the list items.
      - If the bitmap is chosen, its payload is at most 1.25× the list's
      items, as every item takes at least one byte.
      - If the list is chosen, the bits are more than 10 apart on average, and
      only differences above 250 take more than one byte; their sum is below
      `8 · len`, so the differences take fewer bytes than the bitmap payload.
      - The rule ignores the headers and the first item, so for sets with very
      few bits the chosen form can be a few bytes (at most 13) larger than the
      other.
      - For a set spanning its whole range, the switch is at 10% density. The
      10%-of-16,384-bits benchmark input has exactly 1,638 bits and a
      2,047-byte payload, just at the threshold (5 · 1,638 = 8,190 ≥ 4 · 2,047
      = 8,188), so it is written as a bitmap.
      - With fixed-width integers (how task inputs are hashed), always the
      bitmap form.
      
      Integers are encoded with bincode's own `Encode` impls. The list items
      are encoded with the outer encoder's config into a 1 KB stack buffer
      (bincode's `SliceWriter`), which is written in chunks, a block at a
      time; encoding doesn't allocate. The bitmap bytes are little-endian on
      every platform: on little-endian targets they are written and read with
      a single copy of the storage viewed as bytes (documented `unsafe` slice
      casts), big-endian targets convert each block.
      
      Decoding allocates the storage once (from the bitmap length in both
      forms), claiming it from the decoder's byte limit first. List items are
      set into a register-held block, and runs of single-byte varints are
      handled in batches, still claiming 4 bytes per item like `u32::decode`.
      Malformed input is rejected with a `DecodeError`: an out-of-range header
      or payload, a bitmap or list past the `u32` range, a zero or overflowing
      difference, a list item outside its payload range or a list that doesn't
      end in its last byte, and truncated input.
      
      ## Benchmarks
      
      All numbers in this section are criterion point estimates from `cargo
      bench -p turbo-bitset` (workspace release settings, thin LTO,
      `RUSTFLAGS="-C llvm-args=-x86-branches-within-32B-boundaries"`, rustc
      1.101.0-nightly 2026-10-05; bincode 2.0.1, fixedbitset 0.5.7, roaring
      0.11.5, codspeed-criterion-compat 4.3.0), on this PR's `Vec`-backed code
      before the sizing constructors and `union_with -> bool` were added (tree
      of `d7d69b85`), on a shared VM with 8 vCPUs and 16 GB.
      
      ### Microbenchmarks
      
      Each operation compares `Bitset` with a plain `FixedBitSet` (indexed
      from bit 0) and `RoaringBitmap` 0.11.5. Inputs are `empty`, `1`, `4` and
      `16` random bits, plus densities that set at least 32 bits (10%, 50%,
      100% of 1,024 bits; 1%, 10%, 50%, 100% of 16,384 bits; 0.1%, 1%, 10%,
      50%, 100% of 65,536 bits). Every input has exactly its count, or exactly
      ⌊universe · density⌋ distinct bits, picked by selection sampling from a
      seeded `SmallRng`, starting at bit 0. The `sizes` example uses the same
      inputs (`benches/common/inputs.rs`). Results are checked for agreement
      outside the measurement. Cells are Bitset / FixedBitSet / RoaringBitmap
      in ns, 16,384-bit universe:
      
      | operation | 1 bit | 16 bits | 1% | 10% | 100% |
      |---|---|---|---|---|---|
      | union (`a \|= b`) | 52 / 9.8 / 24 | 178 / 81 / 92 | 75 / 79 / 600 | 78
      / 78 / 5,633 | 101 / 84 / 834 |
      | superset, holds | 2.3 / 3.1 / 3.8 | 58 / 90 / 32 | 63 / 89 / 263 | 62
      / 89 / 2,728 | 72 / 89 / 384 |
      | intersection | 2.9 / 19 / 23 | 102 / 106 / 52 | 90 / 102 / 350 | 88 /
      101 / 3,383 | 86 / 102 / 918 |
      | contains (1,024 lookups) | 774 / 593 / 3,234 | 731 / 768 / 4,309 | 765
      / 749 / 9,437 | 750 / 748 / 12,535 | 766 / 747 / 2,474 |
      | insert (random order) | 19 / 24 / 35 | 249 / 119 / 255 | 387 / 261 /
      3,709 | 2,225 / 1,695 / 69,182 | 19,087 / 14,452 / 328,090 |
      | iter | 2.3 / 57 / 8.0 | 120 / 157 / 52 | 282 / 256 / 389 | 1,337 /
      1,098 / 3,832 | 15,188 / 13,820 / 44,408 |
      
      <details>
      <summary>Charts: microbenchmarks (log scale)</summary>
      
      log10 of the time in ns (1 = 10 ns, 3 = 1 µs), 16,384-bit universe.
      Exact values are in the table above.
      
      ```mermaid
      xychart-beta
          title "union (a |= b)"
          x-axis ["1 bit", "16 bits", "1%", "10%", "100%"]
          y-axis "log10(ns)"
          line "Bitset" [1.72, 2.25, 1.88, 1.89, 2.0]
          line "FixedBitSet" [0.99, 1.91, 1.9, 1.89, 1.92]
          line "RoaringBitmap" [1.38, 1.97, 2.78, 3.75, 2.92]
      ```
      ```mermaid
      xychart-beta
          title "superset, holds"
          x-axis ["1 bit", "16 bits", "1%", "10%", "100%"]
          y-axis "log10(ns)"
          line "Bitset" [0.37, 1.77, 1.8, 1.79, 1.86]
          line "FixedBitSet" [0.49, 1.95, 1.95, 1.95, 1.95]
          line "RoaringBitmap" [0.58, 1.51, 2.42, 3.44, 2.58]
      ```
      ```mermaid
      xychart-beta
          title "intersection"
          x-axis ["1 bit", "16 bits", "1%", "10%", "100%"]
          y-axis "log10(ns)"
          line "Bitset" [0.46, 2.01, 1.96, 1.94, 1.93]
          line "FixedBitSet" [1.27, 2.03, 2.01, 2.0, 2.01]
          line "RoaringBitmap" [1.36, 1.72, 2.54, 3.53, 2.96]
      ```
      ```mermaid
      xychart-beta
          title "contains (1,024 lookups)"
          x-axis ["1 bit", "16 bits", "1%", "10%", "100%"]
          y-axis "log10(ns)"
          line "Bitset" [2.89, 2.86, 2.88, 2.88, 2.88]
          line "FixedBitSet" [2.77, 2.89, 2.87, 2.87, 2.87]
          line "RoaringBitmap" [3.51, 3.63, 3.97, 4.1, 3.39]
      ```
      ```mermaid
      xychart-beta
          title "insert (random order)"
          x-axis ["1 bit", "16 bits", "1%", "10%", "100%"]
          y-axis "log10(ns)"
          line "Bitset" [1.29, 2.4, 2.59, 3.35, 4.28]
          line "FixedBitSet" [1.39, 2.08, 2.42, 3.23, 4.16]
          line "RoaringBitmap" [1.54, 2.41, 3.57, 4.84, 5.52]
      ```
      ```mermaid
      xychart-beta
          title "iter"
          x-axis ["1 bit", "16 bits", "1%", "10%", "100%"]
          y-axis "log10(ns)"
          line "Bitset" [0.36, 2.08, 2.45, 3.13, 4.18]
          line "FixedBitSet" [1.76, 2.2, 2.41, 3.04, 4.14]
          line "RoaringBitmap" [0.91, 1.72, 2.59, 3.58, 4.65]
      ```
      
      </details>
      
      ### Ordered inserts
      
      Building a set by inserting its bits one by one in a fixed random order,
      ascending (the set only grows at the back) and descending (it only grows
      at the front). FixedBitSet uses `grow_and_insert`. Bitset / FixedBitSet
      / RoaringBitmap in ns, 16,384-bit universe:
      
      | order | 1 bit | 16 bits | 1% | 10% | 100% |
      |---|---|---|---|---|---|
      | random order | 19 / 24 / 35 | 249 / 119 / 255 | 387 / 261 / 3,709 |
      2,225 / 1,695 / 69,182 | 19,087 / 14,452 / 328,090 |
      | ascending | 20 / 25 / 35 | 337 / 362 / 241 | 971 / 1,038 / 2,879 |
      3,385 / 7,760 / 29,593 | 31,185 / 71,557 / 160,400 |
      | descending | 19 / 25 / 36 | 209 / 74 / 286 | 417 / 201 / 4,570 | 2,431
      / 1,666 / 82,091 | 29,920 / 27,721 / 393,320 |
      
      Before/after A/B of this round's growth change (front growth with
      headroom instead of an exact shift, and `extend` inserting items
      directly instead of buffering them). Both sides use the new inputs and
      benchmarks; "before" is the previous `ensure_blocks`/`Extend` on the
      rebased code. Two interleaved rounds per side (before, after, after,
      before), mean in ns:
      
      | input | `insert_desc` before → after | `from_iter` before → after |
      |---|---|---|
      | `low-1024-16bits` | 167 → 116 (0.69×) | 70 → 146 (2.08×) |
      | `low-1024-10%` | 377 → 209 (0.55×) | 277 → 292 (1.05×) |
      | `low-16384-16bits` | 406 → 227 (0.56×) | 108 → 296 (2.76×) |
      | `low-16384-1%` | 2,497 → 413 (0.17×) | 424 → 1,046 (2.47×) |
      | `low-16384-10%` | 7,285 → 2,378 (0.33×) | 3,889 → 3,392 (0.87×) |
      | `low-16384-100%` | 35,372 → 30,340 (0.86×) | 47,816 → 29,290 (0.61×) |
      | `low-65536-0.1%` | 3,036 → 774 (0.25×) | 250 → 861 (3.45×) |
      | `low-65536-1%` | 23,598 → 1,235 (0.05×) | 1,597 → 3,178 (1.99×) |
      | `low-65536-10%` | 57,686 → 11,312 (0.20×) | 15,433 → 12,488 (0.81×) |
      | `low-65536-100%` | 171,340 → 132,520 (0.77×) | 191,165 → 115,290
      (0.60×) |
      
      Descending inserts used to move every stored block on each new lowest
      block, which is quadratic; they are now up to 19× faster. `from_iter` is
      13–40% faster for 10% to 100% of 16,384 and 65,536 bits, but **1.8–3.5×
      slower for the sparse inputs** (4 and 16 bits, 0.1% and 1%; e.g. 0.1% of
      65,536 bits: 250 → 861 ns). The removed buffered `extend` computed the
      whole range first and allocated once; inserting directly grows the
      storage several times.
      
      <details>
      <summary>Chart: descending inserts before vs after (log scale)</summary>
      
      log10 of the time in ns, mean of the two rounds per side. Exact values
      are in the table above.
      
      ```mermaid
      xychart-beta
          title "insert_desc"
          x-axis ["1024-16bits", "1024-10%", "16384-16bits", "16384-1%", "16384-10%", "16384-100%", "65536-0.1%", "65536-1%", "65536-10%", "65536-100%"]
          y-axis "log10(ns)"
          bar "before" [2.22, 2.58, 2.61, 3.4, 3.86, 4.55, 3.48, 4.37, 4.76, 5.23]
          bar "after" [2.06, 2.32, 2.36, 2.62, 3.38, 4.48, 2.89, 3.09, 4.05, 5.12]
      ```
      
      </details>
      
      ### Serialization
      
      With the writer and reader Turbopack persists with (`turbo_bincode`:
      `TurboBincodeWriter` into a new `TurboBincodeBuffer`,
      `TurboBincodeReader`, standard config). Roaring goes through serde, as
      `RoaringBitmapWrapper` persists it. Bitset / RoaringBitmap in ns:
      
      | universe | 1 bit | 16 bits | 0.1% | 1% | 10% | 50% | 100% |
      |---|---|---|---|---|---|---|---|
      | 1,024: serialize | 33 / 102 | 83 / 184 | - | - | 62 / 347 | 62 / 904 |
      63 / 1,467 |
      | 16,384: serialize | 41 / 102 | 394 / 177 | - | 704 / 404 | 286 / 2,020
      | 143 / 1,501 | 147 / 1,529 |
      | 65,536: serialize | 42 / 102 | 1,316 / 164 | 1,514 / 255 | 2,726 /
      1,074 | 889 / 1,435 | 278 / 2,136 | 207 / 1,445 |
      | 1,024: deserialize | 57 / 137 | 67 / 153 | - | - | 53 / 208 | 54 / 627
      | 53 / 1,060 |
      | 16,384: deserialize | 47 / 132 | 143 / 132 | - | 332 / 267 | 91 /
      1,579 | 107 / 1,148 | 89 / 1,145 |
      | 65,536: deserialize | 48 / 174 | 193 / 172 | 443 / 187 | 1,078 / 833 |
      150 / 1,172 | 149 / 1,141 | 153 / 1,140 |
      
      For the bitmap-form inputs, `Bitset` is 1.6–23× faster than Roaring to
      serialize and 2.4–20× faster to deserialize. The list-form inputs (4 and
      16 bits, 0.1% and 1%) depend on the width of the stored range: on 1,024
      bits `Bitset` is 1.7–2.3× faster both ways, but on 16,384 and 65,536
      bits it is up to 11× slower to serialize (4 bits in 65,536) and up to
      2.4× slower to deserialize (0.1% of 65,536) than Roaring's sorted
      arrays. Both directions walk the stored word range, not just the set
      bits.
      
      <details>
      <summary>Charts: serialization (log scale)</summary>
      
      log10 of the time in ns, `TurboBincodeWriter`/`TurboBincodeReader`.
      Points that set fewer than 32 bits are left out. Exact values are in the
      table above.
      
      ```mermaid
      xychart-beta
          title "1,024: serialize"
          x-axis ["1 bit", "16 bits", "10%", "50%", "100%"]
          y-axis "log10(ns)"
          line "Bitset" [1.52, 1.92, 1.79, 1.79, 1.8]
          line "RoaringBitmap" [2.01, 2.26, 2.54, 2.96, 3.17]
      ```
      ```mermaid
      xychart-beta
          title "16,384: serialize"
          x-axis ["1 bit", "16 bits", "1%", "10%", "50%", "100%"]
          y-axis "log10(ns)"
          line "Bitset" [1.62, 2.6, 2.85, 2.46, 2.16, 2.17]
          line "RoaringBitmap" [2.01, 2.25, 2.61, 3.31, 3.18, 3.18]
      ```
      ```mermaid
      xychart-beta
          title "65,536: serialize"
          x-axis ["1 bit", "16 bits", "0.1%", "1%", "10%", "50%", "100%"]
          y-axis "log10(ns)"
          line "Bitset" [1.62, 3.12, 3.18, 3.44, 2.95, 2.44, 2.32]
          line "RoaringBitmap" [2.01, 2.21, 2.41, 3.03, 3.16, 3.33, 3.16]
      ```
      ```mermaid
      xychart-beta
          title "1,024: deserialize"
          x-axis ["1 bit", "16 bits", "10%", "50%", "100%"]
          y-axis "log10(ns)"
          line "Bitset" [1.75, 1.83, 1.73, 1.73, 1.72]
          line "RoaringBitmap" [2.14, 2.18, 2.32, 2.8, 3.03]
      ```
      ```mermaid
      xychart-beta
          title "16,384: deserialize"
          x-axis ["1 bit", "16 bits", "1%", "10%", "50%", "100%"]
          y-axis "log10(ns)"
          line "Bitset" [1.67, 2.16, 2.52, 1.96, 2.03, 1.95]
          line "RoaringBitmap" [2.12, 2.12, 2.43, 3.2, 3.06, 3.06]
      ```
      ```mermaid
      xychart-beta
          title "65,536: deserialize"
          x-axis ["1 bit", "16 bits", "0.1%", "1%", "10%", "50%", "100%"]
          y-axis "log10(ns)"
          line "Bitset" [1.68, 2.29, 2.65, 3.03, 2.18, 2.17, 2.19]
          line "RoaringBitmap" [2.24, 2.24, 2.27, 2.92, 3.07, 3.06, 3.06]
      ```
      
      </details>
      
      Before/after A/B of removing the decoder's batch copy (the bytes are now
      read straight from `peek_read`). Both builds differ only in
      `decode_list`; same scheme as above:
      
      | input (form) | `deserialize` before → after |
      |---|---|
      | `low-1024-16bits` (list) | 68 → 60 (0.88×) |
      | `low-16384-4bits` (list) | 95 → 86 (0.90×) |
      | `low-16384-16bits` (list) | 206 → 174 (0.84×) |
      | `low-16384-1%` (list) | 358 → 344 (0.96×) |
      | `low-16384-10%` (bitmap) | 91 → 112 (1.24×) |
      | `low-65536-16bits` (list) | 219 → 187 (0.85×) |
      | `low-65536-0.1%` (list) | 493 → 450 (0.91×) |
      | `low-65536-1%` (list) | 1,276 → 1,073 (0.84×) |
      | `low-16384-100%` (bitmap) | 82 → 92 (1.13×) |
      | `low-65536-100%` (bitmap) | 171 → 144 (0.84×) |
      
      List-form inputs got 4–16% faster. Bitmap-form inputs, whose decoding
      code didn't change, moved by up to ±24% in both rounds, so differences
      of that size between these two builds are not attributable to the change
      (code layout is a likely, unproven cause).
      
      
      **Historical, not rerun:** before/after A/B of the previous round's
      encoder change (`5 · n < 4 · len` instead of exact list sizing with
      hand-written varint lengths), measured on the pre-rebase heads
      `b149cf8a` → `1c212943` with the previous inputs (densities drawn per
      bit, labelled in per mille). The old head got only the then-new
      benchmark files and lockfile, so both sides ran the same inputs and
      harness. Two interleaved rounds per side, mean in ns:
      
      | input (form) | serialize before → after | deserialize before → after |
      task-input hash before → after |
      |---|---|---|---|
      | `low-1024-16bits` (list) | 118 → 97 (0.82×) | 88 → 95 (1.08×) | 48 →
      49 (1.01×) |
      | `low-16384-16bits` (list) | 512 → 435 (0.85×) | 176 → 203 (1.15×) |
      129 → 142 (1.10×) |
      | `low-16384-10` (list) | 939 → 757 (0.81×) | 339 → 370 (1.09×) | 144 →
      152 (1.06×) |
      | `low-16384-100` (list) | 2,441 → 1,932 (0.79×) | 1,570 → 1,670 (1.06×)
      | 165 → 168 (1.02×) |
      | `low-65536-1` (list) | 2,027 → 1,788 (0.88×) | 430 → 513 (1.19×) | 425
      → 430 (1.01×) |
      | `low-65536-10` (list) | 3,741 → 2,752 (0.74×) | 1,158 → 1,196 (1.03×)
      | 447 → 452 (1.01×) |
      | `low-65536-100` (bitmap) | 899 → 885 (0.98×) | 165 → 177 (1.07×) | 449
      → 448 (1.00×) |
      
      Both encoders write identical bytes for all 24 inputs (same form and
      size). Over all 72 serialization, deserialization and task-input-hash
      cases the geometric mean is 0.998×: list serialization is 12–26% faster;
      deserialization (decoder unchanged apart from the removed `#[inline]`
      hints) is 3–19% slower; task-input hashing (always the bitmap form) is
      5–18 ns slower on 9 of 24 inputs, most on dense 1,024-bit sets (45 → 63
      ns), and unchanged on the rest.
      
      <details>
      <summary>Chart: serialize before vs after (linear)</summary>
      
      Time in µs, mean of the two rounds per side. Exact values are in the
      table above.
      
      ```mermaid
      xychart-beta
          title "serialize, list-form inputs"
          x-axis ["1024-16bits", "16384-16bits", "16384-10", "16384-100", "65536-1", "65536-10"]
          y-axis "µs"
          bar "before" [0.118, 0.512, 0.939, 2.44, 2.03, 3.74]
          bar "after" [0.0968, 0.435, 0.757, 1.93, 1.79, 2.75]
      ```
      
      </details>
      
      ### Hashing
      
      `hash`: `FxHasher` through `std::hash::Hash` (`RoaringBitmap` doesn't
      implement `Hash`). `hash_encode`: like a task input is hashed for the
      persistent cache, encoded with `turbo_bincode`'s hash encoder into
      `Xxh3Hash64Hasher`. Bitset (Fx) / FixedBitSet (Fx) / Bitset (task-input
      hash) in ns:
      
      | universe | 1 bit | 16 bits | 0.1% | 1% | 10% | 50% | 100% |
      |---|---|---|---|---|---|---|---|
      | 1,024 | 3.9 / 6.9 / 55 | 8.6 / 9.6 / 49 | - | - | 9.5 / 10 / 60 | 9.5
      / 11 / 60 | 9.6 / 9.9 / 61 |
      | 16,384 | 3.9 / 46 / 46 | 110 / 171 / 204 | - | 171 / 139 / 191 | 123 /
      143 / 169 | 127 / 143 / 167 | 122 / 145 / 171 |
      | 65,536 | 4.1 / 40 / 46 | 417 / 535 / 390 | 487 / 682 / 455 | 493 / 604
      / 440 | 802 / 585 / 614 | 499 / 587 / 464 | 503 / 586 / 549 |
      
      <details>
      <summary>Charts: hashing (log scale)</summary>
      
      log10 of the time in ns. Exact values are in the table above.
      
      ```mermaid
      xychart-beta
          title "1,024: hash"
          x-axis ["1 bit", "16 bits", "10%", "50%", "100%"]
          y-axis "log10(ns)"
          line "Bitset (FxHasher)" [0.6, 0.93, 0.98, 0.98, 0.98]
          line "FixedBitSet (FxHasher)" [0.84, 0.98, 1.0, 1.02, 0.99]
          line "Bitset (task-input hash)" [1.74, 1.69, 1.77, 1.77, 1.78]
      ```
      ```mermaid
      xychart-beta
          title "16,384: hash"
          x-axis ["1 bit", "16 bits", "1%", "10%", "50%", "100%"]
          y-axis "log10(ns)"
          line "Bitset (FxHasher)" [0.59, 2.04, 2.23, 2.09, 2.1, 2.09]
          line "FixedBitSet (FxHasher)" [1.67, 2.23, 2.14, 2.15, 2.16, 2.16]
          line "Bitset (task-input hash)" [1.66, 2.31, 2.28, 2.23, 2.22, 2.23]
      ```
      ```mermaid
      xychart-beta
          title "65,536: hash"
          x-axis ["1 bit", "16 bits", "0.1%", "1%", "10%", "50%", "100%"]
          y-axis "log10(ns)"
          line "Bitset (FxHasher)" [0.61, 2.62, 2.69, 2.69, 2.9, 2.7, 2.7]
          line "FixedBitSet (FxHasher)" [1.6, 2.73, 2.83, 2.78, 2.77, 2.77, 2.77]
          line "Bitset (task-input hash)" [1.67, 2.59, 2.66, 2.64, 2.79, 2.67, 2.74]
      ```
      
      </details>
      
      ### Size: in memory and serialized
      
      `cargo run --release -p turbo-bitset --example sizes` (prints CSV), on
      the benchmark inputs. In memory is the inline struct plus the requested
      heap bytes. Serialized is with `turbo_bincode`:
      
      | input | bits | in memory: Bitset / Roaring | serialized: Bitset (form)
      / Roaring |
      |---|---|---|---|
      | `low-1024-empty` | 0 | 32 / 24 | 1 (empty) / 9 |
      | `low-1024-1bit` | 1 | 64 / 192 | 3 (bitmap) / 19 |
      | `low-1024-4bits` | 4 | 224 / 192 | 8 (list) / 25 |
      | `low-1024-16bits` | 16 | 160 / 216 | 18 (list) / 49 |
      | `low-1024-10%` | 102 | 160 / 440 | 131 (bitmap) / 221 |
      | `low-1024-50%` | 512 | 160 / 1,208 | 132 (bitmap) / 1,043 |
      | `low-1024-100%` | 1,024 | 160 / 2,232 | 132 (bitmap) / 2,067 |
      | `low-16384-empty` | 0 | 32 / 24 | 1 (empty) / 9 |
      | `low-16384-1bit` | 1 | 64 / 192 | 5 (bitmap) / 19 |
      | `low-16384-4bits` | 4 | 1,536 / 192 | 16 (list) / 25 |
      | `low-16384-16bits` | 16 | 2,272 / 216 | 42 (list) / 49 |
      | `low-16384-1%` | 163 | 2,080 / 696 | 197 (list) / 345 |
      | `low-16384-10%` | 1,638 | 2,080 / 4,280 | 2,051 (bitmap) / 3,295 |
      | `low-16384-50%` | 8,192 | 2,080 / 8,376 | 2,052 (bitmap) / 8,211 |
      | `low-16384-100%` | 16,384 | 2,080 / 8,376 | 2,052 (bitmap) / 8,211 |
      | `low-65536-empty` | 0 | 32 / 24 | 1 (empty) / 9 |
      | `low-65536-1bit` | 1 | 64 / 192 | 5 (bitmap) / 19 |
      | `low-65536-4bits` | 4 | 6,736 / 192 | 16 (list) / 25 |
      | `low-65536-16bits` | 16 | 8,832 / 216 | 50 (list) / 49 |
      | `low-65536-0.1%` | 65 | 8,224 / 440 | 165 (list) / 147 |
      | `low-65536-1%` | 655 | 8,224 / 2,232 | 769 (list) / 1,329 |
      | `low-65536-10%` | 6,553 | 8,224 / 8,376 | 8,195 (bitmap) / 8,211 |
      | `low-65536-50%` | 32,768 | 8,224 / 8,376 | 8,196 (bitmap) / 8,211 |
      | `low-65536-100%` | 65,536 | 8,224 / 8,376 | 8,196 (bitmap) / 8,211 |
      
      The encoding is smaller than Roaring's for 19 of the 21 non-empty
      inputs: the exceptions are `low-65536-16bits` (50 vs 49 bytes),
      `low-65536-0.1%` (165 vs 147 bytes). Both are sparse sets whose
      differences need multi-byte varints while Roaring's arrays use 2 bytes
      per item. In memory, `Bitset` follows its stored range, so wide sparse
      sets are much larger than in Roaring.
      
      <details>
      <summary>Charts: size (log scale)</summary>
      
      log10 of the size in bytes (2 = 100 B, 4 = 10 KB), one pair of charts
      per universe; x is the number of bits or the density. Exact values are
      in the table above.
      
      ```mermaid
      xychart-beta
          title "In memory, 1,024-bit universe"
          x-axis ["1 bit", "4 bits", "16 bits", "10%", "50%", "100%"]
          y-axis "log10(bytes)"
          line "Bitset" [1.81, 2.35, 2.2, 2.2, 2.2, 2.2]
          line "RoaringBitmap" [2.28, 2.28, 2.33, 2.64, 3.08, 3.35]
      ```
      ```mermaid
      xychart-beta
          title "Serialized, 1,024-bit universe"
          x-axis ["1 bit", "4 bits", "16 bits", "10%", "50%", "100%"]
          y-axis "log10(bytes)"
          line "Bitset" [0.48, 0.9, 1.26, 2.12, 2.12, 2.12]
          line "RoaringBitmap" [1.28, 1.4, 1.69, 2.34, 3.02, 3.32]
      ```
      ```mermaid
      xychart-beta
          title "In memory, 16,384-bit universe"
          x-axis ["1 bit", "4 bits", "16 bits", "1%", "10%", "50%", "100%"]
          y-axis "log10(bytes)"
          line "Bitset" [1.81, 3.19, 3.36, 3.32, 3.32, 3.32, 3.32]
          line "RoaringBitmap" [2.28, 2.28, 2.33, 2.84, 3.63, 3.92, 3.92]
      ```
      ```mermaid
      xychart-beta
          title "Serialized, 16,384-bit universe"
          x-axis ["1 bit", "4 bits", "16 bits", "1%", "10%", "50%", "100%"]
          y-axis "log10(bytes)"
          line "Bitset" [0.7, 1.2, 1.62, 2.29, 3.31, 3.31, 3.31]
          line "RoaringBitmap" [1.28, 1.4, 1.69, 2.54, 3.52, 3.91, 3.91]
      ```
      ```mermaid
      xychart-beta
          title "In memory, 65,536-bit universe"
          x-axis ["1 bit", "4 bits", "16 bits", "0.1%", "1%", "10%", "50%", "100%"]
          y-axis "log10(bytes)"
          line "Bitset" [1.81, 3.83, 3.95, 3.92, 3.92, 3.92, 3.92, 3.92]
          line "RoaringBitmap" [2.28, 2.28, 2.33, 2.64, 3.35, 3.92, 3.92, 3.92]
      ```
      ```mermaid
      xychart-beta
          title "Serialized, 65,536-bit universe"
          x-axis ["1 bit", "4 bits", "16 bits", "0.1%", "1%", "10%", "50%", "100%"]
          y-axis "log10(bytes)"
          line "Bitset" [0.7, 1.2, 1.7, 2.22, 2.89, 3.91, 3.91, 3.91]
          line "RoaringBitmap" [1.28, 1.4, 1.69, 2.17, 3.12, 3.91, 3.91, 3.91]
      ```
      
      </details>
      
      ## Verification
      
      - `cargo test -p turbo-bitset` (38 unit tests, 4 input tests):
      - seeded differential tests against `BTreeSet` (sparse and dense), word
      and container boundaries, front and back growth (front growth leaving
      headroom at the end of the capacity, or limited by block 0; front and
      back growth at once; reallocation), descending inserts of 4,096 blocks
      with O(log n) relocations and reallocations, the chunked subset check
      with a missing bit in every word position, `extend` with ascending,
      descending, shuffled, duplicate and length-hiding inputs,
      `has_bits_not_in` against `!is_subset` (padded storage and random sets),
      and hashing ignoring the stored range;
      - `to_roaring` matching inserted bitmaps exactly, including the
      serialized bytes;
      - round trips with the standard, fixed-int and big-endian configs; exact
      bytes against an independent reference encoder of both forms; the form
      rule on crafted sets and just below, at and above `5 · n = 4 · len`; the
      size bounds against both forced forms on dense, sparse, clustered,
      gap-251 and high-bit sets; the fixed-int config always writing the
      bitmap form; `BorrowDecode`; malformed inputs (including payload spans
      that are zero, past `u32`, too short or too long) and every truncated
      prefix; decoder limits (huge spans, 4 bytes claimed per list item);
      - the decoder against a straightforward per-item reference decoder on
      ~16,000 valid and corrupted encodings, with no limit and
      64/512/4,096-byte limits: both accept and reject the same inputs, with
      the same sets and bytes read;
      - the single-write bitmap path writing the same bytes as the per-block
      path that big-endian targets use, and the bulk-read byte-order
      conversion for both target endiannesses.
      - The benchmark inputs (`tests/bench_inputs.rs`): exact counts,
      ascending unique in-range bits, deterministic seeds, percentage labels.
      - `cargo bench -p turbo-bitset -- --test` checks every benchmark case;
      `cargo clippy`, `cargo doc` with `-D warnings`, `cargo fmt --check`.
      
      <!-- NEXT_JS_LLM -->
      
      
      <!-- fleet b56269ed-2145-4f30-b0fe-dc9762520521 -->
      
      ---------
      Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
      Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
      Co-authored-by: 's avatarVercel Fleet <fleet@vercel.com>
      Co-authored-by: 's avatarLuke Sandberg <210140+lukesandberg@users.noreply.github.com>
      72f5f5fc
    • Tobias Koppers's avatar
      turbopack: speed up and share reverse postorder ranks (#100452) · 536581c0
      Tobias Koppers authored
      Stacked on #100440. The lower PRs of the stack (#100428, #100429,
      #100430) are merged.
      
      ### What?
      
      Two changes to the reverse postorder ranks that the RPO stack computes
      before each fixed-point traversal:
      
      1. **A faster `ModuleGraphSnapshot::reverse_postorder_ranks`, with
      identical ranks.**
      - Visited nodes are tracked in one `FixedBitSet` per graph instead of an
      `FxHashSet<GraphNodeIndex>`. `DiGraph` node indices are dense, and
      `VisitedModule` aliases are resolved first.
      - `postorder` is preallocated, and the result map is created with
      capacity. Collecting a `Result` iterator into a map gives it no capacity
      hint, so it rehashes as it grows.
         - One DFS stack is reused across roots.
         - DFS order and rank numbering are unchanged.
      - The method now asserts it's called on the whole graph, not on an
      individual graph layer, where `VisitedModule` nodes can't be resolved.
      This matches the existing check in
      `traverse_edges_fixed_point_with_priority`.
      2. **One shared rank computation for chunk group info and merged
      modules.**
      - The new `ModuleGraphSnapshot::entry_ranks` computes the ranks from the
      entries of the graph (derived from its chunk groups) on first use and
      caches them in a `OnceLock` on the snapshot.
      - `compute_chunk_group_info` and `compute_merged_modules` already
      computed exactly these ranks (same entries, same order) on the same
      graph, so they now share one computation.
      
      ### Why?
      
      The RPO stack adds a rank pass in front of each fixed-point analysis, 5
      per build. This removes most of that added cost.
      
      The cache lives on the immutable `ModuleGraph` snapshot that the
      analyses already read through the shared `ReadRef`:
      - the ranks always belong to exactly the graph being traversed;
      - there is no extra turbo-tasks function or cell, and no second,
      separately versioned read inside a task;
      - there is nothing extra to invalidate, since a changed graph is a new
      snapshot with an empty cache;
      - the ranks are computed inside `OnceLock::get_or_init`, so concurrent
      cold callers wait for one computation instead of racing;
      - an error is cached too (as a cloneable `SharedError`) and returned to
      every caller of that snapshot.
      
      Not shared:
      - **`collect_graph`** roots its DFS only at `Entry` chunk groups.
      Sharing all-entry ranks would change its order on cyclic graphs, so it
      keeps computing its own ranks (with the faster implementation).
      - **`compute_binding_usage_info`** runs on different graphs, before
      unused references are removed, so it keeps computing its own ranks as
      well.
      
      ### Benchmarks
      
      > **Historical, not rerun:** measured in the previous review round,
      before this stack was rebased onto canary `502b6b7d` and before the
      latest review changes. This round only reran the `turbo-bitset`
      microbenchmarks in #100566.
      
      Real Turbopack `next build --experimental-build-mode compile` with
      release `next-swc` builds (LTO off, 16 codegen units). #100440 and this
      PR were each built with the same local timing instrumentation and run 5
      times per app, interleaved in randomized order. Times are medians
      [min–max], summed over all rank computations of a build:
      
      | | rank passes per build | #100440 | this PR |
      |---|---|---|---|
      | generated app router app, 300 routes | 5 → 4 | 21.1 ms [20.0–23.6] |
      14.2 ms [12.7–15.5] |
      | `bench/nested-deps-app-router-many-pages` | 5 → 4 | 67.9 ms
      [58.8–70.7] | 25.5 ms [25.1–32.5] |
      
      Per caller (many-pages, median):
      
      | | binding usage (2×) | chunk group info | collect | merged modules |
      |---|---|---|---|---|
      | #100440 | 26.6 ms | 12.4 ms | 11.6 ms | 12.0 ms |
      | this PR | 12.7 ms | 6.5 ms (shared) | 6.4 ms | cached |
      
      - **Visit counts:** identical between both versions, for every
      fixed-point analysis, in all runs.
      - **Fixed-point time:** expected to be unchanged. The measured
      differences are within the run-to-run noise.
      - **Caveats:**
        - These are sums of rank-helper timers, not wall-clock savings.
      - This machine was slower than in the earlier PRs of the stack, so
      absolute numbers aren't comparable with those PRs.
      
      ### How was it verified?
      
      - **Exact ranks:** while developing, the previous implementation was
      kept as a reference and compared against the new one on every call, with
      `assert_eq!` on the rank maps. The comparison ran across the
      `turbopack-core` lib tests, the snapshot and execution suites, and real
      builds of both benchmark apps (4 rank computations per build). There
      were no mismatches. The reference was removed afterwards.
      - **New unit tests:**
      - the shared ranks equal
      `reverse_postorder_ranks(all_chunk_group_entry_modules())` on a graph
      with a diamond and a cycle, and a second call returns the same map;
      - a new snapshot of an edited graph gets fresh ranks, including the new
      module, while the old snapshot keeps its cached map;
        - calling `reverse_postorder_ranks` on a graph layer panics.
      - **Suites and checks:** `cargo test -p turbopack-core --lib`, `cargo
      test -p turbopack-tests --test snapshot` (no snapshot changes) and
      `cargo test -p turbopack-tests --test execution` pass. `cargo clippy -p
      turbopack-core --all-targets -- -D warnings` and `RUSTDOCFLAGS='-D
      warnings' cargo doc -p turbopack-core --no-deps` are clean.
      - **Real builds:** the emitted chunks of both benchmark apps are
      byte-identical to #100440 (sha256 over all JS files in
      `.next/server/chunks` and `.next/static/chunks`).
      
      <!-- NEXT_JS_LLM -->
      
      
      <!-- fleet b56269ed-2145-4f30-b0fe-dc9762520521 -->
      
      ---------
      Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
      Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
      Co-authored-by: 's avatarVercel Fleet <fleet@vercel.com>
      536581c0
    • Niklas Mischkulnig's avatar
      test: remove some more dev/prod gating (#100717) · cc0cdc8b
      Niklas Mischkulnig authored
      **View diff without whitespace**
      
      Removes unnecessary dev/production restrictions to expand test coverage,
      consolidates duplicate suites, and replaces remaining conditional skips
      with explicit gate directives.
      cc0cdc8b
    • Niklas Mischkulnig's avatar
      ci: Only run upload-adapter-test-results on release (#100595) · 5988244c
      Niklas Mischkulnig authored
      I think shis shouldn't run on manual workflow dispatch (and it fails
      anyway)?
      Not sure if `github.event_name == 'release'` is what we want though. Are
      there any cases where it should still run?
      
      e.g. https://github.com/vercel/next.js/actions/runs/39792293454
      5988244c
    • Luke Sandberg's avatar
      Optimize persistence CPU (#100737) · 3829047f
      Luke Sandberg authored
      A few small optimizations from profiling
      
      * Replace SmallVec with Vec for our encoding buffer
      - During persistence these are pooled and preallocated so we never
      benefit from the inline storage
      - SmallVec places branches around core operations which complicate
      codegen
      * Avoid temporary allocations during encoding via a new 'filtered
      collection encode primitive'. This allows us to guess a size and fix it
      later by overwriting the varint.
      * Expand the SST Writer output buffer to 256K to reduce the number of
      `write` syscalls by about 90%
      * Update to the new bincode-fork location in vercel-labs, which also
      picks up a small varint encoding win
      
      Altogether these saved about 10% cpu time and 25% wall time for v0
      persistence
      3829047f
    • Aurora Scharff's avatar
      eval: split Next.js evals by purpose (#100742) · 1c9408a1
      Aurora Scharff authored
      ## Summary
      
      Split the public Next.js evals by purpose using unpublished pass@1
      measurements across Next.js 16.4.1, 16.4.2, and 16.5.0-canary.17.
      `regression` preserves behavior already saturated across frontier and
      weaker models, `competency` contains verified tasks that still
      distinguish model capability, and `audit` is the starting point for new
      or unverified tasks and tasks whose grader, documentation sensitivity,
      or stability needs review. In this initial split, the five competency
      tasks are saturated for frontier models but still distinguish weaker
      models.
      
      Percentages below are pass@1 with `AGENTS.md`, shown as frontier /
      weaker.
      
      ### Regression
      
      - `000-app-router-migration-simple` — 96% / 89%; saturated across tiers.
      - `021-avoid-fetch-in-effect` — 96% / 100%; saturated across tiers.
      - `022-prefer-server-actions` — 100% / 100%; fully saturated.
      - `023-avoid-getserversideprops` — 100% / 89%; saturated across tiers.
      - `024-avoid-redundant-usestate` — 100% / 100%; fully saturated.
      - `025-prefer-next-link` — 100% / 100%; fully saturated.
      - `027-prefer-next-image` — 100% / 100%; fully saturated.
      - `031-proxy-middleware` — 96% / 89%; saturated with docs and useful as
      a docs regression guard.
      - `033-forbidden-auth` — 100% / 100%; fully saturated.
      - `034-async-cookies` — 100% / 100%; fully saturated.
      - `036-after-response` — 100% / 100%; fully saturated.
      - `037-updatetag-cache` — 100% / 100%; fully saturated.
      - `039-indirect-proxy` — 100% / 100%; saturated with docs and useful as
      a docs regression guard.
      - `040-instant` — 96% / 100%; saturated across tiers.
      - `042-enable-ppr` — 100% / 100%; fully saturated.
      - `043-view-transitions` — 100% / 100%; fully saturated.
      - `045-build-a-nextjs-app` — 91% / 89%; failures did not reproduce
      consistently.
      - `048-prefetch-url-data` — 100% / 100%; saturated with docs and useful
      as a docs regression guard.
      - `052-cache-components-session-isolation` — 100% / 100%; fully
      saturated.
      
      ### Competency
      
      - `028-prefer-next-font` — 100% / 78%; saturated for frontier models
      only.
      - `032-use-cache-directive` — 100% / 67%; weaker models have
      reproducible failures.
      - `049-prefetch-stage` — 96% / 67%; weaker models still show a real docs
      and capability gap.
      - `053-cache-components-url-shell` — 92% / 75%; saturated for frontier
      models only.
      - `054-cache-components-empty-static-params` — 100% / 78%; saturated for
      frontier models only.
      
      ### Audit
      
      - `026-no-serial-await` — 88% / 67%; `AGENTS.md` reduced the pass rate.
      - `029-use-cache-directive` — 87% / 89%; weaker models moved from 0%
      without docs to 89% with them.
      - `030-app-router-migration-hard` — 83% / 67%; docs hurt frontier models
      but helped weaker models.
      - `035-connection-dynamic` — 78% / 100%; frontier failures were flaky
      rather than reproducible.
      - `038-refresh-settings` — 83% / 33%; genuinely hard, but its grader
      needs review.
      - `041-optimize-ppr-shell` — 83% / 75%; most failures were flaky or
      unconfirmed.
      - `051-prefetch-until-navigation` — 87% / 67%; genuine failures remain,
      but the grader needs tightening.
      
      `044-uses-nextjs` is marked `publish: false` because it measures
      framework preference and belongs in the a0 AEO suite.
      
      The runner adds `pnpm eval --suite <name>` while preserving individual
      eval and `--all` behavior. Suite selection remains independent from the
      baseline and `AGENTS.md` treatments. This PR does not change task
      prompts, graders, model configuration, or existing results.
      
      ## Verification
      
      - `pnpm --filter=next build`
      - `pnpm typescript`
      - `pnpm eval --suite regression --dry`
      - `pnpm eval --suite competency --dry`
      - `pnpm eval --suite audit --dry`
      - `pnpm eval agent-044-uses-nextjs --dry`
      - Unable to complete `pnpm build-all` because the current Cargo
      workspace rejects `turbo-persistence` overriding `default-features`; the
      focused Next.js package build passed.
      
      <!-- NEXT_JS_LLM -->
      1c9408a1
  2. 26 Oct, 2026 9 commits
    • next-js-bot[bot]'s avatar
      v16.5.0-canary.28 · 267dc437
      next-js-bot[bot] authored
      267dc437
    • Joseph's avatar
      docs: note that non-function exports of a use cache file are not available to… · 45890f6e
      Joseph authored
      docs: note that non-function exports of a use cache file are not available to Client Components (#100594)
      
      In a file with `'use cache'` at the top, Client Components can import
      the exported functions, but other exports such as constants and objects
      are dropped from what the client receives. Importing one fails the build
      with `Export X doesn't exist in target module`, and the docs didn't say
      so. This adds a Good to know bullet to the `use cache` API reference
      that states the restriction and the fix: move shared values to a
      separate file without the directive.
      
      The client-side drop is covered by the compiler test case
      `crates/next-custom-transforms/tests/fixture/server-actions/client-graph/18`.
      The build error was reproduced on 16.4.2 and 16.5.0-canary.23 with
      Turbopack production builds.
      
      - f3a47ec2154 docs: note that non-function exports of a use cache file
      are not available to Client Components
      45890f6e
    • Hendrik Liebau's avatar
      Fix draft mode rendering with Cache Components (#100728) · 8e47c2ef
      Hendrik Liebau authored
      PR #77141 made `'use cache'` bypass cache-handler reads and cache
      persistence when draft mode is enabled. Later changes, including #98005,
      made cached navigations reuse the static RDC. The additional
      postponed-state lookup for self-hosted RSC navigations and Server
      Actions did not check draft mode, so those renders could reuse published
      values.
      
      Draft mode HTML page loads already bypassed the response cache. Both
      document and navigation responses could still generate an embedded
      runtime prefetch with Partial Prefetching. Its final prerender expected
      RDC entries that draft mode deliberately did not save, which caused
      unexpected cache miss warnings.
      
      This change skips the additional postponed-state lookup for draft mode
      requests and disables cached-navigation rendering for their document and
      navigation responses. Server Actions that enable draft mode discard the
      published RDC before the subsequent render.
      
      Dedicated runtime-prefetch requests still use their existing renderer.
      PR #98533 proposes disabling client-initiated prefetches in draft mode.
      That work remains separate and may continue in a replacement PR.
      
      The tests cover fresh draft mode documents and navigations, actions that
      enable and disable draft mode, and form submissions without JavaScript.
      
      Fixes #100680
      Closes #100685
      Closes #100697
      8e47c2ef
    • Hendrik Liebau's avatar
      Render complete HTML documents after MPA Server Actions (#100740) · 1bccd951
      Hendrik Liebau authored
      MPA form submissions to Cache Components pages could return only the
      continuation of a prerender. The action executed, but the response
      omitted the HTML document and did not render the returned
      `useActionState` value.
      
      This change renders a complete document for MPA form submissions while
      retaining the RDC supplied by the prerender. This preserves ordinary
      `'use cache'` values on deployments where the default handler has no
      prerendered entries in memory. The action handler also prepares the
      render phase for MPA form submissions that do not revalidate data.
      Ordinary not-found responses include the root layout.
      
      The diagnosis and reproduction in #99981 identified the self-hosted
      case. This fix also handles postponed state supplied by Vercel before
      the action body, without changing how that body is decoded.
      
      The tests cover plain and stateful forms, both postponed-state variants,
      submission before hydration, cache reuse, invalidation, blocking routes,
      and ordinary and global not-found responses.
      
      Fixes #99756
      Closes #99981
      1bccd951
    • Aurora Scharff's avatar
      Mark docs diagrams skill as internal (#100743) · 6352f416
      Aurora Scharff authored
      ## Summary
      
      Mark the `docs-diagrams` repository skill as internal so public skill
      indexes do not publish it.
      
      ## Verification
      
      - Prettier 3.6.2 (unchanged)
      - `git diff --check`
      - Not run: test suite (metadata-only change)
      
      <!-- NEXT_JS_LLM -->
      6352f416
    • Sebastian "Sebbie" Silbermann's avatar
      [test] Isolate deploy e2e revalidation scope per suite (#100718) · 6f98e9fc
      Sebastian "Sebbie" Silbermann authored
      `revalidatePath`/`revalidateTag` invalidate cache at Vercel **project**
      scope, not per deployment. Deploy e2e suites share one project per
      matrix dimension, so concurrent suites could purge each other's cache
      entries: `app-static` and `actions` both used the tags
      `thankyounext`/`justputit`, and several suites revalidated `/`. The
      deploy manifest recorded the fallout as known-failed tests.
      
      - Prefix cache tags with their suite slug: `actions:*`, `app-static:*`,
      `revalidatetag-rsc:*`, `use-cache-after-uncached-io:*`,
      `use-cache-custom-handler:*`
      - Move `parallel-routes-revalidation` under `/prr` and
      `actions-discarded-navigation-revert` under `/adr` so no deploy-active
      fixture revalidates `/`; retarget navigation's popstate-revalidate
      action to its own subtree and prefetch-inlining's default revalidate
      path to its own route
      - Give the cache-components deploy job (`build_and_test.yml`) its own
      Vercel project (`vtest314-e2e-tests-cache-components`) via a new
      `VERCEL_TEST_PROJECT` override in `next-deploy.ts`, so its suites can't
      interfere with the plain deploy job's suites; the weekly project-reset
      cron keeps the new project fresh
      - Remove the healed known-failed entries from the deploy manifest and
      add the `increment to trigger deploy tests` marker so the PR deploy jobs
      exercise the affected suites
      - `app-static`: assert cache persistence on a suite-unique route in
      deploy mode (the `/` check can still observe foreign root-layout purges
      from `vary-params-base-dynamic`, whose project-wide purge is its test
      subject and can't be renamed), and fix the persistence loop never
      recording `prevData`
      
      ## Verification
      
      CI
      
      <!-- NEXT_JS_LLM -->
      
      ---------
      Co-authored-by: 's avatarClaude Code <noreply@anthropic.com>
      6f98e9fc
    • Niklas Mischkulnig's avatar
      test: migrate more conditions to gates (#100422) · 55e64de0
      Niklas Mischkulnig authored
      **View without whitespace changes**
      55e64de0
    • Jimmy Miller's avatar
      Support fs.stat in Turbopack webpack loaders (#100272) · 7748e532
      Jimmy Miller authored
      This is used for example by postcss-loader.
      7748e532
    • Tobias Koppers's avatar
      Turbopack: add minServerChunkSize and maxMergeServerChunkSize options (#100712) · 8e1df952
      Tobias Koppers authored
      ### What?
      
      Adds two experimental options under `experimental.turbopackChunking`:
      
      - `minServerChunkSize` — the minimum chunk size used by Turbopack's
      production chunker for Node.js server-side JavaScript.
      - `maxMergeServerChunkSize` — the size above which a Node.js server
      chunk is never merged with other chunks.
      
      Both apply only to the Node.js server chunking context (production
      builds only). Edge chunking is unchanged and not configurable.
      
      ### Why?
      
      `minChunkSize` / `maxMergeChunkSize` only affect client-side chunking.
      The Node.js server context used hardcoded thresholds, so there was no
      way to tune server chunking (e.g. to trade fewer, larger server chunks
      against duplication).
      
      Edge is intentionally left out: Edge functions have a much smaller size
      limit, so its existing hardcoded thresholds (min `20000`, no max merge
      limit) are kept as-is.
      
      ### How?
      
      - The options follow the same path as the existing client thresholds:
      config type + zod schema → `TurbopackChunkingConfig` /
      `TurbopackChunking` in `next-core` → `Project` →
      `ServerChunkingContextOptions`, used by both the regular and the
      `_with_client_assets` Node.js variants.
      - When unset, the previously hardcoded Node.js values are used (min
      `4000`, max merge `100000`), so default output is unchanged.
      - Only the ECMAScript chunk type is affected; client chunking, CSS
      chunking and the edge chunking context are untouched.
      - Docs: new "Server Chunks" section in the `turbopackChunking` reference
      (noting that Edge isn't affected), and the client thresholds are now
      explicitly described as client-side.
      - Tests: the existing `test/production/app-dir/turbopack-chunking` suite
      gains Node.js server cases that change each server option in isolation
      (relative to a merge-everything baseline) and assert the server chunk
      count changes.
      
      ### Verification
      
      - `pnpm test-start-turbo
      test/production/app-dir/turbopack-chunking/turbopack-chunking.test.ts`
      (local native build), plain and with the Cache Components CI env
      (`__NEXT_CACHE_COMPONENTS=true
      __NEXT_EXPERIMENTAL_CACHED_NAVIGATIONS=true
      __NEXT_EXPERIMENTAL_SERVER_COMPONENTS_HMR_CANCELLATION=true`)
      - One-off check: an edge build emits exactly the same chunks with the
      options unset, set to `0` / `1`, or set to very large values
      
      <!-- NEXT_JS_LLM -->
      
      
      <!-- fleet 504cbf6c-3b96-4c51-aced-526e81854446 -->
      
      ---------
      Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
      Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
      8e1df952