- 12 Mar, 2026 22 commits
-
-
nextjs-bot authored
-
Luke Sandberg authored
## Summary Remove the unused LMDB database backend and the `ReadTransaction` abstraction layer from `turbo-tasks-backend`. This is a pure cleanup that deletes ~1,500 lines of code and eliminates all `unsafe` blocks related to transaction lifetime management. **Motivation:** - This was much more useful in the early days of the persistence layer where comparing against a 'known good' db was useful, but this has rotted and turbo-persistence has gained lots of tests and usage so that this is not needed. - Carrying the code added a lot of complexity to APIs e.g. - The `ReadTransaction` abstraction existed solely to support LMDB's transaction-based read model - The only remaining backend (`TurboKeyValueDatabase` via `turbo-persistence`) already used `()` as its `ReadTransaction` type, making the entire abstraction dead weight - The transaction plumbing required `unsafe` transmute-based lifetime extension in `ExecuteContextImpl`, adding complexity and risk for no benefit - The WriteBatch::Serial branches complicated already subtle code paths -
Andrew Clark authored
Previously, MPA page loads (reload, anchor navigation) while the instant navigation lock was held would skip hydration entirely and hard-reload when the lock released. This meant the page was non-interactive during the locked period and tests couldn't assert on the hydrated React tree. Now the page actually hydrates from the static shell. The server injects a script that fetches the static RSC payload, and the client uses createFromFetch with allowPartialStream so React can hydrate without erroring on the incomplete Suspense boundaries. When the lock is released, a soft refresh fetches the dynamic data (falling back to hard reload if the router hasn't initialized yet). Also refactors the cookie protocol to encode the lock state as a JSON array (`[0]` = pending, `[1, null]` = captured MPA, `[1, {}]` = captured SPA) for better integration with the devtools UI. This is somewhat entangled with the hydration changes so they're grouped together, but can be reviewed as atomic commits. -
Benjamin Woodruff authored
Migrates the content from https://turbopack-rust-docs.vercel.sh/turbo-engine/traits.html over to rustdocs, and tries to update/fix some minor mistakes our outdated docs.   
-
Will Binns-Smith authored
This: - Fetches `data/_app/analyze.data` alongside the current route's data - Merges `_app` modules into the current route's `AnalyzeData` before visualization using a `mergeAnalyzeData()`, which merges the two datasets, reindexing the additional ones and recreating the binary data format expected by the treemap. ## Test plan - [x] Add a large dependency (e.g. `moment`) in `pages/_app.tsx` - [x] Run `next experimental-analyze` - [x] Open the bundle analyzer and select a pages page - [x] Verify moment appears in the treemap - [x] Verify App Router projects (no `_app`) work without errors
-
Janka Uryga authored
Adds extra context to errors that we think may have prevented validation: <img width="816" height="765" alt="Screenshot 2026-03-12 at 22 07 40" src="https://github.com/user-attachments/assets/35ce60d0-4840-496e-b91e-62ada4096f64" /> This is particularly useful for errors where the component comes from node_modules, so it gets ignore-listed. previously, all we got is the message with no stack, no we at least get a location. One notable case of this is `next/dynamic` with `ssr: false`. <img width="802" height="556" alt="Screenshot 2026-03-12 at 22 08 30" src="https://github.com/user-attachments/assets/b1240732-3346-4056-89b2-9a864a3166f2" />
-
JJ Kasper authored
### What? This PR exposes adapter-facing router helpers on `requestMeta` for Node.js entrypoints and updates route modules to merge those request-scoped values with the existing router server context fallback. It also documents the new adapter contract in the `adapterPath` docs. ### Why? When adapters invoke the Node.js handler directly, some router helpers are needed by route handlers but currently only exist behind internal globals. Passing them through `requestMeta` makes those helpers available through the public handler options and still preserves the existing fallback context for the current `relativeProjectDir`. ### How? - add typed `requestMeta` fields for `hostname`, `revalidate`, and `render404` - add a `RouteModule` helper that reads the existing router server context and overlays request-scoped values from `requestMeta` - switch the route-module callsites that read router server context to use the merged result - document how adapters can pass these helpers when calling `handler(req, res, { requestMeta })` -
nextjs-bot authored
-
dependabot[bot] authored
Bumps [tar](https://github.com/isaacs/node-tar) from 7.5.7 to 7.5.11. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/isaacs/node-tar/commit/bf776f673164215074b62749e0fe80e5834588f4"><code>bf776f6</code></a> 7.5.11</li> <li><a href="https://github.com/isaacs/node-tar/commit/f48b5fa3b7985ddab96dc0f2125a4ffc9911b6ad"><code>f48b5fa</code></a> prevent escaping symlinks with drive-relative paths</li> <li><a href="https://github.com/isaacs/node-tar/commit/97cff15d3539a37a4095eb3d287147d9d77c2dc3"><code>97cff15</code></a> docs: more security info</li> <li><a href="https://github.com/isaacs/node-tar/commit/2b72abc1d47c3570e1ad95c9ab557fc4c2e6e4b1"><code>2b72abc</code></a> 7.5.10</li> <li><a href="https://github.com/isaacs/node-tar/commit/7bc755dd85e623c0279e08eb3784909e6d7e4b9f"><code>7bc755d</code></a> parse root off paths before sanitizing .. parts</li> <li><a href="https://github.com/isaacs/node-tar/commit/c8cb84629dee649feedde03f2f4ea48f2e44e778"><code>c8cb846</code></a> update deps</li> <li><a href="https://github.com/isaacs/node-tar/commit/1f0c2c9006b10199cf2686f8ef43e79a1773e1aa"><code>1f0c2c9</code></a> 7.5.9</li> <li><a href="https://github.com/isaacs/node-tar/commit/fbb08518bf290733b68ca4d4135f75becf73fd75"><code>fbb0851</code></a> build minified version as default export</li> <li><a href="https://github.com/isaacs/node-tar/commit/6b8eba0ef367ac937e703238daa6df94ae6f823f"><code>6b8eba0</code></a> 7.5.8</li> <li><a href="https://github.com/isaacs/node-tar/commit/2cb1120bcefe28d7ecc719b41441ade59c52e384"><code>2cb1120</code></a> fix(unpack): improve UnpackSync symlink error "into" path accuracy</li> <li>Additional commits viewable in <a href="https://github.com/isaacs/node-tar/compare/v7.5.7...v7.5.11">compare view</a></li> </ul> </details> <details> <summary>Maintainer changes</summary> <p>This version was pushed to npm by <a href="https://www.npmjs.com/~isaacs">isaacs</a>, a new releaser for tar since your current version.</p> </details> <details> <summary>Install script changes</summary> <p>This version adds <code>prepare</code> script that runs during installation. Review the package contents before updating.</p> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/vercel/next.js/network/alerts). </details> --------- Signed-off-by:
dependabot[bot] <support@github.com> Co-authored-by:
dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by:
Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by:
styfle <229881+styfle@users.noreply.github.com>
-
Benjamin Woodruff authored
Copying content from https://turbopack-rust-docs.vercel.sh/turbo-engine/task_inputs.html Removed some AI slop and some outdated stuff (`Value` is gone, we implemented `TaskInput` on `Arc`). Added a note about serialization of reference counted types. Rendered: 
-
Sebastian "Sebbie" Silbermann authored
-
Zack Tanner authored
Reverts vercel/next.js#91163 Flag is re-landed for test teams
-
JJ Kasper authored
## Summary - export `NEXT_ENABLE_ADAPTER=1` in the `test-new-tests-deploy` workflow job - export `NEXT_ENABLE_ADAPTER=1` in the `test-new-tests-deploy-cache-components` workflow job ## Testing - `git diff --check`
-
Benjamin Woodruff authored
Moving content from: https://turbopack-rust-docs.vercel.sh/turbo-engine/tasks.html Very few changes, mostly updates to the introduction and links. Rendered: 
-
nextjs-bot authored
-
Tobias Koppers authored
## Summary Remove the per-block `madvise(MADV_SEQUENTIAL)` call before decompression in `StaticSortedFile`. The primary motivation is that the kernel splits mmap regions internally when they receive different `madvise` hints. Each per-block `MADV_SEQUENTIAL` call fragments the VMA (Virtual Memory Area) for the file mapping. For larger builds with many compressed blocks, this causes the process to hit the `vm.max_map_count` limit, leading to failures. Additionally, the file-level `MADV_RANDOM` advice is sufficient on its own — the sequential hint for individual compressed blocks adds syscall overhead without meaningful benefit, since the decompressor already performs a linear scan in userspace and the blocks are already mapped in memory. ## Changes - Removed the `madvise(Sequential)` call in `StaticSortedFile::get_block()` for compressed blocks (`turbopack/crates/turbo-persistence/src/static_sorted_file.rs`) ## Test Plan - Existing turbo-persistence tests continue to pass - This is a performance-only change with no functional impact
-
Janka Uryga authored
We forgot to update the autocomplete hint when we changed it from `mode` to `prefetch`.
-
Janka Uryga authored
Initial version of build-time instant validation. During prerendering, if we have `unstable_instant` configs, we perform a complete staged render of the page (including dynamic data). This render mocks request data (`params`/`searchParams`/`cookies`/`headers`) using the `samples` provided in `unstable_instant`. Then, we use the staged chunks to run instant validation in mostly the same way that we would in dev. Most changes in the validation flow are related to samples, which also need to be passed into the client render to mock APIs like `useParams`. ### Providing Mock data We run everything in a fresh workStore+workUnitStore to (hopefully) avoid any data from the prerender from sneaking in. This is quite involved. A big part of the trickiness is that if you do e.g. `(await cookies()).get('myCookie')` but the `samples` don't contain `myCookie`, we want the validation to error and require explicitly saying what state `myCookie` is in. If it's available, use ```ts export const unstable_instant = { // ... samples: { cookies: [{ name: "myCookie", value: "someValue" }] } } ``` If it's not available, use ```ts export const unstable_instant = { // ... samples: { cookies: [{ name: "myCookie", value: null }] } } ``` This explicitness makes it harder to e.g. unintentionally validate a logged-out state. Note that even validating a `prefetch: 'static'` page might require samples, because we need the shared parent for the navigation to be (mostly) complete. We might be able to relax this in the future, but it's not straightforward. Another issue is that a segment might access a missing sample in a "benign" way, i.e. where it wouldn't block validation (or be inside a suspense boundary). Ideally we wouldn't require providing a sample for usages like that, but currently we do. ### Instrumenting APIs to detect missing samples All request APIs are instrumented in this way. - `params` throw when accessing a property that was not declared in `samples`. same for the object from `useParams()`. `in` is allowed because we know the shape of the params object statically, based on the routing structure. - `searchParams` throw when accessing a property that was not declared in `samples` and `in` checks. - objects returned from `cookies()`, `headers()`, `useSearchPrams()` throw on `get()` and `has()`. There's a lot of tests for this behavior in `instant-validation-build`. One gap is that iteration/enumeration is currently not instrumented. We're punting on it for now and will revisit in a follow-up. ### Relation to prerender We kick off the validation after the prerender, in `renderToHTMLOrFlightImpl`. Ideally, we should kick this off somewhere higher up, separately from the prerender itself, but we're leaving that for future work. Note that if we do multiple prerenders for the same page (e.g. due to `generateStaticParams`, we'll only run validation for the first prerender for that route. `generateStaticParams` is not currently integrated into the validation at all, i.e. we don't use those params to validate, so this is safe. All params need to be provided via `samples`. This will likely change in the future, but we're not sure how yet. ### Disabling build validation Build-time instant validation is experimental, and has more potential problems to deal with than dev validation. To allow incremental rollout and testing, we allow disabling it per instant config: ```ts export const unstable_instant = { unstable_disableBuildValidation: true // ... } ``` for parity, there's also `unstable_disableBuildValidation`, but we don't expect that to see much use. -
Sebastian "Sebbie" Silbermann authored
Required for TypeScript 6.0. Older versions of `json-schema-types` ship with TypeScript source instead of declaration files.
-
Sebastian "Sebbie" Silbermann authored
Required to work with TypeScript 6.0 The package itself is vendored into `create-next-app`. Includes 1. https://github.com/alexeyraspopov/picocolors/releases/tag/v1.1.1 1. https://github.com/alexeyraspopov/picocolors/releases/tag/v1.1.0 1. https://github.com/alexeyraspopov/picocolors/releases/tag/v1.0.1
-
Sebastian "Sebbie" Silbermann authored
Co-authored-by:Cursor Agent <cursoragent@cursor.com>
-
zoomdong authored
In utoo case, we need to support react classic jsx transform(To compitable with react 16 or above project). We have a case like: ```ts // index.ts import React from 'react'; export const App = () => <div></div>; ``` Under the classic jsx transform, the jsx will be transformed as follows: ```ts export const App = () => React.createElement("div", /***/); ``` But turbopack's jsx transform was behind typescript transform, so in the stage of typescript transform, the import of react will be deleted by the default config of swc typescript transform(which set `verbatimmodulesyntax` as false by default): https://swc.rs/docs/migrating-from-tsc#verbatimmodulesyntax-true Then the output of classic jsx transform will get a runtime error like `React was not found`. For solve the problem, this pr just support a config `verbatimModuleSyntax` for turbopack's typescript transform, it will not remove user's React import under classic jsx transform. We test the case under utoopack in the following pr: - https://github.com/utooland/utoo/pull/2653 - https://github.com/utooland/utoo/pull/2655 And Rspack also enable `verbatimModuleSyntax` to true by default under 2.0: https://github.com/web-infra-dev/rspack/discussions/9270#discussioncomment-13429924
-
- 11 Mar, 2026 18 commits
-
-
nextjs-bot authored
-
JJ Kasper authored
## Summary - preserve Flight/RSC headers on the Proxy request instance when URL normalization is skipped (`skipProxyUrlNormalize`) - preserve the `_rsc` query on the request URL in that same mode so custom proxy logic can access it - document this behavior in the Proxy API docs, including guidance for `NextResponse.rewrite()` vs custom `fetch` rewrites - add an e2e regression in `skip-trailing-slash-redirect` to verify middleware receives `_rsc` and all RSC headers on the request instance ## Testing - `NEXT_SKIP_ISOLATE=1 NEXT_TEST_MODE=dev pnpm testheadless test/e2e/skip-trailing-slash-redirect/index.test.ts`
-
Benjamin Woodruff authored
Merges the content from https://turbopack-rust-docs.vercel.sh/turbo-engine/cells.html into our rustdocs. I also tried fixing the wording of a few things for accuracy, and adding a bit more detail where warranted. Mostly, this modified our `Vc` documentation. Rendering of `Vc` page: 
-
Niklas Mischkulnig authored
``` let x: Ident = IdentName::new(name, span).into(); ``` should never be done. It uses `SyntaxContext::default()` for the Ident. So from swc's perspective, you are referring to a different variable than you think. When constructing an `Ident`, you should usually always provide a SyntaxContext. I think this would pose a problem in Webpack because it serializes it into a code string again, but for Turbopack these SyntaxContext are important.
-
Luke Sandberg authored
🤦 ♂️ -
Sam Selikoff authored
…of client navs. Allows the initial screen to be dismissed without reloading the app. https://github.com/user-attachments/assets/56b08c57-bd78-4d38-875d-244a450b7325
-
xusd320 authored
## What? Calls `unref()` on the `ThreadsafeFunction`s created in `NapiNextTurbopackCallbacks::from_js` (`throw_turbopack_internal_error` and `on_before_deferred_entries`). ## Why? These `ThreadsafeFunction`s are not unref'd, which means they keep the Node.js event loop alive even after `project_shutdown` completes. Currently this doesn't cause a visible hang in Next.js because turbopack runs inside a worker thread (via `packages/next/src/lib/worker.ts`), and the worker thread is terminated externally. However, if turbopack were ever run in a child process or directly in the main process, the build would hang indefinitely after completion — the Rust side of `stop_and_wait()` finishes, but the Node.js process never exits because the ref'd `ThreadsafeFunction`s prevent the event loop from draining. This happened in `utoo`, caused some CI hang forever, the same fix commit: https://github.com/utooland/utoo/pull/2670/changes/9e8fd08e84470dce6ac59e976db1b556ce7fece8. This fix makes the shutdown behavior more robust by following the same pattern already used in `register_worker_scheduler` in `turbopack-node/src/worker_pool/worker_thread.rs`, where both creator and terminator `ThreadsafeFunction`s are explicitly unref'd. ## How? - Added `Env` parameter to `NapiNextTurbopackCallbacks::from_js()` - Called `unref(&env)` on both `throw_turbopack_internal_error` and `on_before_deferred_entries` after creation - Updated the call site in `project_new` to pass `&env`
-
nextjs-bot authored
-
Will Binns-Smith authored
This adds a dedicated attribute, `failed`, to a span representing if the procedure failed. It's then set if the runner function threw. Test Plan: Added an e2e test
-
Tim Neutkens authored
### What? Fixes webpack dev 404s when parallel routes have children nested inside a route group (e.g. `@slot` + `(b1)/page`) and when a parallel route has `page.tsx` directly inside it (e.g. `@slot/page.tsx`). ### Why? Two separate issues caused 404s for parallel routes with route groups: 1. **Sort order**: The three `appPaths` sort sites used bare `.sort()` (lexicographic ordering). `renderPageComponent` relies on `appPaths[appPaths.length - 1]` being the children/root page. Without route groups this works because `@` (0x40) sorts after lowercase letters — `/@slot/page` comes before `/page`. But when the children page is inside a route group, `(` (0x28) sorts before `@` (0x40), flipping the order: `/(b1)/page < /@slot/page`. Now the last item is the slot page instead of the children page, so the manifest lookup fails and a 404 is returned. 2. **Double `__PAGE__` segment**: When a parallel route like `@slot` has `page.tsx` directly inside it, `resolveParallelSegments` returns `[PAGE_SEGMENT]` (an array) to signal that the loader should recurse into the slot directory to pick up layout/default files. However, the segment key normalization was mapping `PAGE_SEGMENT` to `PAGE_SEGMENT_KEY` (`__PAGE__`), while the recursion would also produce a `__PAGE__` for children — creating a double `__PAGE__` nesting (`slot: ['__PAGE__', { children: ['__PAGE__', ...] }, ...]`). ### How? **Sort order fix**: Adds a `compareAppPaths` comparator in `packages/next/src/shared/lib/router/utils/app-paths.ts` that sorts paths containing `/@` segments before paths without them, ensuring the children page is always last. Replaces the three bare `.sort()` calls with `.sort(compareAppPaths)` in: - `packages/next/src/build/entries.ts` - `packages/next/src/server/lib/router-utils/setup-dev-bundler.ts` - `packages/next/src/server/route-matcher-providers/dev/dev-app-page-route-matcher-provider.ts` **Loader tree fix**: Maps `PAGE_SEGMENT` to `'(slot)'` in the segment key normalization (same as `PARALLEL_VIRTUAL_SEGMENT`), instead of `PAGE_SEGMENT_KEY`. By the time this code is reached, `parallelSegmentKey === PAGE_SEGMENT` only occurs for the array case (the string case is handled by the early return), so this is safe. The actual `__PAGE__` is correctly created by the recursion. **Before:** `slot: ['__PAGE__', { children: ['__PAGE__', ...] }, ...]` **After:** `slot: ['(slot)', { children: ['__PAGE__', ...] }, ...]` Also adds an e2e test that verifies the fix across all modes (Turbopack dev, Turbopack start, webpack dev). -
Niklas Mischkulnig authored
There are unfortunately a few false-positives, because this isn't aware of type information of course.
-
Luke Sandberg authored
In turbo-persistence, if a key block has small keys don't attempt to compress the key block Currently all of our tables have small keys and none of them actually meet our compression threshold, thus all the time we spend attempting to compress them is just wasted CPU time.
-
Tobias Koppers authored
### What? Adds a new `experimental.lightningCssFeatures` config option that lets users control which CSS features lightningcss should always transpile (`include`) or never transpile (`exclude`), regardless of browserslist targets. ```js // next.config.js module.exports = { experimental: { useLightningcss: true, lightningCssFeatures: { include: ['light-dark', 'oklab-colors'], exclude: ['nesting'], }, }, } ``` ### Why? Currently, lightningcss feature transpilation is determined solely by browserslist targets. There's no way to force transpilation of a specific feature (e.g., `light-dark()`) when targeting modern browsers that already support it, or to skip transpilation of a feature that the user wants to preserve as-is. This is useful for: - Forcing polyfill-style transpilation for features with incomplete browser support - Testing transpiled output for specific features - Opting out of specific transforms that may interfere with CSS tooling downstream ### How? **TypeScript config & validation:** - `config-shared.ts`: `LIGHTNINGCSS_FEATURE_NAMES` const array (single source of truth) with `LightningCssFeature` type derived from it, and `LightningCssFeatures` interface - `config-schema.ts`: Zod validation using `z.enum(LIGHTNINGCSS_FEATURE_NAMES)` for both include/exclude arrays **Feature name → bitmask mapping (Rust, shared via NAPI):** - `crates/next-core/src/next_config.rs`: `lightningcss_feature_names_to_mask()` maps feature name strings to `lightningcss::targets::Features` bitflag constants and returns a `Result<u32>` bitmask. Unknown feature names produce an error via `bail!`. - Exposed to JS via NAPI as `lightningcssFeatureNamesToMaskNapi`, callable through `bindings.css.lightning.featureNamesToMask(names)` - Single source of truth for feature name resolution — both webpack and Turbopack paths use this Rust function **Webpack path:** - `global.ts` / `modules.ts` pass config through to loader options - `loader.ts` calls `featureNamesToMask()` via the native bindings to compute include/exclude masks, then passes them to the SWC `transform()` call - The `lightningCssFeatures` without `useLightningcss` warning only shows when using webpack (Turbopack always uses lightningcss) **Turbopack path (Rust):** - `next_config.rs`: `LightningCssFeatures` struct for deserialization, accessor methods converting names → u32 bitmask via `lightningcss_feature_names_to_mask()` - Config flows through `CssOptionsContext` → `ModuleType::Css` → `CssModuleAsset` (using `LightningCssFeatureFlags { include, exclude }` struct) → `process.rs` where bitmasks are merged into `lightningcss::targets::Targets { include, exclude }` - Uses `u32` bitmask representation throughout to avoid adding lightningcss dependency to non-CSS crates **Defaults preserved:** `Nesting | MediaRangeSyntax` for Turbopack, `Nesting` for Webpack. User `include` is OR-ed on top, user `exclude` masks bits off. **Feature names:** 21 individual features (bit 0–20) + 3 composite groups (`selectors`, `media-queries`, `colors`), all in dash-case. ### Test - **Include test** (`test/e2e/app-dir/experimental-lightningcss-features/`) — targets Chrome 123 (which natively supports `light-dark()`) with `include: ['light-dark']`, then asserts the CSS output contains lightningcss transpilation markers (`--lightningcss-light`, `--lightningcss-dark`) instead of raw `light-dark()`. - **Exclude test** (`test/e2e/app-dir/experimental-lightningcss-features-exclude/`) — targets Chrome 100 (which does NOT support `light-dark()` natively) with `exclude: ['light-dark']`, then asserts the CSS output preserves raw `light-dark()` calls and does not contain transpilation markers. -
Luke Sandberg authored
Not that error rendering performance is critical, but it is silly to pass frame information from turbopack to js just to have js pass the same information back to native code to draw a frame.
-
JJ Kasper authored
x-ref: https://github.com/vercel/next.js/pull/91163
-
Benjamin Woodruff authored
Merges the content from https://turbopack-rust-docs.vercel.sh/turbo-engine/concepts.html with the crate-level documentation we already had, and adds some some graphics from https://nextjs.org/blog/turbopack-incremental-computation (currently linking to the vercel blob bucket). ### Rendered 
-
Luke Sandberg authored
When snapshotting we create a number of iterators to encode every `shard` of the `modified` dashmap. Some of these shards end up being empty and rarely they might all be empty. To avoid the overhead of opening a write transaction we detect this prior to persisting. Unfortunately this is done in a single threaded pass and can be expensive since there are often thousands of shards. To improve this we move the filtering into shard production so it can happen concurrently. This is a small improvment, from vercel-site builds timing the `Finished writing to filesystem cache` event we get Before: 6.6s 7.8s 7.1s After: 6.6s 6.5s 6.4s So a small progression
-
Luke Sandberg authored
## Summary Reduces CPU overhead in turbo-persistence SST reads through three optimizations: 1. **Switch from `twox-hash` to `xxhash-rust`** — `twox_hash::XxHash3_64::with_seed(0)` heap-allocates a 192-byte secret buffer on every call. `xxhash_rust::xxh3::Xxh3Default::new()` is a `const fn` with zero allocation. Applied to both `turbo-persistence` and `turbo-tasks-hash`. 2. **Bulk `write()` in `KeyBase::hash` impls** — The hash implementations were feeding bytes one at a time via `write_u8()`, preventing xxh3 from using its optimized bulk ingestion path. Changed all slice-backed types (`&[u8]`, `[u8; N]`, `Vec<u8>`, `Box<[u8]>`) to use a single `state.write(slice)` call. 3. **Binary search improvements in SST lookups:** - **Index block lookup**: Replaced hand-rolled binary search with `as_chunks::<10>()` + stdlib `binary_search_by`, making the code simpler and leveraging the stdlib's optimized search. - **Duplicate key scans**: When collecting all entries with the same key (FIND_ALL mode), the backward/forward scans now use the binary search bounds (`l..m` and `m+1..r`) instead of scanning from `0` to `entry_count`. The binary search already proved entries outside these bounds are different, so this avoids unnecessary comparisons. - **`entry_matches_key` helper**: Equality checks during duplicate scans now use a dedicated function that avoids recomputing the full hash when no hash is stored in the block (block type 2/4). Previously `compare_hash_key` would call `hash_key()` to compute the full hash even when only an equality check was needed. ## Benchmark results Compared against the `fixed_size_blocks` branch using `cargo bench -p turbo-persistence -- 'read/get/key_8'` (8-byte keys, 4-byte values): | Benchmark | Change | |-----------|--------| | 10.67Mi/compacted/miss/cached | **-45%** | | 10.67Mi/compacted/hit/cached | **-35%** | | 10.67Mi/uncompacted/hit/cached | **-35%** | | 10.67Mi/uncompacted/hit/uncached | **-22%** | | 10.67Mi/uncompacted/miss/uncached | **-22%** | | 10.67Mi/uncompacted/miss/cached | **-21%** | | 10.67Mi/20commits/hit/cached | **-26%** | | 10.67Mi/20commits/miss/* | -13% | | 85.33Mi/compacted/hit/uncached | **-56%** | | 85.33Mi/compacted/hit/cached | **-22%** | | 85.33Mi/compacted/miss/* | **-22%** | | 85.33Mi/uncompacted/hit/cached | -10% | | 85.33Mi/20commits/hit/uncached | -12% | | 85.33Mi/20commits/hit/cached | -6% | | 85.33Mi/20commits/miss/* | -1 to -3% | Largest wins on cached paths with smaller datasets where CPU cost (hashing, search) dominates over I/O.
-