Unverified Commit 466cdfaa authored by Will Binns-Smith's avatar Will Binns-Smith Committed by GitHub

Turbopack server hmr: avoid complete `clear` on graph changes (#95546)

Currently, importing a new module causes a complete eviction (`clear()`)
and re-evaluation of server chunks, both in the Turbopack runtime's
module cache and Node's `require.cache`.

Here's what happens today:

- A new module is imported into a chunk's graph
- This changes its `availability_info`, which in dev for non-entry
chunks is encoded into the chunk's filepath
- `VersionedContentMap` works entirely on chunk paths. When we construct
instructions to transition a chunk into its new state, we fail to find
the prior state and fall back to `clear()` as described above.

An ideal version of this is a refactor that implements
`VersionedContentMap` on a per-module basis, not a per-chunk one. This
PR doesn't do that, but achieves consistent module-level updates when a
chunk's availability info changes.

It does this by implementing chunk lists for server entry chunks the
same way the client hmr implementation does: an entry chunk's version
aggregates the versions of its dependent chunks (including dynamically
imported ones), keyed by merger rather than path. Entry chunks do not
encode availability info in their paths, so they are insulated from
missing versions. Updates are applied in Node.js through the same shared
merged-update machinery in the unified hmr runtime that the client
already uses, plus a new `ChunkListUpdate` branch in the server hmr
client to unwrap the merged updates.

Details:

- The merged-update wire format (`EcmascriptMergedUpdate` etc.) moves
from turbopack-browser into shared `turbopack_ecmascript::chunk_list`,
along with runtime-agnostic `ChunkListVersion` and `update_chunk_list`
implementations.
- `turbopack-nodejs` gains a chunk content merger mirroring the browser
one. `EcmascriptBuildNodeEntryChunk`'s versioned content is now a
chunk-list content over its sync and async chunks. It is version-only
and never emitted; the entry chunk still inlines its loader calls.
- The aggregate server hmr subscription now tracks only entry chunks.
Shared chunks under `server/chunks/` ride the entry's `ChunkListUpdate`
as module deltas, so their content-hash paths no longer matter.
- The Node hmr client applies `ChunkListUpdate` by feeding each nested
merged update through the shared apply path. Modules that appear in an
"added" chunk but already exist in the module cache were moved by a
chunk rename and are treated as modified.
- On a successful partial apply, the hot reloader clears the manifest
cache for updated chunks and notifies browsers to refetch RSC, without
clearing `require.cache`.

### Test Plan

Adds a series of e2e tests to catch the described manual test plan in
both entry and dynamic chunks.
parent b340bc74
......@@ -3,10 +3,12 @@ use std::sync::Arc;
use anyhow::Result;
use rustc_hash::FxHashMap;
use turbo_rcstr::RcStr;
use turbo_tasks::{FxIndexMap, ReadRef, ResolvedVc, TraitRef, TryJoinIterExt, Vc};
use turbo_tasks::{FxIndexMap, FxIndexSet, ReadRef, ResolvedVc, TraitRef, TryJoinIterExt, Vc};
use turbo_tasks_fs::FileSystemPath;
use turbo_tasks_hash::{Xxh3Hash64Hasher, encode_base64};
use turbopack_browser::ecmascript::list::content::EcmascriptDevChunkListContent;
use turbopack_core::version::{PartialUpdate, Update, Version, VersionState, VersionedContent};
use turbopack_nodejs::ecmascript::node::entry::chunk_list_content::EcmascriptBuildNodeChunkListContent;
use crate::versioned_content_map::VersionedContentMap;
......@@ -15,6 +17,17 @@ pub struct HmrChunkWithContent {
pub content: ResolvedVc<Box<dyn VersionedContent>>,
}
/// Whether `content` is a chunk list, i.e. an entry point of the chunk graph that
/// an HMR subscription can be anchored on.
///
/// Note this must enumerate every chunk list content type. A new chunking context
/// that introduces one has to be added here, otherwise its chunks silently drop
/// out of the HMR subscription.
pub fn is_entry_chunk_list_content(content: ResolvedVc<Box<dyn VersionedContent>>) -> bool {
ResolvedVc::try_downcast_type::<EcmascriptBuildNodeChunkListContent>(content).is_some()
|| ResolvedVc::try_downcast_type::<EcmascriptDevChunkListContent>(content).is_some()
}
/// Per-chunk versions keyed by path
#[turbo_tasks::value(serialization = "skip", shared)]
pub struct AggregateHmrVersion {
......@@ -80,60 +93,71 @@ impl AggregateHmrVersion {
}
}
pub fn merge_ecmascript_merged_update(
combined_entries: &mut FxHashMap<String, serde_json::Value>,
combined_chunks: &mut FxHashMap<String, serde_json::Value>,
instruction: &serde_json::Value,
) {
let Some(obj) = instruction.as_object() else {
return;
};
if let Some(entries) = obj.get("entries").and_then(|v| v.as_object()) {
for (k, v) in entries {
combined_entries.insert(k.clone(), v.clone());
/// Aggregates per-entry HMR instructions into a single combined `ChunkListUpdate`.
#[derive(Default)]
pub struct ChunkListUpdateBuilder {
chunks: FxHashMap<String, serde_json::Value>,
merged: FxIndexSet<serde_json::Value>,
}
impl ChunkListUpdateBuilder {
pub fn add_instruction(&mut self, instruction: &serde_json::Value) {
let Some(obj) = instruction.as_object() else {
return;
};
match obj.get("type").and_then(|v| v.as_str()) {
Some("ChunkListUpdate") => {
if let Some(chunks) = obj.get("chunks").and_then(|v| v.as_object()) {
for (k, v) in chunks {
self.chunks.insert(k.clone(), v.clone());
}
}
if let Some(merged) = obj.get("merged").and_then(|v| v.as_array()) {
for update in merged {
self.push_merged(update);
}
}
}
Some("EcmascriptMergedUpdate") => {
self.push_merged(instruction);
}
// Unknown instruction shapes are ignored; the caller already
// escalates `Total`/`Missing` updates to a full restart.
_ => {}
}
}
if let Some(chunks) = obj.get("chunks").and_then(|v| v.as_object()) {
for (k, v) in chunks {
combined_chunks.insert(k.clone(), v.clone());
}
fn push_merged(&mut self, update: &serde_json::Value) {
self.merged.insert(update.clone());
}
}
/// Builds an `Update::Partial` whose instruction is a combined
/// `EcmascriptMergedUpdate` covering `entries` and `chunks`. Empty maps are
/// omitted so an empty `entries`/`chunks` field never appears in the payload.
///
/// Passing empty maps produces an instruction with only `type:
/// "EcmascriptMergedUpdate"`, used to advance `VersionState` to `to` without
/// the JS consumer applying anything: it sees a `partial` event with nothing
/// to apply and short-circuits.
pub fn merged_partial_update(
to: TraitRef<Box<dyn Version>>,
entries: FxHashMap<String, serde_json::Value>,
chunks: FxHashMap<String, serde_json::Value>,
) -> Update {
let mut instruction = serde_json::Map::new();
instruction.insert(
"type".to_string(),
serde_json::Value::String("EcmascriptMergedUpdate".to_string()),
);
if !entries.is_empty() {
instruction.insert(
"entries".to_string(),
serde_json::Value::Object(entries.into_iter().collect()),
);
pub fn is_empty(&self) -> bool {
self.chunks.is_empty() && self.merged.is_empty()
}
if !chunks.is_empty() {
pub fn build(self, to: TraitRef<Box<dyn Version>>) -> Update {
let mut instruction = serde_json::Map::new();
instruction.insert(
"chunks".to_string(),
serde_json::Value::Object(chunks.into_iter().collect()),
"type".to_string(),
serde_json::Value::String("ChunkListUpdate".to_string()),
);
if !self.chunks.is_empty() {
instruction.insert(
"chunks".to_string(),
serde_json::Value::Object(self.chunks.into_iter().collect()),
);
}
if !self.merged.is_empty() {
instruction.insert(
"merged".to_string(),
serde_json::Value::Array(self.merged.into_iter().collect()),
);
}
Update::Partial(PartialUpdate {
to,
instruction: Arc::new(serde_json::Value::Object(instruction)),
})
}
Update::Partial(PartialUpdate {
to,
instruction: Arc::new(serde_json::Value::Object(instruction)),
})
}
/// Per-chunk [`Update`]s computed against an `AggregateHmrVersion` snapshot.
......
......@@ -1363,12 +1363,45 @@ impl AppEndpoint {
{
client_assets.extend(assets.all_assets().await?.iter().copied());
}
let mut ssr_client_reference_chunks: Vec<ResolvedVc<Box<dyn OutputAsset>>> = vec![];
for &assets in client_references_chunks_ref
.client_component_ssr_chunks
.values()
{
// TODO(alexkirsz) In which manifest does this go?
server_assets.extend(assets.all_assets().await?.iter().copied());
let all = assets.all_assets().await?;
server_assets.extend(all.iter().copied());
ssr_client_reference_chunks.extend(all.iter().copied());
}
// In development, register a server-side HMR chunk list that owns all
// client-component SSR chunks. This is a bit of a hack to provide a single subscription
// point for these chunks, which aren't normally reachable from the rest of the server-side
// chunk lists.
//
// TODO: This anchor can go away once edges to chunk references are represented explicitly
// in the chunk graph (rsc chunk -> ssr chunk), which would make the SSR chunks reachable
// from the RSC chunk lists directly.
if is_app_page
&& runtime == NextRuntime::NodeJs
&& project
.client_compile_time_info()
.await?
.hot_module_replacement_enabled
{
let ssr_hmr_chunk_list_path = server_path.join(&format!(
"app{original_name}/client-components-ssr.js",
original_name = app_entry.original_name
))?;
let ssr_hmr_chunks = project
.server_chunking_context(process_client_assets)
.server_hmr_chunk_list(
ssr_hmr_chunk_list_path,
Vc::cell(ssr_client_reference_chunks),
)
.to_resolved()
.await?;
server_assets.insert(ssr_hmr_chunks);
}
// In development, register a page-specific HMR chunk list that owns all client
......
......@@ -96,10 +96,7 @@ use turbopack_node::worker_threads_backend;
use turbopack_nodejs::NodeJsChunkingContext;
use crate::{
aggregate_hmr::{
AggregateHmrVersion, DiffResult, diff_chunks_against, merge_ecmascript_merged_update,
merged_partial_update,
},
aggregate_hmr::{AggregateHmrVersion, ChunkListUpdateBuilder, DiffResult, diff_chunks_against},
app::{AppProject, OptionAppProject},
empty::EmptyEndpoint,
entrypoints::Entrypoints,
......@@ -2478,6 +2475,17 @@ impl Project {
})
}
#[turbo_tasks::function]
async fn aggregate_hmr_root_path(
self: Vc<Self>,
target: HmrTarget,
) -> Result<Vc<FileSystemPath>> {
match target {
HmrTarget::Client => bail!("aggregate HMR is not implemented for the client"),
HmrTarget::Server => Ok(self.node_root().await?.join("server/app")?.cell()),
}
}
/// Get HMR content by chunk_name for the specified target.
#[turbo_tasks::function]
async fn hmr_content(
......@@ -2585,7 +2593,7 @@ impl Project {
let Some(map) = this.await?.versioned_content_map else {
bail!("must be in dev mode to hmr")
};
let root = this.hmr_root_path(target).owned().await?;
let root = this.aggregate_hmr_root_path(target).owned().await?;
AggregateHmrVersion::from_map(*map, &root).await
}
let version_op = aggregate_hmr_version_operation(self, target);
......@@ -2602,8 +2610,13 @@ impl Project {
}
/// Aggregate counterpart to [`Self::hmr_update`]: a single `Update` whose
/// `EcmascriptMergedUpdate` is the union of per-chunk diffs under
/// `target`'s root.
/// combined `ChunkListUpdate` is the union of the per-entry-chunk diffs
/// under `target`'s root.
///
/// Each tracked entry chunk's own update is a `ChunkListUpdate` (carrying
/// the module deltas for its shared chunks via the merger) or a bare
/// `EcmascriptMergedUpdate`; both are folded into one `ChunkListUpdate` that
/// the runtime applies exactly as it would a single chunk list.
///
/// All-or-nothing restart: any chunk needing `Total`/`Missing` escalates
/// the whole batch to `Total` (the runtime can't partially restart). New
......@@ -2622,7 +2635,7 @@ impl Project {
let Some(map) = self.await?.versioned_content_map else {
bail!("must be in dev mode to hmr")
};
let root = self.hmr_root_path(target).owned().await?;
let root = self.aggregate_hmr_root_path(target).owned().await?;
let chunks_versioned_content = map.hmr_chunks_in_path(&root).await?;
// No chunks to diff yet (e.g. before any endpoints have been written).
......@@ -2648,13 +2661,10 @@ impl Project {
// the *next* change produces a real diff; returning `Total` instead would
// force a needless full re-evaluation.
if chunk_updates.is_empty() && !has_new_chunks {
return Ok(
merged_partial_update(to_ref, FxHashMap::default(), FxHashMap::default()).cell(),
);
return Ok(ChunkListUpdateBuilder::default().build(to_ref).cell());
}
let mut combined_entries: FxHashMap<String, serde_json::Value> = FxHashMap::default();
let mut combined_chunks: FxHashMap<String, serde_json::Value> = FxHashMap::default();
let mut builder = ChunkListUpdateBuilder::default();
for (_path, update) in chunk_updates {
match &*update {
Update::None => {}
......@@ -2662,20 +2672,16 @@ impl Project {
return Ok(Update::Total(TotalUpdate { to: to_ref }).cell());
}
Update::Partial(PartialUpdate { instruction, .. }) => {
merge_ecmascript_merged_update(
&mut combined_entries,
&mut combined_chunks,
instruction,
);
builder.add_instruction(instruction);
}
}
}
if combined_entries.is_empty() && combined_chunks.is_empty() && !has_new_chunks {
if builder.is_empty() && !has_new_chunks {
return Ok(Update::None.cell());
}
Ok(merged_partial_update(to_ref, combined_entries, combined_chunks).cell())
Ok(builder.build(to_ref).cell())
}
/// Gets a list of all HMR chunk names that can be subscribed to for the
......
......@@ -15,7 +15,7 @@ use turbopack_core::{
version::OptionVersionedContent,
};
use crate::aggregate_hmr::HmrChunkWithContent;
use crate::aggregate_hmr::{HmrChunkWithContent, is_entry_chunk_list_content};
#[derive(
Clone, TraceRawVcs, PartialEq, Eq, ValueDebugFormat, Debug, NonLocalValue, Encode, Decode,
......@@ -90,16 +90,18 @@ impl VersionedContentMap {
.resolved_cell()
}
/// Lists every HMR-eligible chunk under `root` with its [`VersionedContent`],
/// sorted by path.
/// Lists the aggregate-HMR *entry* chunks under `root` with their
/// [`VersionedContent`], sorted by path. Only entry-chunk-list content is
/// returned (see [`is_entry_chunk_list_content`]). Callers scope which
/// entries are included by narrowing `root` (e.g. the aggregate server-HMR
/// subscription passes `server/app` to include App Router entries only).
///
/// `map_path_to_op` is an `FxHashMap`, whose iteration order
/// depends on bucket layout rather than insertion order, so the same set of
/// paths can come out in a different order across calls.
///
/// Since this map contains entries that span server and client contexts, changes for one
/// context can shift the internals of the map, making iteration order different for the same
/// set of paths.
/// `map_path_to_op` is an `FxHashMap`, whose iteration order depends on
/// bucket layout rather than insertion order, so the same set of paths can
/// come out in a different order across calls. Since this map contains
/// entries that span server and client contexts, changes for one context
/// can shift the internals of the map, making iteration order different
/// for the same set of paths.
pub async fn hmr_chunks_in_path(
self: Vc<Self>,
root: &FileSystemPath,
......@@ -116,10 +118,6 @@ impl VersionedContentMap {
.into_iter()
.filter_map(|path| {
let rel = root.get_path_to(&path)?;
// Skip source maps: they aren't HMR-eligible chunks.
if rel.ends_with(".map") {
return None;
}
Some((RcStr::from(rel), path))
})
.map(|(name, path)| async move {
......@@ -132,6 +130,14 @@ impl VersionedContentMap {
return Ok(None);
}
let content = asset.versioned_content().to_resolved().await?;
// *Important*: only chunk lists are subscribed to. Individual chunks are already
// covered by the chunk list that owns them, so including them here
// would produce duplicate updates for the same change.
if !is_entry_chunk_list_content(content) {
return Ok(None);
}
Ok(Some(HmrChunkWithContent {
path: name,
content,
......
......@@ -2400,15 +2400,14 @@ async fn get_source_map_rope(
return Ok(FileContent::NotFound.cell());
};
let Some(chunk_base_unix) = project.node_root().await?.get_path_to(&fs_path) else {
let node_root = project.node_root().await?;
let Some(chunk_base_unix) = node_root.get_path_to(&fs_path).map(ToOwned::to_owned) else {
// The path is not within the dist dir
return Ok(FileContent::NotFound.cell());
};
let client_path = project
.client_relative_path()
.await?
.join(chunk_base_unix)?;
let client_relative_path = project.client_relative_path().await?;
let client_path = client_relative_path.join(&chunk_base_unix)?;
// `fs_path` is the server path: it's inside `node_root`, and `output_fs` is the filesystem
// that `node_root` uses.
......@@ -2421,6 +2420,24 @@ async fn get_source_map_rope(
// chunks.
map = container.get_source_map(client_path, module.clone());
if !map.await?.is_content() {
// An older revision of a chunk's sourcemap may be requested by an HMR client. We
// remove stale entries from the VersionStateMap but a client may be holding onto a
// reference to a stale chunk and requesting its sourcmemap via the error
// overlay.
//
// This exists because we don't have logic for the server hmr client to mark which
// chunks it's no longer using.
//
// Fall back to reading from the filesystem.
let map_relative = format!("{chunk_base_unix}.map");
let server_map = node_root.join(&map_relative)?.read();
if server_map.await?.is_content() {
return Ok(server_map);
}
let client_map = client_relative_path.join(&map_relative)?.read();
if client_map.await?.is_content() {
return Ok(client_map);
}
bail!("chunk/module {sys_path:?} (module: {module:?}) is missing a sourcemap");
}
}
......
......@@ -1456,5 +1456,7 @@
"1455": "Page \"%s\" returned incomplete params from \"generateStaticParams()\". With \"output: export\", every params object must include all dynamic route parameters. Missing: %s. See more info here: https://nextjs.org/docs/messages/generate-static-params",
"1456": "Expected an HTML size for prerendered app route \"%s\"",
"1457": "Expected prerender classification data for PPR route \"%s\"",
"1458": "Expected complete prerender classification for route \"%s\""
"1458": "Expected complete prerender classification for route \"%s\"",
"1459": "[Server HMR] unreachable: unknown HMR instruction type %s",
"1460": "[Server HMR] unreachable: unexpected update instruction type %s"
}
......@@ -234,20 +234,28 @@ export type Update = IssuesUpdate | PartialUpdate
* The runtime file cannot import from this ES module without triggering module semantics,
* so we maintain a copy there. Please keep both definitions in sync.
*/
export interface NodeJsEcmascriptMergedUpdate {
type: 'EcmascriptMergedUpdate'
entries?: Record<
string,
{ code: string; url: string; map?: string | undefined }
>
chunks?: Record<
string,
| { type: 'added' | 'deleted'; modules?: string[] }
| { type: 'partial'; added?: string[]; deleted?: string[] }
>
}
export interface NodeJsChunkListUpdate {
type: 'ChunkListUpdate'
merged?: NodeJsEcmascriptMergedUpdate[]
chunks?: Record<string, { type: 'added' | 'deleted' | 'total' | 'partial' }>
}
export interface NodeJsPartialHmrUpdate extends BaseUpdate {
type: 'partial'
instruction: {
type: 'EcmascriptMergedUpdate'
entries?: Record<
string,
{ code: string; url: string; map?: string | undefined }
>
chunks?: Record<
string,
| { type: 'added' | 'deleted'; modules?: string[] }
| { type: 'partial'; added?: string[]; deleted?: string[] }
>
}
instruction: NodeJsEcmascriptMergedUpdate | NodeJsChunkListUpdate
}
export interface NodeJsRestartHmrUpdate {
......
......@@ -38,6 +38,7 @@ import {
} from './middleware-turbopack'
import { PageNotFoundError } from '../../shared/lib/utils'
import { debounce } from '../utils'
import { clearManifestCache } from '../load-manifest.external'
import { deleteCache } from './require-cache'
import {
clearAllModuleContexts,
......@@ -169,12 +170,45 @@ declare global {
var __turbopack_server_hmr_handlers__: Map<string, unknown> | undefined
}
/**
* Collects the output chunk paths touched by a partial HMR update. Both
* single-chunk `EcmascriptMergedUpdate`s and `ChunkListUpdate`s (which nest
* per-chunk deltas inside `merged`) are flattened so the manifest cache can be
* invalidated for every affected chunk after a successful apply.
*/
function collectUpdatedChunkPaths(
instruction: NodeJsPartialHmrUpdate['instruction']
): string[] {
const paths = new Set<string>()
if (instruction.type === 'EcmascriptMergedUpdate') {
for (const chunkPath of Object.keys(instruction.chunks ?? {})) {
paths.add(chunkPath)
}
} else if (instruction.type === 'ChunkListUpdate') {
for (const chunkPath of Object.keys(instruction.chunks ?? {})) {
paths.add(chunkPath)
}
for (const merged of instruction.merged ?? []) {
for (const chunkPath of Object.keys(merged.chunks ?? {})) {
paths.add(chunkPath)
}
}
} else {
throw new Error(
`[Server HMR] unreachable: unknown HMR instruction type ${(instruction as { type: string }).type}`
)
}
return Array.from(paths)
}
function setupServerHmr(
project: Project,
{
reEvaluateAllModulesExpensive,
onApplied,
}: {
reEvaluateAllModulesExpensive: () => void | Promise<void>
onApplied: (chunkPaths: string[]) => void | Promise<void>
}
) {
async function runSubscription() {
......@@ -205,8 +239,14 @@ function setupServerHmr(
// knows how to apply; `ChunkListUpdate` is browser-only. Anything else is
// unknown to us, so ignore it rather than evicting the module cache.
const instruction = update.instruction
if (!instruction || instruction.type !== 'EcmascriptMergedUpdate') {
continue
if (
!instruction ||
(instruction.type !== 'EcmascriptMergedUpdate' &&
instruction.type !== 'ChunkListUpdate')
) {
throw new Error(
`[Server HMR] unreachable: unexpected update instruction type ${(instruction as { type: string }).type}`
)
}
// No handler registered yet (before first request, or right after
......@@ -222,8 +262,18 @@ function setupServerHmr(
__turbopack_server_hmr_apply__(update)
} catch {
// A matching runtime tried the apply and threw. Evict require.cache
// so the next request loads fresh. (A no-match update is a no-op.)
// so the next request loads fresh, then skip onApplied. (A no-match
// update is a no-op and does not throw.)
await reEvaluateAllModulesExpensive()
continue
}
const updatedChunkPaths = collectUpdatedChunkPaths(instruction)
// An empty partial only advances the version state (e.g. the seed
// transition or a new endpoint); nothing changed on disk, so don't
// invalidate manifests or ping browsers to refetch RSC.
if (updatedChunkPaths.length > 0) {
await onApplied(updatedChunkPaths)
}
} else {
await reEvaluateAllModulesExpensive()
......@@ -584,9 +634,11 @@ export async function createHotReloaderTurbopack(
} else {
// Figure out if the server files have changed
let hasChange = false
const currentPaths = new Set<string>()
for (const { path, contentHash } of writtenEndpoint.serverPaths) {
// We ignore source maps
if (path.endsWith('.map')) continue
currentPaths.add(path)
const localKey = `${key}:${path}`
const localHash = serverPathState.get(localKey)
const globalHash = serverPathState.get(path)
......@@ -607,11 +659,27 @@ export async function createHotReloaderTurbopack(
}
}
const localKeyPrefix = `${key}:`
for (const pathKey of serverPathState.keys()) {
if (
pathKey.startsWith(localKeyPrefix) &&
!currentPaths.has(pathKey.slice(localKeyPrefix.length))
) {
serverPathState.delete(pathKey)
hasChange = true
}
}
if (!hasChange) {
return false
}
}
// Edge does not participate in server HMR.
if (writtenEndpoint.type === 'edge') {
void clearAllModuleContexts()
}
const serverPaths = writtenEndpoint.serverPaths.map(({ path: p }) =>
join(distDir, p)
)
......@@ -752,16 +820,23 @@ export async function createHotReloaderTurbopack(
}
}
function sendEnqueuedMessages() {
function hasCompilationErrors() {
for (const [, issueMap] of currentEntryIssues) {
if (
[...issueMap.values()].filter((i) => i.severity !== 'warning').length >
0
) {
// During compilation errors we want to delay the HMR events until errors are fixed
return
return true
}
}
return false
}
function sendEnqueuedMessages() {
if (hasCompilationErrors()) {
// During compilation errors we want to delay the HMR events until errors are fixed
return
}
for (const client of [
...clientsWithoutHtmlRequestId,
......@@ -1871,6 +1946,7 @@ export async function createHotReloaderTurbopack(
force: forceDeleteCache,
})
},
serverFastRefresh,
},
})
} finally {
......@@ -2019,6 +2095,18 @@ export async function createHotReloaderTurbopack(
process.exit(1)
})
// Tell browsers to refetch RSC (soft refresh, not full page reload).
// Skip while there are outstanding compilation errors: an RSC refetch would
// 500 and force a full-page navigation, losing client state (e.g. recovering
// from a syntax error). A subsequent successful compile/apply fires this
// again to refresh.
function notifyServerComponentChanges() {
if (hasCompilationErrors()) return
hotReloader.send({
type: HMR_MESSAGE_SENT_TO_BROWSER.SERVER_COMPONENT_CHANGES,
})
}
if (serverFastRefresh) {
setupServerHmr(project, {
reEvaluateAllModulesExpensive: async () => {
......@@ -2045,10 +2133,18 @@ export async function createHotReloaderTurbopack(
resetFetch()
// Tell browsers to refetch RSC (soft refresh, not full page reload)
hotReloader.send({
type: HMR_MESSAGE_SENT_TO_BROWSER.SERVER_COMPONENT_CHANGES,
})
notifyServerComponentChanges()
},
onApplied: (chunkPaths: string[]) => {
// Clear the evalManifest() shared cache for each updated chunk so the
// next RSC render picks up the HMR-applied module changes. Unlike
// a full restart, this does NOT clear require.cache — the HMR-applied
// modules in devModuleCache must persist for dep preservation.
for (const chunkPath of chunkPaths) {
clearManifestCache(join(distDir, chunkPath))
}
notifyServerComponentChanges()
},
})
}
......
......@@ -142,6 +142,14 @@ type HandleRouteTypeHooks = {
handleWrittenEndpoint: HandleWrittenEndpoint
subscribeToChanges: StartChangeSubscription
handleServerComponentChanges?: () => void
// When Turbopack server fast refresh is enabled, the aggregate server-HMR
// subscription (setupServerHmr `onApplied` in hot-reloader-turbopack.ts)
// owns the browser refresh signal for app-page RSC changes and only fires
// after the server module cache is refreshed. In that mode the per-page
// `rscHmrEndpoint` subscription must NOT also send SERVER_COMPONENT_CHANGES,
// or every edit triggers two RSC refetches (the first immediately
// superseded).
serverFastRefresh?: boolean
}
export async function handleRouteType({
......@@ -365,6 +373,21 @@ export async function handleRouteType({
}
// Report the next compilation again
readyIds?.delete(pathname)
// When server fast refresh is enabled, the aggregate server-HMR
// subscription sends SERVER_COMPONENT_CHANGES after applying the
// update in-process. Sending here too would double the refresh.
//
// But the aggregate subscription only fires when there is a live
// server-HMR handler registered (i.e. the page has rendered at
// least once). When recovering from a build error the page never
// rendered, so no handler exists, the aggregate stays silent, and
// this per-page send is the only thing that clears the redbox.
// Only suppress when a handler is actually live to own the refresh.
const hasLiveServerHmrHandler =
(globalThis.__turbopack_server_hmr_handlers__?.size ?? 0) > 0
if (hooks?.serverFastRefresh && hasLiveServerHmrHandler) {
return
}
hooks?.handleServerComponentChanges?.()
},
(e) => {
......
......@@ -403,7 +403,7 @@ describe('ReactRefreshLogBox app', () => {
"stack": [
"ClickCount.render Child.js (4:11)",
"Home index.js (6:7)",
"<FIXME-file-protocol>",
"Page app/page.js (4:10)",
],
}
`)
......
......@@ -528,7 +528,7 @@ describe('Error recovery app', () => {
"stack": [
"Child child.js (3:9)",
"Index index.js (6:7)",
"<FIXME-file-protocol>",
"Page app/client/page.js (4:10)",
],
}
`)
......@@ -828,7 +828,7 @@ describe('Error recovery app', () => {
"stack": [
"Foo Foo.js (3:3)",
"FunctionDefault index.js (4:10)",
"<FIXME-file-protocol>",
"Page app/page.js (4:10)",
],
}
`)
......
'use client'
export function ClientGreeting({ text }: { text: string }) {
return <p id="client-greeting">{text}</p>
}
// An always-imported dependency that is never modified during the test.
// Its evaluation timestamp detects whether clear() fired.
export const depEvalTime = Date.now()
import { depEvalTime } from './dep'
const evaluatedAt = Date.now()
export default function Page() {
return (
<div>
<p id="greeting">hello world</p>
<p id="module-eval-time">Module Evaluated At: {evaluatedAt}</p>
<p id="dep-eval-time">Dep Evaluated At: {depEvalTime}</p>
<p id="client-component">not imported yet</p>
</div>
)
}
// An always-imported (synchronous) dependency that is never modified during the
// test. Its evaluation timestamp is used to detect whether clear() fired:
// clear() wipes require.cache and re-evaluates every module, which would change
// this timestamp. A partial HMR apply leaves unmodified modules untouched.
export const depEvalTime = Date.now()
// Added to lazy.ts's import graph during the test. It starts out unreferenced,
// then the test adds an import of it inside the dynamically-imported module.
// This changes the dynamic chunk's content hash and thus its path.
export const lazyNewModuleValue = 'from-lazy-new-module'
// This module is loaded via dynamic import() on the server, so it lands in its
// own chunk under server/chunks/ rather than the entry chunk. Editing it (or
// adding a new import to it) changes that chunk's content hash and therefore its
// path — the exact "availability info / chunk rename" scenario, but for a
// dynamically-imported chunk instead of the synchronous entry chunk graph.
export const lazyValue = 'lazy-v0'
import { depEvalTime } from './dep'
const evaluatedAt = Date.now()
export default async function Page() {
// Load a module in its own chunk via a server-side dynamic import(). The
// chunk lives under server/chunks/ and is not part of the entry chunk's
// synchronous chunk list, so its updates are surfaced through the async
// loader module rather than the entry chunk list.
const { lazyValue } = await import('./lazy')
return (
<div>
<p id="greeting">hello world</p>
<p id="module-eval-time">Module Evaluated At: {evaluatedAt}</p>
<p id="dep-eval-time">Dep Evaluated At: {depEvalTime}</p>
<p id="lazy-value">{lazyValue}</p>
<p id="lazy-new-module-value">not imported yet</p>
</div>
)
}
// An always-imported dependency that is never modified during the test.
// Its evaluation timestamp is used to detect whether clear() fired: clear()
// wipes require.cache and re-evaluates every module, which would change this
// timestamp. A partial HMR apply leaves unmodified modules untouched.
export const depEvalTime = Date.now()
// This module is dynamically imported by the page during HMR tests.
// It starts not being imported, then the test adds the import.
export const newModuleValue = 'from-new-module'
export const newModuleEvalTime = Date.now()
import { depEvalTime } from './dep'
const pageLoadId = Math.random().toString(36).slice(2)
const evaluatedAt = Date.now()
export default function Page() {
return (
<div>
<p id="greeting">hello world</p>
<p id="page-load-id">Page Load ID: {pageLoadId}</p>
<p id="module-eval-time">Module Evaluated At: {evaluatedAt}</p>
<p id="dep-eval-time">Dep Evaluated At: {depEvalTime}</p>
<p id="new-module-value">not imported yet</p>
</div>
)
}
......@@ -124,7 +124,6 @@ const STATS_RELATIVE_PATH = '.next/warm-restart-task-stats.json'
expect(missed).toMatchInlineSnapshot(`
[
"<turbopack_browser::ecmascript::list::content::EcmascriptDevChunkListContent as dyn turbopack_core::version::VersionedContent>::update",
"<turbopack_core::version::FileHashVersion as dyn turbopack_core::version::Version>::id",
"next_api::project::Project::hmr_update",
"next_api::project::Project::hmr_version_state",
"next_napi_bindings::next_api::project::hmr_update_with_issues_operation",
......
......@@ -38,7 +38,7 @@ enum CurrentChunkMethodWithData {
/// Contents of an [`EcmascriptDevChunkList`].
#[turbo_tasks::value]
pub(super) struct EcmascriptDevChunkListContent {
pub struct EcmascriptDevChunkListContent {
current_chunk_method: CurrentChunkMethodWithData,
#[bincode(with = "turbo_bincode::indexmap")]
pub(super) chunks_contents: FxIndexMap<String, ResolvedVc<Box<dyn VersionedContent>>>,
......
pub(crate) mod asset;
pub(crate) mod content;
pub mod content;
pub(crate) mod chunk;
pub(crate) mod content;
pub(crate) mod evaluate;
pub(crate) mod list;
pub mod list;
pub(crate) mod merged;
pub(crate) mod update;
pub(crate) mod version;
......
......@@ -57,10 +57,20 @@ if (handlers.size === 0) {
): void => {
const registry: Map<string, HmrHandlerEntry> =
globalThis.__turbopack_server_hmr_handlers__ ?? new Map()
const updateChunkPaths = Object.keys(update.instruction?.chunks ?? {})
// Chunk paths can appear either directly on the instruction (single-chunk
// updates) or nested inside `merged` entries (chunks covered by a
// merger). Collect both so routing isn't skipped just because a mergeable
// chunk's update only reports its paths inside `merged`.
const updateChunkPaths = new Set<string>([
...Object.keys(update.instruction?.chunks ?? {}),
...(update.instruction?.merged ?? []).flatMap((merged) =>
Object.keys(merged.chunks ?? {})
),
])
const toCall: HmrHandlerEntry[] = []
if (updateChunkPaths.length === 0) {
if (updateChunkPaths.size === 0) {
for (const entry of registry.values()) toCall.push(entry)
} else {
const seen = new Set<string>()
......
......@@ -10,7 +10,7 @@ type NodeJsHmrPayload = {
}
issues: Issue[]
type: 'partial'
instruction: EcmascriptMergedUpdate
instruction: NodeJsEcmascriptMergedUpdate | NodeJsChunkListUpdate
}
/**
......@@ -79,38 +79,79 @@ function handleNodejsUpdate(
}
const instruction = msg.instruction
if (instruction.type !== 'EcmascriptMergedUpdate') {
return
}
try {
const { entries = {}, chunks = {} } = instruction
if (instruction.type === 'ChunkListUpdate') {
// All node ecmascript chunks are mergeable, so a `total`/`partial` here
// means a non-mergeable asset changed in an unsupported way. Escalate
// to a full clear() rather than leave stale factories in memory.
for (const [chunkPath, chunkUpdate] of Object.entries(
instruction.chunks ?? {}
)) {
if (chunkUpdate.type === 'total' || chunkUpdate.type === 'partial') {
throw new Error(
`unsupported '${chunkUpdate.type}' update for chunk ${chunkPath}`
)
}
}
const evalModuleEntry = (entry: EcmascriptModuleEntry) => {
// eslint-disable-next-line no-eval
return (0, eval)(entry.map ? inlineSourcemaps(entry) : entry.code)
if (instruction.merged) {
for (const merged of instruction.merged) {
applyEcmascriptMergedUpdate(merged, moduleFactories, devModuleCache)
}
}
return
}
const { added, modified } = computeChangedModules(
entries,
chunks,
undefined // no chunkModulesMap for Node.js
)
// Use shared HMR update implementation
applyEcmascriptMergedUpdateShared({
added,
modified,
disposedModules: [], // no disposedModules for Node.js (no chunk management)
evalModuleEntry,
instantiateModule,
applyModuleFactoryName: () => {}, // Node doesn't use this
moduleFactories,
devModuleCache,
autoAcceptRootModules: true,
})
if (instruction.type === 'EcmascriptMergedUpdate') {
applyEcmascriptMergedUpdate(instruction, moduleFactories, devModuleCache)
return
}
} catch (e) {
console.error('[Server HMR] Update failed, full reload needed:', e)
throw e
}
}
function applyEcmascriptMergedUpdate(
instruction: NodeJsEcmascriptMergedUpdate,
moduleFactories: ModuleFactories,
devModuleCache: ModuleCache<HotModule>
): void {
const { entries = {}, chunks = {} } = instruction
const evalModuleEntry = (entry: EcmascriptModuleEntry) => {
const code = entry.map ? inlineSourcemaps(entry) : entry.code
// eslint-disable-next-line no-eval
return (0, eval)(`(require) => ${code}`)(require)
}
const { added, modified } = computeChangedModules(
entries,
chunks,
undefined // no chunkModulesMap for Node.js
)
// Modules that appear in an "added" chunk but already exist in the cache
// were moved to a renamed chunk. Treat them as modified so the dependency
// walk runs and they get re-instantiated with the new factory.
for (const [moduleId, entry] of added) {
if (entry != null && devModuleCache[moduleId] != null) {
added.delete(moduleId)
modified.set(moduleId, entry)
}
}
// Use shared HMR update implementation
applyEcmascriptMergedUpdateShared({
added,
modified,
disposedModules: [], // no disposedModules for Node.js (no chunk management)
evalModuleEntry,
instantiateModule,
applyModuleFactoryName: () => {}, // Node doesn't use this
moduleFactories,
devModuleCache,
autoAcceptRootModules: true,
})
}
......@@ -7,16 +7,28 @@
* Keep NodeJsPartialHmrUpdate in sync with the copy in packages/next/src/build/swc/types.ts.
*/
interface NodeJsEcmascriptMergedUpdate {
type: 'EcmascriptMergedUpdate'
entries?: Record<
string,
{ code: string; url: string; map?: string | undefined }
>
chunks?: Record<
string,
| { type: 'added' | 'deleted'; modules?: string[] }
| { type: 'partial'; added?: string[]; deleted?: string[] }
>
}
interface NodeJsChunkListUpdate {
type: 'ChunkListUpdate'
merged?: NodeJsEcmascriptMergedUpdate[]
chunks?: Record<string, { type: 'added' | 'deleted' | 'total' | 'partial' }>
}
interface NodeJsPartialHmrUpdate {
type: 'partial'
instruction: {
type: 'EcmascriptMergedUpdate'
entries: Record<
string,
{ code: string; url: string; map?: string | undefined }
>
chunks?: Record<string, { type: 'partial' }>
}
instruction: NodeJsEcmascriptMergedUpdate | NodeJsChunkListUpdate
}
interface NodeJsRestartHmrUpdate {
......
......@@ -35,7 +35,8 @@ use turbopack_ecmascript::{
use turbopack_ecmascript_runtime::RuntimeType;
use crate::ecmascript::node::{
chunk::EcmascriptBuildNodeChunk, entry::chunk::EcmascriptBuildNodeEntryChunk,
chunk::EcmascriptBuildNodeChunk,
entry::{chunk::EcmascriptBuildNodeEntryChunk, chunk_list::EcmascriptBuildNodeChunkList},
};
/// A builder for [`Vc<NodeJsChunkingContext>`].
......@@ -313,6 +314,31 @@ impl NodeJsChunkingContext {
pub fn asset_prefix(&self) -> Vc<Option<RcStr>> {
Vc::cell(self.asset_prefix.clone())
}
/// Creates a standalone server-HMR tracking anchor at `path` covering
/// `chunks`, without producing an evaluate chunk.
///
/// Unlike the browser's `hmr_chunk_list`, the caller supplies an explicit
/// output `path` so the anchor can be placed alongside the App Router
/// entries it belongs to (under `server/app/`). This is what lets the
/// aggregate server-HMR subscription scope tracking to App Router: the
/// anchor for client-component SSR chunks (which are physically emitted
/// under the shared `server/chunks/ssr/`) is registered under the app
/// entry's directory so it rides the same App Router scope.
#[turbo_tasks::function]
pub async fn server_hmr_chunk_list(
self: ResolvedVc<Self>,
path: FileSystemPath,
chunks: Vc<OutputAssets>,
) -> Result<Vc<Box<dyn OutputAsset>>> {
#[cfg(debug_assertions)]
if !matches!(*self.runtime_type().await?, RuntimeType::Development) {
bail!("server_hmr_chunk_list can only be used in development");
}
Ok(Vc::upcast(EcmascriptBuildNodeChunkList::new(
*self, path, chunks,
)))
}
}
impl NodeJsChunkingContext {
......
......@@ -7,7 +7,9 @@ use turbopack_core::{
code_builder::{Code, CodeBuilder},
output::OutputAsset,
source_map::{GenerateSourceMap, SourceMapAsset},
version::{Update, Version, VersionedContent},
version::{
MergeableVersionedContent, Update, Version, VersionedContent, VersionedContentMerger,
},
};
use turbopack_ecmascript::{
chunk::{EcmascriptChunkContent, EcmascriptChunkContentEntries},
......@@ -16,8 +18,8 @@ use turbopack_ecmascript::{
};
use super::{
chunk::EcmascriptBuildNodeChunk, update::update_node_chunk,
version::EcmascriptBuildNodeChunkVersion,
chunk::EcmascriptBuildNodeChunk, merged::merger::EcmascriptBuildNodeChunkContentMerger,
update::update_node_chunk, version::EcmascriptBuildNodeChunkVersion,
};
use crate::NodeJsChunkingContext;
......@@ -140,3 +142,11 @@ impl VersionedContent for EcmascriptBuildNodeChunkContent {
Ok(update_node_chunk(self, from_version).await?.cell())
}
}
#[turbo_tasks::value_impl]
impl MergeableVersionedContent for EcmascriptBuildNodeChunkContent {
#[turbo_tasks::function]
fn get_merger(&self) -> Vc<Box<dyn VersionedContentMerger>> {
Vc::upcast(EcmascriptBuildNodeChunkContentMerger::new())
}
}
......@@ -14,11 +14,15 @@ use turbopack_core::{
OutputAssetsWithReferenced,
},
source_map::{GenerateSourceMap, SourceMapAsset},
version::VersionedContent,
};
use turbopack_ecmascript::{chunk::EcmascriptChunkPlaceable, utils::StringifyJs};
use turbopack_ecmascript_runtime::RuntimeType;
use super::runtime::EcmascriptBuildNodeRuntimeChunk;
use super::{
chunk_list_content::EcmascriptBuildNodeChunkListContent,
runtime::EcmascriptBuildNodeRuntimeChunk,
};
use crate::NodeJsChunkingContext;
/// An Ecmascript chunk that loads a list of parallel chunks, then instantiates
......@@ -223,6 +227,23 @@ impl Asset for EcmascriptBuildNodeEntryChunk {
FileContent::Content(File::from(code.source_code().clone())).cell(),
))
}
#[turbo_tasks::function]
fn versioned_content(self: Vc<Self>) -> Vc<Box<dyn VersionedContent>> {
Vc::upcast(self.chunk_list_content())
}
}
#[turbo_tasks::value_impl]
impl EcmascriptBuildNodeEntryChunk {
#[turbo_tasks::function]
fn chunk_list_content(&self) -> Vc<EcmascriptBuildNodeChunkListContent> {
EcmascriptBuildNodeChunkListContent::new(
*self.chunking_context,
*self.other_chunks,
*self.references,
)
}
}
#[turbo_tasks::value_impl]
......
use turbo_rcstr::rcstr;
use turbo_tasks::{ResolvedVc, ValueToString, Vc};
use turbo_tasks_fs::{File, FileContent, FileSystemPath};
use turbopack_core::{
asset::{Asset, AssetContent},
output::{OutputAsset, OutputAssets, OutputAssetsReference, OutputAssetsWithReferenced},
version::VersionedContent,
};
use super::chunk_list_content::EcmascriptBuildNodeChunkListContent;
use crate::NodeJsChunkingContext;
/// A server-side HMR tracking anchor for a set of chunks built outside any
/// entry's own module graph (e.g. client-component SSR chunks produced via
/// separate `chunk_group(IsolatedMerged)` calls).
///
/// It is never `require()`d at runtime — it exists only so the aggregate
/// server-HMR subscription has a *stable-ident* [`VersionedContent`]
/// ([`EcmascriptBuildNodeChunkListContent`], whose version is a
/// `ChunkListVersion`) covering these chunks. The stable ident means adding,
/// removing, or renaming a covered chunk doesn't force a `TotalUpdate`
/// (restart); edits ride the merged `ChunkListUpdate` just like the entry's own
/// chunk list.
#[turbo_tasks::value(shared)]
#[derive(ValueToString)]
#[value_to_string("Ecmascript Build Node Chunk List")]
pub(crate) struct EcmascriptBuildNodeChunkList {
pub(super) chunking_context: ResolvedVc<NodeJsChunkingContext>,
/// Explicit output path for this tracking anchor. The caller controls
/// placement so the anchor can live alongside the entries it belongs to
/// (e.g. under `server/app/`), which is how the aggregate server-HMR
/// subscription scopes what it tracks.
pub(super) path: FileSystemPath,
pub(super) chunks: ResolvedVc<OutputAssets>,
}
#[turbo_tasks::value_impl]
impl EcmascriptBuildNodeChunkList {
#[turbo_tasks::function]
pub fn new(
chunking_context: ResolvedVc<NodeJsChunkingContext>,
path: FileSystemPath,
chunks: ResolvedVc<OutputAssets>,
) -> Vc<Self> {
EcmascriptBuildNodeChunkList {
chunking_context,
path,
chunks,
}
.cell()
}
#[turbo_tasks::function]
fn own_content(&self) -> Vc<EcmascriptBuildNodeChunkListContent> {
EcmascriptBuildNodeChunkListContent::new_from_chunks(*self.chunking_context, *self.chunks)
}
}
#[turbo_tasks::value_impl]
impl OutputAssetsReference for EcmascriptBuildNodeChunkList {
#[turbo_tasks::function]
fn references(&self) -> Vc<OutputAssetsWithReferenced> {
OutputAssetsWithReferenced::from_assets(*self.chunks)
}
}
#[turbo_tasks::value_impl]
impl OutputAsset for EcmascriptBuildNodeChunkList {
#[turbo_tasks::function]
fn path(&self) -> Vc<FileSystemPath> {
// Use the explicit path from the caller. The path deliberately does not
// encode the covered chunks' idents, so it stays stable when a chunk is
// added/removed/renamed (which is what insulates the aggregate HMR
// version from `TotalUpdate`s).
self.path.clone().cell()
}
}
#[turbo_tasks::value_impl]
impl Asset for EcmascriptBuildNodeChunkList {
#[turbo_tasks::function]
fn content(self: Vc<Self>) -> Vc<AssetContent> {
// Never loaded at runtime, but this can't `bail!`: as a real
// `OutputAsset` in `server_assets` it flows through `all_asset_paths`,
// which hashes every asset's content to build the server paths list.
// Erroring here fails the whole endpoint write. Emit an empty stub
// instead.
AssetContent::file(FileContent::Content(File::from(rcstr!(""))).cell())
}
#[turbo_tasks::function]
fn versioned_content(self: Vc<Self>) -> Vc<Box<dyn VersionedContent>> {
Vc::upcast(self.own_content())
}
}
use anyhow::{Result, bail};
use turbo_tasks::{FxIndexMap, ResolvedVc, TryJoinIterExt, Vc};
use turbo_tasks_fs::FileSystemPath;
use turbopack_core::{
asset::{Asset, AssetContent},
chunk::ChunkingContext,
output::{
ExpandOutputAssetsInput, OutputAsset, OutputAssets, OutputAssetsReferences,
expand_output_assets,
},
version::{Update, Version, VersionedContent},
};
use turbopack_ecmascript::chunk_list::{
update::update_chunk_list,
version::{ChunkListVersion, compute_chunk_list_version},
};
use crate::NodeJsChunkingContext;
/// Maps each chunk to its `output_root`-relative path and versioned content.
async fn collect_chunks_contents(
output_root: &FileSystemPath,
chunks: impl Iterator<Item = ResolvedVc<Box<dyn OutputAsset>>>,
) -> Result<FxIndexMap<String, ResolvedVc<Box<dyn VersionedContent>>>> {
chunks
.map(async |chunk| {
let chunk_path = chunk.path().await?;
let Some(path) = output_root.get_path_to(&chunk_path) else {
bail!("chunk {chunk_path} is not inside the output root {output_root}");
};
Ok((
path.to_string(),
chunk.versioned_content().to_resolved().await?,
))
})
.try_join()
.await
.map(FxIndexMap::from_iter)
}
/// Never emitted as an asset — the entry chunk already inlines the `R.c(...)`
/// calls for its shared chunks. This exists purely to give the entry a
/// *stable* [`VersionedContent`] identity keyed off a chunk-list ident, so
/// adding/removing/renaming a shared chunk doesn't force a `TotalUpdate`.
///
/// Tracks both synchronous chunks and chunks reachable via async-loader
/// references (dynamic `import()`), so an edit inside a lazy-loaded module
/// still rides the merged `ChunkListUpdate` instead of being missed.
#[turbo_tasks::value]
pub struct EcmascriptBuildNodeChunkListContent {
#[bincode(with = "turbo_bincode::indexmap")]
pub(super) chunks_contents: FxIndexMap<String, ResolvedVc<Box<dyn VersionedContent>>>,
}
#[turbo_tasks::value_impl]
impl EcmascriptBuildNodeChunkListContent {
#[turbo_tasks::function]
pub async fn new(
chunking_context: ResolvedVc<NodeJsChunkingContext>,
chunks: ResolvedVc<OutputAssets>,
references: ResolvedVc<OutputAssetsReferences>,
) -> Result<Vc<Self>> {
let output_root = chunking_context.output_root().owned().await?;
// Expand async-loader references transitively to reach dynamically
// imported chunks. `inner=false`: only follow Reference edges (async
// loaders), not Asset-adjacent files like source maps that aren't part
// of the module graph and can't be hot-reloaded.
let async_chunks = expand_output_assets(
references
.await?
.iter()
.copied()
.map(ExpandOutputAssetsInput::Reference),
false,
)
.await?;
let chunks_contents = collect_chunks_contents(
&output_root,
chunks.await?.iter().copied().chain(async_chunks),
)
.await?;
Ok(EcmascriptBuildNodeChunkListContent { chunks_contents }.cell())
}
/// Builds a chunk list content directly from a fixed set of `chunks`,
/// without expanding async-loader references. Used by
/// [`super::chunk_list::EcmascriptBuildNodeChunkList`] to track chunks
/// (e.g. client-component SSR chunks) that are already fully enumerated by
/// the caller.
#[turbo_tasks::function]
pub async fn new_from_chunks(
chunking_context: ResolvedVc<NodeJsChunkingContext>,
chunks: Vc<OutputAssets>,
) -> Result<Vc<Self>> {
let output_root = chunking_context.output_root().owned().await?;
let chunks_contents =
collect_chunks_contents(&output_root, chunks.await?.iter().copied()).await?;
Ok(EcmascriptBuildNodeChunkListContent { chunks_contents }.cell())
}
#[turbo_tasks::function]
pub async fn version(&self) -> Result<Vc<ChunkListVersion>> {
compute_chunk_list_version(&self.chunks_contents).await
}
}
#[turbo_tasks::value_impl]
impl VersionedContent for EcmascriptBuildNodeChunkListContent {
#[turbo_tasks::function]
fn content(self: Vc<Self>) -> Result<Vc<AssetContent>> {
bail!("EcmascriptBuildNodeChunkListContent does not have content")
}
#[turbo_tasks::function]
fn version(self: Vc<Self>) -> Vc<Box<dyn Version>> {
Vc::upcast(self.version())
}
#[turbo_tasks::function]
async fn update(
self: ResolvedVc<Self>,
from_version: ResolvedVc<Box<dyn Version>>,
) -> Result<Vc<Update>> {
let this = self.await?;
let to_version = self.version();
update_chunk_list(&this.chunks_contents, to_version, from_version).await
}
}
pub(crate) mod chunk;
pub(crate) mod chunk_list;
pub mod chunk_list_content;
pub(crate) mod runtime;
use anyhow::{Result, bail};
use turbo_tasks::{ResolvedVc, TryJoinIterExt, Vc};
use turbopack_core::version::{Update, Version, VersionedContent};
use super::{
update::update_ecmascript_merged_chunk, version::EcmascriptBuildNodeMergedChunkVersion,
};
use crate::ecmascript::node::content::EcmascriptBuildNodeChunkContent;
/// Composite [`VersionedContent`] that is the result of merging multiple
/// node ecmascript chunk contents together through the
/// [`EcmascriptBuildNodeChunkContentMerger`].
#[turbo_tasks::value(serialization = "skip", shared)]
pub(crate) struct EcmascriptBuildNodeMergedChunkContent {
pub(super) contents: Vec<ResolvedVc<EcmascriptBuildNodeChunkContent>>,
}
#[turbo_tasks::value_impl]
impl EcmascriptBuildNodeMergedChunkContent {
#[turbo_tasks::function]
pub async fn version(&self) -> Result<Vc<EcmascriptBuildNodeMergedChunkVersion>> {
Ok(EcmascriptBuildNodeMergedChunkVersion {
versions: self
.contents
.iter()
.map(|content| async move { content.own_version().await })
.try_join()
.await?,
}
.cell())
}
}
#[turbo_tasks::value_impl]
impl VersionedContent for EcmascriptBuildNodeMergedChunkContent {
#[turbo_tasks::function]
fn content(self: Vc<Self>) -> Result<Vc<turbopack_core::asset::AssetContent>> {
bail!("EcmascriptBuildNodeMergedChunkContent does not have content")
}
#[turbo_tasks::function]
fn version(self: Vc<Self>) -> Vc<Box<dyn Version>> {
Vc::upcast(self.version())
}
#[turbo_tasks::function]
async fn update(
self: Vc<Self>,
from_version: ResolvedVc<Box<dyn Version>>,
) -> Result<Vc<Update>> {
Ok(update_ecmascript_merged_chunk(self, from_version)
.await?
.cell())
}
}
use anyhow::{Result, bail};
use turbo_tasks::{ResolvedVc, TryJoinIterExt, Vc};
use turbopack_core::version::{VersionedContent, VersionedContentMerger, VersionedContents};
use crate::ecmascript::node::{
content::EcmascriptBuildNodeChunkContent,
merged::content::EcmascriptBuildNodeMergedChunkContent,
};
/// Merges multiple [`EcmascriptBuildNodeChunkContent`] into a single
/// [`EcmascriptBuildNodeMergedChunkContent`]. This allows the chunk list to
/// produce a single `EcmascriptMergedUpdate` for multiple chunks updating at
/// the same time.
#[turbo_tasks::value]
pub(crate) struct EcmascriptBuildNodeChunkContentMerger;
#[turbo_tasks::value_impl]
impl EcmascriptBuildNodeChunkContentMerger {
#[turbo_tasks::function]
pub fn new() -> Vc<Self> {
Self::cell(EcmascriptBuildNodeChunkContentMerger)
}
}
#[turbo_tasks::value_impl]
impl VersionedContentMerger for EcmascriptBuildNodeChunkContentMerger {
#[turbo_tasks::function]
async fn merge(
&self,
contents: Vc<VersionedContents>,
) -> Result<Vc<Box<dyn VersionedContent>>> {
let contents = contents
.await?
.iter()
.map(|content| async move {
if let Some(content) =
ResolvedVc::try_downcast_type::<EcmascriptBuildNodeChunkContent>(*content)
{
Ok(content)
} else {
bail!("expected Vc<EcmascriptBuildNodeChunkContent>")
}
})
.try_join()
.await?;
Ok(Vc::upcast(
EcmascriptBuildNodeMergedChunkContent { contents }.cell(),
))
}
}
pub(crate) mod content;
pub(crate) mod merger;
pub(crate) mod update;
pub(crate) mod version;
use std::sync::Arc;
use anyhow::Result;
use turbo_tasks::{FxIndexMap, ReadRef, ResolvedVc, TryJoinIterExt, Vc};
use turbopack_core::{
chunk::{ChunkingContext, ModuleId},
output::OutputAsset,
version::{PartialUpdate, TotalUpdate, Update, Version},
};
use turbopack_ecmascript::chunk_list::merged_update::{
EcmascriptMergedChunkAdded, EcmascriptMergedChunkDeleted, EcmascriptMergedChunkPartial,
EcmascriptMergedChunkUpdate, EcmascriptMergedUpdate, EcmascriptModuleEntry,
};
use crate::ecmascript::node::{
merged::{
content::EcmascriptBuildNodeMergedChunkContent,
version::EcmascriptBuildNodeMergedChunkVersion,
},
update::{NodeChunkUpdate, update_ecmascript_node_chunk_content},
version::EcmascriptBuildNodeChunkVersion,
};
struct MergedModuleMap {
versions: Vec<ReadRef<EcmascriptBuildNodeChunkVersion>>,
}
impl MergedModuleMap {
fn new(versions: Vec<ReadRef<EcmascriptBuildNodeChunkVersion>>) -> Self {
Self { versions }
}
fn get(&self, id: &ModuleId) -> Option<u128> {
for version in &self.versions {
if let Some(hash) = version.entries_hashes.get(id) {
return Some(*hash);
}
}
None
}
}
pub(crate) async fn update_ecmascript_merged_chunk(
content: Vc<EcmascriptBuildNodeMergedChunkContent>,
from_version: ResolvedVc<Box<dyn Version>>,
) -> Result<Update> {
let to_merged_version = content.version();
let from_merged_version = if let Some(from) =
ResolvedVc::try_downcast_type::<EcmascriptBuildNodeMergedChunkVersion>(from_version)
{
from
} else {
return Ok(Update::Total(TotalUpdate {
to: Vc::upcast::<Box<dyn Version>>(to_merged_version)
.into_trait_ref()
.await?,
}));
};
let to = to_merged_version.await?;
let from = from_merged_version.await?;
if from.ptr_eq(&to) {
return Ok(Update::None);
}
let mut from_versions_by_chunk_path: FxIndexMap<_, _> = from
.versions
.iter()
.map(|version| (&*version.chunk_path, version))
.collect();
let merged_module_map = MergedModuleMap::new(from.versions.to_vec());
let content = content.await?;
let to_contents = content
.contents
.iter()
.map(|content| async move {
let entries = content.entries().await?;
let content_ref = content.await?;
let output_root = content_ref.chunking_context.output_root().await?;
let path = content_ref.chunk.path().await?;
Ok((*content, entries, output_root, path))
})
.try_join()
.await?;
let mut merged_update = EcmascriptMergedUpdate::default();
for (content, entries, output_root, path) in &to_contents {
let Some(chunk_path) = output_root.get_path_to(path) else {
continue;
};
let chunk_update = if let Some(from_version) =
from_versions_by_chunk_path.swap_remove(chunk_path)
{
// Reuse the single-chunk diff so the merged path stays in sync with
// the standalone chunk update path.
let to_version = content.own_version().await?;
match update_ecmascript_node_chunk_content(**content, &to_version, from_version).await?
{
NodeChunkUpdate::None => continue,
NodeChunkUpdate::Partial {
added,
modified,
deleted,
} => {
let mut partial = EcmascriptMergedChunkPartial::default();
for (module_id, (module_hash, module_code)) in added {
partial.added.insert(module_id.clone());
// Only ship the code if no other chunk in the group
// already provides this module at the same hash.
if merged_module_map.get(&module_id) != Some(module_hash) {
let entry = EcmascriptModuleEntry::from_code(
&module_id,
module_code,
chunk_path,
)
.await?;
merged_update.entries.insert(module_id, entry);
}
}
partial.deleted.extend(deleted.into_keys());
for (module_id, module_code) in modified {
let entry =
EcmascriptModuleEntry::from_code(&module_id, module_code, chunk_path)
.await?;
merged_update.entries.insert(module_id, entry);
}
EcmascriptMergedChunkUpdate::Partial(partial)
}
}
} else {
let mut added = EcmascriptMergedChunkAdded::default();
for (id, entry) in entries {
let hash = *entry.hash.await?;
added.modules.insert(id.clone());
if merged_module_map.get(id) != Some(hash) {
let entry =
EcmascriptModuleEntry::from_code(id, *entry.code, chunk_path).await?;
merged_update.entries.insert(id.clone(), entry);
}
}
EcmascriptMergedChunkUpdate::Added(added)
};
merged_update.chunks.insert(chunk_path, chunk_update);
}
for (chunk_path, chunk_version) in from_versions_by_chunk_path {
let hashes = &chunk_version.entries_hashes;
merged_update.chunks.insert(
chunk_path,
EcmascriptMergedChunkUpdate::Deleted(EcmascriptMergedChunkDeleted {
modules: hashes.keys().cloned().collect(),
}),
);
}
let update = if merged_update.is_empty() {
Update::None
} else {
Update::Partial(PartialUpdate {
to: Vc::upcast::<Box<dyn Version>>(to_merged_version)
.into_trait_ref()
.await?,
instruction: Arc::new(serde_json::to_value(&merged_update)?),
})
};
Ok(update)
}
use anyhow::Result;
use turbo_rcstr::RcStr;
use turbo_tasks::{ReadRef, TryJoinIterExt, Vc};
use turbo_tasks_hash::{Xxh3Hash64Hasher, encode_base64};
use turbopack_core::version::Version;
use crate::ecmascript::node::version::EcmascriptBuildNodeChunkVersion;
/// The version of a [`super::content::EcmascriptBuildNodeMergedChunkContent`].
#[turbo_tasks::value(serialization = "skip", shared)]
pub(crate) struct EcmascriptBuildNodeMergedChunkVersion {
pub(super) versions: Vec<ReadRef<EcmascriptBuildNodeChunkVersion>>,
}
#[turbo_tasks::value_impl]
impl Version for EcmascriptBuildNodeMergedChunkVersion {
#[turbo_tasks::function]
async fn id(&self) -> Result<Vc<RcStr>> {
let mut hasher = Xxh3Hash64Hasher::new();
hasher.write_value(self.versions.len());
let sorted_ids = {
let mut sorted_ids = self
.versions
.iter()
.map(|version| ReadRef::cell(version.clone()).id())
.try_join()
.await?;
sorted_ids.sort();
sorted_ids
};
for id in sorted_ids {
hasher.write_value(id);
}
let hash = hasher.finish();
let hash = encode_base64(hash);
Ok(Vc::cell(hash.into()))
}
}
pub(crate) mod chunk;
pub(crate) mod content;
pub(crate) mod entry;
pub mod entry;
pub(crate) mod merged;
pub(crate) mod update;
pub(crate) mod version;
use std::sync::Arc;
use anyhow::Result;
use serde::Serialize;
use turbo_tasks::{FxIndexMap, FxIndexSet, ReadRef, ResolvedVc, Vc};
use turbo_tasks_fs::rope::Rope;
use turbo_tasks::{FxIndexMap, ReadRef, ResolvedVc, Vc};
use turbopack_core::{
chunk::ModuleId,
code_builder::Code,
source_map::GenerateSourceMap,
version::{PartialUpdate, TotalUpdate, Update, Version},
};
use turbopack_ecmascript::chunk_list::merged_update::{
EcmascriptMergedChunkPartial, EcmascriptMergedChunkUpdate, EcmascriptMergedUpdate,
EcmascriptModuleEntry,
};
use super::{content::EcmascriptBuildNodeChunkContent, version::EcmascriptBuildNodeChunkVersion};
#[derive(Serialize, Default)]
#[serde(
tag = "type",
rename = "EcmascriptMergedUpdate",
rename_all = "camelCase"
)]
struct EcmascriptMergedUpdate<'a> {
/// A map from module id to latest module entry.
#[serde(skip_serializing_if = "FxIndexMap::is_empty")]
entries: FxIndexMap<ModuleId, EcmascriptModuleEntry>,
/// A map from chunk path to the chunk update.
#[serde(skip_serializing_if = "FxIndexMap::is_empty")]
chunks: FxIndexMap<&'a str, EcmascriptMergedChunkUpdate>,
}
impl EcmascriptMergedUpdate<'_> {
fn is_empty(&self) -> bool {
self.entries.is_empty() && self.chunks.is_empty()
}
}
#[derive(Serialize)]
#[serde(tag = "type", rename_all = "camelCase")]
enum EcmascriptMergedChunkUpdate {
Partial(EcmascriptMergedChunkPartial),
}
#[derive(Serialize, Default)]
#[serde(rename_all = "camelCase")]
struct EcmascriptMergedChunkPartial {
#[serde(skip_serializing_if = "FxIndexSet::is_empty")]
added: FxIndexSet<ModuleId>,
#[serde(skip_serializing_if = "FxIndexSet::is_empty")]
deleted: FxIndexSet<ModuleId>,
}
#[derive(Serialize)]
struct EcmascriptModuleEntry {
#[serde(with = "turbo_tasks_fs::rope::ser_as_string")]
code: Rope,
url: String,
#[serde(with = "turbo_tasks_fs::rope::ser_option_as_string")]
map: Option<Rope>,
}
impl EcmascriptModuleEntry {
async fn from_code(id: &ModuleId, code: Vc<Code>, chunk_path: &str) -> Result<Self> {
let map = &*code.generate_source_map().await?;
let map = map.as_content().map(|f| f.content().clone());
/// serde_qs can't serialize a lone enum when it's [serde::untagged].
#[derive(Serialize)]
struct Id<'a> {
id: &'a ModuleId,
}
let id = serde_qs::to_string(&Id { id }).unwrap();
Ok(EcmascriptModuleEntry {
// Cloning a rope is cheap.
code: code.await?.source_code().clone(),
url: format!("{}?{}", chunk_path, id),
map,
})
}
}
pub(super) async fn update_node_chunk(
content: Vc<EcmascriptBuildNodeChunkContent>,
from_version: ResolvedVc<Box<dyn Version>>,
......@@ -121,7 +56,7 @@ pub(super) async fn update_node_chunk(
} => {
let mut partial = EcmascriptMergedChunkPartial::default();
for (module_id, module_code) in added {
for (module_id, (_hash, module_code)) in added {
partial.added.insert(module_id.clone());
let entry =
......@@ -161,16 +96,18 @@ pub(super) async fn update_node_chunk(
Ok(update)
}
enum NodeChunkUpdate {
pub(crate) enum NodeChunkUpdate {
None,
Partial {
added: FxIndexMap<ModuleId, Vc<Code>>,
/// Added modules, keyed by id, with their content hash (used by the
/// merged-chunk path to dedup code emission across chunks) and code.
added: FxIndexMap<ModuleId, (u128, Vc<Code>)>,
modified: FxIndexMap<ModuleId, Vc<Code>>,
deleted: FxIndexMap<ModuleId, u128>,
},
}
async fn update_ecmascript_node_chunk_content(
pub(super) async fn update_ecmascript_node_chunk_content(
content: Vc<EcmascriptBuildNodeChunkContent>,
to: &ReadRef<EcmascriptBuildNodeChunkVersion>,
from: &ReadRef<EcmascriptBuildNodeChunkVersion>,
......@@ -205,14 +142,14 @@ async fn update_ecmascript_node_chunk_content(
}
// Check for added modules
for (id, _hash) in &to.entries_hashes {
for (id, hash) in &to.entries_hashes {
if !from.entries_hashes.contains_key(id) {
let entries = match &entries_ref {
Some(entries) => entries,
None => entries_ref.insert(content.entries().await?),
};
if let Some(entry) = entries.get(id) {
added.insert(id.clone(), *entry.code);
added.insert(id.clone(), (*hash, *entry.code));
}
}
}
......
......@@ -2,6 +2,6 @@
#![feature(arbitrary_self_types_pointers)]
pub(crate) mod chunking_context;
pub(crate) mod ecmascript;
pub mod ecmascript;
pub use chunking_context::{NodeJsChunkingContext, NodeJsChunkingContextBuilder};
......@@ -1671,34 +1671,63 @@ function initializeServerHmr(moduleFactories, devModuleCache) {
return;
}
const instruction = msg.instruction;
if (instruction.type !== 'EcmascriptMergedUpdate') {
return;
}
try {
const { entries = {}, chunks = {} } = instruction;
const evalModuleEntry = (entry)=>{
// eslint-disable-next-line no-eval
return (0, eval)(entry.map ? inlineSourcemaps(entry) : entry.code);
};
const { added, modified } = computeChangedModules(entries, chunks, undefined // no chunkModulesMap for Node.js
);
// Use shared HMR update implementation
applyEcmascriptMergedUpdateShared({
added,
modified,
disposedModules: [],
evalModuleEntry,
instantiateModule,
applyModuleFactoryName: ()=>{},
moduleFactories,
devModuleCache,
autoAcceptRootModules: true
});
if (instruction.type === 'ChunkListUpdate') {
// All node ecmascript chunks are mergeable, so a `total`/`partial` here
// means a non-mergeable asset changed in an unsupported way. Escalate
// to a full clear() rather than leave stale factories in memory.
for (const [chunkPath, chunkUpdate] of Object.entries(instruction.chunks ?? {})){
if (chunkUpdate.type === 'total' || chunkUpdate.type === 'partial') {
throw new Error(`unsupported '${chunkUpdate.type}' update for chunk ${chunkPath}`);
}
}
if (instruction.merged) {
for (const merged of instruction.merged){
applyEcmascriptMergedUpdate(merged, moduleFactories, devModuleCache);
}
}
return;
}
if (instruction.type === 'EcmascriptMergedUpdate') {
applyEcmascriptMergedUpdate(instruction, moduleFactories, devModuleCache);
return;
}
} catch (e) {
console.error('[Server HMR] Update failed, full reload needed:', e);
throw e;
}
}
function applyEcmascriptMergedUpdate(instruction, moduleFactories, devModuleCache) {
const { entries = {}, chunks = {} } = instruction;
const evalModuleEntry = (entry)=>{
const code = entry.map ? inlineSourcemaps(entry) : entry.code;
// eslint-disable-next-line no-eval
return (0, eval)(`(require) => ${code}`)(require);
};
const { added, modified } = computeChangedModules(entries, chunks, undefined // no chunkModulesMap for Node.js
);
// Modules that appear in an "added" chunk but already exist in the cache
// were moved to a renamed chunk. Treat them as modified so the dependency
// walk runs and they get re-instantiated with the new factory.
for (const [moduleId, entry] of added){
if (entry != null && devModuleCache[moduleId] != null) {
added.delete(moduleId);
modified.set(moduleId, entry);
}
}
// Use shared HMR update implementation
applyEcmascriptMergedUpdateShared({
added,
modified,
disposedModules: [],
evalModuleEntry,
instantiateModule,
applyModuleFactoryName: ()=>{},
moduleFactories,
devModuleCache,
autoAcceptRootModules: true
});
}
/// <reference path="../../shared/runtime/dev-protocol.d.ts" />
/// <reference path="./hmr-client.ts" />
/**
......@@ -1731,9 +1760,16 @@ if (handlers.size === 0) {
// First registration in this generation: install the routing dispatcher.
globalThis.__turbopack_server_hmr_apply__ = (update)=>{
const registry = globalThis.__turbopack_server_hmr_handlers__ ?? new Map();
const updateChunkPaths = Object.keys(update.instruction?.chunks ?? {});
// Chunk paths can appear either directly on the instruction (single-chunk
// updates) or nested inside `merged` entries (chunks covered by a
// merger). Collect both so routing isn't skipped just because a mergeable
// chunk's update only reports its paths inside `merged`.
const updateChunkPaths = new Set([
...Object.keys(update.instruction?.chunks ?? {}),
...(update.instruction?.merged ?? []).flatMap((merged)=>Object.keys(merged.chunks ?? {}))
]);
const toCall = [];
if (updateChunkPaths.length === 0) {
if (updateChunkPaths.size === 0) {
for (const entry of registry.values())toCall.push(entry);
} else {
const seen = new Set();
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment