Unverified Commit 13491b0d authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

Stop serializaing hmr version state (#97146)

Mark VersionedContentMap as `serialization = "none"`

This will cause every dev session to start with an empty map, to ensure
_derived_ data from the old map doesn't survive, the functions
interacting with the state are all marked `session_dependent`.

This is directly motivated by the work on GC which runs into problems
with the State objects storing operations but not consistently
`connect`ing to them on cache hits, which causes the GC to drop them.
Additionally, this prevents eagerly recomputing all server hmr routes on
startup.

In theory this will slow down reconnecting a dev session, but the
underlying operations are still cached and the tasks computing Versions,
so it just the map that needs to be reconstructed, so the cost should be
minor. In fact i would guess that it is a bit faster to reconstruct this
map than to restore all the keys from disk in the case where the map is
large.

<!-- Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:

## For Contributors

We encourage you to use AI to assist you in researching, creating, and
reviewing changes. However, you must review and deeply understand the
contributions you are making. For this reason, **pull request
descriptions from external contributors must be written by a human**.

### Improving Documentation

- Run `pnpm prettier-fix` to fix formatting issues before opening the
PR.
- Read the Docs Contribution Guide to ensure your contribution follows
the docs guidelines:
https://nextjs.org/docs/community/contribution-guide

### Fixing a bug

- Related issues linked using `fixes #number`
- Tests added. See:
https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md

### Adding a feature

- Implements an existing feature request or RFC. Make sure the feature
request has been accepted for implementation before opening a PR. (A
discussion must be opened, see
https://github.com/vercel/next.js/discussions/new?category=ideas)
- Related issues/discussions are linked using `fixes #number`
- e2e tests added
(https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs)
- Documentation added
- Telemetry added. In case of a feature if it's used or not.
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md

### Signed commits

- This repository requires verified commit signatures on protected
branches.
- If this pull request is blocked for unsigned commits, re-sign the
commits and force-push the branch.
- A `Signed-off-by` line in the commit message is not enough.

## For Maintainers

- Minimal description (aim for explaining to someone not on the team to
understand the PR)
- When linking to a Slack thread, you might want to share details of the
conclusion
- Link both the Linear (Fixes NEXT-xxx) and the GitHub issues
- Add review comments if necessary to explain to the reviewer the logic
behind a change

### What?

### Why?

### How?

Closes NEXT-
Fixes #

-->

---

<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
parent 3c2f9be2
......@@ -3,7 +3,10 @@ use std::sync::Arc;
use anyhow::Result;
use rustc_hash::FxHashMap;
use turbo_rcstr::RcStr;
use turbo_tasks::{FxIndexMap, FxIndexSet, ReadRef, ResolvedVc, TraitRef, TryJoinIterExt, Vc};
use turbo_tasks::{
FxIndexMap, FxIndexSet, NonLocalValue, ReadRef, ResolvedVc, TraitRef, TryJoinIterExt, Vc,
debug::ValueDebugFormat, trace::TraceRawVcs,
};
use turbo_tasks_fs::FileSystemPath;
use turbo_tasks_hash::{Xxh3Hash64Hasher, encode_base64};
use turbopack_browser::ecmascript::list::content::EcmascriptDevChunkListContent;
......@@ -12,11 +15,15 @@ use turbopack_nodejs::ecmascript::node::entry::chunk_list_content::EcmascriptBui
use crate::versioned_content_map::VersionedContentMap;
#[derive(TraceRawVcs, PartialEq, Eq, ValueDebugFormat, NonLocalValue)]
pub struct HmrChunkWithContent {
pub path: RcStr,
pub content: ResolvedVc<Box<dyn VersionedContent>>,
}
#[turbo_tasks::value(transparent, serialization = "skip")]
pub struct HmrChunksWithContent(Vec<HmrChunkWithContent>);
/// Whether `content` is a chunk list, i.e. an entry point of the chunk graph that
/// an HMR subscription can be anchored on.
///
......@@ -66,7 +73,7 @@ impl Version for AggregateHmrVersion {
impl AggregateHmrVersion {
pub async fn from_map(
map: Vc<VersionedContentMap>,
root: &FileSystemPath,
root: FileSystemPath,
) -> Result<Vc<Box<dyn Version>>> {
// An empty `versions` map behaves the same as `NotFoundVersion` would in
// `diff_chunks_against`, so no special case is needed here.
......
......@@ -2617,7 +2617,7 @@ impl Project {
bail!("must be in dev mode to hmr")
};
let root = this.aggregate_hmr_root_path(target).owned().await?;
AggregateHmrVersion::from_map(*map, &root).await
AggregateHmrVersion::from_map(*map, root).await
}
let version_op = aggregate_hmr_version_operation(self, target);
......@@ -2659,7 +2659,7 @@ impl Project {
bail!("must be in dev mode to hmr")
};
let root = self.aggregate_hmr_root_path(target).owned().await?;
let chunks_versioned_content = map.hmr_chunks_in_path(&root).await?;
let chunks_versioned_content = map.hmr_chunks_in_path(root).await?;
// No chunks to diff yet (e.g. before any endpoints have been written).
if chunks_versioned_content.is_empty() {
......
......@@ -15,7 +15,9 @@ use turbopack_core::{
version::OptionVersionedContent,
};
use crate::aggregate_hmr::{HmrChunkWithContent, is_entry_chunk_list_content};
use crate::aggregate_hmr::{
HmrChunkWithContent, HmrChunksWithContent, is_entry_chunk_list_content,
};
#[derive(
Clone, TraceRawVcs, PartialEq, Eq, ValueDebugFormat, Debug, NonLocalValue, Encode, Decode,
......@@ -32,9 +34,7 @@ unsafe impl OperationValue for MapEntry {}
#[turbo_tasks::value(transparent, operation)]
struct OptionMapEntry(Option<MapEntry>);
#[derive(
Clone, TraceRawVcs, PartialEq, Eq, ValueDebugFormat, Debug, NonLocalValue, Encode, Decode,
)]
#[derive(Clone, TraceRawVcs, PartialEq, Eq, ValueDebugFormat, Debug, NonLocalValue)]
pub struct PathToOutputOperation(
/// We need to use an operation for outputs as it's stored for later usage and we want to
/// reconnect this operation when it's received from the map again.
......@@ -45,21 +45,8 @@ pub struct PathToOutputOperation(
FxHashMap<FileSystemPath, ExpandedOutputAssetsOperationSet>,
);
#[derive(
Clone,
Default,
TraceRawVcs,
PartialEq,
Eq,
ValueDebugFormat,
Debug,
NonLocalValue,
Encode,
Decode,
)]
struct ExpandedOutputAssetsOperationSet(
#[bincode(with = "turbo_bincode::indexset")] FxIndexSet<OperationVc<ExpandedOutputAssets>>,
);
#[derive(Clone, Default, TraceRawVcs, PartialEq, Eq, ValueDebugFormat, Debug, NonLocalValue)]
struct ExpandedOutputAssetsOperationSet(FxIndexSet<OperationVc<ExpandedOutputAssets>>);
// HACK: This is technically incorrect because the map's key contains a `ResolvedVc`...
unsafe impl OperationValue for PathToOutputOperation {}
......@@ -68,13 +55,26 @@ unsafe impl OperationValue for PathToOutputOperation {}
type OutputOperationToComputeEntry =
FxHashMap<OperationVc<ExpandedOutputAssets>, OperationVc<OptionMapEntry>>;
// TODO: Ideally this structure is never persisted, so new sessions start from scratch and don't
// accumulate entries or force rebuilds of all chunks when a new session is only interested in some
// of them. If this happens, this should have #[turbo_tasks::value(evict = "never")].
#[turbo_tasks::value]
/// Tracks all the output assets produced in a session. This allows us to compute fine grained
/// change information which drives HMR sessions.
///
/// `serialization = "skip"` so that HMR sessions fully restart on each new session.
/// `evict = "never"` in order to ensure that we don't lose track of version state in the middle of
/// a session.
#[turbo_tasks::value(serialization = "skip", evict = "never")]
pub struct VersionedContentMap {
// TODO: turn into a bi-directional multimap, ExpandedOutputAssets ->
// FxIndexSet<FileSystemPath>
// Because the cell is not serialized, these `State`s -- and the sets of invalidators
// registered on them by `State::get` -- start out empty in every new session. A task that
// restores clean would therefore serve stale entries forever, since it never re-runs and so
// never re-registers its invalidator. Every `#[turbo_tasks::function]` that touches these
// must be `session_dependent` so it re-executes once per session and rebuilds that wiring.
//
// Keep those accesses inside this file. A plain `async fn` helper reading a `State` runs in
// its *caller's* task frame, which silently pushes the `session_dependent` obligation onto
// every transitive caller.
map_path_to_op: State<PathToOutputOperation>,
map_op_to_compute_entry: State<OutputOperationToComputeEntry>,
}
......@@ -89,7 +89,10 @@ impl VersionedContentMap {
}
.resolved_cell()
}
}
#[turbo_tasks::value_impl]
impl VersionedContentMap {
/// Lists the aggregate-HMR *entry* chunks under `root` with their
/// [`VersionedContent`], sorted by path. Only entry-chunk-list content is
/// returned (see [`is_entry_chunk_list_content`]). Callers scope which
......@@ -102,10 +105,11 @@ impl VersionedContentMap {
/// entries that span server and client contexts, changes for one context
/// can shift the internals of the map, making iteration order different
/// for the same set of paths.
#[turbo_tasks::function(session_dependent)]
pub async fn hmr_chunks_in_path(
self: Vc<Self>,
root: &FileSystemPath,
) -> Result<Vec<HmrChunkWithContent>> {
root: FileSystemPath,
) -> Result<Vc<HmrChunksWithContent>> {
let this = self.await?;
// `State::get` returns a lock guard, which can't be held across the
// awaits below, so snapshot the keys and release it.
......@@ -146,18 +150,12 @@ impl VersionedContentMap {
.try_flat_join()
.await?;
chunks.sort_by(|a, b| a.path.cmp(&b.path));
Ok(chunks)
Ok(Vc::cell(chunks))
}
}
#[turbo_tasks::value_impl]
impl VersionedContentMap {
/// Inserts output assets into the map and returns a completion that when
/// awaited will emit the assets that were inserted.
//
// TODO: If `VersionedContentMap` becomes transient as described above, these methods should be
// `#[turbo_tasks::function(session_dependent)]`
#[turbo_tasks::function]
#[turbo_tasks::function(session_dependent)]
pub async fn insert_output_assets(
self: ResolvedVc<Self>,
// Output assets to emit
......@@ -182,7 +180,7 @@ impl VersionedContentMap {
/// Creates a [`MapEntry`] (a pre-computed map for optimized lookup) for an output assets
/// operation. When assets change, map_path_to_op is updated.
#[turbo_tasks::function]
#[turbo_tasks::function(session_dependent)]
async fn compute_entry(
&self,
assets_operation: OperationVc<ExpandedOutputAssets>,
......@@ -289,7 +287,7 @@ impl VersionedContentMap {
Ok(Vc::cell(None))
}
#[turbo_tasks::function]
#[turbo_tasks::function(session_dependent)]
pub async fn keys_in_path(&self, root: FileSystemPath) -> Result<Vc<Vec<RcStr>>> {
let keys = {
let map = &self.map_path_to_op.get().0;
......@@ -306,7 +304,7 @@ impl VersionedContentMap {
Ok(Vc::cell(keys))
}
#[turbo_tasks::function]
#[turbo_tasks::function(session_dependent)]
fn raw_get(&self, path: FileSystemPath) -> Vc<OptionMapEntry> {
let assets = {
let map = &self.map_path_to_op.get().0;
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment