Unverified Commit 054aa46c authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

[turbopack] Persistent last_successful_parse (#92852)

## What?

Replaces the `TransientState<ReadRef<ParseResult>>` used by `failsafe_parse` with a persistent `LastSuccessfulSource` turbo-tasks cell that stores the raw bytes of the last successfully parsed version of the module.

## Why?

The previous implementation had three problems:

1. **Memory**: It held a full AST in memory alongside the current one — effectively doubling AST memory for every file with `keep_last_successful_parse` enabled.
2. **Durability**: `TransientState` skips serialization, so the fallback was silently lost on eviction or server restart, meaning the mechanism that prevents whole-graph invalidation on syntax errors didn't survive across restarts.
3. **`final_read_hint` defeat**: `with_consumed_parse_result` uses `final_read_hint()` + `ReadRef::try_unwrap()` to avoid cloning the AST when this is the last reader. But holding a `ReadRef<ParseResult>` in `TransientState` meant there was always a second `Arc` reference alive, so `try_unwrap` always failed and the AST was always cloned — even in the common case where no syntax error had occurred. Now because we are just cloning the Rope, we don't duplicate any memory until things change and even then only if there is an error.

## How?

`LastSuccessfulSource` is a `#[turbo_tasks::value(eq = "manual")]` wrapping `parking_lot::Mutex<Option<Rope>>` and a `SerializationInvalidator` with:
- `PartialEq` always `true` and `Hash` as a no-op — re-celling a freshly constructed `LastSuccessfulSource` inside `failsafe_parse` hits the compare-and-update path and preserves the existing cell (and its interior `Mutex<Option<Rope>>`) across task re-runs. The value is not a field on `EcmascriptModuleAsset`; it lives in the `failsafe_parse` task's own cell storage.
- Serialized via `#[bincode(with = "parking_lot_mutex_bincode")]` on the Mutex field — persisted to the cache and survives both eviction and restarts.
- Calls `serialization_invalidator.invalidate()` on every mutation (`set`/`clear`) so the backend knows to write the updated state back to the persistence layer.

`Rope` is just a ref-counted byte buffer, so cloning it (on cache hit) is a single `Arc::clone`.

### Consistency with the parsed AST

`ParseResult::Ok` now carries a `program_source: Rope` field: the raw bytes that produced the AST, captured atomically inside `parse_file_content`. `failsafe_parse` reads the rope out of `ParseResult::Ok` rather than re-reading `source.content()` separately — otherwise, under eventual consistency, the file could change between the parse task and the rope read, letting `failsafe_parse` cache bytes that didn't match the AST it just stored.

On a successful parse, `program_source` is stored into the cell. On failure, `parse_from_rope` re-derives the AST on demand, reusing the existing `parse_file_content` path. If the fallback itself fails for any reason, the cached bytes are cleared and the real (broken) result is returned.

### Simplification of `SerializationInvalidator`

`SerializationInvalidator` previously captured a `Weak<dyn TurboTasksApi>` and a tokio `Handle` at construction time. Since `invalidate()` is always called from within a turbo-tasks execution context (where the `TURBO_TASKS` task-local is available), these captures are unnecessary. The struct is now just a `TaskId` wrapper with derives for `Clone`, `Hash`, `Eq`, `PartialEq`, `Encode`, `Decode`, `TraceRawVcs`, and `NonLocalValue`.

<!-- NEXT_JS_LLM_PR -->
parent 9e71bba0
......@@ -150,6 +150,8 @@ impl Debug for TaskId {
}
}
unsafe impl crate::NonLocalValue for TaskId {}
pub const TRANSIENT_TASK_BIT: u32 = 0x8000_0000;
impl TaskId {
......
......@@ -101,7 +101,7 @@ pub use crate::{
read_ref::ReadRef,
serialization_invalidation::SerializationInvalidator,
spawn::{JoinHandle, block_for_future, block_in_place, spawn, spawn_blocking, spawn_thread},
state::{State, TransientState},
state::{State, TransientState, parking_lot_mutex_bincode},
task::{
SharedReference, TypedSharedReference,
task_input::{EitherTaskInput, TaskInput},
......
use std::{
hash::Hash,
sync::{Arc, Weak},
};
use bincode::{
Decode, Encode,
de::Decoder,
enc::Encoder,
error::{DecodeError, EncodeError},
impl_borrow_decode,
};
use serde::{Deserialize, Serialize, de::Visitor};
use tokio::runtime::Handle;
use crate::{TaskId, TurboTasksApi, manager::with_turbo_tasks, trace::TraceRawVcs};
#[derive(Clone)]
use std::hash::Hash;
use bincode::{Decode, Encode};
use turbo_tasks_macros::NonLocalValue;
use crate as turbo_tasks;
use crate::{TaskId, manager::with_turbo_tasks, trace::TraceRawVcs};
/// Allows a turbo-tasks value type to notify the backend that its serialized
/// state has changed out-of-band (i.e. without going through the normal
/// output-cell mechanism).
///
/// `invalidate` must always be called from within a turbo-tasks execution
/// context (i.e. inside a `#[turbo_tasks::function]` body or a `State`
/// mutation triggered from one), so `TURBO_TASKS` task-local is always
/// available and we do not need to capture handles at construction time.
#[derive(Clone, Hash, Eq, PartialEq, Encode, Decode, TraceRawVcs, NonLocalValue)]
pub struct SerializationInvalidator {
task: TaskId,
turbo_tasks: Weak<dyn TurboTasksApi>,
handle: Handle,
}
impl Hash for SerializationInvalidator {
fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
self.task.hash(state);
}
}
impl PartialEq for SerializationInvalidator {
fn eq(&self, other: &Self) -> bool {
self.task == other.task
}
}
impl Eq for SerializationInvalidator {}
impl SerializationInvalidator {
pub fn invalidate(&self) {
let SerializationInvalidator {
task,
turbo_tasks,
handle,
} = self;
let _guard = handle.enter();
if let Some(turbo_tasks) = turbo_tasks.upgrade() {
turbo_tasks.invalidate_serialization(*task);
}
with_turbo_tasks(|tt| tt.invalidate_serialization(self.task));
}
pub(crate) fn new(task_id: TaskId) -> Self {
Self {
task: task_id,
turbo_tasks: with_turbo_tasks(Arc::downgrade),
handle: Handle::current(),
}
Self { task: task_id }
}
}
impl TraceRawVcs for SerializationInvalidator {
fn trace_raw_vcs(&self, _context: &mut crate::trace::TraceRawVcsContext) {
// nothing here
}
}
impl Serialize for SerializationInvalidator {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
serializer.serialize_newtype_struct("SerializationInvalidator", &self.task)
}
}
impl<'de> Deserialize<'de> for SerializationInvalidator {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: serde::Deserializer<'de>,
{
struct V;
impl<'de> Visitor<'de> for V {
type Value = SerializationInvalidator;
fn expecting(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
write!(f, "an SerializationInvalidator")
}
fn visit_newtype_struct<D>(self, deserializer: D) -> Result<Self::Value, D::Error>
where
D: serde::Deserializer<'de>,
{
Ok(SerializationInvalidator {
task: TaskId::deserialize(deserializer)?,
turbo_tasks: with_turbo_tasks(Arc::downgrade),
handle: tokio::runtime::Handle::current(),
})
}
}
deserializer.deserialize_newtype_struct("SerializationInvalidator", V)
}
}
impl Encode for SerializationInvalidator {
fn encode<E: Encoder>(&self, encoder: &mut E) -> Result<(), EncodeError> {
Encode::encode(&self.task, encoder)
}
}
impl<Context> Decode<Context> for SerializationInvalidator {
fn decode<D: Decoder<Context = Context>>(decoder: &mut D) -> Result<Self, DecodeError> {
Ok(SerializationInvalidator {
task: Decode::decode(decoder)?,
turbo_tasks: with_turbo_tasks(Arc::downgrade),
handle: tokio::runtime::Handle::current(),
})
}
}
impl_borrow_decode!(SerializationInvalidator);
......@@ -123,7 +123,7 @@ impl<T> Drop for StateRef<'_, T> {
}
}
mod parking_lot_mutex_bincode {
pub mod parking_lot_mutex_bincode {
use bincode::{
BorrowDecode,
de::{BorrowDecoder, Decoder},
......
......@@ -77,8 +77,9 @@ use swc_core::{
use tracing::{Instrument, Level, instrument};
use turbo_rcstr::{RcStr, rcstr};
use turbo_tasks::{
FxDashMap, FxIndexMap, NonLocalValue, ReadRef, ResolvedVc, TaskInput, TryJoinIterExt, Upcast,
ValueToString, Vc, trace::TraceRawVcs, turbofmt,
FxDashMap, FxIndexMap, NonLocalValue, ReadRef, ResolvedVc, SerializationInvalidator, TaskInput,
TryJoinIterExt, Upcast, ValueToString, Vc, get_serialization_invalidator,
parking_lot_mutex_bincode, trace::TraceRawVcs, turbofmt,
};
use turbo_tasks_fs::{FileJsonContent, FileSystemPath, glob::Glob, rope::Rope};
use turbopack_core::{
......@@ -363,7 +364,74 @@ impl EcmascriptModuleAssetBuilder {
}
}
/// Stores the raw bytes of the last successfully parsed version of a module.
///
/// Cached as a turbo-tasks cell inside `failsafe_parse`: the always-equal
/// `PartialEq` impl means that re-running the task does not replace the cell,
/// so the interior `Mutex<Option<Rope>>` (and its stored rope) survive across
/// task executions. A `SerializationInvalidator` keeps the persistence layer
/// in sync with the in-memory mutation.
#[turbo_tasks::value(eq = "manual")]
struct LastSuccessfulSource {
#[bincode(with = "parking_lot_mutex_bincode")]
#[turbo_tasks(trace_ignore, debug_ignore)]
source: parking_lot::Mutex<Option<Rope>>,
/// Notifies the backend when the in-memory `source` changes so that the
/// serialized task state is written back to the persistence layer.
#[turbo_tasks(debug_ignore)]
serialization_invalidator: SerializationInvalidator,
}
impl LastSuccessfulSource {
fn get(&self) -> Option<Rope> {
self.source.lock().clone()
}
fn set(&self, rope: Rope) {
*self.source.lock() = Some(rope);
self.serialization_invalidator.invalidate();
}
fn clear(&self) {
*self.source.lock() = None;
self.serialization_invalidator.invalidate();
}
}
impl Default for LastSuccessfulSource {
fn default() -> Self {
Self {
source: parking_lot::Mutex::new(None),
serialization_invalidator: get_serialization_invalidator(),
}
}
}
impl std::fmt::Debug for LastSuccessfulSource {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("LastSuccessfulSource")
}
}
// Always-equal so that re-celling a freshly constructed `LastSuccessfulSource`
// does not overwrite the existing cell — the interior `Mutex` holds the
// cross-execution state and must survive task re-runs.
impl PartialEq for LastSuccessfulSource {
fn eq(&self, _other: &Self) -> bool {
true
}
}
impl Eq for LastSuccessfulSource {}
// No-op hash to uphold the `Hash` / `Eq` contract (equal values must hash
// identically). The interior `Mutex` content is not part of the cache identity.
impl std::hash::Hash for LastSuccessfulSource {
fn hash<H: std::hash::Hasher>(&self, _state: &mut H) {}
}
#[turbo_tasks::value]
#[derive(Debug)]
pub struct EcmascriptModuleAsset {
pub source: ResolvedVc<Box<dyn Source>>,
pub asset_context: ResolvedVc<Box<dyn AssetContext>>,
......@@ -373,22 +441,6 @@ pub struct EcmascriptModuleAsset {
pub compile_time_info: ResolvedVc<CompileTimeInfo>,
pub side_effect_free_packages: Option<ResolvedVc<Glob>>,
pub inner_assets: Option<ResolvedVc<InnerAssets>>,
#[turbo_tasks(debug_ignore)]
last_successful_parse: turbo_tasks::TransientState<ReadRef<ParseResult>>,
}
impl core::fmt::Debug for EcmascriptModuleAsset {
fn fmt(&self, f: &mut core::fmt::Formatter) -> core::fmt::Result {
f.debug_struct("EcmascriptModuleAsset")
.field("source", &self.source)
.field("asset_context", &self.asset_context)
.field("ty", &self.ty)
.field("transforms", &self.transforms)
.field("options", &self.options)
.field("compile_time_info", &self.compile_time_info)
.field("side_effect_free_packages", &self.side_effect_free_packages)
.field("inner_assets", &self.inner_assets)
.finish()
}
}
#[turbo_tasks::value_trait]
......@@ -498,6 +550,41 @@ impl ModuleTypeResult {
}
}
impl EcmascriptModuleAsset {
/// Attempts to re-parse the module from the last known-good file bytes.
///
/// Returns `None` if no saved source is available or if any step fails, in
/// which case the caller should fall back to the current (broken) result.
/// On failure the cached source is cleared so we don't keep retrying it.
async fn try_parse_last_successful_source(
&self,
last_successful_source: &LastSuccessfulSource,
) -> Option<Vc<ParseResult>> {
let rope = last_successful_source.get()?;
let result: Result<Vc<ParseResult>> = async {
let node_env = self
.compile_time_info
.await?
.defines
.read_process_env(rcstr!("NODE_ENV"))
.owned()
.await?
.unwrap_or_else(|| rcstr!("development"));
crate::parse::parse_from_rope(rope, self.source, self.ty, self.transforms, node_env)
.await
}
.await;
match result {
Ok(result) => Some(result),
Err(_) => {
// A failure is very unexpected, but we don't want to keep bad bytes around
last_successful_source.clear();
None
}
}
}
}
#[turbo_tasks::value_impl]
impl EcmascriptParsable for EcmascriptModuleAsset {
#[turbo_tasks::function]
......@@ -505,15 +592,23 @@ impl EcmascriptParsable for EcmascriptModuleAsset {
let real_result = self.parse().await?;
if self.options.await?.keep_last_successful_parse {
let real_result_value = real_result.await?;
let result_value = if matches!(*real_result_value, ParseResult::Ok { .. }) {
self.last_successful_parse
.set_unconditionally(real_result_value.clone());
real_result_value
// The cell stored here survives re-runs of this task because
// `LastSuccessfulSource`'s `PartialEq` is always-equal: the
// compare-and-update path preserves the existing cell (and its
// interior `Mutex<Option<Rope>>`) whenever this function is
// re-executed.
let last_successful_source = LastSuccessfulSource::default().cell().await?;
if let ParseResult::Ok { program_source, .. } = &*real_result_value {
// Store the bytes that `parse()` actually saw as the
// last-known-good source.
last_successful_source.set(program_source.clone());
Ok(real_result)
} else {
let state_ref = self.last_successful_parse.get();
state_ref.as_ref().unwrap_or(&real_result_value).clone()
};
Ok(ReadRef::cell(result_value))
Ok(self
.try_parse_last_successful_source(&last_successful_source)
.await
.unwrap_or(real_result))
}
} else {
Ok(real_result)
}
......@@ -642,7 +737,6 @@ impl EcmascriptModuleAsset {
compile_time_info,
side_effect_free_packages,
inner_assets: None,
last_successful_parse: Default::default(),
})
}
......@@ -677,7 +771,6 @@ impl EcmascriptModuleAsset {
compile_time_info,
side_effect_free_packages,
inner_assets: Some(inner_assets),
last_successful_parse: Default::default(),
}))
}
}
......
......@@ -147,7 +147,6 @@ impl Visit for IdentCollector {
#[turbo_tasks::value(shared, serialization = "none", eq = "manual", cell = "new")]
#[allow(clippy::large_enum_variant)]
pub enum ParseResult {
// Note: Ok must not contain any Vc as it's snapshot by failsafe_parse
Ok {
#[turbo_tasks(debug_ignore, trace_ignore)]
program: Program,
......@@ -160,6 +159,11 @@ pub enum ParseResult {
#[turbo_tasks(debug_ignore, trace_ignore)]
source_map: Arc<swc_core::common::SourceMap>,
source_mapping_url: Option<RcStr>,
/// Raw bytes of the source that produced this parse, captured atomically
/// with the AST. `failsafe_parse` uses this to recover good parses in development on
/// error.
#[turbo_tasks(debug_ignore, trace_ignore)]
program_source: Rope,
},
Unparsable {
messages: Option<Vec<RcStr>>,
......@@ -333,62 +337,29 @@ async fn parse_internal(
AssetContent::File(file) => match &*file.await? {
FileContent::NotFound => ParseResult::NotFound.cell(),
FileContent::Content(file) => {
match BytesStr::from_utf8(file.content().clone().into_bytes()) {
Ok(string) => {
let transforms = &*transforms.await?;
match parse_file_content(
string,
&fs_path,
ident,
source.ident().await?.query.clone(),
file_path_hash,
source,
ty,
transforms,
node_env.clone(),
loose_errors,
inline_helpers,
)
.await
{
Ok(result) => result,
Err(e) => {
// ast-grep-ignore: no-context-turbofmt
return Err(e).context(
turbofmt!(
"Transforming and/or parsing of {} failed",
source.ident()
)
.await?,
);
}
}
}
Err(error) => {
let error: RcStr = PrettyPrintError(
&anyhow::anyhow!(error).context("failed to convert rope into string"),
)
.to_string()
.into();
ReadSourceIssue {
// Technically we could supply byte offsets to the issue source, but
// that would cause another utf8 error to be produced when we
// attempt to infer line/column
// offsets
source: IssueSource::from_source_only(source),
error: error.clone(),
severity: if loose_errors {
IssueSeverity::Warning
} else {
IssueSeverity::Error
},
}
.resolved_cell()
.emit();
ParseResult::Unparsable {
messages: Some(vec![error]),
}
.cell()
let transforms = &*transforms.await?;
match parse_file_content(
file.content().clone(),
&fs_path,
ident,
source.ident().await?.query.clone(),
file_path_hash,
source,
ty,
transforms,
node_env.clone(),
loose_errors,
inline_helpers,
)
.await
{
Ok(result) => result,
Err(e) => {
// ast-grep-ignore: no-context-turbofmt
return Err(e).context(
turbofmt!("Transforming and/or parsing of {} failed", source.ident())
.await?,
);
}
}
}
......@@ -398,7 +369,7 @@ async fn parse_internal(
}
async fn parse_file_content(
string: BytesStr,
program_source: Rope,
fs_path: &FileSystemPath,
ident: &str,
query: RcStr,
......@@ -410,6 +381,35 @@ async fn parse_file_content(
loose_errors: bool,
inline_helpers: bool,
) -> Result<Vc<ParseResult>> {
let string = match BytesStr::from_utf8(program_source.clone().into_bytes()) {
Ok(s) => s,
Err(error) => {
let error: RcStr = PrettyPrintError(
&anyhow::anyhow!(error).context("failed to convert rope into string"),
)
.to_string()
.into();
ReadSourceIssue {
// Technically we could supply byte offsets to the issue source, but
// that would cause another utf8 error to be produced when we
// attempt to infer line/column
// offsets
source: IssueSource::from_source_only(source),
error: error.clone(),
severity: if loose_errors {
IssueSeverity::Warning
} else {
IssueSeverity::Error
},
}
.resolved_cell()
.emit();
return Ok(ParseResult::Unparsable {
messages: Some(vec![error]),
}
.cell());
}
};
let source_map: Arc<swc_core::common::SourceMap> = Default::default();
let (emitter, collector) = IssueEmitter::new(
source,
......@@ -619,6 +619,7 @@ async fn parse_file_content(
globals: Arc::new(Globals::new()),
source_map,
source_mapping_url: source_mapping_url.map(|s| s.into()),
program_source,
})
},
|f, cx| GLOBALS.set(globals_ref, || HANDLER.set(&handler, || f.poll(cx))),
......@@ -752,6 +753,38 @@ impl Visit for VarDeclWithTsDeclareCollector {
}
}
/// Re-parses a module directly from saved bytes, bypassing `source.content()`.
///
/// Used by `failsafe_parse` to serve the last good AST when the live file has a syntax error.
pub async fn parse_from_rope(
rope: Rope,
source: ResolvedVc<Box<dyn Source>>,
ty: EcmascriptModuleAssetType,
transforms: ResolvedVc<EcmascriptInputTransforms>,
node_env: RcStr,
) -> Result<Vc<ParseResult>> {
let ident_vc = source.ident();
let fs_path = ident_vc.path().owned().await?;
let ident = &*ident_vc.to_string().await?;
let file_path_hash = hash_xxh3_hash64(ident) as u128;
let query = ident_vc.await?.query.clone();
let transforms = &*transforms.await?;
parse_file_content(
rope,
&fs_path,
ident,
query,
file_path_hash,
source,
ty,
transforms,
node_env,
false,
false,
)
.await
}
#[cfg(test)]
mod tests {
use swc_core::{
......
......@@ -625,6 +625,7 @@ async fn analyze_ecmascript_module_internal(
comments,
source_map,
source_mapping_url,
program_source: _,
} = &*parsed
else {
return analysis.build(Default::default(), false).await;
......
......@@ -500,6 +500,7 @@ pub(super) async fn split_module(asset: Vc<EcmascriptModuleAsset>) -> Result<Vc<
eval_context,
source_map,
globals,
program_source,
..
} => {
// If the script file is a common js file, we cannot split the module
......@@ -578,6 +579,7 @@ pub(super) async fn split_module(asset: Vc<EcmascriptModuleAsset>) -> Result<Vc<
source_map: source_map.clone(),
eval_context,
source_mapping_url: None,
program_source: program_source.clone(),
})
})
.collect();
......@@ -623,6 +625,7 @@ pub(crate) async fn part_of_module(
eval_context,
globals,
source_map,
program_source,
..
} = &*modules[0].await?
{
......@@ -703,6 +706,7 @@ pub(crate) async fn part_of_module(
globals: globals.clone(),
source_map: source_map.clone(),
source_mapping_url: None,
program_source: program_source.clone(),
}
.cell());
} else {
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment