Unverified Commit dccb95c3 authored by Benjamin Woodruff's avatar Benjamin Woodruff Committed by GitHub

Turbopack: Cache effect errors, replace the write_lock Mutex with a lazily…

Turbopack: Cache effect errors, replace the write_lock Mutex with a lazily created Notify instance (#93476)

This is a mostly-by-hand refactor of some of the caching logic in
https://github.com/vercel/next.js/pull/92300

- **Cache errors.** We often end up applying effects multiple times in
frequent succession, and many filesystem effects already have their own
backoff/retry loops, which can make them very slow when they fail, so it
seems generally better to cache the failures.
- **Add `EffectLastApplied` enum:** It's a matter of opinion, but I
think the `EffectLastApplied` enum makes it easier to reason about the
different states in the system. `Option` was difficult to reason about,
and it would've been even more confiusing if it was an
`Option<Result<(), Arc<dyn EffectError>>`.
- `tokio::sync::Mutex<()>` is overkill here, `tokio::sync::Notify` is a
simpler (and smaller) primitive that does the core of what we needed the
`Mutex` for. We could also use `EventListener` here, but I didn't.
(`EventListener` is really just smol's version of `Notify`)
- We can lazily allocate the `Notify` instance inside of `last_applied`.
We have to grab this lock anyways to check for the cached fast-path. If
we're not in the fast-path, we can re-use the lock guard to create and
store the `Notify` instance.
- **Perf:** The old code's slow path had to always lock `last_applied`
three times. Now, if there's no concurrent writer (common case), it only
locks twice: Once before application, and once after.
- **Perf:** Use a smallvec when computing unique indices to avoid a
bunch of heap allocations. Unless there's a bug, we don't expect any of
the keys to have more than one value (returns an error).
- **Perf:** Store keys as `Box<[u8]>` instead of `Vec<u8>`, saving 8
bytes per key.

Currently, if `effect.dyn_apply()` panics, we just do our best to mark
the effect as unapplied to avoid a deadlock. This could alternatively
poison the object.

## Memory Usage

Memory usage can increase if we're storing errors (heap allocated,
stored in `Arc`) we previously wouldn't have, but if we assume that
errors are uncommon, this refactor decreases the size of the data
structures involved.

Before this change, eagerly allocating a `tokio::sync::Mutex` was
costing 40 bytes per effect, making the entire `EffectEntryState` struct
64 bytes:

<img
src="https://app.graphite.com/user-attachments/assets/989ac229-9e2f-4370-8e32-b14e10e09248.png"
width=500>
<img
src="https://app.graphite.com/user-attachments/assets/f2e14416-17c3-4961-8c00-fa7a81846fa0.png"
width=500>

After the changes, the entire `EffectLastApplied` enum is 40 bytes

<img
src="https://app.graphite.com/user-attachments/assets/18e4fc0a-e7f8-4fad-b9bc-c22ba3eefb6c.png"
width=500>

After wrapping the `EffectLastApplied` enum in a `parking_lot::Mutex`,
it comes in at 48 bytes (after alignment), 16 bytes smaller than the old
struct.
parent 533693f3
...@@ -9838,6 +9838,7 @@ dependencies = [ ...@@ -9838,6 +9838,7 @@ dependencies = [
"serde_json", "serde_json",
"shrink-to-fit", "shrink-to-fit",
"smallvec", "smallvec",
"thiserror 1.0.69",
"tokio", "tokio",
"tokio-util", "tokio-util",
"tracing", "tracing",
......
...@@ -950,8 +950,8 @@ impl FileSystem for DiskFileSystem { ...@@ -950,8 +950,8 @@ impl FileSystem for DiskFileSystem {
type Error = AnyhowWrapper; type Error = AnyhowWrapper;
type Value = u128; type Value = u128;
fn key(&self) -> Vec<u8> { fn key(&self) -> Box<[u8]> {
self.full_path.as_os_str().as_encoded_bytes().to_vec() self.full_path.as_os_str().as_encoded_bytes().into()
} }
fn value(&self) -> &u128 { fn value(&self) -> &u128 {
...@@ -1126,8 +1126,8 @@ impl FileSystem for DiskFileSystem { ...@@ -1126,8 +1126,8 @@ impl FileSystem for DiskFileSystem {
type Error = AnyhowWrapper; type Error = AnyhowWrapper;
type Value = u128; type Value = u128;
fn key(&self) -> Vec<u8> { fn key(&self) -> Box<[u8]> {
self.full_path.as_os_str().as_encoded_bytes().to_vec() self.full_path.as_os_str().as_encoded_bytes().into()
} }
fn value(&self) -> &u128 { fn value(&self) -> &u128 {
......
...@@ -45,6 +45,7 @@ serde = { workspace = true, features = ["rc", "derive"] } ...@@ -45,6 +45,7 @@ serde = { workspace = true, features = ["rc", "derive"] }
serde_json = { workspace = true } serde_json = { workspace = true }
shrink-to-fit = { workspace = true, features = ["indexmap", "serde_json", "smallvec", "nightly"] } shrink-to-fit = { workspace = true, features = ["indexmap", "serde_json", "smallvec", "nightly"] }
smallvec = { workspace = true } smallvec = { workspace = true }
thiserror = { workspace = true }
tokio = { workspace = true, features = ["full"] } tokio = { workspace = true, features = ["full"] }
tokio-util = { workspace = true } tokio-util = { workspace = true }
tracing = { workspace = true } tracing = { workspace = true }
......
...@@ -2,20 +2,23 @@ use std::{ ...@@ -2,20 +2,23 @@ use std::{
any::Any, any::Any,
error::Error as StdError, error::Error as StdError,
future::Future, future::Future,
mem::{forget, replace},
pin::Pin, pin::Pin,
sync::{Arc, OnceLock}, sync::{Arc, OnceLock},
}; };
use anyhow::{Result, bail}; use anyhow::Result;
use futures::{StreamExt, TryStreamExt}; use futures::{StreamExt, TryStreamExt};
use parking_lot::Mutex; use parking_lot::{Mutex, MutexGuard};
use rustc_hash::{FxHashMap, FxHashSet}; use rustc_hash::{FxHashMap, FxHashSet};
use smallvec::SmallVec;
use tracing::Instrument; use tracing::Instrument;
use turbo_dyn_eq_hash::DynPartialEq; use turbo_dyn_eq_hash::DynPartialEq;
use crate::{ use crate::{
self as turbo_tasks, CollectiblesSource, NonLocalValue, ReadRef, ResolvedVc, TryJoinIterExt, self as turbo_tasks, CollectiblesSource, NonLocalValue, ReadRef, ResolvedVc, TryJoinIterExt,
emit, emit,
event::Event,
manager::{debug_assert_in_top_level_task, debug_assert_not_in_top_level_task}, manager::{debug_assert_in_top_level_task, debug_assert_not_in_top_level_task},
trace::TraceRawVcs, trace::TraceRawVcs,
}; };
...@@ -43,7 +46,7 @@ pub trait Effect: TraceRawVcs + NonLocalValue + Send + Sync + 'static { ...@@ -43,7 +46,7 @@ pub trait Effect: TraceRawVcs + NonLocalValue + Send + Sync + 'static {
type Value: Clone + DynPartialEq + Eq + Send + Sync + 'static; type Value: Clone + DynPartialEq + Eq + Send + Sync + 'static;
/// Unique key identifying this effect's target (e.g., absolute path bytes). /// Unique key identifying this effect's target (e.g., absolute path bytes).
fn key(&self) -> Vec<u8>; fn key(&self) -> Box<[u8]>;
/// Extract the value part of this effect for storage and comparison. /// Extract the value part of this effect for storage and comparison.
fn value(&self) -> &Self::Value; fn value(&self) -> &Self::Value;
...@@ -71,37 +74,30 @@ pub trait Effect: TraceRawVcs + NonLocalValue + Send + Sync + 'static { ...@@ -71,37 +74,30 @@ pub trait Effect: TraceRawVcs + NonLocalValue + Send + Sync + 'static {
pub trait EffectError: StdError + TraceRawVcs + NonLocalValue + Send + Sync + 'static {} pub trait EffectError: StdError + TraceRawVcs + NonLocalValue + Send + Sync + 'static {}
impl<T> EffectError for T where T: StdError + TraceRawVcs + NonLocalValue + Send + Sync + 'static {} impl<T> EffectError for T where T: StdError + TraceRawVcs + NonLocalValue + Send + Sync + 'static {}
/// Per-key entry in the effect state storage. enum EffectLastApplied {
/// Unapplied,
/// - `last_applied`: the value that was last successfully written (sync-readable for fast-path InProgress {
/// dedup) write_event: Event,
/// - `write_lock`: async mutex held during the actual write; ensures only one concurrent write per },
/// key Applied {
struct EffectStateEntry { value: Box<dyn Any + Send + Sync>,
last_applied: Mutex<Option<Box<dyn Any + Send + Sync>>>, result: Result<(), Arc<dyn EffectError>>,
write_lock: tokio::sync::Mutex<()>, },
}
impl Default for EffectStateEntry {
fn default() -> Self {
Self {
last_applied: Mutex::new(None),
write_lock: tokio::sync::Mutex::new(()),
}
}
} }
/// Per-key entry in the effect state storage.
type EffectStateEntry = Arc<Mutex<EffectLastApplied>>;
/// Shared state storage for tracking applied effects. Stored on the filesystem implementation /// Shared state storage for tracking applied effects. Stored on the filesystem implementation
/// (e.g. DiskFileSystemInner). /// (e.g. DiskFileSystemInner).
#[derive(Default)] #[derive(Default)]
pub struct EffectStateStorage { pub struct EffectStateStorage {
effect_state: Mutex<FxHashMap<Vec<u8>, Arc<EffectStateEntry>>>, effect_state: Mutex<FxHashMap<Box<[u8]>, EffectStateEntry>>,
} }
// Private wrapper trait to allow dynamic dispatch of an `Effect`. This is similar to the pattern // Private wrapper trait to allow dynamic dispatch of an `Effect`. This is similar to the pattern
// that the dynosaur crate uses: https://github.com/spastorino/dynosaur // that the dynosaur crate uses: https://github.com/spastorino/dynosaur
trait DynEffect: TraceRawVcs + NonLocalValue + Send + Sync + 'static { trait DynEffect: TraceRawVcs + NonLocalValue + Send + Sync + 'static {
fn key(&self) -> Vec<u8>; fn key(&self) -> Box<[u8]>;
/// Compare `self`'s value against a stored `Box<dyn Any>`, using [`DynPartialEq`]. /// Compare `self`'s value against a stored `Box<dyn Any>`, using [`DynPartialEq`].
fn eq_value_dyn(&self, other: &dyn Any) -> bool; fn eq_value_dyn(&self, other: &dyn Any) -> bool;
fn value_dyn(&self) -> Box<dyn Any + Send + Sync>; fn value_dyn(&self) -> Box<dyn Any + Send + Sync>;
...@@ -113,7 +109,7 @@ impl<T> DynEffect for T ...@@ -113,7 +109,7 @@ impl<T> DynEffect for T
where where
T: Effect, T: Effect,
{ {
fn key(&self) -> Vec<u8> { fn key(&self) -> Box<[u8]> {
Effect::key(self) Effect::key(self)
} }
...@@ -130,11 +126,12 @@ where ...@@ -130,11 +126,12 @@ where
} }
fn dyn_apply<'a>(&'a self) -> DynEffectApplyFuture<'a> { fn dyn_apply<'a>(&'a self) -> DynEffectApplyFuture<'a> {
Box::pin(async move { Effect::apply(self).await.map_err(anyhow::Error::from) }) Box::pin(async move { Effect::apply(self).await.map_err(|err| Arc::new(err) as _) })
} }
} }
type DynEffectApplyFuture<'a> = Pin<Box<dyn Future<Output = Result<()>> + Send + 'a>>; type DynEffectApplyFuture<'a> =
Pin<Box<dyn Future<Output = Result<(), Arc<dyn EffectError>>> + Send + 'a>>;
/// A trait to emit a task effect as collectible. This trait only has one implementation, /// A trait to emit a task effect as collectible. This trait only has one implementation,
/// `EffectInstance` and no other implementation is allowed. The trait is private to this module so /// `EffectInstance` and no other implementation is allowed. The trait is private to this module so
...@@ -246,11 +243,17 @@ pub async fn take_effects(source: impl CollectiblesSource) -> Result<Effects> { ...@@ -246,11 +243,17 @@ pub async fn take_effects(source: impl CollectiblesSource) -> Result<Effects> {
}) })
} }
#[derive(thiserror::Error, Debug, TraceRawVcs, NonLocalValue)]
#[error("Conflicting effects for the same key (key length: {key_len} bytes)")]
struct ConflictingEffectError {
key_len: usize,
}
/// Cached result of grouping effects by key and dedup/conflict detection. /// Cached result of grouping effects by key and dedup/conflict detection.
/// Each entry is (index into `effects`, Arc to per-key state entry). /// Each entry is (index into `effects`, per-key state entry).
/// The `Arc<EffectStateEntry>` is resolved once and cached to avoid repeated map lookups on /// The `EffectStateEntry` is resolved once and cached to avoid repeated map lookups on subsequent
/// subsequent `apply()` calls. /// `apply()` calls.
type UniqueEffectIndices = Result<Vec<(usize, Arc<EffectStateEntry>)>, String>; type UniqueEffectIndices = Result<Vec<(usize, EffectStateEntry)>, Arc<ConflictingEffectError>>;
/// Captured effects from an operation. This struct can be used to return Effects from a turbo-tasks /// Captured effects from an operation. This struct can be used to return Effects from a turbo-tasks
/// function and apply them later. /// function and apply them later.
...@@ -297,7 +300,7 @@ impl Effects { ...@@ -297,7 +300,7 @@ impl Effects {
/// consistency][crate::OperationVc::read_strongly_consistent]. /// consistency][crate::OperationVc::read_strongly_consistent].
/// ///
/// See [`take_effects`] for example usage. /// See [`take_effects`] for example usage.
pub async fn apply(&self) -> Result<()> { pub async fn apply(&self) -> Result<(), Arc<dyn EffectError>> {
debug_assert_in_top_level_task( debug_assert_in_top_level_task(
"Effects::apply must be called from a top-level task to avoid unintended \ "Effects::apply must be called from a top-level task to avoid unintended \
re-executions due to eventual consistency", re-executions due to eventual consistency",
...@@ -310,90 +313,134 @@ impl Effects { ...@@ -310,90 +313,134 @@ impl Effects {
async { async {
// Compute unique (index, state_entry) pairs once; reuse on later calls. // Compute unique (index, state_entry) pairs once; reuse on later calls.
// The Arc<EffectStateEntry> is resolved from the state map on first call and cached // The EffectStateEntry is resolved from the state map on first call and cached
// here, so subsequent apply() calls bypass the map lookup entirely. // here, so subsequent apply() calls bypass the map lookup entirely.
let unique_indices = self.unique_indices.get_or_init(|| { let unique_indices = self
let mut by_key: FxHashMap<Vec<u8>, Vec<usize>> = FxHashMap::default(); .unique_indices
for (i, effect) in self.effects.iter().enumerate() { .get_or_init(|| {
let key = effect.inner.key(); let mut by_key: FxHashMap<Box<[u8]>, SmallVec<[usize; 1]>> =
by_key.entry(key).or_default().push(i); FxHashMap::default();
} for (i, effect) in self.effects.iter().enumerate() {
let key = effect.inner.key();
by_key.entry(key).or_default().push(i);
}
let mut indices = Vec::with_capacity(by_key.len()); let mut indices = Vec::with_capacity(by_key.len());
for (key, group) in by_key { for (key, group) in by_key {
if group.len() > 1 { if group.len() > 1 {
let first_value = self.effects[group[0]].inner.value_dyn(); let first_value = self.effects[group[0]].inner.value_dyn();
for &idx in &group[1..] { for &idx in &group[1..] {
if !self.effects[idx].inner.eq_value_dyn(&*first_value) { if !self.effects[idx].inner.eq_value_dyn(&*first_value) {
return Err(format!( return Err(Arc::new(ConflictingEffectError {
"Conflicting effects for the same key (key length: {} bytes)", key_len: key.len(),
key.len() }));
)); }
} }
} }
let idx = group[0];
let state_storage = self.effects[idx].inner.state_storage();
// Look up or create the per-key state entry and cache the Arc directly.
let entry = state_storage
.effect_state
.lock()
.entry(key)
.or_insert_with(|| Arc::new(Mutex::new(EffectLastApplied::Unapplied)))
.clone();
indices.push((idx, entry));
} }
let idx = group[0]; Ok(indices)
let state_storage = self.effects[idx].inner.state_storage(); })
// Look up or create the per-key state entry and cache the Arc directly. .as_ref()
let entry = state_storage .map_err(|err| err.clone() as Arc<_>)?;
.effect_state
.lock()
.entry(key)
.or_insert_with(|| Arc::new(EffectStateEntry::default()))
.clone();
indices.push((idx, entry));
}
Ok(indices)
});
let unique_indices = match unique_indices {
Ok(indices) => indices,
Err(msg) => bail!("{msg}"),
};
// Apply effects using cached (index, state_entry) pairs. // Apply effects using cached (index, state_entry) pairs.
// Hot path: no map lookup — Arc<EffectStateEntry> is cached in unique_indices. // Hot path: no map lookup — EffectStateEntry is cached in unique_indices.
futures::stream::iter(unique_indices.iter()) futures::stream::iter(unique_indices.iter())
.map(Ok::<_, anyhow::Error>) .map(Ok::<_, Arc<dyn EffectError>>)
.try_for_each_concurrent(APPLY_EFFECTS_CONCURRENCY_LIMIT, async |(idx, entry)| { .try_for_each_concurrent(APPLY_EFFECTS_CONCURRENCY_LIMIT, async |(idx, entry)| {
let effect: &dyn DynEffect = &*self.effects[*idx].inner; let effect: &dyn DynEffect = &*self.effects[*idx].inner;
// Fast path: check if the stored value already matches (sync, no await) // If `effect.dyn_apply` panics or the apply future is dropped before
{ // completion, the guard's drop impl resets the per-key state to `Unapplied`
let stored = entry.last_applied.lock(); // and notifies other waiters via the `Event` it recovers from the previous
if let Some(stored_val) = stored.as_ref() // `InProgress`, so they retry rather than deadlock or observe a stale
&& effect.eq_value_dyn(&**stored_val) // "panic" cache entry.
{ struct EventGuard<'a> {
return Ok(()); entry: &'a EffectStateEntry,
}
} }
impl Drop for EventGuard<'_> {
// Slow path: acquire the write lock and re-check before writing fn drop(&mut self) {
let _write_guard = entry.write_lock.lock().await; let prev_state =
replace(&mut *self.entry.lock(), EffectLastApplied::Unapplied);
{ let EffectLastApplied::InProgress { write_event } = prev_state else {
let stored = entry.last_applied.lock(); unreachable!("EventGuard: prev_state must be InProgress");
if let Some(stored_val) = stored.as_ref() };
&& effect.eq_value_dyn(&**stored_val) write_event.notify(usize::MAX);
{
return Ok(());
} }
} }
// Clear stored value so concurrent fast-path checks won't let begin_in_progress = |mut last_applied_guard: MutexGuard<'_, _>| {
// match against the stale value while we're writing. *last_applied_guard = EffectLastApplied::InProgress {
*entry.last_applied.lock() = None; write_event: Event::new(|| {
|| "effect application in progress".to_string()
// Apply the effect }),
effect.dyn_apply().await?; };
EventGuard { entry }
// Store the new value (sync) };
*entry.last_applied.lock() = Some(effect.value_dyn());
let event_guard = loop {
Ok(()) let listener;
{
let last_applied_guard = entry.lock();
match &*last_applied_guard {
EffectLastApplied::Unapplied => {
break begin_in_progress(last_applied_guard);
}
EffectLastApplied::Applied { value, result } => {
// Fast path: check if the stored value already matches
if effect.eq_value_dyn(&**value) {
return result.clone();
} else {
break begin_in_progress(last_applied_guard);
}
}
EffectLastApplied::InProgress { write_event } => {
// `Event::listen` registers the listener immediately, so
// notifications fired after we drop `last_applied_guard`
// cannot be missed.
listener = write_event.listen();
}
}
};
// We didn't break out of the loop: There's a concurrent in-progress
// application. Wait for it to finish and then read `last_applied` again.
listener.await;
};
// Apply the effect. InProgress is visible to concurrent readers, preventing
// stale fast-path matches.
let effect_result = effect.dyn_apply().await;
// Update the state, clear the EventGuard
let prev_state = replace(
&mut *entry.lock(),
EffectLastApplied::Applied {
value: effect.value_dyn(),
result: effect_result.clone(),
},
);
// don't run the `Drop` impl on `EventGuard`. We'll do the notification
// ourselves.
forget(event_guard);
let EffectLastApplied::InProgress { write_event } = prev_state else {
unreachable!("Effect applied: prev_state must be InProgress");
};
write_event.notify(usize::MAX);
effect_result
}) })
.await?; .await
anyhow::Ok(())
} }
.instrument(span) .instrument(span)
.await .await
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment