Unverified Commit 05553796 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

turbo-tasks: task-storage memory wins (#93720)

## Summary

Four small, independent changes that shrink `TaskStorage` and the data
it owns:

Recommend reviewing commit-by-commit

1. **`Arc<CachedTaskType>` → `triomphe::Arc<CachedTaskType>`.**
`triomphe::Arc` is already a workspace dep used in `ReadRef` /
`SharedReference`. `CachedTaskType` never appears in a `Weak<...>`, so
we can drop the weak count and the CAS in `drop_slow`. Saves one `usize`
per allocation. Migrated via a `CachedTaskTypeArc` newtype so the
bincode `Encode`/`Decode` impls don't need to cross the orphan rule.

2. **Niche-encode `CellDependency`.** The `cell_dependencies` /
`cell_dependents` sets used to hold `(CellRef, Option<u64>)` tuples —
`Option<u64>` cost a full 16 B (8 B discriminant + 8 B value, aligned),
making each element 32 B. A `CellDependency` enum with two variants
(`All(CellRef)` / `Hash(CellRef, u64)`) lets the layout algorithm reuse
the niche on `ValueTypeId` (`NonZero<u16>`) inside
`CellRef.cell.type_id` for the variant tag. Element size drops 32 → 24
B; `LazyField` from 56 → 48 B. The same enum backs both forward and
reverse edges — for `cell_dependents` we re-point `CellRef.task` at the
dependent task.

Added `CellDependency::into_parts()` and use it in
`iter_cell_dependents` / `iter_cell_dependencies` hot loops so the
discriminant is checked once instead of twice via back-to-back
`cell_ref()` + `key()` calls.

3. **`TaskStorage::lazy: Vec<LazyField>` → `TinyVec<LazyField>`.** The
lazy vec only ever holds ~25 elements (one per declared lazy field in
the schema). Swapping `Vec`'s 24 B `(ptr, len, cap)` header for `(ptr,
len: u8, cap: u8)` + 6 B padding gives 16 B. Drops
`size_of::<TaskStorage>()` from 136 → 128 B.

`TinyVec` is hand-rolled so I added a push/iter micro-benchmark to
confirm it doesn't lose performance vs std `Vec`. Results below.

4. **Rightsize collections** → Explore the `AutoSet`/`AutoMap` types in
storage_schema and ensure each one is maximally sized for its natural
alignment.


## Benchmark results

### `next build` on a representative app (15 runs each, M4 Pro,
`caffeinate -dimsu nice -n -20`)

Fresh same-day baseline against branch:

| metric | canary | branch | Δ | 95% CI | significant? |
|---|---:|---:|---:|---|:---:|
| wall time | 40.83s | 41.12s | +0.7% | [−1.07s, +1.64s] | no |
| user time | 282.27s | 283.21s | +0.3% | [−1.02s, +2.89s] | no |
| sys time | 69.38s | 71.26s | +2.7% | [−1.54s, +5.32s] | no |
| **MaxRSS** | **12.47 GB** | **12.04 GB** | **−3.4%** | **[−0.48 GB,
−0.38 GB]** | **yes** |

**MaxRSS is the headline.** −0.43 GB on a 12.5 GB working set, with
t=−17.86 (every branch run lower than every canary run, CV ≤ 0.6% on
both sides). Wall / user / sys are all within noise — this PR is a
memory win with no measurable timing impact.

### `TinyVec` vs `Vec` micro-bench (`turbo-tasks/benches/tiny_vec.rs`,
200 samples each)

| n | Vec push | TinyVec push | Δ% | Vec iter | TinyVec iter | Δ% |
|---:|---:|---:|---:|---:|---:|---:|
| 0 | 1.31ns | 894ps | **−31.8%** | 598ps | 596ps | −0.4% |
| 1 | 16.92ns | 14.75ns | **−12.9%** | 964ps | 952ps | −1.2% |
| 4 | 17.93ns | 15.93ns | **−11.1%** | 1.49ns | 1.50ns | +0.5% |
| 8 | 63.13ns | 45.24ns | **−28.3%** | 1.97ns | 1.96ns | −0.2% |
| 16 | 97.35ns | 79.91ns | **−17.9%** | 3.16ns | 3.14ns | −0.5% |
| 24 | 137.41ns | 119.88ns | **−12.8%** | 4.30ns | 4.30ns | +0.0% |

TinyVec push is 11–32% faster than Vec push across all realistic sizes;
iter is identical. Run with `cargo bench -p turbo-tasks --bench
tiny_vec`.

### `task_overhead/turbo` Criterion bench (M4 Pro, `--sample-size 200`)

| variant | dur | canary | branch | Δ | significant? |
|---|---:|---:|---:|---:|:---:|
| turbo-uncached | 1µs | 9.77 µs | 9.68 µs | −1.0% | yes |
| turbo-uncached | 1000µs | 1.01 ms | 1.01 ms | −0.1% | yes |
| turbo-cached-same-keys | 1µs | 198.6 ns | 191.9 ns | −3.4% | yes |
| turbo-cached-same-keys | 100µs | 226.5 ns | 208.1 ns | −8.1% | yes |
| turbo-cached-different-keys | 1µs | 233.8 ns | 224.1 ns | −4.2% | yes
|
| turbo-cached-different-keys | 100µs | 305.3 ns | 246.9 ns | −19.1% |
yes |
| turbo-uncached-parallel | 10µs | 1.63 µs | 1.54 µs | −5.8% | yes |
| turbo-uncached-parallel | 100µs | 8.41 µs | 7.88 µs | −6.3% | yes |


<!-- NEXT_JS_LLM_PR -->
parent dadc154d
...@@ -12,34 +12,37 @@ use bincode::{ ...@@ -12,34 +12,37 @@ use bincode::{
}; };
use rustc_hash::FxHasher; use rustc_hash::FxHasher;
type InnerMap<K, V> = AutoMap<K, V, BuildHasherDefault<FxHasher>, 1>; type InnerMap<K, V, const I: usize> = AutoMap<K, V, BuildHasherDefault<FxHasher>, I>;
/// A map optimized for reference counting, backed by AutoMap. /// A map optimized for reference counting, backed by AutoMap.
/// ///
/// Entries are automatically removed when their count reaches zero. /// Entries are automatically removed when their count reaches zero.
/// This provides memory-efficient storage for sparse counter data. /// This provides memory-efficient storage for sparse counter data.
///
/// The `I` const generic forwards the inline capacity to the backing `AutoMap`
/// — see the schema field-by-field sizing for the chosen values.
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct CounterMap<K, V>(InnerMap<K, V>); pub struct CounterMap<K, V, const I: usize>(InnerMap<K, V, I>);
impl<K, V> Default for CounterMap<K, V> { impl<K, V, const I: usize> Default for CounterMap<K, V, I> {
fn default() -> Self { fn default() -> Self {
Self(InnerMap::default()) Self(InnerMap::default())
} }
} }
impl<K: Eq + Hash, V: Eq> PartialEq for CounterMap<K, V> { impl<K: Eq + Hash, V: Eq, const I: usize> PartialEq for CounterMap<K, V, I> {
fn eq(&self, other: &Self) -> bool { fn eq(&self, other: &Self) -> bool {
self.0 == other.0 self.0 == other.0
} }
} }
impl<K: Encode, V: Encode> Encode for CounterMap<K, V> { impl<K: Encode, V: Encode, const I: usize> Encode for CounterMap<K, V, I> {
fn encode<E: Encoder>(&self, encoder: &mut E) -> Result<(), EncodeError> { fn encode<E: Encoder>(&self, encoder: &mut E) -> Result<(), EncodeError> {
self.0.encode(encoder) self.0.encode(encoder)
} }
} }
impl<Context, K, V> Decode<Context> for CounterMap<K, V> impl<Context, K, V, const I: usize> Decode<Context> for CounterMap<K, V, I>
where where
K: Decode<Context> + Eq + Hash, K: Decode<Context> + Eq + Hash,
V: Decode<Context>, V: Decode<Context>,
...@@ -80,7 +83,7 @@ impl CounterValue for i32 { ...@@ -80,7 +83,7 @@ impl CounterValue for i32 {
} }
} }
impl<K, V> CounterMap<K, V> { impl<K, V, const I: usize> CounterMap<K, V, I> {
pub fn new() -> Self { pub fn new() -> Self {
Self(AutoMap::default()) Self(AutoMap::default())
} }
...@@ -138,16 +141,16 @@ impl<K, V> CounterMap<K, V> { ...@@ -138,16 +141,16 @@ impl<K, V> CounterMap<K, V> {
} }
} }
impl<K, V> IntoIterator for CounterMap<K, V> { impl<K, V, const I: usize> IntoIterator for CounterMap<K, V, I> {
type Item = (K, V); type Item = (K, V);
type IntoIter = <InnerMap<K, V> as IntoIterator>::IntoIter; type IntoIter = <InnerMap<K, V, I> as IntoIterator>::IntoIter;
fn into_iter(self) -> Self::IntoIter { fn into_iter(self) -> Self::IntoIter {
self.0.into_iter() self.0.into_iter()
} }
} }
impl<K: Hash + Eq, V: CounterValue> CounterMap<K, V> { impl<K: Hash + Eq, V: CounterValue, const I: usize> CounterMap<K, V, I> {
/// Insert a key-value pair. Panics if value is zero (invariant: zero values are not stored). /// Insert a key-value pair. Panics if value is zero (invariant: zero values are not stored).
pub fn insert(&mut self, key: K, value: V) -> Option<V> { pub fn insert(&mut self, key: K, value: V) -> Option<V> {
debug_assert!( debug_assert!(
...@@ -297,7 +300,7 @@ mod tests { ...@@ -297,7 +300,7 @@ mod tests {
#[test] #[test]
fn test_update_count_new_entry() { fn test_update_count_new_entry() {
let mut map: CounterMap<u32, u32> = CounterMap::new(); let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
// Adding new entry crosses zero (from nothing to something) // Adding new entry crosses zero (from nothing to something)
assert!(map.update_count(1, 5)); assert!(map.update_count(1, 5));
assert_eq!(map.get(&1), Some(&5)); assert_eq!(map.get(&1), Some(&5));
...@@ -305,7 +308,7 @@ mod tests { ...@@ -305,7 +308,7 @@ mod tests {
#[test] #[test]
fn test_update_count_increment() { fn test_update_count_increment() {
let mut map: CounterMap<u32, u32> = CounterMap::new(); let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
map.update_count(1, 5); map.update_count(1, 5);
// Incrementing existing entry doesn't cross zero // Incrementing existing entry doesn't cross zero
assert!(!map.update_count(1, 3)); assert!(!map.update_count(1, 3));
...@@ -314,7 +317,7 @@ mod tests { ...@@ -314,7 +317,7 @@ mod tests {
#[test] #[test]
fn test_update_count_removal_on_zero() { fn test_update_count_removal_on_zero() {
let mut map: CounterMap<u32, i32> = CounterMap::new(); let mut map: CounterMap<u32, i32, 1> = CounterMap::new();
map.update_count(1, 5); map.update_count(1, 5);
// Subtracting to zero removes entry and crosses zero // Subtracting to zero removes entry and crosses zero
assert!(map.update_count(1, -5)); assert!(map.update_count(1, -5));
...@@ -324,7 +327,7 @@ mod tests { ...@@ -324,7 +327,7 @@ mod tests {
#[test] #[test]
fn test_update_count_zero_delta_on_empty() { fn test_update_count_zero_delta_on_empty() {
let mut map: CounterMap<u32, u32> = CounterMap::new(); let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
// Adding zero to non-existent entry doesn't create it // Adding zero to non-existent entry doesn't create it
assert!(!map.update_count(1, 0)); assert!(!map.update_count(1, 0));
assert!(map.is_empty()); assert!(map.is_empty());
...@@ -332,14 +335,14 @@ mod tests { ...@@ -332,14 +335,14 @@ mod tests {
#[test] #[test]
fn test_update_and_get_new_entry() { fn test_update_and_get_new_entry() {
let mut map: CounterMap<u32, u32> = CounterMap::new(); let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
assert_eq!(map.update_and_get(1, 5), 5); assert_eq!(map.update_and_get(1, 5), 5);
assert_eq!(map.get(&1), Some(&5)); assert_eq!(map.get(&1), Some(&5));
} }
#[test] #[test]
fn test_update_and_get_increment() { fn test_update_and_get_increment() {
let mut map: CounterMap<u32, u32> = CounterMap::new(); let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
map.update_and_get(1, 5); map.update_and_get(1, 5);
assert_eq!(map.update_and_get(1, 3), 8); assert_eq!(map.update_and_get(1, 3), 8);
assert_eq!(map.get(&1), Some(&8)); assert_eq!(map.get(&1), Some(&8));
...@@ -347,7 +350,7 @@ mod tests { ...@@ -347,7 +350,7 @@ mod tests {
#[test] #[test]
fn test_update_and_get_removal() { fn test_update_and_get_removal() {
let mut map: CounterMap<u32, i32> = CounterMap::new(); let mut map: CounterMap<u32, i32, 1> = CounterMap::new();
map.update_and_get(1, 5); map.update_and_get(1, 5);
assert_eq!(map.update_and_get(1, -5), 0); assert_eq!(map.update_and_get(1, -5), 0);
assert!(map.is_empty()); assert!(map.is_empty());
...@@ -355,7 +358,7 @@ mod tests { ...@@ -355,7 +358,7 @@ mod tests {
#[test] #[test]
fn test_add_entry() { fn test_add_entry() {
let mut map: CounterMap<u32, u32> = CounterMap::new(); let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
map.add_entry(1, 10); map.add_entry(1, 10);
assert_eq!(map.get(&1), Some(&10)); assert_eq!(map.get(&1), Some(&10));
} }
...@@ -363,14 +366,14 @@ mod tests { ...@@ -363,14 +366,14 @@ mod tests {
#[test] #[test]
#[should_panic(expected = "Entry already exists")] #[should_panic(expected = "Entry already exists")]
fn test_add_entry_panics_on_duplicate() { fn test_add_entry_panics_on_duplicate() {
let mut map: CounterMap<u32, u32> = CounterMap::new(); let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
map.add_entry(1, 10); map.add_entry(1, 10);
map.add_entry(1, 20); // Should panic map.add_entry(1, 20); // Should panic
} }
#[test] #[test]
fn test_update_positive_crossing_new_positive() { fn test_update_positive_crossing_new_positive() {
let mut map: CounterMap<u32, i32> = CounterMap::new(); let mut map: CounterMap<u32, i32, 1> = CounterMap::new();
// From nothing to positive - crosses positive boundary // From nothing to positive - crosses positive boundary
assert!(map.update_positive_crossing(1, 5)); assert!(map.update_positive_crossing(1, 5));
assert_eq!(map.get(&1), Some(&5)); assert_eq!(map.get(&1), Some(&5));
...@@ -378,7 +381,7 @@ mod tests { ...@@ -378,7 +381,7 @@ mod tests {
#[test] #[test]
fn test_update_positive_crossing_new_negative() { fn test_update_positive_crossing_new_negative() {
let mut map: CounterMap<u32, i32> = CounterMap::new(); let mut map: CounterMap<u32, i32, 1> = CounterMap::new();
// From nothing to negative - doesn't cross positive boundary // From nothing to negative - doesn't cross positive boundary
assert!(!map.update_positive_crossing(1, -5)); assert!(!map.update_positive_crossing(1, -5));
assert_eq!(map.get(&1), Some(&-5)); assert_eq!(map.get(&1), Some(&-5));
...@@ -386,7 +389,7 @@ mod tests { ...@@ -386,7 +389,7 @@ mod tests {
#[test] #[test]
fn test_update_positive_crossing_stay_positive() { fn test_update_positive_crossing_stay_positive() {
let mut map: CounterMap<u32, i32> = CounterMap::new(); let mut map: CounterMap<u32, i32, 1> = CounterMap::new();
map.update_positive_crossing(1, 5); map.update_positive_crossing(1, 5);
// Staying positive doesn't cross boundary // Staying positive doesn't cross boundary
assert!(!map.update_positive_crossing(1, 3)); assert!(!map.update_positive_crossing(1, 3));
...@@ -395,7 +398,7 @@ mod tests { ...@@ -395,7 +398,7 @@ mod tests {
#[test] #[test]
fn test_update_positive_crossing_to_non_positive() { fn test_update_positive_crossing_to_non_positive() {
let mut map: CounterMap<u32, i32> = CounterMap::new(); let mut map: CounterMap<u32, i32, 1> = CounterMap::new();
map.update_positive_crossing(1, 5); map.update_positive_crossing(1, 5);
// Crossing to non-positive // Crossing to non-positive
assert!(map.update_positive_crossing(1, -8)); assert!(map.update_positive_crossing(1, -8));
...@@ -404,7 +407,7 @@ mod tests { ...@@ -404,7 +407,7 @@ mod tests {
#[test] #[test]
fn test_update_positive_crossing_to_zero_removes() { fn test_update_positive_crossing_to_zero_removes() {
let mut map: CounterMap<u32, i32> = CounterMap::new(); let mut map: CounterMap<u32, i32, 1> = CounterMap::new();
map.update_positive_crossing(1, 5); map.update_positive_crossing(1, 5);
// Crossing to zero removes and crosses boundary // Crossing to zero removes and crosses boundary
assert!(map.update_positive_crossing(1, -5)); assert!(map.update_positive_crossing(1, -5));
...@@ -413,14 +416,14 @@ mod tests { ...@@ -413,14 +416,14 @@ mod tests {
#[test] #[test]
fn test_update_with_create() { fn test_update_with_create() {
let mut map: CounterMap<u32, u32> = CounterMap::new(); let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
map.update_with(1, |_| Some(10)); map.update_with(1, |_| Some(10));
assert_eq!(map.get(&1), Some(&10)); assert_eq!(map.get(&1), Some(&10));
} }
#[test] #[test]
fn test_update_with_modify() { fn test_update_with_modify() {
let mut map: CounterMap<u32, u32> = CounterMap::new(); let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
map.update_with(1, |_| Some(10)); map.update_with(1, |_| Some(10));
map.update_with(1, |v| v.map(|x| x + 5)); map.update_with(1, |v| v.map(|x| x + 5));
assert_eq!(map.get(&1), Some(&15)); assert_eq!(map.get(&1), Some(&15));
...@@ -428,7 +431,7 @@ mod tests { ...@@ -428,7 +431,7 @@ mod tests {
#[test] #[test]
fn test_update_with_remove() { fn test_update_with_remove() {
let mut map: CounterMap<u32, u32> = CounterMap::new(); let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
map.update_with(1, |_| Some(10)); map.update_with(1, |_| Some(10));
map.update_with(1, |_| None); map.update_with(1, |_| None);
assert!(map.is_empty()); assert!(map.is_empty());
...@@ -436,14 +439,14 @@ mod tests { ...@@ -436,14 +439,14 @@ mod tests {
#[test] #[test]
fn test_update_with_no_op() { fn test_update_with_no_op() {
let mut map: CounterMap<u32, u32> = CounterMap::new(); let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
map.update_with(1, |_| None); map.update_with(1, |_| None);
assert!(map.is_empty()); assert!(map.is_empty());
} }
#[test] #[test]
fn test_len_and_is_empty() { fn test_len_and_is_empty() {
let mut map: CounterMap<u32, u32> = CounterMap::new(); let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
assert!(map.is_empty()); assert!(map.is_empty());
assert_eq!(map.len(), 0); assert_eq!(map.len(), 0);
...@@ -457,7 +460,7 @@ mod tests { ...@@ -457,7 +460,7 @@ mod tests {
#[test] #[test]
fn test_iter() { fn test_iter() {
let mut map: CounterMap<u32, u32> = CounterMap::new(); let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
map.update_count(1, 5); map.update_count(1, 5);
map.update_count(2, 10); map.update_count(2, 10);
......
...@@ -34,8 +34,8 @@ use turbo_tasks::{ ...@@ -34,8 +34,8 @@ use turbo_tasks::{
TaskId, TaskPersistence, TaskPriority, TraitTypeId, TurboTasksBackendApi, TurboTasksPanic, TaskId, TaskPersistence, TaskPriority, TraitTypeId, TurboTasksBackendApi, TurboTasksPanic,
ValueTypeId, ValueTypeId,
backend::{ backend::{
Backend, CachedTaskType, CellContent, CellHash, TaskExecutionSpec, TransientTaskType, Backend, CachedTaskType, CachedTaskTypeArc, CellContent, CellHash, TaskExecutionSpec,
TurboTaskContextError, TurboTaskLocalContextError, TurboTasksError, TransientTaskType, TurboTaskContextError, TurboTaskLocalContextError, TurboTasksError,
TurboTasksExecutionError, TurboTasksExecutionErrorMessage, TypedCellContent, TurboTasksExecutionError, TurboTasksExecutionErrorMessage, TypedCellContent,
VerificationMode, VerificationMode,
}, },
...@@ -70,8 +70,8 @@ use crate::{ ...@@ -70,8 +70,8 @@ use crate::{
}, },
backing_storage::{BackingStorage, SnapshotItem, SnapshotMeta, compute_task_type_hash}, backing_storage::{BackingStorage, SnapshotItem, SnapshotMeta, compute_task_type_hash},
data::{ data::{
ActivenessState, CellRef, CollectibleRef, CollectiblesRef, Dirtyness, InProgressCellState, ActivenessState, CellDependency, CellRef, CollectibleRef, CollectiblesRef, Dirtyness,
InProgressState, InProgressStateInner, OutputValue, TransientTask, InProgressCellState, InProgressState, InProgressStateInner, OutputValue, TransientTask,
}, },
error::TaskError, error::TaskError,
utils::{ utils::{
...@@ -785,7 +785,8 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> { ...@@ -785,7 +785,8 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
&& (!task.immutable() || cfg!(feature = "verify_immutable")) && (!task.immutable() || cfg!(feature = "verify_immutable"))
{ {
let reader = reader.unwrap(); let reader = reader.unwrap();
let _ = task.add_cell_dependents((cell, key, reader)); let _ = task
.add_cell_dependents(CellDependency::new(CellRef { task: reader, cell }, key));
drop(task); drop(task);
// Note: We use `task_pair` earlier to lock the task and its reader at the same // Note: We use `task_pair` earlier to lock the task and its reader at the same
...@@ -797,8 +798,9 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> { ...@@ -797,8 +798,9 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
task: task_id, task: task_id,
cell, cell,
}; };
if !reader_task.remove_outdated_cell_dependencies(&(target, key)) { let dep = CellDependency::new(target, key);
let _ = reader_task.add_cell_dependencies((target, key)); if !reader_task.remove_outdated_cell_dependencies(&dep) {
let _ = reader_task.add_cell_dependencies(dep);
} }
drop(reader_task); drop(reader_task);
} }
...@@ -1545,7 +1547,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> { ...@@ -1545,7 +1547,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
// Only now do we force the allocation. // Only now do we force the allocation.
// NOTE: if our caller had to perform resolution, then this will have already // NOTE: if our caller had to perform resolution, then this will have already
// been boxed and take_box just takes it. // been boxed and take_box just takes it.
let task_type = Arc::new(CachedTaskType { let task_type = CachedTaskTypeArc::new(CachedTaskType {
native_fn, native_fn,
this, this,
arg: arg.take_box(), arg: arg.take_box(),
...@@ -1776,7 +1778,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> { ...@@ -1776,7 +1778,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
} }
} }
fn debug_get_cached_task_type(&self, task_id: TaskId) -> Option<Arc<CachedTaskType>> { fn debug_get_cached_task_type(&self, task_id: TaskId) -> Option<CachedTaskTypeArc> {
let task = self.storage.access_mut(task_id); let task = self.storage.access_mut(task_id);
task.get_persistent_task_type().cloned() task.get_persistent_task_type().cloned()
} }
...@@ -2216,7 +2218,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> { ...@@ -2216,7 +2218,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
Some( Some(
// Collect all dependencies on tasks to check if all dependencies are immutable // Collect all dependencies on tasks to check if all dependencies are immutable
task.iter_output_dependencies() task.iter_output_dependencies()
.chain(task.iter_cell_dependencies().map(|(target, _key)| target.task)) .chain(task.iter_cell_dependencies().map(|dep| dep.cell_ref().task))
.collect::<FxHashSet<_>>(), .collect::<FxHashSet<_>>(),
) )
} else { } else {
...@@ -2255,7 +2257,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> { ...@@ -2255,7 +2257,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
// breaking dependency tracking. // breaking dependency tracking.
old_edges.extend( old_edges.extend(
task.iter_outdated_cell_dependencies() task.iter_outdated_cell_dependencies()
.map(|(target, key)| OutdatedEdge::CellDependency(target, key)), .map(OutdatedEdge::CellDependency),
); );
old_edges.extend( old_edges.extend(
task.iter_outdated_output_dependencies() task.iter_outdated_output_dependencies()
......
...@@ -17,7 +17,7 @@ use crate::{ ...@@ -17,7 +17,7 @@ use crate::{
}, },
storage_schema::TaskStorageAccessors, storage_schema::TaskStorageAccessors,
}, },
data::{CellRef, CollectibleRef, CollectiblesRef}, data::{CellDependency, CellRef, CollectibleRef, CollectiblesRef},
}; };
#[derive(Encode, Decode, Clone)] #[derive(Encode, Decode, Clone)]
...@@ -48,7 +48,7 @@ impl Default for CleanupOldEdgesOperation { ...@@ -48,7 +48,7 @@ impl Default for CleanupOldEdgesOperation {
pub enum OutdatedEdge { pub enum OutdatedEdge {
Child(TaskId), Child(TaskId),
Collectible(CollectibleRef, i32), Collectible(CollectibleRef, i32),
CellDependency(CellRef, Option<u64>), CellDependency(CellDependency),
OutputDependency(TaskId), OutputDependency(TaskId),
CollectiblesDependency(CollectiblesRef), CollectiblesDependency(CollectiblesRef),
} }
...@@ -166,27 +166,28 @@ impl CleanupOldEdgesOperation { ...@@ -166,27 +166,28 @@ impl CleanupOldEdgesOperation {
AggregatedDataUpdate::new().collectibles_update(collectibles), AggregatedDataUpdate::new().collectibles_update(collectibles),
)); ));
} }
OutdatedEdge::CellDependency( OutdatedEdge::CellDependency(dep) => {
let (
CellRef { CellRef {
task: cell_task_id, task: cell_task_id,
cell, cell,
}, },
key, key,
) => { ) = dep.into_parts();
{ {
let mut task = ctx.task(cell_task_id, TaskDataCategory::Data); let mut task = ctx.task(cell_task_id, TaskDataCategory::Data);
task.remove_cell_dependents(&(cell, key, task_id)); task.remove_cell_dependents(&CellDependency::new(
}
{
let mut task = ctx.task(task_id, TaskDataCategory::Data);
task.remove_cell_dependencies(&(
CellRef { CellRef {
task: cell_task_id, task: task_id,
cell, cell,
}, },
key, key,
)); ));
} }
{
let mut task = ctx.task(task_id, TaskDataCategory::Data);
task.remove_cell_dependencies(&dep);
}
} }
OutdatedEdge::OutputDependency(output_task_id) => { OutdatedEdge::OutputDependency(output_task_id) => {
#[cfg(feature = "trace_task_output_dependencies")] #[cfg(feature = "trace_task_output_dependencies")]
......
...@@ -19,7 +19,7 @@ use tracing::info_span; ...@@ -19,7 +19,7 @@ use tracing::info_span;
use tracing::trace_span; use tracing::trace_span;
use turbo_tasks::{ use turbo_tasks::{
CellId, DynTaskInputs, FxIndexMap, RawVc, SharedReference, TaskExecutionReason, TaskId, CellId, DynTaskInputs, FxIndexMap, RawVc, SharedReference, TaskExecutionReason, TaskId,
TaskPriority, TurboTasksBackendApi, TurboTasksCallApi, backend::CachedTaskType, TaskPriority, TurboTasksBackendApi, TurboTasksCallApi, backend::CachedTaskTypeArc,
macro_helpers::NativeFunction, macro_helpers::NativeFunction,
}; };
...@@ -103,7 +103,8 @@ pub trait ExecuteContext<'e>: Sized { ...@@ -103,7 +103,8 @@ pub trait ExecuteContext<'e>: Sized {
/// Uses hash-based lookup which may return multiple candidates due to hash collisions, /// Uses hash-based lookup which may return multiple candidates due to hash collisions,
/// then verifies each candidate by comparing the stored `persistent_task_type`. /// then verifies each candidate by comparing the stored `persistent_task_type`.
/// Returns `Some((task_id, task_type))` if a matching task is found, where `task_type` is /// Returns `Some((task_id, task_type))` if a matching task is found, where `task_type` is
/// the existing `Arc<CachedTaskType>` from storage (avoiding a duplicate allocation). /// the existing `CachedTaskTypeArc` from storage (avoiding a duplicate
/// allocation).
/// ///
/// Accepts exploded components so the caller does not need to box the argument before calling. /// Accepts exploded components so the caller does not need to box the argument before calling.
fn task_by_type( fn task_by_type(
...@@ -111,7 +112,7 @@ pub trait ExecuteContext<'e>: Sized { ...@@ -111,7 +112,7 @@ pub trait ExecuteContext<'e>: Sized {
native_fn: &'static NativeFunction, native_fn: &'static NativeFunction,
this: Option<RawVc>, this: Option<RawVc>,
arg: &dyn DynTaskInputs, arg: &dyn DynTaskInputs,
) -> Option<(TaskId, Arc<CachedTaskType>)>; ) -> Option<(TaskId, CachedTaskTypeArc)>;
fn debug_get_task_description(&self, task_id: TaskId) -> String; fn debug_get_task_description(&self, task_id: TaskId) -> String;
} }
...@@ -606,7 +607,7 @@ struct TaskRestoreEntry { ...@@ -606,7 +607,7 @@ struct TaskRestoreEntry {
/// Another thread claimed the meta restore; we must wait in Phase 3. /// Another thread claimed the meta restore; we must wait in Phase 3.
wait_meta: bool, wait_meta: bool,
/// Task type discovered during Phase 1c data restore (used to update task cache in Phase 2). /// Task type discovered during Phase 1c data restore (used to update task cache in Phase 2).
task_type: Option<Arc<CachedTaskType>>, task_type: Option<CachedTaskTypeArc>,
/// This thread performed the restore for at least one category (set in Phase 1c). /// This thread performed the restore for at least one category (set in Phase 1c).
self_restored: bool, self_restored: bool,
} }
...@@ -985,7 +986,7 @@ impl<'e, B: BackingStorage> ExecuteContext<'e> for ExecuteContextImpl<'e, B> { ...@@ -985,7 +986,7 @@ impl<'e, B: BackingStorage> ExecuteContext<'e> for ExecuteContextImpl<'e, B> {
native_fn: &'static NativeFunction, native_fn: &'static NativeFunction,
this: Option<RawVc>, this: Option<RawVc>,
arg: &dyn DynTaskInputs, arg: &dyn DynTaskInputs,
) -> Option<(TaskId, Arc<CachedTaskType>)> { ) -> Option<(TaskId, CachedTaskTypeArc)> {
if !self.backend.should_restore() { if !self.backend.should_restore() {
return None; return None;
} }
...@@ -1032,14 +1033,14 @@ impl<'e, B: BackingStorage> ChildExecuteContext<'e> for ChildExecuteContextImpl< ...@@ -1032,14 +1033,14 @@ impl<'e, B: BackingStorage> ChildExecuteContext<'e> for ChildExecuteContextImpl<
} }
pub enum TaskTypeRef<'l> { pub enum TaskTypeRef<'l> {
Cached(&'l Arc<CachedTaskType>), Cached(&'l CachedTaskTypeArc),
Transient(&'l Arc<TransientTask>), Transient(&'l Arc<TransientTask>),
} }
impl TaskTypeRef<'_> { impl TaskTypeRef<'_> {
pub fn to_owned(&self) -> TaskType { pub fn to_owned(&self) -> TaskType {
match self { match self {
TaskTypeRef::Cached(ty) => TaskType::Cached(Arc::clone(ty)), TaskTypeRef::Cached(ty) => TaskType::Cached((*ty).clone()),
TaskTypeRef::Transient(ty) => TaskType::Transient(Arc::clone(ty)), TaskTypeRef::Transient(ty) => TaskType::Transient(Arc::clone(ty)),
} }
} }
...@@ -1056,7 +1057,7 @@ impl Display for TaskTypeRef<'_> { ...@@ -1056,7 +1057,7 @@ impl Display for TaskTypeRef<'_> {
#[derive(Debug)] #[derive(Debug)]
pub enum TaskType { pub enum TaskType {
Cached(Arc<CachedTaskType>), Cached(CachedTaskTypeArc),
Transient(Arc<TransientTask>), Transient(Arc<TransientTask>),
} }
...@@ -1398,7 +1399,7 @@ impl TaskGuard for TaskGuardImpl<'_> { ...@@ -1398,7 +1399,7 @@ impl TaskGuard for TaskGuardImpl<'_> {
.map(|target| (target, TaskDataCategory::Meta)) .map(|target| (target, TaskDataCategory::Meta))
.chain( .chain(
self.iter_cell_dependencies() self.iter_cell_dependencies()
.map(|(target, _key)| (target.task, TaskDataCategory::All)), .map(|dep| (dep.cell_ref().task, TaskDataCategory::All)),
) )
.chain( .chain(
self.iter_collectibles_dependencies() self.iter_collectibles_dependencies()
......
...@@ -20,7 +20,7 @@ use crate::{ ...@@ -20,7 +20,7 @@ use crate::{
}, },
storage_schema::TaskStorageAccessors, storage_schema::TaskStorageAccessors,
}, },
data::CellRef, data::{CellDependency, CellRef},
}; };
#[derive(Encode, Decode, Clone, Default)] #[derive(Encode, Decode, Clone, Default)]
...@@ -137,16 +137,21 @@ impl UpdateCellOperation { ...@@ -137,16 +137,21 @@ impl UpdateCellOperation {
let mut dependent_tasks: FxIndexMap<TaskId, SmallVec<[Option<u64>; 2]>> = let mut dependent_tasks: FxIndexMap<TaskId, SmallVec<[Option<u64>; 2]>> =
FxIndexMap::default(); FxIndexMap::default();
if !skip_invalidation { if !skip_invalidation {
let tasks_with_keys = let tasks_with_keys = task.iter_cell_dependents().filter_map(|dep| {
task.iter_cell_dependents() let (
.filter_map(|(dependent_cell, key, task)| { CellRef {
task: dependent_task,
cell: dependent_cell,
},
key,
) = dep.into_parts();
(dependent_cell == cell (dependent_cell == cell
&& key.is_none_or(|key_hash| { && key.is_none_or(|key_hash| {
updated_key_hashes_set updated_key_hashes_set
.as_ref() .as_ref()
.is_none_or(|set| set.contains(&key_hash)) .is_none_or(|set| set.contains(&key_hash))
})) }))
.then_some((task, key)) .then_some((dependent_task, key))
}); });
for (task, key) in tasks_with_keys { for (task, key) in tasks_with_keys {
dependent_tasks.entry(task).or_default().push(key); dependent_tasks.entry(task).or_default().push(key);
...@@ -276,14 +281,15 @@ impl Operation for UpdateCellOperation { ...@@ -276,14 +281,15 @@ impl Operation for UpdateCellOperation {
let mut make_stale = false; let mut make_stale = false;
let dependent = ctx.task(dependent_task_id, TaskDataCategory::All); let dependent = ctx.task(dependent_task_id, TaskDataCategory::All);
for key in keys.iter().copied() { for key in keys.iter().copied() {
if dependent.outdated_cell_dependencies_contains(&(cell_ref, key)) { let dep = CellDependency::new(cell_ref, key);
if dependent.outdated_cell_dependencies_contains(&dep) {
// cell dependency is outdated, so it hasn't read the cell yet // cell dependency is outdated, so it hasn't read the cell yet
// and doesn't need to be invalidated. // and doesn't need to be invalidated.
// We do not need to make the task stale in this case. // We do not need to make the task stale in this case.
// But importantly we still need to make the task dirty as it should // But importantly we still need to make the task dirty as it should
// no longer be considered as // no longer be considered as
// "recomputation". // "recomputation".
} else if !dependent.cell_dependencies_contains(&(cell_ref, key)) { } else if !dependent.cell_dependencies_contains(&dep) {
// cell dependency has been removed, so the task doesn't depend on // cell dependency has been removed, so the task doesn't depend on
// the cell anymore and doesn't need // the cell anymore and doesn't need
// to be invalidated // to be invalidated
......
...@@ -12,7 +12,7 @@ use std::{ ...@@ -12,7 +12,7 @@ use std::{
use thread_local::ThreadLocal; use thread_local::ThreadLocal;
use tracing::span::Id; use tracing::span::Id;
use turbo_bincode::TurboBincodeBuffer; use turbo_bincode::TurboBincodeBuffer;
use turbo_tasks::{FxDashMap, TaskId, backend::CachedTaskType, event::Event, parallel}; use turbo_tasks::{FxDashMap, TaskId, backend::CachedTaskTypeArc, event::Event, parallel};
use crate::{ use crate::{
backend::storage_schema::{ backend::storage_schema::{
...@@ -183,7 +183,7 @@ pub struct Storage { ...@@ -183,7 +183,7 @@ pub struct Storage {
/// This is backed by the TaskCache table in the database. /// This is backed by the TaskCache table in the database.
/// ///
/// LockOrdering: See the comments on [map]. /// LockOrdering: See the comments on [map].
pub task_cache: FxDashMap<Arc<CachedTaskType>, TaskId>, pub task_cache: FxDashMap<CachedTaskTypeArc, TaskId>,
} }
impl Storage { impl Storage {
...@@ -253,7 +253,7 @@ impl Storage { ...@@ -253,7 +253,7 @@ impl Storage {
/// Mark a newly allocated task as restored (skip DB queries) and new (include in persistence /// Mark a newly allocated task as restored (skip DB queries) and new (include in persistence
/// snapshots). Optionally sets the `persistent_task_type` eagerly so it's available for /// snapshots). Optionally sets the `persistent_task_type` eagerly so it's available for
/// persistence snapshots without needing to propagate it through `connect_child`. /// persistence snapshots without needing to propagate it through `connect_child`.
pub fn initialize_new_task(&self, task_id: TaskId, task_type: Option<Arc<CachedTaskType>>) { pub fn initialize_new_task(&self, task_id: TaskId, task_type: Option<CachedTaskTypeArc>) {
let mut task = self.access_mut(task_id); let mut task = self.access_mut(task_id);
task.flags.set_restored(TaskDataCategory::All); task.flags.set_restored(TaskDataCategory::All);
task.flags.set_new_task(true); task.flags.set_new_task(true);
...@@ -516,7 +516,7 @@ impl Storage { ...@@ -516,7 +516,7 @@ impl Storage {
// was contended. We defer them until after the map shard lock is released to // was contended. We defer them until after the map shard lock is released to
// avoid a lock cycle with get_or_create_persistent_task, which takes task_cache // avoid a lock cycle with get_or_create_persistent_task, which takes task_cache
// before map. Allocated lazily on first conflict. // before map. Allocated lazily on first conflict.
let mut deferred_task_cache_removals: Vec<Arc<CachedTaskType>> = Vec::new(); let mut deferred_task_cache_removals: Vec<CachedTaskTypeArc> = Vec::new();
// SAFETY: We hold the write lock for the duration of iteration. // SAFETY: We hold the write lock for the duration of iteration.
for bucket in unsafe { shard.iter() } { for bucket in unsafe { shard.iter() } {
// SAFETY: The write lock guard outlives the bucket reference. // SAFETY: The write lock guard outlives the bucket reference.
......
...@@ -77,6 +77,60 @@ impl CollectiblesRef { ...@@ -77,6 +77,60 @@ impl CollectiblesRef {
} }
} }
/// An edge between a [`CellRef`] and a task, optionally narrowed by a hashed sub-key.
///
/// Used both as a forward and reverse edge:
/// - In `cell_dependencies`, the [`CellRef`] is the cell another task owns that this task depends
/// on.
/// - In `cell_dependents`, the [`CellRef`]'s `task` is the dependent task and `cell` is the cell of
/// the storing task; the `task` field is reused as the dependent's id rather than the cell's
/// owning task. The fields encode the same bits either way.
#[derive(Debug, Copy, Clone, Hash, PartialEq, Eq, Encode, Decode)]
pub enum CellDependency {
/// Depend on the cell as a whole.
All(CellRef),
/// Depend only on the sub-value identified by this hash key.
Hash(CellRef, u64),
}
impl CellDependency {
pub fn cell_ref(&self) -> CellRef {
match *self {
CellDependency::All(c) | CellDependency::Hash(c, _) => c,
}
}
pub fn key(&self) -> Option<u64> {
match *self {
CellDependency::All(_) => None,
CellDependency::Hash(_, k) => Some(k),
}
}
/// Decompose into the underlying `(CellRef, Option<u64>)` in a single match.
///
/// Prefer this over back-to-back `cell_ref()` + `key()` calls — the discriminant is
/// checked once instead of twice, which is meaningful in hot loops over
/// `iter_cell_dependents` / `iter_cell_dependencies`.
pub fn into_parts(self) -> (CellRef, Option<u64>) {
match self {
CellDependency::All(c) => (c, None),
CellDependency::Hash(c, k) => (c, Some(k)),
}
}
pub fn new(cell_ref: CellRef, key: Option<u64>) -> Self {
match key {
None => CellDependency::All(cell_ref),
Some(k) => CellDependency::Hash(cell_ref, k),
}
}
pub fn is_transient(&self) -> bool {
self.cell_ref().is_transient()
}
}
#[derive(Debug, Clone, PartialEq, Eq, Encode, Decode)] #[derive(Debug, Clone, PartialEq, Eq, Encode, Decode)]
pub enum OutputValue { pub enum OutputValue {
Cell(CellRef), Cell(CellRef),
......
...@@ -1296,12 +1296,16 @@ fn generate_typed_storage_struct(grouped_fields: &GroupedFields) -> TokenStream ...@@ -1296,12 +1296,16 @@ fn generate_typed_storage_struct(grouped_fields: &GroupedFields) -> TokenStream
quote! {} quote! {}
}; };
// Add lazy vec field if needed (pub(crate) - used by helper methods)
// Note: Serialization is handled manually via encode_data/encode_meta methods
let lazy_field = if has_lazy { let lazy_field = if has_lazy {
let max_lazy: u8 = grouped_fields
.all_lazy()
.count()
.try_into()
.expect("cannot have more than 255 lazy fields");
quote! { quote! {
#[doc = "Lazily-allocated fields stored in a single Vec for memory efficiency"] #[doc = "Lazily-allocated fields stored in a compact TinyVec for memory efficiency"]
lazy: Vec<LazyField>, lazy: TinyVec<LazyField, #max_lazy>,
} }
} else { } else {
quote! {} quote! {}
...@@ -3686,10 +3690,10 @@ fn generate_snapshot_restore_methods(grouped_fields: &GroupedFields) -> TokenStr ...@@ -3686,10 +3690,10 @@ fn generate_snapshot_restore_methods(grouped_fields: &GroupedFields) -> TokenStr
#clone_all_flags #clone_all_flags
// Pre-allocate lazy vec (upper bound - some may be transient and skipped) // Clone all persistent lazy fields (both meta and data).
snapshot.lazy.reserve(self.lazy.len()); // (No pre-`reserve`: the schema has ≤24 lazy fields, so at most 3 grows
// (0→4→8→16→24) total — cheaper than complicating the public API surface
// Clone all persistent lazy fields (both meta and data) // of `TinyVec`.)
for field in &self.lazy { for field in &self.lazy {
match field { match field {
#(#clone_data_lazy_arms)* #(#clone_data_lazy_arms)*
...@@ -3746,7 +3750,7 @@ fn generate_snapshot_restore_methods(grouped_fields: &GroupedFields) -> TokenStr ...@@ -3746,7 +3750,7 @@ fn generate_snapshot_restore_methods(grouped_fields: &GroupedFields) -> TokenStr
// and merge each source variant in O(1). // and merge each source variant in O(1).
let (any_meta, _any_data, index) = Self::build_lazy_index(&self.lazy); let (any_meta, _any_data, index) = Self::build_lazy_index(&self.lazy);
if !any_meta { if !any_meta {
self.lazy.extend(source.lazy); self.lazy.extend_exact(source.lazy);
} else { } else {
for field in source.lazy { for field in source.lazy {
debug_assert!(field.is_persistent() && field.is_meta()); debug_assert!(field.is_persistent() && field.is_meta());
...@@ -3770,7 +3774,7 @@ fn generate_snapshot_restore_methods(grouped_fields: &GroupedFields) -> TokenStr ...@@ -3770,7 +3774,7 @@ fn generate_snapshot_restore_methods(grouped_fields: &GroupedFields) -> TokenStr
// in `self.lazy` is never a collision risk. // in `self.lazy` is never a collision risk.
let (_any_meta, any_data, index) = Self::build_lazy_index(&self.lazy); let (_any_meta, any_data, index) = Self::build_lazy_index(&self.lazy);
if !any_data { if !any_data {
self.lazy.extend(source.lazy); self.lazy.extend_exact(source.lazy);
} else { } else {
for field in source.lazy { for field in source.lazy {
debug_assert!(field.is_persistent() && field.is_data()); debug_assert!(field.is_persistent() && field.is_data());
......
...@@ -65,3 +65,7 @@ criterion = { workspace = true, features = ["async_tokio"] } ...@@ -65,3 +65,7 @@ criterion = { workspace = true, features = ["async_tokio"] }
[[bench]] [[bench]]
name = "mod" name = "mod"
harness = false harness = false
[[bench]]
name = "tiny_vec"
harness = false
//! Direct comparison between `TinyVec<T>` and the standard `Vec<T>` on the
//! operations that `TaskStorage::lazy` actually exercises:
//!
//! * `push` — appending one element at a time, growing through capacity boundaries.
//! * `iter` — linear scan (this is how `find_lazy(id)` works under the hood).
//!
//! These are micro-benchmarks: the values pushed are small `(u8, u64)` pairs to
//! mimic `LazyField`'s ~48 B size without dragging in the entire schema. The
//! goal is to validate that switching `lazy` from `Vec` to `TinyVec` doesn't
//! cost throughput at the API level, since `Vec::push` is heavily optimized
//! and our hand-rolled `TinyVec::push` is not.
use std::hint::black_box;
use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
use turbo_tasks::TinyVec;
#[global_allocator]
static ALLOC: turbo_tasks_malloc::TurboMalloc = turbo_tasks_malloc::TurboMalloc;
/// A stand-in for `LazyField`: 16 B payload so the per-element work mirrors the
/// real storage layout (without needing the whole schema).
#[derive(Clone, Copy)]
#[allow(dead_code)]
struct Item {
tag: u64,
payload: u64,
}
fn make_item(i: u64) -> Item {
Item {
tag: i,
payload: i.wrapping_mul(0x9E37_79B9_7F4A_7C15),
}
}
/// Push N items into a fresh `Vec`. Returns the populated container so the
/// allocator drop cost is included in the measurement.
fn push_vec(n: usize) -> Vec<Item> {
let mut v: Vec<Item> = Vec::new();
for i in 0..n {
v.push(make_item(i as u64));
}
v
}
fn push_tinyvec(n: usize) -> TinyVec<Item> {
let mut v: TinyVec<Item> = TinyVec::default();
for i in 0..n {
v.push(make_item(i as u64));
}
v
}
/// Sum all items via iter — the linear scan pattern.
#[allow(clippy::ptr_arg)] // for clarity
fn iter_vec(v: &Vec<Item>) -> u64 {
let mut acc: u64 = 0;
for it in v.iter() {
acc = acc.wrapping_add(it.tag).wrapping_add(it.payload);
}
acc
}
fn iter_tinyvec(v: &TinyVec<Item>) -> u64 {
let mut acc: u64 = 0;
for it in v.iter() {
acc = acc.wrapping_add(it.tag).wrapping_add(it.payload);
}
acc
}
pub fn bench(c: &mut Criterion) {
// Sizes chosen to cover the realistic `TaskStorage::lazy` range:
// 0 — empty (steady state for many tasks)
// 1 — single field set (very common)
// 4 — Vec's first grow boundary (1 -> 2 -> 4 -> 8 ...)
// 8 — past the first few grows, full cache line worth of items
// 16 — fits in our `u8` cap with headroom
// 24 — close to the realistic max (~25 lazy fields in the schema)
let sizes = [0usize, 1, 4, 8, 16, 24];
// --- push -----------------------------------------------------------------
let mut group = c.benchmark_group("tiny_vec/push");
group.sample_size(200);
for &n in &sizes {
group.bench_with_input(BenchmarkId::new("Vec", n), &n, |b, &n| {
b.iter(|| {
let v = push_vec(black_box(n));
black_box(v);
});
});
group.bench_with_input(BenchmarkId::new("TinyVec", n), &n, |b, &n| {
b.iter(|| {
let v = push_tinyvec(black_box(n));
black_box(v);
});
});
}
group.finish();
// --- iter -----------------------------------------------------------------
let mut group = c.benchmark_group("tiny_vec/iter");
group.sample_size(200);
for &n in &sizes {
// Pre-fill once outside the timed region.
let v: Vec<Item> = push_vec(n);
let tv: TinyVec<Item> = push_tinyvec(n);
group.bench_with_input(BenchmarkId::new("Vec", n), &n, |b, _| {
b.iter(|| black_box(iter_vec(black_box(&v))));
});
group.bench_with_input(BenchmarkId::new("TinyVec", n), &n, |b, _| {
b.iter(|| black_box(iter_tinyvec(black_box(&tv))));
});
}
group.finish();
}
criterion_group!(tiny_vec_benches, bench);
criterion_main!(tiny_vec_benches);
use std::{ use std::{
borrow::Cow, borrow::{Borrow, Cow},
error::Error, error::Error,
fmt::{self, Debug, Display}, fmt::{self, Debug, Display},
future::Future, future::Future,
hash::{BuildHasher, BuildHasherDefault, Hash}, hash::{BuildHasher, BuildHasherDefault, Hash},
ops::Deref,
pin::Pin, pin::Pin,
sync::Arc, sync::Arc,
}; };
...@@ -131,6 +132,76 @@ impl_encode_for_turbo_bincode_encode!(CachedTaskType); ...@@ -131,6 +132,76 @@ impl_encode_for_turbo_bincode_encode!(CachedTaskType);
impl_decode_for_turbo_bincode_decode!(CachedTaskType); impl_decode_for_turbo_bincode_decode!(CachedTaskType);
impl_borrow_decode!(CachedTaskType); impl_borrow_decode!(CachedTaskType);
/// A reference-counted pointer to a [`CachedTaskType`] using `triomphe::Arc`.
///
/// `triomphe::Arc` saves one `usize` per allocation (no weak count) and avoids the weak-count
/// CAS in `drop_slow` compared to `std::sync::Arc`. We never need `Weak<CachedTaskType>`, so
/// the trade-off is favorable.
#[derive(Clone, Debug, Hash, PartialEq, Eq)]
pub struct CachedTaskTypeArc(pub triomphe::Arc<CachedTaskType>);
impl CachedTaskTypeArc {
pub fn new(value: CachedTaskType) -> Self {
Self(triomphe::Arc::new(value))
}
pub fn count(&self) -> usize {
triomphe::Arc::count(&self.0)
}
}
impl AsRef<CachedTaskType> for CachedTaskTypeArc {
fn as_ref(&self) -> &CachedTaskType {
&self.0
}
}
impl Deref for CachedTaskTypeArc {
type Target = CachedTaskType;
#[inline]
fn deref(&self) -> &CachedTaskType {
&self.0
}
}
impl Borrow<CachedTaskType> for CachedTaskTypeArc {
#[inline]
fn borrow(&self) -> &CachedTaskType {
&self.0
}
}
impl Display for CachedTaskTypeArc {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
Display::fmt(&**self, f)
}
}
impl Encode for CachedTaskTypeArc {
fn encode<E: Encoder>(&self, encoder: &mut E) -> Result<(), EncodeError> {
<CachedTaskType as Encode>::encode(self, encoder)
}
}
impl<Context> Decode<Context> for CachedTaskTypeArc {
fn decode<D: Decoder<Context = Context>>(decoder: &mut D) -> Result<Self, DecodeError> {
Ok(Self::new(<CachedTaskType as Decode<Context>>::decode(
decoder,
)?))
}
}
impl<'de, Context> bincode::BorrowDecode<'de, Context> for CachedTaskTypeArc {
fn borrow_decode<D: bincode::de::BorrowDecoder<'de, Context = Context>>(
decoder: &mut D,
) -> Result<Self, DecodeError> {
Ok(Self::new(<CachedTaskType as bincode::BorrowDecode<
'de,
Context,
>>::borrow_decode(decoder)?))
}
}
// Manual implementation is needed because of a borrow issue with `Box<dyn Trait>`: // Manual implementation is needed because of a borrow issue with `Box<dyn Trait>`:
// https://github.com/rust-lang/rust/issues/31740 // https://github.com/rust-lang/rust/issues/31740
impl PartialEq for CachedTaskType { impl PartialEq for CachedTaskType {
......
...@@ -55,6 +55,7 @@ mod state; ...@@ -55,6 +55,7 @@ mod state;
pub mod task; pub mod task;
mod task_execution_reason; mod task_execution_reason;
pub mod task_statistics; pub mod task_statistics;
mod tiny_vec;
pub mod trace; pub mod trace;
mod trait_ref; mod trait_ref;
mod triomphe_utils; mod triomphe_utils;
...@@ -108,6 +109,7 @@ pub use crate::{ ...@@ -108,6 +109,7 @@ pub use crate::{
task_input::{EitherTaskInput, TaskInput}, task_input::{EitherTaskInput, TaskInput},
}, },
task_execution_reason::TaskExecutionReason, task_execution_reason::TaskExecutionReason,
tiny_vec::TinyVec,
trait_ref::TraitRef, trait_ref::TraitRef,
value::{TransientInstance, TransientValue}, value::{TransientInstance, TransientValue},
value_type::{Evictability, TraitMethod, TraitType, ValueType, ValueTypePersistence}, value_type::{Evictability, TraitMethod, TraitType, ValueType, ValueTypePersistence},
......
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment