Unverified Commit 2e133c98 authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

turbo-tasks: unemit taken collectibles in a single backend operation (#100437)

### What?

`take_collectibles` / `drop_collectibles` now unemit all taken
collectibles in a single backend operation instead of one operation per
collectible.

### Why?

`TurboTasks::unemit_collectibles` looped over every collectible and
called `Backend::unemit_collectible` for each. In `TurboTasksBackend`
every one of those calls locked the task, built its own aggregation
update queue and propagated a single-entry collectibles update through
the aggregation graph. With many collectibles that means one task lock
and one aggregation graph update per collectible.

### How?

- New required `Backend::unemit_collectibles` trait method.
`TurboTasks::unemit_collectibles` calls it once, and does nothing when
no count is positive.
- `TurboTasksBackend` implements it with a batched
`UpdateCollectibleOperation::run_batch`:
  - locks the task once and checks the root-task restriction up front;
- reconciles each collectible against its outdated count, with the same
per-collectible semantics as before;
- invalidates collectible dependents at most once per collectible type;
- propagates all remaining changes as one aggregated data update, so the
aggregation queue runs once.
- `AggregatedDataUpdate`'s collectibles update is now a `SmallVec<[_;
1]>`, so single-collectible updates don't heap-allocate and the batch is
handed over without conversion.
- The single-collectible `UpdateCollectibleOperation::run` now goes
through the same code path.

New regression test `tests/collectibles_take_batch.rs` covers:
- take and drop;
- the same collectible emitted several times, from one task and from two
tasks;
- several uppers;
- the set of collectibles changing between re-executions;
- a nested take;
- taking from a source with no collectibles.

It passes both before and after this change.

### Benchmark

New criterion benchmark `benches/collectibles.rs`. A root task takes (or
peeks) N collectibles of an operation, below a chain of 8 tasks.
Emission happens outside the timed region.

```
cargo bench -p turbo-tasks-backend --bench mod -- collectibles
```

Medians from two interleaved base/new rounds on one 8-core VM:

| N | base | new | change |
|---|---|---|---|
| 1,000 | 655 µs / 575 µs | 438 µs / 384 µs | −33% / −33% |
| 10,000 | 4.8 ms / 4.9 ms | 3.2 ms / 2.9 ms | −34% / −41% |

For N = 10 and 100 there was no consistent difference. The VM's
run-to-run noise there (±15–25%, also for the unchanged `peek` variant)
is larger than any effect.

### Verification

- `cargo test -p turbo-tasks -p turbo-tasks-backend`
- `cargo clippy -p turbo-tasks -p turbo-tasks-backend --all-targets --
-D warnings`

<!-- NEXT_JS_LLM -->


<!-- fleet 53aa7530-1b3f-423b-8eb2-29bf4b657aba -->

---------
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent fa1eab66
//! Benchmarks for taking collectibles (`OperationVc::take_collectibles`), which unemits all taken
//! collectibles from the taking task and propagates that through the aggregation graph.
//!
//! The `peek` variant runs the same task graph without unemitting. It is a comparison workload,
//! not a subtractive control: taking also stops the collectibles from propagating to the uppers.
//! Run with:
//!
//! ```bash
//! cargo bench -p turbo-tasks-backend --bench mod -- collectibles
//! ```
use std::time::{Duration, Instant};
use anyhow::Result;
use criterion::{BenchmarkId, Criterion, Throughput};
use turbo_rcstr::RcStr;
use turbo_tasks::{CollectiblesSource, ResolvedVc, TurboTasks, ValueToString, Vc, emit};
use turbo_tasks_backend::{BackendOptions, TurboTasksBackend, noop_backing_storage};
/// Number of tasks between the root task and the task taking the collectibles, so that the unemit
/// has to propagate through a few levels of the aggregation graph.
const DEPTH: u32 = 8;
pub fn collectibles(c: &mut Criterion) {
let mut group = c.benchmark_group("turbo_tasks_backend_collectibles");
group.sample_size(20);
group.measurement_time(Duration::from_secs(5));
let rt = tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.unwrap();
for count in [10, 100, 1_000, 10_000] {
group.throughput(Throughput::Elements(count as u64));
for take in [false, true] {
let name = if take { "take" } else { "peek" };
group.bench_with_input(BenchmarkId::new(name, count), &count, |b, &count| {
b.to_async(&rt).iter_custom(|iters| async move {
let tt = TurboTasks::new(TurboTasksBackend::new(
BackendOptions {
storage_mode: None,
..Default::default()
},
noop_backing_storage(),
));
let duration = tt
.run(async move {
// Emit the collectibles once, outside of the measurement.
emitting_operation(count).read_strongly_consistent().await?;
let start = Instant::now();
for iteration in 0..iters as u32 {
let taken = chain_operation(count, take, iteration, DEPTH)
.read_strongly_consistent()
.await?;
assert_eq!(*taken, count);
}
Ok(start.elapsed())
})
.await
.unwrap();
// Stopping and dropping the instance happens outside of the measurement.
tt.stop_and_wait().await;
duration
})
});
}
}
}
#[turbo_tasks::value]
struct Collectible(u32);
#[turbo_tasks::value_impl]
impl ValueToString for Collectible {
#[turbo_tasks::function]
fn to_string(&self) -> Vc<RcStr> {
Vc::cell(self.0.to_string().into())
}
}
#[turbo_tasks::function]
fn emit_all(count: u32) {
for i in 0..count {
emit(ResolvedVc::upcast::<Box<dyn ValueToString>>(
Collectible(i).resolved_cell(),
));
}
}
#[turbo_tasks::function(operation, root)]
async fn emitting_operation(count: u32) -> Result<Vc<()>> {
emit_all(count).as_side_effect().await?;
Ok(Vc::cell(()))
}
/// Takes (or peeks) the collectibles of [`emitting_operation`]. `iteration` makes every call a new
/// task.
#[turbo_tasks::function(operation, root)]
async fn collecting_operation(count: u32, take: bool, iteration: u32) -> Result<Vc<u32>> {
let _ = iteration;
let source = emitting_operation(count);
source.connect().await?;
let collectibles = if take {
source.take_collectibles::<Box<dyn ValueToString>>()
} else {
source.peek_collectibles::<Box<dyn ValueToString>>()
};
Ok(Vc::cell(collectibles.len() as u32))
}
/// A chain of `depth` tasks above [`collecting_operation`].
#[turbo_tasks::function(operation, root)]
async fn chain_operation(count: u32, take: bool, iteration: u32, depth: u32) -> Result<Vc<u32>> {
if depth == 0 {
Ok(collecting_operation(count, take, iteration).connect())
} else {
Ok(chain_operation(count, take, iteration, depth - 1).connect())
}
}
...@@ -4,6 +4,7 @@ ...@@ -4,6 +4,7 @@
use criterion::{Criterion, criterion_group, criterion_main}; use criterion::{Criterion, criterion_group, criterion_main};
pub(crate) mod cell_fan_out; pub(crate) mod cell_fan_out;
pub(crate) mod collectibles;
pub(crate) mod gc; pub(crate) mod gc;
pub(crate) mod overhead; pub(crate) mod overhead;
pub(crate) mod scope_stress; pub(crate) mod scope_stress;
...@@ -14,6 +15,6 @@ criterion_group!( ...@@ -14,6 +15,6 @@ criterion_group!(
name = turbo_tasks_backend_stress; name = turbo_tasks_backend_stress;
config = Criterion::default(); config = Criterion::default();
targets = stress::fibonacci, scope_stress::scope_stress, overhead::overhead, gc::gc, targets = stress::fibonacci, scope_stress::scope_stress, overhead::overhead, gc::gc,
cell_fan_out::cell_fan_out, state_fan_out::state_fan_out cell_fan_out::cell_fan_out, state_fan_out::state_fan_out, collectibles::collectibles
); );
criterion_main!(turbo_tasks_backend_stress); criterion_main!(turbo_tasks_backend_stress);
...@@ -39,10 +39,10 @@ use turbo_tasks::{ ...@@ -39,10 +39,10 @@ use turbo_tasks::{
TRANSIENT_TASK_BIT, TaskExecutionReason, TaskId, TaskPersistence, TaskPriority, TraitTypeId, TRANSIENT_TASK_BIT, TaskExecutionReason, TaskId, TaskPersistence, TaskPriority, TraitTypeId,
TurboTasks, TurboTasksCallApi, TurboTasksPanic, ValueTypeId, TurboTasks, TurboTasksCallApi, TurboTasksPanic, ValueTypeId,
backend::{ backend::{
Backend, CachedTaskType, CachedTaskTypeArc, CellContent, CellHash, TaskExecutionSpec, Backend, CachedTaskType, CachedTaskTypeArc, CellContent, CellHash, TaskCollectiblesMap,
TransientTaskType, TurboTaskContextError, TurboTaskLocalContextError, TurboTasksError, TaskExecutionSpec, TransientTaskType, TurboTaskContextError, TurboTaskLocalContextError,
TurboTasksExecutionError, TurboTasksExecutionErrorMessage, TypedCellContent, TurboTasksError, TurboTasksExecutionError, TurboTasksExecutionErrorMessage,
VerificationMode, TypedCellContent, VerificationMode,
}, },
event::{Event, EventDescription, EventListener}, event::{Event, EventDescription, EventListener},
macro_helpers::NativeFunction, macro_helpers::NativeFunction,
...@@ -3425,6 +3425,40 @@ impl TurboTasksBackend { ...@@ -3425,6 +3425,40 @@ impl TurboTasksBackend {
); );
} }
fn unemit_collectibles(
&self,
collectible_type: TraitTypeId,
collectibles: &TaskCollectiblesMap,
task_id: TaskId,
turbo_tasks: &TurboTasks<TurboTasksBackend>,
) {
let updates: SmallVec<[_; 1]> = collectibles
.iter()
.filter(|&(_, &count)| count > 0)
.map(|(&collectible, &count)| {
self.assert_valid_collectible(task_id, collectible);
let Some((collectible_task, cell)) = collectible.as_task_cell() else {
panic!("Collectibles need to be resolved");
};
(
CollectibleRef {
collectible_type,
cell: CellRef {
task: collectible_task,
cell,
},
},
-count,
)
})
.collect();
operation::UpdateCollectibleOperation::run_batch(
task_id,
updates,
self.execute_context(turbo_tasks),
);
}
fn update_task_cell( fn update_task_cell(
&self, &self,
task_id: TaskId, task_id: TaskId,
...@@ -3928,6 +3962,16 @@ impl Backend for TurboTasksBackend { ...@@ -3928,6 +3962,16 @@ impl Backend for TurboTasksBackend {
self.unemit_collectible(collectible_type, collectible, count, task_id, turbo_tasks) self.unemit_collectible(collectible_type, collectible, count, task_id, turbo_tasks)
} }
fn unemit_collectibles(
&self,
collectible_type: TraitTypeId,
collectibles: &TaskCollectiblesMap,
task_id: TaskId,
turbo_tasks: &TurboTasks<Self>,
) {
self.unemit_collectibles(collectible_type, collectibles, task_id, turbo_tasks)
}
fn update_task_cell( fn update_task_cell(
&self, &self,
task_id: TaskId, task_id: TaskId,
......
...@@ -75,6 +75,9 @@ const AGGREGATION_UPDATE_CATEGORY: TaskDataCategory = TaskDataCategory::All; ...@@ -75,6 +75,9 @@ const AGGREGATION_UPDATE_CATEGORY: TaskDataCategory = TaskDataCategory::All;
type TaskIdVec = SmallVec<[TaskId; 4]>; type TaskIdVec = SmallVec<[TaskId; 4]>;
type TaskIdWithCountVec = SmallVec<[(TaskId, u32); 2]>; type TaskIdWithCountVec = SmallVec<[(TaskId, u32); 2]>;
/// Collectible count changes of an [`AggregatedDataUpdate`]. Most updates change a single
/// collectible, so one entry is stored inline.
pub type CollectiblesUpdateVec = SmallVec<[(CollectibleRef, i32); 1]>;
/// Returns true, when a node is aggregating its children and a partial subgraph. /// Returns true, when a node is aggregating its children and a partial subgraph.
pub fn is_aggregating_node(aggregation_number: u32) -> bool { pub fn is_aggregating_node(aggregation_number: u32) -> bool {
...@@ -348,9 +351,11 @@ struct DirtyContainerUpdate { ...@@ -348,9 +351,11 @@ struct DirtyContainerUpdate {
/// ///
/// The order of the result is unspecified. Counts are added without overflow checks; their sum is /// The order of the result is unspecified. Counts are added without overflow checks; their sum is
/// not expected to overflow. /// not expected to overflow.
fn sum_collectibles_update<K: Hash + Eq + Copy>(mut update: Vec<(K, i32)>) -> Vec<(K, i32)> { fn sum_collectibles_update<K: Hash + Eq + Copy>(
mut update: SmallVec<[(K, i32); 1]>,
) -> SmallVec<[(K, i32); 1]> {
if update.len() < 2 { if update.len() < 2 {
update.retain(|&(_, count)| count != 0); update.retain(|&mut (_, count)| count != 0);
return update; return update;
} }
let mut sums: FxHashMap<K, i32> = let mut sums: FxHashMap<K, i32> =
...@@ -369,7 +374,7 @@ pub struct AggregatedDataUpdate { ...@@ -369,7 +374,7 @@ pub struct AggregatedDataUpdate {
/// One of the inner tasks has changed its dirty state or aggregated dirty state. /// One of the inner tasks has changed its dirty state or aggregated dirty state.
dirty_container_update: Option<DirtyContainerUpdate>, dirty_container_update: Option<DirtyContainerUpdate>,
/// One of the inner tasks has changed its collectibles count or aggregated collectibles count. /// One of the inner tasks has changed its collectibles count or aggregated collectibles count.
collectibles_update: Vec<(CollectibleRef, i32)>, collectibles_update: CollectiblesUpdateVec,
} }
impl AggregatedDataUpdate { impl AggregatedDataUpdate {
...@@ -390,8 +395,9 @@ impl AggregatedDataUpdate { ...@@ -390,8 +395,9 @@ impl AggregatedDataUpdate {
.unwrap_or_default(); .unwrap_or_default();
// Allocate an upper bound of the number of entries up front, so the update isn't // Allocate an upper bound of the number of entries up front, so the update isn't
// reallocated while the task is locked. // reallocated while the task is locked.
let mut collectibles_update = let mut collectibles_update = CollectiblesUpdateVec::with_capacity(
Vec::with_capacity(task.collectibles_len() + task.aggregated_collectibles_len()); task.collectibles_len() + task.aggregated_collectibles_len(),
);
collectibles_update.extend(task.iter_collectibles().map(|(&k, &v)| (k, v))); collectibles_update.extend(task.iter_collectibles().map(|(&k, &v)| (k, v)));
for (collectible, count) in task.iter_aggregated_collectibles() { for (collectible, count) in task.iter_aggregated_collectibles() {
if *count > 0 { if *count > 0 {
...@@ -670,7 +676,7 @@ impl AggregatedDataUpdate { ...@@ -670,7 +676,7 @@ impl AggregatedDataUpdate {
pub fn new() -> Self { pub fn new() -> Self {
Self { Self {
dirty_container_update: None, dirty_container_update: None,
collectibles_update: Vec::new(), collectibles_update: SmallVec::new(),
} }
} }
...@@ -690,7 +696,7 @@ impl AggregatedDataUpdate { ...@@ -690,7 +696,7 @@ impl AggregatedDataUpdate {
} }
/// Adds a collectibles update to the update. /// Adds a collectibles update to the update.
pub fn collectibles_update(mut self, collectibles_update: Vec<(CollectibleRef, i32)>) -> Self { pub fn collectibles_update(mut self, collectibles_update: CollectiblesUpdateVec) -> Self {
self.collectibles_update = collectibles_update; self.collectibles_update = collectibles_update;
self self
} }
...@@ -3588,6 +3594,7 @@ fn retry_loop(mut retry: u16, mut f: impl FnMut() -> ControlFlow<()>) -> Result< ...@@ -3588,6 +3594,7 @@ fn retry_loop(mut retry: u16, mut f: impl FnMut() -> ControlFlow<()>) -> Result<
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use rustc_hash::{FxHashMap, FxHashSet}; use rustc_hash::{FxHashMap, FxHashSet};
use smallvec::{SmallVec, smallvec};
use turbo_tasks::{CellId, TaskId, TraitTypeId, ValueTypeId}; use turbo_tasks::{CellId, TaskId, TraitTypeId, ValueTypeId};
use super::{AggregatedDataUpdate, sum_collectibles_update}; use super::{AggregatedDataUpdate, sum_collectibles_update};
...@@ -3657,7 +3664,7 @@ mod tests { ...@@ -3657,7 +3664,7 @@ mod tests {
/// the unsummed ones: equal final counts, equal sets of keys whose presence changed, and every /// the unsummed ones: equal final counts, equal sets of keys whose presence changed, and every
/// such key still flips. /// such key still flips.
fn assert_equivalent(start: &FxHashMap<u32, i32>, update: &[(u32, i32)]) { fn assert_equivalent(start: &FxHashMap<u32, i32>, update: &[(u32, i32)]) {
let summed = sum_collectibles_update(update.to_vec()); let summed = sum_collectibles_update(SmallVec::from_slice(update));
let mut expected = FxHashMap::default(); let mut expected = FxHashMap::default();
for &(key, count) in update { for &(key, count) in update {
*expected.entry(key).or_insert(0) += count; *expected.entry(key).or_insert(0) += count;
...@@ -3688,28 +3695,45 @@ mod tests { ...@@ -3688,28 +3695,45 @@ mod tests {
#[test] #[test]
fn optimize_sums_per_key_and_drops_zeros() { fn optimize_sums_per_key_and_drops_zeros() {
let update = [(1, -1), (2, 1), (1, 1), (3, 2), (2, -1), (3, 1)]; let update = [(1, -1), (2, 1), (1, 1), (3, 2), (2, -1), (3, 1)];
let summed = sum_collectibles_update(update.to_vec()); let summed = sum_collectibles_update(SmallVec::from_slice(&update));
assert_eq!(summed, vec![(3, 3)]); assert_eq!(summed.as_slice(), [(3, 3)]);
} }
#[test] #[test]
fn optimize_mixed_signs_and_counts() { fn optimize_mixed_signs_and_counts() {
let update = [(1, -2), (1, 1), (1, 1), (2, 5), (2, -3), (4, -1)]; let update = [(1, -2), (1, 1), (1, 1), (2, 5), (2, -3), (4, -1)];
let summed = sum_collectibles_update(update.to_vec()); let summed = sum_collectibles_update(SmallVec::from_slice(&update));
assert_eq!(as_map(&summed), [(2, 2), (4, -1)].into_iter().collect()); assert_eq!(as_map(&summed), [(2, 2), (4, -1)].into_iter().collect());
} }
#[test] #[test]
fn optimize_empty_and_single_entry() { fn optimize_empty_and_single_entry() {
assert_eq!(sum_collectibles_update(Vec::<(u32, i32)>::new()), vec![]); let empty = sum_collectibles_update(SmallVec::<[(u32, i32); 1]>::new());
assert_eq!(sum_collectibles_update(vec![(7, 3)]), vec![(7, 3)]); assert!(empty.is_empty());
assert_eq!(sum_collectibles_update(vec![(7, 0)]), vec![]); assert!(!empty.spilled());
let single = sum_collectibles_update(smallvec![(7u32, 3)]);
assert_eq!(single.as_slice(), [(7, 3)]);
assert!(!single.spilled());
assert!(sum_collectibles_update(smallvec![(7u32, 0)]).is_empty());
}
#[test]
fn optimize_reuses_the_allocation() {
let update: SmallVec<[(u32, i32); 1]> = smallvec![(1, 1), (2, 1), (1, 1), (3, -1)];
assert!(update.spilled());
let ptr = update.as_ptr();
let summed = sum_collectibles_update(update);
assert_eq!(summed.as_ptr(), ptr);
assert_eq!(
as_map(&summed),
[(1, 2), (2, 1), (3, -1)].into_iter().collect()
);
} }
#[test] #[test]
fn optimize_keeps_disjoint_keys() { fn optimize_keeps_disjoint_keys() {
let update: Vec<(u32, i32)> = (0..200).map(|k| (k, (k as i32 % 3) + 1)).collect(); let update: Vec<(u32, i32)> = (0..200).map(|k| (k, (k as i32 % 3) + 1)).collect();
let summed = sum_collectibles_update(update.clone()); let summed = sum_collectibles_update(SmallVec::from_slice(&update));
assert_eq!(as_map(&summed), as_map(&update)); assert_eq!(as_map(&summed), as_map(&update));
} }
...@@ -3722,7 +3746,7 @@ mod tests { ...@@ -3722,7 +3746,7 @@ mod tests {
let update = from_task_update(&own, &aggregated); let update = from_task_update(&own, &aggregated);
assert_eq!(update.len(), 2004); assert_eq!(update.len(), 2004);
let summed = sum_collectibles_update(update.clone()); let summed = sum_collectibles_update(SmallVec::from_slice(&update));
assert_eq!( assert_eq!(
as_map(&summed), as_map(&summed),
(1000..1004).map(|k| (k, 1)).collect::<FxHashMap<_, _>>() (1000..1004).map(|k| (k, 1)).collect::<FxHashMap<_, _>>()
...@@ -3772,7 +3796,7 @@ mod tests { ...@@ -3772,7 +3796,7 @@ mod tests {
let b = collectible(2); let b = collectible(2);
let c = collectible(3); let c = collectible(3);
let update = AggregatedDataUpdate::new() let update = AggregatedDataUpdate::new()
.collectibles_update(vec![(a, -1), (b, 2), (a, 1), (c, -1), (b, -1)]) .collectibles_update(smallvec![(a, -1), (b, 2), (a, 1), (c, -1), (b, -1)])
.dirty_container_update(task_id, 1, 0) .dirty_container_update(task_id, 1, 0)
.optimize(); .optimize();
assert_eq!( assert_eq!(
...@@ -3796,7 +3820,7 @@ mod tests { ...@@ -3796,7 +3820,7 @@ mod tests {
assert!( assert!(
AggregatedDataUpdate::new() AggregatedDataUpdate::new()
.collectibles_update(vec![(a, -1), (a, 1)]) .collectibles_update(smallvec![(a, -1), (a, 1)])
.optimize() .optimize()
.is_empty() .is_empty()
); );
......
...@@ -9,7 +9,7 @@ use crate::{ ...@@ -9,7 +9,7 @@ use crate::{
backend::{ backend::{
TaskDataCategory, TaskDataCategory,
operation::{ operation::{
AggregatedDataUpdate, ExecuteContext, TaskGuard, AggregatedDataUpdate, CollectiblesUpdateVec, ExecuteContext, TaskGuard,
aggregation_update::{ aggregation_update::{
AggregationUpdateJob, AggregationUpdateQueue, InnerOfUppersLostFollowersJob, AggregationUpdateJob, AggregationUpdateQueue, InnerOfUppersLostFollowersJob,
get_aggregation_number, get_uppers, is_aggregating_node, get_aggregation_number, get_uppers, is_aggregating_node,
...@@ -225,7 +225,7 @@ fn cleanup_old_edges_inner( ...@@ -225,7 +225,7 @@ fn cleanup_old_edges_inner(
} }
} }
OutdatedEdge::Collectible(collectible, count) => { OutdatedEdge::Collectible(collectible, count) => {
let mut collectibles = Vec::new(); let mut collectibles = CollectiblesUpdateVec::new();
collectibles.push((collectible, -count)); collectibles.push((collectible, -count));
outdated.retain(|e| match e { outdated.retain(|e| match e {
OutdatedEdge::Collectible(collectible, count) => { OutdatedEdge::Collectible(collectible, count) => {
......
...@@ -1835,8 +1835,8 @@ impl TaskStorageAccessors for TaskGuard<'_> { ...@@ -1835,8 +1835,8 @@ impl TaskStorageAccessors for TaskGuard<'_> {
pub use self::{ pub use self::{
aggregation_update::{ aggregation_update::{
AggregatedDataUpdate, AggregationUpdateJob, get_aggregation_number, get_uppers, AggregatedDataUpdate, AggregationUpdateJob, CollectiblesUpdateVec, get_aggregation_number,
is_aggregating_node, is_root_node, get_uppers, is_aggregating_node, is_root_node,
}, },
cleanup_old_edges::{ cleanup_old_edges::{
OutdatedEdge, capture_all_edges, cleanup_old_edges, cleanup_old_edges_deletions_only, OutdatedEdge, capture_all_edges, cleanup_old_edges, cleanup_old_edges_deletions_only,
......
use std::cmp::min; use std::cmp::min;
use smallvec::SmallVec; use smallvec::{SmallVec, smallvec};
use turbo_tasks::TaskId; use turbo_tasks::{TaskId, TraitTypeId};
use crate::{ use crate::{
backend::{ backend::{
TaskDataCategory, TaskDataCategory,
operation::{ operation::{
AggregatedDataUpdate, AggregationUpdateJob, AggregationUpdateQueue, ExecuteContext, AggregatedDataUpdate, AggregationUpdateJob, AggregationUpdateQueue,
CollectiblesUpdateVec, ExecuteContext,
}, },
storage_schema::TaskStorageAccessors, storage_schema::TaskStorageAccessors,
}, },
data::CollectibleRef, data::CollectibleRef,
utils::swap_retain,
}; };
pub struct UpdateCollectibleOperation; pub struct UpdateCollectibleOperation;
impl UpdateCollectibleOperation { impl UpdateCollectibleOperation {
pub fn run( pub fn run(task_id: TaskId, collectible: CollectibleRef, count: i32, ctx: ExecuteContext<'_>) {
Self::run_batch(task_id, smallvec![(collectible, count)], ctx);
}
/// Applies all count changes of `collectibles` to the task in a single operation: the task is
/// locked once, and a single aggregated data update is propagated through the aggregation
/// graph.
pub fn run_batch(
task_id: TaskId, task_id: TaskId,
collectible: CollectibleRef, mut collectibles: CollectiblesUpdateVec,
mut count: i32,
mut ctx: ExecuteContext<'_>, mut ctx: ExecuteContext<'_>,
) { ) {
if collectibles.is_empty() {
return;
}
let mut task = ctx.task(task_id, TaskDataCategory::All); let mut task = ctx.task(task_id, TaskDataCategory::All);
if count < 0 if collectibles.iter().any(|&(_, count)| count < 0)
&& task && task
.get_persistent_task_type() .get_persistent_task_type()
.is_some_and(|t| !t.native_fn.is_root) .is_some_and(|t| !t.native_fn.is_root)
...@@ -38,23 +49,36 @@ impl UpdateCollectibleOperation { ...@@ -38,23 +49,36 @@ impl UpdateCollectibleOperation {
); );
} }
let mut queue = AggregationUpdateQueue::new(); let mut queue = AggregationUpdateQueue::new();
let mut invalidated_types: SmallVec<[TraitTypeId; 1]> = SmallVec::new();
swap_retain(&mut collectibles, |(collectible, count)| {
let collectible = *collectible;
let outdated = task.get_outdated_collectibles(&collectible).copied(); let outdated = task.get_outdated_collectibles(&collectible).copied();
if let Some(outdated) = outdated { if let Some(outdated) = outdated {
if count > 0 && outdated > 0 { if *count > 0 && outdated > 0 {
let shared = min(count, outdated); let shared = min(*count, outdated);
let _ = task.update_outdated_collectibles_positive_crossing(collectible, -shared); let _ =
count -= shared; task.update_outdated_collectibles_positive_crossing(collectible, -shared);
} else if count < 0 && outdated < 0 { *count -= shared;
let shared = min(-count, -outdated); } else if *count < 0 && outdated < 0 {
let _ = task.update_outdated_collectibles_positive_crossing(collectible, shared); let shared = min(-*count, -outdated);
count += shared; let _ =
task.update_outdated_collectibles_positive_crossing(collectible, shared);
*count += shared;
} else { } else {
// Not reduced from outdated // Not reduced from outdated
} }
} }
if count != 0 { if *count == 0 {
if task.update_collectibles_positive_crossing(collectible, count) { return false;
let ty = collectible.collectible_type; }
if task.update_collectibles_positive_crossing(collectible, *count)
&& !invalidated_types.contains(&collectible.collectible_type)
{
invalidated_types.push(collectible.collectible_type);
}
true
});
for ty in invalidated_types {
let dependent: SmallVec<[TaskId; 4]> = task let dependent: SmallVec<[TaskId; 4]> = task
.iter_collectibles_dependents() .iter_collectibles_dependents()
.filter_map(|(collectible_type, task)| (collectible_type == ty).then_some(task)) .filter_map(|(collectible_type, task)| (collectible_type == ty).then_some(task))
...@@ -67,9 +91,10 @@ impl UpdateCollectibleOperation { ...@@ -67,9 +91,10 @@ impl UpdateCollectibleOperation {
}) })
} }
} }
if !collectibles.is_empty() {
queue.extend(AggregationUpdateJob::data_update( queue.extend(AggregationUpdateJob::data_update(
&mut task, &mut task,
AggregatedDataUpdate::new().collectibles_update(vec![(collectible, count)]), AggregatedDataUpdate::new().collectibles_update(collectibles),
)); ));
} }
......
#![feature(arbitrary_self_types)]
#![feature(arbitrary_self_types_pointers)]
#![allow(clippy::needless_return)] // tokio macro-generated code doesn't respect this
//! Taking the collectibles of an operation unemits all of them at once. This covers that every
//! taken collectible is hidden from the uppers of the taking task, including collectibles that
//! are emitted multiple times, also when the set of taken collectibles changes. It also covers
//! dropping collectibles, taking from a source without collectibles, and taking from a source
//! whose own collectibles were already taken (so its aggregated counts aren't all positive).
use anyhow::Result;
use turbo_rcstr::RcStr;
use turbo_tasks::{CollectiblesSource, ResolvedVc, State, ValueToString, Vc, emit};
use turbo_tasks_testing::{Registration, register, run_once};
static REGISTRATION: Registration = register!();
const UPPERS: u32 = 4;
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn taken_collectibles_are_unemitted_together() {
run_once(&REGISTRATION, async || {
let input = ChangingInput {
state: State::new(20),
}
.resolved_cell();
for count in [20, 5, 30, 0, 10] {
input.await?.state.set(count);
for drop in [false, true] {
for upper in 0..UPPERS {
let read = upper_operation(input, drop, upper)
.read_strongly_consistent()
.await?;
// `count` distinct collectibles plus the shared one. Dropping doesn't report
// what was removed.
let expected_taken = if drop { 0 } else { count + 1 };
assert_eq!(
read.taken, expected_taken,
"drop {drop}, upper {upper}, count {count}"
);
assert_eq!(
read.visible,
vec![format!("keep {count}")],
"drop {drop}, upper {upper}, count {count}"
);
}
}
let outer = outer_taking_operation(input)
.read_strongly_consistent()
.await?;
assert_eq!(*outer, vec![format!("keep {count}")], "count {count}");
}
assert_eq!(
*empty_taking_operation().read_strongly_consistent().await?,
0
);
anyhow::Ok(())
})
.await
.unwrap()
}
#[turbo_tasks::value]
struct ChangingInput {
state: State<u32>,
}
#[turbo_tasks::value]
struct Collectible {
value: RcStr,
}
#[turbo_tasks::value_impl]
impl ValueToString for Collectible {
#[turbo_tasks::function]
fn to_string(&self) -> Vc<RcStr> {
Vc::cell(self.value.clone())
}
}
fn collectible(value: String) -> ResolvedVc<Box<dyn ValueToString>> {
ResolvedVc::upcast(
Collectible {
value: value.into(),
}
.resolved_cell(),
)
}
#[turbo_tasks::function]
fn shared_collectible() -> Vc<Box<dyn ValueToString>> {
*collectible("shared".to_string())
}
/// Emits the shared collectible twice and `count` distinct ones.
#[turbo_tasks::function]
async fn emit_many(count: u32) -> Result<()> {
let shared = shared_collectible().to_resolved().await?;
emit(shared);
emit(shared);
for index in 0..count {
emit(collectible(format!("item {index} of {count}")));
}
Ok(())
}
/// Emits the shared collectible once more from another task.
#[turbo_tasks::function]
async fn emit_shared_again() -> Result<()> {
emit(shared_collectible().to_resolved().await?);
Ok(())
}
#[turbo_tasks::function(operation, root)]
async fn child_operation(input: ResolvedVc<ChangingInput>) -> Result<Vc<u32>> {
let count = *input.await?.state.get();
emit_many(count).await?;
emit_shared_again().await?;
Ok(Vc::cell(count))
}
#[turbo_tasks::function]
async fn emit_keep(count: u32) -> Result<()> {
emit(collectible(format!("keep {count}")));
Ok(())
}
/// Takes (or drops) all collectibles of the child operation, and emits one collectible of its own.
#[turbo_tasks::function(operation, root)]
async fn taking_operation(input: ResolvedVc<ChangingInput>, drop: bool) -> Result<Vc<u32>> {
let child = child_operation(input);
let count = *child.connect().await?;
let taken = if drop {
child.drop_collectibles::<Box<dyn ValueToString>>();
0
} else {
child.take_collectibles::<Box<dyn ValueToString>>().len() as u32
};
emit_keep(count).await?;
Ok(Vc::cell(taken))
}
#[turbo_tasks::value(transparent)]
struct Strings(Vec<String>);
/// Takes the collectibles of [`taking_operation`], which has already taken the collectibles of
/// its child.
#[turbo_tasks::function(operation, root)]
async fn outer_taking_operation(input: ResolvedVc<ChangingInput>) -> Result<Vc<Strings>> {
let taking = taking_operation(input, false);
taking.connect().await?;
let mut taken = Vec::new();
for collectible in taking.take_collectibles::<Box<dyn ValueToString>>() {
taken.push(collectible.to_string().await?.to_string());
}
taken.sort();
Ok(Vc::cell(taken))
}
#[turbo_tasks::function(operation, root)]
fn empty_operation() -> Vc<()> {
Vc::cell(())
}
#[turbo_tasks::function(operation, root)]
async fn empty_taking_operation() -> Result<Vc<u32>> {
let empty = empty_operation();
empty.connect().await?;
Ok(Vc::cell(
empty.take_collectibles::<Box<dyn ValueToString>>().len() as u32,
))
}
#[turbo_tasks::value]
struct UpperRead {
taken: u32,
visible: Vec<String>,
}
/// One of several uppers that connect the same taking operation.
#[turbo_tasks::function(operation, root)]
async fn upper_operation(
input: ResolvedVc<ChangingInput>,
drop: bool,
upper: u32,
) -> Result<Vc<UpperRead>> {
let _ = upper;
let taking = taking_operation(input, drop);
let taken = *taking.connect().await?;
let mut visible = Vec::new();
for collectible in taking.peek_collectibles::<Box<dyn ValueToString>>() {
visible.push(collectible.to_string().await?.to_string());
}
visible.sort();
Ok(UpperRead { taken, visible }.cell())
}
...@@ -703,6 +703,17 @@ pub trait Backend: Sized + Sync + Send { ...@@ -703,6 +703,17 @@ pub trait Backend: Sized + Sync + Send {
turbo_tasks: &TurboTasks<Self>, turbo_tasks: &TurboTasks<Self>,
); );
/// Unemits every collectible with a positive count in `collectibles` from `task`, as if
/// [`Backend::unemit_collectible`] was called for each of them, but applied as a single
/// operation.
fn unemit_collectibles(
&self,
trait_type: TraitTypeId,
collectibles: &TaskCollectiblesMap,
task: TaskId,
turbo_tasks: &TurboTasks<Self>,
);
fn update_task_cell( fn update_task_cell(
&self, &self,
task: TaskId, task: TaskId,
......
...@@ -2014,18 +2014,16 @@ impl<B: Backend + 'static> TurboTasksApi for TurboTasks<B> { ...@@ -2014,18 +2014,16 @@ impl<B: Backend + 'static> TurboTasksApi for TurboTasks<B> {
} }
fn unemit_collectibles(&self, trait_type: TraitTypeId, collectibles: &TaskCollectiblesMap) { fn unemit_collectibles(&self, trait_type: TraitTypeId, collectibles: &TaskCollectiblesMap) {
for (&collectible, &count) in collectibles { if !collectibles.values().any(|&count| count > 0) {
if count > 0 { return;
self.backend.unemit_collectible( }
self.backend.unemit_collectibles(
trait_type, trait_type,
collectible, collectibles,
count as u32, current_task("unemitting collectibles"),
current_task("emitting collectible"),
self, self,
); );
} }
}
}
fn read_own_task_cell(&self, task: TaskId, index: CellId) -> Result<TypedCellContent> { fn read_own_task_cell(&self, task: TaskId, index: CellId) -> Result<TypedCellContent> {
self.try_read_own_task_cell(task, index) self.try_read_own_task_cell(task, index)
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment