turbo-tasks-backend: optimize aggregated updates outside of the lock (#100165)
### What? Net the collectible counts of an `AggregatedDataUpdate` in a new `AggregatedDataUpdate::optimize()`, called after the follower task's lock has been released. ### Why? When an aggregating task gains (or loses) an upper, `AggregatedDataUpdate::from_task` builds an update from the task's own collectible counts plus `+1` for every positive aggregated collectible. - **Cancelling entries:** a task that takes or drops the collectibles of its subgraph holds an own `-1` for each of them, so most of those entries cancel out. An earlier large `next build` investigation found one such update with 106,268 entries, of which only 4 had a nonzero net count. - **Serialization:** in a large `next build`, ~6,000 uppers connect the same task. `from_task` runs while that follower's lock is held, so anything done per update while the lock is held serializes across all workers. - **Why the earlier version didn't help:** a previous version of this PR netted inside `from_task`, still under the lock. It didn't improve emit time. ### How? - **`from_task`:** back to its original form (own entries, then `+1` per positive aggregated entry). It keeps a `Vec::with_capacity` upper bound so the vector isn't reallocated while the task is locked. - **`optimize()`:** sums the collectible entries per collectible, drops zero sums, and keeps the dirty container update unchanged. It reuses the vector's allocation. - **Call sites:** `optimize()` is called right after the follower lock is dropped, at the sites that apply the update to uppers afterwards. These are the new-follower paths and the lost-follower paths. In the lost-follower paths, `invert()` also moved out of the lock, after `optimize()`. The `balance_edge` paths, which hold both locks through `apply`, are unchanged. - **Semantics:** applying the optimized update (or its inversion) gives the same final counts in the upper, and the same set of collectibles whose presence changes. Intermediate flips within one update can be skipped. Result order is unspecified. **Tests:** - Unit tests for summing, cancellation, mixed counts, empty and single entries, and disjoint keys. - Equivalence of final counts and presence changes for add and invert. - The real `optimize().invert()` path on `CollectibleRef`s, including the dirty container update. - An integration test that reads an operation which takes its child's collectibles, through several root uppers. **Measurement** on canary `8521c6ae` (`bench/nested-deps-app-router-many-pages`, 32 CPUs, cold untraced builds, 1 warm-up plus 4 interleaved rounds, median of the paired per-round changes vs canary): | | change | |---|---:| | `turbopack-emit` | **−32.2%** (17.4 s → 11.9 s) | | "Compiled in" | −11.2% | | wall | −10.0% | All three were faster in all 4 rounds. Output (`.next` file set) and issues are unchanged vs canary. This is one VM and 4 rounds, and the gain isn't broken down between the preallocation and moving work out of the lock. <!-- NEXT_JS_LLM --> <!-- fleet d8fee2f9-18db-4936-b59a-f8417b8f4bbc --> --------- Co-authored-by:vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by:
Tobias Koppers <1365881+sokra@users.noreply.github.com>
Showing
This diff is collapsed.
Please
register
or
sign in
to comment