Unverified Commit 75d2ed53 authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

turbopack: emit assets before duplicate check in emit_assets (#93875)

### What?

Reworks `emit_assets` in `crates/next-core/src/emit.rs` so that:

- The actual `emit()` / `emit_rebase()` write runs **before**
`check_duplicates()`.
- Per-duplicate diff checks inside `check_duplicates` run in parallel.
- At most one `EmitConflictIssue` is emitted per asset path, even when
multiple duplicates conflict.
- The conflict-issue emission is wrapped in a `turbo_tasks::function`
and reported via `as_side_effect()`.

### Why?

Under eventual consistency, all children might be removed from
`emit_assets`. This causes all `content()` and `emit()` tasks being
inactive and keeping the stale value (which is potentially an error).

On recomputation of `emit_assets()` this only makes very few `content()`
tasks active, as tasks are still in a stale error state and make
`emit_assets()` exit early. So it takes N recomputations of
`emit_assets()` to be up-to-date, where N is the number of duplicate
assets. This can slow down builds drastically and did cause build time
up to 30min.

Emitting the asset first ensures the file is written to disk even if the
subsequent duplicate diagnostic crashes under eventual consistency.
Doing the diff checks in parallel and folding conflicts into a single
issue keeps the diagnostic path cheap and avoids noisy duplicate output
for the same path. Tracking the issue emission as a turbo-tasks side
effect means it is properly attributed to a task instead of being
emitted ad-hoc from `emit_assets`.

### How?

- Swap the order in both branches of `emit_assets` (server-side `emit`
and client-side `emit_rebase`) so the write happens first, then
`check_duplicates` runs as a diagnostic afterwards. A comment is left
explaining the ordering requirement.
- `check_duplicates` now returns `Result<()>` instead of returning the
picked asset. The caller picks `assets.first()` directly, since
`check_duplicates` only validated equivalence — it didn't choose a
different asset.
- The inner loop in `check_duplicates` uses `iter.map(async |next| { ...
}).try_flat_join().await?` to run the diff comparisons in parallel and
collect conflict details. `ext` is computed once before the loop and
cloned into each task.
- Only the first conflict from the collected list is reported, instead
of one issue per conflicting asset.
- The `EmitConflictIssue` emission is moved into a local
`#[turbo_tasks::function] fn emit_conflict_issue(...)` invoked via
`.as_side_effect().await?`, so the issue is tracked as a task side
effect.

<!-- NEXT_JS_LLM_PR -->

---------
Co-authored-by: 's avatarClaude Opus 4.7 (1M context) <noreply@anthropic.com>
parent e2fca6c6
...@@ -100,15 +100,21 @@ pub async fn emit_assets( ...@@ -100,15 +100,21 @@ pub async fn emit_assets(
path: &FileSystemPath, path: &FileSystemPath,
assets: AssetVec, assets: AssetVec,
node_root: &FileSystemPath, node_root: &FileSystemPath,
) -> Result<ResolvedVc<Box<dyn OutputAsset>>> { ) -> Result<()> {
let mut iter = assets.into_iter(); let mut iter = assets.into_iter();
let first = iter.next().unwrap(); let first = iter.next().unwrap();
for next in iter { let ext: RcStr = path.extension().unwrap_or_default().into();
let ext: RcStr = path.extension().unwrap_or_default().into(); let conflicts = iter
if let Some(detail) = assets_diff(*next, *first, ext, node_root.clone()) .map(async |next| {
.owned() assets_diff(*next, *first, ext.clone(), node_root.clone())
.await? .owned()
{ .await
})
.try_flat_join()
.await?;
if let Some(detail) = conflicts.into_iter().next() {
#[turbo_tasks::function]
fn emit_conflict_issue(path: FileSystemPath, detail: RcStr) {
EmitConflictIssue { EmitConflictIssue {
asset_path: path.clone(), asset_path: path.clone(),
detail, detail,
...@@ -116,8 +122,11 @@ pub async fn emit_assets( ...@@ -116,8 +122,11 @@ pub async fn emit_assets(
.resolved_cell() .resolved_cell()
.emit(); .emit();
} }
emit_conflict_issue(path.clone(), detail)
.as_side_effect()
.await?;
} }
Ok(first) Ok(())
} }
// Use join! instead of try_join! to collect all errors deterministically // Use join! instead of try_join! to collect all errors deterministically
...@@ -129,14 +138,20 @@ pub async fn emit_assets( ...@@ -129,14 +138,20 @@ pub async fn emit_assets(
let node_root = node_root.clone(); let node_root = node_root.clone();
async move { async move {
let asset = check_duplicates(&path, assets, &node_root).await?; let asset = *assets.first().unwrap();
let span = tracing::info_span!( let span = tracing::info_span!(
"emit asset", "emit asset",
name = %path.to_string_ref().await? name = %path.to_string_ref().await?
); );
async move { emit(*asset).as_side_effect().await } async move {
.instrument(span) emit(*asset).as_side_effect().await?;
.await // This need to be after `emit()`, so the asset is emitted even if this
// method crashes due to eventual consistency.
check_duplicates(&path, assets, &node_root).await?;
Ok(())
}
.instrument(span)
.await
} }
}) })
.try_join(), .try_join(),
...@@ -148,18 +163,22 @@ pub async fn emit_assets( ...@@ -148,18 +163,22 @@ pub async fn emit_assets(
let client_output_path = client_output_path.clone(); let client_output_path = client_output_path.clone();
async move { async move {
let asset = check_duplicates(&path, assets, &node_root).await?;
let span = tracing::info_span!( let span = tracing::info_span!(
"emit asset", "emit asset",
name = %path.to_string_ref().await? name = %path.to_string_ref().await?
); );
async move { async move {
let asset = *assets.first().unwrap();
// Client assets are emitted to the client output path, which is // Client assets are emitted to the client output path, which is
// prefixed with _next. We need to rebase them to // prefixed with _next. We need to rebase them to
// remove that prefix. // remove that prefix.
emit_rebase(*asset, client_relative_path, client_output_path) emit_rebase(*asset, client_relative_path, client_output_path)
.as_side_effect() .as_side_effect()
.await .await?;
// This need to be after `emit_rebase()`, so the asset is emitted even if
// this method crashes due to eventual consistency.
check_duplicates(&path, assets, &node_root).await?;
Ok(())
} }
.instrument(span) .instrument(span)
.await .await
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment