Unverified Commit be19b72b authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

turbopack: report chunk group consistency errors instead of panicking (#100923)

## Summary

Under eventual consistency, chunk group data from different graph
versions can briefly be paired with each other. Two lookups still panic
in that case. This PR makes both of them return errors, so the
computation is retried once the data is consistent again.

- **Merged-parent lookups.** `ChunkGroupKey::ident_str` resolved the
parent of an `IsolatedMerged`/`SharedMerged` key with
`keys.index(parent)`. Some callers render a key whose parent index comes
from another (e.g. newer) `ChunkGroupInfo` than `keys`, for example
`ChunkGroupInfo::get_index_of`'s debug failure path and
`chunk_group_content_operation`. That panicked with
`chunk_group_info.rs: index out of bounds: the len is 28 but the index
is 29` during `next build`.
  - The lookup is now checked and returns an error.
- `keys` is narrowed to `&FxIndexSet<ChunkGroupKey>` (also for
`debug_str`).
  - `ChunkGroup::debug_str` resolves parents through `chunk_group_at`.
- The output of a successful render is unchanged, so asset names don't
change.
- **`AvailableChunkGroups::assert_module_graph`.** This was a
`debug_assert_eq!`. It now returns `Result`, and a graph mismatch is an
error that names both graphs. The check is a cheap identity comparison,
so it runs in all builds, not only debug builds.
- `chunk_groups` and `with_chunk_group` now return `Result`, and callers
propagate the error.
- In `chunk_group_content_operation` the check runs once up front
instead of in the per-module closures.

#100928 / #100949 already cover the index panics in
`get_chunk_groups_for_module` and `get_chunk_group`.

## Verification

- `cargo test -p turbopack-core`. Adds
`merged_chunk_group_with_unknown_parent_is_an_error`, which panicked at
the same site as CI before this change.
- `cargo clippy -p turbopack-core -p next-api -p next-napi-bindings
--tests`

<!-- NEXT_JS_LLM -->


<!-- fleet 8482d6ce-bad2-4131-98e3-4a391357a1c9 -->
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent 7c5089fe
...@@ -93,13 +93,12 @@ impl AvailabilityInfo { ...@@ -93,13 +93,12 @@ impl AvailabilityInfo {
}, },
ChunkableModuleOrBatch::None(_) => return Ok(false), ChunkableModuleOrBatch::None(_) => return Ok(false),
}; };
let available = available_chunk_groups.chunk_groups(module_graph)?;
Ok(chunk_group_info Ok(chunk_group_info
.module_chunk_groups .module_chunk_groups
.await? .await?
.get(&module) .get(&module)
.is_some_and(|groups| { .is_some_and(|groups| !groups.is_disjoint(available)))
!groups.is_disjoint(available_chunk_groups.chunk_groups(module_graph))
}))
} }
pub fn in_async_module(self) -> Self { pub fn in_async_module(self) -> Self {
...@@ -173,7 +172,7 @@ impl AvailabilityInfo { ...@@ -173,7 +172,7 @@ impl AvailabilityInfo {
Some(available_chunk_groups) => { Some(available_chunk_groups) => {
let available_chunk_groups = available_chunk_groups.await?; let available_chunk_groups = available_chunk_groups.await?;
let canonical = let canonical =
relevant.canonicalize(available_chunk_groups.chunk_groups(module_graph)); relevant.canonicalize(available_chunk_groups.chunk_groups(module_graph)?);
if canonical.is_zero() { if canonical.is_zero() {
None None
} else { } else {
......
...@@ -15,7 +15,7 @@ use crate::module_graph::{ ...@@ -15,7 +15,7 @@ use crate::module_graph::{
/// ///
/// The bitmap holds indices into the [`crate::module_graph::chunk_group_info::ChunkGroupInfo`] of a /// The bitmap holds indices into the [`crate::module_graph::chunk_group_info::ChunkGroupInfo`] of a
/// single [`ModuleGraph`]. The same index means something entirely different in another graph, so /// single [`ModuleGraph`]. The same index means something entirely different in another graph, so
/// the graph is stored alongside the bitmap and every operation asserts that it matches. Use /// the graph is stored alongside the bitmap and every operation checks that it matches. Use
/// [`AvailableChunkGroups::translate`] to move availability to a different graph. /// [`AvailableChunkGroups::translate`] to move availability to a different graph.
#[turbo_tasks::value] #[turbo_tasks::value]
#[derive(Clone, Debug)] #[derive(Clone, Debug)]
...@@ -26,22 +26,31 @@ pub struct AvailableChunkGroups { ...@@ -26,22 +26,31 @@ pub struct AvailableChunkGroups {
} }
impl AvailableChunkGroups { impl AvailableChunkGroups {
/// Panics in debug builds when `module_graph` is not the graph these chunk group indices /// Errors when `module_graph` is not the graph these chunk group indices belong to. Chunk
/// belong to. Chunk group indices of a different graph would silently resolve to unrelated /// group indices of a different graph would silently resolve to unrelated chunk groups, which
/// chunk groups, which drops modules from the output. /// drops modules from the output.
pub fn assert_module_graph(&self, module_graph: ResolvedVc<ModuleGraph>) { ///
debug_assert_eq!( /// This is an error rather than a panic: with eventual consistency a graph can briefly be
self.module_graph, module_graph, /// paired with availability computed for another graph, and the computation is retried once
"AvailableChunkGroups was created for a different ModuleGraph. Chunk group indices \ /// the data is consistent again.
are only valid within the ChunkGroupInfo of the graph that produced them; use \ pub fn assert_module_graph(&self, module_graph: ResolvedVc<ModuleGraph>) -> Result<()> {
AvailableChunkGroups::translate to move them to another graph." if self.module_graph != module_graph {
); bail!(
"AvailableChunkGroups was created for a different ModuleGraph. Chunk group \
indices are only valid within the ChunkGroupInfo of the graph that produced \
them; use AvailableChunkGroups::translate to move them to another graph. \
(expected {:?}, got {:?})",
self.module_graph,
module_graph
);
}
Ok(())
} }
/// The chunk groups that are available, for `module_graph`. /// The chunk groups that are available, for `module_graph`.
pub fn chunk_groups(&self, module_graph: ResolvedVc<ModuleGraph>) -> &Bitset { pub fn chunk_groups(&self, module_graph: ResolvedVc<ModuleGraph>) -> Result<&Bitset> {
self.assert_module_graph(module_graph); self.assert_module_graph(module_graph)?;
&self.chunk_groups Ok(&self.chunk_groups)
} }
} }
...@@ -86,15 +95,15 @@ impl AvailableChunkGroups { ...@@ -86,15 +95,15 @@ impl AvailableChunkGroups {
&self, &self,
module_graph: ResolvedVc<ModuleGraph>, module_graph: ResolvedVc<ModuleGraph>,
chunk_group: u32, chunk_group: u32,
) -> Vc<Self> { ) -> Result<Vc<Self>> {
self.assert_module_graph(module_graph); self.assert_module_graph(module_graph)?;
let mut chunk_groups = self.chunk_groups.clone(); let mut chunk_groups = self.chunk_groups.clone();
chunk_groups.insert(chunk_group); chunk_groups.insert(chunk_group);
Self { Ok(Self {
module_graph: self.module_graph, module_graph: self.module_graph,
chunk_groups, chunk_groups,
} }
.cell() .cell())
} }
/// Re-expresses this availability in terms of `module_graph`'s /// Re-expresses this availability in terms of `module_graph`'s
...@@ -232,6 +241,13 @@ mod tests { ...@@ -232,6 +241,13 @@ mod tests {
} }
} }
fn shared_merged(parent: usize, merge_tag: &str) -> ChunkGroupKey {
ChunkGroupKey::SharedMerged {
parent: ChunkGroupId::from(parent),
merge_tag: merge_tag.into(),
}
}
/// The hash `AvailableChunkGroups::hash` produces for the chunk groups `ids` of `keys`. /// The hash `AvailableChunkGroups::hash` produces for the chunk groups `ids` of `keys`.
/// ///
/// Hashes the keys the way `ChunkGroupInfo::chunk_group_ident_hashes` does. Every key here is /// Hashes the keys the way `ChunkGroupInfo::chunk_group_ident_hashes` does. Every key here is
...@@ -371,6 +387,25 @@ mod tests { ...@@ -371,6 +387,25 @@ mod tests {
assert_ne!(hash_of(&keys, &[0]).await, hash_of(&keys, &[0, 1]).await); assert_ne!(hash_of(&keys, &[0]).await, hash_of(&keys, &[0, 1]).await);
} }
#[tokio::test]
async fn merged_chunk_group_with_unknown_parent_is_an_error() {
// A key can carry a parent index from another (e.g. newer) `ChunkGroupInfo` than `keys`.
for key in [isolated_merged(1, "x"), shared_merged(1, "x")] {
let keys = FxIndexSet::from_iter([key]);
let error = keys[0]
.ident_str(&keys)
.await
.expect_err("an unknown merged parent must be reported as an error");
assert!(
error
.to_string()
.contains("parent chunk group 1 of a merged chunk group"),
"unexpected error: {error:?}"
);
}
}
#[tokio::test] #[tokio::test]
async fn hash_never_contains_a_chunk_group_index() { async fn hash_never_contains_a_chunk_group_index() {
// A merged group identifies its parent by index. Two graphs that agree on the groups but // A merged group identifies its parent by index. Two graphs that agree on the groups but
......
...@@ -326,13 +326,16 @@ async fn chunk_group_content_operation( ...@@ -326,13 +326,16 @@ async fn chunk_group_content_operation(
let available_chunk_groups = match availability_info.available_chunk_groups() { let available_chunk_groups = match availability_info.available_chunk_groups() {
Some(available_chunk_groups) => { Some(available_chunk_groups) => {
let available_chunk_groups = available_chunk_groups.await?; let available_chunk_groups = available_chunk_groups.await?;
// The bitmap indexes this graph's `ChunkGroupInfo`; indices of another graph would
// resolve to unrelated chunk groups and silently drop modules.
available_chunk_groups.assert_module_graph(module_graph);
Some(available_chunk_groups) Some(available_chunk_groups)
} }
None => None, None => None,
}; };
// The bitmap indexes this graph's `ChunkGroupInfo`; indices of another graph would resolve
// to unrelated chunk groups and silently drop modules.
let available_chunk_groups = available_chunk_groups
.as_ref()
.map(|available| available.chunk_groups(module_graph))
.transpose()?;
let module_chunk_groups = chunk_group_info.module_chunk_groups.await?; let module_chunk_groups = chunk_group_info.module_chunk_groups.await?;
let async_loader_chunk_groups = chunk_group_info.async_loader_chunk_groups.await?; let async_loader_chunk_groups = chunk_group_info.async_loader_chunk_groups.await?;
...@@ -395,9 +398,8 @@ async fn chunk_group_content_operation( ...@@ -395,9 +398,8 @@ async fn chunk_group_content_operation(
.get_first_module(chunkable_node) .get_first_module(chunkable_node)
.and_then(|module| module_chunk_groups.get(&module)) .and_then(|module| module_chunk_groups.get(&module))
.is_some_and(|groups| { .is_some_and(|groups| {
available_chunk_groups.as_ref().is_some_and(|available| { available_chunk_groups
!groups.is_disjoint(available.chunk_groups(module_graph)) .is_some_and(|available| !groups.is_disjoint(available))
})
}); });
let Some((_, edge)) = parent_info else { let Some((_, edge)) = parent_info else {
...@@ -439,9 +441,8 @@ async fn chunk_group_content_operation( ...@@ -439,9 +441,8 @@ async fn chunk_group_content_operation(
let is_async_loader_available = async_loader_chunk_groups let is_async_loader_available = async_loader_chunk_groups
.get(&ResolvedVc::upcast(chunkable_module)) .get(&ResolvedVc::upcast(chunkable_module))
.is_some_and(|groups| { .is_some_and(|groups| {
available_chunk_groups.as_ref().is_some_and(|available| { available_chunk_groups
!groups.is_disjoint(available.chunk_groups(module_graph)) .is_some_and(|available| !groups.is_disjoint(available))
})
}); });
if !is_async_loader_available { if !is_async_loader_available {
state.async_modules.insert(chunkable_module); state.async_modules.insert(chunkable_module);
......
...@@ -516,8 +516,12 @@ impl ChunkGroup { ...@@ -516,8 +516,12 @@ impl ChunkGroup {
} => { } => {
format!( format!(
"ChunkGroup::IsolatedMerged({}, {}, {:?})", "ChunkGroup::IsolatedMerged({}, {}, {:?})",
Box::pin(chunk_group_info.chunk_groups[*parent].debug_str(chunk_group_info)) Box::pin(
.await?, chunk_group_info
.chunk_group_at(*parent)?
.debug_str(chunk_group_info)
)
.await?,
merge_tag, merge_tag,
entries entries
.iter() .iter()
...@@ -533,8 +537,12 @@ impl ChunkGroup { ...@@ -533,8 +537,12 @@ impl ChunkGroup {
} => { } => {
format!( format!(
"ChunkGroup::SharedMerged({}, {}, {:?})", "ChunkGroup::SharedMerged({}, {}, {:?})",
Box::pin(chunk_group_info.chunk_groups[*parent].debug_str(chunk_group_info)) Box::pin(
.await?, chunk_group_info
.chunk_group_at(*parent)?
.debug_str(chunk_group_info)
)
.await?,
merge_tag, merge_tag,
entries entries
.iter() .iter()
...@@ -580,10 +588,11 @@ impl ChunkGroupKey { ...@@ -580,10 +588,11 @@ impl ChunkGroupKey {
/// names from this (through [`ChunkGroupInfo::chunk_group_ident_hashes`]), so two chunk groups /// names from this (through [`ChunkGroupInfo::chunk_group_ident_hashes`]), so two chunk groups
/// must render equally exactly when they are the same chunk group. Changing the rendering /// must render equally exactly when they are the same chunk group. Changing the rendering
/// renames assets; adding anything graph-local to it is a correctness bug. /// renames assets; adding anything graph-local to it is a correctness bug.
pub async fn ident_str( ///
&self, /// Errors when the parent of a merged chunk group is not in `keys`. That happens when the
keys: &(impl std::ops::Index<usize, Output = Self> + ?Sized), /// key comes from a different (e.g. newer) [`ChunkGroupInfo`] than `keys`, which is transient
) -> Result<String> { /// with eventual consistency.
pub async fn ident_str(&self, keys: &FxIndexSet<ChunkGroupKey>) -> Result<String> {
Ok(match self { Ok(match self {
ChunkGroupKey::Entry(entries) => format!( ChunkGroupKey::Entry(entries) => format!(
"Entry({:?})", "Entry({:?})",
...@@ -605,7 +614,7 @@ impl ChunkGroupKey { ...@@ -605,7 +614,7 @@ impl ChunkGroupKey {
ChunkGroupKey::IsolatedMerged { parent, merge_tag } => { ChunkGroupKey::IsolatedMerged { parent, merge_tag } => {
format!( format!(
"IsolatedMerged {{ parent: {}, merge_tag: {:?} }}", "IsolatedMerged {{ parent: {}, merge_tag: {:?} }}",
Box::pin(keys.index(parent.0 as usize).ident_str(keys)).await?, Box::pin(Self::merged_parent(keys, parent.0 as usize)?.ident_str(keys)).await?,
merge_tag merge_tag
) )
} }
...@@ -623,7 +632,7 @@ impl ChunkGroupKey { ...@@ -623,7 +632,7 @@ impl ChunkGroupKey {
ChunkGroupKey::SharedMerged { parent, merge_tag } => { ChunkGroupKey::SharedMerged { parent, merge_tag } => {
format!( format!(
"SharedMerged {{ parent: {}, merge_tag: {:?} }}", "SharedMerged {{ parent: {}, merge_tag: {:?} }}",
Box::pin(keys.index(parent.0 as usize).ident_str(keys)).await?, Box::pin(Self::merged_parent(keys, parent.0 as usize)?.ident_str(keys)).await?,
merge_tag merge_tag
) )
} }
...@@ -635,12 +644,21 @@ impl ChunkGroupKey { ...@@ -635,12 +644,21 @@ impl ChunkGroupKey {
/// Delegates to [`Self::ident_str`], which already renders the chunk group unambiguously. /// Delegates to [`Self::ident_str`], which already renders the chunk group unambiguously.
/// Debug output is free to diverge from it, but note that `ident_str` must stay stable because /// Debug output is free to diverge from it, but note that `ident_str` must stay stable because
/// asset idents are derived from it -- change this wrapper rather than `ident_str` itself. /// asset idents are derived from it -- change this wrapper rather than `ident_str` itself.
pub async fn debug_str( pub async fn debug_str(&self, keys: &FxIndexSet<ChunkGroupKey>) -> Result<String> {
&self,
keys: &(impl std::ops::Index<usize, Output = Self> + ?Sized),
) -> Result<String> {
self.ident_str(keys).await self.ident_str(keys).await
} }
/// The parent key of a merged chunk group. `parent` can be out of range when the merged key
/// comes from a different [`ChunkGroupInfo`] than `keys`, so this is an error, not a panic.
fn merged_parent(keys: &FxIndexSet<ChunkGroupKey>, parent: usize) -> Result<&ChunkGroupKey> {
keys.get_index(parent).with_context(|| {
format!(
"parent chunk group {parent} of a merged chunk group is not a known chunk group \
({} chunk groups)",
keys.len()
)
})
}
} }
#[turbo_tasks::task_input] #[turbo_tasks::task_input]
...@@ -1017,12 +1035,9 @@ pub async fn compute_chunk_group_info( ...@@ -1017,12 +1035,9 @@ pub async fn compute_chunk_group_info(
let mut result = vec![]; let mut result = vec![];
result.push("Chunk Groups:".to_string()); result.push("Chunk Groups:".to_string());
for (i, (key, _)) in chunk_groups_map.iter().enumerate() { let keys = chunk_groups_map.keys().cloned().collect::<FxIndexSet<_>>();
result.push(format!( for (i, key) in keys.iter().enumerate() {
" {:?}: {}", result.push(format!(" {:?}: {}", i, key.debug_str(&keys).await?));
i,
key.debug_str(&chunk_groups_map.keys()).await?
));
} }
result.push("# Module buckets:".to_string()); result.push("# Module buckets:".to_string());
for (key, modules) in buckets.iter() { for (key, modules) in buckets.iter() {
......
...@@ -4399,7 +4399,7 @@ pub mod tests { ...@@ -4399,7 +4399,7 @@ pub mod tests {
{ {
let chunk_groups = async |availability: AvailabilityInfo| -> Result<Bitset> { let chunk_groups = async |availability: AvailabilityInfo| -> Result<Bitset> {
Ok(match availability.available_chunk_groups() { Ok(match availability.available_chunk_groups() {
Some(available) => available.await?.chunk_groups(module_graph).clone(), Some(available) => available.await?.chunk_groups(module_graph)?.clone(),
None => Bitset::new(), None => Bitset::new(),
}) })
}; };
...@@ -4414,11 +4414,11 @@ pub mod tests { ...@@ -4414,11 +4414,11 @@ pub mod tests {
} }
} }
let available = match result.available_chunk_groups() { let available = match result.available_chunk_groups() {
Some(available) => describe(available.await?.chunk_groups(module_graph)), Some(available) => describe(available.await?.chunk_groups(module_graph)?),
None => vec![], None => vec![],
}; };
narrowed_input.push(match availability.available_chunk_groups() { narrowed_input.push(match availability.available_chunk_groups() {
Some(available) => describe(available.await?.chunk_groups(module_graph)), Some(available) => describe(available.await?.chunk_groups(module_graph)?),
None => vec![], None => vec![],
}); });
narrowed.push((availability, result, available)); narrowed.push((availability, result, available));
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment