Unverified Commit be19b72b authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

turbopack: report chunk group consistency errors instead of panicking (#100923)

## Summary

Under eventual consistency, chunk group data from different graph
versions can briefly be paired with each other. Two lookups still panic
in that case. This PR makes both of them return errors, so the
computation is retried once the data is consistent again.

- **Merged-parent lookups.** `ChunkGroupKey::ident_str` resolved the
parent of an `IsolatedMerged`/`SharedMerged` key with
`keys.index(parent)`. Some callers render a key whose parent index comes
from another (e.g. newer) `ChunkGroupInfo` than `keys`, for example
`ChunkGroupInfo::get_index_of`'s debug failure path and
`chunk_group_content_operation`. That panicked with
`chunk_group_info.rs: index out of bounds: the len is 28 but the index
is 29` during `next build`.
  - The lookup is now checked and returns an error.
- `keys` is narrowed to `&FxIndexSet<ChunkGroupKey>` (also for
`debug_str`).
  - `ChunkGroup::debug_str` resolves parents through `chunk_group_at`.
- The output of a successful render is unchanged, so asset names don't
change.
- **`AvailableChunkGroups::assert_module_graph`.** This was a
`debug_assert_eq!`. It now returns `Result`, and a graph mismatch is an
error that names both graphs. The check is a cheap identity comparison,
so it runs in all builds, not only debug builds.
- `chunk_groups` and `with_chunk_group` now return `Result`, and callers
propagate the error.
- In `chunk_group_content_operation` the check runs once up front
instead of in the per-module closures.

#100928 / #100949 already cover the index panics in
`get_chunk_groups_for_module` and `get_chunk_group`.

## Verification

- `cargo test -p turbopack-core`. Adds
`merged_chunk_group_with_unknown_parent_is_an_error`, which panicked at
the same site as CI before this change.
- `cargo clippy -p turbopack-core -p next-api -p next-napi-bindings
--tests`

<!-- NEXT_JS_LLM -->


<!-- fleet 8482d6ce-bad2-4131-98e3-4a391357a1c9 -->
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent 7c5089fe
......@@ -93,13 +93,12 @@ impl AvailabilityInfo {
},
ChunkableModuleOrBatch::None(_) => return Ok(false),
};
let available = available_chunk_groups.chunk_groups(module_graph)?;
Ok(chunk_group_info
.module_chunk_groups
.await?
.get(&module)
.is_some_and(|groups| {
!groups.is_disjoint(available_chunk_groups.chunk_groups(module_graph))
}))
.is_some_and(|groups| !groups.is_disjoint(available)))
}
pub fn in_async_module(self) -> Self {
......@@ -173,7 +172,7 @@ impl AvailabilityInfo {
Some(available_chunk_groups) => {
let available_chunk_groups = available_chunk_groups.await?;
let canonical =
relevant.canonicalize(available_chunk_groups.chunk_groups(module_graph));
relevant.canonicalize(available_chunk_groups.chunk_groups(module_graph)?);
if canonical.is_zero() {
None
} else {
......
......@@ -15,7 +15,7 @@ use crate::module_graph::{
///
/// The bitmap holds indices into the [`crate::module_graph::chunk_group_info::ChunkGroupInfo`] of a
/// single [`ModuleGraph`]. The same index means something entirely different in another graph, so
/// the graph is stored alongside the bitmap and every operation asserts that it matches. Use
/// the graph is stored alongside the bitmap and every operation checks that it matches. Use
/// [`AvailableChunkGroups::translate`] to move availability to a different graph.
#[turbo_tasks::value]
#[derive(Clone, Debug)]
......@@ -26,22 +26,31 @@ pub struct AvailableChunkGroups {
}
impl AvailableChunkGroups {
/// Panics in debug builds when `module_graph` is not the graph these chunk group indices
/// belong to. Chunk group indices of a different graph would silently resolve to unrelated
/// chunk groups, which drops modules from the output.
pub fn assert_module_graph(&self, module_graph: ResolvedVc<ModuleGraph>) {
debug_assert_eq!(
self.module_graph, module_graph,
"AvailableChunkGroups was created for a different ModuleGraph. Chunk group indices \
are only valid within the ChunkGroupInfo of the graph that produced them; use \
AvailableChunkGroups::translate to move them to another graph."
);
/// Errors when `module_graph` is not the graph these chunk group indices belong to. Chunk
/// group indices of a different graph would silently resolve to unrelated chunk groups, which
/// drops modules from the output.
///
/// This is an error rather than a panic: with eventual consistency a graph can briefly be
/// paired with availability computed for another graph, and the computation is retried once
/// the data is consistent again.
pub fn assert_module_graph(&self, module_graph: ResolvedVc<ModuleGraph>) -> Result<()> {
if self.module_graph != module_graph {
bail!(
"AvailableChunkGroups was created for a different ModuleGraph. Chunk group \
indices are only valid within the ChunkGroupInfo of the graph that produced \
them; use AvailableChunkGroups::translate to move them to another graph. \
(expected {:?}, got {:?})",
self.module_graph,
module_graph
);
}
Ok(())
}
/// The chunk groups that are available, for `module_graph`.
pub fn chunk_groups(&self, module_graph: ResolvedVc<ModuleGraph>) -> &Bitset {
self.assert_module_graph(module_graph);
&self.chunk_groups
pub fn chunk_groups(&self, module_graph: ResolvedVc<ModuleGraph>) -> Result<&Bitset> {
self.assert_module_graph(module_graph)?;
Ok(&self.chunk_groups)
}
}
......@@ -86,15 +95,15 @@ impl AvailableChunkGroups {
&self,
module_graph: ResolvedVc<ModuleGraph>,
chunk_group: u32,
) -> Vc<Self> {
self.assert_module_graph(module_graph);
) -> Result<Vc<Self>> {
self.assert_module_graph(module_graph)?;
let mut chunk_groups = self.chunk_groups.clone();
chunk_groups.insert(chunk_group);
Self {
Ok(Self {
module_graph: self.module_graph,
chunk_groups,
}
.cell()
.cell())
}
/// Re-expresses this availability in terms of `module_graph`'s
......@@ -232,6 +241,13 @@ mod tests {
}
}
fn shared_merged(parent: usize, merge_tag: &str) -> ChunkGroupKey {
ChunkGroupKey::SharedMerged {
parent: ChunkGroupId::from(parent),
merge_tag: merge_tag.into(),
}
}
/// The hash `AvailableChunkGroups::hash` produces for the chunk groups `ids` of `keys`.
///
/// Hashes the keys the way `ChunkGroupInfo::chunk_group_ident_hashes` does. Every key here is
......@@ -371,6 +387,25 @@ mod tests {
assert_ne!(hash_of(&keys, &[0]).await, hash_of(&keys, &[0, 1]).await);
}
#[tokio::test]
async fn merged_chunk_group_with_unknown_parent_is_an_error() {
// A key can carry a parent index from another (e.g. newer) `ChunkGroupInfo` than `keys`.
for key in [isolated_merged(1, "x"), shared_merged(1, "x")] {
let keys = FxIndexSet::from_iter([key]);
let error = keys[0]
.ident_str(&keys)
.await
.expect_err("an unknown merged parent must be reported as an error");
assert!(
error
.to_string()
.contains("parent chunk group 1 of a merged chunk group"),
"unexpected error: {error:?}"
);
}
}
#[tokio::test]
async fn hash_never_contains_a_chunk_group_index() {
// A merged group identifies its parent by index. Two graphs that agree on the groups but
......
......@@ -326,13 +326,16 @@ async fn chunk_group_content_operation(
let available_chunk_groups = match availability_info.available_chunk_groups() {
Some(available_chunk_groups) => {
let available_chunk_groups = available_chunk_groups.await?;
// The bitmap indexes this graph's `ChunkGroupInfo`; indices of another graph would
// resolve to unrelated chunk groups and silently drop modules.
available_chunk_groups.assert_module_graph(module_graph);
Some(available_chunk_groups)
}
None => None,
};
// The bitmap indexes this graph's `ChunkGroupInfo`; indices of another graph would resolve
// to unrelated chunk groups and silently drop modules.
let available_chunk_groups = available_chunk_groups
.as_ref()
.map(|available| available.chunk_groups(module_graph))
.transpose()?;
let module_chunk_groups = chunk_group_info.module_chunk_groups.await?;
let async_loader_chunk_groups = chunk_group_info.async_loader_chunk_groups.await?;
......@@ -395,9 +398,8 @@ async fn chunk_group_content_operation(
.get_first_module(chunkable_node)
.and_then(|module| module_chunk_groups.get(&module))
.is_some_and(|groups| {
available_chunk_groups.as_ref().is_some_and(|available| {
!groups.is_disjoint(available.chunk_groups(module_graph))
})
available_chunk_groups
.is_some_and(|available| !groups.is_disjoint(available))
});
let Some((_, edge)) = parent_info else {
......@@ -439,9 +441,8 @@ async fn chunk_group_content_operation(
let is_async_loader_available = async_loader_chunk_groups
.get(&ResolvedVc::upcast(chunkable_module))
.is_some_and(|groups| {
available_chunk_groups.as_ref().is_some_and(|available| {
!groups.is_disjoint(available.chunk_groups(module_graph))
})
available_chunk_groups
.is_some_and(|available| !groups.is_disjoint(available))
});
if !is_async_loader_available {
state.async_modules.insert(chunkable_module);
......
......@@ -516,8 +516,12 @@ impl ChunkGroup {
} => {
format!(
"ChunkGroup::IsolatedMerged({}, {}, {:?})",
Box::pin(chunk_group_info.chunk_groups[*parent].debug_str(chunk_group_info))
.await?,
Box::pin(
chunk_group_info
.chunk_group_at(*parent)?
.debug_str(chunk_group_info)
)
.await?,
merge_tag,
entries
.iter()
......@@ -533,8 +537,12 @@ impl ChunkGroup {
} => {
format!(
"ChunkGroup::SharedMerged({}, {}, {:?})",
Box::pin(chunk_group_info.chunk_groups[*parent].debug_str(chunk_group_info))
.await?,
Box::pin(
chunk_group_info
.chunk_group_at(*parent)?
.debug_str(chunk_group_info)
)
.await?,
merge_tag,
entries
.iter()
......@@ -580,10 +588,11 @@ impl ChunkGroupKey {
/// names from this (through [`ChunkGroupInfo::chunk_group_ident_hashes`]), so two chunk groups
/// must render equally exactly when they are the same chunk group. Changing the rendering
/// renames assets; adding anything graph-local to it is a correctness bug.
pub async fn ident_str(
&self,
keys: &(impl std::ops::Index<usize, Output = Self> + ?Sized),
) -> Result<String> {
///
/// Errors when the parent of a merged chunk group is not in `keys`. That happens when the
/// key comes from a different (e.g. newer) [`ChunkGroupInfo`] than `keys`, which is transient
/// with eventual consistency.
pub async fn ident_str(&self, keys: &FxIndexSet<ChunkGroupKey>) -> Result<String> {
Ok(match self {
ChunkGroupKey::Entry(entries) => format!(
"Entry({:?})",
......@@ -605,7 +614,7 @@ impl ChunkGroupKey {
ChunkGroupKey::IsolatedMerged { parent, merge_tag } => {
format!(
"IsolatedMerged {{ parent: {}, merge_tag: {:?} }}",
Box::pin(keys.index(parent.0 as usize).ident_str(keys)).await?,
Box::pin(Self::merged_parent(keys, parent.0 as usize)?.ident_str(keys)).await?,
merge_tag
)
}
......@@ -623,7 +632,7 @@ impl ChunkGroupKey {
ChunkGroupKey::SharedMerged { parent, merge_tag } => {
format!(
"SharedMerged {{ parent: {}, merge_tag: {:?} }}",
Box::pin(keys.index(parent.0 as usize).ident_str(keys)).await?,
Box::pin(Self::merged_parent(keys, parent.0 as usize)?.ident_str(keys)).await?,
merge_tag
)
}
......@@ -635,12 +644,21 @@ impl ChunkGroupKey {
/// Delegates to [`Self::ident_str`], which already renders the chunk group unambiguously.
/// Debug output is free to diverge from it, but note that `ident_str` must stay stable because
/// asset idents are derived from it -- change this wrapper rather than `ident_str` itself.
pub async fn debug_str(
&self,
keys: &(impl std::ops::Index<usize, Output = Self> + ?Sized),
) -> Result<String> {
pub async fn debug_str(&self, keys: &FxIndexSet<ChunkGroupKey>) -> Result<String> {
self.ident_str(keys).await
}
/// The parent key of a merged chunk group. `parent` can be out of range when the merged key
/// comes from a different [`ChunkGroupInfo`] than `keys`, so this is an error, not a panic.
fn merged_parent(keys: &FxIndexSet<ChunkGroupKey>, parent: usize) -> Result<&ChunkGroupKey> {
keys.get_index(parent).with_context(|| {
format!(
"parent chunk group {parent} of a merged chunk group is not a known chunk group \
({} chunk groups)",
keys.len()
)
})
}
}
#[turbo_tasks::task_input]
......@@ -1017,12 +1035,9 @@ pub async fn compute_chunk_group_info(
let mut result = vec![];
result.push("Chunk Groups:".to_string());
for (i, (key, _)) in chunk_groups_map.iter().enumerate() {
result.push(format!(
" {:?}: {}",
i,
key.debug_str(&chunk_groups_map.keys()).await?
));
let keys = chunk_groups_map.keys().cloned().collect::<FxIndexSet<_>>();
for (i, key) in keys.iter().enumerate() {
result.push(format!(" {:?}: {}", i, key.debug_str(&keys).await?));
}
result.push("# Module buckets:".to_string());
for (key, modules) in buckets.iter() {
......
......@@ -4399,7 +4399,7 @@ pub mod tests {
{
let chunk_groups = async |availability: AvailabilityInfo| -> Result<Bitset> {
Ok(match availability.available_chunk_groups() {
Some(available) => available.await?.chunk_groups(module_graph).clone(),
Some(available) => available.await?.chunk_groups(module_graph)?.clone(),
None => Bitset::new(),
})
};
......@@ -4414,11 +4414,11 @@ pub mod tests {
}
}
let available = match result.available_chunk_groups() {
Some(available) => describe(available.await?.chunk_groups(module_graph)),
Some(available) => describe(available.await?.chunk_groups(module_graph)?),
None => vec![],
};
narrowed_input.push(match availability.available_chunk_groups() {
Some(available) => describe(available.await?.chunk_groups(module_graph)),
Some(available) => describe(available.await?.chunk_groups(module_graph)?),
None => vec![],
});
narrowed.push((availability, result, available));
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment