Unverified Commit 84112add authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

fix(turbopack): avoid panics from transient chunk group indices (#100928)

### What?

In `next dev` with Turbopack lazy dynamic imports
(`experimental.turbopackLazyDynamicImports` and/or
`turbopackLazyDynamicImportsSSR`), apps with nested `React.lazy(() =>
import(...))` client components log Rust panics while lazy imports
activate:

```
thread 'tokio-rt-worker' panicked at turbopack/crates/turbopack-core/src/module_graph/chunk_group_info.rs:111:51:
index out of bounds: the len is 31 but the index is 31
```

The dev server recovers and pages render, but the output is noisy and
the panics hide real errors. This PR turns that out-of-range lookup into
a contextual, transient error and adds regression coverage. It doesn't
touch #100705 or canonicalization (#100705 was reverted in #100898). The
panic happens on canary without it.

### Why?

`compute_chunk_group_info` creates the `module_chunk_groups` and
`chunk_group_indices` child cells with `ResolvedVc::cell(...)` while it
builds the result, and creates the outer `ChunkGroupInfo` cell, which
holds `chunk_groups` inline, at the end. Each cell is published when it
is written (`turbo-tasks-backend/src/backend/operation/update_cell.rs`).
So while lazy activation recomputes the graph, an eventually consistent
reader of `get_chunk_groups_for_module` (or `get_chunk_group`) can pair
the **old** inline `chunk_groups` with a **new** child cell. If the new
graph has more chunk groups, `self.chunk_groups[index]` is out of range.

I confirmed this with temporary instrumentation that wasn't committed,
on a fixture like the new test:
- 34 `compute_chunk_group_info` executions. Each was internally
consistent (max index = len − 1).
- 1147 reads, of which 59 paired cells from different executions. 3 were
out of range (the panics), and 56 had in-range indices.
- For all 26 affected modules, the last read was consistent and newer
than every torn read. The reader depends on both cells and is
recomputed, so strongly consistent consumers end up with the correct
result.

### How?

- `ChunkGroupInfo::chunk_group_at` replaces direct indexing in
`get_chunk_groups_for_module` and `get_chunk_group`.
- An index from a child cell that the inline `chunk_groups` doesn't have
becomes an `anyhow` error with context, never an empty or partial
result.
  - The dependencies and keyed reads are the same as before.
- **This is a guard, not a consistent snapshot.** Torn reads with
in-range indices are still possible in the middle of a recomputation,
and they are corrected by recomputation as before. Making them
impossible needs coherent publication or snapshot identity across the
cells, which is a separate design.

**Tests**
- `test/development/app-dir/lazy-dynamic-nested-imports`: nested
`React.lazy` imports with `x` reached from `a`, `c` and `y`, with both
lazy flags enabled.
  - Checks every level in the SSR HTML.
- Hydrates every level in the browser with `retry()`, and asserts that
no request returns ≥ 400 and that there are no transport errors. Browser
cancellations (`net::ERR_ABORTED`) are ignored; they happen without this
change too.
  - Asserts that the CLI output contains no `panicked` and no `⨯`.
- Without the fix it fails on the panic assertion (39 panics). With the
fix it passed 3 runs in a row.
- A deterministic Rust test
(`test_chunk_groups_for_module_with_mixed_generation_cells`) pairs a
real graph's membership cell with a truncated inline group list. It
panics before the fix and returns the error after it.

**Verification** (on canary `a14cbc44`)
- 12 dev sessions of a nested lazy fixture: 2 shapes × client lazy / SSR
lazy / both × 2 sessions. All clean: 0 panics, 0 `⨯`, SSR and hydration
OK. Before the fix almost every lazy session panicked.
- `test/development/app-dir/lazy-dynamic-imports` (19/19) and
`lazy-dynamic-chunk-list-collision` (1/1).
- `cargo nextest -p turbopack-core -p next-core`: 289 passed. `-p
turbopack-tests`: 458 passed, no snapshot changes. fmt, check and clippy
pass with `-D warnings`.
- Production output is unchanged: for the fixture's 25 client files,
path and SHA-256 are identical before and after.

**Not fixed here**
- A build with extra logging `await`s still hit an SSR `ChunkLoadError`
for a lazy proxy chunk in 1 of 3 sessions, with no panic. That is a
separate issue that I'm tracking separately.

<!-- NEXT_JS_LLM -->


<!-- fleet 4f67c6f7-bbe2-454b-9a17-822a60ef8f7e -->

---------
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: 's avatarLuke Sandberg <210140+lukesandberg@users.noreply.github.com>
parent 2a1f6afa
import type { ReactNode } from 'react'
export default function Root({ children }: { children: ReactNode }) {
return (
<html>
<body>{children}</body>
</html>
)
}
'use client'
import { lazy, Suspense } from 'react'
const A = lazy(() => import('../../lib/a'))
const B = lazy(() => import('../../lib/b'))
const C = lazy(() => import('../../lib/c'))
export default function Page() {
return (
<main>
<p id="page">page-0</p>
<Suspense fallback={null}>
<A />
<B />
<C />
</Suspense>
</main>
)
}
'use client'
import { lazy, Suspense } from 'react'
const A = lazy(() => import('../../lib/a'))
const B = lazy(() => import('../../lib/b'))
const C = lazy(() => import('../../lib/c'))
export default function Page() {
return (
<main>
<p id="page">page-1</p>
<Suspense fallback={null}>
<A />
<B />
<C />
</Suspense>
</main>
)
}
'use client'
import { lazy, Suspense } from 'react'
const A = lazy(() => import('../../lib/a'))
const B = lazy(() => import('../../lib/b'))
const C = lazy(() => import('../../lib/c'))
export default function Page() {
return (
<main>
<p id="page">page-2</p>
<Suspense fallback={null}>
<A />
<B />
<C />
</Suspense>
</main>
)
}
'use client'
import { lazy, Suspense } from 'react'
const A = lazy(() => import('../../lib/a'))
const B = lazy(() => import('../../lib/b'))
const C = lazy(() => import('../../lib/c'))
export default function Page() {
return (
<main>
<p id="page">page-3</p>
<Suspense fallback={null}>
<A />
<B />
<C />
</Suspense>
</main>
)
}
'use client'
import { lazy, Suspense } from 'react'
const A = lazy(() => import('../../lib/a'))
const B = lazy(() => import('../../lib/b'))
const C = lazy(() => import('../../lib/c'))
export default function Page() {
return (
<main>
<p id="page">page-4</p>
<Suspense fallback={null}>
<A />
<B />
<C />
</Suspense>
</main>
)
}
import { nextTestSetup } from 'e2e-utils'
import { retry } from 'next-test-utils'
// Every page lazily imports a, b and c. a and c lazily import x, b lazily imports y, which lazily
// imports x, and x lazily imports x2. So x is reached from several nested parents on every page.
const LEVELS = ['a', 'b', 'c', 'y', 'x', 'x2'].map((level) => `level-${level}`)
const PAGES = [0, 1, 2, 3, 4].map((i) => `/page-${i}`)
;(process.env.IS_TURBOPACK_TEST ? describe : describe.skip)(
'lazy-dynamic-nested-imports',
() => {
const { next } = nextTestSetup({
files: __dirname,
})
it('server-renders every nested level', async () => {
for (const page of PAGES) {
const $ = await next.render$(page)
for (const level of LEVELS) {
const texts = $(`.${level}-text`)
.toArray()
.map((element) => $(element).text())
expect(texts.length).toBeGreaterThan(0)
expect(texts.every((text) => text === level)).toBe(true)
}
}
})
it('hydrates every nested level without failed requests', async () => {
for (const page of PAGES) {
const failed: string[] = []
const browser = await next.browser(page, {
beforePageLoad(p) {
p.on('response', (response) => {
if (response.status() >= 400) {
failed.push(`${response.status()} ${response.url()}`)
}
})
p.on('requestfailed', (request) => {
const errorText = request.failure()?.errorText ?? 'request failed'
// Some chunk requests are aborted by the browser while lazy imports activate, also
// without this fix; a cancelled request is not a failed one.
if (errorText !== 'net::ERR_ABORTED') {
failed.push(`${errorText} ${request.url()}`)
}
})
},
})
// Activating the nested lazy imports can rebuild and reload the page several times, which
// resets every button, so allow more than the default 3s for it to settle.
await retry(async () => {
// Buttons only react once their level has hydrated; click the ones that haven't yet.
const labels: string[] = await browser.eval(`(() => {
const buttons = [...document.querySelectorAll('button.level-button')]
for (const button of buttons) {
if (!button.textContent.endsWith('-clicked')) button.click()
}
return buttons.map((button) => button.textContent)
})()`)
for (const level of LEVELS) {
expect(labels).toContain(`${level}-clicked`)
}
expect(labels.every((label) => label.endsWith('-clicked'))).toBe(true)
}, 30_000)
expect(failed).toEqual([])
await browser.close()
}
})
it('does not panic or log errors while activating nested lazy imports', () => {
expect(next.cliOutput).not.toContain('panicked')
expect(next.cliOutput).not.toContain('⨯')
})
}
)
'use client'
import { lazy, Suspense } from 'react'
import { Level } from './level'
const X = lazy(() => import('./x'))
export default function A() {
return (
<Level name="level-a">
<Suspense fallback={null}>
<X />
</Suspense>
</Level>
)
}
'use client'
import { lazy, Suspense } from 'react'
import { Level } from './level'
const Y = lazy(() => import('./y'))
export default function B() {
return (
<Level name="level-b">
<Suspense fallback={null}>
<Y />
</Suspense>
</Level>
)
}
'use client'
import { lazy, Suspense } from 'react'
import { Level } from './level'
const X = lazy(() => import('./x'))
export default function C() {
return (
<Level name="level-c">
<Suspense fallback={null}>
<X />
</Suspense>
</Level>
)
}
'use client'
import { useState, type ReactNode } from 'react'
// A nested level: renders its name, and a button that only reacts after hydration. A level can be
// rendered more than once on a page (x is reached through a, c and y), so it is identified by class.
export function Level({
name,
children,
}: {
name: string
children?: ReactNode
}) {
const [clicked, setClicked] = useState(false)
return (
<div>
<p className={`${name}-text`}>{name}</p>
<button
className={`level-button ${name}-button`}
onClick={() => setClicked(true)}
>
{clicked ? `${name}-clicked` : name}
</button>
{children}
</div>
)
}
'use client'
import { lazy, Suspense } from 'react'
import { Level } from './level'
const X2 = lazy(() => import('./x2'))
export default function X() {
return (
<Level name="level-x">
<Suspense fallback={null}>
<X2 />
</Suspense>
</Level>
)
}
'use client'
import { Level } from './level'
export default function X2() {
return <Level name="level-x2" />
}
'use client'
import { lazy, Suspense } from 'react'
import { Level } from './level'
const X = lazy(() => import('./x'))
export default function Y() {
return (
<Level name="level-y">
<Suspense fallback={null}>
<X />
</Suspense>
</Level>
)
}
import type { NextConfig } from 'next'
const nextConfig: NextConfig = {
experimental: {
turbopackLazyDynamicImports: true,
turbopackLazyDynamicImportsSSR: true,
},
}
export default nextConfig
...@@ -109,11 +109,13 @@ impl ChunkGroupInfo { ...@@ -109,11 +109,13 @@ impl ChunkGroupInfo {
Ok(Vc::cell( Ok(Vc::cell(
groups groups
.iter() .iter()
.map(|index| ChunkGroupWithIndex { .map(|index| {
index, Ok(ChunkGroupWithIndex {
chunk_group: self.chunk_groups[index as usize].clone(), index,
chunk_group: self.chunk_group_at(index as usize)?.clone(),
})
}) })
.collect(), .collect::<Result<_>>()?,
)) ))
} }
...@@ -144,6 +146,24 @@ impl ChunkGroupInfo { ...@@ -144,6 +146,24 @@ impl ChunkGroupInfo {
} }
impl ChunkGroupInfo { impl ChunkGroupInfo {
/// Returns the chunk group at `index`, for an index read from one of the child cells
/// ([`ChunkGroupInfo::module_chunk_groups`], [`ChunkGroupInfo::chunk_group_indices`], ...).
///
/// [`compute_chunk_group_info`] creates those child cells before it creates this
/// `ChunkGroupInfo` cell (with the inline [`ChunkGroupInfo::chunk_groups`]) at the end of the
/// computation. So while it recomputes, an eventually consistent reader can pair this
/// `ChunkGroupInfo` with a child cell from the newer execution, and the index can be out of
/// range. The reader depends on both and is recomputed once both are updated, so an error here
/// is transient, but indexing would panic.
fn chunk_group_at(&self, index: usize) -> Result<&ChunkGroup> {
self.chunk_groups.get_index(index).with_context(|| {
format!(
"chunk group index {index} is out of range ({} chunk groups)",
self.chunk_groups.len()
)
})
}
/// Returns the index of the chunk group identified by `key` in /// Returns the index of the chunk group identified by `key` in
/// [`ChunkGroupInfo::chunk_groups`]. /// [`ChunkGroupInfo::chunk_groups`].
/// ///
...@@ -181,7 +201,7 @@ impl ChunkGroupInfo { ...@@ -181,7 +201,7 @@ impl ChunkGroupInfo {
/// the entries of a merged group) always come from the module graph rather than the caller. /// the entries of a merged group) always come from the module graph rather than the caller.
pub async fn get_chunk_group(self: Vc<Self>, key: ChunkGroupKey) -> Result<ChunkGroup> { pub async fn get_chunk_group(self: Vc<Self>, key: ChunkGroupKey) -> Result<ChunkGroup> {
let index = self.get_index_of(key).await?; let index = self.get_index_of(key).await?;
Ok(self.await?.chunk_groups[index].clone()) Ok(self.await?.chunk_group_at(index)?.clone())
} }
} }
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment