Unverified Commit d3118aba authored by vercel[bot]'s avatar vercel[bot] Committed by GitHub

Reap crashed Turbopack plugin workers instead of hanging forever

When a worker-thread plugin runtime dies unexpectedly (uncaught exception,
process.exit, OOM), nothing told the Rust worker pool about it: the dead
Worker stayed in loaderWorkers, the operation waiting for its response
hung forever, and every later evaluation could be handed the dead worker,
wedging the dev server's compilation permanently. This is the crash-time
sibling of #96592, which covered evaluation and IPC errors reported over
the channel.

JavaScript now attaches 'error'/'exit' listeners to pooled workers and
reports any death while the worker is still registered (intentional
terminations remove the worker from the map first, so they are never
reported; the thread id is captured at creation since it resets to -1 by
the time 'exit' fires). On the Rust side, a dead-worker set records
reaped ids and handle_worker_death classifies the death:

- busy: closes the in-flight task's channel (via a watch channel, which
  cannot lose the wakeup), so its pending recv fails and surfaces through
  the existing subprocess-crash path in pull_operation - synthesized
  issue, disallow_reuse, fresh worker on the next evaluation;
- idle: removed from the idle list and pool statistics;
- created but unassigned (in transit to an acquirer or routed to a
  waiter): total worker count decremented;
- boot failure: the pending creation for the same pool is failed (so
  create_worker errors instead of waiting forever) and the booting
  statistics are balanced - no phantom workers after a boot crash.

Every acquisition and return path checks the dead set so a reaped id is
never handed out again: acquire_worker retries after dead candidates
(idle pop, waiter handoff, fresh boot), and WorkerOperation's drop skips
returning a worker that died while checked out.

Verified end to end with turbopackPluginRuntimeStrategy: 'workerThreads':

- before: a loader calling process.exit(1) hangs the request forever and
  wedges all later compilations;
- after: the request fails in ~1s with "Node.js subprocess crashed while
  evaluating loaders", and the next request is served by a fresh worker
  in ~50ms. A worker that dies between tasks (idle) is reaped the same
  way.

The new development test covers the crash path: baseline (stock binary)
fails by hanging; candidate fails the evaluation and recovers. The #96592
error-cleanup test still passes. Autoreview panel findings addressed:
watch-channel close, dead-set validation on all handoff paths,
options-tagged pending creations, clean-exit reaping.

Validation: pnpm build-all (TS + native), cargo fmt --check, cargo clippy
-p turbopack-node --all-targets -D warnings, cargo test -p turbopack-node
--lib, candidate integration test pass, exact baseline binary fails by
hanging.
Co-Authored-By: 's avatarMarcos Hernanz <96699542+marcoshernanz@users.noreply.github.com>
parent 865d623a
...@@ -39,7 +39,8 @@ export declare function registerWorkerScheduler( ...@@ -39,7 +39,8 @@ export declare function registerWorkerScheduler(
creator: (arg: NapiWorkerCreation) => any, creator: (arg: NapiWorkerCreation) => any,
terminator: (arg: NapiWorkerTermination) => any terminator: (arg: NapiWorkerTermination) => any
): void ): void
export declare function workerCreated(workerId: number): void export declare function workerCreated(creation: NapiWorkerTermination): void
export declare function workerDied(termination: NapiWorkerTermination): void
export interface NapiWorkerCreation { export interface NapiWorkerCreation {
options: NapiWorkerOptions options: NapiWorkerOptions
} }
......
...@@ -22,6 +22,10 @@ export async function runLoaderWorkerPool( ...@@ -22,6 +22,10 @@ export async function runLoaderWorkerPool(
workerData: { workerData: {
bindingPath, bindingPath,
cwd, cwd,
// The worker reports these exact options when it has booted:
// process.argv is inherited from the parent and cannot be used to
// identify the entrypoint.
filename,
}, },
}) })
...@@ -31,7 +35,30 @@ export async function runLoaderWorkerPool( ...@@ -31,7 +35,30 @@ export async function runLoaderWorkerPool(
const workers = const workers =
loaderWorkers[poolId] || (loaderWorkers[poolId] = new Map()) loaderWorkers[poolId] || (loaderWorkers[poolId] = new Map())
workers.set(worker.threadId, worker) // The thread id must be captured up-front: by the time the 'error' and
// 'exit' events fire, `worker.threadId` has already reset to -1.
const threadId = worker.threadId
workers.set(threadId, worker)
const reportWorkerDeath = () => {
// Intentional terminations remove the worker from the map before
// terminating it, so a worker that is no longer registered died on
// purpose and must not be reported again. Any exit while still
// registered is unexpected, including a clean `process.exit(0)`.
// `workerDied` may also be absent when the loaded native binding
// predates it; in that case there is nothing to notify.
if (!workers.has(threadId) || typeof bindings.workerDied !== 'function') {
return
}
workers.delete(threadId)
bindings.workerDied({
options: { filename, cwd },
workerId: threadId,
})
}
worker.once('error', reportWorkerDeath)
worker.once('exit', reportWorkerDeath)
}, },
(termination) => { (termination) => {
const { const {
......
import { ReactNode } from 'react'
export default function Root({ children }: { children: ReactNode }) {
return (
<html>
<body>{children}</body>
</html>
)
}
const data = require('../input.crashprobe')
export default function Page() {
return <p>{data.default}</p>
}
module.exports = function crashLoader(source) {
const value = source.trim()
// Kills the worker before the evaluation result is reported back.
if (value === 'exit-during-eval') {
process.exit(1)
}
// Lets the evaluation succeed, then kills the worker asynchronously.
if (value === 'exit-after-eval') {
setTimeout(() => process.exit(1), 250)
return `export default "exit-after-eval"`
}
return `export default ${JSON.stringify(value)}`
}
/** @type {import('next').NextConfig} */
module.exports = {
experimental: {
turbopackPluginRuntimeStrategy: 'workerThreads',
},
turbopack: {
rules: {
'*.crashprobe': {
as: '*.js',
loaders: [
{
loader: require.resolve('./crash-loader.js'),
},
],
},
},
},
}
import { nextTestSetup } from 'e2e-utils'
import { retry, waitFor } from 'next-test-utils'
describe('turbopack worker thread crash recovery', () => {
const { next, isTurbopack } = nextTestSetup({
files: __dirname,
})
const itOnlyTurbopack = isTurbopack ? it : it.skip
itOnlyTurbopack(
'fails the evaluation instead of hanging when a loader worker dies, and recovers',
async () => {
expect((await next.render$('/'))('p').text()).toBe('initial')
// The loader calls process.exit(1), killing the worker thread before it
// reports the evaluation result. Without worker-death reaping this
// request would hang forever.
await next.patchFile('input.crashprobe', 'exit-during-eval')
await waitFor(1500)
// Trigger the evaluation. It fails the compile rather than hanging.
await next.fetch('/')
await retry(
async () => {
expect(next.cliOutput).toContain(
'Node.js subprocess crashed while evaluating'
)
},
30_000,
500
)
// The pool must hand out a fresh worker after the crash: a new
// evaluation with a working probe succeeds.
await next.patchFile('input.crashprobe', 'recovered')
await retry(async () => {
expect((await next.render$('/'))('p').text()).toBe('recovered')
})
},
60_000
)
})
...@@ -11,7 +11,10 @@ const binding: Binding = require( ...@@ -11,7 +11,10 @@ const binding: Binding = require(
/* turbopackIgnore: true */ workerData.bindingPath /* turbopackIgnore: true */ workerData.bindingPath
) )
binding.workerCreated(workerId) binding.workerCreated({
options: { filename: workerData.filename, cwd: workerData.cwd },
workerId,
})
export const run = async ( export const run = async (
moduleFactory: () => Promise<{ moduleFactory: () => Promise<{
......
...@@ -11,7 +11,10 @@ export interface TaskMessage { ...@@ -11,7 +11,10 @@ export interface TaskMessage {
export interface Binding { export interface Binding {
recvTaskMessageInWorker(workerId: number): Promise<TaskMessage> recvTaskMessageInWorker(workerId: number): Promise<TaskMessage>
sendTaskMessage(msg: TaskMessage): void sendTaskMessage(msg: TaskMessage): void
workerCreated(workerId: number): void workerCreated(creation: {
options: { filename: string; cwd: string }
workerId: number
}): void
} }
// Export this, maybe in the future, we can add an implementation via web worker on browser // Export this, maybe in the future, we can add an implementation via web worker on browser
......
...@@ -44,6 +44,15 @@ impl NodeJsPoolStats { ...@@ -44,6 +44,15 @@ impl NodeJsPoolStats {
self.booting_workers = self.booting_workers.saturating_sub(1); self.booting_workers = self.booting_workers.saturating_sub(1);
} }
/// A worker that was being booted died before it could be used: both the
/// booting counter and the total worker counter have to be decremented or
/// the pool statistics leak a phantom worker per boot crash.
#[allow(unused)]
pub fn failed_booting_worker(&mut self) {
self.booting_workers = self.booting_workers.saturating_sub(1);
self.workers = self.workers.saturating_sub(1);
}
pub fn remove_worker(&mut self) { pub fn remove_worker(&mut self) {
self.workers = self.workers.saturating_sub(1); self.workers = self.workers.saturating_sub(1);
} }
......
...@@ -15,7 +15,7 @@ use turbo_rcstr::RcStr; ...@@ -15,7 +15,7 @@ use turbo_rcstr::RcStr;
use crate::worker_pool::{ use crate::worker_pool::{
WorkerOptions, WorkerOptions,
operation::{TaskMessage, WORKER_POOL_OPERATION}, operation::{TaskMessage, WORKER_POOL_OPERATION, WorkerDeath},
}; };
static WORKER_CREATOR: OnceLock<ThreadsafeFunction<NapiWorkerCreation, ErrorStrategy::Fatal>> = static WORKER_CREATOR: OnceLock<ThreadsafeFunction<NapiWorkerCreation, ErrorStrategy::Fatal>> =
...@@ -25,7 +25,11 @@ static WORKER_TERMINATOR: OnceLock< ...@@ -25,7 +25,11 @@ static WORKER_TERMINATOR: OnceLock<
ThreadsafeFunction<NapiWorkerTermination, ErrorStrategy::Fatal>, ThreadsafeFunction<NapiWorkerTermination, ErrorStrategy::Fatal>,
> = OnceLock::new(); > = OnceLock::new();
static PENDING_CREATIONS: OnceLock<Mutex<VecDeque<oneshot::Sender<u32>>>> = OnceLock::new(); /// Creations are tagged with their pool options: boot completion (and boot
/// failure) must be paired with the creation for the same pool, not just the
/// oldest one globally.
static PENDING_CREATIONS: OnceLock<Mutex<VecDeque<(Arc<WorkerOptions>, oneshot::Sender<u32>)>>> =
OnceLock::new();
// Allow dead_code for test builds where napi exports are not entry points // Allow dead_code for test builds where napi exports are not entry points
#[allow(dead_code)] #[allow(dead_code)]
...@@ -71,7 +75,7 @@ pub async fn create_worker(options: Arc<WorkerOptions>) -> anyhow::Result<u32> { ...@@ -71,7 +75,7 @@ pub async fn create_worker(options: Arc<WorkerOptions>) -> anyhow::Result<u32> {
.lock() .lock()
.entry(options.clone()) .entry(options.clone())
.or_default(); .or_default();
pending.lock().push_back(tx); pending.lock().push_back((options.clone(), tx));
} }
if let Some(creator) = WORKER_CREATOR.get() { if let Some(creator) = WORKER_CREATOR.get() {
...@@ -86,17 +90,51 @@ pub async fn create_worker(options: Arc<WorkerOptions>) -> anyhow::Result<u32> { ...@@ -86,17 +90,51 @@ pub async fn create_worker(options: Arc<WorkerOptions>) -> anyhow::Result<u32> {
} }
let worker_id = rx.await?; let worker_id = rx.await?;
// The boot completed: balance the booting counter here (not in the
// caller) so that every later death classification sees a uniform state
// for created workers. A boot that fails never reaches this point; the
// death path balances it with the matching pending creation instead.
if let Some(state) = WORKER_POOL_OPERATION
.pools
.lock()
.get(options.as_ref())
.cloned()
{
state.stats.lock().finished_booting_worker();
}
Ok(worker_id) Ok(worker_id)
} }
/// Called from a worker once it finished booting. The worker reports the
/// options it was created with (its entry filename and cwd) so the creation
/// can be paired with the pending request for the same pool instead of
/// blindly taking the oldest slot of any pool.
// Allow dead_code for test builds where napi exports are not entry points // Allow dead_code for test builds where napi exports are not entry points
#[allow(dead_code)] #[allow(dead_code)]
#[napi] #[napi]
pub fn worker_created(worker_id: u32) { pub fn worker_created(creation: NapiWorkerTermination) {
if let Some(pending) = PENDING_CREATIONS.get() let NapiWorkerTermination { options, worker_id } = creation;
&& let Some(tx) = pending.lock().pop_front() WORKER_POOL_OPERATION.mark_worker_created(worker_id);
{ if let Some(pending) = PENDING_CREATIONS.get() {
let _ = tx.send(worker_id); let tx = {
let mut pending = pending.lock();
let position = pending.iter().position(|(pending_options, _)| {
pending_options.filename == options.filename && pending_options.cwd == options.cwd
});
// Fall back to the oldest slot if the reported options do not
// match any pending creation exactly (e.g. path normalization
// differences): a mispaired worker is interchangeable, a pending
// creation that is never completed hangs.
position
.or(if pending.is_empty() { None } else { Some(0) })
.and_then(|position| pending.remove(position))
.map(|(_, tx)| tx)
};
if let Some(tx) = tx {
let _ = tx.send(worker_id);
}
} }
} }
...@@ -142,6 +180,56 @@ pub struct NapiWorkerTermination { ...@@ -142,6 +180,56 @@ pub struct NapiWorkerTermination {
pub worker_id: u32, pub worker_id: u32,
} }
/// Called from JavaScript when a pooled worker died unexpectedly (uncaught
/// exception, `process.exit`, OOM). Intentional terminations do not reach
/// this: the terminator removes the worker from the JavaScript-side map
/// before terminating it, so the death is not reported.
// Allow dead_code for test builds where napi exports are not entry points
#[allow(dead_code)]
#[napi]
pub fn worker_died(termination: NapiWorkerTermination) {
let NapiWorkerTermination { options, worker_id } = termination;
let worker_options = Arc::new(WorkerOptions {
filename: options.filename.clone(),
cwd: options.cwd.clone(),
});
match WORKER_POOL_OPERATION.handle_worker_death(worker_options.clone(), worker_id) {
WorkerDeath::BootFailed => {
// The worker never reported itself as created, so it died while
// booting. Fail the pending creation *for the same pool* (failing
// an unrelated pool's creation would both hang this one and leak
// the other): dropping the sender errors the receiver, and the
// pool's booting statistics are balanced so no phantom worker
// remains.
if let Some(pending) = PENDING_CREATIONS.get() {
let failed = {
let mut pending = pending.lock();
pending
.iter()
.position(|(pending_options, _)| {
pending_options.filename == options.filename
&& pending_options.cwd == options.cwd
})
.and_then(|position| pending.remove(position))
};
if let Some((failed_options, tx)) = failed {
drop(tx);
if let Some(state) = WORKER_POOL_OPERATION
.pools
.lock()
.get(failed_options.as_ref())
.cloned()
{
state.stats.lock().failed_booting_worker();
}
}
}
}
WorkerDeath::Busy | WorkerDeath::Idle | WorkerDeath::CreatedUnassigned => {}
}
}
// Allow dead_code for test builds where napi exports are not entry points // Allow dead_code for test builds where napi exports are not entry points
#[allow(dead_code)] #[allow(dead_code)]
#[napi(object)] #[napi(object)]
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment