Unverified Commit 7d525155 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

turbopack: report an actionable error when Node.js child processes cannot connect (#99887)

## What

Improve the error message reported when a Node.js worker process fails
to connect to turbopack. Use a dedicated exit code to signal this.

## Why

Tools like codex like to launch next in a sandbox with limited
permissions. We cannot really support this but we can report more
actionable error messages. The current error is awful:

```
FATAL: An unexpected Turbopack error occurred:
Failed to write app endpoint /page

Caused by:
- [project]/test/tmp/next-test-1791478247265-510/app/globals.css [app-client] (css)
- creating new process
- node process exited before we could connect to it with exit status: 0

Debug info:
- Execution of get_all_written_entrypoints_with_issues_operation failed
- Execution of EntrypointsOperation::new failed
- Execution of all_entrypoints_write_to_disk_operation failed
- Execution of output_assets_operation failed
- Execution of <AppEndpoint as Endpoint>::output failed
- Failed to write app endpoint /page
- Execution of AppEndpoint::output failed
- Execution of whole_app_module_graph_operation failed
- Execution of *Project::get_all_additional_entries failed
- Execution of ModuleGraph::from_graphs failed
- Execution of ModuleGraph::from_graphs_inner failed
- Execution of SingleModuleGraph::new_with_entries failed
- [project]/test/tmp/next-test-1791478247265-510/app/globals.css [app-client] (css)
- Execution of primary_chunkable_referenced_modules failed
- Execution of <CssModule as Module>::references failed
- Execution of parse_css failed
- Execution of <PostCssTransformedAsset as Asset>::content failed
- Execution of PostCssTransformedAsset::process failed
- Execution of evaluate_webpack_loader failed
- creating new process
- node process exited before we could connect to it with exit status: 0
  Process output:

  Process error output:

> Build error occurred
Error [TurbopackInternalError]: Failed to write app endpoint /page

Caused by:
- [project]/test/tmp/next-test-1791478247265-510/app/globals.css [app-client] (css)
- creating new process
- node process exited before we could connect to it with exit status: 0

Debug info:
- Execution of get_all_written_entrypoints_with_issues_operation failed
- Execution of EntrypointsOperation::new failed
- Execution of all_entrypoints_write_to_disk_operation failed
- Execution of output_assets_operation failed
- Execution of <AppEndpoint as Endpoint>::output failed
- Failed to write app endpoint /page
- Execution of AppEndpoint::output failed
- Execution of whole_app_module_graph_operation failed
- Execution of *Project::get_all_additional_entries failed
- Execution of ModuleGraph::from_graphs failed
- Execution of ModuleGraph::from_graphs_inner failed
- Execution of SingleModuleGraph::new_with_entries failed
- [project]/test/tmp/next-test-1791478247265-510/app/globals.css [app-client] (css)
- Execution of primary_chunkable_referenced_modules failed
- Execution of <CssModule as Module>::references failed
- Execution of parse_css failed
- Execution of <PostCssTransformedAsset as Asset>::content failed
- Execution of PostCssTransformedAsset::process failed
- Execution of evaluate_webpack_loader failed
- creating new process
- node process exited before we could connect to it with exit status: 0
  Process output:

  Process error output:
    at <unknown> (TurbopackInternalError: Failed to write app endpoint /page)
    at throwTurbopackInternalError (../../../packages/next/dist/shared/lib/turbopack/internal-error.js:38:17) {
  location: undefined
}
destroyed next instance: 7.653ms
```

Now we report a more useful message that has some suggestions
```
./app/globals.css
Error: Node.js worker processes could not connect to Turbopack
Turbopack is configured to run webpack loaders and PostCSS transforms in child processes that communicate with it over a loopback TCP connection, but the child process was unable to connect while evaluating loaders [postcss]:
connect ENETUNREACH 127.0.0.1
This is likely caused by a sandbox or firewall that blocks connections. Allow local connections, or run this code using worker threads by setting `experimental.turbopackPluginRuntimeStrategy: 'workerThreads'` in your Next.js config.

https://nextjs.org/docs/app/api-reference/config/next-config-js/turbopackPluginRuntimeStrategy
```


<!-- NEXT_JS_LLM -->

---------
Co-authored-by: 's avatarvercel-gh-bot-3[bot] <282332853+vercel-gh-bot-3[bot]@users.noreply.github.com>
parent a6906597
import './globals.css'
export default function RootLayout({ children }) {
return (
<html>
<body>{children}</body>
</html>
)
}
export default function Page() {
return <p className="hello">hello world</p>
}
// Preloaded into every Node.js process of the build (via NODE_OPTIONS) to
// emulate an environment where binding/connecting to a local port is denied,
// e.g. a sandbox with no usable loopback interface.
//
// Binding/listening keeps working (that is what Turbopack's Rust side does),
// only outgoing connections to the loopback interface fail with ENETUNREACH,
// exactly like `connect(127.0.0.1)` inside a network namespace whose loopback
// device is down.
const net = require('node:net')
const originalConnect = net.Socket.prototype.connect
net.Socket.prototype.connect = function connect(...args) {
let host
if (typeof args[0] === 'object' && args[0] !== null) {
host = args[0].host
} else if (typeof args[1] === 'string') {
host = args[1]
}
const isLocal =
host === undefined ||
host === 'localhost' ||
host === '127.0.0.1' ||
host === '::1'
if (!isLocal) {
return originalConnect.apply(this, args)
}
process.nextTick(() => {
const error = new Error(`connect ENETUNREACH ${host ?? '127.0.0.1'}`)
error.code = 'ENETUNREACH'
this.destroy(error)
})
return this
}
module.exports = () => ({ postcssPlugin: 'noop-postcss-plugin' })
module.exports.postcss = true
// A PostCSS config is required to make Turbopack run the CSS through its
// Node.js side process (`PostCssTransformedAsset`), which is the code path
// that fails when the side process cannot use a local port.
module.exports = { plugins: [require('./noop-postcss-plugin')] }
import path from 'path'
import { nextTestSetup } from 'e2e-utils'
import { retry } from 'next-test-utils'
// Turbopack runs PostCSS in a Node.js side process that talks to the Rust
// process over a local TCP socket. When connecting to that local port is
// denied (e.g. a sandbox without a usable loopback interface), the build should
// fail with an actionable diagnostic rather than an internal compiler error.
// @force-gate turbopack
describe('turbopack css node process connect failure', () => {
const { next, isNextDev } = nextTestSetup({
files: path.join(__dirname, 'fixture'),
skipStart: true,
skipDeployment: true,
})
it('reports an actionable error when the CSS side process cannot use a local port', async () => {
const env = {
NODE_OPTIONS: `--require "${path.join(
next.testDir,
'deny-local-port-connect.js'
)}"`,
}
let cliOutput: string
if (isNextDev) {
await next.start({ env })
// Compiling the page runs PostCSS on its global CSS.
await next.fetch('/')
await retry(async () => {
expect(next.cliOutput).toContain(
'Node.js worker processes could not connect to Turbopack'
)
})
cliOutput = next.cliOutput
} else {
const result = await next.build({ env })
expect(result.exitCode).toBe(1)
cliOutput = result.cliOutput
}
expect(cliOutput).toContain(
'Node.js worker processes could not connect to Turbopack'
)
// The issue is attributed to the file being transformed, and says which
// transform was running.
expect(cliOutput).toContain('./app/globals.css')
expect(cliOutput).toContain('while evaluating loaders [postcss]')
// The underlying reason reported by the child process.
expect(cliOutput).toContain('connect ENETUNREACH 127.0.0.1')
// The suggested workaround.
expect(cliOutput).toContain(
"experimental.turbopackPluginRuntimeStrategy: 'workerThreads'"
)
expect(cliOutput).toContain(
'https://nextjs.org/docs/app/api-reference/config/next-config-js/turbopackPluginRuntimeStrategy'
)
// This is not a Turbopack bug, so don't present it as one.
expect(cliOutput).not.toContain('TurbopackInternalError')
expect(cliOutput).not.toContain('https://bugs.nextjs.org/')
})
})
import { writeSync } from 'node:fs'
import { createConnection } from 'node:net'
import { Writable } from 'node:stream'
import { structuredError } from '../error'
......@@ -18,6 +19,15 @@ export type Ipc<TIncoming, TOutgoing> = {
sendReady(): Promise<void>
}
/**
* Exit code used when we cannot connect to the parent process, so that it can
* report an actionable error. Node.js reserves most small exit codes, so this
* uses `EX_UNAVAILABLE` from sysexits.h.
*
* Keep in sync with `CONNECT_FAILED_EXIT_CODE` in `process_pool/mod.rs`.
*/
const CONNECT_FAILED_EXIT_CODE = 69
function createIpc<TIncoming, TOutgoing>(
port: number
): Ipc<TIncoming, TOutgoing> {
......@@ -26,6 +36,16 @@ function createIpc<TIncoming, TOutgoing>(
host: '127.0.0.1',
})
// If we never connect there is nobody to report errors to, so exit right
// away with a dedicated exit code. This must be synchronous: other handlers
// (the 'close' handler below, or `sendError` after failed writes) would
// otherwise race us and exit with a different code.
const onConnectError = (err: Error) => {
writeSync(process.stderr.fd, `${err.message}\n`)
process.exit(CONNECT_FAILED_EXIT_CODE)
}
socket.once('error', onConnectError)
/**
* A writable stream that writes to the socket.
* We don't write directly to the socket because we need to
......@@ -60,6 +80,7 @@ function createIpc<TIncoming, TOutgoing>(
let state: State = { type: 'waiting' }
let buffer: Buffer = Buffer.alloc(0)
socket.once('connect', () => {
socket.off('error', onConnectError)
socket.setNoDelay(true)
socket.on('data', (chunk) => {
buffer = Buffer.concat([buffer, chunk])
......
......@@ -22,7 +22,7 @@ use turbopack_core::{
context::AssetContext,
file_source::FileSource,
ident::AssetIdent,
issue::{Issue, IssueExt, IssueSource, IssueStage, StyledString},
issue::{Issue, IssueExt, IssueSeverity, IssueSource, IssueStage, StyledString},
module::Module,
module_graph::{
GraphEntries, ModuleGraph,
......@@ -297,6 +297,37 @@ async fn common_node_env(env: Vc<Box<dyn ProcessEnv>>) -> Result<Vc<EnvMap>> {
Ok(Vc::cell(filtered))
}
/// A Node.js process could not connect back to us over the loopback interface.
///
/// Backends return this from [`EvaluateOperation::operation`] so that it can be
/// reported as an actionable issue. This is not a bug in the evaluated code and
/// retrying will not help, it usually means a sandbox blocks local networking.
#[derive(Debug)]
pub struct NodeJsConnectError {
/// The error output of the child process, which contains the reason the
/// connection failed.
pub stderr: String,
}
impl std::fmt::Display for NodeJsConnectError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(
f,
"The Node.js child process could not connect to the parent process\n{}",
self.stderr
)
}
}
impl std::error::Error for NodeJsConnectError {}
impl NodeJsConnectError {
/// Finds a [`NodeJsConnectError`] anywhere in the cause chain of `err`.
pub fn find(err: &anyhow::Error) -> Option<&Self> {
err.chain().find_map(|cause| cause.downcast_ref::<Self>())
}
}
struct PoolErrorHandler;
/// Number of attempts before we start slowing down the retry.
......@@ -308,7 +339,8 @@ impl futures_retry::ErrorHandler<anyhow::Error> for PoolErrorHandler {
type OutError = anyhow::Error;
fn handle(&mut self, attempt: usize, err: anyhow::Error) -> RetryPolicy<Self::OutError> {
if attempt >= MAX_ATTEMPTS {
// A process that cannot connect to us will not be able to on a retry either.
if attempt >= MAX_ATTEMPTS || NodeJsConnectError::find(&err).is_some() {
RetryPolicy::ForwardError(err)
} else if attempt >= MAX_FAST_ATTEMPTS {
RetryPolicy::WaitRetry(Duration::from_secs(1))
......@@ -330,6 +362,8 @@ pub trait EvaluateContext {
}
fn args(&self) -> &[ResolvedVc<JsonValue>];
fn cwd(&self) -> Vc<FileSystemPath>;
/// The source that issues about this evaluation are attributed to.
fn context_source_for_issue(&self) -> ResolvedVc<Box<dyn Source>>;
fn emit_error(
&self,
error: StructuredError,
......@@ -381,7 +415,7 @@ pub async fn custom_evaluate(evaluate_context: impl EvaluateContext) -> Result<V
// worker. So we retry picking workers from the pools until we succeed
// sending the job.
let (mut operation, _) = FutureRetry::new(
let operation = FutureRetry::new(
async || {
let mut operation = pool.operation().await?;
operation
......@@ -395,8 +429,27 @@ pub async fn custom_evaluate(evaluate_context: impl EvaluateContext) -> Result<V
},
PoolErrorHandler,
)
.await
.map_err(|(e, _)| e)?;
.await;
let mut operation = match operation {
Ok((operation, _)) => operation,
Err((err, _)) => {
if let Some(connect_error) = NodeJsConnectError::find(&err) {
NodeJsConnectIssue {
source: IssueSource::from_source_only(
evaluate_context.context_source_for_issue(),
),
crash_context: evaluate_context.crash_context_prefix(),
// Connection errors happen first thing on startup, so we shouldn't be too
// worried about this being overly long.
stderr: connect_error.stderr.as_str().trim().into(),
}
.resolved_cell()
.emit();
return Ok(Vc::cell(None));
}
return Err(err);
}
};
// The evaluation sent an initial intermediate value without completing. We'll
// need to spawn a new thread to continually pull data out of the process,
......@@ -677,6 +730,10 @@ impl EvaluateContext for BasicEvaluateContext {
self.cwd.clone().cell()
}
fn context_source_for_issue(&self) -> ResolvedVc<Box<dyn Source>> {
self.context_source_for_issue
}
fn keep_alive(&self) -> bool {
!self.args.is_empty()
}
......@@ -769,3 +826,76 @@ impl Issue for EvaluationIssue {
Some(self.source)
}
}
/// Node.js child processes could not connect back to the parent process, which
/// makes it impossible to run any Node.js code (webpack loaders, PostCSS, ...).
#[turbo_tasks::value(shared)]
pub struct NodeJsConnectIssue {
/// The source being transformed when the connection failed. The failure is
/// unrelated to its contents, but this tells the user which transform hit it.
pub source: IssueSource,
/// What was being evaluated, e.g. the loader chain.
pub crash_context: Option<RcStr>,
/// Error output of the child process, describing why the connection failed.
pub stderr: RcStr,
}
#[async_trait]
#[turbo_tasks::value_impl]
impl Issue for NodeJsConnectIssue {
fn severity(&self) -> IssueSeverity {
IssueSeverity::Error
}
async fn title(&self) -> Result<StyledString> {
Ok(StyledString::Text(rcstr!(
"Node.js worker processes could not connect to Turbopack"
)))
}
fn stage(&self) -> IssueStage {
IssueStage::Transform
}
async fn file_path(&self) -> Result<FileSystemPath> {
self.source.file_path().await
}
fn source(&self) -> Option<IssueSource> {
Some(self.source)
}
async fn description(&self) -> Result<Option<StyledString>> {
let mut first_line = vec![StyledString::Text(rcstr!(
"Turbopack is configured to run webpack loaders and PostCSS transforms in child \
processes that communicate with it over a loopback TCP connection, but the child \
process was unable to connect"
))];
if let Some(crash_context) = &self.crash_context {
first_line.push(StyledString::Text(rcstr!(" while evaluating ")));
first_line.push(StyledString::Code(crash_context.clone()));
};
first_line.push(StyledString::Text(rcstr!(":")));
Ok(Some(StyledString::Stack(vec![
StyledString::Line(first_line),
StyledString::Code(self.stderr.clone()),
StyledString::Line(vec![
StyledString::Text(rcstr!(
"This is likely caused by a sandbox or firewall that blocks connections. \
Allow local connections, or run this code using worker threads by setting "
)),
StyledString::Code(rcstr!(
"experimental.turbopackPluginRuntimeStrategy: 'workerThreads'"
)),
StyledString::Text(rcstr!(" in your Next.js config.")),
]),
])))
}
fn documentation_link(&self) -> RcStr {
rcstr!(
"https://nextjs.org/docs/app/api-reference/config/next-config-js/turbopackPluginRuntimeStrategy"
)
}
}
......@@ -33,7 +33,7 @@ use turbopack_ecmascript::magic_identifier::unmangle_identifiers;
use crate::{
AssetsForSourceMapping,
backend::{CreatePoolFuture, CreatePoolOptions, NodeBackend},
evaluate::{EvaluateOperation, EvaluatePool, Operation},
evaluate::{EvaluateOperation, EvaluatePool, NodeJsConnectError, Operation},
format::FormattingMode,
pool_stats::{AcquiredPermits, NodeJsPoolStats, PoolStatsSnapshot},
source_map::apply_source_mapping,
......@@ -85,6 +85,10 @@ impl PartialEq for NodeJsPoolProcess {
const CONNECT_TIMEOUT: Duration = Duration::from_secs(30);
/// Exit code used by the child process when it cannot connect to the port we
/// listen on (see `js/src/child_process/index.ts`).
const CONNECT_FAILED_EXIT_CODE: i32 = 69;
#[derive(Clone, PartialEq, Eq, Hash)]
struct OutputEntry {
data: Arc<[u8]>,
......@@ -380,6 +384,9 @@ impl NodeJsPoolProcess {
match status {
Ok(status) => {
let (stdout, stderr) = get_output(&mut child).await?;
if status.code() == Some(CONNECT_FAILED_EXIT_CODE) {
return Err(NodeJsConnectError { stderr }.into());
}
bail!("node process exited before we could connect to it with {status}\nProcess output:\n{stdout}\nProcess error output:\n{stderr}");
}
Err(err) => {
......
......@@ -771,6 +771,10 @@ impl EvaluateContext for WebpackLoaderContext {
self.cwd.clone().cell()
}
fn context_source_for_issue(&self) -> ResolvedVc<Box<dyn Source>> {
self.context_source_for_issue
}
fn keep_alive(&self) -> bool {
true
}
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment