Unverified Commit 34bfd9bb authored by Niklas Mischkulnig's avatar Niklas Mischkulnig Committed by GitHub

Turbopack: ignore fully dynamic alternatives when tracing file paths (#100651)

## Summary

With Turbopack, a file read whose path is _either_ an unknown value _or_
a known file traced the **entire project** into `.nft.json`, including
`public/`. It also printed "Dynamic filesystem access causes tracing of
the whole project". Two examples:

- `readFileSync(options.path ?? path.join(process.cwd(),
'manifest.json'))`
- a default parameter `p = path.join(process.cwd(), 'manifest.json')`
that is then read

The default-parameter form regressed in 16.4: #97867 models default
params as "caller's argument or the default". The `??` form behaved like
this before.

A bare unknown path (`readFileSync(p)`) is already skipped when
`ignore_dynamic_requests` is on, as it is for Next.js server and client
code. The mixed case slipped through because `has_constant_parts()` is
true as soon as _any_ alternative is constant. The unbounded alternative
then resolved to every file in the project.

### Approach

A new `Pattern::filter_non_dynamic()` in `turbopack-core` keeps only the
top-level alternatives that have constant parts. When dynamic requests
are ignored, the file-tracing handlers trace that filtered pattern. The
handlers are: fs reads, `fs.readdir`, `path.join`, `path.resolve`,
`child_process.spawn` (its file reference) and `child_process.fork`. As
a result:

- the known file is traced, and the unknown alternative is ignored, just
as a bare unknown path is;
- fully dynamic patterns behave as before: a "very dynamic" lint, then
skip;
- partially bounded paths such as `path.join(process.cwd(), name)` still
trace the project and still warn;
- module requests (`require`, `import()`, `new URL`, workers), externals
tracing, standalone nft (`ignore_dynamic_requests` off) and webpack are
unchanged.

### Behavior change

A path that only exists in the dropped dynamic alternative is no longer
traced. It now ends up exactly where a bare `readFileSync(foo)` already
does. The only thing it got from the old behavior was a whole-project
trace, which was a failure in its own right, not a working fallback. For
the same reason the whole-project warning goes away for these cases.

`child_process.fork(x ?? './worker.js')` creates a module reference, so
in that case this also narrows what gets bundled to `./worker.js`.

### Tests

- `Pattern::filter_non_dynamic` has unit tests.
- `test/production/standalone-mode/tracing-static-files` covers the
`??`, default-parameter and bare-dynamic forms by reading the page's
`.nft.json`:
  - **Turbopack:** the known data files are traced.
- **Both bundlers:** the unreferenced `data/unrelated.txt` and
`next.config.js` aren't traced.

  Without the fix, Turbopack lists the whole project.
- The same suite has an external package (`serverExternalPackages`) with
the same patterns, covering the externals tracing context where
`ignore_dynamic_requests` is off. There, the dynamic alternatives are
kept and traced from the reading module's directory, but they don't
escape into the project. Project-root files referenced via
`process.cwd()` from external packages aren't traced; that predates this
PR and is asserted as current behavior.

## Verification

- Manual check: in an app whose instrumentation hook reads a manifest
through a default parameter, the trace went from the whole project (260
MB from a large `public/` asset) to just the manifest and the generated
chunks.

<!-- NEXT_JS_LLM -->


<!-- fleet ad3254c1-cab0-4e7e-89a5-84a673c8e16a -->

---------
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
parent caa60a93
import path from 'path' import path from 'path'
import 'foo' import 'foo'
import 'external-dynamic'
import {
readNullishFallback,
readDefaultParam,
readBareDynamic,
} from './read-alternatives'
const projectDir = process.cwd() const projectDir = process.cwd()
path.join(projectDir, 'data', 'static-from-app-cwd.txt') path.join(projectDir, 'data', 'static-from-app-cwd.txt')
...@@ -8,6 +14,10 @@ path.join(projectDir, 'data', 'static-from-app-cwd.txt') ...@@ -8,6 +14,10 @@ path.join(projectDir, 'data', 'static-from-app-cwd.txt')
path.join('data', 'static-from-app-rel-join.txt') path.join('data', 'static-from-app-rel-join.txt')
path.join('data', 'static-from-app-rel-read.txt') path.join('data', 'static-from-app-rel-read.txt')
readNullishFallback()
readDefaultParam()
readBareDynamic()
export default function Page() { export default function Page() {
return 'hello' return 'hello'
} }
import fs from 'fs'
import path from 'path'
// Each of these reads either a statically known file or a fully dynamic path.
// The known file should be traced, but the dynamic alternative should be
// ignored (like a bare dynamic path is) instead of tracing the whole project.
export function readNullishFallback(dynamicPath) {
return fs.readFileSync(
dynamicPath ??
path.join(process.cwd(), 'data', 'static-from-app-nullish.txt'),
'utf8'
)
}
export function readDefaultParam(
filePath = path.join(
process.cwd(),
'data',
'static-from-app-default-param.txt'
)
) {
return fs.readFileSync(filePath, 'utf8')
}
export function readBareDynamic(dynamicPath) {
try {
return fs.readFileSync(dynamicPath, 'utf8')
} catch {
return undefined
}
}
This file is not read by any server code and must not be traced.
const fs = require('fs')
exports.readBareDynamic = function readBareDynamic(dynamicPath) {
try {
return fs.readFileSync(dynamicPath, 'utf8')
} catch {
return undefined
}
}
const fs = require('fs')
const path = require('path')
exports.readDefaultParam = function readDefaultParam(
filePath = path.join(
process.cwd(),
'data',
'static-from-ext-default-param.txt'
)
) {
return fs.readFileSync(filePath, 'utf8')
}
const path = require('path')
const { readNullishFallback } = require('./nullish/read')
const { readDefaultParam } = require('./default-param/read')
const { readBareDynamic } = require('./bare-dynamic/read')
// This package is listed in `serverExternalPackages`, so it is traced with the
// externals tracing context instead of being bundled.
const projectDir = process.cwd()
path.join(projectDir, 'data', 'static-from-ext-cwd.txt')
path.join('data', 'static-from-ext-rel-join.txt')
path.join('data', 'static-from-ext-rel-read.txt')
readNullishFallback()
readDefaultParam()
readBareDynamic()
module.exports = 'external-dynamic'
const fs = require('fs')
const path = require('path')
exports.readNullishFallback = function readNullishFallback(dynamicPath) {
return fs.readFileSync(
dynamicPath ??
path.join(process.cwd(), 'data', 'static-from-ext-nullish.txt'),
'utf8'
)
}
{
"name": "external-dynamic",
"version": "0.0.0",
"main": "index.js"
}
/** /**
* @type {import('next').NextConfig} * @type {import('next').NextConfig}
*/ */
const nextConfig = {} const nextConfig = {
serverExternalPackages: ['external-dynamic'],
}
module.exports = nextConfig module.exports = nextConfig
{ {
"dependencies": { "dependencies": {
"external-dynamic": "file:./external-dynamic",
"foo": "file:./foo" "foo": "file:./foo"
} }
} }
...@@ -9,10 +9,12 @@ describe('standalone mode - tracing-static-files', () => { ...@@ -9,10 +9,12 @@ describe('standalone mode - tracing-static-files', () => {
skipStart: true, skipStart: true,
}) })
it('should trace process.cwd calls in node_modules', async () => { beforeAll(async () => {
let { exitCode } = await next.build() const { exitCode } = await next.build()
expect(exitCode).toBe(0) if (exitCode !== 0) throw new Error(`Fixture build failed: ${exitCode}`)
})
it('should trace process.cwd calls in node_modules', async () => {
let trace = await next.readJSON('.next/server/app/page.js.nft.json') let trace = await next.readJSON('.next/server/app/page.js.nft.json')
// should trace process.cwd and relative calls relative to the root // should trace process.cwd and relative calls relative to the root
...@@ -38,4 +40,59 @@ describe('standalone mode - tracing-static-files', () => { ...@@ -38,4 +40,59 @@ describe('standalone mode - tracing-static-files', () => {
expect(trace.files).not.toContain('../../../app/static-from-pkg.txt') expect(trace.files).not.toContain('../../../app/static-from-pkg.txt')
expect(trace.files).not.toContain('../../../foo/foo.txt') expect(trace.files).not.toContain('../../../foo/foo.txt')
}) })
it('should ignore fully dynamic alternatives of fs paths', async () => {
let trace = await next.readJSON('.next/server/app/page.js.nft.json')
if (process.env.IS_TURBOPACK_TEST) {
// `dynamic ?? path.join(process.cwd(), ...)` and a parameter defaulting to
// `path.join(process.cwd(), ...)` should still trace the known file
expect(trace.files).toContain('../../../data/static-from-app-nullish.txt')
expect(trace.files).toContain(
'../../../data/static-from-app-default-param.txt'
)
}
// the dynamic alternative should not cause the whole project to be traced
expect(trace.files).not.toContain('../../../data/unrelated.txt')
expect(trace.files).not.toContain('../../../next.config.js')
})
it('should keep dynamic alternatives of fs paths in external packages', async () => {
let trace = await next.readJSON('.next/server/app/page.js.nft.json')
// External packages are traced without the project root: `process.cwd()` is
// unknown there and relative paths resolve from the module's directory, so
// project files referenced by the package are currently not traced.
for (const name of [
'cwd',
'rel-join',
'rel-read',
'nullish',
'default-param',
]) {
expect(trace.files).not.toContain(
`../../../data/static-from-ext-${name}.txt`
)
}
if (process.env.IS_TURBOPACK_TEST) {
// External packages are traced without ignoring dynamic requests, so the
// dynamic alternatives are kept and traced relative to the reading
// module's directory, which pulls in otherwise unreferenced files there.
for (const name of ['nullish', 'default-param', 'bare-dynamic']) {
expect(trace.files).toContainEqual(
expect.stringMatching(
new RegExp(
`/node_modules/external-dynamic/${name}/sentinel-${name}\\.txt$`
)
)
)
}
}
// ...but they don't cause the whole project to be traced
expect(trace.files).not.toContain('../../../data/unrelated.txt')
expect(trace.files).not.toContain('../../../next.config.js')
})
}) })
...@@ -169,6 +169,28 @@ impl Pattern { ...@@ -169,6 +169,28 @@ impl Pattern {
} }
} }
/// Returns the alternatives that contain significant constant parts (everything except `/`).
///
/// The pattern is expected to be normalized so that alternatives are at the top level.
pub fn filter_non_dynamic(&self) -> Option<Pattern> {
if let Pattern::Alternatives(list) = self {
let mut non_dynamic_alternatives = list
.iter()
.filter(|alternative| alternative.has_constant_parts())
.cloned()
.collect::<Vec<_>>();
match non_dynamic_alternatives.len() {
0 => None,
1 => non_dynamic_alternatives.pop(),
_ => Some(Pattern::Alternatives(non_dynamic_alternatives)),
}
} else if self.has_constant_parts() {
Some(self.clone())
} else {
None
}
}
pub fn constant_prefix(&self) -> &str { pub fn constant_prefix(&self) -> &str {
// The normalized pattern is an Alternative of maximally merged // The normalized pattern is an Alternative of maximally merged
// Concatenations, so extracting the first/only Concatenation child // Concatenations, so extracting the first/only Concatenation child
...@@ -2083,6 +2105,46 @@ mod tests { ...@@ -2083,6 +2105,46 @@ mod tests {
); );
} }
#[test]
fn filter_non_dynamic() {
let constant = Pattern::Constant(rcstr!("./file.json"));
let partial = Pattern::Concatenation(vec![
Pattern::Constant(rcstr!("./prefix/")),
Pattern::Dynamic,
]);
let dynamic = Pattern::Concatenation(vec![
Pattern::Dynamic,
Pattern::Constant(rcstr!("/")),
Pattern::DynamicNoSlash,
]);
assert_eq!(constant.filter_non_dynamic(), Some(constant.clone()));
assert_eq!(partial.filter_non_dynamic(), Some(partial.clone()));
assert_eq!(Pattern::Dynamic.filter_non_dynamic(), None);
assert_eq!(Pattern::DynamicNoSlash.filter_non_dynamic(), None);
assert_eq!(Pattern::Constant(rcstr!("/")).filter_non_dynamic(), None);
assert_eq!(dynamic.filter_non_dynamic(), None);
assert_eq!(Pattern::Alternatives(vec![]).filter_non_dynamic(), None);
assert_eq!(
Pattern::Alternatives(vec![Pattern::Dynamic, dynamic.clone()]).filter_non_dynamic(),
None
);
assert_eq!(
Pattern::Alternatives(vec![Pattern::Dynamic, constant.clone()]).filter_non_dynamic(),
Some(constant.clone())
);
assert_eq!(
Pattern::Alternatives(vec![
Pattern::Dynamic,
constant.clone(),
dynamic,
partial.clone(),
])
.filter_non_dynamic(),
Some(Pattern::Alternatives(vec![constant, partial]))
);
}
#[test] #[test]
fn with_normalized_path() { fn with_normalized_path() {
assert!( assert!(
......
...@@ -2582,19 +2582,24 @@ where ...@@ -2582,19 +2582,24 @@ where
let args = linked_args().await?; let args = linked_args().await?;
if !args.is_empty() { if !args.is_empty() {
let pat = js_value_to_pattern(&args[0]); let pat = js_value_to_pattern(&args[0]);
if !pat.has_constant_parts() { let pat = match pat.filter_non_dynamic() {
let (args, hints) = explain_args(args); Some(filtered) if ignore_dynamic_requests => filtered,
handler.span_warn_with_code( Some(_) => pat,
span, None => {
&format!("fs.{name}({args}) is very dynamic{hints}",), let (args, hints) = explain_args(args);
DiagnosticId::Lint( handler.span_warn_with_code(
errors::failed_to_analyze::ecmascript::FS_METHOD.to_string(), span,
), &format!("fs.{name}({args}) is very dynamic{hints}",),
); DiagnosticId::Lint(
if ignore_dynamic_requests { errors::failed_to_analyze::ecmascript::FS_METHOD.to_string(),
return Ok(()); ),
);
if ignore_dynamic_requests {
return Ok(());
}
pat
} }
} };
let (static_files_only_context_dir, fallback_context_dir) = let (static_files_only_context_dir, fallback_context_dir) =
get_traced_project_dirs().await?; get_traced_project_dirs().await?;
analysis.add_reference( analysis.add_reference(
...@@ -2622,19 +2627,24 @@ where ...@@ -2622,19 +2627,24 @@ where
let args = linked_args().await?; let args = linked_args().await?;
if !args.is_empty() { if !args.is_empty() {
let pat = js_value_to_pattern(&args[0]); let pat = js_value_to_pattern(&args[0]);
if !pat.has_constant_parts() { let pat = match pat.filter_non_dynamic() {
let (args, hints) = explain_args(args); Some(filtered) if ignore_dynamic_requests => filtered,
handler.span_warn_with_code( Some(_) => pat,
span, None => {
&format!("fs.readdir({args}) is very dynamic{hints}"), let (args, hints) = explain_args(args);
DiagnosticId::Lint( handler.span_warn_with_code(
errors::failed_to_analyze::ecmascript::FS_METHOD.to_string(), span,
), &format!("fs.readdir({args}) is very dynamic{hints}"),
); DiagnosticId::Lint(
if ignore_dynamic_requests { errors::failed_to_analyze::ecmascript::FS_METHOD.to_string(),
return Ok(()); ),
);
if ignore_dynamic_requests {
return Ok(());
}
pat
} }
} };
let (static_files_only_context_dir, fallback_context_dir) = let (static_files_only_context_dir, fallback_context_dir) =
get_traced_project_dirs().await?; get_traced_project_dirs().await?;
analysis.add_reference( analysis.add_reference(
...@@ -2682,19 +2692,24 @@ where ...@@ -2682,19 +2692,24 @@ where
.await?; .await?;
let pat = js_value_to_pattern(&linked_func_call); let pat = js_value_to_pattern(&linked_func_call);
if !pat.has_constant_parts() { let pat = match pat.filter_non_dynamic() {
let (args, hints) = explain_args(args); Some(filtered) if ignore_dynamic_requests => filtered,
handler.span_warn_with_code( Some(_) => pat,
span, None => {
&format!("path.resolve({args}) is very dynamic{hints}",), let (args, hints) = explain_args(args);
DiagnosticId::Lint( handler.span_warn_with_code(
errors::failed_to_analyze::ecmascript::PATH_METHOD.to_string(), span,
), &format!("path.resolve({args}) is very dynamic{hints}",),
); DiagnosticId::Lint(
if ignore_dynamic_requests { errors::failed_to_analyze::ecmascript::PATH_METHOD.to_string(),
return Ok(()); ),
);
if ignore_dynamic_requests {
return Ok(());
}
pat
} }
} };
let (static_files_only_context_dir, fallback_context_dir) = let (static_files_only_context_dir, fallback_context_dir) =
get_traced_project_dirs().await?; get_traced_project_dirs().await?;
analysis.add_reference( analysis.add_reference(
...@@ -2737,19 +2752,24 @@ where ...@@ -2737,19 +2752,24 @@ where
) )
.await?; .await?;
let pat = js_value_to_pattern(&linked_func_call); let pat = js_value_to_pattern(&linked_func_call);
if !pat.has_constant_parts() { let pat = match pat.filter_non_dynamic() {
let (args, hints) = explain_args(args); Some(filtered) if ignore_dynamic_requests => filtered,
handler.span_warn_with_code( Some(_) => pat,
span, None => {
&format!("path.join({args}) is very dynamic{hints}",), let (args, hints) = explain_args(args);
DiagnosticId::Lint( handler.span_warn_with_code(
errors::failed_to_analyze::ecmascript::PATH_METHOD.to_string(), span,
), &format!("path.join({args}) is very dynamic{hints}",),
); DiagnosticId::Lint(
if ignore_dynamic_requests { errors::failed_to_analyze::ecmascript::PATH_METHOD.to_string(),
return Ok(()); ),
);
if ignore_dynamic_requests {
return Ok(());
}
pat
} }
} };
let (static_files_only_context_dir, fallback_context_dir) = let (static_files_only_context_dir, fallback_context_dir) =
get_traced_project_dirs().await?; get_traced_project_dirs().await?;
analysis.add_reference( analysis.add_reference(
...@@ -2810,11 +2830,15 @@ where ...@@ -2810,11 +2830,15 @@ where
); );
} }
} }
let dynamic = !pat.has_constant_parts(); let pat = match pat.filter_non_dynamic() {
if dynamic { Some(filtered) if ignore_dynamic_requests => Some(filtered),
show_dynamic_warning = true; Some(_) => Some(pat),
} None => {
if !dynamic || !ignore_dynamic_requests { show_dynamic_warning = true;
(!ignore_dynamic_requests).then_some(pat)
}
};
if let Some(pat) = pat {
let (static_files_only_context_dir, fallback_context_dir) = let (static_files_only_context_dir, fallback_context_dir) =
get_traced_project_dirs().await?; get_traced_project_dirs().await?;
analysis.add_reference( analysis.add_reference(
...@@ -2860,19 +2884,25 @@ where ...@@ -2860,19 +2884,25 @@ where
if !args.is_empty() { if !args.is_empty() {
let first_arg = &args[0]; let first_arg = &args[0];
let pat = js_value_to_pattern(first_arg); let pat = js_value_to_pattern(first_arg);
if !pat.has_constant_parts() { let pat = match pat.filter_non_dynamic() {
let (args, hints) = explain_args(args); Some(filtered) if ignore_dynamic_requests => filtered,
handler.span_warn_with_code( Some(_) => pat,
span, None => {
&format!("child_process.fork({args}) is very dynamic{hints}",), let (args, hints) = explain_args(args);
DiagnosticId::Lint( handler.span_warn_with_code(
errors::failed_to_analyze::ecmascript::CHILD_PROCESS_SPAWN.to_string(), span,
), &format!("child_process.fork({args}) is very dynamic{hints}",),
); DiagnosticId::Lint(
if ignore_dynamic_requests { errors::failed_to_analyze::ecmascript::CHILD_PROCESS_SPAWN
return Ok(()); .to_string(),
),
);
if ignore_dynamic_requests {
return Ok(());
}
pat
} }
} };
let error_mode = if in_try { let error_mode = if in_try {
ResolveErrorMode::Warn ResolveErrorMode::Warn
} else { } else {
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment