Unverified Commit ef851206 authored by Benjamin Woodruff ‮'s avatar Benjamin Woodruff Committed by GitHub

Turbopack: Improve file existence error handling in realpath_with_links and in…

Turbopack: Improve file existence error handling in realpath_with_links and in module resolution (#97717)

Fixes some regressions from
https://github.com/vercel/next.js/pull/97395.

Human summary of changes:

- Plainly return `NotFound` for `realpath_with_link` whenever any step
in the resolution algorithm fails, matching the behavior the libc uses.
- When probing in the module resolution algorithm, explicitly check for
`NotFound`.
- Fully remove the extremely non-standard and confusing behavior of
trying to return the last processed path from `realpath_with_links` if a
failure occurs.
- Modify `realpath` to return a `RealPathError` instead of just an
`anyhow::Error`, so that callers that care about the actual error kind
but not all of the intermediate links can still use it instead of
`realpath_with_links`.
- Rename `RealPathResultError` to `RealPathError`, and make it actually
implement `Display` so that it's easier to coerce to an `anyhow::Error`.
This requires including a tiny bit more data on this object, but in the
hot path it's just one extra clone of a `FileSystemPath`.

Human discussion:
https://vercel.slack.com/archives/C09R44U5HQW/p1787322213695299

AI slop summary of why #97395 caused a regression:

> The regression came from newly added missing-target detection in
b2e835de.
>
> Before that commit, realpath_with_links:
>
> 1. Resolved directory symlinks.
> 2. Continued resolving the remaining path beneath the resolved
directory.
> 3. If that remaining path did not exist, returned the resolved path
successfully.
> 4. Let the caller’s subsequent existence check report that the
candidate was absent.
>
> The commit changed read_link so it no longer checked the symlink
target’s type. To detect dangling symlinks, realpath_with_links added
this condition:
>
> ```
> if (followed_link || !symlinks.is_empty())
>     && entry_type == FileSystemEntryType::NotFound
> {
>     // report a missing symlink target
> }
> ```
>
> The problem is that !symlinks.is_empty() does not mean the missing
path is a symlink target. It only means that some earlier path component
was a symlink.
>
> For the reported path:
>
> `node_modules/@workflow/core/dist/serialization/package.json`
>
> resolution worked like this:
>
> 1. Resolve node_modules/@workflow/core, a valid directory symlink.
> 2. Continue into dist/serialization, which exists.
> 3. Probe the optional dist/serialization/package.json, which does not
exist.
> 4. Because the earlier @workflow/core component was a symlink,
symlinks was nonempty.
> 5. The new condition incorrectly classified the missing package.json
as a dangling symlink target.
> 6. Turbopack aborted instead of treating package.json as an absent
resolution candidate and continuing to serialization.js.
>
> So the regression was not caused by resolving the directory symlink
incorrectly. It was caused by using “we traversed any symlink” as
evidence that a later missing descendant was the target of a dangling
symlink.
parent 8911fe36
......@@ -120,7 +120,8 @@ pub async fn trace_endpoint(
// where
// node_modules/.pnpm/node_modules/@libsql/client is a symlink
let parent_path = referenced_chunk_path.parent();
if parent_path.realpath().await? != parent_path {
let resolved_parent_path = parent_path.realpath().await??;
if resolved_parent_path != parent_path {
turbo_tasks::turbobail!(
"Encountered file inside of symlink in NFT list: {parent_path} is a \
symlink, but {referenced_chunk_path} was created inside of it"
......
use anyhow::Result;
use anyhow::{Result, bail};
use tracing::Instrument;
use turbo_rcstr::RcStr;
use turbo_tasks::{OptionVcExt, ResolvedVc, TryJoinIterExt, ValueToStringRef, Vc};
use turbo_tasks_fs::{
DirectoryContent, DirectoryEntry, FileSystemEntryType, FileSystemPath, FileSystemPathOption,
RealPathErrorType,
};
use crate::next_import_map::get_next_package;
......@@ -106,12 +107,24 @@ pub async fn find_pages_structure(
page_extensions: Vc<Vec<RcStr>>,
next_mode: Vc<crate::mode::NextMode>,
) -> Result<Vc<PagesStructure>> {
let pages_root = project_root.join("pages")?.realpath().await?;
let pages_root = if *pages_root.get_type().await? == FileSystemEntryType::Directory {
async fn realpath_if_exists(path: &FileSystemPath) -> Result<Option<FileSystemPath>> {
match path.realpath().await? {
Ok(path) => Ok(Some(path)),
Err(error) if matches!(error.kind(), RealPathErrorType::NotFound) => Ok(None),
Err(error) => bail!(error),
}
}
let pages_root = realpath_if_exists(&project_root.join("pages")?).await?;
let pages_root = if let Some(pages_root) = pages_root
&& *pages_root.get_type().await? == FileSystemEntryType::Directory
{
Some(pages_root)
} else {
let src_pages_root = project_root.join("src/pages")?.realpath().await?;
if *src_pages_root.get_type().await? == FileSystemEntryType::Directory {
let src_pages_root = realpath_if_exists(&project_root.join("src/pages")?).await?;
if let Some(src_pages_root) = src_pages_root
&& *src_pages_root.get_type().await? == FileSystemEntryType::Directory
{
Some(src_pages_root)
} else {
// If neither pages nor src/pages exists, we still want to generate
......
# Symbolic File Links
This test represents what the Next.js file structure may look like when run
under a build orchestrator, such as bazel, where its sandbox sets up all files
to be symlinks to their original source.
../../somewhere-else/src/app/layout.tsx
\ No newline at end of file
../../somewhere-else/src/app/page.tsx
\ No newline at end of file
../somewhere-else/src/i18n.ts
\ No newline at end of file
import { nextTestSetup } from 'e2e-utils'
describe('symbolic-file-links', () => {
const { next } = nextTestSetup({
files: __dirname,
})
// Recommended for tests that need a full browser
it('should work using browser', async () => {
const browser = await next.browser('/')
expect(await browser.elementByCss('p').text()).toBe('hello world')
})
// In case you need the full HTML. Can also use $.html() with cheerio.
it('should work with html', async () => {
const html = await next.render('/')
expect(html).toContain('hello world')
})
})
# Symbolic Links
The `/file-symlink` route represents what the Next.js file structure may look
like when run under a build orchestrator, such as Bazel, where its sandbox sets
up source files as symlinks to their original source.
The `/directory-symlink` route covers resolving a module through a directory
symlink under `src`.
import message from '../../symlinked-package/serialization'
export default function Page() {
return <p>{message}</p>
}
../../somewhere-else/src/app/file-symlink/page.tsx
\ No newline at end of file
somewhere-else/src/i18n.ts
\ No newline at end of file
export default function Root({ children }: { children: React.ReactNode }) {
return (
<html>
<body>{children}</body>
</html>
)
}
throw new Error(
'The serialization file should take precedence over this directory'
)
somewhere-else/src/symlinked-package
\ No newline at end of file
import { nextTestSetup } from 'e2e-utils'
describe('symbolic-links', () => {
const { next } = nextTestSetup({
files: __dirname,
})
it('should render a route whose page is a file symlink', async () => {
const browser = await next.browser('/file-symlink')
expect(await browser.elementByCss('p').text()).toBe(
'hello from a file symlink'
)
})
it('should render a route that imports through a directory symlink', async () => {
const html = await next.render('/directory-symlink')
expect(html).toContain('hello from a directory symlink')
})
})
......@@ -1719,8 +1719,8 @@ mod tests {
use super::extract_effects_operation;
use crate::{
DiskFileSystem, FileSystem, FileSystemEntryType, FileSystemPath, LinkContent,
LinkTarget, RealPathResultError, WriteLinkContent, WriteLinkTarget,
WriteLinkTargetType, canonicalize_to_rcstr,
LinkTarget, RealPathErrorType, WriteLinkContent, WriteLinkTarget, WriteLinkTargetType,
canonicalize_to_rcstr,
};
#[turbo_tasks::function(operation, root)]
......@@ -1864,7 +1864,7 @@ mod tests {
}
/// `read_link` never looks at the target, so a dangling link still reads back as a valid
/// [`LinkContent::Link`]. Resolving it is what discovers the target is missing.
/// [`LinkContent::Link`]. Resolving it reports the missing target.
#[cfg(unix)]
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_dangling_symlink() {
......@@ -1873,8 +1873,10 @@ mod tests {
let scratch = tempfile::tempdir().unwrap();
let path = scratch.path().to_owned();
create_dir_all(path.join("sub")).unwrap();
create_dir_all(path.join("target-dir")).unwrap();
symlink("missing.txt", path.join("sub/link-dangling")).unwrap();
symlink("link-dangling", path.join("sub/link-chain")).unwrap();
symlink("../target-dir", path.join("sub/link-dir")).unwrap();
let root = canonicalize_to_rcstr(&path).unwrap();
......@@ -1899,44 +1901,36 @@ mod tests {
);
assert_eq!(target.target_type().await?, FileSystemEntryType::NotFound,);
// `realpath` follows the link, so it must report the missing target rather than
// succeeding with a path that doesn't exist.
// `realpath` follows the link and reports the missing target.
let result = link_path.realpath_with_links().await?;
assert!(
matches!(
&result.path_result,
Err(RealPathResultError::Invalid { reason })
if reason == "a symlink target does not exist"
),
"realpath must report the missing target as an invalid chain: {:?}",
result.path_result
);
let error = result.path_result.as_ref().unwrap_err();
let message = error.as_error_message(&link_path, &result).await?;
assert!(
message.contains("could not be resolved: a symlink target does not exist"),
"unexpected error message: {message}"
);
assert!(matches!(
result.path_result.as_ref().unwrap_err().kind(),
RealPathErrorType::NotFound
));
// The same missing target after another link is still an invalid chain, never a
// missing initial link.
// The same missing target after another link is also reported as not found.
let chain_path = root_path.join("sub/link-chain")?;
let result = chain_path.realpath_with_links().await?;
assert!(
matches!(
&result.path_result,
Err(RealPathResultError::Invalid { reason })
if reason == "a symlink target does not exist"
),
"realpath must report a missing target later in a chain as invalid: {:?}",
result.path_result
);
assert!(matches!(
result.path_result.as_ref().unwrap_err().kind(),
RealPathErrorType::NotFound
));
// Resolving a path that simply doesn't exist is not an error, though: there is no
// link involved, so it resolves to itself.
// A missing path beneath a resolved directory link is reported as not found.
let missing_in_linked_dir = root_path.join("sub/link-dir/package.json")?;
let result = missing_in_linked_dir.realpath_with_links().await?;
assert!(matches!(
result.path_result.as_ref().unwrap_err().kind(),
RealPathErrorType::NotFound
));
// A path that simply doesn't exist is also reported as not found.
let missing = root_path.join("sub/missing.txt")?;
let result = missing.realpath_with_links().await?;
assert_eq!(result.path_result, Ok(missing));
assert!(matches!(
result.path_result.as_ref().unwrap_err().kind(),
RealPathErrorType::NotFound
));
Ok(())
}
......
......@@ -59,7 +59,10 @@ pub use crate::{
},
disk::{DiskFileSystem, canonicalize_to_rcstr, validate_path_length},
null_fs::NullFileSystem,
path::{FileSystemPath, FileSystemPathOption, RealPathResult, RealPathResultError, rebase},
path::{
FileSystemPath, FileSystemPathOption, RealPathError, RealPathErrorType,
RealPathWithLinksResult, rebase,
},
read_glob::ReadGlobResult,
virtual_fs::VirtualFileSystem,
watcher::{DiskWatcherConfig, DiskWatcherPathMatcher, DiskWatcherRecursiveMode},
......@@ -120,9 +123,7 @@ impl DirectoryEntry {
let real_path = match &result.path_result {
Ok(path) => path,
Err(error) => {
return Ok(DirectoryEntry::Error(
error.as_error_message(symlink, result).await?,
));
return Ok(DirectoryEntry::Error(RcStr::from(error.to_string())));
}
};
Ok(match *real_path.get_type().await? {
......
This diff is collapsed.
......@@ -135,7 +135,7 @@ async fn dir_references(package_dir: FileSystemPath) -> Result<Vc<ModuleReferenc
Ok(path) => {
results.insert(path.clone());
}
Err(e) => bail!(e.as_error_message(file, &realpath).await?),
Err(error) => bail!(error.clone()),
}
}
PatternMatch::Directory(..) => {}
......
......@@ -20,7 +20,7 @@ use turbo_tasks::{
FxIndexMap, NonLocalValue, ReadRef, ResolvedVc, TryFlatJoinIterExt, TryJoinIterExt,
ValueToString, ValueToStringRef, Vc, trace::TraceRawVcs,
};
use turbo_tasks_fs::{FileSystemEntryType, FileSystemPath};
use turbo_tasks_fs::{FileSystemEntryType, FileSystemPath, RealPathErrorType};
use turbo_unix_path::normalize_request;
use crate::{
......@@ -1120,7 +1120,9 @@ async fn type_exists(
ty: FileSystemEntryType,
refs: Option<&mut Vec<ResolvedVc<Box<dyn Source>>>>,
) -> Result<Option<FileSystemPath>> {
let path = realpath(fs_path, refs).await?;
let Some(path) = realpath_if_exists(fs_path, refs).await? else {
return Ok(None);
};
Ok(if *path.get_type().await? == ty {
Some(path)
} else {
......@@ -1128,10 +1130,10 @@ async fn type_exists(
})
}
async fn realpath(
async fn realpath_if_exists(
fs_path: &FileSystemPath,
refs: Option<&mut Vec<ResolvedVc<Box<dyn Source>>>>,
) -> Result<FileSystemPath> {
) -> Result<Option<FileSystemPath>> {
let result = fs_path.realpath_with_links().await?;
if let Some(refs) = refs {
refs.extend(
......@@ -1148,8 +1150,9 @@ async fn realpath(
);
}
match &result.path_result {
Ok(path) => Ok(path.clone()),
Err(e) => bail!(e.as_error_message(fs_path, &result).await?),
Ok(path) => Ok(Some(path.clone())),
Err(error) if matches!(error.kind(), RealPathErrorType::NotFound) => Ok(None),
Err(error) => bail!(error.clone()),
}
}
......@@ -1378,14 +1381,17 @@ async fn find_package(
.await?;
for m in &*matches {
if let PatternMatch::Directory(_, package_dir) = m {
let Some(dir) = realpath_if_exists(
package_dir,
collect_affecting_sources.then_some(&mut affecting_sources),
)
.await?
else {
continue;
};
packages.push(FindPackageItem::PackageDirectory {
name: get_package_name(&fs_path, package_dir)?,
dir: realpath(
package_dir,
collect_affecting_sources
.then_some(&mut affecting_sources),
)
.await?,
dir,
});
}
}
......@@ -1408,23 +1414,31 @@ async fn find_package(
for m in &*matches {
match m {
PatternMatch::Directory(_, package_dir) => {
let Some(resolved_dir) = realpath_if_exists(
package_dir,
collect_affecting_sources.then_some(&mut affecting_sources),
)
.await?
else {
continue;
};
packages.push(FindPackageItem::PackageDirectory {
name: get_package_name(dir, package_dir)?,
dir: realpath(
package_dir,
collect_affecting_sources.then_some(&mut affecting_sources),
)
.await?,
dir: resolved_dir,
});
}
PatternMatch::File(_, package_file) => {
let Some(file) = realpath_if_exists(
package_file,
collect_affecting_sources.then_some(&mut affecting_sources),
)
.await?
else {
continue;
};
packages.push(FindPackageItem::PackageFile {
name: get_package_name(dir, package_file)?,
file: realpath(
package_file,
collect_affecting_sources.then_some(&mut affecting_sources),
)
.await?,
file,
});
}
}
......@@ -1447,13 +1461,17 @@ async fn find_package(
.await?;
for m in &matches {
if let PatternMatch::File(_, package_file) = m {
let Some(file) = realpath_if_exists(
package_file,
collect_affecting_sources.then_some(&mut affecting_sources),
)
.await?
else {
continue;
};
packages.push(FindPackageItem::PackageFile {
name: get_package_name(dir, package_file)?,
file: realpath(
package_file,
collect_affecting_sources.then_some(&mut affecting_sources),
)
.await?,
file,
});
}
}
......@@ -1511,7 +1529,7 @@ pub async fn resolve_raw(
let result = &*path.realpath_with_links().await?;
let path = match &result.path_result {
Ok(path) => path,
Err(e) => bail!(e.as_error_message(path, result).await?),
Err(error) => bail!(error.clone()),
};
let request_key = RequestKey::new(request);
let source = ResolvedVc::upcast(FileSource::new(path.clone()).to_resolved().await?);
......@@ -3117,7 +3135,7 @@ async fn resolved(
let result = &*fs_path.realpath_with_links().await?;
let path = match &result.path_result {
Ok(path) => path,
Err(e) => bail!(e.as_error_message(&fs_path, result).await?),
Err(error) => bail!(error.clone()),
};
let path_ref = path.clone();
......@@ -3438,19 +3456,63 @@ mod tests {
use turbo_tasks_backend::{BackendOptions, TurboTasksBackend, noop_backing_storage};
use turbo_tasks_fs::{DiskFileSystem, FileContent, FileSystem, FileSystemPath};
use super::*;
use crate::{
asset::AssetContent,
module::Module,
raw_module::RawModule,
resolve::{
ModuleResolveResult, ModuleResolveResultBuilder, ModuleResolveResultItem, RequestKey,
ResolveResult, ResolveResultItem, node::node_esm_resolve_options, parse::Request,
pattern::Pattern,
},
source::Source,
asset::AssetContent, module::Module, raw_module::RawModule, source::Source,
virtual_source::VirtualSource,
};
#[cfg(unix)]
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_missing_paths_through_symlinks_do_not_error() {
use std::os::unix::fs::symlink;
#[turbo_tasks::value]
struct MissingPathsResult {
missing_file: bool,
dangling_package: bool,
}
let scratch = tempfile::tempdir().unwrap();
create_dir_all(scratch.path().join("package")).unwrap();
symlink("package", scratch.path().join("linked-package")).unwrap();
symlink("missing-package", scratch.path().join("dangling-package")).unwrap();
let path = RcStr::from(scratch.path().to_str().unwrap());
let tt = turbo_tasks::TurboTasks::new(TurboTasksBackend::new(
BackendOptions::default(),
noop_backing_storage(),
));
#[turbo_tasks::function(operation, root)]
async fn missing_paths_through_symlinks_operation(
path: RcStr,
) -> Result<Vc<MissingPathsResult>> {
let fs = DiskFileSystem::new(rcstr!("temp"), Vc::cell(path));
let root = fs.root().owned().await?;
let missing_file = root.join("linked-package/package.json")?;
let dangling_package = root.join("dangling-package")?;
Ok(MissingPathsResult {
missing_file: realpath_if_exists(&missing_file, None).await?.is_none(),
dangling_package: realpath_if_exists(&dangling_package, None).await?.is_none(),
}
.cell())
}
tt.run_once(async move {
let missing = missing_paths_through_symlinks_operation(path)
.read_strongly_consistent()
.await?;
assert!(missing.missing_file);
assert!(missing.dangling_package);
anyhow::Ok(())
})
.await
.unwrap();
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_explicit_js_resolves_to_ts() {
resolve_relative_request_test(TestParams {
......@@ -3813,7 +3875,7 @@ mod tests {
force_in_lookup_dir,
fragment,
} => {
super::resolve_relative_request(
resolve_relative_request(
lookup_path,
request,
options,
......
......@@ -186,7 +186,7 @@ async fn resolve_reference_from_dir(
}
let path: FileSystemPath = match &realpath.path_result {
Ok(path) => path.clone(),
Err(e) => bail!(e.as_error_message(file, &realpath).await?),
Err(error) => bail!(error.clone()),
};
results.push((
RequestKey::new(matched_path.clone()),
......
......@@ -191,9 +191,7 @@ async fn resolve_node_pre_gyp_files(
format!("deps/lib/{key}").into(),
Vc::upcast(FileSource::new(match &realpath_with_links.path_result {
Ok(path) => path.clone(),
Err(e) => {
bail!(e.as_error_message(dylib, &realpath_with_links).await?)
}
Err(error) => bail!(error.clone()),
})),
);
}
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment