Unverified Commit f0c1ffc4 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

Remove ineffective turbo-tasks (#91341)

## Remove ineffective turbo-tasks

Identifies and removes turbo-tasks functions where the task overhead exceeds the value they provide. Each turbo-task carries ~4-6μs execution overhead per miss and ~200-500ns per cache hit, plus allocations and bookkeeping.

### What?

Removes 22 `#[turbo_tasks::function]` implementations across resolve plugins, chunk items, and resolve-result helpers — converting them to plain methods or inlining their work. Changes fall into a few buckets:

- **ResolvePlugin condition handling** (`AfterResolvePluginCondition::matches`, `BeforeResolvePluginCondition::matches`, `after_resolve_condition`, `before_resolve_condition`): conditions now store the resolved `Glob` as a `ReadRef<Glob>` on the plugin struct at construction, so `matches` is a pure sync function and the per-plugin `*_resolve_condition` getters are trivial field reads (no longer turbo-tasks). The `after_resolve` / `before_resolve` hooks themselves stay as `#[turbo_tasks::function]` — they synthesize virtual sources/modules and need memoization on `(self, lookup_path, reference_type, request)` to avoid distinct cells producing duplicate module-graph idents.
    - The basic theory here is that the right level of caching is at `resolve` and at the hook bodies themselves, not the conditions or condition getters.
    - `AfterResolvePluginCondition` and `BeforeResolvePluginCondition` are marked `serialization = "none"` because `ReadRef` cannot be persisted; plugin construction is cheap enough to re-derive on restore.
- **ChunkItem trait methods** (`chunking_context`, `ty`, `content_with_async_module_info`): returned constants or simple field reads, zero cache hits and no `.await` calls (no invalidation value).
- **ResolveResult / ModuleResolveResult helpers** (`primary_modules`, `first_module`, `first_source`, `primary_sources`, `is_unresolvable`, `primary_output_assets`): simple iterators over already-resolved data; converted to plain methods. Added a `Duplicate(usize)` variant to `ModuleResolveResultItem` to handle dedup at construction time instead of in a separate task.
    - The basic idea here is that it is reasonable to consume `ResolveResult/ModuleResolveResult` monolithically, and we get little to no benefit from fine grained access. e.g. `is_unresolved()` in theory that is a valuable turbotask, but since it rarely changes but generally if we change how we resolve an import then we have to regenerate code, so saving a few boolean conditions is unlikely to be very valuable.
- Misc: `EcmascriptModuleAsset::analyze`, `is_types_resolving_enabled`, `next_server::resolve::condition`.

### Impact (vercel-site build, dev first-compile)

| Metric | Before | After | Δ |
|---|---:|---:|---:|
| Total cache hits | 30,885,827 | 29,201,314 | −1,684,513 |
| Total cache misses | 6,473,123 | 5,953,626 | **−519,497** |
| Overall hit rate | 82.67% | 83.06% | +0.39 pp |
| Registered task functions | 1,294 | 1,272 | −22 |

The 22 removed tasks were collectively responsible for ~519K misses per build — each miss previously paying the full execution overhead. Most of the work from `EcmascriptModuleAsset::analyze` naturally migrated into `analyze_ecmascript_module` (the task it was wrapping; +129K hits there).

### On-disk cache size (persistent caching)

Each removed task also stops allocating cache cells on disk. Measured on the same vercel-site build with `.next/cache/turbopack` (persistent cache enabled):

| | Size |
|---|---:|
| canary | 2.56 GiB |
| this branch | 2.46 GiB |
| **saved** | **~100 MiB (−3.81%)** |

### Build-time wall clock and peak memory

Ran `pnpm next build --experimental-build-mode=compile` 5 times on each branch

**Peak RSS — clear reduction:**

| | canary | branch | Δ |
|---|---:|---:|---:|
| min | 19.18 GiB | 18.94 GiB | |
| **median** | **19.22 GiB** | **19.01 GiB** | **−217 MiB (−1.10%)** |
| mean | 19.21 GiB | 19.02 GiB | −199 MiB (−1.01%) |
| max | 19.23 GiB | 19.13 GiB | |

Every branch run has lower RSS than every canary run — the distributions don't overlap. Welch's t = −6.03.

**Wall time — no measurable change:**

| | canary | branch | Δ |
|---|---:|---:|---:|
| min | 62.03s | 60.78s | |
| **median** | **62.61s** | **62.65s** | **+0.04s (+0.06%)** |
| mean | 62.83s | 63.80s | +0.96s (+1.53%) |
| max | 64.25s | 68.23s | |
| stddev | 0.84s | 3.42s | |

Median is flat. The mean difference is within noise (Welch's t = +0.61, n = 5). Branch run-to-run variance is higher — one 68.23s outlier pulls the mean up — so this is neither a regression nor a measurable speedup at this sample size.

<!-- NEXT_JS_LLM_PR -->
parent e83bc939
......@@ -860,7 +860,6 @@ impl AppProject {
None,
ResolveErrorMode::Error,
)
.to_resolved()
.await?
.first_module()
.await?
......
......@@ -242,10 +242,11 @@ impl ServerNftJsonAsset {
None,
ResolveErrorMode::Error,
)
.await?
.primary_modules()
.await?
.into_iter()
.map(|m| **m))
.map(|m| *m))
})
.try_flat_join()
.await?,
......
......@@ -566,6 +566,7 @@ impl PagesProject {
None,
)
.await?
.await?
.first_module()
.await?
.context("expected Next.js client runtime to resolve to a module")?;
......
......@@ -40,7 +40,6 @@ impl RuntimeEntry {
None,
ResolveErrorMode::Error,
)
.to_resolved()
.await?
.primary_modules()
.await?;
......
use std::{io::Write, iter::once};
use anyhow::{Context, Result, bail};
use async_trait::async_trait;
use indoc::writedoc;
use turbo_rcstr::{RcStr, rcstr};
use turbo_tasks::{ResolvedVc, ValueToString, Vc};
......@@ -320,12 +321,10 @@ impl ChunkItem for EcmascriptClientReferenceProxyChunkItem {
self.inner_module.ident()
}
#[turbo_tasks::function]
fn chunking_context(&self) -> Vc<Box<dyn ChunkingContext>> {
*self.chunking_context
}
#[turbo_tasks::function]
fn ty(&self) -> Vc<Box<dyn ChunkType>> {
Vc::upcast(Vc::<EcmascriptChunkType>::default())
}
......@@ -336,21 +335,19 @@ impl ChunkItem for EcmascriptClientReferenceProxyChunkItem {
}
}
#[async_trait]
#[turbo_tasks::value_impl]
impl EcmascriptChunkItem for EcmascriptClientReferenceProxyChunkItem {
#[turbo_tasks::function]
fn content(&self) -> Vc<EcmascriptChunkItemContent> {
self.inner_chunk_item.content()
}
#[turbo_tasks::function]
fn content_with_async_module_info(
async fn content_with_async_module_info(
&self,
async_module_info: Option<Vc<AsyncModuleInfo>>,
estimated: bool,
) -> Vc<EcmascriptChunkItemContent> {
) -> Result<Vc<EcmascriptChunkItemContent>> {
self.inner_chunk_item
.into_trait_ref()
.await?
.content_with_async_module_info(async_module_info, estimated)
.await
}
}
......
......@@ -52,24 +52,27 @@ struct NextFontLocalFontFileOptions {
#[turbo_tasks::value]
pub(crate) struct NextFontLocalResolvePlugin {
root: FileSystemPath,
condition: ResolvedVc<BeforeResolvePluginCondition>,
}
#[turbo_tasks::value_impl]
impl NextFontLocalResolvePlugin {
#[turbo_tasks::function]
pub fn new(root: FileSystemPath) -> Vc<Self> {
NextFontLocalResolvePlugin { root }.cell()
pub async fn new(root: FileSystemPath) -> Result<Vc<Self>> {
let condition = BeforeResolvePluginCondition::from_request_glob(Glob::new(
rcstr!("{next,@vercel/turbopack-next/internal}/font/local/*"),
GlobOptions::default(),
))
.to_resolved()
.await?;
Ok(NextFontLocalResolvePlugin { root, condition }.cell())
}
}
#[turbo_tasks::value_impl]
impl BeforeResolvePlugin for NextFontLocalResolvePlugin {
#[turbo_tasks::function]
fn before_resolve_condition(&self) -> Vc<BeforeResolvePluginCondition> {
BeforeResolvePluginCondition::from_request_glob(Glob::new(
rcstr!("{next,@vercel/turbopack-next/internal}/font/local/*"),
GlobOptions::default(),
))
*self.condition
}
#[turbo_tasks::function]
......
......@@ -1269,7 +1269,7 @@ pub async fn try_get_next_package(
Request::parse(Pattern::Constant(rcstr!("next/package.json"))),
node_cjs_resolve_options(root.clone()),
);
if let Some(source) = &*result.first_source().await? {
if let Some(source) = result.await?.first_source() {
Ok(Vc::cell(Some(source.ident().await?.path.parent())))
} else {
MissingNextFolderIssue {
......
......@@ -42,41 +42,38 @@ pub enum ExternalPredicate {
/// possible to resolve them at runtime.
#[turbo_tasks::value]
pub(crate) struct ExternalCjsModulesResolvePlugin {
root: FileSystemPath,
predicate: ResolvedVc<ExternalPredicate>,
import_externals: bool,
condition: ResolvedVc<AfterResolvePluginCondition>,
}
#[turbo_tasks::value_impl]
impl ExternalCjsModulesResolvePlugin {
#[turbo_tasks::function]
pub fn new(
pub async fn new(
root: FileSystemPath,
predicate: ResolvedVc<ExternalPredicate>,
import_externals: bool,
) -> Vc<Self> {
ExternalCjsModulesResolvePlugin {
) -> Result<Vc<Self>> {
let condition = AfterResolvePluginCondition::new_with_glob(
root,
Glob::new(rcstr!("**/node_modules/**"), GlobOptions::default()),
)
.to_resolved()
.await?;
Ok(ExternalCjsModulesResolvePlugin {
predicate,
import_externals,
condition,
}
.cell()
.cell())
}
}
#[turbo_tasks::function]
fn condition(root: FileSystemPath) -> Vc<AfterResolvePluginCondition> {
AfterResolvePluginCondition::new_with_glob(
root,
Glob::new(rcstr!("**/node_modules/**"), GlobOptions::default()),
)
}
#[turbo_tasks::value_impl]
impl AfterResolvePlugin for ExternalCjsModulesResolvePlugin {
#[turbo_tasks::function]
fn after_resolve_condition(&self) -> Vc<AfterResolvePluginCondition> {
condition(self.root.clone())
*self.condition
}
#[turbo_tasks::function]
......@@ -85,7 +82,7 @@ impl AfterResolvePlugin for ExternalCjsModulesResolvePlugin {
fs_path: FileSystemPath,
lookup_path: FileSystemPath,
reference_type: ReferenceType,
request: ResolvedVc<Request>,
request: Vc<Request>,
) -> Result<Vc<ResolveResultOption>> {
let request_value = &*request.await?;
let Request::Module {
......@@ -112,10 +109,7 @@ impl AfterResolvePlugin for ExternalCjsModulesResolvePlugin {
let predicate = self.predicate.await?;
let must_be_external = match &*predicate {
ExternalPredicate::AllExcept(exceptions) => {
if *condition(self.root.clone())
.matches(lookup_path.clone())
.await?
{
if self.condition.await?.matches(&lookup_path) {
return Ok(ResolveResultOption::none());
}
......@@ -214,7 +208,7 @@ impl AfterResolvePlugin for ExternalCjsModulesResolvePlugin {
Ok(ResolveResultOption::none())
};
let mut request = *request;
let mut request = request;
let mut request_str = request_str.to_string();
let node_resolve_options = if is_esm {
......@@ -230,7 +224,7 @@ impl AfterResolvePlugin for ExternalCjsModulesResolvePlugin {
node_resolve_options,
);
let Some(result_from_original_location) =
*node_resolved_from_original_location.first_source().await?
node_resolved_from_original_location.await?.first_source()
else {
if is_esm
&& !package_subpath.is_empty()
......@@ -288,7 +282,7 @@ impl AfterResolvePlugin for ExternalCjsModulesResolvePlugin {
request,
node_resolve_options,
);
let resolves_equal = if let Some(result) = *node_resolved.first_source().await? {
let resolves_equal = if let Some(result) = node_resolved.await?.first_source() {
let ident = result.ident().await?;
let cjs_path = &ident.path;
cjs_path == path
......
......@@ -92,33 +92,35 @@ impl Issue for InvalidImportModuleIssue {
#[turbo_tasks::value]
pub(crate) struct NextExternalResolvePlugin {
project_path: FileSystemPath,
condition: ResolvedVc<AfterResolvePluginCondition>,
}
#[turbo_tasks::value_impl]
impl NextExternalResolvePlugin {
#[turbo_tasks::function]
pub fn new(project_path: FileSystemPath) -> Vc<Self> {
NextExternalResolvePlugin { project_path }.cell()
pub async fn new(project_path: FileSystemPath) -> Result<Vc<Self>> {
let condition = AfterResolvePluginCondition::new_with_glob(
project_path.root().owned().await?,
Glob::new(
rcstr!("**/next/dist/**/*.{external,runtime.dev,runtime.prod}.js"),
GlobOptions::default(),
),
)
.to_resolved()
.await?;
Ok(NextExternalResolvePlugin { condition }.cell())
}
}
#[turbo_tasks::value_impl]
impl AfterResolvePlugin for NextExternalResolvePlugin {
#[turbo_tasks::function]
async fn after_resolve_condition(&self) -> Result<Vc<AfterResolvePluginCondition>> {
Ok(AfterResolvePluginCondition::new_with_glob(
self.project_path.root().owned().await?,
Glob::new(
rcstr!("**/next/dist/**/*.{external,runtime.dev,runtime.prod}.js"),
GlobOptions::default(),
),
))
fn after_resolve_condition(&self) -> Vc<AfterResolvePluginCondition> {
*self.condition
}
#[turbo_tasks::function]
async fn after_resolve(
&self,
self: Vc<Self>,
fs_path: FileSystemPath,
_lookup_path: FileSystemPath,
_reference_type: ReferenceType,
......@@ -146,33 +148,38 @@ impl AfterResolvePlugin for NextExternalResolvePlugin {
#[turbo_tasks::value]
pub(crate) struct NextNodeSharedRuntimeResolvePlugin {
root: FileSystemPath,
server_context_type: ServerContextType,
condition: ResolvedVc<AfterResolvePluginCondition>,
}
#[turbo_tasks::value_impl]
impl NextNodeSharedRuntimeResolvePlugin {
#[turbo_tasks::function]
pub fn new(root: FileSystemPath, server_context_type: ServerContextType) -> Vc<Self> {
NextNodeSharedRuntimeResolvePlugin {
root,
pub async fn new(
root: FileSystemPath,
server_context_type: ServerContextType,
) -> Result<Vc<Self>> {
let condition = AfterResolvePluginCondition::new_with_glob(
root.root().owned().await?,
Glob::new(
rcstr!("**/next/dist/**/*.shared-runtime.js"),
GlobOptions::default(),
),
)
.to_resolved()
.await?;
Ok(NextNodeSharedRuntimeResolvePlugin {
server_context_type,
condition,
}
.cell()
.cell())
}
}
#[turbo_tasks::value_impl]
impl AfterResolvePlugin for NextNodeSharedRuntimeResolvePlugin {
#[turbo_tasks::function]
async fn after_resolve_condition(&self) -> Result<Vc<AfterResolvePluginCondition>> {
Ok(AfterResolvePluginCondition::new_with_glob(
self.root.root().owned().await?,
Glob::new(
rcstr!("**/next/dist/**/*.shared-runtime.js"),
GlobOptions::default(),
),
))
fn after_resolve_condition(&self) -> Vc<AfterResolvePluginCondition> {
*self.condition
}
#[turbo_tasks::function]
......@@ -225,32 +232,34 @@ impl AfterResolvePlugin for NextNodeSharedRuntimeResolvePlugin {
/// telemetry events if there is a match.
#[turbo_tasks::value]
pub(crate) struct ModuleFeatureReportResolvePlugin {
root: FileSystemPath,
condition: ResolvedVc<BeforeResolvePluginCondition>,
}
#[turbo_tasks::value_impl]
impl ModuleFeatureReportResolvePlugin {
#[turbo_tasks::function]
pub fn new(root: FileSystemPath) -> Vc<Self> {
ModuleFeatureReportResolvePlugin { root }.cell()
pub async fn new(_root: FileSystemPath) -> Result<Vc<Self>> {
let condition = BeforeResolvePluginCondition::from_modules(Vc::cell(
FEATURE_MODULES
.keys()
.map(|k| (*k).into())
.collect::<Vec<RcStr>>(),
))
.to_resolved()
.await?;
Ok(ModuleFeatureReportResolvePlugin { condition }.cell())
}
}
#[turbo_tasks::value_impl]
impl BeforeResolvePlugin for ModuleFeatureReportResolvePlugin {
#[turbo_tasks::function]
fn before_resolve_condition(&self) -> Vc<BeforeResolvePluginCondition> {
BeforeResolvePluginCondition::from_modules(Vc::cell(
FEATURE_MODULES
.keys()
.map(|k| (*k).into())
.collect::<Vec<RcStr>>(),
))
*self.condition
}
#[turbo_tasks::function]
async fn before_resolve(
&self,
self: Vc<Self>,
_lookup_path: FileSystemPath,
_reference_type: ReferenceType,
request: Vc<Request>,
......@@ -269,6 +278,7 @@ impl BeforeResolvePlugin for ModuleFeatureReportResolvePlugin {
.find(|sub_path| path.is_match(sub_path));
if let Some(sub_path) = sub_path {
// This is not accurate. we only emit one diagnostic per request not per import
ModuleFeatureTelemetry::new(format!("{module}{sub_path}").into(), 1)
.resolved_cell()
.emit();
......@@ -282,33 +292,35 @@ impl BeforeResolvePlugin for ModuleFeatureReportResolvePlugin {
#[turbo_tasks::value]
pub(crate) struct NextSharedRuntimeResolvePlugin {
root: FileSystemPath,
condition: ResolvedVc<AfterResolvePluginCondition>,
}
#[turbo_tasks::value_impl]
impl NextSharedRuntimeResolvePlugin {
#[turbo_tasks::function]
pub fn new(root: FileSystemPath) -> Vc<Self> {
NextSharedRuntimeResolvePlugin { root }.cell()
pub async fn new(root: FileSystemPath) -> Result<Vc<Self>> {
let condition = AfterResolvePluginCondition::new_with_glob(
root.root().owned().await?,
Glob::new(
rcstr!("**/next/dist/esm/**/*.shared-runtime.js"),
GlobOptions::default(),
),
)
.to_resolved()
.await?;
Ok(NextSharedRuntimeResolvePlugin { condition }.cell())
}
}
#[turbo_tasks::value_impl]
impl AfterResolvePlugin for NextSharedRuntimeResolvePlugin {
#[turbo_tasks::function]
async fn after_resolve_condition(&self) -> Result<Vc<AfterResolvePluginCondition>> {
Ok(AfterResolvePluginCondition::new_with_glob(
self.root.root().owned().await?,
Glob::new(
rcstr!("**/next/dist/esm/**/*.shared-runtime.js"),
GlobOptions::default(),
),
))
fn after_resolve_condition(&self) -> Vc<AfterResolvePluginCondition> {
*self.condition
}
#[turbo_tasks::function]
async fn after_resolve(
&self,
self: Vc<Self>,
fs_path: FileSystemPath,
_lookup_path: FileSystemPath,
_reference_type: ReferenceType,
......
......@@ -135,8 +135,10 @@ pub async fn get_swc_ecma_transform_rule_impl(
)
.await?;
let Some(plugin_module) =
&*plugin_wasm_module_resolve_result.first_module().await?
let Some(plugin_module) = plugin_wasm_module_resolve_result
.await?
.first_module()
.await?
else {
// Ignore unresolvable plugin modules, handle_resolve_error has already emitted
// an issue.
......
......@@ -260,7 +260,7 @@ async fn detect_react_compiler_target(
node_cjs_resolve_options(project_path.root().owned().await?),
);
let Some(source) = &*react_pkg_result.first_source().await? else {
let Some(source) = react_pkg_result.await?.first_source() else {
return Ok(None);
};
......@@ -335,7 +335,7 @@ pub async fn resolve_babel_plugin_react_compiler(
Request::parse(Pattern::Constant(BABEL_PLUGIN_REACT_COMPILER_PACKAGE_JSON)),
node_cjs_resolve_options(project_path.root().owned().await?),
);
let Some(source) = &*babel_plugin_result.first_source().await? else {
let Some(source) = babel_plugin_result.await?.first_source() else {
BabelPluginReactCompilerResolutionIssue {
failed_resolution: BABEL_PLUGIN_REACT_COMPILER,
config_file_path: next_config
......
......@@ -204,7 +204,7 @@ async fn track_glob_internal(
types.push(path.get_type())
}
}
// The most likely case of this is actually a sylink resolution error, it is
// The most likely case of this is actually a symlink resolution error, it is
// fine to ignore since the mere act of attempting to resolve it has triggered
// the ncecessary dependencies. If this file is actually a dependency we should
// get an error in the actual webpack loader when it reads it.
......
......@@ -63,9 +63,9 @@ pub async fn assert_can_resolve_react_refresh(
request,
resolve_options,
)
.first_source();
.await?;
if result.await?.is_some() {
if result.first_source().is_some() {
return Ok(ResolveReactRefreshResult::Found(request.to_resolved().await?).cell());
}
}
......
......@@ -40,7 +40,6 @@ impl RuntimeEntry {
None,
ResolveErrorMode::Error,
)
.to_resolved()
.await?
.primary_modules()
.await?;
......
......@@ -295,6 +295,7 @@ async fn build_internal(
origin
.resolve_asset(request_vc, origin.resolve_options(), ty)
.await?
.await?
.first_module()
.await?
.with_context(|| {
......
......@@ -138,15 +138,12 @@ pub async fn create_web_entry_source(
.into_iter()
.map(|request| async move {
let ty = ReferenceType::Entry(EntryReferenceSubType::Web);
Ok(origin
origin
.resolve_asset(request, origin.resolve_options(), ty)
.await?
.to_resolved()
.await?
.primary_modules()
.await?
.first()
.copied())
.first_module()
.await
})
.try_flat_join()
.await?;
......
......@@ -127,13 +127,26 @@ pub async fn make_chunk_group(
})
.try_join()
.await?;
let async_loader_chunk_items = async_loaders.iter().map(|&chunk_item| {
ChunkItemOrBatchWithAsyncModuleInfo::ChunkItem(ChunkItemWithAsyncModuleInfo {
chunk_item,
module: None,
async_info: None,
let async_loader_chunk_items = async_loaders
.iter()
.map(async |&chunk_item| {
let chunk_type = chunk_item
.into_trait_ref()
.await?
.ty()
.to_resolved()
.await?;
Ok(ChunkItemOrBatchWithAsyncModuleInfo::ChunkItem(
ChunkItemWithAsyncModuleInfo {
chunk_item,
chunk_type,
module: None,
async_info: None,
},
))
})
});
.try_join()
.await?;
let referenced_output_assets = traced_modules
.into_iter()
......
......@@ -11,7 +11,7 @@ use turbo_tasks::{
};
use crate::{
chunk::{ChunkItem, ChunkItemWithAsyncModuleInfo, ChunkType, ChunkableModule, ChunkingContext},
chunk::{ChunkItemWithAsyncModuleInfo, ChunkType, ChunkableModule, ChunkingContext},
module_graph::{
ModuleGraph,
async_module_info::AsyncModulesInfo,
......@@ -46,8 +46,15 @@ pub async fn attach_async_info_to_chunkable_module(
.as_chunk_item(module_graph, chunking_context)
.to_resolved()
.await?;
let chunk_type = chunk_item
.into_trait_ref()
.await?
.ty()
.to_resolved()
.await?;
Ok(ChunkItemWithAsyncModuleInfo {
chunk_item,
chunk_type,
module: Some(module),
async_info,
})
......@@ -103,10 +110,9 @@ impl ChunkItemOrBatchWithAsyncModuleInfo {
&self,
) -> Result<ChunkItemOrBatchWithAsyncModuleInfoByChunkType> {
Ok(match self {
Self::ChunkItem(item) => Either::Left(smallvec![(
item.chunk_item.ty().to_resolved().await?,
Self::ChunkItem(item.clone())
)]),
Self::ChunkItem(item) => {
Either::Left(smallvec![(item.chunk_type, Self::ChunkItem(*item))])
}
Self::Batch(batch) => Either::Right(batch.split_by_chunk_type().await?),
})
}
......@@ -167,17 +173,15 @@ impl ChunkItemBatchWithAsyncModuleInfo {
let Some((_, first)) = iter.next() else {
return Ok(Vc::cell(SmallVec::new()));
};
let chunk_type = first.chunk_item.ty().to_resolved().await?;
while let Some((i, item)) = iter.next() {
let ty = item.chunk_item.ty().to_resolved().await?;
let chunk_type = first.chunk_type;
for (i, item) in iter.by_ref() {
let ty = item.chunk_type;
if ty != chunk_type {
let mut map = FxIndexMap::default();
map.insert(chunk_type, this.chunk_items[..i].to_vec());
map.insert(ty, vec![item.clone()]);
map.insert(ty, vec![*item]);
for (_, item) in iter {
map.entry(item.chunk_item.ty().to_resolved().await?)
.or_default()
.push(item.clone());
map.entry(item.chunk_type).or_default().push(*item);
}
return Ok(Vc::cell(
map.into_iter()
......
......@@ -60,7 +60,7 @@ pub async fn expand_batches(
);
let asset_ident = item.chunk_item.asset_ident().to_string();
Ok(ChunkItemOrBatchWithInfo::ChunkItem {
chunk_item: item.clone(),
chunk_item: *item,
size: *size.await?,
asset_ident: asset_ident.owned().await?,
})
......
......@@ -84,6 +84,7 @@ async fn batch_size(
.map(
|&ChunkItemWithAsyncModuleInfo {
chunk_item,
chunk_type: _,
async_info,
module: _,
}| {
......@@ -106,18 +107,21 @@ async fn plain_chunk_items_with_info(
ChunkItemOrBatchWithAsyncModuleInfo::ChunkItem(chunk_item_with_info) => {
let ChunkItemWithAsyncModuleInfo {
chunk_item,
chunk_type,
async_info,
module: _,
} = chunk_item_with_info;
let asset_ident = chunk_item.asset_ident().to_string();
let ty = chunk_item.ty();
let chunk_item_size =
ty.chunk_item_size(chunking_context, *chunk_item, async_info.map(|info| *info));
let chunk_item_size = chunk_type.chunk_item_size(
chunking_context,
*chunk_item,
async_info.map(|info| *info),
);
ChunkItemsWithInfo {
by_type: smallvec![(
ty.to_resolved().await?,
chunk_type,
smallvec![ChunkItemOrBatchWithInfo::ChunkItem {
chunk_item: chunk_item_with_info,
size: *chunk_item_size.await?,
......@@ -162,6 +166,7 @@ async fn plain_chunk_items_with_info_with_type(
ChunkItemOrBatchWithAsyncModuleInfo::ChunkItem(chunk_item_with_info) => {
let &ChunkItemWithAsyncModuleInfo {
chunk_item,
chunk_type: _,
async_info,
module: _,
} = chunk_item_with_info;
......@@ -172,7 +177,7 @@ async fn plain_chunk_items_with_info_with_type(
Ok((
ty,
smallvec![ChunkItemOrBatchWithInfo::ChunkItem {
chunk_item: chunk_item_with_info.clone(),
chunk_item: *chunk_item_with_info,
size: *chunk_item_size.await?,
asset_ident: asset_ident.owned().await?,
}],
......@@ -413,7 +418,7 @@ async fn make_chunk(
.into_iter()
.map(|item| match item {
ChunkItemOrBatchWithInfo::ChunkItem { chunk_item, .. } => {
ChunkItemOrBatchWithAsyncModuleInfo::ChunkItem(chunk_item.clone())
ChunkItemOrBatchWithAsyncModuleInfo::ChunkItem(*chunk_item)
}
&ChunkItemOrBatchWithInfo::Batch { batch, .. } => {
ChunkItemOrBatchWithAsyncModuleInfo::Batch(batch)
......
......@@ -48,7 +48,7 @@ pub async fn make_style_production_chunks(
} else {
make_chunk(
vec![&ChunkItemOrBatchWithInfo::ChunkItem {
chunk_item: chunk_item.clone(),
chunk_item: *chunk_item,
size: 0,
asset_ident: rcstr!(""),
}],
......
......@@ -160,13 +160,13 @@ impl ChunkGroupResult {
#[turbo_tasks::value_impl]
impl ChunkGroupResult {
#[turbo_tasks::function]
pub async fn output_assets_with_referenced(&self) -> Result<Vc<OutputAssetsWithReferenced>> {
Ok(OutputAssetsWithReferenced {
pub fn output_assets_with_referenced(&self) -> Vc<OutputAssetsWithReferenced> {
OutputAssetsWithReferenced {
assets: self.assets,
referenced_assets: self.referenced_assets,
references: self.references,
}
.cell())
.cell()
}
#[turbo_tasks::function]
......
......@@ -8,7 +8,7 @@ pub(crate) mod chunking_context;
pub(crate) mod data;
pub(crate) mod evaluate;
use std::fmt::Display;
use std::{fmt::Display, hash::Hash};
use anyhow::{Result, bail};
use auto_hash_map::AutoSet;
......@@ -453,16 +453,14 @@ pub trait ChunkItem: OutputAssetsReference {
}
/// The type of chunk this item should be assembled into.
#[turbo_tasks::function]
fn ty(self: Vc<Self>) -> Vc<Box<dyn ChunkType>>;
fn ty(&self) -> Vc<Box<dyn ChunkType>>;
/// A temporary method to retrieve the module associated with this
/// ChunkItem. TODO: Remove this as part of the chunk refactoring.
#[turbo_tasks::function]
fn module(self: Vc<Self>) -> Vc<Box<dyn Module>>;
#[turbo_tasks::function]
fn chunking_context(self: Vc<Self>) -> Vc<Box<dyn ChunkingContext>>;
fn chunking_context(&self) -> Vc<Box<dyn ChunkingContext>>;
}
#[turbo_tasks::value_trait]
......@@ -514,10 +512,11 @@ impl AsyncModuleInfo {
}
#[derive(
Debug, Clone, PartialEq, Eq, Hash, TraceRawVcs, TaskInput, NonLocalValue, Encode, Decode,
Debug, Clone, Copy, PartialEq, Eq, Hash, TraceRawVcs, TaskInput, NonLocalValue, Encode, Decode,
)]
pub struct ChunkItemWithAsyncModuleInfo {
pub chunk_item: ResolvedVc<Box<dyn ChunkItem>>,
pub chunk_type: ResolvedVc<Box<dyn ChunkType>>,
pub module: Option<ResolvedVc<Box<dyn ChunkableModule>>>,
pub async_info: Option<ResolvedVc<AsyncModuleInfo>>,
}
......@@ -535,6 +534,8 @@ where
async fn id(self: Vc<Self>) -> Result<ModuleId> {
let chunk_item = Vc::upcast_non_strict(self);
chunk_item
.into_trait_ref()
.await?
.chunking_context()
.chunk_item_id_strategy()
.await?
......
......@@ -86,7 +86,6 @@ pub async fn children_from_module_references(
for &module in reference
.resolve_reference()
.to_resolved()
.await?
.primary_modules()
.await?
......
......@@ -12,7 +12,7 @@ use turbo_tasks::{
use crate::{
chunk::{
ChunkItem, ChunkItemBatchWithAsyncModuleInfo, ChunkItemWithAsyncModuleInfo, ChunkType,
ChunkItemBatchWithAsyncModuleInfo, ChunkItemWithAsyncModuleInfo, ChunkType,
ChunkableModule, ChunkingContext, chunk_item_batch::attach_async_info_to_chunkable_module,
},
module::{Module, StyleModule, StyleType},
......@@ -232,8 +232,8 @@ pub async fn compute_style_groups(
chunking_context,
)
.await?;
let ty = chunk_item.chunk_item.ty();
let size = *ty
let size = *chunk_item
.chunk_type
.chunk_item_size(chunking_context, *chunk_item.chunk_item, None)
.await?;
Ok((chunk_item, size))
......@@ -271,7 +271,7 @@ pub async fn compute_style_groups(
// The list of modules and chunk items that go into the new chunk
let mut new_chunk_modules = [module].into_iter().collect::<FxHashSet<_>>();
let mut new_chunk_items = vec![info.chunk_item.as_ref().unwrap().clone()];
let mut new_chunk_items = vec![info.chunk_item.unwrap()];
// The current size of the new chunk
let mut current_size = info.size;
......@@ -374,7 +374,7 @@ pub async fn compute_style_groups(
}
}
new_chunk_items.push(info.chunk_item.as_ref().unwrap().clone());
new_chunk_items.push(info.chunk_item.unwrap());
new_chunk_modules.insert(module);
*ordered_modules_with_state.get_mut(&module).unwrap() = true;
continue 'outer;
......
......@@ -178,15 +178,7 @@ pub async fn primary_referenced_modules(module: Vc<Box<dyn Module>>) -> Result<V
.references()
.await?
.iter()
.map(|reference| async {
reference
.resolve_reference()
.to_resolved()
.await?
.primary_modules()
.owned()
.await
})
.map(|reference| async { reference.resolve_reference().await?.primary_modules().await })
.try_join()
.await?
.into_iter()
......@@ -231,7 +223,7 @@ pub async fn primary_chunkable_referenced_modules(
let resolved = reference
.resolve_reference()
.await?
.primary_modules_ref()
.primary_modules()
.await?;
let binding_usage = if include_binding_usage {
trait_ref.binding_usage()
......
......@@ -28,7 +28,7 @@ pub async fn handle_resolve_error(
) -> Result<Vc<ModuleResolveResult>> {
Ok(match result.await {
Ok(result_ref) => {
if result_ref.is_unresolvable_ref() {
if result_ref.is_unresolvable() {
emit_unresolvable_issue(
error_mode,
origin,
......@@ -71,7 +71,7 @@ pub async fn handle_resolve_source_error(
) -> Result<Vc<ResolveResult>> {
Ok(match result.await {
Ok(result_ref) => {
if result_ref.is_unresolvable_ref() {
if result_ref.is_unresolvable() {
emit_unresolvable_issue(
error_mode,
origin,
......
use anyhow::Result;
use rustc_hash::FxHashSet;
use turbo_rcstr::RcStr;
use turbo_tasks::{ResolvedVc, Vc};
use turbo_tasks::{ReadRef, ResolvedVc, Vc};
use turbo_tasks_fs::{FileSystemPath, glob::Glob};
use crate::{
......@@ -10,12 +9,17 @@ use crate::{
};
/// A condition which determines if the hooks of a resolve plugin gets called.
#[turbo_tasks::value(shared)]
///
/// The glob is read at construction time and stored as a `ReadRef`, so `matches` is a pure
/// sync function. `serialization = "skip"` because serializing a `ReadRef` is wasteful and
/// recomputing this is very cheap.
#[turbo_tasks::value(serialization = "skip")]
pub enum AfterResolvePluginCondition {
Glob {
root: FileSystemPath,
glob: ResolvedVc<Glob>,
glob: ReadRef<Glob>,
},
// these variants are used by utoo
Always,
Never,
}
......@@ -23,38 +27,31 @@ pub enum AfterResolvePluginCondition {
#[turbo_tasks::value_impl]
impl AfterResolvePluginCondition {
#[turbo_tasks::function]
pub fn new_with_glob(root: FileSystemPath, glob: ResolvedVc<Glob>) -> Vc<Self> {
AfterResolvePluginCondition::Glob { root, glob }.cell()
pub async fn new_with_glob(root: FileSystemPath, glob: ResolvedVc<Glob>) -> Result<Vc<Self>> {
let glob = glob.await?;
Ok(AfterResolvePluginCondition::Glob { root, glob }.cell())
}
}
#[turbo_tasks::value_impl]
impl AfterResolvePluginCondition {
#[turbo_tasks::function]
pub async fn matches(&self, fs_path: FileSystemPath) -> Result<Vc<bool>> {
/// Test whether `fs_path` matches this condition.
pub fn matches(&self, fs_path: &FileSystemPath) -> bool {
match self {
AfterResolvePluginCondition::Glob { root, glob } => {
let path = fs_path;
if let Some(path) = root.get_path_to(&path)
&& glob.await?.matches(path)
{
return Ok(Vc::cell(true));
}
Ok(Vc::cell(false))
root.get_path_to(fs_path).is_some_and(|p| glob.matches(p))
}
AfterResolvePluginCondition::Always => Ok(Vc::cell(true)),
AfterResolvePluginCondition::Never => Ok(Vc::cell(false)),
AfterResolvePluginCondition::Always => true,
AfterResolvePluginCondition::Never => false,
}
}
}
/// A condition which determines if the hooks of a resolve plugin gets called.
#[turbo_tasks::value(shared)]
#[turbo_tasks::value(shared, serialization = "skip")]
pub enum BeforeResolvePluginCondition {
Request(ResolvedVc<Glob>),
Modules(FxHashSet<RcStr>),
Request(ReadRef<Glob>),
Modules(ReadRef<Vec<RcStr>>),
// These are used by utoo
Always,
Never,
}
......@@ -63,26 +60,25 @@ pub enum BeforeResolvePluginCondition {
impl BeforeResolvePluginCondition {
#[turbo_tasks::function]
pub async fn from_modules(modules: ResolvedVc<Vec<RcStr>>) -> Result<Vc<Self>> {
Ok(BeforeResolvePluginCondition::Modules(modules.await?.iter().cloned().collect()).cell())
Ok(BeforeResolvePluginCondition::Modules(modules.await?).cell())
}
#[turbo_tasks::function]
pub fn from_request_glob(glob: ResolvedVc<Glob>) -> Vc<Self> {
BeforeResolvePluginCondition::Request(glob).cell()
pub async fn from_request_glob(glob: ResolvedVc<Glob>) -> Result<Vc<Self>> {
Ok(BeforeResolvePluginCondition::Request(glob.await?).cell())
}
}
#[turbo_tasks::value_impl]
impl BeforeResolvePluginCondition {
#[turbo_tasks::function]
pub async fn matches(&self, request: Vc<Request>) -> Result<Vc<bool>> {
Ok(Vc::cell(match self {
BeforeResolvePluginCondition::Request(glob) => match request.await?.request() {
Some(request) => glob.await?.matches(request.as_str()),
/// Test whether `request` matches this condition.
pub fn matches(&self, request: &Request) -> bool {
match self {
BeforeResolvePluginCondition::Request(glob) => match request.request() {
Some(request) => glob.matches(request.as_str()),
None => false,
},
BeforeResolvePluginCondition::Modules(modules) => {
if let Request::Module { module, .. } = &*request.await? {
if let Request::Module { module, .. } = request {
modules.iter().any(|m| module.is_match(m))
} else {
false
......@@ -90,14 +86,13 @@ impl BeforeResolvePluginCondition {
}
BeforeResolvePluginCondition::Always => true,
BeforeResolvePluginCondition::Never => false,
}))
}
}
}
#[turbo_tasks::value_trait]
pub trait BeforeResolvePlugin {
#[turbo_tasks::function]
fn before_resolve_condition(self: Vc<Self>) -> Vc<BeforeResolvePluginCondition>;
fn before_resolve_condition(&self) -> Vc<BeforeResolvePluginCondition>;
#[turbo_tasks::function]
fn before_resolve(
......@@ -111,8 +106,7 @@ pub trait BeforeResolvePlugin {
#[turbo_tasks::value_trait]
pub trait AfterResolvePlugin {
/// A condition which determines if the hooks gets called.
#[turbo_tasks::function]
fn after_resolve_condition(self: Vc<Self>) -> Vc<AfterResolvePluginCondition>;
fn after_resolve_condition(&self) -> Vc<AfterResolvePluginCondition>;
/// This hook gets called when a full filepath has been resolved and the
/// condition matches. If a value is returned it replaces the resolve
......
......@@ -248,12 +248,10 @@ impl ChunkItem for CssModuleChunkItem {
self.module.ident()
}
#[turbo_tasks::function]
fn chunking_context(&self) -> Vc<Box<dyn ChunkingContext>> {
*self.chunking_context
}
#[turbo_tasks::function]
fn ty(&self) -> Vc<Box<dyn ChunkType>> {
Vc::upcast(Vc::<CssChunkType>::default())
}
......@@ -278,7 +276,6 @@ impl CssChunkItem for CssModuleChunkItem {
{
for &module in import_ref
.resolve_reference()
.to_resolved()
.await?
.primary_modules()
.await?
......@@ -300,7 +297,6 @@ impl CssChunkItem for CssModuleChunkItem {
{
for &module in compose_ref
.resolve_reference()
.to_resolved()
.await?
.primary_modules()
.await?
......
......@@ -284,15 +284,16 @@ impl EcmascriptChunkPlaceable for EcmascriptCssModule {
original: original_name,
from,
} => {
let resolved_module = from.resolve_reference().first_module().await?;
let resolved_module =
from.resolve_reference().await?.first_module().await?;
let Some(resolved_module) = &*resolved_module else {
let Some(resolved_module) = resolved_module else {
// Issue already emitted by CssModuleComposeReference::resolve_reference
continue;
};
let Some(css_module) =
ResolvedVc::try_downcast_type::<EcmascriptCssModule>(*resolved_module)
ResolvedVc::try_downcast_type::<EcmascriptCssModule>(resolved_module)
else {
// Issue already emitted by CssModuleComposeReference::resolve_reference
continue;
......
......@@ -46,10 +46,10 @@ impl ModuleReference for CssModuleComposeReference {
None,
);
let resolved = result.first_module().await?;
let resolved = result.await?.first_module().await?;
let file_path = self.origin.origin_path().to_resolved().await?;
if let Some(module) = &*resolved {
if ResolvedVc::try_downcast_type::<EcmascriptCssModule>(*module).is_none() {
if let Some(module) = resolved {
if ResolvedVc::try_downcast_type::<EcmascriptCssModule>(module).is_none() {
CssModuleComposesIssue {
severity: IssueSeverity::Error,
file_path,
......
......@@ -58,7 +58,7 @@ impl UrlAssetReference {
self: Vc<Self>,
chunking_context: Vc<Box<dyn ChunkingContext>>,
) -> Result<Vc<ReferencedAsset>> {
if let Some(module) = *self.resolve_reference().first_module().await?
if let Some(module) = self.resolve_reference().await?.first_module().await?
&& let Some(embeddable) = ResolvedVc::try_downcast::<Box<dyn CssEmbed>>(module)
{
return Ok(ReferencedAsset::Some(
......
......@@ -57,12 +57,18 @@ impl ChunkType for EcmascriptChunkType {
else {
bail!("Chunk item is not an ecmascript chunk item but reporting chunk type ecmascript");
};
Ok(Vc::cell(
chunk_item
.content_with_async_module_info(async_module_info, true)
.await
.map_or(0, |content| round_chunk_item_size(content.inner_code.len())),
))
let chunk_item = chunk_item.into_trait_ref().await?;
let size = match chunk_item
.content_with_async_module_info(async_module_info, true)
.await
{
Ok(content) => {
let content = content.await?;
round_chunk_item_size(content.inner_code.len())
}
Err(_) => 0,
};
Ok(Vc::cell(size))
}
}
......
use std::io::Write;
use anyhow::{Result, bail};
use async_trait::async_trait;
use bincode::{Decode, Encode};
use smallvec::SmallVec;
use turbo_rcstr::{RcStr, rcstr};
......@@ -240,6 +241,7 @@ impl EcmascriptChunkItemWithAsyncInfo {
) -> Result<EcmascriptChunkItemWithAsyncInfo> {
let ChunkItemWithAsyncModuleInfo {
chunk_item,
chunk_type: _,
module: _,
async_info,
} = chunk_item;
......@@ -255,23 +257,18 @@ impl EcmascriptChunkItemWithAsyncInfo {
}
}
#[async_trait]
#[turbo_tasks::value_trait]
pub trait EcmascriptChunkItem: ChunkItem + OutputAssetsReference {
#[turbo_tasks::function]
fn content(self: Vc<Self>) -> Vc<EcmascriptChunkItemContent>;
/// Fetches the content of the chunk item with async module info.
/// When `estimated` is true, it's ok to provide an estimated content, since it's only used for
/// compute the chunking. When `estimated` is true, this function should not invoke other
/// chunking operations that would cause cycles.
#[turbo_tasks::function]
fn content_with_async_module_info(
self: Vc<Self>,
_async_module_info: Option<Vc<AsyncModuleInfo>>,
_estimated: bool,
) -> Vc<EcmascriptChunkItemContent> {
self.content()
}
async fn content_with_async_module_info(
&self,
async_module_info: Option<Vc<AsyncModuleInfo>>,
estimated: bool,
) -> Result<Vc<EcmascriptChunkItemContent>>;
}
pub trait EcmascriptChunkItemExt {
......@@ -295,13 +292,18 @@ async fn module_factory_with_code_generation_issue(
chunk_item: Vc<Box<dyn EcmascriptChunkItem>>,
async_module_info: Option<Vc<AsyncModuleInfo>>,
) -> Result<Vc<PersistedCode>> {
let content = match chunk_item
.content_with_async_module_info(async_module_info, false)
.await
{
Ok(item) => item.module_factory().await,
Err(err) => Err(err),
};
async fn get_content(
chunk_item: Vc<Box<dyn EcmascriptChunkItem>>,
async_module_info: Option<Vc<AsyncModuleInfo>>,
) -> Result<ResolvedVc<PersistedCode>> {
let chunk_item_ref = chunk_item.into_trait_ref().await?;
let content = chunk_item_ref
.content_with_async_module_info(async_module_info, false)
.await?
.await?;
content.module_factory().await
}
let content = get_content(chunk_item, async_module_info).await;
Ok(match content {
Ok(factory) => *factory,
Err(error) => {
......@@ -372,7 +374,6 @@ impl ChunkItem for EcmascriptModuleChunkItem {
.chunk_item_content_ident(*self.chunking_context, *self.module_graph)
}
#[turbo_tasks::function]
fn ty(&self) -> Vc<Box<dyn ChunkType>> {
Vc::upcast(Vc::<EcmascriptChunkType>::default())
}
......@@ -382,7 +383,6 @@ impl ChunkItem for EcmascriptModuleChunkItem {
Vc::upcast(*self.module)
}
#[turbo_tasks::function]
fn chunking_context(&self) -> Vc<Box<dyn ChunkingContext>> {
*self.chunking_context
}
......@@ -397,25 +397,19 @@ impl OutputAssetsReference for EcmascriptModuleChunkItem {
}
}
#[async_trait]
#[turbo_tasks::value_impl]
impl EcmascriptChunkItem for EcmascriptModuleChunkItem {
#[turbo_tasks::function]
fn content(&self) -> Vc<EcmascriptChunkItemContent> {
self.module
.chunk_item_content(*self.chunking_context, *self.module_graph, None, false)
}
#[turbo_tasks::function]
fn content_with_async_module_info(
async fn content_with_async_module_info(
&self,
async_module_info: Option<Vc<AsyncModuleInfo>>,
estimated: bool,
) -> Vc<EcmascriptChunkItemContent> {
self.module.chunk_item_content(
) -> Result<Vc<EcmascriptChunkItemContent>> {
Ok(self.module.chunk_item_content(
*self.chunking_context,
*self.module_graph,
async_module_info,
estimated,
)
))
}
}
......@@ -778,11 +778,6 @@ impl EcmascriptModuleAsset {
*self.source
}
#[turbo_tasks::function]
pub fn analyze(self: Vc<Self>) -> Vc<AnalyzeEcmascriptModuleResult> {
analyze_ecmascript_module(self, None)
}
#[turbo_tasks::function]
pub fn options(&self) -> Vc<EcmascriptOptions> {
*self.options
......@@ -790,6 +785,10 @@ impl EcmascriptModuleAsset {
}
impl EcmascriptModuleAsset {
pub fn analyze(self: Vc<Self>) -> Vc<AnalyzeEcmascriptModuleResult> {
analyze_ecmascript_module(self, None)
}
pub async fn parse(&self) -> Result<Vc<ParseResult>> {
let options = self.options.await?;
let node_env = self
......
......@@ -331,7 +331,7 @@ impl ReferencedAsset {
pub async fn from_resolve_result(resolve_result: Vc<ModuleResolveResult>) -> Result<Self> {
// TODO handle multiple keyed results
let result = resolve_result.await?;
if result.is_unresolvable_ref() {
if result.is_unresolvable() {
return Ok(ReferencedAsset::Unresolvable);
}
for (_, result) in result.primary.iter() {
......@@ -475,7 +475,7 @@ impl ModuleReference for EsmAssetReference {
loader_request,
origin.resolve_options(),
);
let loader_fs_path = if let Some(source) = *resolved.first_source().await? {
let loader_fs_path = if let Some(source) = resolved.await?.first_source() {
source.ident().await?.path.clone()
} else {
bail!("Unable to resolve turbopackLoader '{}'", loader.loader);
......@@ -544,7 +544,7 @@ impl ModuleReference for EsmAssetReference {
.await?;
if let Some(ModulePart::Export(export_name)) = &self.export_name {
for &module in result.primary_modules().await? {
for &module in result.await?.primary_modules().await?.iter() {
if let Some(module) = ResolvedVc::try_downcast(module)
&& *is_export_missing(*module, export_name.clone()).await?
{
......
......@@ -3622,10 +3622,9 @@ async fn require_resolve_visitor(
.await?;
let mut values =
resolved
.primary_sources()
.await?
.iter()
.map(|&source| async move {
.primary_sources()
.map(|source| async move {
Ok(require_resolve(source.ident().await?.path.clone()).into())
})
.try_join()
......
......@@ -85,7 +85,7 @@ pub(crate) enum PatternMapping {
/// ```js
/// require(`./images/${name}.png`)
/// ```
Map(#[bincode(with = "turbo_bincode::indexmap")] FxIndexMap<String, SinglePatternMapping>),
Map(#[bincode(with = "turbo_bincode::indexmap")] FxIndexMap<RcStr, SinglePatternMapping>),
}
#[derive(
......@@ -231,7 +231,7 @@ enum ImportMode {
}
fn create_context_map(
map: &FxIndexMap<String, SinglePatternMapping>,
map: &FxIndexMap<RcStr, SinglePatternMapping>,
key_expr: &Expr,
import_mode: ImportMode,
) -> Expr {
......@@ -304,6 +304,7 @@ async fn to_single_pattern_mapping(
origin: Vc<Box<dyn ResolveOrigin>>,
chunking_context: Vc<Box<dyn ChunkingContext>>,
resolve_item: &ModuleResolveResultItem,
primary: &[(turbopack_core::resolve::RequestKey, ModuleResolveResultItem)],
resolve_type: ResolveType,
) -> Result<SinglePatternMapping> {
let module = match resolve_item {
......@@ -328,6 +329,16 @@ async fn to_single_pattern_mapping(
.to_unstyled_string(),
));
}
ModuleResolveResultItem::Duplicate(first) => {
return Box::pin(to_single_pattern_mapping(
origin,
chunking_context,
&primary[*first].1,
primary,
resolve_type,
))
.await;
}
ModuleResolveResultItem::Empty | ModuleResolveResultItem::Custom(_) => {
// TODO implement mapping
CodeGenerationIssue {
......@@ -405,24 +416,37 @@ impl PatternMapping {
.cell()),
1 if !request.request_pattern().await?.has_dynamic_parts() => {
let resolve_item = &result.primary.first().unwrap().1;
let single_pattern_mapping =
to_single_pattern_mapping(origin, chunking_context, resolve_item, resolve_type)
.await?;
let single_pattern_mapping = to_single_pattern_mapping(
origin,
chunking_context,
resolve_item,
&result.primary,
resolve_type,
)
.await?;
Ok(PatternMapping::Single(single_pattern_mapping).cell())
}
_ => {
let primary = &result.primary;
let mut set = HashSet::new();
let map = result
.primary
let items: Vec<(RcStr, &ModuleResolveResultItem)> = primary
.iter()
.filter_map(|(k, v)| {
let request = k.request.as_ref()?;
set.insert(request).then(|| (request.to_string(), v))
set.insert(request).then(|| (request.clone(), v))
})
.collect();
let map = items
.into_iter()
.map(|(k, v)| async move {
let single_pattern_mapping =
to_single_pattern_mapping(origin, chunking_context, v, resolve_type)
.await?;
let single_pattern_mapping = to_single_pattern_mapping(
origin,
chunking_context,
v,
primary,
resolve_type,
)
.await?;
Ok((k, single_pattern_mapping))
})
.try_join()
......
......@@ -661,7 +661,7 @@ impl EvaluateContext for WebpackLoaderContext {
options,
);
if let Some(source) = *resolved.first_source().await? {
if let Some(source) = resolved.await?.first_source() {
if let Some(path) = self.cwd.get_relative_path_to(&source.ident().await?.path) {
Ok(ResponseMessage::Resolve { path })
} else {
......@@ -702,7 +702,7 @@ impl EvaluateContext for WebpackLoaderContext {
)
.await?;
let Some(module) = *resolved.first_module().await? else {
let Some(module) = resolved.await?.first_module().await? else {
bail!(
"importModule: unable to resolve {} in {}",
request,
......
......@@ -95,10 +95,9 @@ async fn resolve_node_pre_gyp_files(
collect_affecting_sources,
true,
)
.first_source()
.await?;
let compile_target = compile_target.await?;
if let Some(config_asset) = *config
if let Some(config_asset) = config.first_source()
&& let AssetContent::File(file) = &*config_asset.content().await?
&& let FileContent::Content(config_file) = &*file.await?
{
......@@ -273,9 +272,11 @@ async fn resolve_node_gyp_build_files(
collect_affecting_sources,
true,
);
if let [binding_gyp] = &gyp_file.primary_sources().await?[..] {
let gyp_file = gyp_file.await?;
let mut primary_sources = gyp_file.primary_sources();
if let (Some(binding_gyp), None) = (primary_sources.next(), primary_sources.next()) {
let mut merged_affecting_sources = if collect_affecting_sources {
gyp_file.await?.get_affecting_sources().collect::<Vec<_>>()
gyp_file.get_affecting_sources().collect::<Vec<_>>()
} else {
Vec::new()
};
......@@ -399,9 +400,8 @@ async fn resolve_node_bindings_files(
collect_affecting_sources,
true,
)
.first_source()
.await?;
if let Some(asset) = *resolved
if let Some(asset) = resolved.first_source()
&& let AssetContent::File(file) = &*asset.content().await?
&& let FileContent::Content(_) = &*file.await?
{
......
......@@ -25,7 +25,7 @@ use turbopack_core::{
pattern::Pattern,
resolve,
},
source::{OptionSource, Source},
source::Source,
};
use crate::ecmascript::get_condition_maps;
......@@ -88,7 +88,7 @@ pub async fn read_tsconfigs(
configs.push((parsed_data, tsconfig));
if let Some(extends) = json["extends"].as_str() {
let resolved = resolve_extends(*tsconfig, extends, resolve_options).await?;
if let Some(source) = *resolved.await? {
if let Some(source) = resolved {
data = source.content().file_content();
tsconfig = source;
continue;
......@@ -118,7 +118,7 @@ async fn resolve_extends(
tsconfig: Vc<Box<dyn Source>>,
extends: &str,
resolve_options: Vc<ResolveOptions>,
) -> Result<Vc<OptionSource>> {
) -> Result<Option<ResolvedVc<Box<dyn Source>>>> {
let parent_dir = tsconfig.ident().await?.path.parent();
let request = Request::parse_string(extends.into());
......@@ -148,18 +148,18 @@ async fn resolve_extends(
Request::Empty => {
let request = Request::parse_string(rcstr!("./tsconfig"));
Ok(resolve(parent_dir,
ReferenceType::TypeScript(TypeScriptReferenceSubType::Undefined), request, resolve_options).first_source())
ReferenceType::TypeScript(TypeScriptReferenceSubType::Undefined), request, resolve_options).await?.first_source())
}
// All other types are treated as module imports, and potentially joined with
// "tsconfig.json". This includes "relative" imports like '.' and '..'.
_ => {
let mut result = resolve(parent_dir.clone(), ReferenceType::TypeScript(TypeScriptReferenceSubType::Undefined), request, resolve_options).first_source();
if result.await?.is_none() {
let request = Request::parse_string(format!("{extends}/tsconfig").into());
result = resolve(parent_dir, ReferenceType::TypeScript(TypeScriptReferenceSubType::Undefined), request, resolve_options).first_source();
let result = resolve(parent_dir.clone(), ReferenceType::TypeScript(TypeScriptReferenceSubType::Undefined), request, resolve_options).await?;
if let Some(source) = result.first_source() {
return Ok(Some(source));
}
Ok(result)
let request = Request::parse_string(format!("{extends}/tsconfig").into());
Ok(resolve(parent_dir, ReferenceType::TypeScript(TypeScriptReferenceSubType::Undefined), request, resolve_options).await?.first_source())
}
}
}
......@@ -169,19 +169,21 @@ async fn resolve_extends_rooted_or_relative(
request: Vc<Request>,
resolve_options: Vc<ResolveOptions>,
path: &str,
) -> Result<Vc<OptionSource>> {
let mut result = resolve(
) -> Result<Option<ResolvedVc<Box<dyn Source>>>> {
let result = resolve(
lookup_path.clone(),
ReferenceType::TypeScript(TypeScriptReferenceSubType::Undefined),
request,
resolve_options,
)
.first_source();
.await?;
let mut result = result.first_source();
// If the file doesn't end with ".json" and we can't find the file, then we have
// to try again with it.
// https://github.com/microsoft/TypeScript/blob/611a912d/src/compiler/commandLineParser.ts#L3305
if !path.ends_with(".json") && result.await?.is_none() {
if !path.ends_with(".json") && result.is_none() {
let request = Request::parse_string(format!("{path}.json").into());
result = resolve(
lookup_path.clone(),
......@@ -189,6 +191,7 @@ async fn resolve_extends_rooted_or_relative(
request,
resolve_options,
)
.await?
.first_source();
}
Ok(result)
......@@ -424,7 +427,7 @@ pub async fn type_resolve(
request,
options,
);
if !*result1.is_unresolvable().await? {
if !result1.await?.is_unresolvable() {
result1
} else {
resolve(
......
......@@ -421,20 +421,12 @@ impl ModuleAssetContext {
*self.resolve_options_context
}
#[turbo_tasks::function]
pub async fn is_types_resolving_enabled(&self) -> Result<Vc<bool>> {
let resolve_options_context = self.resolve_options_context.await?;
Ok(Vc::cell(
resolve_options_context.enable_types && resolve_options_context.enable_typescript,
))
}
#[turbo_tasks::function]
pub async fn with_types_resolving_enabled(self: Vc<Self>) -> Result<Vc<ModuleAssetContext>> {
if *self.is_types_resolving_enabled().await? {
let this = self.await?;
if this.is_types_resolving_enabled().await? {
return Ok(self);
}
let this = self.await?;
let resolve_options_context = *this
.resolve_options_context
.with_types_enabled()
......@@ -452,6 +444,10 @@ impl ModuleAssetContext {
}
impl ModuleAssetContext {
async fn is_types_resolving_enabled(&self) -> Result<bool> {
let resolve_options_context = self.resolve_options_context.await?;
Ok(resolve_options_context.enable_types && resolve_options_context.enable_typescript)
}
async fn process_with_transition_rules(
self: Vc<Self>,
source: ResolvedVc<Box<dyn Source>>,
......@@ -1025,8 +1021,8 @@ impl AssetContext for ModuleAssetContext {
);
let mut result = self.process_resolve_result(*result.to_resolved().await?, reference_type);
if *self.is_types_resolving_enabled().await? {
let this = self.await?;
if this.is_types_resolving_enabled().await? {
let types_result = type_resolve(
Vc::upcast(PlainResolveOrigin::new(Vc::upcast(self), origin_path)),
request,
......
......@@ -12,11 +12,20 @@ Then run this script with the path to the stats.json file to get a report on cac
The JSON format contains entries like:
{ "task_name": { "cache_hit": N, "cache_miss": N } }
Usage:
analyze_cache_effectiveness.py <stats.json>
analyze_cache_effectiveness.py --diff <before.json> <after.json> [--top N]
In --diff mode, the script reports the tasks whose hits or misses changed the
most between the two runs — useful for evaluating the impact of removing or
adding `turbo_tasks` caching on specific functions.
"""
import argparse
import json
import sys
from typing import List, Tuple
from typing import Dict, List
from dataclasses import dataclass
......@@ -54,6 +63,10 @@ def load_task_stats(file_path: str) -> List[TaskStats]:
return tasks
def load_task_stats_map(file_path: str) -> Dict[str, TaskStats]:
return {t.name: t for t in load_task_stats(file_path)}
def analyze_tasks(tasks: List[TaskStats]) -> List[TaskStats]:
"""Analyze all tasks and return sorted by wasted cache overhead.
......@@ -114,27 +127,213 @@ def print_analysis(tasks: List[TaskStats]):
print(f"Tasks with <50% hit rate: {low_hit_rate_count}")
def main():
if len(sys.argv) != 2:
print("Usage: python analyze_cache_effectiveness.py <stats.json>")
sys.exit(1)
@dataclass
class TaskDiff:
name: str
before_hit: int
after_hit: int
before_miss: int
after_miss: int
@property
def delta_hit(self) -> int:
return self.after_hit - self.before_hit
file_path = sys.argv[1]
@property
def delta_miss(self) -> int:
return self.after_miss - self.before_miss
try:
tasks = load_task_stats(file_path)
tasks = analyze_tasks(tasks)
print_analysis(tasks)
@staticmethod
def _rate(hit: int, miss: int) -> float:
total = hit + miss
return hit / total if total > 0 else 0.0
@property
def before_rate(self) -> float:
return self._rate(self.before_hit, self.before_miss)
@property
def after_rate(self) -> float:
return self._rate(self.after_hit, self.after_miss)
@property
def delta_rate(self) -> float:
return self.after_rate - self.before_rate
def compute_diff(
before: Dict[str, TaskStats], after: Dict[str, TaskStats]
) -> List[TaskDiff]:
names = set(before) | set(after)
diffs = []
zero = TaskStats(name="", cache_hit=0, cache_miss=0)
for name in names:
b = before.get(name, zero)
a = after.get(name, zero)
diffs.append(
TaskDiff(
name=name,
before_hit=b.cache_hit,
after_hit=a.cache_hit,
before_miss=b.cache_miss,
after_miss=a.cache_miss,
)
)
return diffs
def _print_diff_section(
title: str, diffs: List[TaskDiff], key, top: int, reverse: bool
):
print("=" * 100)
print(title)
print("=" * 100)
ordered = sorted(diffs, key=key, reverse=reverse)
header = (
f"{'Δhit':>12} {'Δmiss':>12} {'hit b→a':>22} {'miss b→a':>22} "
f"{'rate b→a':>18} Task"
)
print(header)
print("-" * len(header))
shown = 0
for d in ordered:
# Stop once the signed delta crosses zero in the direction we care about.
v = key(d)
if reverse and v <= 0:
break
if not reverse and v >= 0:
break
hit_str = f"{d.before_hit:,}→{d.after_hit:,}"
miss_str = f"{d.before_miss:,}→{d.after_miss:,}"
rate_str = f"{d.before_rate:.0%}→{d.after_rate:.0%}"
print(
f"{d.delta_hit:>+12,} {d.delta_miss:>+12,} "
f"{hit_str:>22} {miss_str:>22} {rate_str:>18} {d.name}"
)
shown += 1
if shown >= top:
break
if shown == 0:
print("(none)")
print()
def print_diff(
before: Dict[str, TaskStats], after: Dict[str, TaskStats], top: int
):
diffs = compute_diff(before, after)
before_hits = sum(t.cache_hit for t in before.values())
after_hits = sum(t.cache_hit for t in after.values())
before_misses = sum(t.cache_miss for t in before.values())
after_misses = sum(t.cache_miss for t in after.values())
before_total = before_hits + before_misses
after_total = after_hits + after_misses
before_rate = before_hits / before_total if before_total > 0 else 0.0
after_rate = after_hits / after_total if after_total > 0 else 0.0
only_before = set(before) - set(after)
only_after = set(after) - set(before)
except FileNotFoundError:
print(f"Error: File '{file_path}' not found")
print("Cache statistics diff (before → after)")
print()
print(
f"Hits: {before_hits:>12,} → {after_hits:>12,} "
f"({after_hits - before_hits:+,})"
)
print(
f"Misses: {before_misses:>12,} → {after_misses:>12,} "
f"({after_misses - before_misses:+,})"
)
print(
f"Hit rate: {before_rate:>11.2%} → {after_rate:>11.2%} "
f"({(after_rate - before_rate) * 100:+.2f} pp)"
)
print(
f"Tasks: {len(before):>12,} → {len(after):>12,} "
f"(only in before: {len(only_before)}, only in after: {len(only_after)})"
)
print()
_print_diff_section(
f"Top {top} INCREASES IN HITS (after > before)",
diffs,
key=lambda d: d.delta_hit,
top=top,
reverse=True,
)
_print_diff_section(
f"Top {top} INCREASES IN MISSES (after > before)",
diffs,
key=lambda d: d.delta_miss,
top=top,
reverse=True,
)
_print_diff_section(
f"Top {top} DECREASES IN HITS (after < before)",
diffs,
key=lambda d: d.delta_hit,
top=top,
reverse=False,
)
_print_diff_section(
f"Top {top} DECREASES IN MISSES (after < before)",
diffs,
key=lambda d: d.delta_miss,
top=top,
reverse=False,
)
def main():
parser = argparse.ArgumentParser(
description=(
"Analyze turbo_tasks cache effectiveness, or diff two stats files "
"to see which tasks gained or lost hits/misses."
),
)
parser.add_argument(
"--diff",
action="store_true",
help="Diff mode: compare two stats files and report largest changes.",
)
parser.add_argument(
"--top",
type=int,
default=25,
help="Number of entries to show per section in --diff mode (default: 25).",
)
parser.add_argument(
"files",
nargs="+",
help=(
"Path to stats JSON. In default mode, one file. "
"In --diff mode, two files: <before> <after>."
),
)
args = parser.parse_args()
try:
if args.diff:
if len(args.files) != 2:
parser.error("--diff requires exactly two files: <before> <after>")
before = load_task_stats_map(args.files[0])
after = load_task_stats_map(args.files[1])
print_diff(before, after, args.top)
else:
if len(args.files) != 1:
parser.error("default mode requires exactly one stats file")
tasks = load_task_stats(args.files[0])
tasks = analyze_tasks(tasks)
print_analysis(tasks)
except FileNotFoundError as e:
print(f"Error: File not found: {e.filename}")
sys.exit(1)
except json.JSONDecodeError as e:
print(f"Error parsing JSON: {e}")
sys.exit(1)
except Exception as e:
print(f"Error: {e}")
sys.exit(1)
if __name__ == "__main__":
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment