Unverified Commit 236a76dd authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

[turbopack] Remove `turbo_tasks::function` from ModuleReference getters (#91229)

### What?

Refactors the `ModuleReference` trait to make `chunking_type()` and `binding_usage()` methods return direct values instead of `Vc<T>` wrapped values, removing the need for async task functions.

Also removes the `get_referenced_asset` task from `EsmAssetReference`, inlining its logic into the callers.

### Why?

This change simplifies the API by eliminating unnecessary async overhead for methods that typically return simple, computed values. The previous implementation required `#[turbo_tasks::function]` annotations and `Vc<T>` wrappers even when the methods didn't need to perform async operations or benefit from caching.

### Impact

| Metric | Base | Change | Delta |
|--------|------|--------|-------|
| Hits | 35,678,143 | 35,845,124 | **+166,981** |
| Misses | 9,418,378 | 7,910,986 | **-1,507,392** |
| Total | 45,096,521 | 43,756,110 | **-1,340,411** |
| Task types | 1,306 | 1,277 | **-29** |

29 task types were removed, eliminating **2.6M total task invocations** (1.1M hits + 1.5M misses):

- **`chunking_type`** — 21 task types removed across all `ModuleReference` implementors (~952k invocations)
- **`binding_usage`** — 6 task types removed (~527k invocations)
- **`BindingUsage::all`** — helper task removed (~36k invocations)
- **`EsmAssetReference::get_referenced_asset`** — removed and inlined (~1.08M invocations: 628k hits + 451k misses)

The removed `get_referenced_asset` hits reappear as +628k hits on `EsmAssetReference::resolve_reference` and `ReferencedAsset::from_resolve_result` (with zero increase in misses), confirming the work is now served from cache through the existing callers.

No tasks had increased misses — the removal is clean with no cache invalidation spillover.

I also ran some builds to measure latency

```
# This branch
$ hyperfine -p 'rm -rf .next' -w 2 -r 10  'pnpm next build --turbopack --experimental-build-mode=compile'
Benchmark 1: pnpm next build --turbopack --experimental-build-mode=compile
  Time (mean ± σ):     52.752 s ±  0.658 s    [User: 376.575 s, System: 106.375 s]
  Range (min … max):   51.913 s … 54.161 s    10 runs

# on canary
$ hyperfine -p 'rm -rf .next' -w 2 -r 10  'pnpm next build --turbopack --experimental-build-mode=compile'
Benchmark 1: pnpm next build --turbopack --experimental-build-mode=compile
  Time (mean ± σ):     54.675 s ±  1.394 s    [User: 389.273 s, System: 114.642 s]
  Range (min … max):   53.434 s … 58.189 s    10 runs
```

so a solid win of almost 2 seconds

MaxRSS also went from 16,474,324,992 bytes to 16,359,309,312 bytes (from one measurement) so a savings of ~100M of max heap size.


### How?

- Changed `chunking_type()` method signature from `Vc<ChunkingTypeOption>` to `Option<ChunkingType>`
- Changed `binding_usage()` method signature from `Vc<BindingUsage>` to `BindingUsage`
- Removed `ChunkingTypeOption` type alias as it's no longer needed
- Updated all implementations across the codebase to return direct values instead of wrapped ones
- Removed `#[turbo_tasks::function]` annotations from these methods
- Updated call sites to use `into_trait_ref().await?` pattern when accessing these methods from `Vc<dyn ModuleReference>`
- Removed `EsmAssetReference::get_referenced_asset`, inlining its logic into callers
- Added validation for `turbopack-chunking-type` annotation values in import analysis
- Fixed cache effectiveness analysis script
parent 00067f40
...@@ -7,7 +7,7 @@ use turbo_rcstr::{RcStr, rcstr}; ...@@ -7,7 +7,7 @@ use turbo_rcstr::{RcStr, rcstr};
use turbo_tasks::{ use turbo_tasks::{
FxIndexMap, ReadRef, ResolvedVc, TryFlatJoinIterExt, TryJoinIterExt, ValueToString, Vc, FxIndexMap, ReadRef, ResolvedVc, TryFlatJoinIterExt, TryJoinIterExt, ValueToString, Vc,
graph::{AdjacencyMap, GraphTraversal, Visit}, graph::{AdjacencyMap, GraphTraversal, Visit},
turbobail, turbofmt, turbofmt,
}; };
use turbo_tasks_fs::{ use turbo_tasks_fs::{
DirectoryEntry, File, FileContent, FileSystem, FileSystemPath, DirectoryEntry, File, FileContent, FileSystem, FileSystemPath,
...@@ -287,7 +287,7 @@ impl Asset for NftJsonAsset { ...@@ -287,7 +287,7 @@ impl Asset for NftJsonAsset {
&*current_path.get_type().await?, &*current_path.get_type().await?,
FileSystemEntryType::Symlink FileSystemEntryType::Symlink
) { ) {
turbobail!( turbo_tasks::turbobail!(
"Encountered file inside of symlink in NFT list: {current_path} \ "Encountered file inside of symlink in NFT list: {current_path} \
is a symlink, but {referenced_chunk_path} was created inside of \ is a symlink, but {referenced_chunk_path} was created inside of \
it" it"
......
...@@ -8,7 +8,7 @@ use turbopack_core::{ ...@@ -8,7 +8,7 @@ use turbopack_core::{
asset::{Asset, AssetContent}, asset::{Asset, AssetContent},
chunk::{ chunk::{
AsyncModuleInfo, ChunkItem, ChunkableModule, ChunkingContext, ChunkingType, AsyncModuleInfo, ChunkItem, ChunkableModule, ChunkingContext, ChunkingType,
ChunkingTypeOption, EvaluatableAsset, EvaluatableAsset,
}, },
ident::AssetIdent, ident::AssetIdent,
module::{Module, ModuleSideEffects}, module::{Module, ModuleSideEffects},
...@@ -164,11 +164,10 @@ impl ModuleReference for HmrEntryModuleReference { ...@@ -164,11 +164,10 @@ impl ModuleReference for HmrEntryModuleReference {
*ModuleResolveResult::module(self.module) *ModuleResolveResult::module(self.module)
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(&self) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Parallel {
Vc::cell(Some(ChunkingType::Parallel {
inherit_async: false, inherit_async: false,
hoisted: false, hoisted: false,
})) })
} }
} }
...@@ -4,7 +4,7 @@ use turbo_tasks::{ResolvedVc, ValueToString, Vc}; ...@@ -4,7 +4,7 @@ use turbo_tasks::{ResolvedVc, ValueToString, Vc};
use turbo_tasks_fs::FileContent; use turbo_tasks_fs::FileContent;
use turbopack_core::{ use turbopack_core::{
asset::{Asset, AssetContent}, asset::{Asset, AssetContent},
chunk::{ChunkGroupType, ChunkingType, ChunkingTypeOption}, chunk::{ChunkGroupType, ChunkingType},
ident::AssetIdent, ident::AssetIdent,
module::{Module, ModuleSideEffects}, module::{Module, ModuleSideEffects},
reference::{ModuleReference, ModuleReferences}, reference::{ModuleReference, ModuleReferences},
...@@ -99,11 +99,10 @@ impl ModuleReference for CssClientReference { ...@@ -99,11 +99,10 @@ impl ModuleReference for CssClientReference {
*ModuleResolveResult::module(self.module) *ModuleResolveResult::module(self.module)
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(&self) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Isolated {
Vc::cell(Some(ChunkingType::Isolated {
_ty: ChunkGroupType::Evaluated, _ty: ChunkGroupType::Evaluated,
merge_tag: Some(rcstr!("client")), merge_tag: Some(rcstr!("client")),
})) })
} }
} }
...@@ -9,7 +9,7 @@ use turbopack_core::{ ...@@ -9,7 +9,7 @@ use turbopack_core::{
asset::AssetContent, asset::AssetContent,
chunk::{ chunk::{
AsyncModuleInfo, ChunkGroupType, ChunkItem, ChunkType, ChunkableModule, ChunkingContext, AsyncModuleInfo, ChunkGroupType, ChunkItem, ChunkType, ChunkableModule, ChunkingContext,
ChunkingType, ChunkingTypeOption, ChunkingType,
}, },
code_builder::CodeBuilder, code_builder::CodeBuilder,
context::AssetContext, context::AssetContext,
...@@ -384,11 +384,10 @@ impl ModuleReference for EcmascriptClientReference { ...@@ -384,11 +384,10 @@ impl ModuleReference for EcmascriptClientReference {
*ModuleResolveResult::module(self.module) *ModuleResolveResult::module(self.module)
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(&self) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Isolated {
Vc::cell(Some(ChunkingType::Isolated {
_ty: self.ty, _ty: self.ty,
merge_tag: self.merge_tag.clone(), merge_tag: self.merge_tag.clone(),
})) })
} }
} }
use turbo_tasks::{ResolvedVc, ValueToString, Vc}; use turbo_tasks::{ResolvedVc, ValueToString, Vc};
use turbopack_core::{ use turbopack_core::{
chunk::{ChunkingType, ChunkingTypeOption}, chunk::ChunkingType, module::Module, reference::ModuleReference, resolve::ModuleResolveResult,
module::Module,
reference::ModuleReference,
resolve::ModuleResolveResult,
}; };
#[turbo_tasks::value] #[turbo_tasks::value]
...@@ -27,11 +24,10 @@ impl ModuleReference for NextServerComponentModuleReference { ...@@ -27,11 +24,10 @@ impl ModuleReference for NextServerComponentModuleReference {
fn resolve_reference(&self) -> Vc<ModuleResolveResult> { fn resolve_reference(&self) -> Vc<ModuleResolveResult> {
*ModuleResolveResult::module(self.asset) *ModuleResolveResult::module(self.asset)
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(&self) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Shared {
Vc::cell(Some(ChunkingType::Shared {
inherit_async: true, inherit_async: true,
merge_tag: None, merge_tag: None,
})) })
} }
} }
...@@ -2,10 +2,7 @@ use once_cell::sync::Lazy; ...@@ -2,10 +2,7 @@ use once_cell::sync::Lazy;
use turbo_rcstr::{RcStr, rcstr}; use turbo_rcstr::{RcStr, rcstr};
use turbo_tasks::{ResolvedVc, ValueToString, Vc}; use turbo_tasks::{ResolvedVc, ValueToString, Vc};
use turbopack_core::{ use turbopack_core::{
chunk::{ChunkingType, ChunkingTypeOption}, chunk::ChunkingType, module::Module, reference::ModuleReference, resolve::ModuleResolveResult,
module::Module,
reference::ModuleReference,
resolve::ModuleResolveResult,
}; };
#[turbo_tasks::value] #[turbo_tasks::value]
...@@ -32,11 +29,10 @@ impl ModuleReference for NextServerUtilityModuleReference { ...@@ -32,11 +29,10 @@ impl ModuleReference for NextServerUtilityModuleReference {
*ModuleResolveResult::module(self.asset) *ModuleResolveResult::module(self.asset)
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(&self) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Shared {
Vc::cell(Some(ChunkingType::Shared {
inherit_async: true, inherit_async: true,
merge_tag: Some(NEXT_SERVER_UTILITY_MERGE_TAG.clone()), merge_tag: Some(NEXT_SERVER_UTILITY_MERGE_TAG.clone()),
})) })
} }
} }
...@@ -111,6 +111,14 @@ export declare function codeFrameColumns( ...@@ -111,6 +111,14 @@ export declare function codeFrameColumns(
location: NapiCodeFrameLocation, location: NapiCodeFrameLocation,
options?: NapiCodeFrameOptions | undefined | null options?: NapiCodeFrameOptions | undefined | null
): string | null ): string | null
/**
* Convert an array of dash-case feature name strings to a lightningcss
* `Features` bitmask (u32). Called from the webpack lightningcss-loader to
* avoid duplicating the name-to-bit mapping in JavaScript.
*/
export declare function lightningcssFeatureNamesToMaskNapi(
names: Array<string>
): number
export declare function lockfileTryAcquireSync( export declare function lockfileTryAcquireSync(
path: string, path: string,
content?: string | undefined | null content?: string | undefined | null
...@@ -248,7 +256,7 @@ export interface NapiProjectOptions { ...@@ -248,7 +256,7 @@ export interface NapiProjectOptions {
isPersistentCachingEnabled: boolean isPersistentCachingEnabled: boolean
/** The version of Next.js that is running. */ /** The version of Next.js that is running. */
nextVersion: RcStr nextVersion: RcStr
/** Whether server-side HMR is enabled (requires --experimental-server-fast-refresh). */ /** Whether server-side HMR is enabled (--experimental-server-fast-refresh). */
serverHmr?: boolean serverHmr?: boolean
} }
/** [NapiProjectOptions] with all fields optional. */ /** [NapiProjectOptions] with all fields optional. */
...@@ -294,8 +302,6 @@ export interface NapiPartialProjectOptions { ...@@ -294,8 +302,6 @@ export interface NapiPartialProjectOptions {
* debugging/profiling purposes. * debugging/profiling purposes.
*/ */
noMangling?: boolean noMangling?: boolean
/** Whether server-side HMR is enabled (requires --experimental-server-fast-refresh). */
serverHmr?: boolean
} }
export interface NapiDefineEnv { export interface NapiDefineEnv {
client: Array<NapiOptionEnvVar> client: Array<NapiOptionEnvVar>
......
...@@ -377,9 +377,6 @@ impl ChunkingType { ...@@ -377,9 +377,6 @@ impl ChunkingType {
} }
} }
#[turbo_tasks::value(transparent)]
pub struct ChunkingTypeOption(Option<ChunkingType>);
pub struct ChunkGroupContent { pub struct ChunkGroupContent {
pub chunkable_items: Vec<ChunkableModuleOrBatch>, pub chunkable_items: Vec<ChunkableModuleOrBatch>,
pub batch_groups: Vec<ResolvedVc<ModuleBatchGroup>>, pub batch_groups: Vec<ResolvedVc<ModuleBatchGroup>>,
......
...@@ -68,7 +68,8 @@ pub async fn children_from_module_references( ...@@ -68,7 +68,8 @@ pub async fn children_from_module_references(
let mut children = FxIndexSet::default(); let mut children = FxIndexSet::default();
let references = references.await?; let references = references.await?;
for &reference in &*references { for &reference in &*references {
let key = match &*reference.chunking_type().await? { let trait_ref = reference.into_trait_ref().await?;
let key = match &trait_ref.chunking_type() {
None => key.clone(), None => key.clone(),
Some(ChunkingType::Parallel { inherit_async, .. }) => { Some(ChunkingType::Parallel { inherit_async, .. }) => {
if *inherit_async { if *inherit_async {
......
...@@ -9,7 +9,7 @@ use turbo_tasks::{ ...@@ -9,7 +9,7 @@ use turbo_tasks::{
}; };
use crate::{ use crate::{
chunk::{ChunkingType, ChunkingTypeOption}, chunk::ChunkingType,
module::{Module, Modules}, module::{Module, Modules},
output::{ output::{
ExpandOutputAssetsInput, ExpandedOutputAssets, OutputAsset, OutputAssets, ExpandOutputAssetsInput, ExpandedOutputAssets, OutputAsset, OutputAssets,
...@@ -34,14 +34,12 @@ pub trait ModuleReference: ValueToString { ...@@ -34,14 +34,12 @@ pub trait ModuleReference: ValueToString {
// TODO think about different types // TODO think about different types
// fn kind(&self) -> Vc<AssetReferenceType>; // fn kind(&self) -> Vc<AssetReferenceType>;
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(self: Vc<Self>) -> Vc<ChunkingTypeOption> { None
Vc::cell(None)
} }
#[turbo_tasks::function] fn binding_usage(&self) -> BindingUsage {
fn binding_usage(self: Vc<Self>) -> Vc<BindingUsage> { BindingUsage::default()
BindingUsage::all()
} }
} }
...@@ -97,22 +95,22 @@ impl SingleModuleReference { ...@@ -97,22 +95,22 @@ impl SingleModuleReference {
pub struct SingleChunkableModuleReference { pub struct SingleChunkableModuleReference {
asset: ResolvedVc<Box<dyn Module>>, asset: ResolvedVc<Box<dyn Module>>,
description: RcStr, description: RcStr,
export: ResolvedVc<ExportUsage>, export: ExportUsage,
} }
#[turbo_tasks::value_impl] #[turbo_tasks::value_impl]
impl SingleChunkableModuleReference { impl SingleChunkableModuleReference {
#[turbo_tasks::function] #[turbo_tasks::function]
pub fn new( pub async fn new(
asset: ResolvedVc<Box<dyn Module>>, asset: ResolvedVc<Box<dyn Module>>,
description: RcStr, description: RcStr,
export: ResolvedVc<ExportUsage>, export: Vc<ExportUsage>,
) -> Vc<Self> { ) -> Result<Vc<Self>> {
Self::cell(SingleChunkableModuleReference { Ok(Self::cell(SingleChunkableModuleReference {
asset, asset,
description, description,
export, export: export.owned().await?,
}) }))
} }
} }
...@@ -123,21 +121,18 @@ impl ModuleReference for SingleChunkableModuleReference { ...@@ -123,21 +121,18 @@ impl ModuleReference for SingleChunkableModuleReference {
*ModuleResolveResult::module(self.asset) *ModuleResolveResult::module(self.asset)
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(self: Vc<Self>) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Parallel {
Vc::cell(Some(ChunkingType::Parallel {
inherit_async: true, inherit_async: true,
hoisted: false, hoisted: false,
})) })
} }
#[turbo_tasks::function] fn binding_usage(&self) -> BindingUsage {
async fn binding_usage(&self) -> Result<Vc<BindingUsage>> { BindingUsage {
Ok(BindingUsage {
import: ImportUsage::TopLevel, import: ImportUsage::TopLevel,
export: self.export.owned().await?, export: self.export.clone(),
} }
.cell())
} }
} }
...@@ -225,9 +220,8 @@ impl ModuleReference for TracedModuleReference { ...@@ -225,9 +220,8 @@ impl ModuleReference for TracedModuleReference {
*ModuleResolveResult::module(self.module) *ModuleResolveResult::module(self.module)
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(&self) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Traced)
Vc::cell(Some(ChunkingType::Traced))
} }
} }
...@@ -295,7 +289,8 @@ pub async fn primary_chunkable_referenced_modules( ...@@ -295,7 +289,8 @@ pub async fn primary_chunkable_referenced_modules(
.await? .await?
.iter() .iter()
.map(|reference| async { .map(|reference| async {
if let Some(chunking_type) = &*reference.chunking_type().await? { let trait_ref = reference.into_trait_ref().await?;
if let Some(chunking_type) = &trait_ref.chunking_type() {
if !include_traced && matches!(chunking_type, ChunkingType::Traced) { if !include_traced && matches!(chunking_type, ChunkingType::Traced) {
return Ok(None); return Ok(None);
} }
...@@ -306,7 +301,7 @@ pub async fn primary_chunkable_referenced_modules( ...@@ -306,7 +301,7 @@ pub async fn primary_chunkable_referenced_modules(
.primary_modules_ref() .primary_modules_ref()
.await?; .await?;
let binding_usage = if include_binding_usage { let binding_usage = if include_binding_usage {
reference.binding_usage().owned().await? trait_ref.binding_usage()
} else { } else {
BindingUsage::default() BindingUsage::default()
}; };
......
use turbo_tasks::{ResolvedVc, ValueToString, Vc}; use turbo_tasks::{ResolvedVc, ValueToString, Vc};
use turbopack_core::{ use turbopack_core::{
chunk::{ChunkingType, ChunkingTypeOption}, chunk::ChunkingType,
reference::ModuleReference, reference::ModuleReference,
reference_type::CssReferenceSubType, reference_type::CssReferenceSubType,
resolve::{ModuleResolveResult, origin::ResolveOrigin, parse::Request}, resolve::{ModuleResolveResult, origin::ResolveOrigin, parse::Request},
...@@ -42,11 +42,10 @@ impl ModuleReference for CssModuleComposeReference { ...@@ -42,11 +42,10 @@ impl ModuleReference for CssModuleComposeReference {
) )
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(self: Vc<Self>) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Parallel {
Vc::cell(Some(ChunkingType::Parallel {
inherit_async: false, inherit_async: false,
hoisted: false, hoisted: false,
})) })
} }
} }
...@@ -8,7 +8,7 @@ use lightningcss::{ ...@@ -8,7 +8,7 @@ use lightningcss::{
}; };
use turbo_tasks::{ResolvedVc, ValueToString, Vc}; use turbo_tasks::{ResolvedVc, ValueToString, Vc};
use turbopack_core::{ use turbopack_core::{
chunk::{ChunkingContext, ChunkingType, ChunkingTypeOption}, chunk::{ChunkingContext, ChunkingType},
issue::IssueSource, issue::IssueSource,
reference::ModuleReference, reference::ModuleReference,
reference_type::{CssReferenceSubType, ImportContext}, reference_type::{CssReferenceSubType, ImportContext},
...@@ -142,12 +142,11 @@ impl ModuleReference for ImportAssetReference { ...@@ -142,12 +142,11 @@ impl ModuleReference for ImportAssetReference {
)) ))
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(self: Vc<Self>) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Parallel {
Vc::cell(Some(ChunkingType::Parallel {
inherit_async: false, inherit_async: false,
hoisted: false, hoisted: false,
})) })
} }
} }
......
use turbo_tasks::{ResolvedVc, ValueToString, Vc}; use turbo_tasks::{ResolvedVc, ValueToString, Vc};
use turbopack_core::{ use turbopack_core::{
chunk::{ChunkingType, ChunkingTypeOption}, chunk::ChunkingType, module::Module, reference::ModuleReference, resolve::ModuleResolveResult,
module::Module,
reference::ModuleReference,
resolve::ModuleResolveResult,
}; };
/// A reference to an internal CSS asset. /// A reference to an internal CSS asset.
...@@ -30,11 +27,10 @@ impl ModuleReference for InternalCssAssetReference { ...@@ -30,11 +27,10 @@ impl ModuleReference for InternalCssAssetReference {
*ModuleResolveResult::module(self.module) *ModuleResolveResult::module(self.module)
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(self: Vc<Self>) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Parallel {
Vc::cell(Some(ChunkingType::Parallel {
inherit_async: false, inherit_async: false,
hoisted: false, hoisted: false,
})) })
} }
} }
...@@ -11,7 +11,7 @@ use rustc_hash::FxHashMap; ...@@ -11,7 +11,7 @@ use rustc_hash::FxHashMap;
use turbo_rcstr::RcStr; use turbo_rcstr::RcStr;
use turbo_tasks::{ResolvedVc, ValueToString, Vc}; use turbo_tasks::{ResolvedVc, ValueToString, Vc};
use turbopack_core::{ use turbopack_core::{
chunk::{ChunkingContext, ChunkingType, ChunkingTypeOption}, chunk::{ChunkingContext, ChunkingType},
issue::IssueSource, issue::IssueSource,
output::OutputAsset, output::OutputAsset,
reference::ModuleReference, reference::ModuleReference,
...@@ -86,12 +86,11 @@ impl ModuleReference for UrlAssetReference { ...@@ -86,12 +86,11 @@ impl ModuleReference for UrlAssetReference {
) )
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(self: Vc<Self>) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Parallel {
Vc::cell(Some(ChunkingType::Parallel {
inherit_async: false, inherit_async: false,
hoisted: false, hoisted: false,
})) })
} }
} }
......
...@@ -5,7 +5,12 @@ use rustc_hash::{FxHashMap, FxHashSet}; ...@@ -5,7 +5,12 @@ use rustc_hash::{FxHashMap, FxHashSet};
use smallvec::SmallVec; use smallvec::SmallVec;
use swc_core::{ use swc_core::{
atoms::Wtf8Atom, atoms::Wtf8Atom,
common::{BytePos, Span, Spanned, SyntaxContext, comments::Comments, source_map::SmallPos}, common::{
BytePos, Span, Spanned, SyntaxContext,
comments::Comments,
errors::{DiagnosticId, HANDLER},
source_map::SmallPos,
},
ecma::{ ecma::{
ast::*, ast::*,
atoms::{Atom, atom}, atoms::{Atom, atom},
...@@ -15,7 +20,9 @@ use swc_core::{ ...@@ -15,7 +20,9 @@ use swc_core::{
}; };
use turbo_rcstr::{RcStr, rcstr}; use turbo_rcstr::{RcStr, rcstr};
use turbo_tasks::{FxIndexMap, FxIndexSet, ResolvedVc}; use turbo_tasks::{FxIndexMap, FxIndexSet, ResolvedVc};
use turbopack_core::{issue::IssueSource, loader::WebpackLoaderItem, source::Source}; use turbopack_core::{
chunk::ChunkingType, issue::IssueSource, loader::WebpackLoaderItem, source::Source,
};
use super::{JsValue, ModuleValue, top_level_await::has_top_level_await}; use super::{JsValue, ModuleValue, top_level_await::has_top_level_await};
use crate::{ use crate::{
...@@ -113,6 +120,23 @@ impl ImportAnnotations { ...@@ -113,6 +120,23 @@ impl ImportAnnotations {
PropName::Str(s) => s.value.clone(), PropName::Str(s) => s.value.clone(),
_ => continue, _ => continue,
}; };
// Validate known annotation values
if key == *ANNOTATION_CHUNKING_TYPE {
let value = str.value.to_string_lossy();
if value != "parallel" && value != "none" {
HANDLER.with(|handler| {
handler.span_warn_with_code(
kv.value.span(),
&format!(
"unknown turbopack-chunking-type: \"{value}\", \
expected \"parallel\" or \"none\""
),
DiagnosticId::Error("turbopack-chunking-type".into()),
);
});
continue;
}
}
map.insert(key, str.value.clone()); map.insert(key, str.value.clone());
} }
} }
...@@ -171,9 +195,23 @@ impl ImportAnnotations { ...@@ -171,9 +195,23 @@ impl ImportAnnotations {
.map(|v| v.to_string_lossy()) .map(|v| v.to_string_lossy())
} }
/// Returns the content on the chunking-type annotation /// Returns the chunking type override from the `turbopack-chunking-type` annotation.
pub fn chunking_type(&self) -> Option<&Wtf8Atom> { ///
self.get(&ANNOTATION_CHUNKING_TYPE) /// - `None` — no annotation present
/// - `Some(None)` — annotation is `"none"` (opt out of chunking)
/// - `Some(Some(..))` — explicit chunking type (e.g. `"parallel"`)
///
/// Unknown values are rejected during [`ImportAnnotations::parse`] and omitted.
pub fn chunking_type(&self) -> Option<Option<ChunkingType>> {
let chunking_type = self.get(&ANNOTATION_CHUNKING_TYPE)?;
if chunking_type == "none" {
Some(None)
} else {
Some(Some(ChunkingType::Parallel {
inherit_async: true,
hoisted: true,
}))
}
} }
/// Returns the content on the type attribute /// Returns the content on the type attribute
......
...@@ -15,7 +15,7 @@ use turbo_tasks::{ ...@@ -15,7 +15,7 @@ use turbo_tasks::{
trace::TraceRawVcs, trace::TraceRawVcs,
}; };
use turbopack_core::{ use turbopack_core::{
chunk::{ChunkingContext, ChunkingType, ChunkingTypeOption}, chunk::{ChunkingContext, ChunkingType},
issue::IssueSource, issue::IssueSource,
reference::ModuleReference, reference::ModuleReference,
reference_type::CommonJsReferenceSubType, reference_type::CommonJsReferenceSubType,
...@@ -74,12 +74,11 @@ impl ModuleReference for AmdDefineAssetReference { ...@@ -74,12 +74,11 @@ impl ModuleReference for AmdDefineAssetReference {
) )
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(self: Vc<Self>) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Parallel {
Vc::cell(Some(ChunkingType::Parallel {
inherit_async: false, inherit_async: false,
hoisted: false, hoisted: false,
})) })
} }
} }
......
...@@ -88,7 +88,8 @@ struct AsyncModuleIdents( ...@@ -88,7 +88,8 @@ struct AsyncModuleIdents(
async fn get_inherit_async_referenced_asset( async fn get_inherit_async_referenced_asset(
r: ResolvedVc<Box<dyn ModuleReference>>, r: ResolvedVc<Box<dyn ModuleReference>>,
) -> Result<Option<ReadRef<ReferencedAsset>>> { ) -> Result<Option<ReadRef<ReferencedAsset>>> {
let Some(ty) = &*r.chunking_type().await? else { let trait_ref = r.into_trait_ref().await?;
let Some(ty) = &trait_ref.chunking_type() else {
return Ok(None); return Ok(None);
}; };
if !matches!( if !matches!(
......
...@@ -9,7 +9,7 @@ use turbo_tasks::{ ...@@ -9,7 +9,7 @@ use turbo_tasks::{
NonLocalValue, ResolvedVc, ValueToString, Vc, debug::ValueDebugFormat, trace::TraceRawVcs, NonLocalValue, ResolvedVc, ValueToString, Vc, debug::ValueDebugFormat, trace::TraceRawVcs,
}; };
use turbopack_core::{ use turbopack_core::{
chunk::{ChunkingContext, ChunkingType, ChunkingTypeOption}, chunk::{ChunkingContext, ChunkingType},
issue::IssueSource, issue::IssueSource,
reference::ModuleReference, reference::ModuleReference,
reference_type::CommonJsReferenceSubType, reference_type::CommonJsReferenceSubType,
...@@ -68,12 +68,11 @@ impl ModuleReference for CjsAssetReference { ...@@ -68,12 +68,11 @@ impl ModuleReference for CjsAssetReference {
) )
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(self: Vc<Self>) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Parallel {
Vc::cell(Some(ChunkingType::Parallel {
inherit_async: false, inherit_async: false,
hoisted: false, hoisted: false,
})) })
} }
} }
...@@ -116,12 +115,11 @@ impl ModuleReference for CjsRequireAssetReference { ...@@ -116,12 +115,11 @@ impl ModuleReference for CjsRequireAssetReference {
) )
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(self: Vc<Self>) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Parallel {
Vc::cell(Some(ChunkingType::Parallel {
inherit_async: false, inherit_async: false,
hoisted: false, hoisted: false,
})) })
} }
} }
...@@ -239,12 +237,11 @@ impl ModuleReference for CjsRequireResolveAssetReference { ...@@ -239,12 +237,11 @@ impl ModuleReference for CjsRequireResolveAssetReference {
) )
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(self: Vc<Self>) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Parallel {
Vc::cell(Some(ChunkingType::Parallel {
inherit_async: false, inherit_async: false,
hoisted: false, hoisted: false,
})) })
} }
} }
......
...@@ -13,7 +13,7 @@ use turbo_rcstr::{RcStr, rcstr}; ...@@ -13,7 +13,7 @@ use turbo_rcstr::{RcStr, rcstr};
use turbo_tasks::{ResolvedVc, ValueToString, Vc, turbobail}; use turbo_tasks::{ResolvedVc, ValueToString, Vc, turbobail};
use turbo_tasks_fs::FileSystemPath; use turbo_tasks_fs::FileSystemPath;
use turbopack_core::{ use turbopack_core::{
chunk::{ChunkingContext, ChunkingType, ChunkingTypeOption, ModuleChunkItemIdExt}, chunk::{ChunkingContext, ChunkingType, ModuleChunkItemIdExt},
issue::{ issue::{
Issue, IssueExt, IssueSeverity, IssueSource, IssueStage, OptionIssueSource, Issue, IssueExt, IssueSeverity, IssueSource, IssueStage, OptionIssueSource,
OptionStyledString, StyledString, OptionStyledString, StyledString,
...@@ -397,11 +397,6 @@ impl EsmAssetReference { ...@@ -397,11 +397,6 @@ impl EsmAssetReference {
is_pure_import: true, is_pure_import: true,
} }
} }
}
#[turbo_tasks::value_impl]
impl EsmAssetReference {
#[turbo_tasks::function]
pub(crate) fn get_referenced_asset(self: Vc<Self>) -> Vc<ReferencedAsset> { pub(crate) fn get_referenced_asset(self: Vc<Self>) -> Vc<ReferencedAsset> {
ReferencedAsset::from_resolve_result(self.resolve_reference()) ReferencedAsset::from_resolve_result(self.resolve_reference())
} }
...@@ -501,31 +496,16 @@ impl ModuleReference for EsmAssetReference { ...@@ -501,31 +496,16 @@ impl ModuleReference for EsmAssetReference {
Ok(result) Ok(result)
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(&self) -> Result<Vc<ChunkingTypeOption>> { self.annotations
Ok(Vc::cell( .chunking_type()
if let Some(chunking_type) = self.annotations.chunking_type() { .unwrap_or(Some(ChunkingType::Parallel {
if chunking_type == "parallel" { inherit_async: true,
Some(ChunkingType::Parallel { hoisted: true,
inherit_async: true, }))
hoisted: true,
})
} else if chunking_type == "none" {
None
} else {
bail!("unknown chunking_type: {}", chunking_type.to_string_lossy());
}
} else {
Some(ChunkingType::Parallel {
inherit_async: true,
hoisted: true,
})
},
))
} }
#[turbo_tasks::function] fn binding_usage(&self) -> BindingUsage {
fn binding_usage(&self) -> Vc<BindingUsage> {
BindingUsage { BindingUsage {
import: self.import_usage.clone(), import: self.import_usage.clone(),
export: match &self.export_name { export: match &self.export_name {
...@@ -534,7 +514,6 @@ impl ModuleReference for EsmAssetReference { ...@@ -534,7 +514,6 @@ impl ModuleReference for EsmAssetReference {
_ => ExportUsage::All, _ => ExportUsage::All,
}, },
} }
.cell()
} }
} }
...@@ -555,7 +534,7 @@ impl EsmAssetReference { ...@@ -555,7 +534,7 @@ impl EsmAssetReference {
} }
// only chunked references can be imported // only chunked references can be imported
if this.annotations.chunking_type().is_none_or(|v| v != "none") { if !matches!(this.annotations.chunking_type(), Some(None)) {
let import_externals = this.import_externals; let import_externals = this.import_externals;
let referenced_asset = self.get_referenced_asset().await?; let referenced_asset = self.get_referenced_asset().await?;
......
...@@ -9,7 +9,7 @@ use turbo_tasks::{ ...@@ -9,7 +9,7 @@ use turbo_tasks::{
NonLocalValue, ResolvedVc, ValueToString, Vc, debug::ValueDebugFormat, trace::TraceRawVcs, NonLocalValue, ResolvedVc, ValueToString, Vc, debug::ValueDebugFormat, trace::TraceRawVcs,
}; };
use turbopack_core::{ use turbopack_core::{
chunk::{ChunkingContext, ChunkingType, ChunkingTypeOption}, chunk::{ChunkingContext, ChunkingType},
environment::ChunkLoading, environment::ChunkLoading,
issue::IssueSource, issue::IssueSource,
reference::ModuleReference, reference::ModuleReference,
...@@ -94,18 +94,15 @@ impl ModuleReference for EsmAsyncAssetReference { ...@@ -94,18 +94,15 @@ impl ModuleReference for EsmAsyncAssetReference {
.await .await
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(&self) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Async)
Vc::cell(Some(ChunkingType::Async))
} }
#[turbo_tasks::function] fn binding_usage(&self) -> BindingUsage {
fn binding_usage(&self) -> Vc<BindingUsage> {
BindingUsage { BindingUsage {
import: Default::default(), import: Default::default(),
export: self.export_usage.clone(), export: self.export_usage.clone(),
} }
.cell()
} }
} }
......
...@@ -5,7 +5,7 @@ use turbo_tasks::{ ...@@ -5,7 +5,7 @@ use turbo_tasks::{
NonLocalValue, ResolvedVc, ValueToString, Vc, debug::ValueDebugFormat, trace::TraceRawVcs, NonLocalValue, ResolvedVc, ValueToString, Vc, debug::ValueDebugFormat, trace::TraceRawVcs,
}; };
use turbopack_core::{ use turbopack_core::{
chunk::{ChunkingContext, ChunkingTypeOption, ModuleChunkItemIdExt}, chunk::{ChunkingContext, ChunkingType, ModuleChunkItemIdExt},
reference::ModuleReference, reference::ModuleReference,
resolve::ModuleResolveResult, resolve::ModuleResolveResult,
}; };
...@@ -25,11 +25,15 @@ use crate::{ ...@@ -25,11 +25,15 @@ use crate::{
#[value_to_string("module id of {inner}")] #[value_to_string("module id of {inner}")]
pub struct EsmModuleIdAssetReference { pub struct EsmModuleIdAssetReference {
inner: ResolvedVc<EsmAssetReference>, inner: ResolvedVc<EsmAssetReference>,
chunking_type: Option<ChunkingType>,
} }
impl EsmModuleIdAssetReference { impl EsmModuleIdAssetReference {
pub fn new(inner: ResolvedVc<EsmAssetReference>) -> Self { pub fn new(inner: ResolvedVc<EsmAssetReference>, chunking_type: Option<ChunkingType>) -> Self {
EsmModuleIdAssetReference { inner } EsmModuleIdAssetReference {
inner,
chunking_type,
}
} }
} }
...@@ -40,9 +44,8 @@ impl ModuleReference for EsmModuleIdAssetReference { ...@@ -40,9 +44,8 @@ impl ModuleReference for EsmModuleIdAssetReference {
self.inner.resolve_reference() self.inner.resolve_reference()
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(&self) -> Vc<ChunkingTypeOption> { self.chunking_type.clone()
self.inner.chunking_type()
} }
} }
......
...@@ -9,7 +9,7 @@ use turbo_tasks::{ ...@@ -9,7 +9,7 @@ use turbo_tasks::{
trace::TraceRawVcs, trace::TraceRawVcs,
}; };
use turbopack_core::{ use turbopack_core::{
chunk::{ChunkingContext, ChunkingType, ChunkingTypeOption, ModuleChunkItemIdExt}, chunk::{ChunkingContext, ChunkingType, ModuleChunkItemIdExt},
environment::Rendering, environment::Rendering,
issue::IssueSource, issue::IssueSource,
reference::ModuleReference, reference::ModuleReference,
...@@ -99,12 +99,11 @@ impl ModuleReference for UrlAssetReference { ...@@ -99,12 +99,11 @@ impl ModuleReference for UrlAssetReference {
) )
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(&self) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Parallel {
Vc::cell(Some(ChunkingType::Parallel {
inherit_async: false, inherit_async: false,
hoisted: false, hoisted: false,
})) })
} }
} }
......
...@@ -16,7 +16,7 @@ use turbo_tasks::{ ...@@ -16,7 +16,7 @@ use turbo_tasks::{
trace::TraceRawVcs, trace::TraceRawVcs,
}; };
use turbopack_core::{ use turbopack_core::{
chunk::{ChunkingContext, ChunkingType, ChunkingTypeOption, ModuleChunkItemIdExt}, chunk::{ChunkingContext, ChunkingType, ModuleChunkItemIdExt},
issue::IssueSource, issue::IssueSource,
reference::ModuleReference, reference::ModuleReference,
reference_type::{CommonJsReferenceSubType, EcmaScriptModulesReferenceSubType}, reference_type::{CommonJsReferenceSubType, EcmaScriptModulesReferenceSubType},
...@@ -112,12 +112,11 @@ impl ModuleReference for ModuleHotReferenceAssetReference { ...@@ -112,12 +112,11 @@ impl ModuleReference for ModuleHotReferenceAssetReference {
self.resolve().await self.resolve().await
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(self: Vc<Self>) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Parallel {
Vc::cell(Some(ChunkingType::Parallel {
inherit_async: false, inherit_async: false,
hoisted: false, hoisted: false,
})) })
} }
} }
......
...@@ -68,6 +68,7 @@ use turbo_tasks::{ ...@@ -68,6 +68,7 @@ use turbo_tasks::{
}; };
use turbo_tasks_fs::FileSystemPath; use turbo_tasks_fs::FileSystemPath;
use turbopack_core::{ use turbopack_core::{
chunk::ChunkingType,
compile_time_info::{ compile_time_info::{
CompileTimeDefineValue, CompileTimeDefines, CompileTimeInfo, DefinableNameSegment, CompileTimeDefineValue, CompileTimeDefines, CompileTimeInfo, DefinableNameSegment,
DefinableNameSegmentRef, FreeVarReference, FreeVarReferences, FreeVarReferencesMembers, DefinableNameSegmentRef, FreeVarReference, FreeVarReferences, FreeVarReferencesMembers,
...@@ -1509,8 +1510,14 @@ async fn analyze_ecmascript_module_internal( ...@@ -1509,8 +1510,14 @@ async fn analyze_ecmascript_module_internal(
}; };
if let Some("__turbopack_module_id__") = export.as_deref() { if let Some("__turbopack_module_id__") = export.as_deref() {
let chunking_type = r.await?.annotations.chunking_type().unwrap_or(Some(
ChunkingType::Parallel {
inherit_async: true,
hoisted: true,
},
));
analysis.add_reference_code_gen( analysis.add_reference_code_gen(
EsmModuleIdAssetReference::new(*r), EsmModuleIdAssetReference::new(*r, chunking_type),
ast_path.into(), ast_path.into(),
) )
} else { } else {
......
...@@ -4,7 +4,7 @@ use turbo_rcstr::rcstr; ...@@ -4,7 +4,7 @@ use turbo_rcstr::rcstr;
use turbo_tasks::{ResolvedVc, ValueToString, Vc}; use turbo_tasks::{ResolvedVc, ValueToString, Vc};
use turbo_tasks_fs::FileSystemPath; use turbo_tasks_fs::FileSystemPath;
use turbopack_core::{ use turbopack_core::{
chunk::{ChunkingType, ChunkingTypeOption}, chunk::ChunkingType,
file_source::FileSource, file_source::FileSource,
issue::IssueSource, issue::IssueSource,
raw_module::RawModule, raw_module::RawModule,
...@@ -78,9 +78,8 @@ impl ModuleReference for FileSourceReference { ...@@ -78,9 +78,8 @@ impl ModuleReference for FileSourceReference {
.await .await
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(&self) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Traced)
Vc::cell(Some(ChunkingType::Traced))
} }
} }
...@@ -207,8 +206,7 @@ impl ModuleReference for DirAssetReference { ...@@ -207,8 +206,7 @@ impl ModuleReference for DirAssetReference {
.await .await
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(&self) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Traced)
Vc::cell(Some(ChunkingType::Traced))
} }
} }
...@@ -22,8 +22,8 @@ use turbo_tasks::{ ...@@ -22,8 +22,8 @@ use turbo_tasks::{
use turbo_tasks_fs::{DirectoryContent, DirectoryEntry, FileSystemPath}; use turbo_tasks_fs::{DirectoryContent, DirectoryEntry, FileSystemPath};
use turbopack_core::{ use turbopack_core::{
chunk::{ chunk::{
AsyncModuleInfo, ChunkableModule, ChunkingContext, ChunkingType, ChunkingTypeOption, AsyncModuleInfo, ChunkableModule, ChunkingContext, ChunkingType, MinifyType,
MinifyType, ModuleChunkItemIdExt, ModuleChunkItemIdExt,
}, },
ident::AssetIdent, ident::AssetIdent,
issue::IssueSource, issue::IssueSource,
...@@ -295,12 +295,11 @@ impl ModuleReference for RequireContextAssetReference { ...@@ -295,12 +295,11 @@ impl ModuleReference for RequireContextAssetReference {
*ModuleResolveResult::module(ResolvedVc::upcast(self.inner)) *ModuleResolveResult::module(ResolvedVc::upcast(self.inner))
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(&self) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Parallel {
Vc::cell(Some(ChunkingType::Parallel {
inherit_async: false, inherit_async: false,
hoisted: false, hoisted: false,
})) })
} }
} }
...@@ -373,12 +372,11 @@ impl ModuleReference for ResolvedModuleReference { ...@@ -373,12 +372,11 @@ impl ModuleReference for ResolvedModuleReference {
*self.0 *self.0
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(&self) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Parallel {
Vc::cell(Some(ChunkingType::Parallel {
inherit_async: false, inherit_async: false,
hoisted: false, hoisted: false,
})) })
} }
} }
......
...@@ -12,7 +12,7 @@ use turbo_tasks::{ ...@@ -12,7 +12,7 @@ use turbo_tasks::{
}; };
use turbo_tasks_fs::FileSystemPath; use turbo_tasks_fs::FileSystemPath;
use turbopack_core::{ use turbopack_core::{
chunk::{ChunkableModule, ChunkingContext, ChunkingType, ChunkingTypeOption, EvaluatableAsset}, chunk::{ChunkableModule, ChunkingContext, ChunkingType, EvaluatableAsset},
context::AssetContext, context::AssetContext,
issue::{IssueExt, IssueSeverity, IssueSource, StyledString, code_gen::CodeGenerationIssue}, issue::{IssueExt, IssueSeverity, IssueSource, StyledString, code_gen::CodeGenerationIssue},
module::Module, module::Module,
...@@ -256,12 +256,11 @@ impl ModuleReference for WorkerAssetReference { ...@@ -256,12 +256,11 @@ impl ModuleReference for WorkerAssetReference {
.cell()) .cell())
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(self: Vc<Self>) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Parallel {
Vc::cell(Some(ChunkingType::Parallel {
inherit_async: false, inherit_async: false,
hoisted: false, hoisted: false,
})) })
} }
} }
......
...@@ -7,7 +7,7 @@ use swc_core::{ ...@@ -7,7 +7,7 @@ use swc_core::{
}; };
use turbo_tasks::{NonLocalValue, ResolvedVc, ValueToString, Vc, trace::TraceRawVcs}; use turbo_tasks::{NonLocalValue, ResolvedVc, ValueToString, Vc, trace::TraceRawVcs};
use turbopack_core::{ use turbopack_core::{
chunk::{ChunkingContext, ChunkingType, ChunkingTypeOption, ModuleChunkItemIdExt}, chunk::{ChunkingContext, ChunkingType, ModuleChunkItemIdExt},
module::Module, module::Module,
reference::ModuleReference, reference::ModuleReference,
resolve::{BindingUsage, ExportUsage, ImportUsage, ModulePart, ModuleResolveResult}, resolve::{BindingUsage, ExportUsage, ImportUsage, ModulePart, ModuleResolveResult},
...@@ -40,7 +40,7 @@ enum EcmascriptModulePartReferenceMode { ...@@ -40,7 +40,7 @@ enum EcmascriptModulePartReferenceMode {
pub struct EcmascriptModulePartReference { pub struct EcmascriptModulePartReference {
module: ResolvedVc<Box<dyn EcmascriptChunkPlaceable>>, module: ResolvedVc<Box<dyn EcmascriptChunkPlaceable>>,
part: ModulePart, part: ModulePart,
export_usage: ResolvedVc<ExportUsage>, export_usage: ExportUsage,
mode: EcmascriptModulePartReferenceMode, mode: EcmascriptModulePartReferenceMode,
} }
...@@ -48,11 +48,11 @@ pub struct EcmascriptModulePartReference { ...@@ -48,11 +48,11 @@ pub struct EcmascriptModulePartReference {
impl EcmascriptModulePartReference { impl EcmascriptModulePartReference {
// Create new [EcmascriptModuleFacadeModule]s as necessary // Create new [EcmascriptModuleFacadeModule]s as necessary
#[turbo_tasks::function] #[turbo_tasks::function]
pub fn new_part( pub async fn new_part(
module: ResolvedVc<Box<dyn EcmascriptChunkPlaceable>>, module: ResolvedVc<Box<dyn EcmascriptChunkPlaceable>>,
part: ModulePart, part: ModulePart,
export_usage: ResolvedVc<ExportUsage>, export_usage: Vc<ExportUsage>,
) -> Vc<Self> { ) -> Result<Vc<Self>> {
debug_assert!(matches!( debug_assert!(matches!(
part, part,
ModulePart::Locals ModulePart::Locals
...@@ -60,29 +60,29 @@ impl EcmascriptModulePartReference { ...@@ -60,29 +60,29 @@ impl EcmascriptModulePartReference {
| ModulePart::RenamedExport { .. } | ModulePart::RenamedExport { .. }
| ModulePart::RenamedNamespace { .. } | ModulePart::RenamedNamespace { .. }
)); ));
EcmascriptModulePartReference { Ok(EcmascriptModulePartReference {
module, module,
part, part,
export_usage, export_usage: export_usage.owned().await?,
mode: EcmascriptModulePartReferenceMode::Synthesize, mode: EcmascriptModulePartReferenceMode::Synthesize,
} }
.cell() .cell())
} }
// A reference to the given module, without any intermediary synthesized modules. // A reference to the given module, without any intermediary synthesized modules.
#[turbo_tasks::function] #[turbo_tasks::function]
pub fn new_normal( pub async fn new_normal(
module: ResolvedVc<Box<dyn EcmascriptChunkPlaceable>>, module: ResolvedVc<Box<dyn EcmascriptChunkPlaceable>>,
part: ModulePart, part: ModulePart,
export_usage: ResolvedVc<ExportUsage>, export_usage: Vc<ExportUsage>,
) -> Vc<Self> { ) -> Result<Vc<Self>> {
EcmascriptModulePartReference { Ok(EcmascriptModulePartReference {
module, module,
part, part,
export_usage, export_usage: export_usage.owned().await?,
mode: EcmascriptModulePartReferenceMode::Normal, mode: EcmascriptModulePartReferenceMode::Normal,
} }
.cell() .cell())
} }
} }
...@@ -127,21 +127,18 @@ impl ModuleReference for EcmascriptModulePartReference { ...@@ -127,21 +127,18 @@ impl ModuleReference for EcmascriptModulePartReference {
Ok(*ModuleResolveResult::module(module)) Ok(*ModuleResolveResult::module(module))
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(self: Vc<Self>) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Parallel {
Vc::cell(Some(ChunkingType::Parallel {
inherit_async: true, inherit_async: true,
hoisted: true, hoisted: true,
})) })
} }
#[turbo_tasks::function] fn binding_usage(&self) -> BindingUsage {
async fn binding_usage(&self) -> Result<Vc<BindingUsage>> { BindingUsage {
Ok(BindingUsage {
import: ImportUsage::TopLevel, import: ImportUsage::TopLevel,
export: self.export_usage.owned().await?, export: self.export_usage.clone(),
} }
.cell())
} }
} }
...@@ -174,15 +171,15 @@ impl EcmascriptModulePartReference { ...@@ -174,15 +171,15 @@ impl EcmascriptModulePartReference {
)); ));
} }
let export_usage = this.export_usage.await?; let export_usage = &this.export_usage;
if merged_index.is_some() && matches!(*export_usage, ExportUsage::Evaluation) { if merged_index.is_some() && matches!(export_usage, ExportUsage::Evaluation) {
// No need to import, the module was already executed and is available in the same scope // No need to import, the module was already executed and is available in the same scope
// hoisting group (unless it's a namespace import) // hoisting group (unless it's a namespace import)
} else { } else {
let ident = referenced_asset let ident = referenced_asset
.get_ident( .get_ident(
chunking_context, chunking_context,
match &*export_usage { match export_usage {
ExportUsage::Named(export) => Some(export.clone()), ExportUsage::Named(export) => Some(export.clone()),
ExportUsage::PartialNamespaceObject(_) ExportUsage::PartialNamespaceObject(_)
| ExportUsage::All | ExportUsage::All
......
...@@ -5,7 +5,7 @@ use turbo_tasks::{ResolvedVc, TryJoinIterExt, ValueToString, Vc}; ...@@ -5,7 +5,7 @@ use turbo_tasks::{ResolvedVc, TryJoinIterExt, ValueToString, Vc};
use turbopack_core::{ use turbopack_core::{
chunk::{ chunk::{
AsyncModuleInfo, ChunkData, ChunkGroupType, ChunkableModule, ChunkingContext, AsyncModuleInfo, ChunkData, ChunkGroupType, ChunkableModule, ChunkingContext,
ChunkingContextExt, ChunkingType, ChunkingTypeOption, ChunksData, EvaluatableAsset, ChunkingContextExt, ChunkingType, ChunksData, EvaluatableAsset,
availability_info::AvailabilityInfo, availability_info::AvailabilityInfo,
}, },
context::AssetContext, context::AssetContext,
...@@ -331,14 +331,13 @@ impl ModuleReference for WorkerModuleReference { ...@@ -331,14 +331,13 @@ impl ModuleReference for WorkerModuleReference {
*ModuleResolveResult::module(self.module) *ModuleResolveResult::module(self.module)
} }
#[turbo_tasks::function] fn chunking_type(&self) -> Option<ChunkingType> {
fn chunking_type(&self) -> Vc<ChunkingTypeOption> { Some(ChunkingType::Isolated {
Vc::cell(Some(ChunkingType::Isolated {
_ty: match self.worker_type { _ty: match self.worker_type {
WorkerType::SharedWebWorker | WorkerType::WebWorker => ChunkGroupType::Evaluated, WorkerType::SharedWebWorker | WorkerType::WebWorker => ChunkGroupType::Evaluated,
WorkerType::NodeWorkerThread => ChunkGroupType::Entry, WorkerType::NodeWorkerThread => ChunkGroupType::Entry,
}, },
merge_tag: None, merge_tag: None,
})) })
} }
} }
...@@ -6,21 +6,17 @@ This script analyzes task statistics to identify which tasks are not getting ...@@ -6,21 +6,17 @@ This script analyzes task statistics to identify which tasks are not getting
significant benefit from caching and would be candidates for removing the significant benefit from caching and would be candidates for removing the
caching layer. caching layer.
To use this script, run: a build with `NEXT_TURBOPACK_TASK_STATISTICS=path/to/stats.json` set To use this script, run a build with `NEXT_TURBOPACK_TASK_STATISTICS=path/to/stats.json` set
Then run this script with the path to the stats.json file to get a report on optimization opportunities. Then run this script with the path to the stats.json file to get a report on cache effectiveness.
Based on benchmarking data from the `turbopack/crates/turbo-tasks-backend/benches/overhead.rs` benchmark we have the following estimates: The JSON format contains entries like:
- Cache hit cost: 200-500ns { "task_name": { "cache_hit": N, "cache_miss": N } }
- Execution overhead: 4-6us
- Measurement overhead: 260ns-750ns
This script assumes the best case scenario and reports on the potential time savings from removing the caching layer.
""" """
import json import json
import sys import sys
from typing import Dict, List, Tuple from typing import List, Tuple
from dataclasses import dataclass from dataclasses import dataclass
...@@ -29,8 +25,6 @@ class TaskStats: ...@@ -29,8 +25,6 @@ class TaskStats:
name: str name: str
cache_hit: int cache_hit: int
cache_miss: int cache_miss: int
executions: int
duration_ns: int
@property @property
def total_operations(self) -> int: def total_operations(self) -> int:
...@@ -42,18 +36,6 @@ class TaskStats: ...@@ -42,18 +36,6 @@ class TaskStats:
return 0.0 return 0.0
return self.cache_hit / self.total_operations return self.cache_hit / self.total_operations
@property
def avg_execution_time_ns(self) -> int:
MEASUREMENT_OVERHEAD = 750 # OVerhead implicit in the reported duration
if self.executions == 0:
return 0
return max(0, (self.duration_ns - MEASUREMENT_OVERHEAD * self.executions) // self.executions)
def parse_duration(duration_dict: Dict) -> int:
"""Convert duration dict to nanoseconds."""
return duration_dict.get("secs", 0) * 1_000_000_000 + duration_dict.get("nanos", 0)
def load_task_stats(file_path: str) -> List[TaskStats]: def load_task_stats(file_path: str) -> List[TaskStats]:
"""Load and parse task statistics from JSON file.""" """Load and parse task statistics from JSON file."""
...@@ -62,127 +44,87 @@ def load_task_stats(file_path: str) -> List[TaskStats]: ...@@ -62,127 +44,87 @@ def load_task_stats(file_path: str) -> List[TaskStats]:
tasks = [] tasks = []
for task_name, stats in data.items(): for task_name, stats in data.items():
duration_ns = parse_duration(stats["duration"])
task = TaskStats( task = TaskStats(
name=task_name, name=task_name,
cache_hit=stats["cache_hit"], cache_hit=stats["cache_hit"],
cache_miss=stats["cache_miss"], cache_miss=stats["cache_miss"],
executions=stats["executions"],
duration_ns=duration_ns
) )
tasks.append(task) tasks.append(task)
return tasks return tasks
def calculate_cache_effectiveness(task: TaskStats) -> float: def analyze_tasks(tasks: List[TaskStats]) -> List[TaskStats]:
""" """Analyze all tasks and return sorted by wasted cache overhead.
Calculate the effectiveness of caching for a task.
Returns:
Time savings from removing caching (negative means caching is beneficial)
"""
# Constants based on benchmarking
# These are optimistic estimates
CACHE_HIT_COST_NS = 500 # Average of 200-500ns
EXECUTION_OVERHEAD_NS = 6000 # Average of 4-6us (caching layer overhead)
MEASUREMENT_OVERHEAD = 750 # OVerhead implicit in the reported duration
if task.total_operations == 0:
return 0.0
# Current cost with caching
# Cache hits: just the cache lookup cost
# Cache misses: cache overhead + actual execution time
cache_hit_cost = task.cache_hit * CACHE_HIT_COST_NS
cache_miss_cost = task.cache_miss * (EXECUTION_OVERHEAD_NS + task.avg_execution_time_ns)
current_total_cost = cache_hit_cost + cache_miss_cost
# Cost without caching (all operations would be direct executions, no overhead)
no_cache_cost = task.total_operations * task.avg_execution_time_ns
# Time savings from removing caching (positive means we save time by removing cache)
time_savings = current_total_cost - no_cache_cost
return time_savings
def analyze_tasks(tasks: List[TaskStats]) -> List[Tuple[TaskStats, float]]: Tasks with the most wasted overhead are ranked first. Wasted overhead is
"""Analyze all tasks and return sorted by potential time savings.""" estimated as cache misses (each miss pays lookup cost but gets no benefit)
results = [] plus cache hits weighted by their relative cheapness compared to a miss.
for task in tasks: In practice this sorts by: most cache misses first, breaking ties by lower
results.append((task, calculate_cache_effectiveness(task))) hit rate.
"""
# Sort by time savings (descending - highest savings first) # Sort by cache_miss descending, then by hit rate ascending
results.sort(key=lambda x: x[1], reverse=True) tasks.sort(key=lambda t: (-t.cache_miss, t.cache_hit_rate))
return tasks
return results
def format_time(nanoseconds: float) -> str:
"""Format time in appropriate units (ns, μs, ms, s)."""
sign = "-" if nanoseconds < 0 else ""
nanoseconds = abs(nanoseconds)
if nanoseconds >= 1_000_000_000: # >= 1 second
return f"{sign}{nanoseconds / 1_000_000_000:.2f}s"
elif nanoseconds >= 1_000_000: # >= 1 millisecond
return f"{sign}{nanoseconds / 1_000_000:.2f}ms"
elif nanoseconds >= 1_000: # >= 1 microsecond
return f"{sign}{nanoseconds / 1_000:.1f}μs"
else: # nanoseconds
return f"{sign}{nanoseconds:.0f}ns"
def print_analysis(results: List[Tuple[TaskStats, float]]): def print_analysis(tasks: List[TaskStats]):
"""Print the analysis results.""" """Print the analysis results."""
print("Tasks ranked by estimated time savings from removing caching layer") print("Tasks ranked by cache effectiveness (worst first)")
print() print()
if not results: if not tasks:
print("No tasks would benefit from removing caching.") print("No tasks found.")
return return
# Print header # Print header
header = (f"{'Savings':<10} {'Hit Rate':<8} {'Exec Time':<10} " header = (f"{'Hit Rate':<10} {'Hits':<10} {'Misses':<10} "
f"{'Operations':<10} {'Task Name'}") f"{'Total':<10} {'Task Name'}")
print(header) print(header)
print("-" * len(header)) print("-" * len(header))
total_hits = 0
total_misses = 0
low_hit_rate_count = 0
# Print results # Print results
for (task, time_savings) in results: for task in tasks:
savings_str = format_time(time_savings)
hit_rate_str = f"{task.cache_hit_rate:.1%}" hit_rate_str = f"{task.cache_hit_rate:.1%}"
exec_time_str = format_time(task.avg_execution_time_ns) hits_str = f"{task.cache_hit:,}"
operations_str = f"{task.total_operations:,}" misses_str = f"{task.cache_miss:,}"
total_str = f"{task.total_operations:,}"
print(f"{hit_rate_str:<10} {hits_str:<10} {misses_str:<10} "
f"{total_str:<10} {task.name}")
print(f"{savings_str:<10} {hit_rate_str:<8} {exec_time_str:<10} " total_hits += task.cache_hit
f"{operations_str:<10} {task.name}") total_misses += task.cache_miss
if task.cache_hit_rate < 0.5:
low_hit_rate_count += 1
total_ops = total_hits + total_misses
overall_hit_rate = total_hits / total_ops if total_ops > 0 else 0.0
# Print summary # Print summary
total_savings = sum(time_savings if time_savings > 0 else 0 for _, time_savings in results)
print()
print(f"Summary: {sum(1 if time_savings > 0 else 0 for _, time_savings in results)} tasks would benefit from removing caching")
print(f"Total potential savings: {format_time(total_savings)}")
print() print()
print("Legend:") print(f"Total tasks: {len(tasks)}")
print("- Savings: Time saved by removing caching layer") print(f"Total cache misses: {total_misses:,}")
print("- Hit Rate: Percentage of operations that were cache hits") print(f"Overall cache hit rate: {overall_hit_rate:.1%} ({total_hits:,} hits / {total_ops:,} total)")
print("- Exec Time: Average execution time per operation") print(f"Tasks with <50% hit rate: {low_hit_rate_count}")
print("- Operations: Total number of cache hits + misses")
def main(): def main():
if len(sys.argv) != 2: if len(sys.argv) != 2:
print("Usage: python analyze_cache_effectiveness.py <stats-durations.json>") print("Usage: python analyze_cache_effectiveness.py <stats.json>")
sys.exit(1) sys.exit(1)
file_path = sys.argv[1] file_path = sys.argv[1]
try: try:
tasks = load_task_stats(file_path) tasks = load_task_stats(file_path)
results = analyze_tasks(tasks) tasks = analyze_tasks(tasks)
print_analysis(results) print_analysis(tasks)
except FileNotFoundError: except FileNotFoundError:
print(f"Error: File '{file_path}' not found") print(f"Error: File '{file_path}' not found")
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment