Unverified Commit be9b2ddb authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

[Turbopack] refactor filesystem writes to an effect based system (#72847)

### Why?

Before we just wrote the files in `FileSystem::write()`, but that caused some issues when two `FileSystem::write()` tasks are active (or even temporarily active) at the same time. They invalidate each other and basically caused a loop of writes.

For a long time we wanted to have a better solution for that, since that problem causes compilation to hang with 100% cpu load, which is quite annoying.

Also the eventual consistency can cause some unexpected writes to be done when the content or filename are on a temporary value that has not stabilised yet.

The problem applies to every side effect in tasks.

### What?

To solve that this refactoring changes the way side effects in tasks are handled.

Side effects in tasks should not be directly executed anymore. Instead one should emit an "effect" (`effect(async { ... })` helper method) to declare a side effect that should be executed. Top level execution is usually wrapped in a strongly consistent read of the top level task. Here one need to apply the "effects" (`apply_effects(operation).await?` helper method). Since this happens after the strongly consistent read, only effects that have been stabilised are executed.

It's also possible to call `apply_effects` somewhere nested in the execution (after a strongly consistent read) if side effects are necessary for further execution. e. g. we do that to emit the files for the node.js pool before executing the pool.

Internally effects are collectibles that are propagated through the call graph. Each effect can only be executed once, so updates to the graph will only apply new effects while existing effects are skipped as they were already executed.

The actual change is pretty simple, but `FileSystem::write` no longer returns a `Vc<Completion>` but nothing instead. This makes the change needed a bit bigger.

Good news: To emit a whole (potential circular) graph we could use recursion now instead of needing to walk the graph while avoiding cycles. This might allow some performance improvements for incremental builds.
parent 2186f7e0
......@@ -7,7 +7,7 @@ use next_api::{
route::{Endpoint, WrittenEndpoint},
};
use tracing::Instrument;
use turbo_tasks::{Completion, ReadRef, Vc, VcValueType};
use turbo_tasks::{get_effects, Completion, Effects, ReadRef, Vc, VcValueType};
use turbopack_core::{
diagnostics::PlainDiagnostic,
error::PrettyPrintError,
......@@ -104,10 +104,12 @@ async fn strongly_consistent_catch_collectables<R: VcValueType + Send>(
Option<ReadRef<R>>,
Arc<Vec<ReadRef<PlainIssue>>>,
Arc<Vec<ReadRef<PlainDiagnostic>>>,
Arc<Effects>,
)> {
let result = source.strongly_consistent().await;
let issues = get_issues(source).await?;
let diagnostics = get_diagnostics(source).await?;
let effects = Arc::new(get_effects(source).await?);
let result = if result.is_err() && issues.iter().any(|i| i.severity <= IssueSeverity::Error) {
None
......@@ -115,7 +117,7 @@ async fn strongly_consistent_catch_collectables<R: VcValueType + Send>(
Some(result?)
};
Ok((result, issues, diagnostics))
Ok((result, issues, diagnostics, effects))
}
#[turbo_tasks::value(serialization = "none")]
......@@ -123,6 +125,7 @@ struct WrittenEndpointWithIssues {
written: Option<ReadRef<WrittenEndpoint>>,
issues: Arc<Vec<ReadRef<PlainIssue>>>,
diagnostics: Arc<Vec<ReadRef<PlainDiagnostic>>>,
effects: Arc<Effects>,
}
#[turbo_tasks::function]
......@@ -130,12 +133,13 @@ async fn get_written_endpoint_with_issues(
endpoint: Vc<Box<dyn Endpoint>>,
) -> Result<Vc<WrittenEndpointWithIssues>> {
let write_to_disk = endpoint.write_to_disk();
let (written, issues, diagnostics) =
let (written, issues, diagnostics, effects) =
strongly_consistent_catch_collectables(write_to_disk).await?;
Ok(WrittenEndpointWithIssues {
written,
issues,
diagnostics,
effects,
}
.cell())
}
......@@ -149,13 +153,15 @@ pub async fn endpoint_write_to_disk(
let endpoint = ***endpoint;
let (written, issues, diags) = turbo_tasks
.run_once(async move {
let operation = get_written_endpoint_with_issues(endpoint);
let WrittenEndpointWithIssues {
written,
issues,
diagnostics,
} = &*get_written_endpoint_with_issues(endpoint)
.strongly_consistent()
.await?;
effects,
} = &*operation.strongly_consistent().await?;
effects.apply().await?;
Ok((written.clone(), issues.clone(), diagnostics.clone()))
})
.await
......@@ -180,9 +186,10 @@ pub fn endpoint_server_changed_subscribe(
func,
move || {
async move {
subscribe_issues_and_diags(endpoint, issues)
.strongly_consistent()
.await
let operation = subscribe_issues_and_diags(endpoint, issues);
let result = operation.strongly_consistent().await?;
result.effects.apply().await?;
Ok(result)
}
.instrument(tracing::info_span!("server changes subscription"))
},
......@@ -191,6 +198,7 @@ pub fn endpoint_server_changed_subscribe(
changed: _,
issues,
diagnostics,
effects: _,
} = &*ctx.value;
Ok(vec![TurbopackResult {
......@@ -210,6 +218,7 @@ struct EndpointIssuesAndDiags {
changed: Option<ReadRef<Completion>>,
issues: Arc<Vec<ReadRef<PlainIssue>>>,
diagnostics: Arc<Vec<ReadRef<PlainDiagnostic>>>,
effects: Arc<Effects>,
}
impl PartialEq for EndpointIssuesAndDiags {
......@@ -233,12 +242,13 @@ async fn subscribe_issues_and_diags(
let changed = endpoint.server_changed();
if should_include_issues {
let (changed_value, issues, diagnostics) =
let (changed_value, issues, diagnostics, effects) =
strongly_consistent_catch_collectables(changed).await?;
Ok(EndpointIssuesAndDiags {
changed: changed_value,
issues,
diagnostics,
effects,
}
.cell())
} else {
......@@ -247,6 +257,7 @@ async fn subscribe_issues_and_diags(
changed: Some(changed_value),
issues: Arc::new(vec![]),
diagnostics: Arc::new(vec![]),
effects: Arc::new(Effects::default()),
}
.cell())
}
......
......@@ -24,7 +24,7 @@ use tokio::{io::AsyncWriteExt, time::Instant};
use tracing::Instrument;
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt, EnvFilter, Registry};
use turbo_rcstr::RcStr;
use turbo_tasks::{Completion, ReadRef, TransientInstance, UpdateInfo, Vc};
use turbo_tasks::{get_effects, Completion, Effects, ReadRef, TransientInstance, UpdateInfo, Vc};
use turbo_tasks_fs::{
util::uri_from_file, DiskFileSystem, FileContent, FileSystem, FileSystemPath,
};
......@@ -623,6 +623,7 @@ struct EntrypointsWithIssues {
entrypoints: ReadRef<Entrypoints>,
issues: Arc<Vec<ReadRef<PlainIssue>>>,
diagnostics: Arc<Vec<ReadRef<PlainDiagnostic>>>,
effects: Arc<Effects>,
}
#[turbo_tasks::function]
......@@ -633,10 +634,12 @@ async fn get_entrypoints_with_issues(
let entrypoints = entrypoints_operation.strongly_consistent().await?;
let issues = get_issues(entrypoints_operation).await?;
let diagnostics = get_diagnostics(entrypoints_operation).await?;
let effects = Arc::new(get_effects(entrypoints_operation).await?);
Ok(EntrypointsWithIssues {
entrypoints,
issues,
diagnostics,
effects,
}
.cell())
}
......@@ -653,13 +656,14 @@ pub fn project_entrypoints_subscribe(
func,
move || {
async move {
let operation = get_entrypoints_with_issues(container);
let EntrypointsWithIssues {
entrypoints,
issues,
diagnostics,
} = &*get_entrypoints_with_issues(container)
.strongly_consistent()
.await?;
effects,
} = &*operation.strongly_consistent().await?;
effects.apply().await?;
Ok((entrypoints.clone(), issues.clone(), diagnostics.clone()))
}
.instrument(tracing::info_span!("entrypoints subscription"))
......@@ -718,6 +722,7 @@ struct HmrUpdateWithIssues {
update: ReadRef<Update>,
issues: Arc<Vec<ReadRef<PlainIssue>>>,
diagnostics: Arc<Vec<ReadRef<PlainDiagnostic>>>,
effects: Arc<Effects>,
}
#[turbo_tasks::function]
......@@ -730,10 +735,12 @@ async fn hmr_update(
let update = update_operation.strongly_consistent().await?;
let issues = get_issues(update_operation).await?;
let diagnostics = get_diagnostics(update_operation).await?;
let effects = Arc::new(get_effects(update_operation).await?);
Ok(HmrUpdateWithIssues {
update,
issues,
diagnostics,
effects,
}
.cell())
}
......@@ -760,14 +767,15 @@ pub fn project_hmr_events(
let project = project.project().resolve().await?;
let state = project.hmr_version_state(identifier.clone(), session);
let update = hmr_update(project, identifier.clone(), state)
.strongly_consistent()
.await?;
let operation = hmr_update(project, identifier.clone(), state);
let update = operation.strongly_consistent().await?;
let HmrUpdateWithIssues {
update,
issues,
diagnostics,
effects,
} = &*update;
effects.apply().await?;
match &**update {
Update::Missing | Update::None => {}
Update::Total(TotalUpdate { to }) => {
......@@ -832,6 +840,7 @@ struct HmrIdentifiersWithIssues {
identifiers: ReadRef<Vec<RcStr>>,
issues: Arc<Vec<ReadRef<PlainIssue>>>,
diagnostics: Arc<Vec<ReadRef<PlainDiagnostic>>>,
effects: Arc<Effects>,
}
#[turbo_tasks::function]
......@@ -842,10 +851,12 @@ async fn get_hmr_identifiers_with_issues(
let hmr_identifiers = hmr_identifiers_operation.strongly_consistent().await?;
let issues = get_issues(hmr_identifiers_operation).await?;
let diagnostics = get_diagnostics(hmr_identifiers_operation).await?;
let effects = Arc::new(get_effects(hmr_identifiers_operation).await?);
Ok(HmrIdentifiersWithIssues {
identifiers: hmr_identifiers,
issues,
diagnostics,
effects,
}
.cell())
}
......@@ -861,13 +872,14 @@ pub fn project_hmr_identifiers_subscribe(
turbo_tasks.clone(),
func,
move || async move {
let operation = get_hmr_identifiers_with_issues(container);
let HmrIdentifiersWithIssues {
identifiers,
issues,
diagnostics,
} = &*get_hmr_identifiers_with_issues(container)
.strongly_consistent()
.await?;
effects,
} = &*operation.strongly_consistent().await?;
effects.apply().await?;
Ok((identifiers.clone(), issues.clone(), diagnostics.clone()))
},
......
......@@ -1647,10 +1647,10 @@ impl Endpoint for AppEndpoint {
let node_root_ref = &node_root.await?;
this.app_project
let _ = this
.app_project
.project()
.emit_all_output_assets(Vc::cell(output_assets))
.await?;
.emit_all_output_assets(Vc::cell(output_assets));
let node_root = this.app_project.project().node_root();
let server_paths = all_server_paths(output_assets, node_root)
......
......@@ -231,9 +231,7 @@ impl Endpoint for InstrumentationEndpoint {
let this = self.await?;
let output_assets = self.output_assets();
let _ = output_assets.resolve().await?;
this.project
.emit_all_output_assets(Vc::cell(output_assets))
.await?;
let _ = this.project.emit_all_output_assets(Vc::cell(output_assets));
let node_root = this.project.node_root();
let server_paths = all_server_paths(output_assets, node_root)
......
......@@ -276,9 +276,7 @@ impl Endpoint for MiddlewareEndpoint {
let this = self.await?;
let output_assets = self.output_assets();
let _ = output_assets.resolve().await?;
this.project
.emit_all_output_assets(Vc::cell(output_assets))
.await?;
let _ = this.project.emit_all_output_assets(Vc::cell(output_assets));
let node_root = this.project.node_root();
let server_paths = all_server_paths(output_assets, node_root)
......
......@@ -64,7 +64,7 @@ impl NftJsonAsset {
let output_fs = this.output_fs.await?;
let nft_folder = self.ident().path().parent().await?;
if let Some(subdir) = output_fs.root.strip_prefix(&*project_fs.root) {
if let Some(subdir) = output_fs.root().strip_prefix(&**project_fs.root()) {
Ok(this
.project_fs
.root()
......
......@@ -1290,10 +1290,10 @@ impl Endpoint for PageEndpoint {
// single operation
let output_assets = self.output_assets();
this.pages_project
let _ = this
.pages_project
.project()
.emit_all_output_assets(Vc::cell(output_assets))
.await?;
.emit_all_output_assets(Vc::cell(output_assets));
let node_root = this.pages_project.project().node_root();
let server_paths = all_server_paths(output_assets, node_root)
......
......@@ -1164,7 +1164,7 @@ impl Project {
pub async fn emit_all_output_assets(
self: Vc<Self>,
output_assets: Vc<OutputAssetsOperation>,
) -> Result<Vc<()>> {
) -> Result<()> {
let span = tracing::info_span!("emitting");
async move {
let all_output_assets = all_assets_from_entries_operation(output_assets);
......@@ -1173,27 +1173,23 @@ impl Project {
let node_root = self.node_root();
if let Some(map) = self.await?.versioned_content_map {
let _ = map
.insert_output_assets(
all_output_assets,
node_root,
client_relative_path,
node_root,
)
.resolve()
.await?;
let _ = map.insert_output_assets(
all_output_assets,
node_root,
client_relative_path,
node_root,
);
Ok(Vc::cell(()))
Ok(())
} else {
let _ = emit_assets(
*all_output_assets.await?,
node_root,
client_relative_path,
node_root,
)
.resolve()
.await?;
Ok(Vc::cell(()))
);
Ok(())
}
}
.instrument(span)
......
......@@ -5,8 +5,8 @@ use next_core::emit_assets;
use serde::{Deserialize, Serialize};
use turbo_rcstr::RcStr;
use turbo_tasks::{
debug::ValueDebugFormat, trace::TraceRawVcs, Completion, FxIndexSet, ResolvedVc, State,
TryFlatJoinIterExt, TryJoinIterExt, ValueDefault, ValueToString, Vc,
debug::ValueDebugFormat, trace::TraceRawVcs, FxIndexSet, ResolvedVc, State, TryFlatJoinIterExt,
TryJoinIterExt, ValueDefault, ValueToString, Vc,
};
use turbo_tasks_fs::FileSystemPath;
use turbopack_core::{
......@@ -24,8 +24,8 @@ pub struct OutputAssetsOperation(Vc<OutputAssets>);
#[derive(Clone, TraceRawVcs, PartialEq, Eq, ValueDebugFormat, Serialize, Deserialize, Debug)]
struct MapEntry {
// must not be resolved
assets_operation: Vc<OutputAssets>,
side_effects: Vc<Completion>,
/// Precomputed map for quick access to output asset by filepath
path_to_asset: HashMap<ResolvedVc<FileSystemPath>, Vc<Box<dyn OutputAsset>>>,
}
......@@ -74,7 +74,7 @@ impl VersionedContentMap {
node_root: Vc<FileSystemPath>,
client_relative_path: Vc<FileSystemPath>,
client_output_path: Vc<FileSystemPath>,
) -> Result<Vc<Completion>> {
) -> Result<()> {
let this = self.await?;
let compute_entry = self.compute_entry(
assets_operation,
......@@ -85,10 +85,7 @@ impl VersionedContentMap {
let assets = *assets_operation.await?;
this.map_op_to_compute_entry
.update_conditionally(|map| map.insert(assets, compute_entry) != Some(compute_entry));
let Some(entry) = &*compute_entry.await? else {
unreachable!("compute_entry always returns Some(MapEntry)")
};
Ok(entry.side_effects)
Ok(())
}
/// Creates a [`MapEntry`] (a pre-computed map for optimized lookup) for an output assets
......@@ -143,10 +140,9 @@ impl VersionedContentMap {
});
// Make sure all written client assets are up-to-date
let side_effects = emit_assets(assets, node_root, client_relative_path, client_output_path);
let _ = emit_assets(assets, node_root, client_relative_path, client_output_path);
let map_entry = Vc::cell(Some(MapEntry {
assets_operation: assets,
side_effects,
path_to_asset: entries.into_iter().collect(),
}));
Ok(map_entry)
......@@ -195,12 +191,9 @@ impl VersionedContentMap {
let result = self.raw_get(*path).await?;
if let Some(MapEntry {
assets_operation: _,
side_effects,
path_to_asset,
}) = &*result
{
side_effects.await?;
if let Some(asset) = path_to_asset.get(&path) {
return Ok(Vc::cell(Some(asset.to_resolved().await?)));
} else {
......
......@@ -2,7 +2,7 @@ use anyhow::Result;
use tracing::Instrument;
use turbo_tasks::{
graph::{AdjacencyMap, GraphTraversal},
Completion, Completions, ResolvedVc, TryFlatJoinIterExt, ValueToString, Vc,
ResolvedVc, TryFlatJoinIterExt, ValueToString, Vc,
};
use turbo_tasks_fs::{rebase, FileSystemPath};
use turbopack_core::{
......@@ -21,13 +21,13 @@ pub fn emit_all_assets(
node_root: Vc<FileSystemPath>,
client_relative_path: Vc<FileSystemPath>,
client_output_path: Vc<FileSystemPath>,
) -> Vc<Completion> {
emit_assets(
) {
let _ = emit_assets(
all_assets_from_entries(assets),
node_root,
client_relative_path,
client_output_path,
)
);
}
/// Emits all assets transitively reachable from the given chunks, that are
......@@ -41,42 +41,39 @@ pub async fn emit_assets(
node_root: Vc<FileSystemPath>,
client_relative_path: Vc<FileSystemPath>,
client_output_path: Vc<FileSystemPath>,
) -> Result<Vc<Completion>> {
Ok(Vc::<Completions>::cell(
assets
.await?
.iter()
.copied()
.map(|asset| async move {
let asset = asset.resolve().await?;
let path = asset.ident().path();
let span =
tracing::info_span!("emit asset", name = path.to_string().await?.as_str());
async move {
let path = path.await?;
Ok(if path.is_inside_ref(&*node_root.await?) {
Some(emit(asset))
} else if path.is_inside_ref(&*client_relative_path.await?) {
// Client assets are emitted to the client output path, which is prefixed
// with _next. We need to rebase them to remove that
// prefix.
Some(emit_rebase(asset, client_relative_path, client_output_path))
} else {
None
})
}
.instrument(span)
.await
})
.try_flat_join()
.await?,
)
.completed())
) -> Result<()> {
let _: Vec<Vc<()>> = assets
.await?
.iter()
.copied()
.map(|asset| async move {
let asset = asset.resolve().await?;
let path = asset.ident().path();
let span = tracing::info_span!("emit asset", name = %path.to_string().await?);
async move {
let path = path.await?;
Ok(if path.is_inside_ref(&*node_root.await?) {
Some(emit(asset))
} else if path.is_inside_ref(&*client_relative_path.await?) {
// Client assets are emitted to the client output path, which is prefixed
// with _next. We need to rebase them to remove that
// prefix.
Some(emit_rebase(asset, client_relative_path, client_output_path))
} else {
None
})
}
.instrument(span)
.await
})
.try_flat_join()
.await?;
Ok(())
}
#[turbo_tasks::function]
fn emit(asset: Vc<Box<dyn OutputAsset>>) -> Vc<Completion> {
asset.content().write(asset.ident().path())
fn emit(asset: Vc<Box<dyn OutputAsset>>) {
let _ = asset.content().write(asset.ident().path());
}
#[turbo_tasks::function]
......@@ -84,10 +81,11 @@ async fn emit_rebase(
asset: Vc<Box<dyn OutputAsset>>,
from: Vc<FileSystemPath>,
to: Vc<FileSystemPath>,
) -> Result<Vc<Completion>> {
) -> Result<()> {
let path = rebase(asset.ident().path(), from, to);
let content = asset.content();
Ok(content.resolve().await?.write(path.resolve().await?))
let _ = content.resolve().await?.write(path.resolve().await?);
Ok(())
}
/// Walks the asset graph from multiple assets and collect all referenced
......
......@@ -164,16 +164,12 @@ impl FileSystem for AttachedFileSystem {
}
#[turbo_tasks::function(fs)]
fn write(self: Vc<Self>, path: Vc<FileSystemPath>, content: Vc<FileContent>) -> Vc<Completion> {
fn write(self: Vc<Self>, path: Vc<FileSystemPath>, content: Vc<FileContent>) -> Vc<()> {
self.get_inner_fs_path(path).write(content)
}
#[turbo_tasks::function(fs)]
fn write_link(
self: Vc<Self>,
path: Vc<FileSystemPath>,
target: Vc<LinkContent>,
) -> Vc<Completion> {
fn write_link(self: Vc<Self>, path: Vc<FileSystemPath>, target: Vc<LinkContent>) -> Vc<()> {
self.get_inner_fs_path(path).write_link(target)
}
......
......@@ -76,20 +76,12 @@ impl FileSystem for EmbeddedFileSystem {
}
#[turbo_tasks::function]
fn write(
&self,
_path: Vc<FileSystemPath>,
_content: Vc<FileContent>,
) -> Result<Vc<Completion>> {
fn write(&self, _path: Vc<FileSystemPath>, _content: Vc<FileContent>) -> Result<Vc<()>> {
bail!("Writing is not possible to the embedded filesystem")
}
#[turbo_tasks::function]
fn write_link(
&self,
_path: Vc<FileSystemPath>,
_target: Vc<LinkContent>,
) -> Result<Vc<Completion>> {
fn write_link(&self, _path: Vc<FileSystemPath>, _target: Vc<LinkContent>) -> Result<Vc<()>> {
bail!("Writing is not possible to the embedded filesystem")
}
......
This diff is collapsed.
......@@ -66,20 +66,12 @@ impl FileSystem for VirtualFileSystem {
}
#[turbo_tasks::function]
fn write(
&self,
_fs_path: Vc<FileSystemPath>,
_content: Vc<FileContent>,
) -> Result<Vc<Completion>> {
fn write(&self, _fs_path: Vc<FileSystemPath>, _content: Vc<FileContent>) -> Result<Vc<()>> {
bail!("Writing is not possible on the virtual file system")
}
#[turbo_tasks::function]
fn write_link(
&self,
_fs_path: Vc<FileSystemPath>,
_target: Vc<LinkContent>,
) -> Result<Vc<Completion>> {
fn write_link(&self, _fs_path: Vc<FileSystemPath>, _target: Vc<LinkContent>) -> Result<Vc<()>> {
bail!("Writing is not possible on the virtual file system")
}
......
......@@ -16,16 +16,13 @@ use notify::{
};
use rayon::iter::{IntoParallelIterator, ParallelIterator};
use serde::{Deserialize, Serialize};
use tokio::sync::RwLock;
use tracing::instrument;
use turbo_rcstr::RcStr;
use turbo_tasks::{spawn_thread, Invalidator};
use crate::{
format_absolute_fs_path,
invalidation::{WatchChange, WatchStart},
invalidator_map::InvalidatorMap,
path_to_key,
path_to_key, DiskFileSystemInner,
};
enum DiskWatcherInternal {
......@@ -137,13 +134,9 @@ impl DiskWatcher {
/// - Doesn't emit duplicate create events
/// - Doesn't emit Modify events after a Create event
pub(crate) fn start_watching(
self: Arc<Self>,
name: RcStr,
root_path: PathBuf,
report_invalidation_reason: Option<(RcStr, PathBuf)>,
invalidation_lock: Arc<RwLock<()>>,
invalidator_map: Arc<InvalidatorMap>,
dir_invalidator_map: Arc<InvalidatorMap>,
&self,
inner: Arc<DiskFileSystemInner>,
report_invalidation_reason: bool,
poll_interval: Option<Duration>,
) -> Result<()> {
let mut watcher_guard = self.watcher.lock().unwrap();
......@@ -169,7 +162,7 @@ impl DiskWatcher {
// below will be monitored for changes.
#[cfg(any(target_os = "macos", target_os = "windows"))]
{
watcher.watch(&root_path, RecursiveMode::Recursive)?;
watcher.watch(inner.root_path(), RecursiveMode::Recursive)?;
}
#[cfg(not(any(target_os = "macos", target_os = "windows")))]
......@@ -182,17 +175,17 @@ impl DiskWatcher {
{
let _span = tracing::info_span!("invalidate filesystem").entered();
let span = tracing::Span::current();
let invalidator_map = take(&mut *invalidator_map.lock().unwrap());
let dir_invalidator_map = take(&mut *dir_invalidator_map.lock().unwrap());
let invalidator_map = take(&mut *inner.invalidator_map.lock().unwrap());
let dir_invalidator_map = take(&mut *inner.dir_invalidator_map.lock().unwrap());
let iter = invalidator_map
.into_par_iter()
.chain(dir_invalidator_map.into_par_iter());
let handle = tokio::runtime::Handle::current();
if report_invalidation_reason.is_some() {
if report_invalidation_reason {
iter.flat_map(|(path, invalidators)| {
let _span = span.clone().entered();
let reason = WatchStart {
name: name.clone(),
name: inner.name.clone(),
path: path.into(),
};
invalidators
......@@ -221,14 +214,10 @@ impl DiskWatcher {
drop(watcher_guard);
spawn_thread(move || {
self.watch_thread(
rx,
root_path,
report_invalidation_reason,
invalidation_lock,
invalidator_map,
dir_invalidator_map,
)
inner
.clone()
.watcher
.watch_thread(rx, inner, report_invalidation_reason)
});
Ok(())
......@@ -248,11 +237,8 @@ impl DiskWatcher {
fn watch_thread(
&self,
rx: Receiver<notify::Result<notify::Event>>,
root_path: PathBuf,
report_invalidation_reason: Option<(RcStr, PathBuf)>,
invalidation_lock: Arc<RwLock<()>>,
invalidator_map: Arc<InvalidatorMap>,
dir_invalidator_map: Arc<InvalidatorMap>,
inner: Arc<DiskFileSystemInner>,
report_invalidation_reason: bool,
) {
let mut batched_invalidate_path = HashSet::new();
let mut batched_invalidate_path_dir = HashSet::new();
......@@ -377,8 +363,10 @@ impl DiskWatcher {
println!("watch error ({:?}): {:?} ", paths, kind);
if paths.is_empty() {
batched_invalidate_path_and_children.insert(root_path.clone());
batched_invalidate_path_and_children_dir.insert(root_path.clone());
batched_invalidate_path_and_children
.insert(inner.root_path().to_path_buf());
batched_invalidate_path_and_children_dir
.insert(inner.root_path().to_path_buf());
} else {
batched_invalidate_path_and_children.extend(paths.clone());
batched_invalidate_path_and_children_dir.extend(paths.clone());
......@@ -413,33 +401,37 @@ impl DiskWatcher {
#[cfg(not(any(target_os = "macos", target_os = "windows")))]
{
for path in batched_new_paths.drain() {
let _ = self.restore_if_watching(&path, &root_path);
let _ = self.restore_if_watching(&path, inner.root_path());
}
}
let _lock = invalidation_lock.blocking_write();
let _lock = inner.invalidation_lock.blocking_write();
{
let mut invalidator_map = invalidator_map.lock().unwrap();
let mut invalidator_map = inner.invalidator_map.lock().unwrap();
invalidate_path(
&report_invalidation_reason,
&inner,
report_invalidation_reason,
&mut invalidator_map,
batched_invalidate_path.drain(),
);
invalidate_path_and_children_execute(
&report_invalidation_reason,
&inner,
report_invalidation_reason,
&mut invalidator_map,
batched_invalidate_path_and_children.drain(),
);
}
{
let mut dir_invalidator_map = dir_invalidator_map.lock().unwrap();
let mut dir_invalidator_map = inner.dir_invalidator_map.lock().unwrap();
invalidate_path(
&report_invalidation_reason,
&inner,
report_invalidation_reason,
&mut dir_invalidator_map,
batched_invalidate_path_dir.drain(),
);
invalidate_path_and_children_execute(
&report_invalidation_reason,
&inner,
report_invalidation_reason,
&mut dir_invalidator_map,
batched_invalidate_path_and_children_dir.drain(),
);
......@@ -450,12 +442,13 @@ impl DiskWatcher {
#[instrument(parent = None, level = "info", name = "DiskFileSystem file change", skip_all, fields(name = display(path.display())))]
fn invalidate(
report_invalidation_reason: &Option<(RcStr, PathBuf)>,
inner: &DiskFileSystemInner,
report_invalidation_reason: bool,
path: &Path,
invalidator: Invalidator,
) {
if let Some((name, root_path)) = report_invalidation_reason {
if let Some(path) = format_absolute_fs_path(path, name, root_path) {
if report_invalidation_reason {
if let Some(path) = format_absolute_fs_path(path, &inner.name, inner.root_path()) {
invalidator.invalidate_with_reason(WatchChange { path });
return;
}
......@@ -464,7 +457,8 @@ fn invalidate(
}
fn invalidate_path(
report_invalidation_reason: &Option<(RcStr, PathBuf)>,
inner: &DiskFileSystemInner,
report_invalidation_reason: bool,
invalidator_map: &mut HashMap<String, HashSet<Invalidator>>,
paths: impl Iterator<Item = PathBuf>,
) {
......@@ -473,13 +467,14 @@ fn invalidate_path(
if let Some(invalidators) = invalidator_map.remove(&key) {
invalidators
.into_iter()
.for_each(|i| invalidate(report_invalidation_reason, &path, i));
.for_each(|i| invalidate(inner, report_invalidation_reason, &path, i));
}
}
}
fn invalidate_path_and_children_execute(
report_invalidation_reason: &Option<(RcStr, PathBuf)>,
inner: &DiskFileSystemInner,
report_invalidation_reason: bool,
invalidator_map: &mut HashMap<String, HashSet<Invalidator>>,
paths: impl Iterator<Item = PathBuf>,
) {
......@@ -488,7 +483,7 @@ fn invalidate_path_and_children_execute(
for (_, invalidators) in invalidator_map.extract_if(|key, _| key.starts_with(&path_key)) {
invalidators
.into_iter()
.for_each(|i| invalidate(report_invalidation_reason, &path, i));
.for_each(|i| invalidate(inner, report_invalidation_reason, &path, i));
}
}
}
use std::{borrow::Cow, collections::HashSet, future::Future, panic, pin::Pin};
use anyhow::{anyhow, Result};
use auto_hash_map::AutoSet;
use parking_lot::Mutex;
use tokio::task::JoinHandle;
use tracing::{Instrument, Span};
use turbo_tasks_macros::{TraceRawVcs, ValueDebugFormat};
use crate::{
self as turbo_tasks, emit, manager::turbo_tasks_future_scope, CollectiblesSource, ReadRef,
TryJoinIterExt, Vc,
};
/// A trait to emit a task effect as collectible. This trait only has one
/// implementation, `EffectInstance` and no other implementation is allowed.
/// The trait is private to this module so that no other implementation can be
/// added.
#[turbo_tasks::value_trait]
trait Effect {}
/// A future that represents the effect of a task. The future is executed when
/// the effect is applied.
type EffectFuture = Pin<Box<dyn Future<Output = Result<()>> + Send + Sync + 'static>>;
/// The inner state of an effect instance if it has not been applied yet.
struct EffectInner {
future: EffectFuture,
span: Span,
}
/// The Effect instance collectible that is emitted for effects.
#[turbo_tasks::value(serialization = "none", cell = "new", eq = "manual")]
struct EffectInstance {
#[turbo_tasks(trace_ignore, debug_ignore)]
inner: Mutex<Option<EffectInner>>,
}
impl EffectInstance {
fn new(future: impl Future<Output = Result<()>> + Send + Sync + 'static) -> Self {
Self {
inner: Mutex::new(Some(EffectInner {
future: Box::pin(future),
span: Span::current(),
})),
}
}
fn apply(&self) -> Option<JoinHandle<Result<()>>> {
let future = self.inner.lock().take();
future.map(|EffectInner { future, span }| {
tokio::spawn(
turbo_tasks_future_scope(turbo_tasks::turbo_tasks(), future).instrument(span),
)
})
}
}
#[turbo_tasks::value_impl]
impl Effect for EffectInstance {}
/// Schedules an effect to be applied. The passed future is executed once `apply_effects` is called.
///
/// The effect will only executed once. The passed future is executed outside of the current task
/// and can't read any Vcs. These need to be read before. ReadRefs can be passed into the future.
///
/// Effects are executed in parallel, so they might need to use async locking to avoid problems.
/// Order of execution of multiple effects is not defined. You must not use mutliple conflicting
/// effects to avoid non-deterministic behavior.
pub fn effect(future: impl Future<Output = Result<()>> + Send + Sync + 'static) {
emit::<Box<dyn Effect>>(Vc::upcast(EffectInstance::new(future).cell()));
}
/// Applies all effects that have been emitted by an operations.
///
/// Usually it's important that effects are strongly consistent, so one want to use `apply_effects`
/// only on operations that have been strongly consistently read before.
///
/// The order of execution is not defined and effects are executed in parallel.
///
/// `apply_effects` must only be used in a "once" task. When used in a "root" task, a
/// combination of `get_effects` and `Effects::apply` must be used.
///
/// # Example
///
/// ```rust
/// let operation = some_turbo_tasks_function(args);
/// let result = operation.strongly_consistent().await?;
/// apply_effects(operation).await?;
/// ```
pub async fn apply_effects(source: impl CollectiblesSource) -> Result<()> {
let effects: AutoSet<Vc<Box<dyn Effect>>> = source.take_collectibles();
if effects.is_empty() {
return Ok(());
}
let span = tracing::info_span!("apply effects", count = effects.len());
async move {
let mut first_error = anyhow::Ok(());
for effect in effects {
let Some(effect) = Vc::try_resolve_downcast_type::<EffectInstance>(effect).await?
else {
panic!("Effect must only be implemented by EffectInstance");
};
apply_effect(&effect.await?, &mut first_error).await;
}
first_error
}
.instrument(span)
.await
}
/// Capture effects from an turbo-tasks operation. Since this captures collectibles it might
/// invalidate the current task when effects are changing or even temporary change.
///
/// Therefore it's important to wrap this in a strongly consistent read before applying the effects
/// with `Effects::apply`.
///
/// # Example
///
/// ```rust
/// async fn some_turbo_tasks_function_with_effects(args: Args) -> Result<ResultWithEffects> {
/// let operation = some_turbo_tasks_function(args);
/// let result = operation.strongly_consistent().await?;
/// let effects = get_effects(operation).await?;
/// Ok(ResultWithEffects { result, effects })
/// }
///
/// let result_with_effects = some_turbo_tasks_function_with_effects(args).strongly_consistent().await?;
/// result_with_effects.effects.apply().await?;
/// ```
pub async fn get_effects(source: impl CollectiblesSource) -> Result<Effects> {
let effects: AutoSet<Vc<Box<dyn Effect>>> = source.take_collectibles();
let effects = effects
.into_iter()
.map(|effect| async move {
if let Some(effect) = Vc::try_resolve_downcast_type::<EffectInstance>(effect).await? {
Ok(effect.await?)
} else {
panic!("Effect must only be implemented by EffectInstance");
}
})
.try_join()
.await?;
Ok(Effects { effects })
}
/// Captured effects from an operation. This struct can be used to return Effects from a turbo-tasks
/// function and apply them later.
#[derive(TraceRawVcs, Default, ValueDebugFormat)]
pub struct Effects {
#[turbo_tasks(trace_ignore, debug_ignore)]
effects: Vec<ReadRef<EffectInstance>>,
}
impl PartialEq for Effects {
fn eq(&self, other: &Self) -> bool {
if self.effects.len() != other.effects.len() {
return false;
}
let effect_ptrs = self
.effects
.iter()
.map(ReadRef::ptr)
.collect::<HashSet<_>>();
other
.effects
.iter()
.all(|e| effect_ptrs.contains(&ReadRef::ptr(e)))
}
}
impl Eq for Effects {}
impl Effects {
/// Applies all effects that have been captured by this struct.
pub async fn apply(&self) -> Result<()> {
let _span = tracing::info_span!("apply effects", count = self.effects.len());
let mut first_error = anyhow::Ok(());
for effect in self.effects.iter() {
apply_effect(effect, &mut first_error).await;
}
first_error
}
}
async fn apply_effect(
effect: &ReadRef<EffectInstance>,
first_error: &mut std::result::Result<(), anyhow::Error>,
) {
if let Some(join_handle) = effect.apply() {
match join_handle.await {
Ok(Err(err)) if first_error.is_ok() => {
*first_error = Err(err);
}
Err(err) if first_error.is_ok() => {
let any = err.into_panic();
let panic = match any.downcast::<String>() {
Ok(owned) => Some(Cow::Owned(*owned)),
Err(any) => match any.downcast::<&'static str>() {
Ok(str) => Some(Cow::Borrowed(*str)),
Err(_) => None,
},
};
*first_error = Err(if let Some(panic) = panic {
anyhow!("Task effect panicked: {panic}")
} else {
anyhow!("Task effect panicked")
});
}
_ => {}
}
}
}
#[cfg(test)]
mod tests {
use crate::{apply_effects, get_effects, CollectiblesSource};
#[test]
#[allow(dead_code)]
fn is_sync_and_send() {
fn assert_sync<T: Sync + Send>(_: T) {}
fn check_apply_effects<T: CollectiblesSource + Send + Sync>(t: T) {
assert_sync(apply_effects(t));
}
fn check_get_effects<T: CollectiblesSource + Send + Sync>(t: T) {
assert_sync(get_effects(t));
}
}
}
......@@ -43,6 +43,7 @@ mod completion;
pub mod debug;
mod display;
pub mod duration_span;
mod effect;
pub mod event;
pub mod graph;
mod id;
......@@ -85,6 +86,7 @@ use auto_hash_map::AutoSet;
pub use collectibles::CollectiblesSource;
pub use completion::{Completion, Completions};
pub use display::ValueToString;
pub use effect::{apply_effects, effect, get_effects, Effects};
pub use id::{
ExecutionId, FunctionId, LocalTaskId, SessionId, TaskId, TraitTypeId, ValueTypeId,
TRANSIENT_TASK_BIT,
......
......@@ -1672,6 +1672,13 @@ pub fn turbo_tasks_scope<T>(tt: Arc<dyn TurboTasksApi>, f: impl FnOnce() -> T) -
TURBO_TASKS.sync_scope(tt, f)
}
pub fn turbo_tasks_future_scope<T>(
tt: Arc<dyn TurboTasksApi>,
f: impl Future<Output = T>,
) -> impl Future<Output = T> {
TURBO_TASKS.scope(tt, f)
}
pub fn with_turbo_tasks_for_testing<T>(
tt: Arc<dyn TurboTasksApi>,
current_task: TaskId,
......
......@@ -234,6 +234,10 @@ impl<T> ReadRef<T> {
pub fn ptr_eq(&self, other: &ReadRef<T>) -> bool {
triomphe::Arc::ptr_eq(&self.0, &other.0)
}
pub fn ptr(&self) -> *const T {
&*self.0 as *const T
}
}
impl<T> ReadRef<T>
......
......@@ -8,7 +8,8 @@ use std::{
use anyhow::{bail, Context, Result};
use turbo_rcstr::RcStr;
use turbo_tasks::{
ReadConsistency, ResolvedVc, TransientInstance, TryJoinIterExt, TurboTasks, Value, Vc,
apply_effects, ReadConsistency, ResolvedVc, TransientInstance, TryJoinIterExt, TurboTasks,
Value, Vc,
};
use turbo_tasks_fs::FileSystem;
use turbo_tasks_memory::MemoryBackend;
......@@ -128,7 +129,9 @@ impl TurbopackBuildBuilder {
);
// Await the result to propagate any errors.
build_result.await?;
build_result.strongly_consistent().await?;
apply_effects(build_result).await?;
let issue_reporter: Vc<Box<dyn IssueReporter>> =
Vc::upcast(ConsoleUi::new(TransientInstance::new(LogOptions {
......
use anyhow::Result;
use turbo_rcstr::RcStr;
use turbo_tasks::{Completion, ResolvedVc, Vc};
use turbo_tasks::{ResolvedVc, Vc};
use turbo_tasks_fs::{
FileContent, FileJsonContent, FileLinesContent, FileSystemPath, LinkContent, LinkType,
};
......@@ -86,17 +86,22 @@ impl AssetContent {
}
#[turbo_tasks::function]
pub async fn write(self: Vc<Self>, path: Vc<FileSystemPath>) -> Result<Vc<Completion>> {
pub async fn write(self: Vc<Self>, path: Vc<FileSystemPath>) -> Result<()> {
let this = self.await?;
Ok(match &*this {
AssetContent::File(file) => path.write(**file),
AssetContent::Redirect { target, link_type } => path.write_link(
LinkContent::Link {
target: target.clone(),
link_type: *link_type,
}
.cell(),
),
})
match &*this {
AssetContent::File(file) => {
let _ = path.write(**file);
}
AssetContent::Redirect { target, link_type } => {
let _ = path.write_link(
LinkContent::Link {
target: target.clone(),
link_type: *link_type,
}
.cell(),
);
}
}
Ok(())
}
}
......@@ -251,7 +251,7 @@ pub async fn fileify_source_map(
.await?
.context("Expected the chunking context to have a DiskFileSystem")?
.await?;
let prefix = format!("{}[{}]/", SOURCE_MAP_PREFIX, context_fs.name);
let prefix = format!("{}[{}]/", SOURCE_MAP_PREFIX, context_fs.name());
let mut transformed = flattened.into_owned();
let mut updates = IndexMap::new();
......
......@@ -39,20 +39,12 @@ impl FileSystem for ServerFileSystem {
}
#[turbo_tasks::function]
fn write(
&self,
_fs_path: Vc<FileSystemPath>,
_content: Vc<FileContent>,
) -> Result<Vc<Completion>> {
fn write(&self, _fs_path: Vc<FileSystemPath>, _content: Vc<FileContent>) -> Result<Vc<()>> {
bail!("Writing is not possible to the marker filesystem for the server")
}
#[turbo_tasks::function]
fn write_link(
&self,
_fs_path: Vc<FileSystemPath>,
_target: Vc<LinkContent>,
) -> Result<Vc<Completion>> {
fn write_link(&self, _fs_path: Vc<FileSystemPath>, _target: Vc<LinkContent>) -> Result<Vc<()>> {
bail!("Writing is not possible to the marker filesystem for the server")
}
......
......@@ -10,7 +10,9 @@ use hyper::{
};
use mime::Mime;
use tokio_util::io::{ReaderStream, StreamReader};
use turbo_tasks::{util::SharedError, CollectiblesSource, ReadRef, TransientInstance, Vc};
use turbo_tasks::{
apply_effects, util::SharedError, CollectiblesSource, ReadRef, TransientInstance, Vc,
};
use turbo_tasks_bytes::Bytes;
use turbo_tasks_fs::FileContent;
use turbopack_core::{
......@@ -80,6 +82,7 @@ pub async fn process_request_with_content_source(
let request = http_request_to_source_request(request).await?;
let result = get_from_source(source, TransientInstance::new(request));
let resolved_result = result.resolve_strongly_consistent().await?;
apply_effects(result).await?;
let side_effects: AutoSet<Vc<Box<dyn ContentSourceSideEffect>>> = result.peek_collectibles();
handle_issues(
result,
......
......@@ -33,7 +33,8 @@ use socket2::{Domain, Protocol, Socket, Type};
use tokio::task::JoinHandle;
use tracing::{event, info_span, Instrument, Level, Span};
use turbo_tasks::{
run_once_with_reason, trace::TraceRawVcs, util::FormatDuration, TurboTasksApi, Vc,
apply_effects, run_once_with_reason, trace::TraceRawVcs, util::FormatDuration, TurboTasksApi,
Vc,
};
use turbopack_core::{
error::PrettyPrintError,
......@@ -211,6 +212,7 @@ impl DevServerBuilder {
let path = uri.path().to_string();
let source = source_provider.get_source();
let resolved_source = source.resolve_strongly_consistent().await?;
apply_effects(source).await?;
handle_issues(
source,
issue_reporter,
......
use std::{borrow::Cow, ops::ControlFlow, thread::available_parallelism, time::Duration};
use std::{
borrow::Cow, ops::ControlFlow, sync::Arc, thread::available_parallelism, time::Duration,
};
use anyhow::{anyhow, bail, Result};
use async_stream::try_stream as generator;
......@@ -12,8 +14,8 @@ use parking_lot::Mutex;
use serde::{de::DeserializeOwned, Deserialize, Serialize};
use serde_json::Value as JsonValue;
use turbo_tasks::{
duration_span, fxindexmap, mark_finished, prevent_gc, util::SharedError, Completion, RawVc,
ResolvedVc, TaskInput, TryJoinIterExt, Value, Vc,
duration_span, fxindexmap, get_effects, mark_finished, prevent_gc, util::SharedError,
Completion, Effects, RawVc, ResolvedVc, TaskInput, TryJoinIterExt, Value, Vc,
};
use turbo_tasks_bytes::{Bytes, Stream};
use turbo_tasks_env::ProcessEnv;
......@@ -28,7 +30,7 @@ use turbopack_core::{
ident::AssetIdent,
issue::{Issue, IssueExt, IssueStage, OptionStyledString, StyledString},
module::Module,
output::OutputAssets,
output::{OutputAsset, OutputAssets},
reference_type::{InnerAssets, ReferenceType},
virtual_source::VirtualSource,
};
......@@ -77,19 +79,20 @@ pub struct JavaScriptStreamSender {
#[derive(Clone, Debug)]
pub struct JavaScriptEvaluation(#[turbo_tasks(trace_ignore)] JavaScriptStream);
#[turbo_tasks::value]
struct EmittedEvaluatePoolAssets {
bootstrap: ResolvedVc<Box<dyn OutputAsset>>,
output_root: ResolvedVc<FileSystemPath>,
entrypoint: ResolvedVc<FileSystemPath>,
}
#[turbo_tasks::function]
/// Pass the file you cared as `runtime_entries` to invalidate and reload the
/// evaluated result automatically.
pub async fn get_evaluate_pool(
async fn emit_evaluate_pool_assets(
module_asset: ResolvedVc<Box<dyn Module>>,
cwd: Vc<FileSystemPath>,
env: Vc<Box<dyn ProcessEnv>>,
asset_context: Vc<Box<dyn AssetContext>>,
chunking_context: Vc<Box<dyn ChunkingContext>>,
runtime_entries: Option<Vc<EvaluatableAssets>>,
additional_invalidation: Vc<Completion>,
debug: bool,
) -> Result<Vc<NodeJsPool>> {
) -> Result<Vc<EmittedEvaluatePoolAssets>> {
let runtime_asset = asset_context
.process(
Vc::upcast(FileSource::new(embed_file_path("ipc/evaluate.ts".into()))),
......@@ -108,7 +111,7 @@ pub async fn get_evaluate_pool(
} else {
Cow::Owned(format!("{file_name}.js"))
};
let path = chunking_context.output_root().join(file_name.into());
let entrypoint = chunking_context.output_root().join(file_name.into());
let entry_module = asset_context
.process(
Vc::upcast(VirtualSource::new(
......@@ -124,10 +127,6 @@ pub async fn get_evaluate_pool(
)
.module();
let (Some(cwd), Some(entrypoint)) = (to_sys_path(cwd).await?, to_sys_path(path).await?) else {
panic!("can only evaluate from a disk filesystem");
};
let runtime_entries = {
let globals_module = asset_context
.process(
......@@ -153,7 +152,7 @@ pub async fn get_evaluate_pool(
};
let bootstrap = chunking_context.root_entry_chunk_group_asset(
path,
entrypoint,
entry_module,
OutputAssets::empty(),
runtime_entries,
......@@ -161,10 +160,84 @@ pub async fn get_evaluate_pool(
let output_root = chunking_context.output_root().to_resolved().await?;
let _ = emit_package_json(*output_root);
// Invalidate pool when code content changes
content_changed(Vc::upcast(bootstrap)).await?;
let _ = emit(bootstrap, *output_root);
let assets_for_source_mapping = internal_assets_for_source_mapping(bootstrap, *output_root)
Ok(EmittedEvaluatePoolAssets {
bootstrap: bootstrap.to_resolved().await?,
output_root,
entrypoint: entrypoint.to_resolved().await?,
}
.cell())
}
#[turbo_tasks::value(serialization = "none")]
struct EmittedEvaluatePoolAssetsWithEffects {
bootstrap: ResolvedVc<Box<dyn OutputAsset>>,
output_root: ResolvedVc<FileSystemPath>,
entrypoint: ResolvedVc<FileSystemPath>,
effects: Arc<Effects>,
}
#[turbo_tasks::function]
async fn emit_evaluate_pool_assets_with_effects(
module_asset: Vc<Box<dyn Module>>,
asset_context: Vc<Box<dyn AssetContext>>,
chunking_context: Vc<Box<dyn ChunkingContext>>,
runtime_entries: Option<Vc<EvaluatableAssets>>,
) -> Result<Vc<EmittedEvaluatePoolAssetsWithEffects>> {
let operation = emit_evaluate_pool_assets(
module_asset,
asset_context,
chunking_context,
runtime_entries,
);
let result = operation.strongly_consistent().await?;
let effects = Arc::new(get_effects(operation).await?);
Ok(EmittedEvaluatePoolAssetsWithEffects {
bootstrap: result.bootstrap,
output_root: result.output_root,
entrypoint: result.entrypoint,
effects,
}
.cell())
}
#[turbo_tasks::function]
/// Pass the file you cared as `runtime_entries` to invalidate and reload the
/// evaluated result automatically.
pub async fn get_evaluate_pool(
module_asset: Vc<Box<dyn Module>>,
cwd: Vc<FileSystemPath>,
env: Vc<Box<dyn ProcessEnv>>,
asset_context: Vc<Box<dyn AssetContext>>,
chunking_context: Vc<Box<dyn ChunkingContext>>,
runtime_entries: Option<Vc<EvaluatableAssets>>,
additional_invalidation: Vc<Completion>,
debug: bool,
) -> Result<Vc<NodeJsPool>> {
let EmittedEvaluatePoolAssetsWithEffects {
bootstrap,
output_root,
entrypoint,
ref effects,
} = *emit_evaluate_pool_assets_with_effects(
module_asset,
asset_context,
chunking_context,
runtime_entries,
)
.strongly_consistent()
.await?;
effects.apply().await?;
let (Some(cwd), Some(entrypoint)) = (to_sys_path(cwd).await?, to_sys_path(*entrypoint).await?)
else {
panic!("can only evaluate from a disk filesystem");
};
// Invalidate pool when code content changes
content_changed(Vc::upcast(*bootstrap)).await?;
let assets_for_source_mapping = internal_assets_for_source_mapping(*bootstrap, *output_root)
.to_resolved()
.await?;
let pool = NodeJsPool::new(
......@@ -322,12 +395,12 @@ pub async fn compute(
};
let stream = generator! {
let pool = evaluate_context.pool();
let pool_operation = evaluate_context.pool();
let mut state = Default::default();
// Read this strongly consistent, since we don't want to run inconsistent
// node.js code.
let pool = pool.strongly_consistent().await?;
let pool = pool_operation.strongly_consistent().await?;
let args = evaluate_context.args().iter().try_join().await?;
// Assume this is a one-off operation, so we can kill the process
......
......@@ -11,12 +11,13 @@ pub use node_entry::{NodeEntry, NodeRenderingEntries, NodeRenderingEntry};
use turbo_rcstr::RcStr;
use turbo_tasks::{
graph::{AdjacencyMap, GraphTraversal},
Completion, Completions, FxIndexSet, ResolvedVc, TryJoinIterExt, ValueToString, Vc,
FxIndexSet, ResolvedVc, TryJoinIterExt, ValueToString, Vc,
};
use turbo_tasks_env::ProcessEnv;
use turbo_tasks_fs::{to_sys_path, File, FileSystemPath};
use turbopack_core::{
asset::{Asset, AssetContent},
changed::content_changed,
chunk::{ChunkingContext, ChunkingContextExt, EvaluatableAssets},
module::Module,
output::{OutputAsset, OutputAssets, OutputAssetsSet},
......@@ -41,16 +42,11 @@ pub mod transforms;
async fn emit(
intermediate_asset: Vc<Box<dyn OutputAsset>>,
intermediate_output_path: Vc<FileSystemPath>,
) -> Result<Vc<Completion>> {
Ok(Vc::<Completions>::cell(
internal_assets(intermediate_asset, intermediate_output_path)
.strongly_consistent()
.await?
.iter()
.map(|a| a.content().write(a.ident().path()))
.collect(),
)
.completed())
) -> Result<()> {
for asset in internal_assets(intermediate_asset, intermediate_output_path).await? {
let _ = asset.content().write(asset.ident().path());
}
Ok(())
}
/// List of the all assets of the "internal" subgraph and a list of boundary
......@@ -196,7 +192,7 @@ async fn separate_assets(
/// Emit a basic package.json that sets the type of the package to commonjs.
/// Currently code generated for Node is CommonJS, while authored code may be
/// ESM, for example.
fn emit_package_json(dir: Vc<FileSystemPath>) -> Vc<Completion> {
fn emit_package_json(dir: Vc<FileSystemPath>) -> Vc<()> {
emit(
Vc::upcast(VirtualOutputAsset::new(
dir.join("package.json".into()),
......@@ -219,7 +215,7 @@ pub async fn get_renderer_pool(
) -> Result<Vc<NodeJsPool>> {
emit_package_json(intermediate_output_path).await?;
let emit = emit(intermediate_asset, *output_root);
let _ = emit(intermediate_asset, *output_root);
let assets_for_source_mapping =
internal_assets_for_source_mapping(intermediate_asset, *output_root);
......@@ -237,8 +233,9 @@ pub async fn get_renderer_pool(
entrypoint.to_string().await?
);
};
// Invalidate pool when code content changes
content_changed(Vc::upcast(intermediate_asset)).await?;
emit.await?;
Ok(NodeJsPool::new(
cwd,
entrypoint,
......
......@@ -3,7 +3,7 @@ use std::{fs, path::PathBuf};
use criterion::{Bencher, BenchmarkId, Criterion};
use regex::Regex;
use turbo_rcstr::RcStr;
use turbo_tasks::{ReadConsistency, TurboTasks, Value, Vc};
use turbo_tasks::{apply_effects, ReadConsistency, TurboTasks, Value, Vc};
use turbo_tasks_fs::{DiskFileSystem, FileSystem, NullFileSystem};
use turbo_tasks_memory::MemoryBackend;
use turbopack::{
......@@ -112,7 +112,9 @@ fn bench_emit(b: &mut Bencher, bench_input: &BenchInput) {
.module();
let rebased = RebasedAsset::new(Vc::upcast(module), input_dir, output_dir);
emit_with_completion(Vc::upcast(rebased), output_dir).await?;
let emit = emit_with_completion(Vc::upcast(rebased), output_dir);
emit.strongly_consistent().await?;
apply_effects(emit).await?;
Ok::<Vc<()>, _>(Default::default())
});
......
......@@ -31,7 +31,7 @@ use graph::{aggregate, AggregatedGraph, AggregatedGraphNodeContent};
use module_options::{ModuleOptions, ModuleOptionsContext, ModuleRuleEffect, ModuleType};
use tracing::Instrument;
use turbo_rcstr::RcStr;
use turbo_tasks::{Completion, ResolvedVc, Value, ValueToString, Vc};
use turbo_tasks::{ResolvedVc, Value, ValueToString, Vc};
use turbo_tasks_fs::{glob::Glob, FileSystemPath};
pub use turbopack_core::condition;
use turbopack_core::{
......@@ -781,54 +781,49 @@ impl AssetContext for ModuleAssetContext {
}
#[turbo_tasks::function]
pub fn emit_with_completion(
asset: Vc<Box<dyn OutputAsset>>,
output_dir: Vc<FileSystemPath>,
) -> Vc<Completion> {
emit_assets_aggregated(asset, output_dir)
pub fn emit_with_completion(asset: Vc<Box<dyn OutputAsset>>, output_dir: Vc<FileSystemPath>) {
let _ = emit_assets_aggregated(asset, output_dir);
}
#[turbo_tasks::function]
fn emit_assets_aggregated(
asset: Vc<Box<dyn OutputAsset>>,
output_dir: Vc<FileSystemPath>,
) -> Vc<Completion> {
fn emit_assets_aggregated(asset: Vc<Box<dyn OutputAsset>>, output_dir: Vc<FileSystemPath>) {
let aggregated = aggregate(asset);
emit_aggregated_assets(aggregated, output_dir)
let _ = emit_aggregated_assets(aggregated, output_dir);
}
#[turbo_tasks::function]
async fn emit_aggregated_assets(
aggregated: Vc<AggregatedGraph>,
output_dir: Vc<FileSystemPath>,
) -> Result<Vc<Completion>> {
Ok(match &*aggregated.content().await? {
AggregatedGraphNodeContent::Asset(asset) => emit_asset_into_dir(**asset, output_dir),
) -> Result<()> {
match &*aggregated.content().await? {
AggregatedGraphNodeContent::Asset(asset) => {
let _ = emit_asset_into_dir(**asset, output_dir);
}
AggregatedGraphNodeContent::Children(children) => {
for aggregated in children {
emit_aggregated_assets(**aggregated, output_dir).await?;
let _ = emit_aggregated_assets(**aggregated, output_dir);
}
Completion::new()
}
})
}
Ok(())
}
#[turbo_tasks::function]
pub fn emit_asset(asset: Vc<Box<dyn OutputAsset>>) -> Vc<Completion> {
asset.content().write(asset.ident().path())
pub fn emit_asset(asset: Vc<Box<dyn OutputAsset>>) {
let _ = asset.content().write(asset.ident().path());
}
#[turbo_tasks::function]
pub async fn emit_asset_into_dir(
asset: Vc<Box<dyn OutputAsset>>,
output_dir: Vc<FileSystemPath>,
) -> Result<Vc<Completion>> {
) -> Result<()> {
let dir = &*output_dir.await?;
Ok(if asset.ident().path().await?.is_inside_ref(dir) {
emit_asset(asset)
} else {
Completion::new()
})
if asset.ident().path().await?.is_inside_ref(dir) {
let _ = emit_asset(asset);
}
Ok(())
}
type OutputAssetSet = HashSet<Vc<Box<dyn OutputAsset>>>;
......
......@@ -30,7 +30,9 @@ use rstest_reuse::{
use serde::{Deserialize, Serialize};
use tokio::{process::Command, time::timeout};
use turbo_rcstr::RcStr;
use turbo_tasks::{backend::Backend, ReadRef, ResolvedVc, TurboTasks, Value, ValueToString, Vc};
use turbo_tasks::{
apply_effects, backend::Backend, ReadRef, ResolvedVc, TurboTasks, Value, ValueToString, Vc,
};
use turbo_tasks_fs::{DiskFileSystem, FileSystem, FileSystemPath};
use turbo_tasks_memory::MemoryBackend;
use turbopack::{
......@@ -462,7 +464,9 @@ fn node_file_trace<B: Backend + 'static>(
print_graph(ResolvedVc::upcast(rebased)).await?;
emit_with_completion(*ResolvedVc::upcast(rebased), output_dir).await?;
let emit = emit_with_completion(*ResolvedVc::upcast(rebased), output_dir);
emit.strongly_consistent().await?;
apply_effects(emit).await?;
#[cfg(not(feature = "bench_against_node_nft"))]
{
......@@ -657,10 +661,12 @@ fn clean_stderr(str: &str) -> String {
lazy_static! {
static ref EXPERIMENTAL_WARNING: Regex =
Regex::new(r"\(node:\d+\) ExperimentalWarning:").unwrap();
static ref DEPRECATION_WARNING: Regex =
Regex::new(r"\(node:\d+\) \[DEP\d+] DeprecationWarning:").unwrap();
}
EXPERIMENTAL_WARNING
.replace_all(str, "(node:XXXX) ExperimentalWarning:")
.to_string()
let str = EXPERIMENTAL_WARNING.replace_all(str, "(node:XXXX) ExperimentalWarning:");
let str = DEPRECATION_WARNING.replace_all(&str, "(node:XXXX) [DEPXXXX] DeprecationWarning:");
str.to_string()
}
fn diff(expected: &str, actual: &str) -> String {
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment