Unverified Commit 3d018005 authored by dan's avatar dan Committed by GitHub

Turbopack: stop copying sourcesContent into every serialized source map (#95934)

This reduces peak memory consumption with the original repro case in
[vercel/next.js#81161](https://github.com/vercel/next.js/issues/81161).

If you walk the 15 routes in a quick succession and capture vmmap
--summary at the end of each fast sweep (before memory is released),
with this change the number reliably goes down from 3 GB to 2.6 GB:

```
before	1	3.0G	17
after	1	2.6G	16
before	2	3.0G	16
after	2	2.6G	16
before	3	3.1G	18
after	3	2.6G	17
before	4	3.1G	17
after	4	2.6G	16
before	5	3.1G	17
after	5	2.6G	17
before	6	3.0G	17
after	6	2.5G	17
before	7	3.1G	17
after	7	2.6G	17
before	8	3.0G	25
after	8	2.6G	24
before: [3.0, 3.0, 3.1, 3.1, 3.1, 3.0, 3.1, 3.0] 
after:  [2.6, 2.6, 2.6, 2.6, 2.6, 2.5, 2.6, 2.6]
mean before=3.050G after=2.587G  diff=0.462G (-15.2%)
exact permutation test: p = 2/12870 = 0.00016
```

More runs on this app:

<img width="938" height="623" alt="Screenshot 2026-07-22 at 20 59 12"
src="https://github.com/user-attachments/assets/01c41656-eb5a-41f9-8871-b4ce262c54d5"
/>

And here's an internal app showing ~20% dev memory win at peak (although
reclaiming memory settles it):

<img width="832" height="402" alt="Screenshot 2026-07-22 at 16 45 34"
src="https://github.com/user-attachments/assets/8c39f668-d431-43f9-b533-d39dfaa97c15"
/>

---

<details>

Module source maps are serialized to a JSON rope right after code
generation and treated as opaque bytes from then on. Since
`sourcesContent` — the full original source text — is inlined into that
JSON, every later step that modifies the map has to copy all of it:
rewriting the `sources` URLs for each chunking context copies the entire
map per context, and embedding the module map into a chunk's sectioned
map copies it again, once per layer that includes the module. In a dev
session on a dependency-heavy app, each module's source text ends up
resident ~5 times over, and source maps end up dominating the memory the
task store holds for codegen output (in one measured app, maps were 57%
of module factory bytes, and `sourcesContent` alone was 36%).

This PR stops copying the text. `StructuredSourceMap` keeps a map in
field form until the moment it is actually emitted:

- Fields we never touch (`mappings`, `names`, `version`, ...) are stored
as verbatim raw JSON snippets, so producers' bytes round-trip untouched
and re-emission is rope sharing rather than re-serialization.
- `sources` is the one field rewrites modify, so it is stored typed, and
the per-chunking-context "fileify" rewrites become a plain string
rewrite of that field instead of a parse–modify–reserialize of the whole
map.
- `sourcesContent` entries are individual, already-JSON-escaped ropes.
Rewritten maps and chunk maps share them; the source text is escaped
once per module and never copied again.

Two construction paths, chosen by where the bytes come from: maps we
generate ourselves are built directly from the `swc_sourcemap` object
(`from_swc_map` takes the contents out, serializes the small remainder,
and never round-trips source text through JSON), while maps that arrive
as bytes (prebuilt maps next to external code, the
single-complete-original fast path) go through `from_json`. The emitter
writes fields in exactly `swc_sourcemap`'s order and format, pinned by
golden tests that assert byte-identity with `to_writer` — so emitted
chunks and maps are byte-for-byte unchanged, and no snapshot fixtures
change. The CSS pipeline gets the same treatment; the single-file output
path is left alone since it serializes straight into the final artifact
and there is nothing to share.

The win is proportional to how much duplicated source-map text an app
holds, so results range accordingly. On a dependency-heavy reproduction
app (13 routes sharing large wallet SDKs; rapid route sweep in `next
dev`, macOS physical footprint, 8 interleaved cold runs per side): 3.05
GB → 2.59 GB mean (−15%), with zero overlap between the two
distributions (exact permutation test p ≈ 0.00016) and per-route
latencies unchanged. On `bench/basic-app` (small first-party modules): a
full sweep of all 42 routes drops the post-sweep footprint 3.4–3.5 GB →
2.4–2.5 GB (−29%, interleaved runs); cold compile −4%, idle footprint
identical (eviction reclaims both sides equally at idle). On
`bench/heavy-npm-deps` (heavy deps disabled in its default config) and
on very large first-party apps whose memory is dominated by the JS heap
and application state rather than map text, the change measures flat —
as expected, since there is little duplicated text for sharing to
recover. `small_apps` build benchmarks are flat except `date-fns-all` at
+1–2% (~2.5 ms): that app is 1000+ tiny modules and `from_swc_map` adds
one small-skeleton JSON parse per module, with no redundancy to win back
in a single-context one-shot build. The single-complete-original fast
path (per prebuilt module with a shipped map, and per minified chunk) is
cost-neutral: the structured map is built by splitting the raw map's
serde fields directly (`from_serialize`), so the serialized document is
never materialized or re-parsed — 93 ms per 5 iterations on a synthetic
21.7 MB map, matching the pre-PR baseline exactly, with byte-identical
output pinned by a property test. Serialized task-store size is
unchanged (bincode expands sharing back out), so this is purely a
live-memory change.

Verified: snapshot suite passes with zero fixture changes, execution
suite matches canary (the single pre-existing failure reproduces on the
base tag), unit suites pass including new golden byte-identity tests,
and emitted `.map` files were validated end-to-end in a running dev
server (multi-section chunk maps, fileified `file://` sources,
`sourcesContent` present and byte-identical between base and this
branch).

</details>
parent 99cf207a
...@@ -22,7 +22,7 @@ use turbopack_core::{ ...@@ -22,7 +22,7 @@ use turbopack_core::{
module_graph::ModuleGraph, module_graph::ModuleGraph,
reference_type::ReferenceType, reference_type::ReferenceType,
source::{OptionSource, Source}, source::{OptionSource, Source},
source_map::GenerateSourceMap, source_map::{GenerateSourceMap, structured::StructuredSourceMap},
}; };
use turbopack_ecmascript::{ use turbopack_ecmascript::{
EcmascriptInputTransforms, EcmascriptInputTransforms,
...@@ -258,6 +258,9 @@ impl EcmascriptChunkPlaceable for RawEcmascriptModule { ...@@ -258,6 +258,9 @@ impl EcmascriptChunkPlaceable for RawEcmascriptModule {
None None
}; };
let source_map = source_map
.map(|map| StructuredSourceMap::from_json(&map))
.transpose()?;
Ok(EcmascriptChunkItemContent { Ok(EcmascriptChunkItemContent {
source_map, source_map,
inner_code: code.into_source_code(), inner_code: code.into_source_code(),
......
...@@ -9,22 +9,66 @@ use anyhow::Result; ...@@ -9,22 +9,66 @@ use anyhow::Result;
use bincode::{Decode, Encode}; use bincode::{Decode, Encode};
use tracing::instrument; use tracing::instrument;
use turbo_rcstr::RcStr; use turbo_rcstr::RcStr;
use turbo_tasks::{ResolvedVc, Vc}; use turbo_tasks::{NonLocalValue, ResolvedVc, Vc, trace::TraceRawVcs};
use turbo_tasks_fs::{ use turbo_tasks_fs::{
File, FileContent, File, FileContent,
rope::{Rope, RopeBuilder}, rope::{Rope, RopeBuilder},
}; };
use turbo_tasks_hash::hash_xxh3_hash128; use turbo_tasks_hash::{DeterministicHash, DeterministicHasher, hash_xxh3_hash128};
use crate::{ use crate::{
debug_id::generate_debug_id, debug_id::generate_debug_id,
output::OutputAsset, output::OutputAsset,
source_map::{GenerateSourceMap, SourceMap, SourceMapAsset}, source_map::{GenerateSourceMap, SourceMap, SourceMapAsset, structured::StructuredSourceMap},
source_pos::SourcePos, source_pos::SourcePos,
}; };
/// A per-section source map: either an opaque serialized map or a structured one whose
/// `sourcesContent` is shared rather than copied when the section is embedded.
#[derive(Clone, Debug, PartialEq, Eq, Encode, Decode, TraceRawVcs, NonLocalValue)]
pub enum SectionMap {
Raw(Rope),
Structured(Box<StructuredSourceMap>),
}
impl From<Rope> for SectionMap {
fn from(map: Rope) -> Self {
SectionMap::Raw(map)
}
}
impl From<StructuredSourceMap> for SectionMap {
fn from(map: StructuredSourceMap) -> Self {
SectionMap::Structured(Box::new(map))
}
}
impl DeterministicHash for SectionMap {
fn deterministic_hash<H: DeterministicHasher>(&self, state: &mut H) {
match self {
SectionMap::Raw(map) => {
state.write_u8(0);
map.deterministic_hash(state);
}
SectionMap::Structured(map) => {
state.write_u8(1);
map.deterministic_hash(state);
}
}
}
}
impl SectionMap {
pub fn to_rope(&self) -> Rope {
match self {
SectionMap::Raw(map) => map.clone(),
SectionMap::Structured(map) => map.to_rope(),
}
}
}
/// A mapping of byte-offset in the code string to an associated source map. /// A mapping of byte-offset in the code string to an associated source map.
pub type Mapping = (usize, Option<Rope>); pub type Mapping = (usize, Option<SectionMap>);
/// Code stores combined output code and the source map of that output code. /// Code stores combined output code and the source map of that output code.
#[turbo_tasks::value(shared, serialization = "hash")] #[turbo_tasks::value(shared, serialization = "hash")]
...@@ -166,8 +210,8 @@ impl CodeBuilder { ...@@ -166,8 +210,8 @@ impl CodeBuilder {
/// Pushes original user code with an optional source map if one is /// Pushes original user code with an optional source map if one is
/// available. If it's not, this is no different than pushing Synthetic /// available. If it's not, this is no different than pushing Synthetic
/// code. /// code.
pub fn push_source(&mut self, code: &Rope, map: Option<Rope>) { pub fn push_source<M: Into<SectionMap>>(&mut self, code: &Rope, map: Option<M>) {
self.push_map(map); self.push_map(map.map(Into::into));
self.code += code; self.code += code;
} }
...@@ -205,7 +249,7 @@ impl CodeBuilder { ...@@ -205,7 +249,7 @@ impl CodeBuilder {
/// original code section. By inserting an empty source map when reaching a /// original code section. By inserting an empty source map when reaching a
/// synthetic section directly after an original section, we tell Chrome /// synthetic section directly after an original section, we tell Chrome
/// that the previous map ended at this point. /// that the previous map ended at this point.
fn push_map(&mut self, map: Option<Rope>) { fn push_map(&mut self, map: Option<SectionMap>) {
let Some(mappings) = self.mappings.as_mut() else { let Some(mappings) = self.mappings.as_mut() else {
return; return;
}; };
...@@ -339,7 +383,7 @@ impl Code { ...@@ -339,7 +383,7 @@ impl Code {
last_byte_pos = *byte_pos; last_byte_pos = *byte_pos;
if let Some(map) = map { if let Some(map) = map {
sections.push((pos, map.clone())) sections.push((pos, map.to_rope()))
} else { } else {
// We don't need an empty source map when column is 0 or the next char is a newline. // We don't need an empty source map when column is 0 or the next char is a newline.
if pos.column != 0 if pos.column != 0
......
...@@ -31,6 +31,7 @@ use crate::{ ...@@ -31,6 +31,7 @@ use crate::{
}; };
pub(crate) mod source_map_asset; pub(crate) mod source_map_asset;
pub mod structured;
pub mod utils; pub mod utils;
pub use source_map_asset::SourceMapAsset; pub use source_map_asset::SourceMapAsset;
......
This diff is collapsed.
...@@ -10,7 +10,7 @@ use turbo_tasks::{ResolvedVc, turbofmt}; ...@@ -10,7 +10,7 @@ use turbo_tasks::{ResolvedVc, turbofmt};
use turbo_tasks_fs::{DiskFileSystem, FileContent, FileSystemPath, rope::Rope}; use turbo_tasks_fs::{DiskFileSystem, FileContent, FileSystemPath, rope::Rope};
use url::Url; use url::Url;
use crate::SOURCE_URL_PROTOCOL_STR; use crate::{SOURCE_URL_PROTOCOL_STR, source_map::structured::StructuredSourceMap};
pub fn add_default_ignore_list(map: &mut swc_sourcemap::SourceMap) { pub fn add_default_ignore_list(map: &mut swc_sourcemap::SourceMap) {
let mut ignored_ids = HashSet::new(); let mut ignored_ids = HashSet::new();
...@@ -223,27 +223,23 @@ fn unencoded_str_to_raw_value(unencoded: &str) -> Box<RawValue> { ...@@ -223,27 +223,23 @@ fn unencoded_str_to_raw_value(unencoded: &str) -> Box<RawValue> {
.expect("serde_json::to_string should produce valid JSON") .expect("serde_json::to_string should produce valid JSON")
} }
/// Helper function to transform turbopack:/// file references in a sourcemap. fn uri_encode_path(path: &str) -> String {
/// Handles parsing the sourcemap, resolving the filesystem, applying transformations, and path.split('/')
/// serializing back. .map(|s| urlencoding::encode(s))
/// The transform function is given the source string as found in the sourcemap (i.e. a URI). .collect::<Vec<_>>()
.join("/")
}
/// Applies the standard `turbopack:///[fs]/` source-URL transform to a structured map,
/// sharing every other field (notably `sourcesContent`) with the input.
async fn transform_relative_files<F>( async fn transform_relative_files<F>(
map: Option<&Rope>, map: &StructuredSourceMap,
context_path: &FileSystemPath, context_path: &FileSystemPath,
mut transform: F, mut transform: F,
) -> Result<Option<Rope>> ) -> Result<StructuredSourceMap>
where where
F: FnMut(&DiskFileSystem, &str) -> Result<String>, F: FnMut(&DiskFileSystem, &str) -> Result<String>,
{ {
let Some(map) = map else {
return Ok(None);
};
let Ok(mut map): serde_json::Result<SourceMapJson> = serde_json::from_reader(map.read()) else {
// Silently ignore invalid sourcemaps
return Ok(None);
};
let context_fs = context_path.fs; let context_fs = context_path.fs;
let context_fs = &*ResolvedVc::try_downcast_type::<DiskFileSystem>(context_fs) let context_fs = &*ResolvedVc::try_downcast_type::<DiskFileSystem>(context_fs)
.context("Expected the chunking context to have a DiskFileSystem")? .context("Expected the chunking context to have a DiskFileSystem")?
...@@ -251,32 +247,21 @@ where ...@@ -251,32 +247,21 @@ where
let prefix = format!("{}///[{}]/", SOURCE_URL_PROTOCOL_STR, context_fs.name()); let prefix = format!("{}///[{}]/", SOURCE_URL_PROTOCOL_STR, context_fs.name());
let mut apply_transform = |src: &mut String| -> Result<()> { map.rewrite_sources(|src| {
if let Some(src_rest) = src.strip_prefix(&prefix) { if let Some(src_rest) = src.strip_prefix(&prefix) {
*src = transform(context_fs, src_rest)?; Ok(Some(transform(context_fs, src_rest)?))
} } else {
Ok(()) Ok(None)
};
for src in map.sources.iter_mut().flatten().flatten() {
apply_transform(src)?;
}
for section in map.sections.iter_mut().flatten() {
for src in section.map.sources.iter_mut().flatten().flatten() {
apply_transform(src)?;
}
} }
})
Ok(Some(Rope::from(serde_json::to_vec(&map)?)))
} }
/// Turns `turbopack:///[project]` references in sourcemap sources into absolute `file://` uris. This /// Turns `turbopack:///[project]` references in the map's sources into absolute `file://` uris.
/// is useful for debugging environments.
pub async fn absolute_fileify_source_map( pub async fn absolute_fileify_source_map(
map: Option<&Rope>, map: &StructuredSourceMap,
context_path: FileSystemPath, context_path: FileSystemPath,
) -> Result<Option<Rope>> { ) -> Result<StructuredSourceMap> {
transform_relative_files(map, &context_path, |context_fs, src_rest| { transform_relative_files(map, &context_path.clone(), |context_fs, src_rest| {
let path = context_path.join(src_rest)?; let path = context_path.join(src_rest)?;
// `to_sys_path` returns a win32 path on Windows. `Url::from_file_path` can also handle // `to_sys_path` returns a win32 path on Windows. `Url::from_file_path` can also handle
...@@ -291,36 +276,23 @@ pub async fn absolute_fileify_source_map( ...@@ -291,36 +276,23 @@ pub async fn absolute_fileify_source_map(
.await .await
} }
fn uri_encode_path(path: &str) -> String { /// Turns `turbopack:///[project]` references in the map's sources into `./`-relative uris.
path.split('/')
.map(|s| urlencoding::encode(s))
.collect::<Vec<_>>()
.join("/")
}
/// Turns `turbopack:///[project]` references in sourcemap sources into relative './' prefixed uris.
/// This is useful in server environments and especially build environments.
pub async fn relative_fileify_source_map( pub async fn relative_fileify_source_map(
map: Option<&Rope>, map: &StructuredSourceMap,
context_path: FileSystemPath, context_path: FileSystemPath,
relative_path_to_output_root: RcStr, relative_path_to_output_root: RcStr,
) -> Result<Option<Rope>> { ) -> Result<StructuredSourceMap> {
let relative_path_to_output_root = relative_path_to_output_root let relative_path_to_output_root = relative_path_to_output_root
.split('/') .split('/')
.map(|s| urlencoding::encode(s)) .map(|s| urlencoding::encode(s))
.collect::<Vec<_>>() .collect::<Vec<_>>()
.join("/"); .join("/");
transform_relative_files(map, &context_path, |_context_fs, src_rest| { transform_relative_files(map, &context_path, |_context_fs, src_rest| {
// NOTE: we just include the relative path prefix here instead of using `sourceRoot`
// since the spec on sourceRoot is broken.
// TODO(bgw): this shouldn't be necessary to uri encode since the strings we get out of the
// source map should already be uri encoded, however in the case of the turbopack scheme in
// particular we are inconsistent so be defensive here.
let src_rest = uri_encode_path(src_rest); let src_rest = uri_encode_path(src_rest);
if relative_path_to_output_root.is_empty() { if relative_path_to_output_root.is_empty() {
Ok(src_rest.to_string()) Ok(src_rest.to_string())
} else { } else {
Ok(format!("{relative_path_to_output_root}/{src_rest}",)) Ok(format!("{relative_path_to_output_root}/{src_rest}"))
} }
}) })
.await .await
......
...@@ -347,7 +347,7 @@ impl CssChunkItem for CssModuleChunkItem { ...@@ -347,7 +347,7 @@ impl CssChunkItem for CssModuleChunkItem {
inner_code: output_code.to_owned().into(), inner_code: output_code.to_owned().into(),
imports, imports,
import_context: self.module.await?.import_context, import_context: self.module.await?.import_context,
source_map: *source_map, source_map: source_map.clone(),
} }
.cell()) .cell())
} else { } else {
...@@ -358,7 +358,7 @@ impl CssChunkItem for CssModuleChunkItem { ...@@ -358,7 +358,7 @@ impl CssChunkItem for CssModuleChunkItem {
.into(), .into(),
imports: vec![], imports: vec![],
import_context: None, import_context: None,
source_map: FileContent::NotFound.resolved_cell(), source_map: None,
} }
.cell()) .cell())
} }
......
...@@ -31,6 +31,7 @@ use turbopack_core::{ ...@@ -31,6 +31,7 @@ use turbopack_core::{
server_fs::ServerFileSystem, server_fs::ServerFileSystem,
source_map::{ source_map::{
GenerateSourceMap, GenerateSourceMap,
structured::StructuredSourceMap,
utils::{absolute_fileify_source_map, relative_fileify_source_map}, utils::{absolute_fileify_source_map, relative_fileify_source_map},
}, },
}; };
...@@ -97,28 +98,26 @@ impl CssChunk { ...@@ -97,28 +98,26 @@ impl CssChunk {
let close = write_import_context(&mut body, content.import_context).await?; let close = write_import_context(&mut body, content.import_context).await?;
let chunking_context = self.chunking_context(); let chunking_context = self.chunking_context();
let source_map = content.source_map.await?; let source_map = match (
let source_map = source_map.as_content().map(|f| f.content()); *chunking_context.source_map_source_type().await?,
let source_map = match *chunking_context.source_map_source_type().await? { &content.source_map,
SourceMapSourceType::AbsoluteFileUri => { ) {
absolute_fileify_source_map( (SourceMapSourceType::AbsoluteFileUri, Some(map)) => Some(
source_map, absolute_fileify_source_map(map, chunking_context.root_path().owned().await?)
chunking_context.root_path().owned().await?, .await?,
) ),
.await? (SourceMapSourceType::RelativeUri, Some(map)) => Some(
}
SourceMapSourceType::RelativeUri => {
relative_fileify_source_map( relative_fileify_source_map(
source_map, map,
chunking_context.root_path().owned().await?, chunking_context.root_path().owned().await?,
chunking_context chunking_context
.relative_path_from_chunk_root_to_project_root() .relative_path_from_chunk_root_to_project_root()
.owned() .owned()
.await?, .await?,
) )
.await? .await?,
} ),
SourceMapSourceType::TurbopackUri => source_map.cloned(), (_, map) => map.clone(),
}; };
body.push_source(&content.inner_code, source_map); body.push_source(&content.inner_code, source_map);
...@@ -452,7 +451,7 @@ pub struct CssChunkItemContent { ...@@ -452,7 +451,7 @@ pub struct CssChunkItemContent {
pub import_context: Option<ResolvedVc<ImportContext>>, pub import_context: Option<ResolvedVc<ImportContext>>,
pub imports: Vec<CssImport>, pub imports: Vec<CssImport>,
pub inner_code: Rope, pub inner_code: Rope,
pub source_map: ResolvedVc<FileContent>, pub source_map: Option<StructuredSourceMap>,
} }
#[turbo_tasks::value_trait] #[turbo_tasks::value_trait]
......
...@@ -66,14 +66,7 @@ impl SingleItemCssChunk { ...@@ -66,14 +66,7 @@ impl SingleItemCssChunk {
let content = this.item.content().await?; let content = this.item.content().await?;
let close = write_import_context(&mut code, content.import_context).await?; let close = write_import_context(&mut code, content.import_context).await?;
code.push_source( code.push_source(&content.inner_code, content.source_map.clone());
&content.inner_code,
content
.source_map
.await?
.as_content()
.map(|f| f.content().clone()),
);
write!(code, "{close}")?; write!(code, "{close}")?;
let c = code.build().cell(); let c = code.build().cell();
......
...@@ -5,7 +5,7 @@ use lightningcss::css_modules::CssModuleReference; ...@@ -5,7 +5,7 @@ use lightningcss::css_modules::CssModuleReference;
use swc_core::common::{BytePos, FileName, LineCol, SourceMap}; use swc_core::common::{BytePos, FileName, LineCol, SourceMap};
use turbo_rcstr::{RcStr, rcstr}; use turbo_rcstr::{RcStr, rcstr};
use turbo_tasks::{FxIndexMap, ResolvedVc, Vc, turbofmt}; use turbo_tasks::{FxIndexMap, ResolvedVc, Vc, turbofmt};
use turbo_tasks_fs::{FileSystemPath, rope::Rope}; use turbo_tasks_fs::FileSystemPath;
use turbopack_core::{ use turbopack_core::{
chunk::{AsyncModuleInfo, ChunkableModule, ChunkingContext, ModuleChunkItemIdExt}, chunk::{AsyncModuleInfo, ChunkableModule, ChunkingContext, ModuleChunkItemIdExt},
context::{AssetContext, ProcessResult}, context::{AssetContext, ProcessResult},
...@@ -16,6 +16,7 @@ use turbopack_core::{ ...@@ -16,6 +16,7 @@ use turbopack_core::{
reference_type::{CssReferenceSubType, ReferenceType}, reference_type::{CssReferenceSubType, ReferenceType},
resolve::{origin::ResolveOrigin, parse::Request}, resolve::{origin::ResolveOrigin, parse::Request},
source::{OptionSource, Source}, source::{OptionSource, Source},
source_map::structured::StructuredSourceMap,
}; };
use turbopack_ecmascript::{ use turbopack_ecmascript::{
chunk::{ chunk::{
...@@ -369,7 +370,7 @@ impl ResolveOrigin for EcmascriptCssModule { ...@@ -369,7 +370,7 @@ impl ResolveOrigin for EcmascriptCssModule {
} }
} }
fn generate_minimal_source_map(filename: String, source: String) -> Result<Rope> { fn generate_minimal_source_map(filename: String, source: String) -> Result<StructuredSourceMap> {
let mut mappings = vec![]; let mut mappings = vec![];
// Start from 1 because 0 is reserved for dummy spans in SWC. // Start from 1 because 0 is reserved for dummy spans in SWC.
let mut pos = 1; let mut pos = 1;
......
...@@ -19,7 +19,7 @@ use swc_core::base::sourcemap::SourceMapBuilder; ...@@ -19,7 +19,7 @@ use swc_core::base::sourcemap::SourceMapBuilder;
use tracing::Instrument; use tracing::Instrument;
use turbo_rcstr::{RcStr, rcstr}; use turbo_rcstr::{RcStr, rcstr};
use turbo_tasks::{FxIndexMap, ResolvedVc, ValueToString, Vc}; use turbo_tasks::{FxIndexMap, ResolvedVc, ValueToString, Vc};
use turbo_tasks_fs::{File, FileContent, FileSystemPath, rope::Rope}; use turbo_tasks_fs::{FileContent, FileSystemPath};
use turbopack_core::{ use turbopack_core::{
SOURCE_URL_PROTOCOL, SOURCE_URL_PROTOCOL,
asset::{Asset, AssetContent}, asset::{Asset, AssetContent},
...@@ -33,7 +33,7 @@ use turbopack_core::{ ...@@ -33,7 +33,7 @@ use turbopack_core::{
reference_type::ImportContext, reference_type::ImportContext,
resolve::origin::ResolveOrigin, resolve::origin::ResolveOrigin,
source::Source, source::Source,
source_map::utils::add_default_ignore_list, source_map::{structured::StructuredSourceMap, utils::add_default_ignore_list},
source_pos::SourcePos, source_pos::SourcePos,
}; };
...@@ -46,7 +46,7 @@ use crate::{ ...@@ -46,7 +46,7 @@ use crate::{
}, },
}; };
pub type CssOutput = (ToCssResult, Option<Rope>); pub type CssOutput = (ToCssResult, Option<StructuredSourceMap>);
#[turbo_tasks::value(transparent)] #[turbo_tasks::value(transparent)]
struct LightningCssTargets( struct LightningCssTargets(
...@@ -191,12 +191,13 @@ pub enum CssWithPlaceholderResult { ...@@ -191,12 +191,13 @@ pub enum CssWithPlaceholderResult {
} }
#[turbo_tasks::value(shared, serialization = "skip")] #[turbo_tasks::value(shared, serialization = "skip")]
#[allow(clippy::large_enum_variant)] // This is a turbo-tasks value
pub enum FinalCssResult { pub enum FinalCssResult {
Ok { Ok {
#[turbo_tasks(trace_ignore)] #[turbo_tasks(trace_ignore)]
output_code: String, output_code: String,
source_map: ResolvedVc<FileContent>, source_map: Option<StructuredSourceMap>,
}, },
Unparsable, Unparsable,
NotFound, NotFound,
...@@ -329,11 +330,7 @@ pub async fn finalize_css( ...@@ -329,11 +330,7 @@ pub async fn finalize_css(
Ok(FinalCssResult::Ok { Ok(FinalCssResult::Ok {
output_code: result.code, output_code: result.code,
source_map: if let Some(srcmap) = srcmap { source_map: srcmap,
FileContent::Content(File::from(srcmap)).resolved_cell()
} else {
FileContent::NotFound.resolved_cell()
},
} }
.cell()) .cell())
} }
...@@ -733,7 +730,9 @@ impl lightningcss::visitor::Visitor<'_> for CssValidator { ...@@ -733,7 +730,9 @@ impl lightningcss::visitor::Visitor<'_> for CssValidator {
} }
} }
fn generate_css_source_map(source_map: &parcel_sourcemap::SourceMap) -> Result<Rope> { fn generate_css_source_map(
source_map: &parcel_sourcemap::SourceMap,
) -> Result<StructuredSourceMap> {
let mut builder = SourceMapBuilder::new(None); let mut builder = SourceMapBuilder::new(None);
for src in source_map.get_sources() { for src in source_map.get_sources() {
...@@ -758,9 +757,7 @@ fn generate_css_source_map(source_map: &parcel_sourcemap::SourceMap) -> Result<R ...@@ -758,9 +757,7 @@ fn generate_css_source_map(source_map: &parcel_sourcemap::SourceMap) -> Result<R
let mut map = builder.into_sourcemap(); let mut map = builder.into_sourcemap();
add_default_ignore_list(&mut map); add_default_ignore_list(&mut map);
let mut result = vec![]; StructuredSourceMap::from_swc_map(map)
map.to_writer(&mut result)?;
Ok(Rope::from(result))
} }
#[turbo_tasks::value] #[turbo_tasks::value]
......
...@@ -20,7 +20,10 @@ use turbopack_core::{ ...@@ -20,7 +20,10 @@ use turbopack_core::{
module::Module, module::Module,
module_graph::ModuleGraph, module_graph::ModuleGraph,
output::OutputAssetsReference, output::OutputAssetsReference,
source_map::utils::{absolute_fileify_source_map, relative_fileify_source_map}, source_map::{
structured::StructuredSourceMap,
utils::{absolute_fileify_source_map, relative_fileify_source_map},
},
}; };
use crate::{ use crate::{
...@@ -46,7 +49,7 @@ pub enum RewriteSourcePath { ...@@ -46,7 +49,7 @@ pub enum RewriteSourcePath {
#[derive(Default, Clone)] #[derive(Default, Clone)]
pub struct EcmascriptChunkItemContent { pub struct EcmascriptChunkItemContent {
pub inner_code: Rope, pub inner_code: Rope,
pub source_map: Option<Rope>, pub source_map: Option<StructuredSourceMap>,
pub additional_ids: SmallVec<[ModuleId; 1]>, pub additional_ids: SmallVec<[ModuleId; 1]>,
pub options: EcmascriptChunkItemOptions, pub options: EcmascriptChunkItemOptions,
pub rewrite_source_path: RewriteSourcePath, pub rewrite_source_path: RewriteSourcePath,
...@@ -165,19 +168,14 @@ impl EcmascriptChunkItemContent { ...@@ -165,19 +168,14 @@ impl EcmascriptChunkItemContent {
code += " try {\n"; code += " try {\n";
} }
let source_map = match &self.rewrite_source_path { let source_map = match (&self.rewrite_source_path, &self.source_map) {
RewriteSourcePath::AbsoluteFilePath(path) => { (RewriteSourcePath::AbsoluteFilePath(path), Some(map)) => {
absolute_fileify_source_map(self.source_map.as_ref(), path.clone()).await? Some(absolute_fileify_source_map(map, path.clone()).await?)
} }
RewriteSourcePath::RelativeFilePath(path, relative_path) => { (RewriteSourcePath::RelativeFilePath(path, relative_path), Some(map)) => {
relative_fileify_source_map( Some(relative_fileify_source_map(map, path.clone(), relative_path.clone()).await?)
self.source_map.as_ref(),
path.clone(),
relative_path.clone(),
)
.await?
} }
RewriteSourcePath::None => self.source_map.clone(), (_, map) => map.clone(),
}; };
code.push_source(&self.inner_code, source_map); code.push_source(&self.inner_code, source_map);
......
...@@ -98,7 +98,7 @@ use turbopack_core::{ ...@@ -98,7 +98,7 @@ use turbopack_core::{
reference_type::InnerAssets, reference_type::InnerAssets,
resolve::{FindContextFileResult, find_context_file, origin::ResolveOrigin, package_json}, resolve::{FindContextFileResult, find_context_file, origin::ResolveOrigin, package_json},
source::Source, source::Source,
source_map::GenerateSourceMap, source_map::{GenerateSourceMap, structured::StructuredSourceMap},
}; };
use crate::{ use crate::{
...@@ -933,7 +933,7 @@ impl ResolveOrigin for EcmascriptModuleAsset { ...@@ -933,7 +933,7 @@ impl ResolveOrigin for EcmascriptModuleAsset {
#[turbo_tasks::value(shared)] #[turbo_tasks::value(shared)]
pub struct EcmascriptModuleContent { pub struct EcmascriptModuleContent {
pub inner_code: Rope, pub inner_code: Rope,
pub source_map: Option<Rope>, pub source_map: Option<StructuredSourceMap>,
pub is_esm: bool, pub is_esm: bool,
pub strict: bool, pub strict: bool,
pub additional_ids: SmallVec<[ModuleId; 1]>, pub additional_ids: SmallVec<[ModuleId; 1]>,
......
...@@ -163,7 +163,7 @@ pub fn minify(code: Code, source_maps: bool, mangle: Option<MangleType>) -> Resu ...@@ -163,7 +163,7 @@ pub fn minify(code: Code, source_maps: bool, mangle: Option<MangleType>) -> Resu
)?), )?),
); );
} else { } else {
builder.push_source(&src.into(), None); builder.push_source(&src.into(), None::<turbo_tasks_fs::rope::Rope>);
} }
Ok(builder.build()) Ok(builder.build())
} }
......
...@@ -40,7 +40,7 @@ use turbopack_core::{ ...@@ -40,7 +40,7 @@ use turbopack_core::{
asset::{Asset, AssetContent}, asset::{Asset, AssetContent},
issue::{Issue, IssueExt, IssueSeverity, IssueSource, IssueStage, StyledString}, issue::{Issue, IssueExt, IssueSeverity, IssueSource, IssueStage, StyledString},
source::Source, source::Source,
source_map::utils::add_default_ignore_list, source_map::{structured::StructuredSourceMap, utils::add_default_ignore_list},
}; };
use turbopack_swc_utils::emitter::IssueEmitter; use turbopack_swc_utils::emitter::IssueEmitter;
...@@ -171,6 +171,11 @@ pub enum ParseResult { ...@@ -171,6 +171,11 @@ pub enum ParseResult {
NotFound, NotFound,
} }
/// Generates a [`StructuredSourceMap`] for the transformed code, whose `sourcesContent`
/// entries are individual shared ropes instead of being embedded in the serialized JSON. This
/// keeps later `sources` URL rewrites and map embedding from copying the source text of every
/// module. Serialize with [`StructuredSourceMap::to_rope`] where raw bytes are needed.
///
/// `original_source_maps_complete` indicates whether the `original_source_maps` cover the whole /// `original_source_maps_complete` indicates whether the `original_source_maps` cover the whole
/// map, i.e. whether every module that ended up in `mappings` had an original sourcemap. /// map, i.e. whether every module that ended up in `mappings` had an original sourcemap.
#[instrument(level = "info", name = "generate source map", skip_all)] #[instrument(level = "info", name = "generate source map", skip_all)]
...@@ -181,7 +186,7 @@ pub fn generate_js_source_map<'a>( ...@@ -181,7 +186,7 @@ pub fn generate_js_source_map<'a>(
original_source_maps_complete: bool, original_source_maps_complete: bool,
inline_sources_content: bool, inline_sources_content: bool,
names: FxHashMap<BytePos, Atom>, names: FxHashMap<BytePos, Atom>,
) -> Result<Rope> { ) -> Result<StructuredSourceMap> {
let original_source_maps = original_source_maps let original_source_maps = original_source_maps
.into_iter() .into_iter()
.map(|map| map.to_bytes()) .map(|map| map.to_bytes())
...@@ -211,14 +216,8 @@ pub fn generate_js_source_map<'a>( ...@@ -211,14 +216,8 @@ pub fn generate_js_source_map<'a>(
); );
if original_source_maps.is_empty() { if original_source_maps.is_empty() {
// We don't convert sourcemap::SourceMap into raw_sourcemap::SourceMap because we don't
// need to adjust mappings
add_default_ignore_list(&mut new_mappings); add_default_ignore_list(&mut new_mappings);
StructuredSourceMap::from_swc_map(new_mappings)
let mut result = vec![];
new_mappings.to_writer(&mut result)?;
Ok(Rope::from(result))
} else if fast_path_single_original_source_map { } else if fast_path_single_original_source_map {
let mut map = original_source_maps.into_iter().next().unwrap(); let mut map = original_source_maps.into_iter().next().unwrap();
// TODO: Make this more efficient // TODO: Make this more efficient
...@@ -227,16 +226,15 @@ pub fn generate_js_source_map<'a>( ...@@ -227,16 +226,15 @@ pub fn generate_js_source_map<'a>(
// TODO: Enable this when we have a way to handle the ignore list // TODO: Enable this when we have a way to handle the ignore list
// add_default_ignore_list(&mut map); // add_default_ignore_list(&mut map);
let map = map.into_raw_sourcemap(); let map = map.into_raw_sourcemap();
let result = serde_json::to_vec(&map)?; // The fallback covers raw maps with fields the structured form does not know.
Ok(Rope::from(result)) StructuredSourceMap::from_serialize(&map)
.or_else(|_| StructuredSourceMap::from_json_slice(&serde_json::to_vec(&map)?))
} else { } else {
let mut map = new_mappings.adjust_mappings_from_multiple(original_source_maps); let mut map = new_mappings.adjust_mappings_from_multiple(original_source_maps);
add_default_ignore_list(&mut map); add_default_ignore_list(&mut map);
let mut result = vec![]; StructuredSourceMap::from_swc_map(map)
map.to_writer(&mut result)?;
Ok(Rope::from(result))
} }
} }
......
...@@ -121,7 +121,7 @@ impl GenerateSourceMap for SingleFileEcmascriptOutput { ...@@ -121,7 +121,7 @@ impl GenerateSourceMap for SingleFileEcmascriptOutput {
true, true,
Default::default(), Default::default(),
)?; )?;
Ok(FileContent::Content(File::from(map)).cell()) Ok(FileContent::Content(File::from(map.to_rope())).cell())
} }
} }
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment