Unverified Commit e2e8ba9a authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

turbo-trace-server: reduce allocation overhead when loading large trace files (#92896)

### What?

Reduces allocation overhead in `turbo-trace-server` when reading large
trace files. Measured improvement: **114 MB/s → 156 MB/s** (~37%
throughput increase) on a 16 GB trace with 46 million spans.

### Why?

Profiling showed the reader thread spending significant time in heap
allocation and deallocation during the initial bulk-load phase. The hot
spots were:

1. `to_string()` / `into_owned()` in `process_internal_row_queue`
allocating a new `String` for every span name, category, and arg
key/value — even when the same string (e.g. `"turbo_tasks::function"`)
appeared in millions of spans.
2. `Vec<SelfTimeEntry>::push` growing one-at-a-time, triggering many
`realloc` calls.
3. `OnceLock<Box<SpanTimeData>>` — inherent to the lazy-init design;
mitigated by the allocator switch.
4. `Vec<TraceRow>::push` in the deserialization loop re-growing during
the initial read.
5. `process_internal_row` allocating a fresh `Vec` queue on every call
(one per row).
6. `Vec<Span>::push` and hash map rehashing in `Store::add_span`.
7. `into_static()` cloning forcing borrowed `Cow` strings to heap for
queued rows.

### How?

**1. mimalloc as global allocator (`main.rs`, `Cargo.toml`)**

Added `turbo-tasks-malloc` as a dependency with a default-on
`custom_allocator` feature (matching the pattern in `turbopack-cli` and
`turbopack-nft`). Set `#[global_allocator] static ALLOC: TurboMalloc =
TurboMalloc`. Also wired `TurboMalloc::thread_stop()` via rayon's
`exit_handler` so each worker thread returns its mimalloc thread-local
heap to the global pool on exit.

**2. `RcStrInterning` in `turbo-rcstr`**

New `RcStrInterning` struct backed by `FxHashSet<RcStr>`. Methods:
`intern(&str)`, `intern_cow(Cow<str>)`, `intern_display(&Display)`.
Strings shorter than the inline threshold (7 bytes on 64-bit) are
already zero-allocation inline atoms and skip the set entirely.

**3. `String` → `RcStr` for span fields**

Changed `Span.name`, `Span.category`, and `Span.args` key/value types
from `String`/`Vec<(String, String)>` to `RcStr`/`Vec<(RcStr, RcStr)>`.
Same for `SpanNames` and `SpanBottomUpBuilder`. All `to_string()` /
`into_owned()` calls in `process_internal_row_queue` are replaced with
`interner.intern()` / `intern_cow()` / `intern_display()` calls on a
per-`TurbopackFormat` `RcStrInterning`. Repeated names like
`"turbo_tasks::function"` now share a single allocation via refcount
bump instead of a fresh heap allocation each time. The `into_static()`
path for queued rows also benefits: cloning an already-interned `RcStr`
is a refcount increment, not a string copy.

**4. Capacity hints**

Pre-allocated major collections to avoid repeated reallocation during
the initial bulk load:
- `Store::spans`: `Vec::with_capacity(131_072)`
- `TurbopackFormat::id_mapping`: `with_capacity(131_072)`
- `TurbopackFormat::outdated_spans`: `with_capacity(8_192)`
- `TurbopackFormat::queued_rows`: `with_capacity(1_024)`
- `TurbopackFormat::thread_stacks`, `thread_allocation_counters`:
`with_capacity(64)`
- `TurbopackFormat::self_time_started`: `with_capacity(256)`

**5. Reused queue Vec (hot spot #8)**

`TurbopackFormat` now holds `row_queue: Vec<InternalRow<'static>>` as a
field. `process_internal_row` takes it via `take` and puts it back after
the loop, retaining the allocated capacity across calls instead of
allocating a new `Vec::new()` for every row processed.

**6. `TraceFormat::create_reused()` hook (hot spot #4)**

Added an overridable `create_reused()` method to the `TraceFormat`
trait. `TurbopackFormat` overrides it to return
`Vec::with_capacity(4_096)`, giving the row deserialization buffer
reasonable initial capacity to avoid repeated doubling during the
initial bulk read.

**7. `SelfTimeTree` leaf capacity (hot spot #2)**

New child nodes created at split time get
`Vec::with_capacity(SPLIT_COUNT / 2)` = 64 entries, matching the
expected post-split occupancy and avoiding incremental doubling after
every split.

<!-- NEXT_JS_LLM_PR -->

---------
Co-authored-by: 's avatarTobias Koppers <sokra@users.noreply.github.com>
Co-authored-by: 's avatarClaude <noreply@anthropic.com>
parent 7873aeaa
...@@ -10808,6 +10808,8 @@ dependencies = [ ...@@ -10808,6 +10808,8 @@ dependencies = [
"serde", "serde",
"serde_json", "serde_json",
"tungstenite 0.21.0", "tungstenite 0.21.0",
"turbo-rcstr",
"turbo-tasks-malloc",
"turbopack-trace-utils", "turbopack-trace-utils",
"zstd", "zstd",
] ]
......
...@@ -614,6 +614,76 @@ mod napi_impl { ...@@ -614,6 +614,76 @@ mod napi_impl {
} }
} }
/// Runtime string interning table.
///
/// Deduplicates strings by storing them in an `FxHashSet<RcStr>`. Strings
/// shorter than the inline threshold are already zero-allocation, so only
/// longer strings benefit from interning.
pub struct RcStrInterning {
set: rustc_hash::FxHashSet<RcStr>,
}
impl Default for RcStrInterning {
fn default() -> Self {
Self::new()
}
}
impl RcStrInterning {
/// Create a new empty interning table.
pub fn new() -> Self {
Self {
set: rustc_hash::FxHashSet::default(),
}
}
/// Intern a string slice. Returns a cheap-to-clone [`RcStr`].
///
/// Strings below the inline threshold are returned directly (they are
/// already zero-allocation inline atoms). Longer strings are looked up
/// in the interning table and deduplicated.
pub fn intern(&mut self, s: &str) -> RcStr {
if s.len() < tagged_value::MAX_INLINE_LEN {
// Inline atom — no allocation needed, don't bother with the set.
return RcStr::from(s);
}
if let Some(existing) = self.set.get(s) {
return existing.clone();
}
let rc = RcStr::from(s);
self.set.insert(rc.clone());
rc
}
/// Intern an owned `String`. When the string is not yet interned, avoids
/// an extra copy compared to [`intern`](Self::intern).
fn intern_owned(&mut self, s: String) -> RcStr {
if s.len() < tagged_value::MAX_INLINE_LEN {
return RcStr::from(s);
}
if let Some(existing) = self.set.get(s.as_str()) {
return existing.clone();
}
let rc = RcStr::from(s);
self.set.insert(rc.clone());
rc
}
/// Intern a `Cow<str>`. When the cow is `Owned`, avoids an extra copy
/// if the string is not yet interned.
pub fn intern_cow(&mut self, s: std::borrow::Cow<'_, str>) -> RcStr {
match s {
std::borrow::Cow::Borrowed(s) => self.intern(s),
std::borrow::Cow::Owned(s) => self.intern_owned(s),
}
}
/// Intern the [`Display`](std::fmt::Display) output of a value.
pub fn intern_display(&mut self, v: &impl std::fmt::Display) -> RcStr {
self.intern_owned(v.to_string())
}
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use std::mem::ManuallyDrop; use std::mem::ManuallyDrop;
...@@ -762,4 +832,38 @@ mod tests { ...@@ -762,4 +832,38 @@ mod tests {
// Decoded via peek_read path should find the static constant // Decoded via peek_read path should find the static constant
assert_eq!(decoded.tag(), STATIC_TAG); assert_eq!(decoded.tag(), STATIC_TAG);
} }
#[test]
fn test_interning() {
let mut interner = RcStrInterning::new();
// Short strings are always inline (no interning needed)
let a = interner.intern("hi");
let b = interner.intern("hi");
assert_eq!(a, b);
// Long strings should be deduplicated to the same allocation.
let long = "this is a long string that exceeds inline threshold";
let c = interner.intern(long);
let d = interner.intern(long);
assert_eq!(c, d);
assert!(std::ptr::eq(c.as_str().as_ptr(), d.as_str().as_ptr()));
// intern_cow with borrowed — same allocation as c
let e = interner.intern_cow(std::borrow::Cow::Borrowed(long));
assert_eq!(e, c);
assert!(std::ptr::eq(e.as_str().as_ptr(), c.as_str().as_ptr()));
// intern_cow with owned — same allocation as c (no new alloc)
let f = interner.intern_cow(std::borrow::Cow::Owned(long.to_string()));
assert_eq!(f, c);
assert!(std::ptr::eq(f.as_str().as_ptr(), c.as_str().as_ptr()));
// intern_display — a fresh long string, verify it is interned too
let long2 = "another long string that exceeds the inline threshold here";
let g = interner.intern_display(&long2);
let h = interner.intern_display(&long2);
assert_eq!(g, h);
assert!(std::ptr::eq(g.as_str().as_ptr(), h.as_str().as_ptr()));
}
} }
...@@ -6,6 +6,10 @@ license = "MIT" ...@@ -6,6 +6,10 @@ license = "MIT"
edition = "2024" edition = "2024"
autobenches = false autobenches = false
[features]
default = ["custom_allocator"]
custom_allocator = ["turbo-tasks-malloc/custom_allocator"]
[[bin]] [[bin]]
name = "turbo-trace-server" name = "turbo-trace-server"
path = "src/main.rs" path = "src/main.rs"
...@@ -24,6 +28,8 @@ rustc-demangle = "0.1" ...@@ -24,6 +28,8 @@ rustc-demangle = "0.1"
rustc-hash = { workspace = true } rustc-hash = { workspace = true }
serde = { workspace = true } serde = { workspace = true }
serde_json = { workspace = true } serde_json = { workspace = true }
turbo-rcstr = { workspace = true }
turbo-tasks-malloc = { workspace = true, default-features = false }
tungstenite = { version = "0.21.0" } tungstenite = { version = "0.21.0" }
turbopack-trace-utils = { workspace = true } turbopack-trace-utils = { workspace = true }
zstd = { version = "0.13.0" } zstd = { version = "0.13.0" }
use std::{env, sync::Arc}; use std::{env, sync::Arc};
use hashbrown::HashMap; use hashbrown::HashMap;
use turbo_rcstr::RcStr;
use crate::{ use crate::{
span::{SpanBottomUp, SpanIndex}, span::{SpanBottomUp, SpanIndex},
...@@ -11,7 +12,7 @@ use crate::{ ...@@ -11,7 +12,7 @@ use crate::{
pub struct SpanBottomUpBuilder { pub struct SpanBottomUpBuilder {
// These values won't change after creation: // These values won't change after creation:
pub self_spans: Vec<SpanIndex>, pub self_spans: Vec<SpanIndex>,
pub children: HashMap<(String, String), SpanBottomUpBuilder>, pub children: HashMap<(RcStr, RcStr), SpanBottomUpBuilder>,
pub example_span: SpanIndex, pub example_span: SpanIndex,
} }
...@@ -43,7 +44,7 @@ pub fn build_bottom_up_graph<'a>( ...@@ -43,7 +44,7 @@ pub fn build_bottom_up_graph<'a>(
.ok() .ok()
.and_then(|s| s.parse().ok()) .and_then(|s| s.parse().ok())
.unwrap_or(usize::MAX); .unwrap_or(usize::MAX);
let mut roots: HashMap<(String, String), SpanBottomUpBuilder> = HashMap::default(); let mut roots: HashMap<(RcStr, RcStr), SpanBottomUpBuilder> = HashMap::default();
// unfortunately there is a rustc bug that fails the typechecking here // unfortunately there is a rustc bug that fails the typechecking here
// when using Either<impl Iterator, impl Iterator>. This error appears // when using Either<impl Iterator, impl Iterator>. This error appears
...@@ -64,7 +65,7 @@ pub fn build_bottom_up_graph<'a>( ...@@ -64,7 +65,7 @@ pub fn build_bottom_up_graph<'a>(
.from_key(&StringTupleRef(category, name)) .from_key(&StringTupleRef(category, name))
.or_insert_with(|| { .or_insert_with(|| {
( (
(category.to_string(), name.to_string()), (RcStr::from(category), RcStr::from(name)),
SpanBottomUpBuilder::new(child.index()), SpanBottomUpBuilder::new(child.index()),
) )
}); });
...@@ -80,7 +81,7 @@ pub fn build_bottom_up_graph<'a>( ...@@ -80,7 +81,7 @@ pub fn build_bottom_up_graph<'a>(
.from_key(&StringTupleRef(category, title)) .from_key(&StringTupleRef(category, title))
.or_insert_with(|| { .or_insert_with(|| {
( (
(category.to_string(), title.to_string()), (RcStr::from(category), RcStr::from(title)),
SpanBottomUpBuilder::new(example_span), SpanBottomUpBuilder::new(example_span),
) )
}); });
......
#![feature(box_patterns)] #![feature(box_patterns)]
#![feature(bufreader_peek)] #![feature(bufreader_peek)]
#[global_allocator]
static ALLOC: turbo_tasks_malloc::TurboMalloc = turbo_tasks_malloc::TurboMalloc;
use std::{hash::BuildHasherDefault, sync::Arc}; use std::{hash::BuildHasherDefault, sync::Arc};
use indexmap::{IndexMap, IndexSet}; use indexmap::{IndexMap, IndexSet};
......
...@@ -4,6 +4,7 @@ use anyhow::{Context, Result, bail}; ...@@ -4,6 +4,7 @@ use anyhow::{Context, Result, bail};
use indexmap::map::Entry; use indexmap::map::Entry;
use rustc_demangle::demangle; use rustc_demangle::demangle;
use rustc_hash::{FxHashMap, FxHashSet}; use rustc_hash::{FxHashMap, FxHashSet};
use turbo_rcstr::{RcStr, rcstr};
use super::TraceFormat; use super::TraceFormat;
use crate::{FxIndexMap, span::SpanIndex, store_container::StoreContainer, timestamp::Timestamp}; use crate::{FxIndexMap, span::SpanIndex, store_container::StoreContainer, timestamp::Timestamp};
...@@ -273,8 +274,8 @@ impl TraceFormat for HeaptrackFormat { ...@@ -273,8 +274,8 @@ impl TraceFormat for HeaptrackFormat {
let span_index = store.add_span( let span_index = store.add_span(
Some(parent.span_index), Some(parent.span_index),
self.last_timestamp, self.last_timestamp,
"".to_string(), RcStr::default(),
"recursion".to_string(), rcstr!("recursion"),
Vec::new(), Vec::new(),
&mut outdated_spans, &mut outdated_spans,
); );
...@@ -333,16 +334,16 @@ impl TraceFormat for HeaptrackFormat { ...@@ -333,16 +334,16 @@ impl TraceFormat for HeaptrackFormat {
.get(*function_index) .get(*function_index)
.context("function not found")?; .context("function not found")?;
args.push(( args.push((
"location".to_string(), rcstr!("location"),
format!("{function} @ {file}:{line}"), RcStr::from(format!("{function} @ {file}:{line}")),
)); ));
} }
let span_index = store.add_span( let span_index = store.add_span(
parent, parent,
self.last_timestamp, self.last_timestamp,
module.to_string(), RcStr::from(module.as_str()),
name, RcStr::from(name),
args, args,
&mut outdated_spans, &mut outdated_spans,
); );
......
...@@ -25,6 +25,10 @@ const MIN_INITIAL_REPORT_SIZE: u64 = 100 * 1024 * 1024; ...@@ -25,6 +25,10 @@ const MIN_INITIAL_REPORT_SIZE: u64 = 100 * 1024 * 1024;
trait TraceFormat { trait TraceFormat {
type Reused: Default; type Reused: Default;
/// Create the initial reused buffer. Override to pre-allocate capacity.
fn create_reused() -> Self::Reused {
Self::Reused::default()
}
fn read(&mut self, buffer: &[u8], reuse: &mut Self::Reused) -> Result<usize>; fn read(&mut self, buffer: &[u8], reuse: &mut Self::Reused) -> Result<usize>;
fn stats(&self) -> String { fn stats(&self) -> String {
String::new() String::new()
...@@ -46,7 +50,7 @@ where ...@@ -46,7 +50,7 @@ where
T::Reused: 'static, T::Reused: 'static,
{ {
fn create_reused(&self) -> ErasedReused { fn create_reused(&self) -> ErasedReused {
Box::new(T::Reused::default()) Box::new(T::create_reused())
} }
fn read(&mut self, buffer: &[u8], reuse: &mut ErasedReused) -> Result<usize> { fn read(&mut self, buffer: &[u8], reuse: &mut ErasedReused) -> Result<usize> {
......
...@@ -2,6 +2,7 @@ use std::{borrow::Cow, fmt::Display, sync::Arc}; ...@@ -2,6 +2,7 @@ use std::{borrow::Cow, fmt::Display, sync::Arc};
use rustc_hash::{FxHashMap, FxHashSet}; use rustc_hash::{FxHashMap, FxHashSet};
use serde::Deserialize; use serde::Deserialize;
use turbo_rcstr::{RcStr, rcstr};
use super::TraceFormat; use super::TraceFormat;
use crate::{FxIndexMap, span::SpanIndex, store_container::StoreContainer, timestamp::Timestamp}; use crate::{FxIndexMap, span::SpanIndex, store_container::StoreContainer, timestamp::Timestamp};
...@@ -62,13 +63,13 @@ impl TraceFormat for NextJsFormat { ...@@ -62,13 +63,13 @@ impl TraceFormat for NextJsFormat {
let index = store.add_span( let index = store.add_span(
parent, parent,
timestamp, timestamp,
"nextjs".to_string(), rcstr!("nextjs"),
name.into_owned(), RcStr::from(name.into_owned()),
tags.iter() tags.iter()
.map(|(k, v)| { .map(|(k, v)| {
( (
k.to_string(), RcStr::from(k.as_ref()),
v.as_ref().map(|v| v.to_string()).unwrap_or_default(), RcStr::from(v.as_ref().map(|v| v.to_string()).unwrap_or_default()),
) )
}) })
.collect(), .collect(),
......
...@@ -8,6 +8,7 @@ use std::{ ...@@ -8,6 +8,7 @@ use std::{
use anyhow::Result; use anyhow::Result;
use rustc_hash::{FxHashMap, FxHashSet}; use rustc_hash::{FxHashMap, FxHashSet};
use turbo_rcstr::{RcStr, RcStrInterning};
use turbopack_trace_utils::tracing::{TraceRow, TraceValue}; use turbopack_trace_utils::tracing::{TraceRow, TraceValue};
use super::TraceFormat; use super::TraceFormat;
...@@ -124,31 +125,32 @@ impl InternalRowType<'_> { ...@@ -124,31 +125,32 @@ impl InternalRowType<'_> {
} }
} }
#[derive(Default)]
struct QueuedRows {
rows: Vec<InternalRow<'static>>,
}
pub struct TurbopackFormat { pub struct TurbopackFormat {
store: Arc<StoreContainer>, store: Arc<StoreContainer>,
id_mapping: FxHashMap<u64, SpanIndex>, id_mapping: FxHashMap<u64, SpanIndex>,
queued_rows: FxHashMap<u64, QueuedRows>, queued_rows: FxHashMap<u64, Vec<InternalRow<'static>>>,
outdated_spans: FxHashSet<SpanIndex>, outdated_spans: FxHashSet<SpanIndex>,
thread_stacks: FxHashMap<u64, Vec<u64>>, thread_stacks: FxHashMap<u64, Vec<u64>>,
thread_allocation_counters: FxHashMap<u64, AllocationInfo>, thread_allocation_counters: FxHashMap<u64, AllocationInfo>,
self_time_started: FxHashMap<(u64, u64), Timestamp>, self_time_started: FxHashMap<(u64, u64), Timestamp>,
/// Scratch buffer reused across `process_internal_row` calls to avoid
/// per-call allocation. Always empty between calls.
row_queue: Vec<InternalRow<'static>>,
interner: RcStrInterning,
} }
impl TurbopackFormat { impl TurbopackFormat {
pub fn new(store: Arc<StoreContainer>) -> Self { pub fn new(store: Arc<StoreContainer>) -> Self {
Self { Self {
store, store,
id_mapping: FxHashMap::default(), id_mapping: FxHashMap::with_capacity_and_hasher(131_072, Default::default()),
queued_rows: FxHashMap::default(), queued_rows: FxHashMap::with_capacity_and_hasher(1_024, Default::default()),
outdated_spans: FxHashSet::default(), outdated_spans: FxHashSet::with_capacity_and_hasher(8_192, Default::default()),
thread_stacks: FxHashMap::default(), thread_stacks: FxHashMap::with_capacity_and_hasher(64, Default::default()),
thread_allocation_counters: FxHashMap::default(), thread_allocation_counters: FxHashMap::with_capacity_and_hasher(64, Default::default()),
self_time_started: FxHashMap::default(), self_time_started: FxHashMap::with_capacity_and_hasher(256, Default::default()),
row_queue: Vec::new(),
interner: RcStrInterning::new(),
} }
} }
...@@ -356,21 +358,23 @@ impl TurbopackFormat { ...@@ -356,21 +358,23 @@ impl TurbopackFormat {
} }
fn process_internal_row(&mut self, store: &mut StoreWriteGuard, row: InternalRow<'_>) { fn process_internal_row(&mut self, store: &mut StoreWriteGuard, row: InternalRow<'_>) {
let mut queue = Vec::new(); // Reclaim capacity from the previous call; the field is always empty between calls.
queue.push(row); let mut queue = take(&mut self.row_queue);
self.process_internal_row_queue(store, row, &mut queue);
while !queue.is_empty() { while !queue.is_empty() {
let q = take(&mut queue); let batch = take(&mut queue);
for row in q { for row in batch {
self.process_internal_row_queue(store, row, &mut queue); self.process_internal_row_queue(store, row, &mut queue);
} }
} }
self.row_queue = queue; // save capacity for next call
} }
fn process_internal_row_queue( fn process_internal_row_queue(
&mut self, &mut self,
store: &mut StoreWriteGuard, store: &mut StoreWriteGuard,
row: InternalRow<'_>, row: InternalRow<'_>,
queue: &mut Vec<InternalRow<'_>>, queue: &mut Vec<InternalRow<'static>>,
) { ) {
let id = if let Some(id) = row.id { let id = if let Some(id) = row.id {
if let Some(id) = self.id_mapping.get(&id) { if let Some(id) = self.id_mapping.get(&id) {
...@@ -379,7 +383,6 @@ impl TurbopackFormat { ...@@ -379,7 +383,6 @@ impl TurbopackFormat {
self.queued_rows self.queued_rows
.entry(id) .entry(id)
.or_default() .or_default()
.rows
.push(row.into_static()); .push(row.into_static());
return; return;
} }
...@@ -397,19 +400,17 @@ impl TurbopackFormat { ...@@ -397,19 +400,17 @@ impl TurbopackFormat {
let span_id = store.add_span( let span_id = store.add_span(
id, id,
ts, ts,
target.into_owned(), self.interner.intern_cow(target),
name.into_owned(), self.interner.intern_cow(name),
values values
.iter() .iter()
.map(|(k, v)| (k.to_string(), v.to_string())) .map(|(k, v)| (self.interner.intern(k), self.interner.intern_display(v)))
.collect(), .collect(),
&mut self.outdated_spans, &mut self.outdated_spans,
); );
self.id_mapping.insert(new_id, span_id); self.id_mapping.insert(new_id, span_id);
if let Some(QueuedRows { rows }) = self.queued_rows.remove(&new_id) { if let Some(rows) = self.queued_rows.remove(&new_id) {
for row in rows { queue.extend(rows);
queue.push(row);
}
} }
} }
InternalRowType::Record { ref values } => { InternalRowType::Record { ref values } => {
...@@ -417,7 +418,7 @@ impl TurbopackFormat { ...@@ -417,7 +418,7 @@ impl TurbopackFormat {
id.unwrap(), id.unwrap(),
values values
.iter() .iter()
.map(|(k, v)| (k.to_string(), v.to_string())) .map(|(k, v)| (self.interner.intern(k), self.interner.intern_display(v)))
.collect(), .collect(),
&mut self.outdated_spans, &mut self.outdated_spans,
); );
...@@ -438,17 +439,17 @@ impl TurbopackFormat { ...@@ -438,17 +439,17 @@ impl TurbopackFormat {
); );
let name = values let name = values
.swap_remove("name") .swap_remove("name")
.and_then(|v| v.as_str().map(|s| s.to_string())) .and_then(|v| v.as_str().map(|s| self.interner.intern(s)))
.unwrap_or("event".into()); .unwrap_or_else(|| RcStr::from("event"));
let id = store.add_span( let id = store.add_span(
id, id,
ts.saturating_sub(duration), ts.saturating_sub(duration),
"event".into(), RcStr::from("event"),
name, name,
values values
.iter() .iter()
.map(|(k, v)| (k.to_string(), v.to_string())) .map(|(k, v)| (self.interner.intern(k), self.interner.intern_display(v)))
.collect(), .collect(),
&mut self.outdated_spans, &mut self.outdated_spans,
); );
...@@ -489,6 +490,15 @@ impl TurbopackFormat { ...@@ -489,6 +490,15 @@ impl TurbopackFormat {
impl TraceFormat for TurbopackFormat { impl TraceFormat for TurbopackFormat {
type Reused = Vec<TraceRow<'static>>; type Reused = Vec<TraceRow<'static>>;
fn create_reused() -> Vec<TraceRow<'static>> {
// Pre-allocate for a typical batch size to avoid repeated doubling during initial read.
Vec::with_capacity(4_096)
}
fn stats(&self) -> String {
format!("{} spans", self.id_mapping.len())
}
fn read(&mut self, mut buffer: &[u8], reuse: &mut Self::Reused) -> Result<usize> { fn read(&mut self, mut buffer: &[u8], reuse: &mut Self::Reused) -> Result<usize> {
reuse.clear(); reuse.clear();
let mut reuse = ClearOnDrop(reuse); let mut reuse = ClearOnDrop(reuse);
......
...@@ -77,10 +77,18 @@ impl<T> SelfTimeTree<T> { ...@@ -77,10 +77,18 @@ impl<T> SelfTimeTree<T> {
let start = self.entries.iter().min_by_key(|e| e.start).unwrap().start; let start = self.entries.iter().min_by_key(|e| e.start).unwrap().start;
let end = self.entries.iter().max_by_key(|e| e.end).unwrap().end; let end = self.entries.iter().max_by_key(|e| e.end).unwrap().end;
let middle = (start + end) / 2; let middle = (start + end) / 2;
// Pre-allocate half the split threshold: after distributing, each child
// typically receives ~SPLIT_COUNT / 2 entries.
self.children = Some(Box::new(SelfTimeChildren { self.children = Some(Box::new(SelfTimeChildren {
left: SelfTimeTree::new(), left: SelfTimeTree {
entries: Vec::with_capacity(SPLIT_COUNT / 2),
..SelfTimeTree::default()
},
split_point: middle, split_point: middle,
right: SelfTimeTree::new(), right: SelfTimeTree {
entries: Vec::with_capacity(SPLIT_COUNT / 2),
..SelfTimeTree::default()
},
spanning_entries: 0, spanning_entries: 0,
})); }));
} }
......
...@@ -4,6 +4,7 @@ use std::{ ...@@ -4,6 +4,7 @@ use std::{
}; };
use hashbrown::HashMap; use hashbrown::HashMap;
use turbo_rcstr::RcStr;
use crate::timestamp::Timestamp; use crate::timestamp::Timestamp;
...@@ -14,9 +15,9 @@ pub struct Span { ...@@ -14,9 +15,9 @@ pub struct Span {
pub parent: Option<SpanIndex>, pub parent: Option<SpanIndex>,
pub depth: u32, pub depth: u32,
pub start: Timestamp, pub start: Timestamp,
pub category: String, pub category: RcStr,
pub name: String, pub name: RcStr,
pub args: Vec<(String, String)>, pub args: Vec<(RcStr, RcStr)>,
// This might change during writing: // This might change during writing:
pub events: Vec<SpanEvent>, pub events: Vec<SpanEvent>,
...@@ -64,14 +65,14 @@ pub struct SpanTimeData { ...@@ -64,14 +65,14 @@ pub struct SpanTimeData {
pub struct SpanExtra { pub struct SpanExtra {
pub graph: OnceLock<Vec<SpanGraphEvent>>, pub graph: OnceLock<Vec<SpanGraphEvent>>,
pub bottom_up: OnceLock<Vec<Arc<SpanBottomUp>>>, pub bottom_up: OnceLock<Vec<Arc<SpanBottomUp>>>,
pub search_index: OnceLock<HashMap<String, Vec<SpanIndex>>>, pub search_index: OnceLock<HashMap<RcStr, Vec<SpanIndex>>>,
} }
#[derive(Default)] #[derive(Default)]
pub struct SpanNames { pub struct SpanNames {
// These values are computed when accessed (and maybe deleted during writing): // These values are computed when accessed (and maybe deleted during writing):
pub nice_name: OnceLock<(String, String)>, pub nice_name: OnceLock<(RcStr, RcStr)>,
pub group_name: OnceLock<(String, String)>, pub group_name: OnceLock<(RcStr, RcStr)>,
} }
impl Span { impl Span {
......
...@@ -8,6 +8,7 @@ use std::{ ...@@ -8,6 +8,7 @@ use std::{
use hashbrown::HashMap; use hashbrown::HashMap;
use rayon::iter::{IntoParallelIterator, IntoParallelRefIterator, ParallelIterator}; use rayon::iter::{IntoParallelIterator, IntoParallelRefIterator, ParallelIterator};
use rustc_hash::FxHashSet; use rustc_hash::FxHashSet;
use turbo_rcstr::{RcStr, rcstr};
use crate::{ use crate::{
FxIndexMap, FxIndexMap,
...@@ -90,26 +91,26 @@ impl<'a> SpanRef<'a> { ...@@ -90,26 +91,26 @@ impl<'a> SpanRef<'a> {
.args .args
.iter() .iter()
.find(|&(k, _)| k == "name") .find(|&(k, _)| k == "name")
.map(|(_, v)| v.as_str()) .map(|(_, v)| v)
{ {
if matches!(self.span.name.as_str(), "turbo_tasks::function") { if matches!(self.span.name.as_str(), "turbo_tasks::function") {
(self.span.name.clone(), name.to_string()) (self.span.name.clone(), name.clone())
} else if matches!( } else if matches!(
self.span.name.as_str(), self.span.name.as_str(),
"turbo_tasks::resolve_call" | "turbo_tasks::resolve_trait_call" "turbo_tasks::resolve_call" | "turbo_tasks::resolve_trait_call"
) { ) {
(self.span.name.clone(), format!("*{name}")) (self.span.name.clone(), format!("*{name}").into())
} else { } else {
( (
self.span.category.clone(), self.span.category.clone(),
format!("{} {name}", self.span.name), format!("{} {name}", self.span.name).into(),
) )
} }
} else { } else {
(self.span.category.to_string(), self.span.name.to_string()) (self.span.category.clone(), self.span.name.clone())
} }
}); });
(category, title) (category.as_str(), title.as_str())
} }
pub fn group_name(&self) -> (&'a str, &'a str) { pub fn group_name(&self) -> (&'a str, &'a str) {
...@@ -120,8 +121,8 @@ impl<'a> SpanRef<'a> { ...@@ -120,8 +121,8 @@ impl<'a> SpanRef<'a> {
.args .args
.iter() .iter()
.find(|&(k, _)| k == "name") .find(|&(k, _)| k == "name")
.map(|(_, v)| v.to_string()) .map(|(_, v)| v.clone())
.unwrap_or_else(|| self.span.name.to_string()); .unwrap_or_else(|| self.span.name.clone());
(self.span.name.clone(), name) (self.span.name.clone(), name)
} else if matches!( } else if matches!(
self.span.name.as_str(), self.span.name.as_str(),
...@@ -132,11 +133,11 @@ impl<'a> SpanRef<'a> { ...@@ -132,11 +133,11 @@ impl<'a> SpanRef<'a> {
.args .args
.iter() .iter()
.find(|&(k, _)| k == "name") .find(|&(k, _)| k == "name")
.map(|(_, v)| format!("*{v}")) .map(|(_, v)| RcStr::from(format!("*{v}")))
.unwrap_or_else(|| self.span.name.to_string()); .unwrap_or_else(|| self.span.name.clone());
( (
self.span.category.clone(), self.span.category.clone(),
format!("{} {name}", self.span.name), format!("{} {name}", self.span.name).into(),
) )
} else { } else {
(self.span.category.clone(), self.span.name.clone()) (self.span.category.clone(), self.span.name.clone())
...@@ -422,7 +423,7 @@ impl<'a> SpanRef<'a> { ...@@ -422,7 +423,7 @@ impl<'a> SpanRef<'a> {
}) })
} }
fn search_index(&self) -> &HashMap<String, Vec<SpanIndex>> { fn search_index(&self) -> &HashMap<RcStr, Vec<SpanIndex>> {
self.extra().search_index.get_or_init(|| { self.extra().search_index.get_or_init(|| {
let mut all_spans = Vec::new(); let mut all_spans = Vec::new();
all_spans.push(self.index); all_spans.push(self.index);
...@@ -442,41 +443,58 @@ impl<'a> SpanRef<'a> { ...@@ -442,41 +443,58 @@ impl<'a> SpanRef<'a> {
enum SpanOrMap<'a> { enum SpanOrMap<'a> {
Span(SpanRef<'a>), Span(SpanRef<'a>),
Map(HashMap<String, Vec<SpanIndex>>), Map(HashMap<RcStr, Vec<SpanIndex>>),
} }
fn add_span_to_map<'a>(index: &mut HashMap<String, Vec<SpanIndex>>, span: SpanRef<'a>) { /// Insert `span_index` into `index` under the given key.
if !span.is_root() { ///
let (cat, name) = span.nice_name(); /// `lookup` is the string used for the hash lookup. `make_key` produces
if !cat.is_empty() { /// the stored map key when a new entry is created (may differ from
index /// `lookup`, e.g. `"name=foo"` stored under the hash of `"foo"`).
.raw_entry_mut() fn push_to_index(
.from_key(cat) index: &mut HashMap<RcStr, Vec<SpanIndex>>,
.and_modify(|_, v| v.push(span.index())) lookup: &str,
.or_insert_with(|| (cat.to_string(), vec![span.index()])); make_key: impl FnOnce() -> RcStr,
} span_index: SpanIndex,
if !name.is_empty() { ) {
index index
.raw_entry_mut() .raw_entry_mut()
.from_key(name) .from_key(lookup)
.and_modify(|_, v| v.push(span.index())) .and_modify(|_, v| v.push(span_index))
.or_insert_with(|| (format!("name={name}"), vec![span.index()])); .or_insert_with(|| (make_key(), vec![span_index]));
} }
for (name, value) in span.span.args.iter() {
index fn add_span_to_map<'a>(index: &mut HashMap<RcStr, Vec<SpanIndex>>, span: SpanRef<'a>) {
.raw_entry_mut() if span.is_root() {
.from_key(value) return;
.and_modify(|_, v| v.push(span.index())) }
.or_insert_with(|| (format!("{name}={value}"), vec![span.index()])); let (cat, name) = span.nice_name();
} if !cat.is_empty() {
if !span.is_complete() && span.span.name != "thread" { push_to_index(index, cat, || RcStr::from(cat), span.index());
let name = "incomplete_span"; }
index if !name.is_empty() {
.raw_entry_mut() push_to_index(
.from_key(name) index,
.and_modify(|_, v| v.push(span.index())) name,
.or_insert_with(|| (name.to_string(), vec![span.index()])); || RcStr::from(format!("name={name}")),
} span.index(),
);
}
for (k, v) in span.span.args.iter() {
push_to_index(
index,
v.as_str(),
|| RcStr::from(format!("{k}={v}")),
span.index(),
);
}
if !span.is_complete() && span.span.name != "thread" {
push_to_index(
index,
"incomplete_span",
|| rcstr!("incomplete_span"),
span.index(),
);
} }
} }
......
...@@ -6,6 +6,7 @@ use std::{ ...@@ -6,6 +6,7 @@ use std::{
}; };
use rustc_hash::FxHashSet; use rustc_hash::FxHashSet;
use turbo_rcstr::{RcStr, rcstr};
use crate::{ use crate::{
self_time_tree::SelfTimeTree, self_time_tree::SelfTimeTree,
...@@ -42,8 +43,8 @@ fn new_root_span() -> Span { ...@@ -42,8 +43,8 @@ fn new_root_span() -> Span {
parent: None, parent: None,
depth: 0, depth: 0,
start: Timestamp::MAX, start: Timestamp::MAX,
category: "".into(), category: RcStr::default(),
name: "(root)".into(), name: rcstr!("(root)"),
args: vec![], args: vec![],
events: vec![], events: vec![],
is_complete: true, is_complete: true,
...@@ -66,7 +67,11 @@ fn new_root_span() -> Span { ...@@ -66,7 +67,11 @@ fn new_root_span() -> Span {
impl Store { impl Store {
pub fn new() -> Self { pub fn new() -> Self {
Self { Self {
spans: vec![new_root_span()], spans: {
let mut v = Vec::with_capacity(131_072);
v.push(new_root_span());
v
},
self_time_tree: env::var("NO_CORRECTED_TIME") self_time_tree: env::var("NO_CORRECTED_TIME")
.ok() .ok()
.is_none() .is_none()
...@@ -96,9 +101,9 @@ impl Store { ...@@ -96,9 +101,9 @@ impl Store {
&mut self, &mut self,
parent: Option<SpanIndex>, parent: Option<SpanIndex>,
start: Timestamp, start: Timestamp,
category: String, category: RcStr,
name: String, name: RcStr,
args: Vec<(String, String)>, args: Vec<(RcStr, RcStr)>,
outdated_spans: &mut FxHashSet<SpanIndex>, outdated_spans: &mut FxHashSet<SpanIndex>,
) -> SpanIndex { ) -> SpanIndex {
let id = SpanIndex::new(self.spans.len()).unwrap(); let id = SpanIndex::new(self.spans.len()).unwrap();
...@@ -152,7 +157,7 @@ impl Store { ...@@ -152,7 +157,7 @@ impl Store {
pub fn add_args( pub fn add_args(
&mut self, &mut self,
span_index: SpanIndex, span_index: SpanIndex,
args: Vec<(String, String)>, args: Vec<(RcStr, RcStr)>,
outdated_spans: &mut FxHashSet<SpanIndex>, outdated_spans: &mut FxHashSet<SpanIndex>,
) { ) {
let span = &mut self.spans[span_index.get()]; let span = &mut self.spans[span_index.get()];
......
use hashbrown::Equivalent; use hashbrown::Equivalent;
use turbo_rcstr::RcStr;
#[derive(Hash)] #[derive(Hash)]
pub struct StringTupleRef<'a>(pub &'a str, pub &'a str); pub struct StringTupleRef<'a>(pub &'a str, pub &'a str);
impl<'a> Equivalent<(String, String)> for StringTupleRef<'a> { impl<'a> Equivalent<(RcStr, RcStr)> for StringTupleRef<'a> {
fn equivalent(&self, other: &(String, String)) -> bool { fn equivalent(&self, other: &(RcStr, RcStr)) -> bool {
self.0 == other.0 && self.1 == other.1 other.0 == self.0 && other.1 == self.1
} }
} }
...@@ -22,7 +23,7 @@ mod string_tuple_ref_tests { ...@@ -22,7 +23,7 @@ mod string_tuple_ref_tests {
let s = RandomState::new(); let s = RandomState::new();
assert_eq!( assert_eq!(
s.hash_one(StringTupleRef("abc", "def")), s.hash_one(StringTupleRef("abc", "def")),
s.hash_one(&("abc".to_string(), "def".to_string())) s.hash_one(&(RcStr::from("abc"), RcStr::from("def")))
); );
} }
} }
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment