Unverified Commit 66776bd9 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

[turbo-trace-server] optimize loading even more (#93332)

## What

Reduce time and memory spent loading large traces in
`turbopack-trace-server`, focused on the row-ingestion hot path
(`reader/turbopack.rs`).

**Eager interning at parse time**
- New `OwnedTraceValue` enum (lifetime-free) and `SpanArgs` reuse for
`InternalRow` value lists. All borrowed `Cow<str>` keys/values are
interned to `RcStr` once, at parse time, via a single `intern_span_args`
helper.
- `InternalRow`/`InternalRowType` shed their `'a` parameter. The
`into_static` path is gone — previously, every row that had to be queued
waiting for its parent allocated a fresh `String` for each Cow field,
then the eventual processor re-interned those strings. Now both costs
are gone: one intern, no String allocation.
- `Event` rows pre-extract `duration` (from `TraceValue::UInt`) and
`name` (from `TraceValue::String`) at parse time, so the per-event
`FxIndexMap` `swap_remove` lookup goes away entirely.

**Inline queue flush instead of working-queue extend**
- `process_internal_row` and `process_internal_row_queue` collapsed into
a single recursive method. When a `Start` row flushes queued children
(i.e. orphaned rows whose parent has now appeared), those rows are
processed inline via direct recursion instead of being `extend`ed into a
working queue and re-driven. Eliminates the `Vec<InternalRow>::extend`
memcpy that showed up as a top memmove site, and keeps the just-added
`Span` hot in cache while its child events are applied.
- `row_queue` field and the `take`/swap driver loop on `TurbopackFormat`
removed.

**New ChunkedVec storage for Span**
- reduces costs when adding Span objects, since they are large and
expensive to move
- reduces peak heap due to resizing (in exchange for larger min sizes)

## Result

Loading a 10 GB trace: **53s → 45s** wall clock, **21.5 GB → 18.8 GB**
peak heap. Hitting 220mb/s loading speed

<!-- NEXT_JS_LLM_PR -->
parent 65340b27
//! A push-only vector that grows in fixed-size chunks instead of one
//! contiguous reallocating buffer.
//!
//! The trade-off vs. `Vec`:
//! - One pointer indirection per indexed access (chunk lookup → element).
//! - Slightly larger per-element overhead from chunk pointers (negligible at 64K elements/chunk).
//! - References returned by `index`/`index_mut` are stable across `push` (a future-useful property;
//! not currently relied on).
//!
//! API is intentionally minimal — only the operations the trace server
//! needs (`push`, `len`, indexed access, `get`, `truncate`).
use std::{
mem::MaybeUninit,
ops::{Index, IndexMut},
};
/// Number of elements per chunk. Power of two so `idx / CHUNK_SIZE` and
/// `idx % CHUNK_SIZE` compile to a shift and a mask.
const CHUNK_SIZE: usize = 1 << 16;
/// Returns the chunk index and intra-chunk offset for an element index.
#[inline]
fn split_index(idx: usize) -> (usize, usize) {
(idx / CHUNK_SIZE, idx % CHUNK_SIZE)
}
type Chunk<T> = Box<[MaybeUninit<T>; CHUNK_SIZE]>;
/// Allocate a fresh chunk on the heap without ever materializing a
/// `CHUNK_SIZE`-element array on the stack.
fn new_chunk<T>() -> Chunk<T> {
// SAFETY: `Box<MaybeUninit<[MaybeUninit<T>; N]>>` and
// `Box<[MaybeUninit<T>; N]>` have identical layout; the outer
// `MaybeUninit` is just deferring initialization of the array of
// uninitialized slots, which trivially satisfies "init".
unsafe {
let raw: Box<MaybeUninit<[MaybeUninit<T>; CHUNK_SIZE]>> = Box::new_uninit();
raw.assume_init()
}
}
pub struct ChunkedVec<T> {
chunks: Vec<Chunk<T>>,
len: usize,
}
impl<T> ChunkedVec<T> {
pub fn new() -> Self {
Self {
chunks: Vec::new(),
len: 0,
}
}
pub fn len(&self) -> usize {
self.len
}
/// Append an element. Returns the index it was placed at.
pub fn push(&mut self, value: T) -> usize {
let idx = self.len;
let (chunk_idx, off) = split_index(idx);
if off == 0 {
// Crossing into a new chunk — allocate it.
debug_assert_eq!(chunk_idx, self.chunks.len());
self.chunks.push(new_chunk());
}
self.chunks[chunk_idx][off].write(value);
self.len += 1;
idx
}
pub fn get(&self, idx: usize) -> Option<&T> {
if idx >= self.len {
return None;
}
let (chunk_idx, off) = split_index(idx);
// SAFETY: `idx < self.len` ⇒ slot was previously written by
// `push` and not freed by `truncate`.
Some(unsafe { self.chunks[chunk_idx][off].assume_init_ref() })
}
}
impl<T> Default for ChunkedVec<T> {
fn default() -> Self {
Self::new()
}
}
impl<T> Drop for ChunkedVec<T> {
fn drop(&mut self) {
// Drop every initialized slot in [new_len, old_len). Walk the
// chunks one by one so we visit each `MaybeUninit<T>` exactly
// once.
let (last_chunk, last_chunk_len) = split_index(self.len);
for (chunk_index, chunk) in self.chunks.iter_mut().enumerate() {
let chunk_end = if chunk_index == last_chunk {
last_chunk_len
} else {
CHUNK_SIZE
};
for slot in &mut chunk[0..chunk_end] {
// SAFETY: the slot was initialized by a prior `push`
// and has not yet been dropped by `truncate`.
unsafe { slot.assume_init_drop() };
}
}
}
}
impl<T> Index<usize> for ChunkedVec<T> {
type Output = T;
#[inline]
fn index(&self, idx: usize) -> &T {
assert!(idx < self.len, "index out of bounds: {idx} >= {}", self.len);
let (chunk_idx, off) = split_index(idx);
// SAFETY: `idx < self.len` ⇒ slot is initialized.
unsafe { self.chunks[chunk_idx][off].assume_init_ref() }
}
}
impl<T> IndexMut<usize> for ChunkedVec<T> {
#[inline]
fn index_mut(&mut self, idx: usize) -> &mut T {
assert!(idx < self.len, "index out of bounds: {idx} >= {}", self.len);
let (chunk_idx, off) = split_index(idx);
// SAFETY: `idx < self.len` ⇒ slot is initialized.
unsafe { self.chunks[chunk_idx][off].assume_init_mut() }
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn empty() {
let v: ChunkedVec<u32> = ChunkedVec::new();
assert_eq!(v.len(), 0);
assert!(v.get(0).is_none());
}
#[test]
fn push_within_first_chunk() {
let mut v = ChunkedVec::new();
for i in 0..1000u32 {
assert_eq!(v.push(i), i as usize);
}
assert_eq!(v.len(), 1000);
assert_eq!(v[0], 0);
assert_eq!(v[999], 999);
assert_eq!(v.get(1000), None);
}
#[test]
fn push_across_chunk_boundary() {
let mut v = ChunkedVec::new();
// Push enough to span three chunks.
let total = 3 * CHUNK_SIZE + 17;
for i in 0..total {
v.push(i);
}
assert_eq!(v.len(), total);
assert_eq!(v[0], 0);
assert_eq!(v[CHUNK_SIZE - 1], CHUNK_SIZE - 1);
assert_eq!(v[CHUNK_SIZE], CHUNK_SIZE);
assert_eq!(v[2 * CHUNK_SIZE], 2 * CHUNK_SIZE);
assert_eq!(v[total - 1], total - 1);
}
#[test]
fn index_mut_writes_through() {
let mut v = ChunkedVec::new();
for i in 0..(CHUNK_SIZE + 5) {
v.push(i);
}
v[CHUNK_SIZE + 3] = 9999;
assert_eq!(v[CHUNK_SIZE + 3], 9999);
}
#[test]
fn drops_elements_on_drop() {
use std::rc::Rc;
let counter = Rc::new(());
{
let mut v: ChunkedVec<Rc<()>> = ChunkedVec::new();
// Span multiple chunks so we exercise the multi-chunk drop path.
let total = 2 * CHUNK_SIZE + 5;
for _ in 0..total {
v.push(counter.clone());
}
assert_eq!(Rc::strong_count(&counter), total + 1);
} // ChunkedVec drop runs here, dropping the remaining CHUNK_SIZE clones.
assert_eq!(Rc::strong_count(&counter), 1);
}
}
...@@ -17,6 +17,7 @@ use self::{ ...@@ -17,6 +17,7 @@ use self::{
}; };
mod bottom_up; mod bottom_up;
mod chunked_vec;
mod lazy_sorted_vec; mod lazy_sorted_vec;
mod reader; mod reader;
mod self_time_tree; mod self_time_tree;
......
...@@ -12,6 +12,7 @@ use rustc_hash::FxHasher; ...@@ -12,6 +12,7 @@ use rustc_hash::FxHasher;
use self::{reader::TraceReader, server::serve, store_container::StoreContainer}; use self::{reader::TraceReader, server::serve, store_container::StoreContainer};
mod bottom_up; mod bottom_up;
mod chunked_vec;
mod lazy_sorted_vec; mod lazy_sorted_vec;
mod reader; mod reader;
mod self_time_tree; mod self_time_tree;
......
...@@ -9,6 +9,7 @@ use rustc_hash::FxHashSet; ...@@ -9,6 +9,7 @@ use rustc_hash::FxHashSet;
use turbo_rcstr::{RcStr, rcstr}; use turbo_rcstr::{RcStr, rcstr};
use crate::{ use crate::{
chunked_vec::ChunkedVec,
self_time_tree::SelfTimeTree, self_time_tree::SelfTimeTree,
span::{Span, SpanArgs, SpanEvent, SpanIndex, SpanTimeData}, span::{Span, SpanArgs, SpanEvent, SpanIndex, SpanTimeData},
span_ref::SpanRef, span_ref::SpanRef,
...@@ -33,7 +34,7 @@ type MemorySample = (Timestamp, u64, u8); ...@@ -33,7 +34,7 @@ type MemorySample = (Timestamp, u64, u8);
const MAX_MEMORY_SAMPLES: usize = 200; const MAX_MEMORY_SAMPLES: usize = 200;
pub struct Store { pub struct Store {
pub(crate) spans: Vec<Span>, pub(crate) spans: ChunkedVec<Span>,
pub(crate) self_time_tree: Option<SelfTimeTree<SpanIndex>>, pub(crate) self_time_tree: Option<SelfTimeTree<SpanIndex>>,
max_self_time_lookup_time: AtomicU64, max_self_time_lookup_time: AtomicU64,
/// Global sorted list of memory samples (timestamp, memory_bytes). /// Global sorted list of memory samples (timestamp, memory_bytes).
...@@ -66,12 +67,10 @@ fn new_root_span() -> Span { ...@@ -66,12 +67,10 @@ fn new_root_span() -> Span {
impl Store { impl Store {
pub fn new() -> Self { pub fn new() -> Self {
let mut spans = ChunkedVec::new();
spans.push(new_root_span());
Self { Self {
spans: { spans,
let mut v = Vec::with_capacity(131_072);
v.push(new_root_span());
v
},
self_time_tree: env::var("NO_CORRECTED_TIME") self_time_tree: env::var("NO_CORRECTED_TIME")
.ok() .ok()
.is_none() .is_none()
...@@ -82,8 +81,8 @@ impl Store { ...@@ -82,8 +81,8 @@ impl Store {
} }
pub fn reset(&mut self) { pub fn reset(&mut self) {
self.spans.truncate(1); self.spans = ChunkedVec::new();
self.spans[0] = new_root_span(); self.spans.push(new_root_span());
if let Some(tree) = self.self_time_tree.as_mut() { if let Some(tree) = self.self_time_tree.as_mut() {
*tree = SelfTimeTree::new(); *tree = SelfTimeTree::new();
} }
...@@ -163,7 +162,7 @@ impl Store { ...@@ -163,7 +162,7 @@ impl Store {
pub fn add_args( pub fn add_args(
&mut self, &mut self,
span_index: SpanIndex, span_index: SpanIndex,
args: Vec<(RcStr, RcStr)>, args: SpanArgs,
outdated_spans: &mut FxHashSet<SpanIndex>, outdated_spans: &mut FxHashSet<SpanIndex>,
) { ) {
let span = &mut self.spans[span_index.get()]; let span = &mut self.spans[span_index.get()];
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment