Unverified Commit deb98d7e authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

turbo-trace-server: speed up loading compressed traces (#99775)

### What?

Makes loading compressed (`.zst` / `.gz`) traces in `turbo-trace-server`
much faster. On a large trace, compressed loads now beat raw loads from
before this change. Raw loads get faster too.

| 12M-span synthetic trace (warm cache, median) | before | after |
|---|---|---|
| raw | ~17s | ~13.5s |
| `.zst` | ~26s | ~14s |
| `.gz` | ~26s | ~12s |

Smaller synthetic traces (2–3M spans) improve as well:
- raw goes from 3.5–4.9s to 2.2–3.3s;
- compressed goes from 2.2–3.7s to 2.1–2.9s.

### Why?

Opening compressed traces was very slow. There were two causes.

1. **Per-batch work was quadratic.**
- After every batch of rows, `Store::invalidate_outdated_spans` cleared
the cached corrected self time of **every** self-time event of each
touched span, plus its ancestors and the root.
   - For a span with millions of events, that is a full scan per batch.
2. **Compressed input arrived in tiny batches.**
- zstd/gzip decoders return about one input buffer of output per `read`,
roughly 450 KB.
   - Raw files arrived in 64 MB reads.
- So compressed traces ran that full scan about 100× more often. On the
trace above, invalidation alone took ~12.7s for `.zst` versus ~1.5s for
raw.

### How?

**Constant-time cache invalidation.**
- Each span now has a generation counter for the cached corrected self
times of its self-time events.
- Invalidating a span just bumps the counter. A cached per-event value
only counts if it was computed for the span's current generation.
- The value and its generation are stored as two atomics in the event,
replacing the `OnceLock`, so `SpanEvent` stays 32 bytes. `Span` grows by
8 bytes.
- The counter only changes under the store's write lock. Concurrent
readers can at worst both compute and store the same value.
- Invalidation semantics are unchanged: the touched span, its ancestors
and the root. The new store tests pass against both the old and the new
implementation.

**Decompression on a background thread** (`reader/threaded.rs`).
- zstd and gzip streams are decoded on a dedicated thread. It fills
batches and sends them to the reader over a bounded channel, so
decompression overlaps parsing.
- Buffers are recycled.
- Errors are sent after the data that came before them, so an error the
decoder reports only once is never lost. gzip checksum errors are an
example.
- After EOF or an error, the thread pauses until the reader asks for
more data. This keeps the existing "wait for the file to grow, then
retry" behavior.
- The reader sees a panic on the decoder thread as an error, not as EOF.
- Dropping the decoder closes the channels and joins the thread, both
when the thread is paused and when it is blocked on a full channel.
- For progress reporting, the reported position is the compressed-file
offset the decoder thread has consumed. It never decreases and can run
ahead of parsing by the queued batches.
- STOP_AT polling now reads the file size from a cloned file handle
instead of seeking to the end.

**1 MB batches for all formats.** Once per-batch invalidation is cheap,
small batches are faster: 1 MB batches parse about 1.3–1.6× faster than
64 MB ones, for raw and compressed input alike.

### Testing

`cargo test -p turbopack-trace-server` passes, including new tests for:
- **Store:**
- cached values are reused until a span is invalidated, then recomputed;
  - invalidation reaches ancestors and the root;
  - concurrent corrected-self-time reads match serial results.
- **Threaded decoder:**
  - gzip and zstd streams round-trip exactly;
- errors arrive after the preceding data (one-shot errors and a real
corrupt gzip CRC);
  - truncated streams deliver their readable prefix;
  - `Interrupted` reads are retried, and reading resumes after EOF;
  - a worker panic surfaces as an error;
- dropping the decoder shuts the thread down while it is paused or
backpressured;
  - the reported position is monotonic.
- **Reader:**
  - raw, zstd and gzip versions of a trace load identically;
  - raw live tailing picks up appended data;
  - truncated compressed files load their prefix.

Checked manually against the previous binary on synthetic traces:
- span counts are identical for every format;
- truncated and corrupt-checksum files load the same number of spans;
- STOP_AT still works.

Known limitations, unchanged by this PR:
- live tailing of compressed traces doesn't pick up appended data;
- gzip checksum errors are still treated like EOF without a warning.

<!-- NEXT_JS_LLM -->


<!-- fleet 7727f609-8d7f-49aa-8f67-2aefbafe3f2e -->
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent 3439bde9
......@@ -32,10 +32,6 @@ impl<T> LazySortedVec<T> {
pub fn retain_unordered(&mut self, mut f: impl FnMut(&T) -> bool) {
self.vec.get_mut().retain(|t| f(t));
}
pub fn iter_mut_unordered(&mut self) -> std::slice::IterMut<'_, T> {
self.vec.get_mut().iter_mut()
}
}
impl<T: Ord> Deref for LazySortedVec<T> {
......
This diff is collapsed.
use std::{
num::{NonZeroU64, NonZeroUsize},
sync::{Arc, OnceLock},
sync::{
Arc, OnceLock,
atomic::{AtomicU64, Ordering},
},
};
use hashbrown::HashMap;
......@@ -79,6 +82,10 @@ pub struct SpanTimeData {
pub total_time: OnceLock<Timestamp>,
pub corrected_self_time: OnceLock<Timestamp>,
pub corrected_total_time: OnceLock<Timestamp>,
/// Generation of the cached `corrected_self_time` values of this span's self-time events.
/// Bumping it invalidates all of them in O(1) instead of iterating the (possibly huge)
/// event list; see [`SpanEventSelfTime::cached_corrected_self_time`].
pub self_time_events_generation: u64,
}
#[derive(Default)]
......@@ -180,10 +187,31 @@ impl Span {
pub struct SpanEventSelfTime {
pub start: Timestamp,
pub duration: NonZeroU64,
pub corrected_self_time: OnceLock<Timestamp>,
/// Cached corrected self time. Only valid if `corrected_generation` equals the owning
/// span's `self_time_events_generation + 1`; `0` means it was never computed.
corrected_self_time: AtomicU64,
corrected_generation: AtomicU64,
}
impl SpanEventSelfTime {
/// Returns the cached corrected self time if it was computed for `span_generation` (the
/// owning span's current `self_time_events_generation`).
pub fn cached_corrected_self_time(&self, span_generation: u64) -> Option<Timestamp> {
// Acquire pairs with the Release in `set_corrected_self_time`, so the value read below
// is the one stored for this generation.
(self.corrected_generation.load(Ordering::Acquire) == span_generation + 1)
.then(|| Timestamp::from_value(self.corrected_self_time.load(Ordering::Relaxed)))
}
/// Caches `value` as the corrected self time for `span_generation`. Concurrent readers may
/// race to compute and store the same value; that is benign since the generation can only
/// change while the store is write-locked.
pub fn set_corrected_self_time(&self, span_generation: u64, value: Timestamp) {
self.corrected_self_time.store(*value, Ordering::Relaxed);
self.corrected_generation
.store(span_generation + 1, Ordering::Release);
}
pub fn end(&self) -> Timestamp {
Timestamp::from_value(*self.start + self.duration.get())
}
......@@ -194,8 +222,8 @@ pub enum SpanEvent {
Child { start: Timestamp, index: SpanIndex },
}
// 32 bytes = 8 (start) + 8 (duration) + 16 (OnceLock<Timestamp>) for the
// SelfTime variant; the Child variant fits in 16 and uses the niche, so no
// 32 bytes = 8 (start) + 8 (duration) + 8 (cached corrected self time) + 8 (cache generation)
// for the SelfTime variant; the Child variant fits in 16 and uses the niche, so no
// extra discriminant byte is needed.
const _: () = assert!(std::mem::size_of::<SpanEvent>() == 32);
......@@ -207,7 +235,8 @@ impl SpanEvent {
Some(SpanEvent::SelfTime(SpanEventSelfTime {
start,
duration,
corrected_self_time: OnceLock::new(),
corrected_self_time: AtomicU64::new(0),
corrected_generation: AtomicU64::new(0),
}))
}
......
......@@ -138,6 +138,7 @@ impl<'a> SpanRef<'a> {
self_time: SpanEventSelfTimeRef {
store: self.store,
self_time,
generation: self.span.time_data.self_time_events_generation,
},
},
SpanEvent::Child { index, .. } => SpanEventRef::Child {
......@@ -234,6 +235,7 @@ impl<'a> SpanRef<'a> {
pub fn corrected_self_time(&self) -> Timestamp {
let store = self.store;
let generation = self.span.time_data.self_time_events_generation;
*self.time_data().corrected_self_time.get_or_init(|| {
let mut self_time = self
.span
......@@ -242,7 +244,12 @@ impl<'a> SpanRef<'a> {
.filter_map(|event: &'a SpanEvent| {
if let SpanEvent::SelfTime(self_time) = event {
return Some(
SpanEventSelfTimeRef { store, self_time }.corrected_self_time(),
SpanEventSelfTimeRef {
store,
self_time,
generation,
}
.corrected_self_time(),
);
}
None
......@@ -504,6 +511,8 @@ impl Debug for SpanRef<'_> {
pub struct SpanEventSelfTimeRef<'a> {
store: &'a Store,
self_time: &'a SpanEventSelfTime,
/// `self_time_events_generation` of the owning span.
generation: u64,
}
impl<'a> SpanEventSelfTimeRef<'a> {
......@@ -516,16 +525,20 @@ impl<'a> SpanEventSelfTimeRef<'a> {
}
pub fn corrected_self_time(&self) -> Timestamp {
*self.self_time.corrected_self_time.get_or_init(|| {
// `duration` is `NonZeroU64`, so zero-duration events are filtered
// at construction time (see `SpanEvent::self_time`).
let end = self.self_time.end();
let duration = Timestamp::from_value(self.self_time.duration.get());
self.store.set_max_self_time_lookup(end);
self.store.self_time_tree.as_ref().map_or(duration, |tree| {
tree.lookup_range_corrected_time(self.self_time.start, end)
})
})
if let Some(value) = self.self_time.cached_corrected_self_time(self.generation) {
return value;
}
// `duration` is `NonZeroU64`, so zero-duration events are filtered
// at construction time (see `SpanEvent::self_time`).
let end = self.self_time.end();
let duration = Timestamp::from_value(self.self_time.duration.get());
self.store.set_max_self_time_lookup(end);
let value = self.store.self_time_tree.as_ref().map_or(duration, |tree| {
tree.lookup_range_corrected_time(self.self_time.start, end)
});
self.self_time
.set_corrected_self_time(self.generation, value);
value
}
}
......
......@@ -482,11 +482,10 @@ impl Store {
span.time_data.total_time.take();
span.time_data.corrected_self_time.take();
span.time_data.corrected_total_time.take();
for event in span.events.iter_mut_unordered() {
if let SpanEvent::SelfTime(self_time) = event {
self_time.corrected_self_time.take();
}
}
// Invalidates the cached corrected self time of all self-time events of this span
// in O(1). Iterating the events here made loading quadratic for spans with many
// events, as this runs for every batch read from the trace file.
span.time_data.self_time_events_generation += 1;
span.totals.take();
span.extra.take();
}
......@@ -546,7 +545,10 @@ impl Store {
#[cfg(test)]
mod tests {
use std::thread;
use super::*;
use crate::span_ref::SpanEventRef;
#[test]
fn blocking_total_time_preserves_ranges_and_counted_children() {
......@@ -645,4 +647,174 @@ mod tests {
.is_empty()
);
}
fn ts(micros: u64) -> Timestamp {
Timestamp::from_micros(micros)
}
fn add_span(store: &mut Store, parent: Option<SpanIndex>, start: u64) -> SpanIndex {
store.add_span(
parent,
ts(start),
RcStr::default(),
RcStr::from("span"),
SpanArgs::new(),
&mut FxHashSet::default(),
)
}
/// Adds a self time and drops the resulting outdated spans, so tests control exactly
/// which spans get invalidated.
fn add_self_time_without_invalidation(
store: &mut Store,
span: SpanIndex,
start: u64,
end: u64,
) {
store.add_self_time(span, ts(start), ts(end), &mut FxHashSet::default());
}
fn invalidate(store: &mut Store, spans: &[SpanIndex]) {
store.invalidate_outdated_spans(&spans.iter().copied().collect());
}
/// Corrected self times of the self-time events of `span`.
fn event_corrected_self_times(store: &Store, span: SpanIndex) -> Vec<Timestamp> {
corrected_self_times_of(SpanRef {
span: &store.spans[span.get()],
store,
index: span.get(),
})
}
fn corrected_self_times_of(span: SpanRef<'_>) -> Vec<Timestamp> {
span.events()
.filter_map(|event| match event {
SpanEventRef::SelfTime { self_time } => Some(self_time.corrected_self_time()),
SpanEventRef::Child { .. } => None,
})
.collect()
}
#[test]
fn corrected_self_time_cached_until_invalidated() {
let mut store = Store::new();
let a = add_span(&mut store, None, 0);
add_self_time_without_invalidation(&mut store, a, 0, 10);
invalidate(&mut store, &[a]);
assert_eq!(event_corrected_self_times(&store, a), vec![ts(10)]);
// A concurrent self time halves the corrected self time of `a`...
let b = add_span(&mut store, None, 0);
add_self_time_without_invalidation(&mut store, b, 0, 10);
invalidate(&mut store, &[b]);
assert_eq!(event_corrected_self_times(&store, b), vec![ts(5)]);
// ...but `a` was not invalidated, so its cached value is still used.
assert_eq!(event_corrected_self_times(&store, a), vec![ts(10)]);
invalidate(&mut store, &[a]);
assert_eq!(event_corrected_self_times(&store, a), vec![ts(5)]);
}
#[test]
fn invalidation_propagates_to_ancestors() {
let mut store = Store::new();
let grandparent = add_span(&mut store, None, 0);
let parent = add_span(&mut store, Some(grandparent), 0);
let child = add_span(&mut store, Some(parent), 0);
add_self_time_without_invalidation(&mut store, grandparent, 0, 10);
add_self_time_without_invalidation(&mut store, parent, 10, 20);
add_self_time_without_invalidation(&mut store, child, 20, 30);
invalidate(&mut store, &[grandparent, parent, child]);
for span in [grandparent, parent, child] {
assert_eq!(event_corrected_self_times(&store, span), vec![ts(10)]);
}
// Concurrent self time in an unrelated span, overlapping all three self times.
let other = add_span(&mut store, None, 0);
add_self_time_without_invalidation(&mut store, other, 0, 30);
for span in [grandparent, parent, child] {
assert_eq!(event_corrected_self_times(&store, span), vec![ts(10)]);
}
// Invalidating only the child invalidates its ancestors too.
invalidate(&mut store, &[child]);
for span in [grandparent, parent, child] {
assert_eq!(event_corrected_self_times(&store, span), vec![ts(5)]);
}
}
#[test]
fn invalidation_always_includes_root() {
let mut store = Store::new();
// The public API never adds self time to the root span (its index is 0), but its
// self-time event caches must still be invalidated like any other span's.
store.spans[0]
.events
.push(SpanEvent::self_time(ts(0), ts(10)).unwrap());
let a = add_span(&mut store, None, 0);
add_self_time_without_invalidation(&mut store, a, 0, 10);
invalidate(&mut store, &[a]);
assert_eq!(corrected_self_times_of(store.root_span()), vec![ts(10)]);
// A second concurrent self time halves the corrected time of the root's event, which
// is only picked up once the root is invalidated.
let b = add_span(&mut store, None, 0);
add_self_time_without_invalidation(&mut store, b, 0, 10);
assert_eq!(corrected_self_times_of(store.root_span()), vec![ts(10)]);
// Invalidating any span invalidates the root.
let child = add_span(&mut store, Some(a), 20);
invalidate(&mut store, &[child]);
assert_eq!(corrected_self_times_of(store.root_span()), vec![ts(5)]);
}
/// Builds a store with many overlapping self times.
fn overlapping_store() -> (Store, Vec<SpanIndex>) {
let mut store = Store::new();
let mut spans = Vec::new();
let mut outdated = FxHashSet::default();
for i in 0..200u64 {
let parent = spans.get((i / 10) as usize).copied();
let span = add_span(&mut store, parent, i);
for j in 0..5 {
let start = (i * 7 + j * 13) % 500;
store.add_self_time(span, ts(start), ts(start + 20 + i % 30), &mut outdated);
}
spans.push(span);
}
store.invalidate_outdated_spans(&outdated);
(store, spans)
}
#[test]
fn concurrent_corrected_self_time_matches_serial() {
let (serial_store, spans) = overlapping_store();
let serial: Vec<_> = spans
.iter()
.map(|&span| event_corrected_self_times(&serial_store, span))
.collect();
let (store, spans) = overlapping_store();
thread::scope(|scope| {
let handles: Vec<_> = (0..8)
.map(|offset| {
let store = &store;
let spans = &spans;
scope.spawn(move || {
// Each thread visits the spans in a different order.
let mut result = vec![Vec::new(); spans.len()];
for i in 0..spans.len() {
let i = (i + offset * 25) % spans.len();
result[i] = event_corrected_self_times(store, spans[i]);
}
result
})
})
.collect();
for handle in handles {
assert_eq!(handle.join().unwrap(), serial);
}
});
}
}
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment