Unverified Commit 3cdbd0f2 authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

Turbopack: delta-encode trace timestamps (#99809)

## Summary

Delta-encodes all timestamps in Turbopack trace files. Absolute
microsecond timestamps take 4–5 bytes as varints. The difference to the
previous row of the same thread usually fits in 1–2 bytes.

- **Format:** every row timestamp (`Start`, `End`, `Enter`, `Exit`,
`Event`, `MemorySample`) is now the difference to the previous timestamp
in the same thread local buffer. The serialized `ts` fields are
`DeltaEncodedTimestamp`, an `i64` newtype that postcard zigzag-encodes,
so a clock that goes backwards still works. `TraceRow` is generic over
its timestamp type: `TraceRow<'a>` is the serialized form, and decoding
returns `TraceRow<'a, u64>` with absolute timestamps, so the types show
whether a row still needs decoding.
- **New row:** `TimestampBase { ts }` carries an absolute timestamp. The
writer puts one before the first row with a timestamp in every buffer.
Buffers are flushed independently and interleave in the file, so each
one can be decoded on its own.
- **Large jumps:** a new base is also written when a difference doesn't
fit in an `i64`.
- **Writer:** the encoder state lives in the buffer and is reset when a
buffer is sent, stolen by the writer thread or reused. Removing the last
row (#99780) also removes its base and restores the encoder state.
- **Readers:** the trace server, `turbo-trace-size` and the tests share
`TimestampDecoder`, which keeps its state across incremental reads. A
timestamp before any base is an error. `turbo-trace-size` reports the
error with its offset.

The header is bumped to `TRACEv1` in #99781, at the top of this
format-changing stack.

Stack (on top of #99765):

1. #99777: `no-memory` flag
2. #99778: Merge allocation counters into Enter/Exit rows
3. #99779: Delta-encode allocation counters, skip unchanged ones
4. #99780: Omit Exit+Enter pairs with (almost) no gap
5. #99808: zstd flags
6. #99809: **This PR**: delta-encoded timestamps
7. #99781: Bump the trace header to `TRACEv1`
8. #99810: `upload-trace` accepts compressed traces

## Measurements

`next build` of `bench/heavy-npm-deps`, release bindings built from the
previous PR and from this one. Medians of 3 rounds that alternate
between the two builds:

| mode | before | after | change |
| --- | --- | --- | --- |
| `1` | 23.94 MB | 21.03 MB | −12.2% |
| `1,zstd` | 6.15 MB | 4.72 MB | **−23.2%** |
| `turbo-tasks` | 155.97 MB | 145.28 MB | −6.9% |
| `turbo-tasks,zstd` | 28.14 MB | 22.13 MB | **−21.4%** |

- **Row sizes:** Enter, Exit and End rows get about 2.2 bytes smaller
each.
- **Overhead:** `TimestampBase` rows add 137 B (35 rows) to the `1`
trace and 896 B (214 rows) to the `turbo-tasks` trace.

## Verification

- **Tests:**
- encoder and decoder round trips, including large jumps and incremental
reads;
- buffers that are sent, stolen or reused, and a buffer that starts with
a `Record` (its base comes before the first row with a timestamp);
  - removing the first or last row of a buffer;
  - exact timestamps across 2 threads and many buffers.
- **Captures:** the release-build captures are decoded fully by
`turbo-trace-size` and load completely in the trace server, raw and
zstd. Span counts per name match the previous PR.

<!-- NEXT_JS_LLM -->


<!-- fleet 4a52842b-f1a2-43cb-9779-3655caada6a0 -->

---------
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent f7b9651d
...@@ -88,5 +88,9 @@ cargo run --bin turbo-trace-size --release -- /path/to/your/trace-turbopack.bin ...@@ -88,5 +88,9 @@ cargo run --bin turbo-trace-size --release -- /path/to/your/trace-turbopack.bin
To keep trace files small, an `Exit` of a span that is followed by an `Enter` of the same span on the same thread shortly after is omitted together with that `Enter`. That's common for async spans, which are exited and entered again on every poll. This only happens when nothing was (de)allocated in between and the gap is at most 1µs or 0.1% of the time since the last `Enter` row written on that thread (omitted `Enter` rows don't count; for nested spans, it can be later than the span's own `Enter`, which only makes the allowed gap smaller). The viewer shows the span as entered during these gaps, so its self time can be slightly too large. To keep trace files small, an `Exit` of a span that is followed by an `Enter` of the same span on the same thread shortly after is omitted together with that `Enter`. That's common for async spans, which are exited and entered again on every poll. This only happens when nothing was (de)allocated in between and the gap is at most 1µs or 0.1% of the time since the last `Enter` row written on that thread (omitted `Enter` rows don't count; for nested spans, it can be later than the span's own `Enter`, which only makes the allowed gap smaller). The viewer shows the span as entered during these gaps, so its self time can be slightly too large.
### Timestamps
Timestamps are stored as the (signed) difference to the previous timestamp in the trace file, which needs fewer bytes than absolute timestamps and compresses better. The trace is written in chunks per thread, and in every chunk a `TimestampBase` row with an absolute timestamp comes before the first row with a timestamp, so every chunk can be decoded on its own.
[turbo-trace-viewer]: https://turbo-trace-viewer.vercel.app/ [turbo-trace-viewer]: https://turbo-trace-viewer.vercel.app/
[youtube-tutorial]: https://www.youtube.com/watch?v=PGO2szAye7A [youtube-tutorial]: https://www.youtube.com/watch?v=PGO2szAye7A
use std::sync::Arc; use std::sync::Arc;
use turbopack_trace_utils::tracing::{TraceRow, TraceValue}; use turbopack_trace_utils::tracing::{TimestampEncoder, TraceRow, TraceValue};
use crate::{ use crate::{
QueryOptions, SortMode, query_spans, QueryOptions, SortMode, query_spans,
...@@ -43,16 +43,7 @@ fn reads_active_workers_and_exposes_them_in_span_queries() { ...@@ -43,16 +43,7 @@ fn reads_active_workers_and_exposes_them_in_span_queries() {
}, },
TraceRow::End { ts: 30, id: 1 }, TraceRow::End { ts: 30, id: 1 },
]; ];
let mut bytes = Vec::new(); ingest(&mut format, &rows);
for row in rows {
bytes.extend(postcard::to_stdvec(&row).unwrap());
}
assert_eq!(
format
.read(&bytes, &mut TurbopackFormat::create_reused())
.unwrap(),
bytes.len()
);
let samples = store let samples = store
.read() .read()
...@@ -84,11 +75,21 @@ fn reads_active_workers_and_exposes_them_in_span_queries() { ...@@ -84,11 +75,21 @@ fn reads_active_workers_and_exposes_them_in_span_queries() {
} }
} }
fn ingest(format: &mut TurbopackFormat, rows: &[TraceRow<'_>]) { /// Serializes the rows like the trace writer does for one buffer (with delta encoded timestamps
let bytes: Vec<_> = rows /// and a [`TraceRow::TimestampBase`] before the first timestamp) and reads them.
.iter() fn ingest(format: &mut TurbopackFormat, rows: &[TraceRow<'_, u64>]) {
.flat_map(|row| postcard::to_stdvec(row).unwrap()) let mut encoder = TimestampEncoder::default();
.collect(); let mut bytes = Vec::new();
for row in rows {
// A copy of the row, through its serialized form
let row_bytes = postcard::to_stdvec(row).unwrap();
let row: TraceRow<'_, u64> = postcard::from_bytes(&row_bytes).unwrap();
let (base, row) = encoder.encode_row(row);
if let Some(base) = base {
bytes.extend(postcard::to_stdvec(&base).unwrap());
}
bytes.extend(postcard::to_stdvec(&row).unwrap());
}
assert_eq!( assert_eq!(
format format
.read(&bytes, &mut TurbopackFormat::create_reused()) .read(&bytes, &mut TurbopackFormat::create_reused())
...@@ -103,7 +104,7 @@ fn start( ...@@ -103,7 +104,7 @@ fn start(
ts: u64, ts: u64,
name: &'static str, name: &'static str,
blocking: Option<bool>, blocking: Option<bool>,
) -> TraceRow<'static> { ) -> TraceRow<'static, u64> {
TraceRow::Start { TraceRow::Start {
ts, ts,
id, id,
...@@ -121,7 +122,7 @@ fn duration_event( ...@@ -121,7 +122,7 @@ fn duration_event(
duration: u64, duration: u64,
name: &'static str, name: &'static str,
blocking: Option<bool>, blocking: Option<bool>,
) -> TraceRow<'static> { ) -> TraceRow<'static, u64> {
let mut values = vec![ let mut values = vec![
("name".into(), TraceValue::String(name.into())), ("name".into(), TraceValue::String(name.into())),
("duration".into(), TraceValue::UInt(duration)), ("duration".into(), TraceValue::UInt(duration)),
......
...@@ -419,19 +419,29 @@ impl TraceReader { ...@@ -419,19 +419,29 @@ impl TraceReader {
mod tests { mod tests {
use std::{borrow::Cow, io::Write, time::Duration}; use std::{borrow::Cow, io::Write, time::Duration};
use turbopack_trace_utils::tracing::TraceRow; use turbopack_trace_utils::tracing::{TimestampEncoder, TraceRow};
use super::*; use super::*;
const TIMEOUT: Duration = Duration::from_secs(30); const TIMEOUT: Duration = Duration::from_secs(30);
/// Serialized rows for `count` child spans of a root span, plus the root span itself. /// Serialized rows for `count` child spans of a root span, plus the root span itself. Like a
/// buffer of the trace writer, the timestamps are delta encoded after a
/// [`TraceRow::TimestampBase`].
fn span_rows(first_id: u64, count: u64) -> Vec<u8> { fn span_rows(first_id: u64, count: u64) -> Vec<u8> {
let mut bytes = Vec::new(); let mut bytes = Vec::new();
let mut push = |row: TraceRow<'_>| bytes.extend(postcard::to_stdvec(&row).unwrap()); let mut encoder = TimestampEncoder::default();
let mut push = |row: TraceRow<'_, u64>| {
let (base, row) = encoder.encode_row(row);
if let Some(base) = base {
bytes.extend(postcard::to_stdvec(&base).unwrap());
}
bytes.extend(postcard::to_stdvec(&row).unwrap());
};
for id in first_id..first_id + count { for id in first_id..first_id + count {
let ts = id;
push(TraceRow::Start { push(TraceRow::Start {
ts: id, ts,
id, id,
parent: (id != 1).then_some(1), parent: (id != 1).then_some(1),
name: Cow::Borrowed("span"), name: Cow::Borrowed("span"),
...@@ -439,19 +449,19 @@ mod tests { ...@@ -439,19 +449,19 @@ mod tests {
values: Vec::new(), values: Vec::new(),
}); });
push(TraceRow::Enter { push(TraceRow::Enter {
ts: id, ts,
id, id,
thread_id: 1, thread_id: 1,
allocations: None, allocations: None,
}); });
push(TraceRow::Exit { push(TraceRow::Exit {
ts: id + 1, ts: ts + 1,
id, id,
thread_id: 1, thread_id: 1,
allocations: None, allocations: None,
}); });
if id != 1 { if id != 1 {
push(TraceRow::End { ts: id + 1, id }); push(TraceRow::End { ts: ts + 1, id });
} }
} }
bytes bytes
......
...@@ -8,7 +8,7 @@ use std::{ ...@@ -8,7 +8,7 @@ use std::{
use anyhow::Result; use anyhow::Result;
use rustc_hash::{FxHashMap, FxHashSet}; use rustc_hash::{FxHashMap, FxHashSet};
use turbo_rcstr::{RcStr, RcStrInterning, rcstr}; use turbo_rcstr::{RcStr, RcStrInterning, rcstr};
use turbopack_trace_utils::tracing::{Allocations, TraceRow, TraceValue}; use turbopack_trace_utils::tracing::{Allocations, TimestampDecoder, TraceRow, TraceValue};
use super::TraceFormat; use super::TraceFormat;
use crate::{ use crate::{
...@@ -69,6 +69,8 @@ pub struct TurbopackFormat { ...@@ -69,6 +69,8 @@ pub struct TurbopackFormat {
// Only the top frame has an active self-time start; suspended frames have None. // Only the top frame has an active self-time start; suspended frames have None.
thread_stacks: FxHashMap<u64, Vec<(u64, Option<Timestamp>)>>, thread_stacks: FxHashMap<u64, Vec<(u64, Option<Timestamp>)>>,
interner: RcStrInterning, interner: RcStrInterning,
/// Resolves the timestamps of the rows, which are relative to each other in the stream.
timestamps: TimestampDecoder,
} }
impl TurbopackFormat { impl TurbopackFormat {
...@@ -86,6 +88,7 @@ impl TurbopackFormat { ...@@ -86,6 +88,7 @@ impl TurbopackFormat {
outdated_spans: FxHashSet::with_capacity_and_hasher(8_192, Default::default()), outdated_spans: FxHashSet::with_capacity_and_hasher(8_192, Default::default()),
thread_stacks: FxHashMap::with_capacity_and_hasher(64, Default::default()), thread_stacks: FxHashMap::with_capacity_and_hasher(64, Default::default()),
interner: RcStrInterning::new(), interner: RcStrInterning::new(),
timestamps: TimestampDecoder::default(),
} }
} }
...@@ -103,7 +106,7 @@ impl TurbopackFormat { ...@@ -103,7 +106,7 @@ impl TurbopackFormat {
.collect() .collect()
} }
fn process(&mut self, store: &mut StoreWriteGuard, row: TraceRow<'_>) { fn process(&mut self, store: &mut StoreWriteGuard, row: TraceRow<'_, u64>) {
match row { match row {
TraceRow::Start { TraceRow::Start {
ts, ts,
...@@ -264,6 +267,8 @@ impl TurbopackFormat { ...@@ -264,6 +267,8 @@ impl TurbopackFormat {
active_worker_threads, active_worker_threads,
); );
} }
// Already applied by the timestamp decoder when the row was read
TraceRow::TimestampBase { .. } => {}
} }
} }
...@@ -412,9 +417,9 @@ impl TurbopackFormat { ...@@ -412,9 +417,9 @@ impl TurbopackFormat {
} }
impl TraceFormat for TurbopackFormat { impl TraceFormat for TurbopackFormat {
type Reused = Vec<TraceRow<'static>>; type Reused = Vec<TraceRow<'static, u64>>;
fn create_reused() -> Vec<TraceRow<'static>> { fn create_reused() -> Vec<TraceRow<'static, u64>> {
// Pre-allocate for a typical batch size to avoid repeated doubling during initial read. // Pre-allocate for a typical batch size to avoid repeated doubling during initial read.
Vec::with_capacity(4_096) Vec::with_capacity(4_096)
} }
...@@ -444,15 +449,16 @@ impl TraceFormat for TurbopackFormat { ...@@ -444,15 +449,16 @@ impl TraceFormat for TurbopackFormat {
let mut reuse = ClearOnDrop(reuse); let mut reuse = ClearOnDrop(reuse);
// Safety: The Vec is empty and is cleared on leaving this scope, so it's safe to cast the // Safety: The Vec is empty and is cleared on leaving this scope, so it's safe to cast the
// lifetime of data, since there is no data and data can't leave this function. // lifetime of data, since there is no data and data can't leave this function.
let rows = let rows = unsafe {
unsafe { transmute::<&mut Vec<TraceRow<'_>>, &mut Vec<TraceRow<'_>>>(&mut *reuse) }; transmute::<&mut Vec<TraceRow<'_, u64>>, &mut Vec<TraceRow<'_, u64>>>(&mut *reuse)
};
let mut bytes_read = 0; let mut bytes_read = 0;
loop { loop {
match postcard::take_from_bytes(buffer) { match postcard::take_from_bytes::<TraceRow<'_>>(buffer) {
Ok((row, remaining)) => { Ok((row, remaining)) => {
bytes_read += buffer.len() - remaining.len(); bytes_read += buffer.len() - remaining.len();
buffer = remaining; buffer = remaining;
rows.push(row); rows.push(self.timestamps.decode(row)?);
} }
Err(err) => { Err(err) => {
if matches!(err, postcard::Error::DeserializeUnexpectedEnd) { if matches!(err, postcard::Error::DeserializeUnexpectedEnd) {
...@@ -504,7 +510,9 @@ impl<T> DerefMut for ClearOnDrop<'_, T> { ...@@ -504,7 +510,9 @@ impl<T> DerefMut for ClearOnDrop<'_, T> {
mod tests { mod tests {
use std::{borrow::Cow, sync::Arc}; use std::{borrow::Cow, sync::Arc};
use turbopack_trace_utils::tracing::{Allocations, TraceRow}; use turbopack_trace_utils::tracing::{
Allocations, DeltaEncodedTimestamp, TimestampEncoder, TraceRow,
};
use crate::{ use crate::{
reader::{TraceFormat, turbopack::TurbopackFormat}, reader::{TraceFormat, turbopack::TurbopackFormat},
...@@ -513,7 +521,7 @@ mod tests { ...@@ -513,7 +521,7 @@ mod tests {
timestamp::Timestamp, timestamp::Timestamp,
}; };
fn start(id: u64, parent: Option<u64>, name: &'static str) -> TraceRow<'static> { fn start(id: u64, parent: Option<u64>, name: &'static str) -> TraceRow<'static, u64> {
TraceRow::Start { TraceRow::Start {
ts: 0, ts: 0,
id, id,
...@@ -533,13 +541,28 @@ mod tests { ...@@ -533,13 +541,28 @@ mod tests {
}) })
} }
/// Reads the rows (without header) and returns the self allocations, allocation counts and /// Serializes the rows like the trace writer does: with delta encoded timestamps and a
/// deallocations of the spans with the given names. /// [`TraceRow::TimestampBase`] row before the first one.
fn read(rows: &[TraceRow<'_>], names: &[&str]) -> Vec<(u64, u64, u64)> { fn encode(rows: &[TraceRow<'_, u64>]) -> Vec<u8> {
let mut encoder = TimestampEncoder::default();
let mut data = Vec::new(); let mut data = Vec::new();
for row in rows { for row in rows {
data.extend(postcard::to_stdvec(row).unwrap()); // A copy of the row, through its serialized form
let bytes = postcard::to_stdvec(row).unwrap();
let row: TraceRow<'_, u64> = postcard::from_bytes(&bytes).unwrap();
let (base, row) = encoder.encode_row(row);
if let Some(base) = base {
data.extend(postcard::to_stdvec(&base).unwrap());
}
data.extend(postcard::to_stdvec(&row).unwrap());
} }
data
}
/// Reads the rows (without header) and returns the self allocations, allocation counts and
/// deallocations of the spans with the given names.
fn read(rows: &[TraceRow<'_, u64>], names: &[&str]) -> Vec<(u64, u64, u64)> {
let data = encode(rows);
let store = Arc::new(StoreContainer::new()); let store = Arc::new(StoreContainer::new());
let mut format = TurbopackFormat::new(store.clone()); let mut format = TurbopackFormat::new(store.clone());
let mut reuse = TurbopackFormat::create_reused(); let mut reuse = TurbopackFormat::create_reused();
...@@ -890,6 +913,67 @@ mod tests { ...@@ -890,6 +913,67 @@ mod tests {
assert_eq!(read(&rows, &["span"]), vec![(220, 22, 90)]); assert_eq!(read(&rows, &["span"]), vec![(220, 22, 90)]);
} }
#[test]
fn decodes_timestamps_across_incremental_reads() {
let rows: [TraceRow<'_, u64>; 4] = [
TraceRow::Start {
ts: 10,
id: 1,
parent: None,
name: Cow::Borrowed("split"),
target: Cow::Borrowed(""),
values: Vec::new(),
},
TraceRow::Enter {
ts: 15,
id: 1,
thread_id: 1,
allocations: None,
},
TraceRow::Exit {
ts: 25,
id: 1,
thread_id: 1,
allocations: None,
},
TraceRow::End { ts: 30, id: 1 },
];
let data = encode(&rows);
let base: TraceRow<'_> = TraceRow::TimestampBase { ts: 10 };
let base_len = postcard::to_stdvec(&base).unwrap().len();
let store = Arc::new(StoreContainer::new());
let mut format = TurbopackFormat::new(store.clone());
let mut reuse = TurbopackFormat::create_reused();
// The first read only contains the base row, the second read the rest
let read = TraceFormat::read(&mut format, &data[..base_len], &mut reuse).unwrap();
assert_eq!(read, base_len);
let read = TraceFormat::read(&mut format, &data[base_len..], &mut reuse).unwrap();
assert_eq!(read, data.len() - base_len);
let store = store.read();
let span = (0..store.spans.len())
.filter_map(|i| store.spans.get(i))
.find(|span| &*span.name == "split")
.unwrap();
assert_eq!(span.start, Timestamp::from_micros(10));
// Entered from 15 to 25
assert_eq!(span.time_data.self_time, Timestamp::from_micros(10));
assert_eq!(span.time_data.self_end, Timestamp::from_micros(25));
assert!(span.is_complete);
}
#[test]
fn rejects_timestamps_without_base() {
let row: TraceRow<'_> = TraceRow::End {
ts: DeltaEncodedTimestamp(4),
id: 1,
};
let data = postcard::to_stdvec(&row).unwrap();
let store = Arc::new(StoreContainer::new());
let mut format = TurbopackFormat::new(store);
let mut reuse = TurbopackFormat::create_reused();
assert!(TraceFormat::read(&mut format, &data, &mut reuse).is_err());
}
#[test] #[test]
fn attributes_allocations_of_enter_and_exit_rows() { fn attributes_allocations_of_enter_and_exit_rows() {
let enter = |ts, id, allocations| TraceRow::Enter { let enter = |ts, id, allocations| TraceRow::Enter {
......
...@@ -20,7 +20,7 @@ use std::{ ...@@ -20,7 +20,7 @@ use std::{
use anyhow::{Context, Result, bail}; use anyhow::{Context, Result, bail};
use flate2::bufread::MultiGzDecoder; use flate2::bufread::MultiGzDecoder;
use rustc_hash::{FxHashMap, FxHashSet}; use rustc_hash::{FxHashMap, FxHashSet};
use turbopack_trace_utils::tracing::{Allocations, TraceRow, TraceValue}; use turbopack_trace_utils::tracing::{Allocations, TimestampDecoder, TraceRow, TraceValue};
/// The magic bytes at the start of every (uncompressed) trace file. /// The magic bytes at the start of every (uncompressed) trace file.
pub const TRACE_HEADER: &[u8] = b"TRACEv0"; pub const TRACE_HEADER: &[u8] = b"TRACEv0";
...@@ -37,10 +37,11 @@ pub enum RowKind { ...@@ -37,10 +37,11 @@ pub enum RowKind {
Event, Event,
Record, Record,
MemorySample, MemorySample,
TimestampBase,
} }
impl RowKind { impl RowKind {
pub const ALL: [RowKind; 7] = [ pub const ALL: [RowKind; 8] = [
RowKind::Start, RowKind::Start,
RowKind::End, RowKind::End,
RowKind::Enter, RowKind::Enter,
...@@ -48,9 +49,10 @@ impl RowKind { ...@@ -48,9 +49,10 @@ impl RowKind {
RowKind::Event, RowKind::Event,
RowKind::Record, RowKind::Record,
RowKind::MemorySample, RowKind::MemorySample,
RowKind::TimestampBase,
]; ];
pub fn of(row: &TraceRow<'_>) -> Self { pub fn of<T>(row: &TraceRow<'_, T>) -> Self {
match row { match row {
TraceRow::Start { .. } => RowKind::Start, TraceRow::Start { .. } => RowKind::Start,
TraceRow::End { .. } => RowKind::End, TraceRow::End { .. } => RowKind::End,
...@@ -59,6 +61,7 @@ impl RowKind { ...@@ -59,6 +61,7 @@ impl RowKind {
TraceRow::Event { .. } => RowKind::Event, TraceRow::Event { .. } => RowKind::Event,
TraceRow::Record { .. } => RowKind::Record, TraceRow::Record { .. } => RowKind::Record,
TraceRow::MemorySample { .. } => RowKind::MemorySample, TraceRow::MemorySample { .. } => RowKind::MemorySample,
TraceRow::TimestampBase { .. } => RowKind::TimestampBase,
} }
} }
...@@ -71,6 +74,7 @@ impl RowKind { ...@@ -71,6 +74,7 @@ impl RowKind {
RowKind::Event => "Event", RowKind::Event => "Event",
RowKind::Record => "Record", RowKind::Record => "Record",
RowKind::MemorySample => "MemorySample", RowKind::MemorySample => "MemorySample",
RowKind::TimestampBase => "TimestampBase",
} }
} }
} }
...@@ -268,6 +272,8 @@ impl TraceSizeAnalyzer { ...@@ -268,6 +272,8 @@ impl TraceSizeAnalyzer {
// Index into `buffer` up to which rows have been consumed. // Index into `buffer` up to which rows have been consumed.
let mut start = 0; let mut start = 0;
let mut header_checked = false; let mut header_checked = false;
// Timestamps aren't analyzed, but decoding them validates the stream.
let mut timestamps = TimestampDecoder::default();
loop { loop {
let bytes_read = match reader.read(&mut chunk) { let bytes_read = match reader.read(&mut chunk) {
Ok(n) => n, Ok(n) => n,
...@@ -297,6 +303,13 @@ impl TraceSizeAnalyzer { ...@@ -297,6 +303,13 @@ impl TraceSizeAnalyzer {
match postcard::take_from_bytes::<TraceRow<'_>>(remaining) { match postcard::take_from_bytes::<TraceRow<'_>>(remaining) {
Ok((row, rest)) => { Ok((row, rest)) => {
let row_bytes = remaining.len() - rest.len(); let row_bytes = remaining.len() - rest.len();
let row = match timestamps.decode(row) {
Ok(row) => row,
Err(err) => {
let offset = self.uncompressed_size - (buffer.len() - start) as u64;
bail!("Invalid trace row at uncompressed offset {offset}: {err}");
}
};
self.add_row(&row, row_bytes as u64); self.add_row(&row, row_bytes as u64);
start += row_bytes; start += row_bytes;
} }
...@@ -316,7 +329,7 @@ impl TraceSizeAnalyzer { ...@@ -316,7 +329,7 @@ impl TraceSizeAnalyzer {
} }
/// Accounts a single decoded row that took `bytes` bytes in the stream. /// Accounts a single decoded row that took `bytes` bytes in the stream.
pub fn add_row(&mut self, row: &TraceRow<'_>, bytes: u64) { pub fn add_row(&mut self, row: &TraceRow<'_, u64>, bytes: u64) {
self.row_kinds[RowKind::of(row) as usize].add(bytes); self.row_kinds[RowKind::of(row) as usize].add(bytes);
match row { match row {
TraceRow::Start { TraceRow::Start {
...@@ -369,7 +382,7 @@ impl TraceSizeAnalyzer { ...@@ -369,7 +382,7 @@ impl TraceSizeAnalyzer {
} }
self.components.event_values += self.add_values(values); self.components.event_values += self.add_values(values);
} }
TraceRow::MemorySample { .. } => {} TraceRow::MemorySample { .. } | TraceRow::TimestampBase { .. } => {}
} }
} }
...@@ -887,15 +900,18 @@ mod tests { ...@@ -887,15 +900,18 @@ mod tests {
use std::borrow::Cow; use std::borrow::Cow;
use flate2::{Compression as GzLevel, write::GzEncoder}; use flate2::{Compression as GzLevel, write::GzEncoder};
use turbopack_trace_utils::tracing::{Allocations, TraceRow, TraceValue}; use turbopack_trace_utils::tracing::{
Allocations, DeltaEncodedTimestamp, TimestampEncoder, TraceRow, TraceValue,
};
use crate::{ use crate::{
Compression, NO_SPAN, RowKind, StringKind, TRACE_HEADER, TraceSizeAnalyzer, UNKNOWN_SPAN, Compression, NO_SPAN, RowKind, StringKind, TRACE_HEADER, TraceSizeAnalyzer, UNKNOWN_SPAN,
allocations_size, value_size, allocations_size, value_size,
}; };
fn sample_rows() -> Vec<TraceRow<'static>> { fn sample_rows() -> Vec<TraceRow<'static, u64>> {
vec![ vec![
TraceRow::TimestampBase { ts: 0 },
TraceRow::Start { TraceRow::Start {
ts: 1, ts: 1,
id: 1, id: 1,
...@@ -996,11 +1012,25 @@ mod tests { ...@@ -996,11 +1012,25 @@ mod tests {
] ]
} }
fn encode(rows: &[TraceRow<'_>]) -> (Vec<u8>, Vec<u64>) { /// Serializes the rows with delta encoded timestamps, like the trace writer. The rows must
/// start with a [`TraceRow::TimestampBase`]. Returns the data and the size of each row.
fn encode(rows: &[TraceRow<'_, u64>]) -> (Vec<u8>, Vec<u64>) {
let mut data = TRACE_HEADER.to_vec(); let mut data = TRACE_HEADER.to_vec();
let mut sizes = Vec::new(); let mut sizes = Vec::new();
let mut encoder = TimestampEncoder::default();
for row in rows { for row in rows {
// A copy of the row, through its serialized form
let bytes = postcard::to_stdvec(row).unwrap(); let bytes = postcard::to_stdvec(row).unwrap();
let row: TraceRow<'_, u64> = postcard::from_bytes(&bytes).unwrap();
let row: TraceRow<'_> = if let TraceRow::TimestampBase { ts } = row {
encoder.encode(ts);
TraceRow::TimestampBase { ts }
} else {
let (base, row) = encoder.encode_row(row);
assert!(base.is_none(), "missing base row");
row
};
let bytes = postcard::to_stdvec(&row).unwrap();
sizes.push(bytes.len() as u64); sizes.push(bytes.len() as u64);
data.extend_from_slice(&bytes); data.extend_from_slice(&bytes);
} }
...@@ -1111,20 +1141,20 @@ mod tests { ...@@ -1111,20 +1141,20 @@ mod tests {
let build = spans.iter().find(|s| s.name == "build").unwrap(); let build = spans.iter().find(|s| s.name == "build").unwrap();
assert_eq!(build.target, "next"); assert_eq!(build.target, "next");
assert_eq!(build.start.count, 2); assert_eq!(build.start.count, 2);
assert_eq!(build.start.bytes, sizes[0] + sizes[2]); assert_eq!(build.start.bytes, sizes[1] + sizes[3]);
// Enter, Event, Record, End, Exit, End // Enter, Event, Record, End, Exit, End
assert_eq!(build.other.count, 6); assert_eq!(build.other.count, 6);
assert_eq!( assert_eq!(
build.other.bytes, build.other.bytes,
sizes[1] + sizes[3..8].iter().sum::<u64>() sizes[2] + sizes[4..9].iter().sum::<u64>()
); );
let unknown = spans.iter().find(|s| s.name == UNKNOWN_SPAN).unwrap(); let unknown = spans.iter().find(|s| s.name == UNKNOWN_SPAN).unwrap();
assert_eq!(unknown.other.count, 1); assert_eq!(unknown.other.count, 1);
assert_eq!(unknown.other.bytes, sizes[8]); assert_eq!(unknown.other.bytes, sizes[9]);
let late = spans.iter().find(|s| s.name == "late").unwrap(); let late = spans.iter().find(|s| s.name == "late").unwrap();
assert_eq!(late.start.bytes, sizes[13]); assert_eq!(late.start.bytes, sizes[14]);
assert_eq!(late.other.count, 1); assert_eq!(late.other.count, 1);
assert_eq!(late.other.bytes, sizes[12]); assert_eq!(late.other.bytes, sizes[13]);
let fields = analyzer.fields_by_size(); let fields = analyzer.fields_by_size();
let path = fields.iter().find(|f| f.key == "path").unwrap(); let path = fields.iter().find(|f| f.key == "path").unwrap();
...@@ -1218,9 +1248,29 @@ mod tests { ...@@ -1218,9 +1248,29 @@ mod tests {
); );
} }
#[test]
fn fails_on_timestamp_without_base() {
let mut data = TRACE_HEADER.to_vec();
let offset = data.len();
let row: TraceRow<'_> = TraceRow::End {
ts: DeltaEncodedTimestamp(4),
id: 1,
};
data.extend(postcard::to_stdvec(&row).unwrap());
let mut analyzer = TraceSizeAnalyzer::new();
let err = analyzer.analyze_reader(&data[..]).unwrap_err();
let err = err.to_string();
assert!(
err.contains(&format!("uncompressed offset {offset}")),
"{err}"
);
assert!(err.contains("timestamp base"), "{err}");
}
#[test] #[test]
fn attributes_reused_ids_and_events_without_span() { fn attributes_reused_ids_and_events_without_span() {
let rows = vec![ let rows: Vec<TraceRow<'_, u64>> = vec![
TraceRow::TimestampBase { ts: 0 },
TraceRow::Start { TraceRow::Start {
ts: 1, ts: 1,
id: 5, id: 5,
...@@ -1255,9 +1305,9 @@ mod tests { ...@@ -1255,9 +1305,9 @@ mod tests {
analyzer.analyze_reader(&data[..]).unwrap(); analyzer.analyze_reader(&data[..]).unwrap();
let spans = analyzer.spans_by_size(); let spans = analyzer.spans_by_size();
let span = |name: &str| *spans.iter().find(|s| s.name == name).unwrap(); let span = |name: &str| *spans.iter().find(|s| s.name == name).unwrap();
assert_eq!(span("a").other.bytes, sizes[1]); assert_eq!(span("a").other.bytes, sizes[2]);
assert_eq!(span("b").other.bytes, sizes[3]); assert_eq!(span("b").other.bytes, sizes[4]);
assert_eq!(span(NO_SPAN).other.bytes, sizes[4]); assert_eq!(span(NO_SPAN).other.bytes, sizes[5]);
assert!(spans.iter().all(|s| s.name != UNKNOWN_SPAN)); assert!(spans.iter().all(|s| s.name != UNKNOWN_SPAN));
} }
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment