Unverified Commit 9da349b7 authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

Turbopack: wait for trace file changes after format errors (#99847)

### What?

`turbo-trace-server` no longer re-reads a trace file every 500 ms after
it was rejected with a format error. It prints the error once, prints
`Waiting for the trace file to change...`, and reads the file again only
once it changes (different size, modification or creation time, or on
unix a different file at the path) or is removed.

### Why?

`TraceReader::try_read` returned `true` after printing `Trace file
error: ...`, so `TraceReader::run` slept 500 ms and read the whole file
again, forever. With the `TRACEv1` header check (#99781), every old
`TRACEv0` trace hit this: the server printed the same error twice a
second (10 times in 5 s locally). Corrupt trace data behaved the same
way.

### How?

- Both format-error paths (format detection, and decoding with
`format.read`) take a metadata snapshot of the open file and poll the
path's metadata once per second until it differs. Only metadata is
compared, so a rewrite that keeps size and timestamps the same (and on
unix the same file) isn't noticed.
- Valid traces, live tailing, I/O errors and `STOP_AT` behave as before.

### Verification

- New unit test: a `TRACEv0` file and a corrupt `TRACEv1` file are not
read again for 2.5 s, and replacing them with a valid trace loads it. It
failed before this change.
- New unit test for change detection: size change, same-size rewrite,
replacement by another file, and removal.
- Manual, release `turbo-trace-server`:
  - `TRACEv0` file for 5 s: error printed once (before: 10 times).
- Corrupt `TRACEv1` file for 3 s: error printed once (before: 6 times).
- Replacing a rejected `TRACEv0` file with a real `TRACEv1` trace while
the server runs loads all 39,490 spans without a restart.
- Only run locally on Linux; Windows and macOS are covered by CI.

<!-- NEXT_JS_LLM -->


<!-- fleet 4a52842b-f1a2-43cb-9779-3655caada6a0 -->
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent a72565ab
...@@ -6,12 +6,12 @@ pub(crate) mod turbopack; ...@@ -6,12 +6,12 @@ pub(crate) mod turbopack;
use std::{ use std::{
any::Any, any::Any,
env, env,
fs::File, fs::{self, File, Metadata},
io::{self, BufReader, Read, Seek, SeekFrom, Write}, io::{self, BufReader, Read, Seek, SeekFrom, Write},
path::PathBuf, path::{Path, PathBuf},
sync::Arc, sync::Arc,
thread::{self, JoinHandle}, thread::{self, JoinHandle},
time::{Duration, Instant}, time::{Duration, Instant, SystemTime},
}; };
use anyhow::Result; use anyhow::Result;
...@@ -152,6 +152,15 @@ impl TraceFile { ...@@ -152,6 +152,15 @@ impl TraceFile {
} }
} }
/// Metadata of the opened file on disk.
fn metadata(&self) -> io::Result<Metadata> {
match self {
Self::Raw(file) => file.get_ref().metadata(),
Self::Compressed { file, .. } => file.metadata(),
Self::Unloaded => unreachable!(),
}
}
/// Size of the opened file on disk. /// Size of the opened file on disk.
fn size(&mut self) -> io::Result<u64> { fn size(&mut self) -> io::Result<u64> {
match self { match self {
...@@ -162,6 +171,39 @@ impl TraceFile { ...@@ -162,6 +171,39 @@ impl TraceFile {
} }
} }
/// Identifies one version of the trace file on disk, to notice when it is written to, truncated or
/// replaced. Only uses metadata, so a rewrite that keeps all of these the same isn't noticed.
#[derive(Debug, PartialEq, Eq)]
struct FileVersion {
len: u64,
modified: Option<SystemTime>,
created: Option<SystemTime>,
/// Device and inode, which identify the file itself on unix.
#[cfg(unix)]
file_id: (u64, u64),
}
impl FileVersion {
fn from_metadata(metadata: &Metadata) -> Self {
Self {
len: metadata.len(),
modified: metadata.modified().ok(),
created: metadata.created().ok(),
#[cfg(unix)]
file_id: {
use std::os::unix::fs::MetadataExt;
(metadata.dev(), metadata.ino())
},
}
}
fn of_path(path: &Path) -> Option<Self> {
fs::metadata(path)
.ok()
.map(|metadata| Self::from_metadata(&metadata))
}
}
pub struct TraceReader { pub struct TraceReader {
store: Arc<StoreContainer>, store: Arc<StoreContainer>,
path: PathBuf, path: PathBuf,
...@@ -291,6 +333,7 @@ impl TraceReader { ...@@ -291,6 +333,7 @@ impl TraceReader {
Ok(None) => continue, Ok(None) => continue,
Err(err) => { Err(err) => {
println!("Trace file error: {err}"); println!("Trace file error: {err}");
self.wait_for_file_change(&file);
return true; return true;
} }
}; };
...@@ -318,6 +361,7 @@ impl TraceReader { ...@@ -318,6 +361,7 @@ impl TraceReader {
} }
Err(err) => { Err(err) => {
println!("Trace file error: {err}"); println!("Trace file error: {err}");
self.wait_for_file_change(&file);
return true; return true;
} }
} }
...@@ -441,6 +485,18 @@ impl TraceReader { ...@@ -441,6 +485,18 @@ impl TraceReader {
} }
} }
/// Waits until the file at the trace path differs from `file`, which was rejected with a
/// format error, so the same error isn't reported again and again. Only checks metadata, at
/// most once per second. Also returns when the file is removed.
fn wait_for_file_change(&self, file: &TraceFile) {
println!("Waiting for the trace file to change...");
let version = file
.metadata()
.ok()
.map(|metadata| FileVersion::from_metadata(&metadata));
wait_for_file_version_change(&self.path, version.as_ref(), Duration::from_secs(1));
}
fn wait_for_new_file(&self, file: &mut TraceFile) { fn wait_for_new_file(&self, file: &mut TraceFile) {
let Ok(pos) = file.stream_position() else { let Ok(pos) = file.stream_position() else {
return; return;
...@@ -457,6 +513,18 @@ impl TraceReader { ...@@ -457,6 +513,18 @@ impl TraceReader {
} }
} }
/// Polls the metadata of `path` every `interval` until it no longer matches `version`, or the file
/// is removed.
fn wait_for_file_version_change(path: &Path, version: Option<&FileVersion>, interval: Duration) {
loop {
thread::sleep(interval);
match FileVersion::of_path(path) {
Some(current) if Some(&current) == version => {}
_ => return,
}
}
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use std::{borrow::Cow, io::Write, time::Duration}; use std::{borrow::Cow, io::Write, time::Duration};
...@@ -556,6 +624,28 @@ mod tests { ...@@ -556,6 +624,28 @@ mod tests {
} }
} }
fn is_finished(&self) -> bool {
self.thread
.as_ref()
.is_none_or(|thread| thread.is_finished())
}
/// Waits for the current [`TraceReader::try_read`] pass to return, then starts the next
/// one on the same file, like [`TraceReader::run`] does.
fn restart_after_return(&mut self) {
let start = Instant::now();
while !self.is_finished() {
assert!(start.elapsed() < TIMEOUT, "reader did not return");
thread::sleep(Duration::from_millis(20));
}
self.thread.take().unwrap().join().unwrap();
let mut reader = TraceReader {
store: self.store.clone(),
path: self.path.clone(),
};
self.thread = Some(thread::spawn(move || reader.try_read()));
}
fn append(&self, data: &[u8]) { fn append(&self, data: &[u8]) {
std::fs::OpenOptions::new() std::fs::OpenOptions::new()
.append(true) .append(true)
...@@ -641,6 +731,92 @@ mod tests { ...@@ -641,6 +731,92 @@ mod tests {
} }
} }
/// A file that is rejected with a format error is not read again until it changes, so the
/// error isn't repeated twice a second. Replacing it with a valid trace loads that trace.
#[test]
fn waits_for_change_after_format_error() {
for (name, data) in [
// Unsupported version (header detection error)
("old-version.trace", b"TRACEv0\x00\x01\x02".to_vec()),
// Corrupt data after a valid header (decode error)
("corrupt.trace", [TRACE_HEADER, &[0xff; 12]].concat()),
] {
let mut reader = ReaderFixture::new(name, &data);
// Previously, `try_read` returned right away and `run` read the file again after
// 500 ms. Several of those retry intervals pass here without the file changing.
thread::sleep(Duration::from_millis(2_500));
assert!(
!reader.is_finished(),
"{name}: reader returned without the file changing"
);
std::fs::write(&reader.path, trace(1_000)).unwrap();
reader.restart_after_return();
reader.wait_for_span_count(1_000);
}
}
/// Each kind of change to a rejected file ends the wait: a different size, a rewrite with the
/// same size (new modification time), a replacement by another file, and removal.
#[test]
fn file_version_detects_changes() {
let dir = env::temp_dir().join(format!("trace-server-test-{}-version", std::process::id()));
fs::create_dir_all(&dir).unwrap();
let path = dir.join("version.trace");
let version = |path: &Path| FileVersion::of_path(path).unwrap();
let modified_at = |path: &Path, time: SystemTime| {
File::options()
.write(true)
.open(path)
.unwrap()
.set_modified(time)
.unwrap();
};
let time = SystemTime::UNIX_EPOCH + Duration::from_secs(1_000_000);
fs::write(&path, b"aaaa").unwrap();
modified_at(&path, time);
let original = version(&path);
assert_eq!(version(&path), original, "unchanged file");
fs::write(&path, b"aaaaa").unwrap();
modified_at(&path, time);
assert_ne!(version(&path), original, "different size");
fs::write(&path, b"bbbb").unwrap();
modified_at(&path, time + Duration::from_secs(1));
assert_ne!(version(&path), original, "same size, rewritten");
fs::write(&path, b"aaaa").unwrap();
modified_at(&path, time);
let before_replace = version(&path);
let replacement = dir.join("replacement.trace");
fs::write(&replacement, b"aaaa").unwrap();
modified_at(&replacement, time);
fs::rename(&replacement, &path).unwrap();
// Same size and modification time, but on unix another inode.
#[cfg(unix)]
assert_ne!(version(&path), before_replace, "replaced file");
#[cfg(not(unix))]
let _ = before_replace;
// Waiting returns once the file is removed.
let waiting_for = version(&path);
let wait_path = path.clone();
let waiter = thread::spawn(move || {
wait_for_file_version_change(&wait_path, Some(&waiting_for), Duration::from_millis(20))
});
thread::sleep(Duration::from_millis(200));
assert!(!waiter.is_finished(), "returned for an unchanged file");
fs::remove_file(&path).unwrap();
let start = Instant::now();
while !waiter.is_finished() {
assert!(start.elapsed() < TIMEOUT, "did not return after removal");
thread::sleep(Duration::from_millis(20));
}
waiter.join().unwrap();
fs::remove_dir_all(&dir).unwrap();
}
#[test] #[test]
fn detects_trace_formats() { fn detects_trace_formats() {
assert_eq!(detect_format(b"").unwrap(), None); assert_eq!(detect_format(b"").unwrap(), None);
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment