Unverified Commit fff4a4d9 authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

turbo-persistence: stop background persisting after unrecoverable failure (#92106)

### What?

When a persist or compaction operation fails in `turbo-persistence`, the
database now:
- Rolls back cleanly (deletes orphan files, restores CURRENT)
- Stops the background persisting process for the session
- Keeps in-memory state consistent with on-disk state at all times
- Deletes superseded files safely (with Windows fallback for open memory
maps)

### Why?

Previously, a failed write operation (e.g. disk full, I/O error) would
leave the database in a broken state:

1. **Misleading error loop** — The `active_write_operation` `AtomicBool`
was left set to `true` after failure, so every subsequent snapshot cycle
printed _"another write operation is already in progress"_ forever,
hiding the real error.

2. **In-memory corruption** — `commit()` mutated `inner.meta_files` and
`inner.current_sequence_number` *before* writing the CURRENT file to
disk. If a disk error occurred between those two steps, the in-memory
state was inconsistent with disk and the rollback had no way to fix it.

3. **Rollback could corrupt committed data** — If `commit()` failed
*after* writing CURRENT (e.g. during old-file deletion or LOG writing),
the rollback would delete the *newly committed* files, corrupting the
database.

4. **Task graph corruption** — `save_snapshot` consumes task cache log
entries. If it failed, those entries were lost, but the background loop
would continue trying to persist — silently skipping those tasks and
corrupting the task graph in storage.

5. **Partially written CURRENT** — If the failure happened mid-write to
the CURRENT file, it could be left with partial/corrupt content, but
nothing restored it.

### How?

**`WriteOperationGuard` RAII (db.rs)**

A new `WriteOperationGuard<'a>` replaces the `AtomicBool` + manual
`try_recover_after_failed_write()` pattern. The guard holds:
- `&'a Mutex<Option<ActiveWriteState>>` — the write slot (`None` = idle,
`Some(Active("write batch"))` = in progress, `Some(Error)` = permanently
disabled)
- `path: &'a Path` — database directory for rollback
- `seq_before: u32` — sequence number at operation start
- `succeeded: bool` — set by `guard.success()`

On `drop`, if not succeeded:
1. Writes `seq_before` back to CURRENT (repairs a partially-written
CURRENT)
2. Deletes all files with `seq > seq_before` (orphans from the failed
operation)
3. Sets the slot to `None` (success) or `Some(Error)` (if cleanup itself
failed)

The `Active` variant carries a `&'static str` name (e.g. `"write
batch"`, `"compaction"`) used in error messages.

**Three-phase `commit()` (db.rs)**

`commit()` is restructured so `inner` is completely unmodified before
the point of no return:

| Phase | What happens | `inner` state | On failure |
|-------|-------------|---------------|------------|
| **A** | Compute `meta_seq_numbers_to_delete` via `sst_filter`. Uses
`apply_filter_collect` (read-only) to update filter state and collect
per-meta-file removal sets without modifying any MetaFile. Only a read
lock on `inner` is needed. | Unchanged | Guard deletes orphan files +
restores CURRENT; `inner` is intact |
| **B** | Write `.del` file and CURRENT to disk. | Unchanged | Same as
above |
| **C** | Apply deferred `retain_entries` (from A's removal sets),
append new metas, remove obsolete metas, bump `current_sequence_number`.
Try to delete superseded files; defer failures. | Updated | CURRENT is
already durable; commit is irreversible |

After CURRENT is written (point of no return), LOG writing errors are
caught and reported via `eprintln!` — they must not propagate because
the `WriteOperationGuard` would then run its rollback and delete the
*newly committed* files.

**`SstFilter::apply_filter_collect` (sst_filter.rs)**

A new read-only variant of `apply_filter` that updates the filter state
and returns a `FxHashSet<u32>` of SST entry sequence numbers to remove
from each meta file, without calling `retain_entries`. The original
`apply_filter` (which mutates the MetaFile) is still used by
`load_directory` and during new-meta-file construction where immediate
mutation is appropriate.

**Deferred file deletion (db.rs)**

Superseded `.sst`/`.meta`/`.blob` files are deleted immediately after
Phase C (once `inner` is updated). On Linux/macOS this always succeeds,
even if concurrent readers have the files memory-mapped. On Windows,
open memory maps prevent deletion — any file that fails is stored as a
`DeferredDeletion` enum (`Sst(u32)` / `Meta(u32)` / `Blob(u32)`) and
retried on the next commit or at shutdown. The `.del` file written
during Phase B ensures crash recovery via `load_directory` regardless.

**Background loop error handling (backend/mod.rs)**

- `snapshot_and_persist()` returns `Result<(Instant, bool),
anyhow::Error>` instead of `Option`. When `save_snapshot` fails, the
error propagates with `?`.
- The background loop matches on the `Result`: on `Err`, it logs the
error and a message that persisting is disabled for this session, then
returns (permanently stopping the background job).
- `has_unrecoverable_write_error()` checks the `ActiveWriteState::Error`
variant to detect permanent failure after compaction errors.

<!-- NEXT_JS_LLM_PR -->

---------
Co-authored-by: 's avatarTobias Koppers <sokra@users.noreply.github.com>
Co-authored-by: 's avatarClaude <noreply@anthropic.com>
parent d4b051d9
use std::collections::hash_map::Entry; use std::collections::hash_map::Entry;
use rustc_hash::FxHashMap; use rustc_hash::{FxHashMap, FxHashSet};
use crate::meta_file::MetaFile; use crate::meta_file::MetaFile;
...@@ -47,6 +47,42 @@ impl SstFilter { ...@@ -47,6 +47,42 @@ impl SstFilter {
} }
} }
/// Like [`apply_filter`](Self::apply_filter) but only updates the filter state without
/// modifying the MetaFile. Returns the set of SST sequence numbers that should be removed
/// from this meta file's entries (they are superseded by a newer meta). Call
/// `meta.retain_entries(|seq| !obsolete.contains(&seq))` later to apply.
pub fn apply_filter_collect(&mut self, meta: &MetaFile) -> FxHashSet<u32> {
let mut to_remove = FxHashSet::default();
// Already obsolete entries need to be considered for usage computation
for seq in meta.obsolete_entries() {
if let Some(state) = self.0.get_mut(seq)
&& matches!(state, SstState::UnusedObsolete)
{
// the obsolete state is used now
*state = SstState::Obsolete;
}
}
for entry in meta.entries() {
let seq = entry.sequence_number();
match self.0.entry(seq) {
Entry::Occupied(mut e) => {
let state = e.get_mut();
if matches!(state, SstState::UnusedObsolete) {
*state = SstState::Obsolete;
}
to_remove.insert(seq);
}
Entry::Vacant(e) => {
e.insert(SstState::Active);
}
}
}
for seq in meta.obsolete_sst_files() {
self.0.entry(*seq).or_insert(SstState::UnusedObsolete);
}
to_remove
}
/// Phase 2: Check if the meta file can be removed based on the filter state after phase 1. /// Phase 2: Check if the meta file can be removed based on the filter state after phase 1.
/// Updates the filter state for the next meta file. Returns true if the meta file can be /// Updates the filter state for the next meta file. Returns true if the meta file can be
/// removed. /// removed.
......
...@@ -2023,18 +2023,19 @@ fn compaction_deletes_superseded_blob() -> Result<()> { ...@@ -2023,18 +2023,19 @@ fn compaction_deletes_superseded_blob() -> Result<()> {
// Compact — the old blob entry is superseded by the newer small value // Compact — the old blob entry is superseded by the newer small value
db.full_compact()?; db.full_compact()?;
// After compaction, the old blob file should be deleted // The new value should still be readable
let result = db.get(0, &vec![1u8])?;
assert_eq!(result.as_deref(), Some(&[99u8][..]));
// After compaction, the old blob file should be deleted immediately.
assert_eq!( assert_eq!(
count_blob_files(path), count_blob_files(path),
0, 0,
"Old blob file should be deleted after compaction" "Old blob file should be deleted after compaction"
); );
// The new value should still be readable
let result = db.get(0, &vec![1u8])?;
assert_eq!(result.as_deref(), Some(&[99u8][..]));
db.shutdown()?; db.shutdown()?;
Ok(()) Ok(())
} }
...@@ -2074,18 +2075,19 @@ fn compaction_deletes_blob_on_tombstone() -> Result<()> { ...@@ -2074,18 +2075,19 @@ fn compaction_deletes_blob_on_tombstone() -> Result<()> {
// Compact — tombstone supersedes the blob entry // Compact — tombstone supersedes the blob entry
db.full_compact()?; db.full_compact()?;
// After compaction, the blob file should be deleted // Key should not be found
let result = db.get(0, &vec![1u8])?;
assert!(result.is_none());
// After compaction, the blob file should be deleted immediately.
assert_eq!( assert_eq!(
count_blob_files(path), count_blob_files(path),
0, 0,
"Blob file should be deleted after compaction" "Blob file should be deleted after compaction"
); );
// Key should not be found
let result = db.get(0, &vec![1u8])?;
assert!(result.is_none());
db.shutdown()?; db.shutdown()?;
Ok(()) Ok(())
} }
...@@ -2130,19 +2132,20 @@ fn compaction_deletes_blob_multi_value_tombstone() -> Result<()> { ...@@ -2130,19 +2132,20 @@ fn compaction_deletes_blob_multi_value_tombstone() -> Result<()> {
// Compact — tombstone prunes the old blob entry // Compact — tombstone prunes the old blob entry
db.full_compact()?; db.full_compact()?;
// After compaction, the old blob file should be deleted // The new value should still be readable
let results = db.get_multiple(0, &vec![1u8].as_slice())?;
assert_eq!(results.len(), 1);
assert_eq!(results[0].as_ref(), &[99u8]);
// After compaction, the old blob file should be deleted immediately.
assert_eq!( assert_eq!(
count_blob_files(path), count_blob_files(path),
0, 0,
"Blob file should be deleted after compaction" "Blob file should be deleted after compaction"
); );
// The new value should still be readable
let results = db.get_multiple(0, &vec![1u8].as_slice())?;
assert_eq!(results.len(), 1);
assert_eq!(results[0].as_ref(), &[99u8]);
db.shutdown()?; db.shutdown()?;
Ok(()) Ok(())
} }
......
...@@ -20,6 +20,7 @@ use crate::{ ...@@ -20,6 +20,7 @@ use crate::{
collector_entry::CollectorEntry, collector_entry::CollectorEntry,
compression::{checksum_block, compress_into_buffer}, compression::{checksum_block, compress_into_buffer},
constants::{MAX_MEDIUM_VALUE_SIZE, THREAD_LOCAL_SIZE_SHIFT}, constants::{MAX_MEDIUM_VALUE_SIZE, THREAD_LOCAL_SIZE_SHIFT},
db::WriteOperationGuard,
key::StoreKey, key::StoreKey,
meta_file::MetaEntryFlags, meta_file::MetaEntryFlags,
meta_file_builder::MetaFileBuilder, meta_file_builder::MetaFileBuilder,
...@@ -66,7 +67,9 @@ enum GlobalCollectorState<K: StoreKey + Send> { ...@@ -66,7 +67,9 @@ enum GlobalCollectorState<K: StoreKey + Send> {
} }
/// A write batch. /// A write batch.
pub struct WriteBatch<K: StoreKey + Send, S: ParallelScheduler, const FAMILIES: usize> { pub struct WriteBatch<'db, K: StoreKey + Send, S: ParallelScheduler, const FAMILIES: usize> {
/// RAII guard that releases the write-operation slot (and rolls back on failure) on drop.
_write_guard: WriteOperationGuard<'db>,
/// Parallel scheduler /// Parallel scheduler
parallel_scheduler: S, parallel_scheduler: S,
/// The database path /// The database path
...@@ -87,11 +90,12 @@ pub struct WriteBatch<K: StoreKey + Send, S: ParallelScheduler, const FAMILIES: ...@@ -87,11 +90,12 @@ pub struct WriteBatch<K: StoreKey + Send, S: ParallelScheduler, const FAMILIES:
new_sst_files: Mutex<Vec<(u32, File)>>, new_sst_files: Mutex<Vec<(u32, File)>>,
} }
impl<K: StoreKey + Send + Sync, S: ParallelScheduler, const FAMILIES: usize> impl<'db, K: StoreKey + Send + Sync, S: ParallelScheduler, const FAMILIES: usize>
WriteBatch<K, S, FAMILIES> WriteBatch<'db, K, S, FAMILIES>
{ {
/// Creates a new write batch for a database with per-family configuration. /// Creates a new write batch for a database with per-family configuration.
pub(crate) fn new( pub(crate) fn new(
write_guard: WriteOperationGuard<'db>,
path: PathBuf, path: PathBuf,
current: u32, current: u32,
parallel_scheduler: S, parallel_scheduler: S,
...@@ -101,6 +105,7 @@ impl<K: StoreKey + Send + Sync, S: ParallelScheduler, const FAMILIES: usize> ...@@ -101,6 +105,7 @@ impl<K: StoreKey + Send + Sync, S: ParallelScheduler, const FAMILIES: usize>
assert!(FAMILIES <= usize_from_u32(u32::MAX)); assert!(FAMILIES <= usize_from_u32(u32::MAX));
}; };
Self { Self {
_write_guard: write_guard,
parallel_scheduler, parallel_scheduler,
db_path: path, db_path: path,
family_configs, family_configs,
...@@ -113,6 +118,14 @@ impl<K: StoreKey + Send + Sync, S: ParallelScheduler, const FAMILIES: usize> ...@@ -113,6 +118,14 @@ impl<K: StoreKey + Send + Sync, S: ParallelScheduler, const FAMILIES: usize>
} }
} }
/// Marks the write operation as successfully completed.
///
/// Must be called before dropping the `WriteBatch` to skip the rollback in the guard's `Drop`
/// impl. Typically called by `TurboPersistence::commit_write_batch` after a successful commit.
pub(crate) fn mark_succeeded(&mut self) {
self._write_guard.success();
}
/// Returns the thread local state for the current thread. /// Returns the thread local state for the current thread.
#[allow(clippy::mut_from_ref)] #[allow(clippy::mut_from_ref)]
fn thread_local_state(&self) -> &mut ThreadLocalState<K, FAMILIES> { fn thread_local_state(&self) -> &mut ThreadLocalState<K, FAMILIES> {
......
...@@ -108,6 +108,12 @@ pub trait BackingStorageSealed: 'static + Send + Sync { ...@@ -108,6 +108,12 @@ pub trait BackingStorageSealed: 'static + Send + Sync {
fn shutdown(&self) -> Result<()> { fn shutdown(&self) -> Result<()> {
Ok(()) Ok(())
} }
/// Returns true if the database is in an unrecoverable error state where a previous write or
/// compaction failed and the rollback also failed, permanently disabling further writes.
fn has_unrecoverable_write_error(&self) -> bool {
false
}
} }
impl<L, R> BackingStorage for Either<L, R> impl<L, R> BackingStorage for Either<L, R>
...@@ -171,4 +177,8 @@ where ...@@ -171,4 +177,8 @@ where
fn shutdown(&self) -> Result<()> { fn shutdown(&self) -> Result<()> {
either::for_both!(self, this => this.shutdown()) either::for_both!(self, this => this.shutdown())
} }
fn has_unrecoverable_write_error(&self) -> bool {
either::for_both!(self, this => this.has_unrecoverable_write_error())
}
} }
...@@ -119,4 +119,10 @@ pub trait KeyValueDatabase { ...@@ -119,4 +119,10 @@ pub trait KeyValueDatabase {
fn shutdown(&self) -> Result<()> { fn shutdown(&self) -> Result<()> {
Ok(()) Ok(())
} }
/// Returns true if the database is in an unrecoverable error state where a previous write or
/// compaction failed and the rollback also failed, permanently disabling further writes.
fn has_unrecoverable_write_error(&self) -> bool {
false
}
} }
...@@ -137,6 +137,10 @@ impl KeyValueDatabase for TurboKeyValueDatabase { ...@@ -137,6 +137,10 @@ impl KeyValueDatabase for TurboKeyValueDatabase {
) )
} }
fn has_unrecoverable_write_error(&self) -> bool {
self.db.has_unrecoverable_write_error()
}
fn shutdown(&self) -> Result<()> { fn shutdown(&self) -> Result<()> {
// Compact the database on shutdown // Compact the database on shutdown
// (Avoid compacting a fresh database since we don't have any usage info yet) // (Avoid compacting a fresh database since we don't have any usage info yet)
...@@ -195,8 +199,12 @@ fn do_compact( ...@@ -195,8 +199,12 @@ fn do_compact(
} }
pub struct TurboWriteBatch<'a> { pub struct TurboWriteBatch<'a> {
batch: batch: turbo_persistence::WriteBatch<
turbo_persistence::WriteBatch<WriteBuffer<'static>, TurboTasksParallelScheduler, FAMILIES>, 'a,
WriteBuffer<'static>,
TurboTasksParallelScheduler,
FAMILIES,
>,
db: &'a Arc<TurboPersistence<TurboTasksParallelScheduler, FAMILIES>>, db: &'a Arc<TurboPersistence<TurboTasksParallelScheduler, FAMILIES>>,
} }
......
...@@ -381,6 +381,10 @@ impl<T: KeyValueDatabase + Send + Sync + 'static> BackingStorageSealed ...@@ -381,6 +381,10 @@ impl<T: KeyValueDatabase + Send + Sync + 'static> BackingStorageSealed
fn shutdown(&self) -> Result<()> { fn shutdown(&self) -> Result<()> {
self.inner.database.shutdown() self.inner.database.shutdown()
} }
fn has_unrecoverable_write_error(&self) -> bool {
self.inner.database.has_unrecoverable_write_error()
}
} }
fn get_next_free_task_id<'a>(batch: &impl ConcurrentWriteBatch<'a>) -> Result<u32, anyhow::Error> { fn get_next_free_task_id<'a>(batch: &impl ConcurrentWriteBatch<'a>) -> Result<u32, anyhow::Error> {
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment