The execution and persistence backend for `turbo-tasks`, the incremental computation framework that powers Turbopack. It manages task scheduling, dependency tracking, invalidation propagation, and persistent caching of computation results.
## Architecture
The crate is organized in three layers:
```
TurboTasksBackend<B> Public API, implements the Backend trait
|-- LmdbKeyValueDatabase (LMDB, for debugging/reproduction)
+-- NoopKvDb (in-memory only, for tests)
```
## Task Storage
Each task's state is stored in a `TaskStorage` struct generated by the `#[task_storage]` macro from a schema definition. Fields are organized into three categories:
-**Transient** — Never serialized (execution state, in-progress cells, events). Memory-only.
Fields can be stored **inline** (always allocated, for hot-path access) or **lazily** (in a `Vec<LazyField>`, for memory efficiency). Boolean flags are packed into a `TaskFlags` bitfield.
## Operations
All graph mutations go through the operation system. Operations are enums that implement `Encode`/`Decode`, making them **resumable**: they can be serialized mid-execution at a suspension point and resumed after a snapshot or restart.
Key operations:
| Operation | Purpose |
|-----------|---------|
| `InvalidateOperation` | Marks tasks dirty, schedules recomputation |
Operations run inside an `ExecuteContext` that provides task locking, storage access, and suspension-point coordination.
## Aggregation Tree
Tasks form a hierarchical aggregation tree used for efficient dirty-state propagation:
-**Leaf nodes** (aggregation number 0..15): Tracked individually by their parent aggregation nodes.
-**Intermediate nodes** (16+): Aggregate dirty counts from their subgraph. Dirty/clean state propagates up through aggregated counters rather than touching every leaf.
-**Root nodes** (`u32::MAX`): Represent the top of the tree.
The aggregation number `{base, distance, effective}` determines a task's position. When a task becomes dirty, the change propagates upward through `aggregated_dirty_container_count` fields, stopping at aggregation boundaries. This avoids O(n) traversal of the entire graph.
## Persistence
Persistence uses **snapshot-based batch writes** rather than write-ahead logging:
1. The backend accumulates modifications in memory during normal operation.
2. When idle, a snapshot is requested: the high bit of `in_progress_operations` is set as a signal.
3. Active operations check `operation_suspend_point()` at safe points and pause.
4. Once all operations are suspended, modified task data is serialized and batch-written to the backing storage.
5. Operations resume after the snapshot completes.
This design minimizes I/O overhead compared to per-operation persistence. Tasks are lazily restored from disk on first access, reducing startup time and memory usage.
### Backing Storage Implementations
**TurboKeyValueDatabase** (default): Uses the `turbo-persistence` crate, a custom high-performance embedded database with adaptive compaction. Supports concurrent write batches and session-aware compaction strategies.
**LmdbKeyValueDatabase** (feature-gated): Uses LMDB (Lightning Memory-Mapped Database). Slower but proven stable — useful for reproducing user-reported bugs in isolation. Augmented with read transaction caching, startup pre-loading, and fresh-database optimizations.
**NoopKvDb**: In-memory only, no persistence. Used for testing.
## Task IDs
Task IDs are split into two ranges:
-`0..TRANSIENT_TASK_BIT` — **Persistent tasks**: Cached across process restarts.
-`TRANSIENT_TASK_BIT..u32::MAX` — **Transient tasks**: Root tasks and one-shot tasks that are never persisted. Filtered out during snapshot serialization.
## Feature Flags
Notable cargo features:
-`lmdb` — Enables the LMDB-based backing storage
-`verify_aggregation_graph` — Runtime verification of aggregation tree invariants