[Turbopack] refactor persistent caching from log based to cow approach (#76234)
### What? Instead of collecting all modifications to the task graph in a log structure (which ends up taking a lot of memory), use a modified and snapshot flag to avoid using more memory than needed. This way we don't store extra memory before a snapshot is requested. Once a snapshot is captured, we store at most twice the memory. We store only the memory of the modified tasks. It works this way: Idle Phase: (before a snapshot is requested) * When a task is modified, we only set the modified flag on that task. Capturing a snapshot want to avoid locking all tasks. So it first switches to the Snapshot Phase. Snapshot Phase: (after a snapshot is captured) * When a task is modified: * If there is already a snapshot, skip * If the task has the modified flag set, clone the task and store the snapshot. * If the task doesn't have the modified flag set, store a None snapshot. To actually capture the snapshot, go over all modified or snapshot tasks (which is stored in a separate map): * If the task has a snapshot (not None) stored, use that. * If the task has only the modified flag set, read the task state and capture a snapshot from that. * There is a race condition here, when the task is modified inbetween. We handle that by having a snapshot flag too. We have all snapshot captured now. To leave Snapshot Phase, we * First unset all modified flags. * Switch to Idle Phase. * Change all stored snapshots to modified.
Showing
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
Please
register
or
sign in
to comment