Unverified Commit f263c562 authored by Benjamin Woodruff's avatar Benjamin Woodruff Committed by GitHub

Turbopack: Various cleanup for turbo-tasks-fs, mostly retry logic and string formatting (#88668)

- https://github.com/vercel/next.js/pull/87661 removed the `tokio::task::spawn_blocking` call from `retry_blocking`, so we can simplify things a lot by removing the `Send + 'static` bounds, and eliminate a bunch of cloning `Path`s into `PathBuf`s.
- `retry_blocking` no longer takes a path argument. After removing the path cloning requirements, it's now easier for callers to capture paths via a closure, and it's more flexible for IO operations that need to pass in more than a single path.
- Adds `retry_blocking_custom`, which I use in #88669 that allows customizing what error kinds we retry on.
- Adds documentation to `retry_blocking` explaining why we need this. It's often unclear to new people on the team why we have to retry filesystem operations (Windows, basically). This was unclear to me when I was new to this code, and other people have asked similar questions.
- We shouldn't use `Path::display` for error messages or tracing spans. It's easier to use `Debug`, and `Debug` ensures that if the error is related to unicode, that information isn't lost in the error message.
- More consistently apply tracing spans using `.instrument`.
- Use the modern inline syntax for some format strings.
- `let foo = RcStr::from(bar)` is preferred over `let foo: RcStr = bar.into()`
parent 4674a435
This diff is collapsed.
use std::{
io::{self, ErrorKind},
path::{Path, PathBuf},
time::Duration,
};
const MAX_RETRY_ATTEMPTS: usize = 10;
pub(crate) async fn retry_blocking<R, F>(path: PathBuf, func: F) -> io::Result<R>
where
F: Fn(&Path) -> io::Result<R> + Send + 'static,
R: Send + 'static,
{
let mut attempt = 1;
/// Retries a blocking io operation up to `MAX_RETRY_ATTEMPTS` in a loop. Retries upon
/// [`ErrorKind::PermissionDenied`] or [`ErrorKind::WouldBlock`]. This default behavior is
/// implemented by [`can_retry`].
///
/// Retry logic is rarely useful on POSIX operating systems, but is often useful on Windows with
/// NTFS. E.g. a file deletion may fail because an AV process has opened the file, and opened files
/// cannot be deleted on Windows. Retries are the only mitigation for such issues.
///
/// This used to use [`tokio::task::spawn_blocking`], but now the IO operation blocks the current
/// thread, as we already wrap callers of this function in
/// [`crate::ConcurrencyLimitedExt::concurrency_limited`] and it
/// appears to be faster to block the tokio thread than to spawn. Most of our IO operations are very
/// short anyways. See this PR: <https://github.com/vercel/next.js/pull/87661>.
pub(crate) async fn retry_blocking<T>(func: impl FnMut() -> io::Result<T>) -> io::Result<T> {
retry_blocking_custom(func, can_retry).await
}
/// A customizable version of [`retry_blocking`] that allows retrying on arbitrary errors.
pub(crate) async fn retry_blocking_custom<T, E>(
mut func: impl FnMut() -> Result<T, E>,
mut can_retry: impl FnMut(&E) -> bool,
) -> Result<T, E> {
let mut attempt = 1;
loop {
return match func(&path) {
Ok(r) => Ok(r),
return match func() {
Ok(val) => Ok(val),
Err(err) => {
if attempt < MAX_RETRY_ATTEMPTS && can_retry(&err) {
tokio::time::sleep(get_retry_wait_time(attempt)).await;
......@@ -29,7 +44,9 @@ where
}
}
fn can_retry(err: &io::Error) -> bool {
/// The default implementation of error checking used by [`retry_blocking`]. Returns true if it
/// would make sense to retry a failed IO operation.
pub fn can_retry(err: &io::Error) -> bool {
matches!(
err.kind(),
ErrorKind::PermissionDenied | ErrorKind::WouldBlock
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment