cjw/Oxicloud
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

fix(storage): a read failure is not proof the blob is gone

Browse Source 
Ed's point, and the most dangerous bug in the batch: NotFound is a
conclusion callers ACT on. Every read path in all three backends
returned it unconditionally.

    // s3, azure, local — all of them
    .map_err(|e| DomainError::new(ErrorKind::NotFound, …))

So a refused connection, a 503, an expired credential, a stale NFS
handle and an unmounted iSCSI target all reported "blob missing". Nine
sites: get / get-range / stat on each backend.

## Why it is disastrous rather than untidy

`backend_migration` probes its source before copying. A transient probe
error used to `continue` — skip the row, record NOTHING, and let the
cursor advance past it at the end of the batch. With `failed` still 0
the run reached `finish_completed` and FLIPPED THE POINTER to a target
missing every blob the outage covered. A migration reporting success
having silently dropped whatever was unreachable at the time.

That path now pauses when the probe error is transient, and records a
finding when it is permanent, so a run can no longer report clean while
having skipped rows.

## Local storage is not exempt

Ed again: a local backend is a PATH, and that path may be an iSCSI or
NVMe-oF LUN, an NFS mount, or a disk with a failing sector. It matters
MORE there than for a remote backend, because `RetryBlobBackend` is only
applied when the active backend is not Local — nothing below retries, so
the classification is the only thing between a flaky mount and a run
concluding the data is gone.

`local_io_error` maps the network-mount family (TimedOut,
HostUnreachable, NetworkDown, ConnectionReset, StaleNetworkFileHandle)
plus Interrupted and ResourceBusy to transient. PermissionDenied,
ReadOnlyFilesystem and StorageFull stay permanent because retrying
changes nothing without an operator, and InvalidData stays permanent
because corruption is a finding worth keeping. A bad sector arrives as
an uncategorised EIO and lands there too, which is right: the useful
outcome is a finding naming the blob, not a run that waits for a disk to
heal.

## Shape of the fix

Only a genuine absence is NotFound — `NoSuchKey` on S3 GET,
`is_not_found` on S3 HEAD, HTTP 404 on Azure, `ErrorKind::NotFound` on
local. Everything else goes through the classifier, so a 403 stays
permanent rather than being retried forever.

Tested at the local layer, which is where the mapping table is dense
enough to get wrong.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Edouard Vanbelle
2026-09-07 22:15:40 +02:00
parent 0cdb2bb0a9
commit 34a2607658
4 changed files with 262 additions and 59 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/infrastructure/services/azure_blob_backend.rs
+35 -16
View File
@@ -239,11 +239,7 @@ impl BlobStorageBackend for AzureBlobBackend {
let first = match pages.next().await {
Some(Ok(response)) => response,
Some(Err(e)) => {
return Err(DomainError::new(
ErrorKind::NotFound,
"Azure",
format!("Failed to get blob {hash}: {e}"),
));
return Err(azure_read_error(format!("Failed to get blob {hash}"), &e));
}
None => {
let empty: BlobStream =
@@ -324,10 +320,9 @@ impl BlobStorageBackend for AzureBlobBackend {
let first = match pages.next().await {
Some(Ok(response)) => response,
Some(Err(e)) => {
return Err(DomainError::new(
ErrorKind::NotFound,
"Azure",
format!("Failed to get blob range {hash}: {e}"),
return Err(azure_read_error(
format!("Failed to get blob range {hash}"),
&e,
));
}
None => {
@@ -415,13 +410,10 @@ impl BlobStorageBackend for AzureBlobBackend {
let hash = hash.to_owned();
Box::pin(async move {
let client = self.blob_client(&hash);
let props = client.get_properties().await.map_err(|e| {
DomainError::new(
ErrorKind::NotFound,
"Azure",
format!("Failed to stat blob {hash}: {e}"),
)
})?;
let props = client
.get_properties()
.await
.map_err(|e| azure_read_error(format!("Failed to stat blob {hash}"), &e))?;
Ok(props.blob.properties.content_length)
})
}
@@ -683,6 +675,33 @@ impl BlobStorageBackend for AzureBlobBackend {
/// second wearing the clothes of the first. So the policy is to retry
/// as if transient and let the bounded attempt cap turn the difference
/// into a Paused run an operator can act on.
/// Read-path variant of [`azure_domain_error`]: only a real 404 is
/// `NotFound`.
///
/// Every Azure read used to label EVERY failure `NotFound` — a refused
/// connection, a 503, an expired SAS token all reported as "blob
/// missing". That is the most dangerous wrong answer available on a read
/// path, because callers ACT on NotFound by concluding the bytes are
/// gone: a migration reading its source would treat an outage as "the
/// source does not have this blob" and move past it.
///
/// Everything that is not a 404 goes through the normal classifier, so a
/// 403 stays permanent instead of being retried.
pub(crate) fn azure_read_error(context: String, err: &azure_core::Error) -> DomainError {
use azure_core::error::ErrorKind as AzKind;
if let AzKind::HttpResponse { status, .. } = err.kind()
&& u16::from(*status) == 404
{
return DomainError::new(
ErrorKind::NotFound,
"Azure",
format!("{context}: not found"),
);
}
azure_domain_error(context, err)
}
pub(crate) fn azure_domain_error(context: String, err: &azure_core::Error) -> DomainError {
use azure_core::error::ErrorKind as AzKind;
src/infrastructure/services/backend_migration_service.rs
+58 -7
View File
@@ -690,21 +690,72 @@ impl RecoverableJobHandler for BackendMigrationService {
.await;
continue;
}
Err(e) if e.is_transient() => {
// PAUSE. Skipping here was a data-loss path.
//
// The old comment called this "a network blip"
// and `continue`d, reasoning that a re-run would
// re-probe. It would not: the cursor advances to
// the batch's last hash regardless, so a skipped
// row is never revisited by THIS run — and unlike
// a copy failure it recorded no finding, so
// `failed` stayed 0, the run reached
// `finish_completed`, and the pointer flipped to
// a target missing every blob the outage
// covered.
//
// That is the worst shape available: a migration
// reporting success while having silently
// dropped whatever was unreachable at the time.
tracing::warn!(
target: "oxicloud::migration",
event = "backend_migration.source_unreachable",
run_id = %store.run_id(),
hash = %hash,
copied = copied_count,
error = %e,
"source unreachable while probing; pausing at the last checkpoint"
);
return RunOutcome::from_domain_error(
cursor.as_ref().map(|s| s.as_bytes()),
&format!(
"source unreachable while probing ({copied_count} blob(s) \
copied so far)"
),
&e,
);
}
Err(e) => {
// Transient probe failure on source is NOT a
// finding — treat like a network blip.
// Skipping this row on this run; a re-run
// will re-probe. If the failure is
// persistent, `blobs_consistency` catches
// it.
// Permanent probe failure. Still skipped rather
// than fatal — one unprobeable blob must not
// abort the migration — but it now records a
// finding, so the run cannot report clean while
// having skipped rows, and `blobs_consistency`
// is not the only thing that would ever notice.
tracing::warn!(
target: "oxicloud::migration",
event = "backend_migration.source_probe_error",
run_id = %store.run_id(),
hash = %hash,
error = %e,
"source blob_exists probe failed; skipping this row"
"source blob_exists probe failed; recording finding, skipping row"
);
failed_count += 1;
record_or_log(
store,
BACKEND_MIGRATION_JOB_NAME,
"migration_failed",
"data_loss",
None,
serde_json::json!({
"hash": hash,
"size": size,
"source": source_kind,
"target": target_kind,
"error": format!("source probe failed: {e}"),
}),
)
.await;
continue;
}
}
src/infrastructure/services/local_blob_backend.rs
+117 -21
View File
@@ -615,13 +615,11 @@ impl BlobStorageBackend for LocalBlobBackend {
let hash = hash.to_owned();
Box::pin(async move {
let blob_path = self.blob_path(&hash);
let file = File::open(&blob_path).await.map_err(|e| {
DomainError::new(
ErrorKind::NotFound,
"Blob",
format!("Failed to open blob {}: {}", hash, e),
)
})?;
// Was unconditional NotFound: a stale NFS handle or an
// unmounted iSCSI target reported the blob as missing.
let file = File::open(&blob_path)
.await
.map_err(|e| local_io_error("Blob", format!("Failed to open blob {hash}"), &e))?;
Ok(Box::pin(ReaderStream::with_capacity(file, STREAM_CHUNK_SIZE)) as BlobStream)
})
}
@@ -636,13 +634,9 @@ impl BlobStorageBackend for LocalBlobBackend {
let hash = hash.to_owned();
Box::pin(async move {
let blob_path = self.blob_path(&hash);
let mut file = File::open(&blob_path).await.map_err(|e| {
DomainError::new(
ErrorKind::NotFound,
"Blob",
format!("Failed to open blob {}: {}", hash, e),
)
})?;
let mut file = File::open(&blob_path)
.await
.map_err(|e| local_io_error("Blob", format!("Failed to open blob {hash}"), &e))?;
file.seek(std::io::SeekFrom::Start(start))
.await
@@ -697,13 +691,9 @@ impl BlobStorageBackend for LocalBlobBackend {
let hash = hash.to_owned();
Box::pin(async move {
let blob_path = self.blob_path(&hash);
let meta = fs::metadata(&blob_path).await.map_err(|e| {
DomainError::new(
ErrorKind::NotFound,
"Blob",
format!("Failed to stat blob {}: {}", hash, e),
)
})?;
let meta = fs::metadata(&blob_path)
.await
.map_err(|e| local_io_error("Blob", format!("Failed to stat blob {hash}"), &e))?;
Ok(meta.len())
})
}
@@ -918,6 +908,60 @@ impl BlobStorageBackend for LocalBlobBackend {
}
}
/// Classify a filesystem error, because "local" does not mean
/// "reliable".
///
/// A local backend is a PATH, and that path may be an iSCSI or NVMe-oF
/// LUN, an NFS mount, or a disk with a failing sector. Those produce
/// errors that clear on their own exactly like a remote 503 does, and
/// treating every one as permanent means a migration off a briefly
/// unreachable mount records data-loss findings for blobs that are
/// perfectly intact.
///
/// It matters more here than for a remote backend, because
/// `RetryBlobBackend` is only applied when the active backend is NOT
/// Local (`di.rs`) — so nothing below this retries, and this
/// classification is the only thing standing between a flaky mount and
/// a run that concludes the data is gone.
///
/// **`NotFound` stays `NotFound`, and nothing else becomes it.** Callers
/// act on that variant by concluding the bytes do not exist.
///
/// Transient: the network-mount family (timeouts, unreachable, reset,
/// stale handle) plus `Interrupted` (EINTR) and `ResourceBusy` (EBUSY).
///
/// Permanent, deliberately: `PermissionDenied` and
/// `ReadOnlyFilesystem` need an operator, retrying changes nothing.
/// `StorageFull` likewise. `InvalidData` is corruption, which is a
/// finding worth keeping. A bad sector surfaces as an uncategorised EIO
/// and therefore lands here too — right, because the useful outcome is
/// a `blob_corrupted`-style finding naming the blob, not a run that
/// pauses forever waiting for a disk to heal.
pub(crate) fn local_io_error(
entity: &'static str,
context: String,
err: &std::io::Error,
) -> DomainError {
use std::io::ErrorKind as Io;
let message = format!("{context}: {err}");
match err.kind() {
Io::NotFound => DomainError::new(ErrorKind::NotFound, entity, message),
Io::TimedOut
| Io::HostUnreachable
| Io::NetworkUnreachable
| Io::NetworkDown
| Io::ConnectionReset
| Io::ConnectionAborted
| Io::NotConnected
| Io::BrokenPipe
| Io::StaleNetworkFileHandle
| Io::Interrupted
| Io::ResourceBusy => DomainError::transient_backend(entity, message),
_ => DomainError::internal_error(entity, message),
}
}
#[cfg(test)]
mod tests {
use super::*;
@@ -1085,4 +1129,56 @@ mod tests {
"resume must start STRICTLY after the given hash"
);
}
/// "Local" does not mean reliable — the path can be an iSCSI LUN or
/// an NFS mount. The two directions this must never confuse:
///
/// * a genuinely absent file must stay `NotFound`, because callers
/// act on that by concluding the bytes do not exist;
/// * an unreachable mount must NOT become `NotFound`, which is what
/// every one of these sites used to return unconditionally.
#[test]
fn local_io_errors_are_classified_not_all_notfound() {
use std::io::{Error, ErrorKind as Io};
let missing = local_io_error("Blob", "open".into(), &Error::from(Io::NotFound));
assert_eq!(missing.kind, ErrorKind::NotFound);
assert!(!missing.is_transient());
// Network-backed mounts and interrupted syscalls: retry helps.
for kind in [
Io::TimedOut,
Io::HostUnreachable,
Io::NetworkDown,
Io::ConnectionReset,
Io::StaleNetworkFileHandle,
Io::Interrupted,
Io::ResourceBusy,
] {
let e = local_io_error("Blob", "open".into(), &Error::from(kind));
assert!(e.is_transient(), "{kind:?} should be retryable");
assert_ne!(
e.kind,
ErrorKind::NotFound,
"{kind:?} must never read as a missing blob"
);
}
// Operator-action or corruption: retrying changes nothing, and a
// finding naming the blob is the useful outcome.
for kind in [
Io::PermissionDenied,
Io::ReadOnlyFilesystem,
Io::StorageFull,
Io::InvalidData,
] {
let e = local_io_error("Blob", "open".into(), &Error::from(kind));
assert!(!e.is_transient(), "{kind:?} should not be retryable");
assert_ne!(
e.kind,
ErrorKind::NotFound,
"{kind:?} is not a missing blob"
);
}
}
}
src/infrastructure/services/s3_blob_backend.rs
+52 -15
View File
@@ -286,11 +286,30 @@ impl BlobStorageBackend for S3BlobBackend {
.send()
.await
.map_err(|e| {
DomainError::new(
ErrorKind::NotFound,
"S3",
format!("Failed to get blob {}: {}", hash, e),
)
// Only a real NoSuchKey is NotFound. This used to
// label EVERY read failure that way — a refused
// connection, a 503, an expired credential all
// reported as "blob missing".
//
// That is the most dangerous wrong answer available
// here, because callers ACT on NotFound by concluding
// the bytes are gone. A migration reading its source
// through this would treat an outage as "the source
// does not have this blob" and move on.
//
// Everything else goes through the normal classifier,
// so a 403 stays permanent rather than being retried
// forever.
if let aws_sdk_s3::error::SdkError::ServiceError(svc) = &e
&& svc.err().is_no_such_key()
{
return DomainError::new(
ErrorKind::NotFound,
"S3",
format!("Failed to get blob {hash}: no such key"),
);
}
s3_domain_error("S3", format!("Failed to get blob {hash}"), &e)
})?;
// Convert S3 ByteStream into a Stream<Item = Result<Bytes, io::Error>>
@@ -325,11 +344,21 @@ impl BlobStorageBackend for S3BlobBackend {
.send()
.await
.map_err(|e| {
DomainError::new(
ErrorKind::NotFound,
"S3",
format!("Failed to get blob range {}: {}", hash, e),
)
// Same rule as the full read: only a real NoSuchKey
// is NotFound. Ranged reads feed CDC reassembly and
// deep verification, so mislabelling an outage here
// reads as "this chunk is gone" — a data-loss
// conclusion drawn from a network problem.
if let aws_sdk_s3::error::SdkError::ServiceError(svc) = &e
&& svc.err().is_no_such_key()
{
return DomainError::new(
ErrorKind::NotFound,
"S3",
format!("Failed to get blob range {hash}: no such key"),
);
}
s3_domain_error("S3", format!("Failed to get blob range {hash}"), &e)
})?;
let reader = output.body.into_async_read();
@@ -407,11 +436,19 @@ impl BlobStorageBackend for S3BlobBackend {
.send()
.await
.map_err(|e| {
DomainError::new(
ErrorKind::NotFound,
"S3",
format!("Failed to stat blob {}: {}", hash, e),
)
// `head_object` reports a missing key as NotFound
// rather than NoSuchKey, so match on the typed
// variant the SDK actually returns here.
if let aws_sdk_s3::error::SdkError::ServiceError(svc) = &e
&& svc.err().is_not_found()
{
return DomainError::new(
ErrorKind::NotFound,
"S3",
format!("Failed to stat blob {hash}: not found"),
);
}
s3_domain_error("S3", format!("Failed to stat blob {hash}"), &e)
})?;
Ok(output.content_length().unwrap_or(0) as u64)