Skip to content

[fm] Add simple disk diagnoser based on zpool health #10460

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
smklein wants to merge 11 commits into
base: main
Choose a base branch
from
Open

[fm] Add simple disk diagnoser based on zpool health #10460

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
793b1ec
[fm] Add simple disk diagnoser based on zpool health
smklein May 18, 2026
dfabfb4
Debuggable facts
smklein May 19, 2026
7555b0b
more logging
smklein May 19, 2026
bcf5b68
CaseFact -> Fact, add created_sitrep_id to facts, fm- prefix
smklein May 19, 2026
a415d27
payload_to, deserialization error context
smklein May 19, 2026
58c780f
case formatting
smklein May 19, 2026
bf5988f
s/Disk/PhysicalDisk
smklein May 20, 2026
325e34b
merge
smklein May 20, 2026
b016d7c
Reuse bulleted JSON formatter for fact payload display
smklein May 20, 2026
aafbf9d
Facts don't case about your prefixes?
smklein May 20, 2026
d021ea9
comments
smklein May 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 28 additions & 18 deletions dev-tools/omdb/tests/successes.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -700,17 +700,18 @@ task: "fm_analysis"
configured period: every <REDACTED_DURATION>m
last completed activation: <REDACTED ITERATIONS>, triggered by <TRIGGERED_BY_REDACTED>
started at <REDACTED_TIMESTAMP> (<REDACTED DURATION>s ago) and ran for <REDACTED DURATION>ms
parent sitrep ID: None
parent sitrep ID: Some(..........<REDACTED_UUID>........... (sitrep))
current inventory collection ID: Some(..........<REDACTED_UUID>........... (collection))
ereport classes consumed: (none)
FAULT MANAGEMENT ANALYSIS SUMMARY
=================================
/!\ analysis failed: FM analysis is not yet implemented
no changes from the current situation report (Some(..........<REDACTED_UUID>........... (sitrep)))
Comment on lines +703 to +708
Copy link
Copy Markdown
Member

@hawkw hawkw May 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm, do we need to be redacting the Some/None-ness of this, as well? it seems plausibly flaky that the OMDB tests may run before there has ever been a sitrep generated, or after we have made the first one?

Copy link
Copy Markdown
Collaborator Author

@smklein smklein May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ugh I don't really want to redact this... How about just waiting until a single sitrep exists?


fault management analysis inputs
--------------------------------
parent sitrep: <none>
parent sitrep: ..........<REDACTED_UUID>...........
inventory collection: ..........<REDACTED_UUID>...........
<FM_INPUT_INV_COMPARISON_REDACTED>
no new ereports since the parent sitrep
no cases copied forward

Expand All @@ -724,28 +725,28 @@ task: "fm_rendezvous"
configured period: every <REDACTED_DURATION>m
last completed activation: <REDACTED ITERATIONS>, triggered by <TRIGGERED_BY_REDACTED>
started at <REDACTED_TIMESTAMP> (<REDACTED DURATION>s ago) and ran for <REDACTED DURATION>ms
(i) no FM situation report loaded, so rendezvous was not performed
current sitrep: ..........<REDACTED_UUID>...........
creating requested alerts:
(i) note: this operation was not executed
started at <REDACTED_TIMESTAMP> (<REDACTED DURATION>s ago) and ran for <REDACTED DURATION>ms
alerts requested: 0
requested in this sitrep: 0
created in this activation: 0
already created: 0
errors: 0
creating requested support bundles:
(i) note: this operation was not executed
started at <REDACTED_TIMESTAMP> (<REDACTED DURATION>s ago) and ran for <REDACTED DURATION>ms
support bundles requested: 0
requested in this sitrep: 0
created in this activation: 0
already created: 0
errors: 0
marking ereports as seen:
(i) note: this operation was not executed
started at <REDACTED_TIMESTAMP> (<REDACTED DURATION>s ago) and ran for <REDACTED DURATION>ms
total ereports in sitrep: 0
not marked when the sitrep was loaded: 0
marked seen by this activation: 0
already marked seen: 0
batch size: 0
batch size: 1000
batches: 0
errors: 0

Expand All @@ -760,6 +761,8 @@ task: "fm_sitrep_gc"
batches: 1
orphaned fm_case rows deleted: 0
batches: 1
orphaned fm_fact rows deleted: 0
batches: 1
orphaned fm_ereport_in_case rows deleted: 0
batches: 1
orphaned fm_support_bundle_request rows deleted: 0
Expand All @@ -775,7 +778,9 @@ task: "fm_sitrep_loader"
configured period: every <REDACTED_DURATION>s
last completed activation: <REDACTED ITERATIONS>, triggered by <TRIGGERED_BY_REDACTED>
started at <REDACTED_TIMESTAMP> (<REDACTED DURATION>s ago) and ran for <REDACTED DURATION>ms
no FM situation report available to load
loaded latest FM situation report as of <REDACTED_TIMESTAMP>:
sitrep ..........<REDACTED_UUID>........... (sitrep) (v1)
made current at: <REDACTED_TIMESTAMP>

task: "instance_reincarnation"
configured period: every <REDACTED_DURATION>m
Expand Down Expand Up @@ -1391,17 +1396,18 @@ task: "fm_analysis"
configured period: every <REDACTED_DURATION>m
last completed activation: <REDACTED ITERATIONS>, triggered by <TRIGGERED_BY_REDACTED>
started at <REDACTED_TIMESTAMP> (<REDACTED DURATION>s ago) and ran for <REDACTED DURATION>ms
parent sitrep ID: None
parent sitrep ID: Some(..........<REDACTED_UUID>........... (sitrep))
current inventory collection ID: Some(..........<REDACTED_UUID>........... (collection))
ereport classes consumed: (none)
FAULT MANAGEMENT ANALYSIS SUMMARY
=================================
/!\ analysis failed: FM analysis is not yet implemented
no changes from the current situation report (Some(..........<REDACTED_UUID>........... (sitrep)))

fault management analysis inputs
--------------------------------
parent sitrep: <none>
parent sitrep: ..........<REDACTED_UUID>...........
inventory collection: ..........<REDACTED_UUID>...........
<FM_INPUT_INV_COMPARISON_REDACTED>
no new ereports since the parent sitrep
no cases copied forward

Expand All @@ -1415,28 +1421,28 @@ task: "fm_rendezvous"
configured period: every <REDACTED_DURATION>m
last completed activation: <REDACTED ITERATIONS>, triggered by <TRIGGERED_BY_REDACTED>
started at <REDACTED_TIMESTAMP> (<REDACTED DURATION>s ago) and ran for <REDACTED DURATION>ms
(i) no FM situation report loaded, so rendezvous was not performed
current sitrep: ..........<REDACTED_UUID>...........
creating requested alerts:
(i) note: this operation was not executed
started at <REDACTED_TIMESTAMP> (<REDACTED DURATION>s ago) and ran for <REDACTED DURATION>ms
alerts requested: 0
requested in this sitrep: 0
created in this activation: 0
already created: 0
errors: 0
creating requested support bundles:
(i) note: this operation was not executed
started at <REDACTED_TIMESTAMP> (<REDACTED DURATION>s ago) and ran for <REDACTED DURATION>ms
support bundles requested: 0
requested in this sitrep: 0
created in this activation: 0
already created: 0
errors: 0
marking ereports as seen:
(i) note: this operation was not executed
started at <REDACTED_TIMESTAMP> (<REDACTED DURATION>s ago) and ran for <REDACTED DURATION>ms
total ereports in sitrep: 0
not marked when the sitrep was loaded: 0
marked seen by this activation: 0
already marked seen: 0
batch size: 0
batch size: 1000
batches: 0
errors: 0

Expand All @@ -1451,6 +1457,8 @@ task: "fm_sitrep_gc"
batches: 1
orphaned fm_case rows deleted: 0
batches: 1
orphaned fm_fact rows deleted: 0
batches: 1
orphaned fm_ereport_in_case rows deleted: 0
batches: 1
orphaned fm_support_bundle_request rows deleted: 0
Expand All @@ -1466,7 +1474,9 @@ task: "fm_sitrep_loader"
configured period: every <REDACTED_DURATION>s
last completed activation: <REDACTED ITERATIONS>, triggered by <TRIGGERED_BY_REDACTED>
started at <REDACTED_TIMESTAMP> (<REDACTED DURATION>s ago) and ran for <REDACTED DURATION>ms
no FM situation report available to load
loaded latest FM situation report as of <REDACTED_TIMESTAMP>:
sitrep ..........<REDACTED_UUID>........... (sitrep) (v1)
made current at: <REDACTED_TIMESTAMP>

task: "instance_reincarnation"
configured period: every <REDACTED_DURATION>m
Expand Down
9 changes: 9 additions & 0 deletions dev-tools/omdb/tests/test_all_output.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -350,6 +350,15 @@ async fn test_omdb_success_cases() {
.field("triggered by", r"[\w ]+")
.section(&["task: \"tuf_artifact_replication\"", "request ringbuf:"]);

// The `fm_analysis` task's input report includes a line comparing the
// current inventory collection against the parent sitrep's collection,
// which can be either "same" or "different" depending on whether a new
// inventory was collected between sitreps. Collapse both forms.
redactor.variable_regex(
"fm_input_inv_comparison",
r" --> (same collection as parent sitrep|different from parent sitrep \(collection [-a-f0-9]+\))",
);

// The `sp_ereport_ingester` task's output depends on how many simulated
// sled agents ahppen to register with Nexus before its first execution.
// These redactions work around the issue described in
Expand Down
43 changes: 41 additions & 2 deletions nexus/db-model/src/fm/case.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@ use super::DiagnosisEngine;
use super::SupportBundleRequest;
use crate::DbTypedUuid;
use crate::ereport;
use nexus_db_schema::schema::{fm_case, fm_ereport_in_case};
use nexus_db_schema::schema::{fm_case, fm_ereport_in_case, fm_fact};
use nexus_types::fm;
use omicron_uuid_kinds::{
CaseEreportKind, CaseKind, EreporterRestartKind, SitrepKind,
CaseEreportKind, CaseKind, EreporterRestartKind, FactKind, SitrepKind,
};

/// Metadata describing a fault management case.
Expand Down Expand Up @@ -64,6 +64,7 @@ impl CaseMetadata {
alerts_requested: _,
support_bundles_requested: _,
ereports: _,
facts: _,
} = case;
Self {
sitrep_id: sitrep_id.into(),
Expand All @@ -76,6 +77,44 @@ impl CaseMetadata {
}
}

/// Diesel row for the `fm_fact` table. See
/// [`nexus_types::fm::case::Fact`] for semantics.
#[derive(Queryable, Insertable, Clone, Debug, Selectable)]
#[diesel(table_name = fm_fact)]
pub struct Fact {
pub id: DbTypedUuid<FactKind>,
/// The sitrep to which this fact belongs.
///
/// This will change as the fact is carried forward from
/// one sitrep to the next.
pub sitrep_id: DbTypedUuid<SitrepKind>,
pub case_id: DbTypedUuid<CaseKind>,
/// Sitrep in which this fact was first added.
///
/// Preserved unchanged when the fact is carried forward; debug-only.
pub created_sitrep_id: DbTypedUuid<SitrepKind>,
pub payload: serde_json::Value,
pub comment: String,
}

impl Fact {
pub fn from_sitrep(
sitrep_id: impl Into<DbTypedUuid<SitrepKind>>,
case_id: impl Into<DbTypedUuid<CaseKind>>,
fact: &fm::case::Fact,
) -> Self {
let fm::case::Fact { id, created_sitrep_id, payload, comment } = fact;
Self {
id: (*id).into(),
sitrep_id: sitrep_id.into(),
case_id: case_id.into(),
created_sitrep_id: (*created_sitrep_id).into(),
payload: payload.clone(),
comment: comment.clone(),
}
}
}

/// An association between an ereport and a case.
#[derive(Queryable, Insertable, Clone, Debug, Selectable)]
#[diesel(table_name = fm_ereport_in_case)]
Expand Down
7 changes: 7 additions & 0 deletions nexus/db-model/src/fm/diagnosis_engine.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,17 @@ impl_enum_type!(
pub enum DiagnosisEngine;

PowerShelf => b"power_shelf"
PhysicalDisk => b"physical_disk"

);

impl From<DiagnosisEngine> for fm::DiagnosisEngineKind {
fn from(de: DiagnosisEngine) -> Self {
match de {
DiagnosisEngine::PowerShelf => fm::DiagnosisEngineKind::PowerShelf,
DiagnosisEngine::PhysicalDisk => {
fm::DiagnosisEngineKind::PhysicalDisk
}
}
}
}
Expand All @@ -39,6 +43,9 @@ impl From<fm::DiagnosisEngineKind> for DiagnosisEngine {
fn from(fm_de: fm::DiagnosisEngineKind) -> Self {
match fm_de {
fm::DiagnosisEngineKind::PowerShelf => DiagnosisEngine::PowerShelf,
fm::DiagnosisEngineKind::PhysicalDisk => {
DiagnosisEngine::PhysicalDisk
}
}
}
}
Expand Down
3 changes: 2 additions & 1 deletion nexus/db-model/src/schema_versions.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ use std::{collections::BTreeMap, sync::LazyLock};
///
/// This must be updated when you change the database schema. Refer to
/// schema/crdb/README.adoc in the root of this repository for details.
pub const SCHEMA_VERSION: Version = Version::new(261, 0, 0);
pub const SCHEMA_VERSION: Version = Version::new(262, 0, 0);

/// List of all past database schema versions, in *reverse* order
///
Expand All @@ -28,6 +28,7 @@ pub static KNOWN_VERSIONS: LazyLock<Vec<KnownVersion>> = LazyLock::new(|| {
// | leaving the first copy as an example for the next person.
// v
// KnownVersion::new(next_int, "unique-dirname-with-the-sql-files"),
KnownVersion::new(262, "fm-disk-de-and-facts"),
KnownVersion::new(261, "remove-add-zones-with-mupdate-override"),
KnownVersion::new(260, "ereport-trim-serial-trailing-nulls"),
KnownVersion::new(259, "vmm-failure-reason"),
Expand Down
Loading
Loading