Skip to content

Idea: derived signals (smoothing / filtering as a new named Signal) #97

Description

@AlexisJanin

Idea, not a plan. Recording a design discussion so it does not have to be re-derived. Nothing here is scheduled; no PR should close this. Came out of #96 (candidate plot types), where it was noted that several "can you plot X" requests are really "plot a computed signal".

The idea

Let a database_options entry produce a new, named Signal from an existing one — smoothing, low-pass, derivative — instead of a new plot type. A derived Signal composes with time_series, loop, psd, spectrogram and the planned histogram all at once, where a plot type buys one behaviour.

Three shapes considered

  • (a) In-place transform, as a per-signal display option — a smoothing key next to period_resampling in signals.<name>. Discarded. It silently changes the numbers under every plot type while the trace still claims to be the measurement.
  • (b) A new derived Signal — the transform produces an additional Signal with a new name, groupable alongside its parent. Preferred. The clinician cannot apply it by accident: they had to name the thing. Metadata.is_derived and Metadata.parent_signal_name already exist for this and are currently set nowhere.
  • (c) Smoothing as a trace_options rendering concern — data untouched, only the drawing smoothed. Cheap and honest about the underlying data, but risks the same misreading as (a): what is on screen is not what was measured.

Prior art already in the tree

period_resampling is exactly shape (a), shipped: configured per signal (with a numerics global default) and applied at Signal construction in signal_container.time_series_from_dataframe. It also carries the warning: spectral._framed_power refuses to compute a PSD or spectrogram on a decimated signal, because step-decimation with no anti-alias filter shows aliased energy that reads as a real rhythm. One in-place transform already forced a hand-written guard into every spectral type.

Caveats found

The one that undercuts the premise:

  • (b) relocates the trap rather than removing it. A derived Signal is first-class, so it can be fed to psd/spectrogram like a measured one — and a smoothed signal's PSD is still a lie about the roll-off, which the existing guard will not catch because it is not decimation. The transform descriptor on the Signal, and the spectral types reading it, are therefore part of this feature, not a follow-up.
  • Metadata inheritance is a correctness rule, not a convenience. A derived signal built from a decimated parent that does not inherit period_resampling silently bypasses the existing refusal. "Inherit everything" is wrong too — smoothing preserves units, a derivative does not.

Two forks to settle before building:

  • Display or analysis? extract stops at format and returns DataFrames; Signals are built afterwards — so a derived signal would appear in plots and be absent from extract_patient output. Pushing derivation into _format does not rescue it: _format is per-datasource, and a parent may live in another one. Display-only means the Signal layer and is cheap; analysis means a DataFrame-level step both pipelines call, and is a much larger build.
  • Where derivation runs argues with ADR-0013. It must run after the datasource loop (a parent may be elsewhere) and before assemble_plot_groups (so groups and derived plots can reference the new signal) — but ADR-0013 puts reference qualification inside assembly's first step, and a derived entry's own references need qualifying before that. Either qualification is hoisted into a phase both call, or the ADR is amended.

Manageable, but they will surface:

  • Every derived signal gets its own subplot by default — assemble_plot_groups gives a plot to any signal no configured group took. "Smoothed ABP over raw ABP" is therefore two config edits for one intent; wants a shorthand.
  • raw_name stops meaning what CONTEXT.md says it means ("its identifier in the source data"). A derived signal's raw_name is in no file — a glossary amendment, not just code.
  • The xlsx half is not free. Plot types get their sheet via SHEET_NAME + CellReader on the definition; a derived-signal section is not a plot type and inherits none of it, so it needs its own reader in database_options_xlsx.py plus the .json parity tests/unit/test_example_assets.py enforces.
  • Chaining and missing parents. Smooth-then-differentiate needs dependency ordering and cycle detection; and since wrapper.main skips a datasource that fails to load, a parent can legitimately be absent at runtime — derivation must degrade with a logged refusal like the derived plot types do, not crash.
  • Annotations key to plot_name plus {datasource_name, raw_name, display_name}, so renaming a derived signal orphans annotations on it. True of groups today, but database_options is shared and edited while annotations.json is per-patient and persistent, so churn is likelier here.

Scope discipline if it is ever built

One parent, one operation, no arithmetic between signals. Two-parent arithmetic needs a common time grid across sources with different sample rates, and resampling onto that grid reintroduces exactly the aliasing problem the spectral guard already refuses. That deserves its own decision, not a config key.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions