mlflow-alerts

A zero-latency, asynchronous alerting and notification plugin framework for MLflow.

This repository is currently in its architectural blueprint and testing phase. The initial test suites are established to define the system's contract and verify the core routing logic before completing the full implementation.

---

🎯 Project Intent

MLflow provides excellent experiment logging and model tracking but lacks native tracking store alerting. Standard webhook mechanisms only cover model registry state changes and require external middleware to format payloads for chat applications.

mlflow-alerts bridges this gap by intercepting MLflow lifecycle events natively. It alerts developers instantly when:

• A validation metric drops below a critical threshold during a training loop.
• A distributed pipeline run crashes or transitions to a FAILED state.
• A model version receives a specific alias tag (e.g., champion), automatically triggering GitOps CI/CD deployment pipelines.

---

🏗️ Architectural Decisions

To prevent the performance bottlenecks typical of custom middleware integrations, the project is bound to two strict design constraints:

1. Zero-Latency Asynchronous Handoff

Logging hundreds of metrics per epoch should not slow down model training. The plugin hooks directly into MLflow's tracking store operations, but it handles data allocations using an isolated, asynchronous, in-memory background worker queue.

• The Result: The plugin hands off the notification packet instantly. The core MLflow database operation returns immediately, ensuring that training loops never stall waiting for network I/O.

2. Throttling & Network Isolation

External chat apps (Slack, Discord) and GitOps endpoints (GitHub Actions) impose strict API rate limits.

• The Result: The background worker acts as a buffer. It serializes, processes, and network-throttles outgoing HTTP payloads. If an external service times out or hits a rate limit, the error is isolated to the background thread. The active training job never encounters a secondary exception or a crash.

---

🧪 Current State & Test Coverage

The repository contains a suite of mock-isolated integration tests written with pytest and unittest.mock to validate the core architecture and routing logic.

The tests enforce:

• test_dispatcher.py: Ensures the background queue dispatches payloads asynchronously and handles network timeouts gracefully.
• test_tracking_store.py: Verifies metrics violating custom thresholds successfully queue alerts while safe metrics are ignored.
• test_registry_store.py: Confirms that model alias promotions map correctly to structured GitHub Actions dispatch payloads.

---

🚀 Installation & Usage

pip install -e .
# or from pypi once published: pip install mlflow-alerts

The package registers as an MLflow plugin under the mlflow-alerts scheme (safe opt-in, does not override core stores by default).

Quick start (direct use in scripts / tests)

import os
import mlflow
from mlflow_alerts import AlertingTrackingStore, AlertingRegistryStore

os.environ["SLACK_WEBHOOK_URL"] = "https://hooks.slack.com/services/..."
os.environ["ALERT_ACCURACY_MIN"] = "0.85"

# Option A: explicit store (for sqlite file backend or testing)
tracking_store = AlertingTrackingStore(db_uri="sqlite:///mlflow.db", artifact_root="./mlruns")
# Use low-level or monkey the client if needed; most users will rely on URI scheme.

# Option B: use the plugin scheme (recommended for server + clients)
mlflow.set_tracking_uri("mlflow-alerts+sqlite:///mlflow.db")
# For a remote server: "mlflow-alerts+postgresql://user:pass@host/mlflow"
# The wrapper unwraps the prefix and delegates to the real backend while adding alerts.

# Same idea for registry if using separate registry URI
mlflow.set_registry_uri("mlflow-alerts+sqlite:///mlflow.db")

Environment configuration (current v0 rules)

• SLACK_WEBHOOK_URL: target for metric and run failure alerts (simple {"text": "..."} payloads).
• ALERT_ACCURACY_MIN: threshold for the val_accuracy metric (example rule; extendable in future).
• GITHUB_DISPATCH_URL (optional): for champion alias GitHub Actions repository_dispatch style payloads.

When a rule matches, the payload is handed off to a background thread immediately. The original log_metric / set_terminated / set_registered_model_alias call returns without waiting for the network request.

Running the tests

pip install -e ".[test]"
python -m pytest tests/ -q

See the test sources for the precise contract and payload shapes.

Limitations / notes

• Currently only a small number of events are intercepted (see tests).
• The entrypoint registration for the mlflow-alerts scheme may emit a benign warning in some MLflow 3.x versions due to internal plugin loading order (the stores remain fully usable).
• For production rate limiting, richer rules, more channels, and log_batch coverage see Phase 1+ work.
• Compared to native MLflow webhooks (Model Registry only): this plugin uniquely covers training-time metric thresholds and run status changes with zero added latency on the hot path.

---

Development

See the detailed implementation plan in the session plan file for architecture rationale, alternatives considered, and next steps (hardening, config, more events).

Contributions should keep the "never block the training loop" and "errors isolated to background" invariants.