CKN plugin: replace oracle plugin and fix event streaming to Patra via CKN broker

[nee1k]

Summary

This branch replaces the oracle_plugin + external ckn-daemon pair with a single consolidated ckn_plugin, and fixes the event-streaming path so camera-traps experiments actually land in the Patra database (PostgreSQL) via the CKN broker and Kafka Connect JDBC sinks.

CKN plugin consolidation

• New external_plugins/ckn_plugin/ combining the oracle plugin's event tracking with the ckn-daemon's Kafka streaming (in-memory state, running experiment metrics: TP/FP/FN, precision/recall/F1, IoU, mAP)
• Installer templates, Makefile, and CI updated to build/deploy ckn_plugin instead of oracle_plugin
• Bounding-box ground truth passed from image_scoring through ctevents; power metrics recorded for newer plugins

Streaming fixes (verified end-to-end)

• Kafka Connect schema envelopes: the JDBC sink connectors run with value.converter.schemas.enable=true, so bare-JSON records failed conversion and were silently dropped (errors.tolerance=all). Oracle events and power summaries are now wrapped in {"schema": ..., "payload": ...} envelopes typed to match the patradb events/power_summary tables. (Note: Connect's JsonConverter expects "double", not "float64".)
• Power-summary shutdown ordering: the power monitor only writes power_summary_report.json after receiving PluginTerminate, but the plugin previously waited for the file before sending it — so the power event never streamed. Now: terminate first, then wait for and stream the summary, then quit.

---

Testing runbook

Reproduces the end-to-end verification: a full camera-traps experiment on a Linux host streaming per-image oracle events and the shutdown power-summary event through cknbroker → Kafka Connect JDBC sinks → the Patra PostgreSQL database (patradb).

Quick start: run_e2e_test.sh

external_plugins/ckn_plugin/run_e2e_test.sh automates everything below (build → render → run → verify → teardown). On any Linux host with Docker + Compose v2 and ~25 GB free disk:

git clone -b ckn_plugin https://github.com/tapis-project/camera-traps.git
cd camera-traps/external_plugins/ckn_plugin

# real powerjoular (reads 0 W on hosts without RAPL, e.g. most cloud VMs)
./run_e2e_test.sh

# or, to see nonzero wattage on a host without RAPL:
SYNTHETIC_POWER=1 ./run_e2e_test.sh

It prints the psql queries to run against patradb at the end (you'll need patradb credentials — see prerequisites below) and a pass/fail summary based on the ckn_plugin log (expects 12 oracle events + 1 power summary + 0 errors). Verified independently on a fresh clone with SYNTHETIC_POWER=1: 12/12 events + power summary streamed, 0 errors, confirmed in patradb.

Override identities/version with env vars if the defaults (example_user / example_device / model 2) don't fit your patradb: USER_ID, DEVICE_ID, MODEL_ID, TRAPS_REL, EXPERIMENT_ID, WORKDIR.

The manual steps below are the same thing spelled out, useful if you want to inspect or modify part of the flow.

0. Prerequisites

• Linux host (x86_64 or arm64) with Docker + Compose v2, ~25 GB free disk, outbound HTTPS (443).
• psql (or any Postgres client) somewhere with the patradb credentials — get them from the Tapis patradb pod definition or a project maintainer. Never commit them.
• The identities used by the experiment must be pre-registered in patradb, or the ingest trigger rejects every row: user_id in users.username, device_id in edge_devices.device_id. The examples below use example_user / example_device, which exist today.
• MODEL_ID accepts two forms (as of patra-knowledge-base@e72413a): either the numeric patradb models.id (e.g. 2), or camera-traps' own native model_id — a Patra model-card UUID, optionally suffixed -model (e.g. ea991e85-feaa-4781-a297-4d7bec1a69b1-model, the form the installer's own defaults and image_generating_plugin actually produce). Either way the model must already exist in patradb (models/model_cards) — this doesn't auto-register new models.
• Both Kafka Connect sinks should report RUNNING before you start:

  curl -s https://cknkafkaconnect.pods.icicleai.tapis.io/connectors/PgSinkConnectorCameraTraps/status
  curl -s https://cknkafkaconnect.pods.icicleai.tapis.io/connectors/PgSinkConnectorPowerSummary/status

1. Build images from this branch

The published tapis/ckn_plugin:0.6.0 and tapis/camera-traps-installer:0.6.0 images predate this branch (the released installer still renders the old oracle_plugin/ckn-daemon stack with a broken broker URL), so build both locally:

git clone -b ckn_plugin https://github.com/tapis-project/camera-traps
cd camera-traps

# ckn plugin with the schema-envelope + shutdown fixes
cd external_plugins/ckn_plugin
docker build -t tapis/ckn_plugin:0.6.0 --build-arg REL=0.6.0 .
cd ../..

# installer with the current templates/defaults
cd installer
docker build -t tapis/camera-traps-installer:local .
cd ..

The locally built tapis/ckn_plugin:0.6.0 shadows the Docker Hub image when compose starts.

2. Create the experiment workspace

mkdir -p ~/ct-e2e
EXPERIMENT_ID=$(python3 -c "import uuid; print(uuid.uuid4())")
echo "experiment: $EXPERIMENT_ID"   # note this down — all verification keys on it

cat > ~/ct-e2e/input.yml <<EOF
install_dir: e2e_test
use_gpu_in_scoring: false
inference_server: false
use_ultralytics: true
local_model_path: ./md_v5a.0.0.pt
user_id: example_user
device_id: example_device
model_id: "2"
experiment_id: $EXPERIMENT_ID
EOF

Why inference_server: false + local_model_path: the default megadetector-server downloads its model from the legacy patraserver endpoint at startup, and that service's Neo4j backend is retired (returns 500) — the container crashloops. The local_model_path branch skips all model downloads and mounts a local weights file instead.

3. Render the install and provide the model file

docker run --rm --user $(id -u):$(id -g) \
  -v $HOME/ct-e2e:/host \
  -e INSTALL_HOST_PATH=$HOME/ct-e2e \
  -e INPUT_FILE=input.yml \
  tapis/camera-traps-installer:local

# extract the weights baked into the ultralytics scoring image and place them
# where the rendered compose mounts them (./md_v5a.0.0.pt in the install dir)
docker pull tapis/image_scoring_plugin_ultralytics_py_3.13:0.6.0
cid=$(docker create tapis/image_scoring_plugin_ultralytics_py_3.13:0.6.0)
docker cp $cid:/md_v5a.0.0.pt ~/ct-e2e/e2e_test/md_v5a.0.0.pt
docker rm $cid

Sanity-check the render: grep -n 'CKN_KAFKA_BROKER\|EXPERIMENT_ID\|image: tapis/ckn_plugin' ~/ct-e2e/e2e_test/docker-compose.yml should show cknbroker.pods.icicleai.tapis.io:443, your experiment UUID, and tapis/ckn_plugin:0.6.0.

Note: the installer refuses to render into an existing install_dir — to rerun, move the .pt file out, rm -rf ~/ct-e2e/e2e_test, and repeat this step (with a fresh experiment_id).

4. Run the experiment

cd ~/ct-e2e/e2e_test
docker compose up -d
docker logs -f ckn_plugin

The stack processes the 12 bundled example images (~2 minutes on CPU). Expected in the ckn_plugin log, in order:

1. Successfully connected to Kafka broker! (SSL to cknbroker...:443)
2. 12× KAFKA EVENT - Publishing to topic 'oracle-events' … Successfully streamed event to Kafka for UUID: …
3. All images processed. Initiating shutdown... → Sent PluginTerminate * event
4. KAFKA POWER EVENT - Publishing to topic 'cameratraps-power-summary' → Power summary successfully streamed to Kafka. (the power monitor writes its summary within ~1 s of the terminate — this ordering is exactly what this branch fixes)
5. Clean exit (all containers end up Exited (0)).

5. Verify rows landed in patradb

export PGPASSWORD=<patradb password>
CONN="host=patradb.pods.icicleai.tapis.io port=443 dbname=patradb user=patradb sslmode=require"

# one row per image (expect 12)
psql "$CONN" -c "SELECT count(*) FROM events WHERE experiment_id='$EXPERIMENT_ID';"

# one power-summary row
psql "$CONN" -x -c "SELECT * FROM power_summary WHERE experiment_id='$EXPERIMENT_ID';"

# normalized experiment row created by the ingest trigger, with running metrics
psql "$CONN" -x -c "SELECT experiment_uid, user_id, edge_device_id, model_id,
  total_images, total_predictions, precision, recall, f1_score
  FROM experiments WHERE experiment_uid='$EXPERIMENT_ID';"

Expected: 12 events rows; 1 power_summary row; 1 experiments row (total_images=12, nonzero metrics). The experiments row updates live during the run, so you can poll it mid-experiment. Also confirm the Connect worker logged no converter/DB errors for your records.

6. Teardown

cd ~/ct-e2e/e2e_test && docker compose down

Known caveats

• Power wattage is 0 on VMs: powerjoular/scaphandre need Intel RAPL (or Jetson jtop). Inside cloud VMs (Jetstream etc.) the counters aren't exposed, so the power event streams with 0 W values. On bare metal the same path carries real wattage.
• Unregistered identities fail loudly by design: the patradb trigger raises for unknown user/device/model; Kafka Connect logs the error and skips the record (errors.tolerance=all), so nothing lands. Register the entities and replay, or use the pre-registered examples.
• Connector status RUNNING does not imply records are landing — per-record failures are swallowed by errors.tolerance=all. Always verify with the SQL queries above and the Connect worker logs.

Optional: synthetic power readings on hosts without RAPL (test-only)

To verify the power path carries nonzero values on a VM, swap the powerjoular measurement container for a synthetic stand-in — no camera-traps code changes; the backend reads the image name from the POWER_JOULAR_IMAGE env var at runtime. The stand-in must honor the same CLI (-tp <pid> -f <output_path>) and emit the 5-column CSV (Date,CPU Utilization,Total Power,CPU Power,GPU Power) one row per second.

# 1. build a fake powerjoular that appends jittered nonzero wattage rows
mkdir -p ~/fake-powerjoular && cd ~/fake-powerjoular

cat > fake_powerjoular.py <<'PYEOF'
#!/usr/bin/env python3
"""
Test-only stand-in for powerjoular, for hosts without Intel RAPL (cloud VMs).

Accepts the same CLI the camera-traps powerjoular backend uses
(`-tp <pid> -f <output_path>`) and appends one CSV row per second in the
5-column format `convert_powerjoular_csv_to_json` expects, with synthetic
nonzero CPU/GPU wattage. Runs until the backend force-removes the container.
"""
import random
import sys
import time
from datetime import datetime

args = sys.argv[1:]
pid = args[args.index("-tp") + 1]
out = args[args.index("-f") + 1]

print(f"fake-powerjoular: monitoring pid {pid}, writing {out}", flush=True)

with open(out, "w") as f:
    f.write("Date,CPU Utilization,Total Power,CPU Power,GPU Power\n")

while True:
    ts = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    util = random.uniform(0.05, 0.95)
    cpu_w = random.uniform(2.0, 6.0)
    gpu_w = random.uniform(0.05, 0.3)
    total_w = cpu_w + gpu_w
    with open(out, "a") as f:
        f.write(f"{ts},{util:.14f},{total_w:.14f},{cpu_w:.14f},{gpu_w:.14f}\n")
    time.sleep(1)
PYEOF

cat > Dockerfile <<'DOCKEOF'
FROM python:3.12-alpine
COPY fake_powerjoular.py /fake_powerjoular.py
ENTRYPOINT ["python", "-u", "/fake_powerjoular.py"]
DOCKEOF

# 2. the backend always pulls POWER_JOULAR_IMAGE before running, so serve the
#    image from a throwaway local registry (localhost is exempt from Docker's TLS requirement)
docker run -d -p 5000:5000 --name e2e-registry registry:2
docker build -t localhost:5000/fake-powerjoular:latest .
docker push localhost:5000/fake-powerjoular:latest

# 3. after rendering the install (step 3 above), point the power monitor at it
sed -i '/TRAPS_TEST_POWER_FUNCTION=1/a\      - POWER_JOULAR_IMAGE=localhost:5000/fake-powerjoular:latest' \
  ~/ct-e2e/e2e_test/docker-compose.yml

# 4. run the experiment as in step 4, then clean up the registry afterwards
docker rm -f e2e-registry

The power_summary row (and experiments.total_cpu_power_w / total_gpu_power_w) for that experiment will then carry the synthetic nonzero wattage. Label such experiments clearly — the power numbers are fabricated for pipeline testing, not measurements.

Reference runs

2026-07-15, Jetstream Ubuntu 22.04 x86_64 host, zero Connect errors in both:

• Experiment 027de96a-f744-48c2-b933-373be3956e65 (real powerjoular): 12/12 oracle events + power summary landed (events=12, power_summary=1, experiments row with precision 0.075 / recall 1.0 / F1 0.14 over 80 predictions); wattage 0 W since the VM exposes no RAPL.
• Experiment 060009f9-6a0f-40f3-b6f1-3e3a2eaecfce (synthetic powerjoular stand-in): 12/12 oracle events + power summary with nonzero wattage (total CPU 11.90 W, total GPU 0.56 W across the three monitored plugins), propagated into experiments.total_cpu_power_w/total_gpu_power_w. Power values are synthetic, for pipeline verification only.

[nee1k]

Update: fixed the MODEL_ID issue Samuel hit.

Root cause: MODEL_ID=2 in the runbook/run_e2e_test.sh defaults was never a camera-traps convention — it's patradb's internal models.id. Camera-traps' actual native model_id (installer defaults, image_generating_plugin) is a Patra model-card UUID with a -model suffix, e.g. ea991e85-feaa-4781-a297-4d7bec1a69b1-model. The CKN ingest trigger in patra-knowledge-base only ever understood the numeric form, so any model_id in the native UUID form — including Samuel's, which is a validly registered model — failed ingestion.

Fixed in patra-knowledge-base (Plale-Lab/patra-knowledge-base@e72413a), applied to the live patradb, and verified: fn_ingest_camera_trap_event() now falls back to resolving model_id via model_cards.uuid (stripping the optional -model suffix) when the numeric lookup misses. No auto-creation reintroduced — an unregistered model still fails loudly.

Verified end-to-end with run_e2e_test.sh MODEL_ID='ea991e85-feaa-4781-a297-4d7bec1a69b1-model' (Samuel's exact reported case): 12/12 oracle events + power summary, experiments.model_id correctly resolved to 22. Also regression-checked the original numeric form (MODEL_ID=2) still works unchanged.

@samuel — both forms of MODEL_ID should work for you now; no changes needed on your end other than pulling in the fix (it's already live in the shared patradb, so nothing to re-run there).