feat(evals): Tier 0 static lint + Tier 1 activation evals for all 18 skills

.claude-plugin/marketplace.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
   "name": "keboola-claude-kit",
-  "version": "1.12.0",
+  "version": "1.12.1",
   "metadata": {
     "description": "Client-facing Claude Kit marketplace for Keboola workflows — component development, data app building, project management and review, and semantic layer toolkit"
   },
@@ -13,28 +13,28 @@
     {
       "name": "component-developer",
       "description": "Comprehensive toolkit for building Keboola Python components with Agent Skills format",
-      "version": "3.3.0",
+      "version": "3.3.1",
       "source": "./plugins/component-developer",
       "category": "development"
     },
     {
       "name": "dataapp-developer",
       "description": "Toolkit for building and deploying Keboola Apps (Streamlit and Python/JS) — full lifecycle: choosing app type, deployment paths, storage access, authentication, DuckDB caching, styling, dashboard patterns, optional Kai chat, and the validate-build-verify dev workflow; and hosting an MCP server as a data app.",
-      "version": "1.5.0",
+      "version": "1.5.1",
       "source": "./plugins/dataapp-developer",
       "category": "development"
     },
     {
       "name": "keboola-cli",
       "description": "Keboola project management and 10-agent review team for SQL, security, performance, financial logic, and template readiness analysis",
-      "version": "1.1.0",
+      "version": "1.1.1",
       "source": "./plugins/keboola-cli",
       "category": "operations"
     },
     {
       "name": "keboola-git",
       "description": "Access Keboola-managed Git (Forgejo) repos for python-js data apps via the kbagent CLI — provision repos, mint push credentials, and copy source between GitHub and Keboola git with the 15MB/build-at-deploy path handled",
-      "version": "1.0.0",
+      "version": "1.0.1",
       "source": "./plugins/keboola-git",
       "category": "development"
     },

.github/workflows/skill-evals.yml

@@ -0,0 +1,59 @@
+name: Skill evals
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  tier0-lint:
+    name: "Tier 0: static lint + case validation"
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Setup uv
+        uses: astral-sh/setup-uv@v5
+
+      - name: Install Python
+        run: uv python install 3.12
+
+      - name: Run static lint + offline activation-case validation
+        working-directory: evals
+        run: uv run --group dev pytest -q
+
+  tier1-activation:
+    name: "Tier 1: skill activation routing"
+    runs-on: ubuntu-latest
+    # The paid tier only runs once the free deterministic tier is green.
+    needs: tier0-lint
+    # Needs an LLM classifier; runs only where the secret is available
+    # (skipped on forks). Deterministic grading, temperature 0.
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Setup uv
+        uses: astral-sh/setup-uv@v5
+
+      - name: Install Python
+        run: uv python install 3.12
+
+      - name: Run activation evals
+        working-directory: evals
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+        run: |
+          if [ -z "$ANTHROPIC_API_KEY" ]; then
+            echo "ANTHROPIC_API_KEY secret not available — skipping Tier 1 (fork PR?)"
+            exit 0
+          fi
+          uv run python activation/run_activation.py --ci
+
+      - name: Upload activation summary
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: activation-summary
+          path: evals/activation/results/summary.json
+          if-no-files-found: ignore

README.md

@@ -162,6 +162,20 @@ We follow semantic versioning. Update version numbers in:
 - `plugins/<name>/.claude-plugin/plugin.json`
 - `plugins/<name>/README.md`
 
+### Skill evals
+
+Every PR runs two eval tiers from [`evals/`](evals/README.md) (see the
+`Skill evals` workflow):
+
+- **Tier 0 — static lint:** frontmatter validity, skill `name` ↔ directory,
+  plugin/marketplace version consistency, dangling references.
+- **Tier 1 — activation:** each skill's `description:` is measured against
+  labeled utterances in `plugins/<plugin>/evals/<skill>/trigger-evals.json`
+  for routing precision/recall. Every skill must have a case set.
+
+Deeper behavior and live E2E tiers run from
+[keboola/KaiBench](https://github.com/keboola/KaiBench) (`docs/skill-evals.md`).
+
 ## License
 
 MIT licensed, see [LICENSE](./LICENSE) file.

evals/.gitignore

@@ -0,0 +1,3 @@
+activation/results/
+.venv/
+__pycache__/

evals/README.md

@@ -0,0 +1,71 @@
+# Skill evals
+
+Empirical quality gates for ai-kit skills: edit a skill → CI measures whether
+an equipped agent behaves better → regressions block the PR.
+
+This is the fast half (Tiers 0–1) of the four-tier design shared with
+[keboola/KaiBench](https://github.com/keboola/KaiBench) (see its
+`docs/skill-evals.md`). The heavy tiers — Tier 2 behavior evals (agent runs a
+case with only the target skill loaded, graded by KaiBench's graders) and
+Tier 3 live E2E — run from KaiBench, which supports `skill`-tagged cases,
+`--skill-dir` skill loading, and per-skill metrics.
+
+| Tier | What | Cost | Where |
+|------|------|------|-------|
+| **0 — Static lint** | Frontmatter validity, name↔directory, manifest/version consistency, dangling references, script conventions | Seconds, no LLM | here, every PR |
+| **1 — Activation** | Routing precision/recall of each skill's `description:` against labeled utterances | ~200 cheap classifier calls | here, every PR (needs `ANTHROPIC_API_KEY`) |
+| 2 — Behavior | Agent + isolated skill on graded cases (`outcome_score` / `process_score`) | Medium | KaiBench |
+| 3 — Live E2E | Live Keboola project ops on the full harness | Slow | KaiBench, nightly |
+
+## Running locally
+
+```bash
+cd evals
+
+# Tier 0 + offline case validation (no API key)
+uv run --group dev pytest -q
+
+# Tier 1 — activation routing (needs ANTHROPIC_API_KEY)
+uv run python activation/run_activation.py            # report only
+uv run python activation/run_activation.py --ci       # gate on thresholds
+uv run python activation/run_activation.py --skill get-started
+uv run python activation/run_activation.py --dry-run  # list cases, no API
+```
+
+The activation run writes `activation/results/summary.json` (overall accuracy,
+per-skill precision/recall, every misroute with the router's actual pick).
+
+## How Tier 1 works
+
+The classifier (Claude Haiku, temperature 0) is shown the **full list of the
+marketplace's skill descriptions** — the same surface the real harness routes
+on — plus one user utterance, and asked which skills it would invoke. Grading
+is deterministic: the skill under test must appear iff the case says
+`should_trigger`. Perfect recall with some co-activation false positives is
+the common failure texture; the misroute list is the backlog for sharpening
+`description:` fields.
+
+CI thresholds (see `--min-accuracy` / `--min-skill-accuracy`): ≥85% overall,
+≥60% per skill.
+
+## Adding cases
+
+Every skill must have `plugins/<plugin>/evals/<skill-dir>/trigger-evals.json`
+(enforced by Tier 0). Format:
+
+```json
+[
+  {"query": "brand new component for the github api, where do i start", "should_trigger": true},
+  {"query": "my component job fails with exit code 2", "should_trigger": false}
+]
+```
+
+Guidelines:
+
+- ≥6 cases, ≥2 positive and ≥2 negative (enforced).
+- Write **hard negatives**: utterances that belong to a *sibling* skill
+  (e.g. debug-component vs develop-component), not obviously unrelated ones.
+- Positives should paraphrase real user phrasing, including the trigger
+  phrases the description promises to catch.
+- When you edit a `description:`, re-run Tier 1 and check the skill's
+  precision/recall — that is the whole point of the loop.

evals/activation/conftest.py

@@ -0,0 +1,4 @@
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))