Skip to content

ci: run Lemonade-dependent checks when LEMONADE_VERSION changes#1871

Open
kovtcharov wants to merge 3 commits into
mainfrom
fix/lemonade-version-ci-triggers
Open

ci: run Lemonade-dependent checks when LEMONADE_VERSION changes#1871
kovtcharov wants to merge 3 commits into
mainfrom
fix/lemonade-version-ci-triggers

Conversation

@kovtcharov

Copy link
Copy Markdown
Collaborator

Why this matters

A Lemonade version bump edits only src/gaia/version.py (LEMONADE_VERSION), which the install-lemonade action reads to provision the runners. But every Lemonade-dependent workflow was path-filtered to feature code (src/gaia/rag/**, llm/**, agents, cpp/**, …) and excluded version.py — so the one change that swaps the Lemonade build on the runners triggered none of the tests that exercise it.

That's the gap that let Lemonade 10.7.0 → 10.8.1 land while the embedding model nomic-embed-text-v2-moe is broken on the bundled llama-server: the embeddings/RAG/API/E2E checks simply never ran on the bump. The break only surfaced days later on unrelated PRs (including a .docx RAG PR) whose authors had no idea it wasn't their change.

Root cause of the embedding break is upstream, not ours — a Vulkan-backend regression on AMD GPUs since llama.cpp build b6524 (llama.cpp #16301, open), surfaced in Lemonade #612 / #941. This PR doesn't fix that; it makes our CI catch that class of regression on the bump PR instead of silently shipping it.

This adds src/gaia/version.py and .github/actions/install-lemonade/** to the push + pull_request path triggers of every Lemonade-using workflow (embeddings, rag, lemonade-server, api, agent-sdk, agent-mcp-server, behavior-e2e, examples, gaia-cli-windows, sd, build_cpp).

Test plan

  • All 11 workflows parse as valid YAML; every paths: block (21 total) now contains src/gaia/version.py (verified programmatically)
  • Reviewer sanity check: the next LEMONADE_VERSION bump PR should now trigger Test Lemonade Embeddings, Test RAG, API Tests, Agent SDK/MCP/Behavior, Lemonade Server Smoke, SD, and C++ build — confirm in that PR's checks list
  • (Separately) the embedding regression itself needs a Lemonade/llama-server fix or a pin to a pre-b6524 build — tracked upstream, out of scope here

A Lemonade version bump edits only src/gaia/version.py (LEMONADE_VERSION),
but every Lemonade-dependent workflow was path-filtered to feature code
(rag/llm/agents/cpp/...) and excluded that file — so the bump that installs
a new Lemonade on the runners triggered NONE of the tests that exercise it.
That is how Lemonade 10.7.0/10.8.x landed while the embedding model
(nomic-embed-text-v2-moe) is broken on the bundled llama-server (>= b6524,
upstream llama.cpp #16301 / lemonade #612): the embeddings/RAG/API/E2E
checks never ran on the bump.

Add src/gaia/version.py and the install-lemonade action to the path
triggers of every Lemonade-using workflow so a version bump runs the full
Lemonade test surface and a regression is caught on the bump PR, not days
later on an unrelated branch.
@github-actions github-actions Bot added devops DevOps/infrastructure changes cpp labels Jun 26, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Verdict: Approve

This closes a real CI gap: a Lemonade version bump only edits version.py, but every Lemonade-dependent workflow was path-filtered to feature code and excluded it — so the bump that swaps the Lemonade build on the runners triggered none of the tests that exercise it (exactly how the broken nomic-embed-text-v2-moe embedding shipped unnoticed). Adding version.py and the install-lemonade action to the triggers fixes that.

Coverage is complete and correct: the 11 workflows changed are precisely the 11 that reference install-lemonade — none missed, none extraneous. The only thing worth a glance is a harmless redundancy (three workflows already match version.py via a broader src/** glob); no functional impact, no change required.

🔍 Technical details

Verification performed:

  • All 11 workflows parse as valid YAML after the change.
  • grep -rl install-lemonade .github/workflows/ returns exactly the 11 files in this PR — full coverage, no Lemonade-using workflow left out.
  • install-lemonade/action.yml:34 does read LEMONADE_VERSION from src/gaia/version.py, so the path trigger matches the actual dependency.

🟢 Minor — redundant version.py entry where a broad glob already covers it (test_agent_sdk.yml, test_gaia_cli_windows.yml, test_examples.yml)
These three already path-filter on src/** (or src/gaia/**), which already matches src/gaia/version.py, so the explicit src/gaia/version.py line is a no-op in them. The install-lemonade/** addition is still meaningful there (it lives outside src/). Harmless and arguably improves readability/intent — fine to leave as-is. No action needed.

Strengths

  • Surgical, scope-clean change — only the paths: blocks touched, both push and pull_request triggers updated consistently across every file.
  • PR description nails the why: leads with user-observable impact (silent embedding regression shipped because the bump ran no tests) and is honest that the upstream Vulkan/llama.cpp break is out of scope — this PR makes CI catch the class of regression, not fix the regression.
  • Test plan is verifiable and was actually run (YAML validity + every paths: block now contains version.py).

… crash on failure

The nomic-embed-text-v2-moe load intermittently aborts llama-server on the
AMD Vulkan backend (transient GPU-driver-state fault; upstream llama.cpp
#16301 / lemonade #612, not a GAIA defect). Retry recovers once GPU state
settles, keeping CI green; debug logging + a job-output dump on each failed
attempt records the exact crash signature the next time it occurs.
…er debug crash on failure"

This reverts commit daa0e46.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cpp devops DevOps/infrastructure changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant