Release 2.1.1

CLAUDE.md

@@ -133,10 +133,16 @@ The RAG pipeline filters out false-positive sources by having the LLM self-repor
 1. `format_context()` (`openrag/core/prompts/chat_prompt_builder.py`) numbers each source (`[Source 1]`, `[Source 2]`, ...) in the context and returns `(formatted_text, included_indices)` — the indices track which docs fit within the token budget
 2. Prompt templates (`openrag/prompts/templates/*.txt`) instruct the LLM to append `[Sources: 1, 3, 5]` at the end of its response
 3. `extract_and_strip_sources_block()` (`openrag/core/utils/source_filtering.py`) strips this tag from the response before sending to the client
-4. `filter_sources_by_citations()` (`openrag/core/utils/source_filtering.py`) filters the source metadata to only include cited sources (falls back to all sources if none match)
+4. `filter_sources_by_citations()` (`openrag/core/utils/source_filtering.py`) filters the source metadata to only include cited sources; if no `[Sources: ...]` tag is found at all, every presented source is kept instead (a missing tag means the model didn't report citations, not that it used none)
 5. For streaming, the OpenAI router buffers the last 100 chars to catch the sources tag before it reaches the client
 
-The `extra` field in API responses is a JSON string: `{"sources": [filtered_source_list]}`.
+The `extra` field in API responses is a JSON string with these keys:
+
+- `sources` — legacy field, kept as-is for existing clients (e.g. Twake): cited sources, or every presented source as a fallback when no `[Sources: ...]` tag was found.
+- `presented_sources` — every source actually shown to the LLM (after `format_context()`/`format_web_context()` truncation), regardless of citation. Always present; a client can fall back to this ("sources consulted") when nothing was cited.
+- `cited_sources` — strictly what the model cited via the tag; unlike `sources`, this never falls back to "everything" — it's `[]` whenever no tag was found. Chainlit is expected to move to this field, falling back to `presented_sources` in its UI when `cited_sources` is empty.
+- `citations_reported` (bool) — `true` only when the model actually emitted a `[Sources: ...]` tag (even an empty/`none` one); `false` when the tag was missing entirely, which is the only case where `sources` falls back to keeping everything. Lets a client tell "the model cited every source" apart from "the model didn't report citations at all".
+- `all_retrieved_sources` — the complete retrieval set, captured before the context-token-budget truncation, so it also includes documents/web results that didn't fit in the prompt (and, on the map-reduce path, the original retrieved docs rather than the LLM-generated summaries). Only included when the request sets `metadata.include_all_retrieved_sources: true` — it's debug/eval telemetry, gated off by default since retrieval is uncapped up to `retriever.top_k` while the context budget only fits a handful of documents.
 
 ### API Routers (`openrag/api/routers/`)
 

infra/charts/openrag-stack/Chart.yaml

@@ -3,8 +3,8 @@ name: openrag-stack
 description: A Helm chart for Kubernetes
 type: application
 
-version: 0.6.1
-appVersion: "2.1.0"
+version: 0.6.2
+appVersion: "2.1.1"
 
 maintainers:
   - name: linagora

infra/charts/openrag-stack/templates/raycluster.yaml

@@ -34,6 +34,13 @@ spec:
               - sh
               - -c
               - |
+                # Keep venv writes group-writable (GID 0) so a later sync under a
+                # different APP_UID can still remove/replace owner-only entries
+                # left by a previous sync. See infra/scripts/entrypoint.sh.
+                # NOTE: only fixes files written from here on — a PVC provisioned
+                # before this fix still has owner-only entries and needs to be
+                # recreated once after upgrading.
+                umask 002
                 if [ -f /app/.venv/.ready ]; then
                   echo "Existing env detected, skipping install."
                 else

infra/charts/openrag-stack/values.yaml

@@ -99,7 +99,7 @@ ray:
     registry: "ghcr.io"
     repository: "linagora/openrag-ray"
     # Pin to a release tag (ideally a digest) for reproducible deploys.
-    tag: "v2.1.0"
+    tag: "v2.1.1"
   resources:
     head:
       requests:
@@ -383,7 +383,7 @@ adminUi:
     repository: "linagoraai/openrag-admin-ui"
     # Pin to a release tag (ideally a digest) for reproducible deploys. Must be a
     # build from infra/docker/ui.Dockerfile (nginx-unprivileged, listens :8080).
-    tag: "v2.1.0"
+    tag: "v2.1.1"
     pullPolicy: IfNotPresent
   replicaCount: 1
   service:
@@ -440,7 +440,7 @@ openrag:
     registry: ""
     repository: "linagoraai/openrag"
     # Pin to a release tag (ideally a digest) for reproducible deploys.
-    tag: "v2.1.0"
+    tag: "v2.1.1"
     pullPolicy: IfNotPresent
   service:
     type: ClusterIP

infra/compose/docker-compose.yaml

@@ -18,7 +18,7 @@ x-openrag-env: &openrag_env
   FONT_PATH: ${FONT_PATH:-/app/data/fonts/GoNotoCurrent-Regular.ttf}
 
 x-openrag: &openrag_template
-  image: linagoraai/openrag:v2.1.0
+  image: linagoraai/openrag:v2.1.1
   # Start as root so entrypoint.sh can grant GID-0 write on the bind-mounted
   # writable dirs (data/, logs/, the HF cache) — which a non-root container
   # can't write when Docker auto-creates them root-owned — then it immediately
@@ -113,7 +113,7 @@ x-vllm: &vllm_template
 services:
   # ── Admin UI (React SPA + nginx, same-origin reverse proxy to the API) ──
   admin-ui:
-    image: linagoraai/openrag-admin-ui:v2.1.0
+    image: linagoraai/openrag-admin-ui:v2.1.1
     build:
       context: ../..
       dockerfile: infra/docker/ui.Dockerfile

infra/scripts/entrypoint.sh

@@ -30,6 +30,18 @@ if [ "$(id -u)" = "0" ]; then
   exec setpriv --reuid "$APP_UID" --regid 0 --clear-groups /app/entrypoint.sh "$@"
 fi
 
+# The persisted openrag_venv volume can be re-synced by different APP_UIDs
+# across the container's lifetime (e.g. a locally-built image bakes in the
+# host UID while a pulled/CI-built image bakes in the Dockerfile default) —
+# both share GID 0. Default umask (022) makes `uv sync` create new venv
+# entries owner-writable only, so a later sync under a different UID can't
+# remove/replace them ("Permission denied" on __editable__*.pth). Force
+# group-writable new files/dirs so any GID-0 UID can always resync.
+# NOTE: this only fixes files written from here on — an openrag_venv volume
+# that already existed before this fix still has owner-only entries and needs
+# to be recreated once after upgrading (delete the volume / let it resync).
+umask 002
+
 ENV_ARGS=()
 if [[ -n "${SHARED_ENV}" ]]; then
   ENV_ARGS+=("--env-file=${SHARED_ENV}")

openrag/api/routers/user/chat.py

@@ -446,7 +446,18 @@ def check_tokens_limit(
 
 **Response:**
 Returns OpenAI-compatible response with additional `extra` field containing:
-- `sources`: Array of source documents with metadata and URLs
+- `sources`: Legacy field, kept for backward compatibility. Cited sources, or
+  every presented source as a fallback when the model didn't report citations
+- `presented_sources`: Array of every source actually shown to the model
+  (after context-budget truncation), regardless of what it cited
+- `cited_sources`: Array of only the sources the model explicitly cited; empty
+  whenever no citations tag was found (never falls back like `sources` does)
+- `citations_reported`: `true` only if the model emitted a citations tag
+  (even an empty one); `false` means `sources` fell back to keeping everything
+- `all_retrieved_sources`: Array of every source retrieval returned, unfiltered
+  by citation or context-budget truncation — only included when the request's
+  `metadata.include_all_retrieved_sources` is `true` (off by default; this is
+  debug/evaluation telemetry and can be large)
 
 **Streaming:**
 Set `stream: true` for Server-Sent Events (SSE) streaming responses.
@@ -552,7 +563,18 @@ async def stream_response():
 
 **Response:**
 Returns OpenAI-compatible response with additional `extra` field containing:
-- `sources`: Array of source documents with metadata and URLs
+- `sources`: Legacy field, kept for backward compatibility. Cited sources, or
+  every presented source as a fallback when the model didn't report citations
+- `presented_sources`: Array of every source actually shown to the model
+  (after context-budget truncation), regardless of what it cited
+- `cited_sources`: Array of only the sources the model explicitly cited; empty
+  whenever no citations tag was found (never falls back like `sources` does)
+- `citations_reported`: `true` only if the model emitted a citations tag
+  (even an empty one); `false` means `sources` fell back to keeping everything
+- `all_retrieved_sources`: Array of every source retrieval returned, unfiltered
+  by citation or context-budget truncation — only included when the request's
+  `metadata.include_all_retrieved_sources` is `true` (off by default; this is
+  debug/evaluation telemetry and can be large)
 
 **Note:** Streaming is not supported for this endpoint.
 """,

openrag/api/schemas/user/chat.py

@@ -43,8 +43,10 @@ class OpenAIChatCompletionRequest(BaseModel):
             "spoken_style_answer": False,
             "websearch": False,
             "llm_override": None,
+            "include_all_retrieved_sources": False,
         },
-        description="Extra custom parameters. Supports 'llm_override' object with an optional 'model' to override the downstream model name. The LLM endpoint and credentials are fixed by server configuration and cannot be overridden by the client.",
+        description="Extra custom parameters. Supports 'llm_override' object with an optional 'model' to override the downstream model name. The LLM endpoint and credentials are fixed by server configuration and cannot be overridden by the client. "
+        "'include_all_retrieved_sources' (default false) adds the full, unfiltered retrieval set to the response's extra.all_retrieved_sources — off by default since it can be large; opt in only for debugging/evaluation.",
     )
 
     @model_validator(mode="after")
@@ -84,3 +86,12 @@ class OpenAICompletionRequest(BaseModel):
     stream: bool | None = Field(False)
     temperature: float | None = Field(0.3)
     top_p: float | None = Field(1.0)
+    metadata: dict[str, Any] | None = Field(
+        {
+            "spoken_style_answer": False,
+            "llm_override": None,
+            "include_all_retrieved_sources": False,
+        },
+        description="Extra custom parameters. Supports 'llm_override' object with an optional 'model' to override the downstream model name. The LLM endpoint and credentials are fixed by server configuration and cannot be overridden by the client. "
+        "'include_all_retrieved_sources' (default false) adds the full, unfiltered retrieval set to the response's extra.all_retrieved_sources — off by default since it can be large; opt in only for debugging/evaluation.",
+    )

openrag/app_front.py

@@ -690,7 +690,12 @@ async def on_message(message: cl.Message):
             async for chunk in stream:
                 if chunk.extra:
                     extra = json.loads(chunk.extra)
-                    if "sources" in extra:
+                    if "cited_sources" in extra:
+                        # Strictly what the model cited; fall back to
+                        # everything it was shown when nothing was cited
+                        # (e.g. it didn't report citations at all).
+                        sources = extra["cited_sources"] or extra.get("presented_sources")
+                    elif "sources" in extra:
                         sources = extra["sources"]
 
                 if chunk.choices and chunk.choices[0].delta.content:

openrag/core/retrieval/pipeline.py

@@ -130,6 +130,13 @@ async def retrieve_docs(
                 if self.reranker_enabled:
                     chunks = await _rerank_chunks(self.reranker, query.query, chunks)
 
+        # `reranker_top_k` is NOT applied here as a final cutoff — only
+        # `top_k` (an explicit caller-supplied value, e.g. map-reduce's
+        # max_total_documents) truncates. On the common no-`top_k` chat path
+        # this returns everything reranked (up to the retriever's own
+        # top_k), which callers sizing a token budget off reranker_top_k
+        # should not assume is bounded by it. Tracked separately:
+        # https://github.com/linagora/openrag/issues/851
         if top_k is not None:
             chunks = chunks[:top_k]
         return chunks

openrag/core/utils/source_filtering.py

@@ -79,15 +79,15 @@ def extract_and_strip_sources_block(
     return cleaned, set()
 
 
-def filter_sources_by_citations(
-    sources: list,
-    citations: set[int] | None,
-    *,
-    allow_uncited: bool = False,
-) -> list:
-    """Keep only sources whose 1-based index was cited."""
+def filter_sources_by_citations(sources: list, citations: set[int] | None) -> list:
+    """Keep only sources whose 1-based index was cited.
+
+    No tag at all (``citations is None``) means the model didn't report which
+    sources it used, not that it used none — the answer may still be grounded
+    in them, so keep everything rather than silently dropping real sources.
+    """
     if citations is None:
-        return sources if allow_uncited else []
+        return sources
     if not citations:
         return []
     return [source for i, source in enumerate(sources, start=1) if i in citations]
@@ -112,11 +112,31 @@ async def stream_with_source_filtering(
     model_name: str,
     buffer_size: int | None = None,
     *,
-    allow_uncited_sources: bool = False,
     citation_protocol_active: bool = True,
+    all_sources: list | None = None,
+    include_all_retrieved: bool = False,
 ):
     """Process an LLM SSE stream and, when active, strip source tags.
 
+    ``sources`` is the prompt-visible (context-budget-truncated) list used to
+    resolve citation indices — reported as-is via ``extra.sources`` (legacy,
+    kept for existing clients: cited sources, or every presented source as a
+    fallback when no tag was found) and ``extra.presented_sources`` (always
+    the raw pre-filter list, so a client can render "sources consulted" when
+    nothing was cited). ``extra.cited_sources`` is the strict version: only
+    what the model actually cited, empty whenever no tag was found — never
+    falling back to "everything" the way ``sources`` does.
+    ``extra.citations_reported`` disambiguates those two empty/full states
+    on the wire: ``true`` only when a ``[Sources: ...]`` tag (even an empty
+    one) was actually found.
+
+    ``all_sources`` — the complete pre-truncation retrieval set — is reported
+    as ``extra.all_retrieved_sources`` only when ``include_all_retrieved`` is
+    true (it defaults to ``sources`` when the caller has nothing more
+    complete to offer). It's gated because a full retrieval dump on every
+    response is debug/eval telemetry, not something most callers need on the
+    hot path.
+
     The terminal flush (tail content + ``extra.sources``) runs exactly once
     after the loop on *every* termination path — a clean ``data: [DONE]``, the
     upstream closing the connection without one, or the upstream generator
@@ -263,8 +283,15 @@ async def stream_with_source_filtering(
     else:
         final_clean, citations = pending, None
 
-    filtered = filter_sources_by_citations(sources, citations, allow_uncited=allow_uncited_sources)
-    extra_payload = {"sources": filtered}
+    filtered = filter_sources_by_citations(sources, citations)
+    extra_payload = {
+        "sources": filtered,
+        "presented_sources": sources,
+        "cited_sources": filtered if citations is not None else [],
+        "citations_reported": citations is not None,
+    }
+    if include_all_retrieved:
+        extra_payload["all_retrieved_sources"] = all_sources if all_sources is not None else sources
     if not saw_done:
         extra_payload["truncated"] = True
         logger.warning(