Skip to content

feat: add chunk-level metadata filtering to hybrid search - #2693

Open
dalaoyang wants to merge 22 commits into
Tencent:mainfrom
dalaoyang:chunk-metadata-filter-api
Open

feat: add chunk-level metadata filtering to hybrid search#2693
dalaoyang wants to merge 22 commits into
Tencent:mainfrom
dalaoyang:chunk-metadata-filter-api

Conversation

@dalaoyang

Copy link
Copy Markdown

What changed

  • Add metadata_filter to hybrid search for chunk/record-level access filtering.
  • Support recursive and / or expressions with eq and in predicates.
  • Persist reserved access_metadata on indexed chunks and apply the predicate before PostgreSQL keyword/vector candidate limits.
  • Fail closed for unsupported or mixed retriever capabilities, and reapply filtering to enrichment paths.
  • Harden the contract with strict JSON AST validation, exact arbitrary-precision number comparison, and bounded in.values payloads.
  • Prevent filtered searches from returning document-wide summaries, knowledge descriptions, or unfiltered child-image enrichment.
  • Update summary, FAQ/document metadata, PostgreSQL copy, migration, API docs, and regression coverage.

Why

Callers need row/chunk-level authorization metadata such as employee nature and department. Different fields are combined with AND, while multiple values in one field are combined with OR through in.

Validation

  • env -u LOG_FORMAT GOTELEMETRY=off go test ./... -count=1
  • GOTELEMETRY=off go vet ./...
  • golangci-lint run --new-from-rev=origin/main ./... — 0 issues
  • git diff --check origin/main...HEAD

Not covered by local tests

Live PostgreSQL/ParadeDB/pgvector behavior, deployed HTTP acceptance, online reindexing, and real summary-model/worker execution still require environment-level validation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant