[feature] ft:fields: introspect configured Lucene fields/facets in a scope

[joewiz]

[This PR was co-authored with Claude Code. -Joe]

Stacked on #6455. This branch is based on feature/lucene-search-index (#6455), because ft:fields reuses that PR's LuceneScope scope-resolution helper. Until #6455 merges, the diff here includes its commits; the ft:fields-specific changes are the final two commits. Once #6455 lands, this rebases onto develop to a clean ft:fields-only diff. Names provisional — happy to rename with the rest of the family.

Summary

Adds ft:fields($scope) as map(*)* — the schema-discovery companion to ft:query-scope/ft:search-scope. Where they search an index scope, this describes it: the configured fields, facets, and vector fields and each one's contract. It's the engine for existdb-openapi's GET /api/search/fields (a _mapping-style "what can I search here, and what is each field's contract?" endpoint) and the field-level-security layer built on top of it.

ft:fields("/db/apps") (: => one map per configured field/facet/vector occurrence :)

map { 
    "field": "site-content", 
    "kind": "field", 
    "element": "xqdoc:function",
    "analyzer": "org.apache.lucene.analysis.core.SimpleAnalyzer",
    "type": "xs:string", 
    "returnable": true() 
}
map { 
    "field": "site-app", 
    "kind": "facet", 
    "element": "topic" 
}
map { 
    "field": "site-embedding", 
    "kind": "vector", 
    "element": "doc",
    "dimension": 384, 
    "similarity": "cosine", 
    "model": "all-MiniLM-L6-v2" 
} …

Motivation / why native (not xconf parsing)

Field names you can scrape from collection.xconf; the contract — which analyzer a field uses, its type, whether it's stored/returnable, which element each field is indexed on, and (for vector fields) its dimension, similarity metric, and embedding model — is the resolved LuceneConfig, which a parser would have to reconstruct (config inheritance across nested collections, analyzer-id resolution, merged qname/wildcard/named indexes). existdb-openapi prototyped this Phase-2 layer with an xconf-parser stand-in and hit exactly those walls; ft:fields reads the resolved config via the broker instead. The requirements below come from that prototype against a real 3-producer corpus.

What it returns, per requirement

• Permission-agnostic, callable by any user (R1). It reads the resolved LuceneConfig via the broker, not /db/system/config (admin-only), so a guest call returns the full catalog. existdb-openapi then applies field-level security (group→allowed-fields) on top. The stand-in had to wrap config reads in system:as-user("admin", …); this removes that.
• Distinguishes field vs facet vs vector (R2): kind: "field" | "facet" | "vector" (the same name is often both field and facet — e.g. site-app is declared as both a <facet dimension> and a <field name>).
• Resolved analyzer, per (field × element) (R3): the analyzer is reported as the concrete class, resolving an @analyzer="simple" id to its class and unwrapping eXist's per-field MetaAnalyzer to the actual default/per-field analyzer. A shared field can be indexed with different analyzers on different elements (StandardAnalyzer vs SimpleAnalyzer), and that variance is preserved.
• Resolved type + returnable (R4): XDM type (default xs:string) and the stored/returnable flag (default true), with defaults resolved.
• Element each field/facet is indexed on (R5).
• Scope = collection path(s), recursive, inheritance-aware (R6): the same $scope model and resolution (getLuceneConfig(broker, docs)) as ft:search-scope.
• One record per (field × element) occurrence (R7): no pre-dedup, so the per-element analyzer/type variance is visible; the API collapses to a field-level view.
• Vector-field metadata (R8): a <vector-field> is reported with kind: "vector" and three extra keys — dimension (xs:integer), similarity ("cosine" | "euclidean" | "dot_product"), and model (the embedding model id, present only when the field embeds text via embedding="local"/"http"; absent for a raw-vector field). This lets a discovery-driven vector-search endpoint resolve the embedding model from ft:fields alone: the client sends {text, field}, the server reads the field's model, embeds the query with it, and runs KNN — no client-side model knowledge. (Driven by existdb-openapi#62.)

What changed (extensions/indexes/lucene)

• Fields.java (ft:fields) — walks the resolved LuceneConfig and emits a map per field/facet/vector field, reusing LuceneScope.resolveScope for the scope→documents→config bridge. The vector branch adds dimension/similarity/model.
• LuceneVectorFieldConfig.getModelId() — new getter exposing the embedding model id (the dimension/similarity getters already existed); returns null for a raw-vector field.
• LuceneConfig.getAllIndexConfigurations() — enumerates all index heads (qname paths + wildcardPaths + namedIndexes); the existing getIndexConfigurations() returned only qname paths.
• LuceneFieldConfig — getType() / isStore() / isBinary() getters (the fields were protected).
• MetaAnalyzer.getConfiguredAnalyzer(field) — exposes the concrete analyzer behind the per-field wrapper for introspection (no behavior change to indexing/search).
• LuceneModule registers the signature; tests in ft-fields.xqm + the LuceneQueryScopeTests runner.

Test plan

• 22 ft:fields XQSuite tests: map shape; field/facet counts; type and returnable (incl. store="no"); the resolved analyzer class for a default-analyzer field (StandardAnalyzer) and an analyzer-id field (SimpleAnalyzer); facet entry shape; unknown scope → empty.
• Vector field: kind="vector" count; dimension is an xs:integer (384); similarity (cosine); model (all-MiniLM-L6-v2). The vector field is configured on an element absent from the test doc, so the XQSuite exercises pure config introspection — no vector extension needed at runtime.
• R1 permission-agnostic: a guest (system:as-user) gets the full catalog (config lives under admin-only /db/system/config, but ft:fields reads the resolved config via the broker).
• 62 tests across the query-scope suite, all green.
• Codacy/PMD clean on the new/changed files.

Related

• [feature] Index-first Lucene search: ft:query-scope (live nodes) and ft:search-scope (ES-shaped map) #6455 — the parent ft:query-scope/ft:search-scope PR this stacks on.
• existdb-openapi field-discovery + FLS design and the feat/api-search-fields-discovery prototype this is the drop-in engine for; existdb-openapi#62 (the vector-search endpoint) drives the vector-field metadata keys.