GoMLX panics on code chunks longer than the model's max sequence length — no input truncation

[madeinoz67]

Describe the bug

Code chunks that tokenize to more tokens than the model's max sequence length cause hugot's GoMLX backend to panic during graph compilation (tensor-shape mismatch), aborting the entire vector index. Inputs aren't truncated/clipped to the model's max_position_embeddings before being fed to GoMLX.

(The daemon silent-fall-to-static from a previous report is fixed in v0.59.1 — provider: local/bge_small, dim: 384 engages correctly now. This is a distinct, remaining bug.)

To Reproduce

1. brew upgrade --cask gortex → v0.59.1 (commit 00aec84)
2. ~/.gortex/config.yaml:

   embedding:
     enabled: true
     provider: local
     variant: bge_small   # BAAI/bge-small-en-v1.5, max_position_embeddings = 512

3. gortex daemon start --embeddings --config ~/.gortex/config.yaml
4. Track a repo with code chunks >512 tokens (e.g. github.com/madeinoz67/go-rag, ~25k nodes — reproduces reliably)

Expected behavior

Chunks exceeding the model's max sequence length (512 for bge-small-en-v1.5) should be truncated/clipped to max_position_embeddings — or sub-chunked to fit (à la chunk_threshold_lines) — not panic GoMLX graph compilation.

Environment:

• OS: macOS 15 (darwin/arm64, Apple Silicon)
• Go version: 1.26.4
• Gortex version: v0.59.1 (commit 00aec84)

Additional context

Daemon engages local/bge_small correctly, then aborts on a long chunk:

serverstack: embeddings enabled, provider: local/bge_small — enabled by flag/env, dim: 384
WARN vector index aborted on chunk failure: hugot run: failed during graph
construction/compilation: compilation failed for Exec
"github.com/knights-analytics/hugot/backends.createGoMLXModelBackend.func1":
graphFn for "...createGoMLXModelBackend.func1" panicked: dimension of axis #1
doesn't match and cannot be broadcast for BinaryOp (Add), got shapes
(Float32)[500 514 384] and (Float32)[1 512 384]

Shape analysis: mismatch on axis #1 = sequence length: [500 514 384] (batch=500, seq=514) vs [1 512 384] (model's 512-wide positional embedding). A 514-token chunk exceeds 512 → GoMLX graph (compiled for 512) can't broadcast → panic.

Confirms it's a long-input bug, not a model/hugot bug: a direct unit test embedding short texts (["hello world", "func main() {}"], <512 tokens) via newHugotProviderWithSpec(bge_small) → EmbedBatch succeeds (real 384-dim vectors). Only chunks >512 tokens panic. The model + hugot pipeline are fine; missing input truncation for long chunks is the issue.

Reference: go-rag (same hugot v0.7.5 + gomlx v0.27.3, byte-identical go.sum) embeds successfully in its integration tests — its fixture chunks stay under 512 tokens, so it never hits this.

Net on Apple Silicon: fall-to-static is fixed, but a full repo embed still aborts because real code chunks routinely exceed 512 tokens. Truncating/sub-chunking inputs to max_position_embeddings before the GoMLX call would resolve it.