fix(context): extract non-ASCII (CJK/Hangul) keywords so context works for Korean queries

[andy-sg]

Problem

codegraph context <task> (and the codegraph_context MCP tool) returns an empty context for non-ASCII task descriptions — Korean, Japanese, Chinese, etc. — even though symbol search works fine for the same input.

Reproduction (before this PR):

// src/auth.ts
export function 로그인(사용자명: string): boolean { return 인증확인(사용자명); }
export function 인증확인(사용자명: string): boolean { return 사용자명.length > 0; }
export class 사용자관리자 { 생성하기(이름: string): string { return 이름; } }

$ codegraph query "로그인"      # ✅ finds the 로그인 function
$ codegraph context "로그인"
## Code Context

**Query:** 로그인
                               # ❌ header + query only, zero symbols

Root cause

Symbol search already handles non-ASCII (FTS5's default unicode61 tokenizer + prefix/LIKE matching, and name COLLATE NOCASE = exact lookup). The break is purely in the keyword-extraction preprocessing that feeds context.

Both extractors are built entirely from ASCII patterns — [a-zA-Z] character classes plus JS's ASCII \b word boundary:

• extractSymbolsFromQuery (src/context/index.ts) — camelCase / snake_case / SCREAMING / acronym / dot-notation / lowercase, all [a-zA-Z]-anchored.
• extractSearchTerms (src/search/query-utils.ts) — final tokenization splits on /[^a-zA-Z0-9]+/, which treats every Hangul/CJK character as a separator, so a non-ASCII query tokenizes to nothing.

Result: a non-Latin description yields zero keywords, so the search/traversal pipeline receives nothing and the context comes back empty.

Fix (additive only)

• extractSymbolsFromQuery: after the existing ASCII patterns, also pull runs of Unicode letters/digits (/[\p{L}\p{N}_]+/gu) and keep only tokens that actually contain a non-ASCII character. ASCII-only tokens stay owned by the existing patterns, so nothing about ASCII behavior changes.
• extractSearchTerms: change the final word split from /[^a-zA-Z0-9]+/ to /[^\p{L}\p{N}]+/u. This is byte-identical for ASCII ([a-zA-Z0-9] ⊂ [\p{L}\p{N}], and every ASCII separator — space, punctuation, _, . — remains a separator) while keeping Unicode letter runs intact. Non-ASCII tokens use a 2-char floor (Hangul/CJK pack a morpheme per character); ASCII keeps its 3-char floor.

Extracted non-ASCII tokens flow straight into the existing search path, which already tokenizes them via unicode61.

Out of scope (deliberately): indexing comments/string-literal bodies into FTS. The FTS index still covers only name / qualified_name / docstring / signature — unchanged. This PR fixes only the context keyword-extraction step.

After this PR, codegraph context "로그인" surfaces the 로그인 entry point, the 인증확인 related symbol, and their code blocks.

Tests

• __tests__/query-utils.test.ts (new): unit coverage for extractSearchTerms — ASCII behavior unchanged (camelCase/snake_case/dot-notation split, stop-word + <3-char drop) plus Korean extraction (single token, multi-word split, 2-char floor, mixed ASCII+Korean).
• __tests__/context.test.ts (added): end-to-end — index a Korean source file and assert findRelevantContext / buildContext now surface 로그인 / 인증확인 / 사용자관리자 (a regression guard for the reported empty-context bug).
• Full suite: 49 files, 1080 passed / 2 skipped, no regressions. Verified the before (empty) → after (symbols surfaced) transition by building both states against the repro repo.