diff --git a/.gitattributes b/.gitattributes index faa0f4b26..e187e861f 100644 --- a/.gitattributes +++ b/.gitattributes @@ -4,3 +4,4 @@ worked/**/*.html linguist-vendored=true graphify-out/**/*.html linguist-vendored=true *.html linguist-detectable=false +graphify-out/graph.json merge=graphify diff --git a/README.md b/README.md index 36a2235ba..ebc1b277b 100644 --- a/README.md +++ b/README.md @@ -109,11 +109,47 @@ What you get out of the box: | **Cross-file links** | `calls` / `imports` / `inherits` / `mixes_in` resolved across ~40 languages via tree-sitter AST | | **Query, path, explain** | Ask a question, trace the path between two things, or explain one concept, all against `graph.json` | | **Rationale + doc refs** | `# NOTE:` / `# WHY:` comments and ADR/RFC citations become first-class nodes linked to the code | +| **Validated knowledge** | `lat.md/` sections, summaries, wiki links, source links, and `@lat` implementation references join the code graph deterministically | | **Beyond code** | Docs, PDFs, images, and video/audio all map into the same graph | | **Local-first** | Code is parsed locally with tree-sitter (no LLM, nothing leaves your machine); only the semantic pass over docs/media calls a backend, and only if you configure one | --- +## Validated knowledge with lat.md + +Graphify automatically recognizes Markdown files inside a project-level +`lat.md/` directory. Each heading becomes a stable `knowledge_section` node, +the first paragraph becomes its searchable summary, and wiki links connect the +curated knowledge to other sections or source files. Explicit implementation +comments connect knowledge back to code: + +```python +# @lat: [[security#Security#Tenant isolation]] +def load_orders(tenant_id): + ... +``` + +Run the normal update command to include the lattice in `graph.json`. Projects +with a `lat.md/` directory are validated automatically after every successful +update; invalid knowledge makes the command exit nonzero. The standalone command +is useful for CI checks and structured JSON diagnostics: + +```bash +graphify update . +graphify check-knowledge . +graphify check-knowledge . --json +graphify query "tenant isolation constraints" +``` + +`check-knowledge` reports broken or ambiguous wiki links, missing source files, +stale or ambiguous `@lat` code mentions, and leaf sections marked +`require-code-mention: true` that have no matching `@lat` reference. The +supported format is compatible with the public lat.md +heading, wiki-link, source-reference, and code-mention conventions while +remaining fully local and deterministic. + +--- + ## Benchmarks | Benchmark | Metric | graphify | Field | @@ -213,6 +249,7 @@ for example `graphify claude install --project` or `graphify codex install --pro | Claude Code (Windows) | `graphify install` (auto-detected) or `graphify install --platform windows` | | CodeBuddy | `graphify install --platform codebuddy` | | Codex | `graphify install --platform codex` | +| Jcode | `graphify jcode install` | | OpenCode | `graphify install --platform opencode` | | Kilo Code | `graphify install --platform kilo` | | GitHub Copilot CLI | `graphify install --platform copilot` | diff --git a/docs/testing/validated-knowledge-ingestion.tdd.md b/docs/testing/validated-knowledge-ingestion.tdd.md new file mode 100644 index 000000000..0f66f657f --- /dev/null +++ b/docs/testing/validated-knowledge-ingestion.tdd.md @@ -0,0 +1,89 @@ +# Validated knowledge ingestion TDD evidence + +## Source + +The user approved implementation of the recommended Graphify and Jcode validated +knowledge plan. This document covers the first Graphify milestone: native +`lat.md` ingestion, integrity checking, implementation linkage, and retrieval. + +## User journeys + +1. As an agent, I can query curated design constraints together with source-code + symbols so that I do not miss project invariants before editing code. +2. As a maintainer, I can validate wiki links, source links, and required + implementation mentions so that project knowledge does not silently drift. +3. As a project adopting lat.md, I can use Graphify's normal update workflow + without running a second retrieval service. + +## Requirement-to-check evidence + +| # | Guarantee | Test or command | Type | Result | +|---|---|---|---|---| +| 1 | Headings become stable knowledge sections with first-paragraph summaries | `test_lattice_markdown_emits_stable_sections_summaries_and_wiki_edges` | Unit | PASS | +| 2 | Inline and fenced examples are ignored while source links become `documents` edges | `test_lattice_ignores_example_links_and_emits_source_documentation_edges` | Integration | PASS | +| 3 | `@lat` comments connect a knowledge section to its implementation file | `test_full_extract_links_at_lat_comment_to_knowledge_section` | Integration | PASS | +| 4 | Cross-file shorthand wiki links resolve to full stable section IDs | `test_full_extract_resolves_cross_file_short_wiki_reference` | Integration | PASS | +| 5 | Broken, ambiguous, and missing implementation references are diagnosed | `test_validate_lattice_reports_broken_ambiguous_and_unimplemented_required_sections` | Unit | PASS | +| 6 | Invalid lattices return structured JSON and exit code 1 | `test_check_knowledge_cli_returns_json_and_nonzero_for_invalid_lattice` | CLI acceptance | PASS | +| 7 | Query scoring searches summaries and returns the summary in bounded output | `test_query_retrieves_lattice_summary_after_normal_extraction` | Public retrieval | PASS | +| 8 | A real valid lattice is accepted through the public CLI | `uv run python -m graphify check-knowledge /home/sergey/.jcode/scratch/graphify-lattice-valid --json` | CLI acceptance | PASS, 2 sections, 0 errors | +| 9 | The implementation is compatible with the official lat.md repository | `validate_lattice(Path('/home/sergey/.jcode/scratch/lat.md-official'))` | Real integration | PASS, 24 files, 192 sections, 0 errors | +| 10 | Dotted lattice filenames remain knowledge references rather than source paths | `test_dotted_lattice_file_reference_is_not_misclassified_as_source` | Integration | PASS | +| 11 | Incremental lattice updates rediscover mentions in unchanged source files | `test_lattice_change_rescans_unchanged_source_mentions` | Incremental integration | PASS | +| 12 | Source links cannot escape the project root | `test_source_reference_cannot_escape_project_root` | Security | PASS | +| 13 | Code-mention validation honors Graphify ignore rules | `test_validation_respects_graphifyignore_when_scanning_code_mentions` | Integration | PASS | +| 14 | Adjacent extraction, CLI, query, and security behavior remains intact | focused regression command below | Regression | PASS, 460 tests | +| 15 | Removed or ambiguous knowledge targets in source comments are diagnosed | `test_validation_reports_stale_and_ambiguous_code_mentions` | Integrity | PASS | +| 16 | `graphify update` returns exit code 1 after rebuilding an invalid lattice | `test_update_automatically_fails_after_rebuild_when_lattice_is_invalid` plus public scratch workflow | CLI acceptance | PASS, both wiki and code diagnostics emitted | +| 17 | Projects without `lat.md/` keep their existing update behavior | `test_update_skips_knowledge_validation_for_projects_without_lattice` | Compatibility | PASS | +| 18 | Valid knowledge passes automatically through the public update workflow | `python -m graphify update .../graphify-knowledge-update-valid-94 --no-cluster` | CLI acceptance | PASS, 2 sections across 1 file | + +## RED evidence + +1. `uv run pytest -q tests/test_lattice_ingest.py` failed during collection with + `ModuleNotFoundError: graphify.lattice_ingest` before production code existed. +2. After the first minimal implementation, the strengthened tests failed because + cross-file shorthand links were pruned and summary-only queries returned + `No matching nodes found.` +3. The source-link compatibility test failed because no `documents` edge was + emitted before Markdown-aware parsing was implemented. +4. Independent review added four regressions which initially failed: incremental + `@lat` rediscovery, dotted lattice filenames, source-root containment, and + ignore-aware validation scanning. +5. The workflow milestone began with two expected failures: stale `@lat` comments + were silently ignored, and `graphify update` returned success for an invalid + knowledge lattice. + +Checkpoint commits: + +- `79fe28b` specifies the initial missing behavior. +- `cee64f3` specifies cross-file resolution and public retrieval behavior. + +## GREEN evidence + +- `uv run pytest -q tests/test_lattice_ingest.py`: 11 passed. +- `uv run pytest -q tests/test_lattice_ingest.py tests/test_manifest_ingest.py tests/test_languages.py tests/test_cli_export.py tests/test_query_cli.py tests/test_security.py`: 460 passed. +- `env -u DEEPSEEK_API_KEY -u DEEPSEEK_BASE_URL uv run pytest -q --ignore=tests/test_falkordb_integration.py`: 4,157 passed, 1 skipped. The excluded integration requires a FalkorDB server with the `GRAPH.QUERY` module; the available localhost service was plain Redis. +- Workflow milestone focused suite: 14 passed. +- Workflow milestone adjacent suite including update/watch behavior: 577 passed. +- Workflow milestone deterministic full suite: 4,160 passed, 1 skipped. +- `uv run ruff check graphify/lattice_ingest.py graphify/extract.py graphify/serve.py graphify/cli.py graphify/__main__.py tests/test_lattice_ingest.py`: passed. +- GREEN implementation checkpoint: `5dcee5a`. + +## Coverage + +`uv run pytest -q tests/test_lattice_ingest.py --cov=graphify.lattice_ingest --cov-report=term-missing --cov-fail-under=80` + +The initial seven-test milestone reached 89% statement coverage. The final +eleven-test suite adds independent-review coverage for incremental, security, +ignore, and dotted-filename edge cases. + +## Known boundaries + +- Graphify validates that referenced source files exist. Symbol-level source-link + validation is deferred to the next milestone, where links will resolve against + Graphify's language-aware symbol index. +- This milestone does not add a separate lat.md semantic index or service. + Curated summaries participate directly in Graphify's existing query scorer. +- Jcode changes are intentionally deferred until the Graphify public graph and + CLI contract is committed and stable. diff --git a/graphify/__main__.py b/graphify/__main__.py index 924ae986d..407cef8a6 100644 --- a/graphify/__main__.py +++ b/graphify/__main__.py @@ -52,6 +52,7 @@ _install_codex_hook, _install_gemini_hook, _install_kilo_plugin, + _install_jcode_hook, _install_opencode_plugin, _install_skill_references, _kilo_config_path, @@ -85,6 +86,7 @@ _uninstall_codex_hook, _uninstall_gemini_hook, _uninstall_kilo_plugin, + _uninstall_jcode_hook, _uninstall_opencode_plugin, claude_install, claude_uninstall, @@ -507,7 +509,7 @@ def _run_cli() -> None: print("Usage: graphify ") print() print("Commands:") - print(" install [--platform P] copy skill to platform config dir (claude|windows|codebuddy|codex|opencode|aider|amp|agents|claw|droid|trae|trae-cn|gemini|cursor|antigravity|hermes|kiro|pi|devin)") + print(" install [--platform P] copy skill to platform config dir (claude|windows|codebuddy|codex|jcode|opencode|aider|amp|agents|claw|droid|trae|trae-cn|gemini|cursor|antigravity|hermes|kiro|pi|devin)") print(" uninstall remove graphify from all detected platforms in one shot") print(" --purge also delete graphify-out/ directory") print(" path \"A\" \"B\" shortest path between two nodes in graph.json") @@ -539,6 +541,8 @@ def _run_cli() -> None: print(" --force overwrite graph.json even if the rebuild has fewer nodes") print(" (also: GRAPHIFY_FORCE=1 env var; use after refactors that delete code)") print(" --no-cluster skip clustering, write raw extraction only") + print(" check-knowledge validate lat.md wiki links and @lat code references") + print(" --json emit machine-readable validation results") print(" cluster-only rerun clustering on an existing graph.json and regenerate report") print(" --no-viz skip graph.html generation (useful for >5000 node graphs / CI)") print(" --graph path to graph.json (default /graphify-out/graph.json)") diff --git a/graphify/cli.py b/graphify/cli.py index 441e4ca36..e465b2af2 100644 --- a/graphify/cli.py +++ b/graphify/cli.py @@ -710,6 +710,52 @@ def _run_hook_guard(kind: str, strict: bool = False) -> None: pass +def _run_jcode_hook_guard() -> bool: + """Redirect Jcode's first raw code lookup to Graphify. + + Jcode sends the tool input JSON on stdin, exports metadata through + ``JCODE_HOOK_*``, and treats exit 2 plus stderr as a blocked tool call. + Return True only for the first matching raw lookup in a session; malformed + input and unsupported tools fail open. + """ + from graphify.paths import out_path + + try: + if not out_path("graph.json").is_file() or _query_stamp_fresh(): + return False + payload = json.loads(sys.stdin.buffer.read().decode("utf-8", "replace")) + if not isinstance(payload, dict): + return False + + tool_name = os.environ.get("JCODE_HOOK_TOOL_NAME", "").strip().lower() + command = str(payload.get("command") or "") + command_lower = command.lower() + if "graphify query" in command_lower: + return False + + is_raw_lookup = tool_name in {"agentgrep", "grep", "read"} + if tool_name == "bash": + is_raw_lookup = any( + token in command_lower + for token in ("grep", "ripgrep", "rg ", "find ", "fd ", "ack ", "ag ") + ) + if not is_raw_lookup: + return False + + session_id = os.environ.get("JCODE_HOOK_SESSION_ID", "").strip() + if not session_id or not _mark_session_denied(f"jcode-{session_id}"): + return False + + sys.stderr.write( + "Graphify knowledge graph is available for this project. " + "Run `graphify query \"\"` before raw " + "search/read, then retry this tool if more detail is needed.\n" + ) + return True + except Exception: + return False + + def _target_is_indexed(file_path: str, root: "Path") -> bool: """Guard the strict deny: only block a read of a file the graph actually indexes. Reads manifest.json (cheap, capped); on any doubt (missing/corrupt/oversized @@ -802,6 +848,25 @@ def _reenter_main() -> None: main() +def _print_knowledge_validation(result: dict) -> None: + """Print one deterministic human-readable knowledge validation report.""" + if result["valid"]: + print( + f"Knowledge lattice valid: {result['sections']} sections " + f"across {result['files']} files." + ) + return + for error in result["errors"]: + print( + f"{error['file']}:{error['line']}: " + f"{error['code']}: {error['message']}" + ) + print( + f"Knowledge lattice invalid: {len(result['errors'])} error(s).", + file=sys.stderr, + ) + + def dispatch_command(cmd: str) -> None: if cmd == "provider": from graphify.llm import _custom_providers_path, BACKENDS @@ -1658,6 +1723,26 @@ def dispatch_command(cmd: str) -> None: else: print(format_diagnostic_report(summary)) + elif cmd == "check-knowledge": + import json as _json + from graphify.lattice_ingest import validate_lattice + + check_path = Path(".") + for arg in sys.argv[2:]: + if not arg.startswith("--"): + check_path = Path(arg) + break + if not check_path.exists() or not check_path.is_dir(): + print(f"error: path not found or not a directory: {check_path}", file=sys.stderr) + sys.exit(1) + result = validate_lattice(check_path) + if "--json" in sys.argv: + print(_json.dumps(result, indent=2, ensure_ascii=False)) + else: + _print_knowledge_validation(result) + if not result["valid"]: + sys.exit(1) + elif cmd == "add": if len(sys.argv) < 3: print( @@ -2115,6 +2200,14 @@ def dispatch_command(cmd: str) -> None: ok = _rebuild_code(watch_path, force=force, no_cluster=no_cluster, block_on_lock=True) if ok: print("Code graph updated. For doc/paper/image changes run /graphify --update in your AI assistant.") + lattice_dir = watch_path.resolve() / "lat.md" + if lattice_dir.is_dir(): + from graphify.lattice_ingest import validate_lattice + + knowledge = validate_lattice(watch_path) + _print_knowledge_validation(knowledge) + if not knowledge["valid"]: + sys.exit(1) if not ( os.environ.get("GEMINI_API_KEY") or os.environ.get("GOOGLE_API_KEY") @@ -2147,6 +2240,10 @@ def dispatch_command(cmd: str) -> None: strict="--strict" in sys.argv[3:], ) sys.exit(0) + elif cmd == "jcode-hook": + # Jcode pre_tool gate: exit 2 blocks once and exposes stderr to the + # model; every unsupported/error path fails open with exit 0. + sys.exit(2 if _run_jcode_hook_guard() else 0) elif cmd == "check-update": if len(sys.argv) < 3: print("Usage: graphify check-update ", file=sys.stderr) diff --git a/graphify/extract.py b/graphify/extract.py index 2ea5b500b..46ca9c94f 100644 --- a/graphify/extract.py +++ b/graphify/extract.py @@ -16,6 +16,13 @@ from .cache import load_cached, save_cached from .mcp_ingest import extract_mcp_config, is_mcp_config_path from .manifest_ingest import extract_package_manifest, is_package_manifest_path +from .lattice_ingest import ( + extract_lattice_code_ref_edges, + extract_lattice_markdown, + is_lattice_markdown_path, + project_source_paths, + resolve_lattice_reference_edges, +) from .resolver_registry import ( LanguageResolver, register as register_language_resolver, @@ -4719,6 +4726,11 @@ def _is_cpp_header(path: Path) -> bool: def _get_extractor(path: Path) -> Any | None: """Return the correct extractor function for a file, or None if unsupported.""" + # A lat.md lattice is curated, validated knowledge rather than ordinary + # prose. Route it before generic Markdown so section ids, summaries and + # wiki-link edges remain compatible with lat.md's public format. + if is_lattice_markdown_path(path): + return extract_lattice_markdown if path.name.lower().endswith(".blade.php"): return extract_blade # MCP config files (.mcp.json, claude_desktop_config.json, ...) are routed @@ -5257,6 +5269,47 @@ def extract( all_nodes.extend(result.get("nodes", [])) all_edges.extend(result.get("edges", [])) all_raw_calls.extend(result.get("raw_calls", [])) + # Bind curated knowledge to implementation files through explicit + # `@lat: [[section]]` comments. The file endpoint uses the same pre-remap id + # recipe as every extractor, so the canonical path remap below updates it + # together with the file node. + resolve_lattice_reference_edges(all_edges, all_nodes) + lattice_changed = any(is_lattice_markdown_path(path) for path in paths) + if lattice_changed: + # Incremental updates often contain only the changed lat.md file. Rescan + # unchanged, ignore-filtered source files so newly-added section ids can + # bind to existing @lat comments in the merged graph. + code_ref_paths = project_source_paths(root) + else: + code_ref_paths = paths + lattice_code_edges = extract_lattice_code_ref_edges(code_ref_paths, all_nodes) + if lattice_changed: + # The incremental extraction result must retain these edges until it is + # merged with the previous graph. Add minimal file endpoints for unchanged + # mentioned sources, otherwise the normal dangling-edge cleanup would + # discard the relationship before the merge can reconnect it. + existing_ids = {str(node.get("id")) for node in all_nodes} + mentioned_paths = { + Path(str(edge["source_file"])) + for edge in lattice_code_edges + if edge.get("source_file") + } + for mentioned_path in sorted(mentioned_paths): + file_id = _file_node_id(mentioned_path) + if file_id in existing_ids: + continue + existing_ids.add(file_id) + all_nodes.append( + { + "id": file_id, + "label": mentioned_path.name, + "type": "file", + "file_type": "code", + "source_file": str(mentioned_path), + "source_location": "L1", + } + ) + all_edges.extend(lattice_code_edges) # Function / method / class def ids for the cross-file indirect_call callable # guard. Built from the `_callable` node marker AFTER the id-remap / disambiguation # passes below (which rewrite node ids), so it can never go stale — see the diff --git a/graphify/install.py b/graphify/install.py index fbe135bcf..f516175b2 100644 --- a/graphify/install.py +++ b/graphify/install.py @@ -455,6 +455,13 @@ def _skill_registration(skill_path: str = "~/.claude/skills/graphify/SKILL.md") "claude_md": False, "skill_refs": "agents", }, + "jcode": { + # Jcode follows Agent Skills and discovers global skills here. + "skill_file": "skill-agents.md", + "skill_dst": Path(".jcode") / "skills" / "graphify" / "SKILL.md", + "claude_md": False, + "skill_refs": "agents", + }, "devin": { # Monolith: devin ships the full SKILL.md inline, no references/ sidecar. "skill_file": "skill-devin.md", @@ -610,6 +617,9 @@ def install(platform: str = "claude", *, project: bool = False, project_dir: Pat project_dir = project_dir or Path(".") skill_dst = _copy_skill_file(platform, project=project, project_dir=project_dir) + if platform == "jcode" and not project: + _install_jcode_hook() + if platform == "kilo": # Kilo Code also supports a native /graphify command file. command_src = Path(__file__).parent / "command-kilo.md" @@ -680,6 +690,113 @@ def _print_install_usage() -> None: print(f"Platforms: {platforms}") print(" --strict block the first raw file read per session until one " "`graphify query` runs (Claude Code project hook only; needs --project)") + + +def _jcode_config_path() -> Path: + root = os.environ.get("JCODE_HOME") + return (Path(root) if root else Path.home() / ".jcode") / "config.toml" + + +def _render_toml_strings(values: list[str]) -> str: + encoded = [json.dumps(value, ensure_ascii=False) for value in values] + return encoded[0] if len(encoded) == 1 else "[" + ", ".join(encoded) + "]" + + +def _replace_jcode_pre_tool(config_text: str, values: list[str]) -> str: + """Replace only hooks.pre_tool while preserving the rest of config.toml.""" + import tomllib + + parsed = tomllib.loads(config_text) if config_text.strip() else {} + if not isinstance(parsed.get("hooks", {}), dict): + raise ValueError("[hooks] must be a TOML table") + + lines = config_text.splitlines() + section_start = next( + (i for i, line in enumerate(lines) if line.strip() == "[hooks]"), None + ) + rendered = "pre_tool = " + _render_toml_strings(values) if values else "" + if section_start is None: + if not values: + return config_text + prefix = config_text.rstrip() + return (prefix + "\n\n" if prefix else "") + "[hooks]\n" + rendered + "\n" + + section_end = next( + (i for i in range(section_start + 1, len(lines)) + if lines[i].strip().startswith("[")), + len(lines), + ) + assignment_start = next( + (i for i in range(section_start + 1, section_end) + if re.match(r"^\s*pre_tool\s*=", lines[i])), + None, + ) + if assignment_start is None: + if values: + lines.insert(section_end, rendered) + else: + assignment_end = assignment_start + 1 + for candidate_end in range(assignment_start + 1, section_end + 1): + snippet = "[hooks]\n" + "\n".join(lines[assignment_start:candidate_end]) + try: + candidate = tomllib.loads(snippet).get("hooks", {}).get("pre_tool") + except tomllib.TOMLDecodeError: + continue + if isinstance(candidate, (str, list)): + assignment_end = candidate_end + break + lines[assignment_start:assignment_end] = [rendered] if values else [] + + result = "\n".join(lines) + return result + ("\n" if config_text.endswith("\n") or result else "") + + +def _jcode_hook_values(config_text: str) -> list[str]: + import tomllib + + parsed = tomllib.loads(config_text) if config_text.strip() else {} + current = parsed.get("hooks", {}).get("pre_tool") + if current is None: + return [] + if isinstance(current, str): + return [current] + if isinstance(current, list) and all(isinstance(value, str) for value in current): + return list(current) + raise ValueError("hooks.pre_tool must be a string or an array of strings") + + +def _install_jcode_hook() -> None: + config_path = _jcode_config_path() + config_path.parent.mkdir(parents=True, exist_ok=True) + text = config_path.read_text(encoding="utf-8") if config_path.exists() else "" + exe = _resolve_graphify_exe() + if " " in exe and not exe.startswith('"'): + exe = f'"{exe}"' + try: + values = [v for v in _jcode_hook_values(text) if "jcode-hook" not in v] + values.append(f"{exe} jcode-hook") + updated = _replace_jcode_pre_tool(text, values) + except Exception as exc: + print(f"error: cannot update {config_path}: {exc}", file=sys.stderr) + sys.exit(1) + config_path.write_text(updated, encoding="utf-8") + print(f" Jcode pre_tool -> registered in {config_path}") + + +def _uninstall_jcode_hook() -> None: + config_path = _jcode_config_path() + if not config_path.exists(): + return + text = config_path.read_text(encoding="utf-8") + try: + values = [v for v in _jcode_hook_values(text) if "jcode-hook" not in v] + updated = _replace_jcode_pre_tool(text, values) + except Exception as exc: + print(f"error: cannot update {config_path}: {exc}", file=sys.stderr) + sys.exit(1) + if updated != text: + config_path.write_text(updated, encoding="utf-8") + print(f" Jcode pre_tool -> Graphify hook removed from {config_path}") _CLAUDE_MD_MARKER = "## graphify" _CODEBUDDY_MD_MARKER = "## graphify" _AGENTS_MD_MARKER = "## graphify" @@ -1798,6 +1915,8 @@ def uninstall_all(project_dir: Path | None = None, purge: bool = False) -> None: # The generic agents platform's user-scope skill lives at ~/.agents/skills, # which neither the AGENTS.md cleanup nor amp's removal reaches. _remove_skill_file("agents") + _remove_skill_file("jcode") + _uninstall_jcode_hook() _uninstall_opencode_plugin(pd) _uninstall_codex_hook(pd) @@ -2003,6 +2122,7 @@ def codebuddy_uninstall(project_dir: Path | None = None, *, project: bool = Fals "install", "kilo", "kiro", + "jcode", "opencode", "pi", "skills", @@ -2134,6 +2254,18 @@ def dispatch_install_cli(cmd: str) -> bool: else: print("Usage: graphify codebuddy [install|uninstall]", file=sys.stderr) sys.exit(1) + elif cmd == "jcode": + subcmd = sys.argv[2] if len(sys.argv) > 2 else "" + if subcmd == "install": + install(platform="jcode") + elif subcmd == "uninstall": + removed = _remove_skill_file("jcode") + if removed: + print("skill removed") + _uninstall_jcode_hook() + else: + print("Usage: graphify jcode [install|uninstall]", file=sys.stderr) + sys.exit(1) elif cmd == "gemini": subcmd = sys.argv[2] if len(sys.argv) > 2 else "" if subcmd == "install": diff --git a/graphify/lattice_ingest.py b/graphify/lattice_ingest.py new file mode 100644 index 000000000..521629011 --- /dev/null +++ b/graphify/lattice_ingest.py @@ -0,0 +1,513 @@ +"""Deterministic ingestion and validation for ``lat.md`` knowledge lattices. + +Graphify treats curated lattice sections as first-class graph nodes while keeping +lat.md's Markdown files as the source of truth. No Node.js runtime or lat CLI is +required: the supported interchange subset is headings, first-paragraph +summaries, ``[[wiki links]]``, ``@lat`` code comments, and the +``require-code-mention`` frontmatter flag. +""" + +from __future__ import annotations + +import re +from pathlib import Path +from typing import Any, Iterable + +from graphify.extractors.base import _make_id + +__all__ = [ + "extract_lattice_markdown", + "extract_lattice_code_ref_edges", + "is_lattice_markdown_path", + "project_source_paths", + "resolve_lattice_reference_edges", + "validate_lattice", +] + +_HEADING_RE = re.compile(r"^(#{1,6})\s+(.+?)\s*$") +_WIKI_RE = re.compile(r"\[\[([^\]|]+?)(?:\|[^\]]+)?\]\]") +_CODE_REF_RE = re.compile(r"(?:#|//)\s*@lat:\s*\[\[([^\]]+)\]\]") +_INLINE_CODE_RE = re.compile(r"`+[^`]*?`+") +_MAX_FILE_BYTES = 2_000_000 + + +def _lattice_dir(path: Path) -> Path | None: + path = path.resolve() + for parent in (path.parent, *path.parents): + if parent.name == "lat.md": + return parent + return None + + +def is_lattice_markdown_path(path: Path) -> bool: + """Return whether *path* is a Markdown file inside a ``lat.md/`` directory.""" + return path.suffix.lower() == ".md" and _lattice_dir(path) is not None + + +def _file_key(path: Path, lattice_dir: Path) -> str: + return path.resolve().relative_to(lattice_dir).with_suffix("").as_posix() + + +def _knowledge_node_id(knowledge_id: str) -> str: + return _make_id("knowledge", knowledge_id) + + +def _edge( + source: str, + target: str, + relation: str, + source_file: str, + line: int, + **extra: Any, +) -> dict[str, Any]: + edge: dict[str, Any] = { + "source": source, + "target": target, + "relation": relation, + "confidence": "EXTRACTED", + "confidence_score": 1.0, + "source_file": source_file, + "source_location": f"L{line}", + "weight": 1.0, + } + edge.update(extra) + return edge + + +def _wiki_refs(lines: list[str], start: int, end: int) -> list[tuple[str, int]]: + """Return real wiki links, excluding fenced and inline-code examples.""" + refs: list[tuple[str, int]] = [] + in_fence = False + for line_number in range(start, end + 1): + line = lines[line_number - 1] + if line.strip().startswith("```"): + in_fence = not in_fence + continue + if in_fence: + continue + searchable = _INLINE_CODE_RE.sub("", line) + refs.extend( + (match.group(1).strip(), line_number) for match in _WIKI_RE.finditer(searchable) + ) + return refs + + +def _source_target(target: str, project_root: Path) -> tuple[Path | None, str | None]: + file_part = target.split("#", 1)[0] + candidate = project_root / file_part + # A dot is legal in a lattice filename (for example operations.v2.md), so a + # suffix alone cannot distinguish knowledge from code. Existing files and + # explicit relative paths such as src/service.py are source references. + if not candidate.is_file() and "/" not in file_part.replace("\\", "/"): + return None, None + try: + resolved = candidate.resolve() + resolved.relative_to(project_root.resolve()) + except (OSError, RuntimeError, ValueError): + return None, "source reference escapes project root" + return resolved, None + + +def extract_lattice_markdown(path: Path) -> dict[str, Any]: + """Extract stable knowledge-section nodes from one lattice Markdown file.""" + lattice_dir = _lattice_dir(path) + if lattice_dir is None or path.suffix.lower() != ".md": + return {"nodes": [], "edges": []} + try: + if path.stat().st_size > _MAX_FILE_BYTES: + return {"nodes": [], "edges": [], "error": "lattice file too large to index"} + text = path.read_text(encoding="utf-8", errors="replace") + except OSError as exc: + return {"nodes": [], "edges": [], "error": f"lattice read error: {exc}"} + + file_key = _file_key(path, lattice_dir) + source_file = str(path) + lines = text.splitlines() + nodes: list[dict[str, Any]] = [] + edges: list[dict[str, Any]] = [] + sections: list[dict[str, Any]] = [] + stack: list[dict[str, Any]] = [] + in_fence = False + + for index, line in enumerate(lines, start=1): + if line.strip().startswith("```"): + in_fence = not in_fence + continue + if in_fence: + continue + match = _HEADING_RE.match(line) + if not match: + continue + depth = len(match.group(1)) + heading = match.group(2).strip() + while stack and stack[-1]["depth"] >= depth: + stack.pop() + parent = stack[-1] if stack else None + knowledge_id = f"{parent['knowledge_id']}#{heading}" if parent else f"{file_key}#{heading}" + section = { + "knowledge_id": knowledge_id, + "heading": heading, + "depth": depth, + "line": index, + "parent": parent, + } + sections.append(section) + stack.append(section) + + for position, section in enumerate(sections): + start = section["line"] + end = sections[position + 1]["line"] - 1 if position + 1 < len(sections) else len(lines) + paragraph: list[str] = [] + started = False + for raw in lines[start:end]: + stripped = raw.strip() + if not stripped: + if started: + break + continue + if stripped.startswith(("```", "---")): + continue + started = True + paragraph.append(stripped) + summary = " ".join(paragraph) + knowledge_id = section["knowledge_id"] + node_id = _knowledge_node_id(knowledge_id) + nodes.append( + { + "id": node_id, + "label": section["heading"], + "file_type": "document", + "type": "knowledge_section", + "knowledge_id": knowledge_id, + "summary": summary, + "source_file": source_file, + "source_location": f"L{section['line']}", + } + ) + parent = section["parent"] + if parent is not None: + edges.append( + _edge( + _knowledge_node_id(parent["knowledge_id"]), + node_id, + "contains", + source_file, + section["line"], + ) + ) + for target, reference_line in _wiki_refs(lines, start, end): + if target.startswith("#"): + target = f"{file_key}{target}" + source_target, source_error = _source_target(target, lattice_dir.parent) + if source_target is not None or source_error is not None: + edges.append( + _edge( + node_id, + _make_id(str(source_target)) + if source_target is not None + else "invalid_source", + "documents", + source_file, + reference_line, + source_target=target, + target_file=( + str(source_target) + if source_target is not None and source_target.is_file() + else None + ), + source_reference_error=source_error, + ) + ) + else: + edges.append( + _edge( + node_id, + _knowledge_node_id(target), + "references", + source_file, + reference_line, + knowledge_target=target, + ) + ) + + return {"nodes": nodes, "edges": edges, "input_tokens": 0, "output_tokens": 0} + + +def _resolve_ref(target: str, knowledge_ids: Iterable[str]) -> tuple[str | None, list[str]]: + ids = list(knowledge_ids) + lowered = {item.lower(): item for item in ids} + exact = lowered.get(target.lower()) + if exact is not None: + return exact, [] + target_lower = target.lower() + # A bare ref names a lattice file before it names an arbitrary heading. + # `[[locate]]` therefore resolves to `tests/locate.md` when that basename is + # unique, even if another document has a child heading named "locate". + if "#" not in target: + file_keys = { + item.split("#", 1)[0] + for item in ids + if item.split("#", 1)[0].lower() == target_lower + or item.split("#", 1)[0].lower().endswith("/" + target_lower) + } + if len(file_keys) == 1: + file_key = next(iter(file_keys)) + roots = [item for item in ids if item.startswith(file_key + "#")] + if roots: + return min(roots, key=lambda item: (item.count("#"), item)), [] + if len(file_keys) > 1: + roots = [ + min( + (item for item in ids if item.startswith(file_key + "#")), + key=lambda item: (item.count("#"), item), + ) + for file_key in sorted(file_keys) + ] + return None, roots + candidates = [ + item + for item in ids + if item.lower().endswith("/" + target_lower) or item.lower().endswith("#" + target_lower) + ] + if not candidates and "#" in target: + file_part, heading_part = target.split("#", 1) + file_suffix = file_part.lower() + heading_suffix = "#" + heading_part.lower() + candidates = [ + item + for item in ids + if item.split("#", 1)[0].lower().endswith(file_suffix) + and item.lower().endswith(heading_suffix) + ] + if len(candidates) == 1: + return candidates[0], [] + return None, sorted(candidates) + + +def resolve_lattice_reference_edges( + edges: Iterable[dict[str, Any]], nodes: Iterable[dict[str, Any]] +) -> None: + """Resolve wiki-link edge targets against the complete extracted lattice. + + Per-file extraction cannot know another file's full heading ancestry. This + post-pass upgrades shorthand such as ``operations#Deployment`` to the stable + section id ``operations#Operations#Deployment`` once all nodes are present. + Broken or ambiguous targets remain dangling and are pruned by the normal + graph builder; ``check-knowledge`` reports the actionable diagnostic. + """ + knowledge_ids = { + str(node["knowledge_id"]) + for node in nodes + if node.get("type") == "knowledge_section" and node.get("knowledge_id") + } + for edge in edges: + if edge.get("relation") != "references" or not edge.get("knowledge_target"): + continue + resolved, ambiguous = _resolve_ref(str(edge["knowledge_target"]), knowledge_ids) + if resolved is not None and not ambiguous: + edge["target"] = _knowledge_node_id(resolved) + edge["resolved_knowledge_target"] = resolved + + +def extract_lattice_code_ref_edges( + paths: Iterable[Path], nodes: Iterable[dict[str, Any]] +) -> list[dict[str, Any]]: + """Create ``knowledge_section --implemented_by--> source file`` edges.""" + knowledge_ids = { + str(node["knowledge_id"]) + for node in nodes + if node.get("type") == "knowledge_section" and node.get("knowledge_id") + } + if not knowledge_ids: + return [] + edges: list[dict[str, Any]] = [] + seen: set[tuple[str, str]] = set() + for path, line_number, target in _iter_code_refs(paths): + resolved, ambiguous = _resolve_ref(target, knowledge_ids) + if resolved is None or ambiguous: + continue + key = (resolved, str(path)) + if key in seen: + continue + seen.add(key) + edges.append( + _edge( + _knowledge_node_id(resolved), + _make_id(str(path)), + "implemented_by", + str(path), + line_number, + knowledge_id=resolved, + ) + ) + return edges + + +def _iter_code_refs(paths: Iterable[Path]) -> Iterable[tuple[Path, int, str]]: + """Yield bounded ``@lat`` references from eligible source files.""" + for path in paths: + if is_lattice_markdown_path(path) or path.suffix.lower() == ".md": + continue + try: + if path.stat().st_size > _MAX_FILE_BYTES: + continue + lines = path.read_text(encoding="utf-8", errors="replace").splitlines() + except OSError: + continue + for line_number, line in enumerate(lines, start=1): + for match in _CODE_REF_RE.finditer(line): + yield path, line_number, match.group(1).strip() + + +def _lattice_files(project_root: Path) -> list[Path]: + lattice_dir = project_root / "lat.md" + if not lattice_dir.is_dir(): + return [] + return sorted(path for path in lattice_dir.rglob("*.md") if path.is_file()) + + +def project_source_paths(project_root: Path) -> list[Path]: + """Return ignore-aware, in-root files eligible for ``@lat`` scanning.""" + from graphify.detect import ignored_predicate + + ignored = ignored_predicate(project_root) + paths: list[Path] = [] + for path in project_root.rglob("*"): + if not path.is_file() or is_lattice_markdown_path(path) or ignored(path): + continue + try: + path.resolve().relative_to(project_root) + except (OSError, RuntimeError, ValueError): + continue + paths.append(path) + return paths + + +def _requires_code_mention(text: str) -> bool: + frontmatter = re.match(r"^---\s*\n(.*?)\n---", text, flags=re.DOTALL) + return bool( + frontmatter and re.search(r"require-code-mention:\s*true", frontmatter.group(1), re.I) + ) + + +def validate_lattice(project_root: Path) -> dict[str, Any]: + """Validate wiki references and required ``@lat`` implementation mentions.""" + project_root = project_root.resolve() + files = _lattice_files(project_root) + extracted = [extract_lattice_markdown(path) for path in files] + nodes = [node for result in extracted for node in result.get("nodes", [])] + knowledge_ids = {str(node["knowledge_id"]) for node in nodes if node.get("knowledge_id")} + errors: list[dict[str, Any]] = [] + + for result in extracted: + for edge in result.get("edges", []): + if edge.get("relation") == "documents": + target = str(edge.get("source_target", "")) + if edge.get("source_reference_error"): + errors.append( + { + "code": "unsafe-source-reference", + "file": edge["source_file"], + "line": int(str(edge["source_location"])[1:]), + "target": target, + "message": f"unsafe source reference [[{target}]]", + } + ) + continue + target_path = Path(str(edge.get("target_file") or "")) + if not target_path.is_file(): + errors.append( + { + "code": "broken-source-reference", + "file": edge["source_file"], + "line": int(str(edge["source_location"])[1:]), + "target": target, + "message": f"broken source reference [[{target}]]", + } + ) + continue + if edge.get("relation") != "references": + continue + target = str(edge.get("knowledge_target", "")) + resolved, ambiguous = _resolve_ref(target, knowledge_ids) + if ambiguous: + errors.append( + { + "code": "ambiguous-reference", + "file": edge["source_file"], + "line": int(str(edge["source_location"])[1:]), + "target": target, + "candidates": ambiguous, + "message": f"ambiguous knowledge reference [[{target}]]", + } + ) + elif resolved is None: + errors.append( + { + "code": "broken-reference", + "file": edge["source_file"], + "line": int(str(edge["source_location"])[1:]), + "target": target, + "message": f"broken knowledge reference [[{target}]]", + } + ) + + source_paths = project_source_paths(project_root) + code_refs = list(_iter_code_refs(source_paths)) + implemented: set[str] = set() + for source_path, line_number, target in code_refs: + resolved, ambiguous = _resolve_ref(target, knowledge_ids) + if ambiguous: + errors.append( + { + "code": "ambiguous-code-reference", + "file": str(source_path), + "line": line_number, + "target": target, + "candidates": ambiguous, + "message": f"ambiguous @lat reference [[{target}]]", + } + ) + elif resolved is None: + errors.append( + { + "code": "broken-code-reference", + "file": str(source_path), + "line": line_number, + "target": target, + "message": f"stale @lat reference [[{target}]]", + } + ) + else: + implemented.add(resolved) + for file_path in files: + text = file_path.read_text(encoding="utf-8", errors="replace") + if not _requires_code_mention(text): + continue + result = extract_lattice_markdown(file_path) + parent_ids = { + edge["source"] for edge in result.get("edges", []) if edge.get("relation") == "contains" + } + for node in result.get("nodes", []): + if node["id"] in parent_ids: + continue + knowledge_id = str(node["knowledge_id"]) + if knowledge_id not in implemented: + errors.append( + { + "code": "missing-code-mention", + "file": str(file_path), + "line": int(str(node["source_location"])[1:]), + "target": knowledge_id, + "message": f"knowledge section [[{knowledge_id}]] has no @lat code mention", + } + ) + + errors.sort(key=lambda item: (item["file"], item["line"], item["code"], item["target"])) + return { + "valid": not errors, + "lattice_dir": str(project_root / "lat.md"), + "files": len(files), + "sections": len(knowledge_ids), + "errors": errors, + } diff --git a/graphify/serve.py b/graphify/serve.py index 3b205d84f..a8bddc0d2 100644 --- a/graphify/serve.py +++ b/graphify/serve.py @@ -295,8 +295,9 @@ def _compute_idf(G: nx.Graph, terms: list[str]) -> dict[str, float]: norm_label = ( data.get("norm_label") or _strip_diacritics(data.get("label") or "") ).lower() + summary = _strip_diacritics(data.get("summary") or "").lower() for t in uncached: - if t in norm_label: + if t in norm_label or t in summary: df[t] += 1 for t in uncached: cache[t] = math.log(1 + N / (1 + df[t])) @@ -330,7 +331,8 @@ def _node_search_text(data: dict, nid: str) -> str: label_tokens = " ".join(_search_tokens(data.get("label") or "")) source = (data.get("source_file") or "").lower() source_tokens = " ".join(_search_tokens(data.get("source_file") or "")) - return "\x00".join((norm_label, label_tokens, str(nid).lower(), source, source_tokens)) + summary = _strip_diacritics(data.get("summary") or "").lower() + return "\x00".join((norm_label, label_tokens, str(nid).lower(), source, source_tokens, summary)) def _get_trigram_index(G: nx.Graph) -> dict: @@ -506,6 +508,7 @@ def _score_query( # driver". label_tokens = " ".join(_search_tokens(data.get("label") or "")) source = (data.get("source_file") or "").lower() + summary = _strip_diacritics(data.get("summary") or "").lower() # `nid_lower` is needed both by the full-query tier (`if joined`) and by # the per-token singleton tier (joined-singlet exact-match check). When # neither runs (`joined` empty AND not collecting seeds) skip the call; @@ -552,19 +555,29 @@ def _score_query( tier_value = 0.0 substr_value = 0.0 source_value = 0.0 + summary_value = 0.0 + label_matched = False if t == norm_label or t == bare_label: tier_value = _EXACT_MATCH_BONUS * w matched += 1 + label_matched = True elif norm_label.startswith(t) or bare_label.startswith(t): tier_value = _PREFIX_MATCH_BONUS * w matched += 1 + label_matched = True elif t in norm_label: substr_value = _SUBSTRING_MATCH_BONUS * w score += substr_value matched += 1 + label_matched = True if t in source: source_value = _SOURCE_MATCH_BONUS * w score += source_value + if t in summary: + summary_value = _SUBSTRING_MATCH_BONUS * w + score += summary_value + if not label_matched: + matched += 1 tiered += tier_value if collect_per_term_seeds and best_by_term is not None: # Singleton score for [t] on this node, mirroring @@ -583,7 +596,7 @@ def _score_query( singleton = _PREFIX_MATCH_BONUS * 10 * w else: singleton = 0.0 - singleton += tier_value + substr_value + source_value + singleton += tier_value + substr_value + source_value + summary_value if singleton > 0: # Tie-break key mirrors the legacy sort+max(degree): # (-singleton, -degree, label_len, nid) — the minimum @@ -1018,6 +1031,9 @@ def _adj(n): f"community={sanitize_label(str(d.get('community_name') or d.get('community', '')))}" f"{learning_suffix}]" ) + summary = str(d.get("summary") or "").strip() + if summary: + line += f" summary={sanitize_label(summary[:500])}" lines.append(line) for u, v in edges: if u in nodes and v in nodes: diff --git a/tests/test_jcode.py b/tests/test_jcode.py new file mode 100644 index 000000000..11075adae --- /dev/null +++ b/tests/test_jcode.py @@ -0,0 +1,149 @@ +"""Native Jcode integration tests. + +Jcode exposes a global skill directory and a blocking ``pre_tool`` hook. The +Graphify integration installs both and uses the hook to redirect the first raw +code search/read in a session to the existing knowledge graph. +""" + +from __future__ import annotations + +import json +import os +from pathlib import Path +import subprocess +import sys +from unittest.mock import patch + + +PYTHON = sys.executable + + +def _run_jcode_hook( + cwd: Path, + *, + tool_name: str, + tool_input: dict, + session_id: str, +) -> subprocess.CompletedProcess[str]: + env = os.environ.copy() + env.update( + { + "JCODE_HOOK_EVENT": "pre_tool", + "JCODE_HOOK_TOOL_NAME": tool_name, + "JCODE_HOOK_SESSION_ID": session_id, + "JCODE_HOOK_CWD": str(cwd), + } + ) + return subprocess.run( + [PYTHON, "-m", "graphify", "jcode-hook"], + cwd=cwd, + env=env, + input=json.dumps(tool_input), + capture_output=True, + text=True, + ) + + +def test_jcode_hook_redirects_first_raw_search_then_fails_open(tmp_path: Path) -> None: + out = tmp_path / "graphify-out" + out.mkdir() + (out / "graph.json").write_text("{}", encoding="utf-8") + + first = _run_jcode_hook( + tmp_path, + tool_name="agentgrep", + tool_input={"query": "MemoryManager", "path": "src"}, + session_id="ses-jcode-search", + ) + second = _run_jcode_hook( + tmp_path, + tool_name="agentgrep", + tool_input={"query": "MemoryManager", "path": "src"}, + session_id="ses-jcode-search", + ) + + assert first.returncode == 2 + assert "graphify query" in first.stderr + assert second.returncode == 0 + assert second.stderr == "" + + +def test_jcode_hook_allows_graphify_queries_and_unrelated_tools(tmp_path: Path) -> None: + out = tmp_path / "graphify-out" + out.mkdir() + (out / "graph.json").write_text("{}", encoding="utf-8") + + graphify_query = _run_jcode_hook( + tmp_path, + tool_name="bash", + tool_input={"command": "graphify query 'memory architecture'"}, + session_id="ses-jcode-query", + ) + unrelated = _run_jcode_hook( + tmp_path, + tool_name="write", + tool_input={"file_path": "notes.txt", "content": "hello"}, + session_id="ses-jcode-write", + ) + + assert graphify_query.returncode == 0 + assert unrelated.returncode == 0 + + +def test_jcode_install_registers_skill_and_pre_tool_hook_idempotently( + tmp_path: Path, + monkeypatch, +) -> None: + from graphify.__main__ import main + + home = tmp_path / "home" + project = tmp_path / "project" + project.mkdir() + config = home / ".jcode" / "config.toml" + config.parent.mkdir(parents=True) + config.write_text( + '[agents]\nmemory_sidecar_enabled = true\n\n[hooks]\npre_tool = "keep-existing-policy"\n', + encoding="utf-8", + ) + + monkeypatch.chdir(project) + with patch("graphify.__main__.Path.home", return_value=home): + monkeypatch.setattr(sys, "argv", ["graphify", "jcode", "install"]) + main() + main() + + skill = home / ".jcode" / "skills" / "graphify" / "SKILL.md" + text = config.read_text(encoding="utf-8") + assert skill.exists() + assert "keep-existing-policy" in text + assert text.count("jcode-hook") == 1 + + +def test_jcode_uninstall_removes_only_graphify_owned_entries( + tmp_path: Path, + monkeypatch, +) -> None: + from graphify.__main__ import main + + home = tmp_path / "home" + project = tmp_path / "project" + project.mkdir() + config = home / ".jcode" / "config.toml" + config.parent.mkdir(parents=True) + config.write_text( + '[hooks]\npre_tool = ["keep-existing-policy", "/usr/bin/graphify jcode-hook"]\n', + encoding="utf-8", + ) + skill = home / ".jcode" / "skills" / "graphify" / "SKILL.md" + skill.parent.mkdir(parents=True) + skill.write_text("graphify", encoding="utf-8") + + monkeypatch.chdir(project) + with patch("graphify.__main__.Path.home", return_value=home): + monkeypatch.setattr(sys, "argv", ["graphify", "jcode", "uninstall"]) + main() + + text = config.read_text(encoding="utf-8") + assert "keep-existing-policy" in text + assert "jcode-hook" not in text + assert not skill.exists() diff --git a/tests/test_lattice_ingest.py b/tests/test_lattice_ingest.py new file mode 100644 index 000000000..d0d579086 --- /dev/null +++ b/tests/test_lattice_ingest.py @@ -0,0 +1,296 @@ +from __future__ import annotations + +import json +import os +import subprocess +import sys +from pathlib import Path + +import pytest + +from graphify.build import build_from_json +from graphify.cli import dispatch_command +from graphify.extract import extract +from graphify.lattice_ingest import ( + extract_lattice_markdown, + is_lattice_markdown_path, + validate_lattice, +) +from graphify.serve import _query_graph_text + + +def _write(path: Path, content: str) -> Path: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(content, encoding="utf-8") + return path + + +def test_lattice_markdown_emits_stable_sections_summaries_and_wiki_edges(tmp_path): + overview = _write( + tmp_path / "lat.md" / "architecture" / "overview.md", + "# Architecture\n\nSystem-wide design constraints.\n\n" + "## Tenant isolation\n\nEvery query is scoped by tenant_id. See [[operations#Deployment]].\n", + ) + _write( + tmp_path / "lat.md" / "operations.md", + "# Operations\n\nOperational guidance.\n\n## Deployment\n\nDeploy through the documented pipeline.\n", + ) + + result = extract_lattice_markdown(overview) + nodes = {node["knowledge_id"]: node for node in result["nodes"] if node.get("knowledge_id")} + + assert is_lattice_markdown_path(overview) + assert "architecture/overview#Architecture" in nodes + assert "architecture/overview#Architecture#Tenant isolation" in nodes + tenant = nodes["architecture/overview#Architecture#Tenant isolation"] + assert tenant["type"] == "knowledge_section" + assert tenant["summary"] == "Every query is scoped by tenant_id. See [[operations#Deployment]]." + assert tenant["source_location"] == "L5" + + relations = {(edge["relation"], edge.get("knowledge_target")) for edge in result["edges"]} + assert ("references", "operations#Deployment") in relations + assert any(edge["relation"] == "contains" for edge in result["edges"]) + + +def test_lattice_ignores_example_links_and_emits_source_documentation_edges(tmp_path): + source = _write(tmp_path / "src" / "service.py", "def enforce():\n return True\n") + spec = _write( + tmp_path / "lat.md" / "security.md", + "# Security\n\nDocuments [[src/service.py#enforce]].\n\n" + "## Syntax examples\n\nInline `[[not-a-reference]]` and fenced examples are ignored.\n\n" + "```md\n[[also-not-a-reference]]\n```\n", + ) + + result = extract([spec, source], cache_root=tmp_path, root=tmp_path, parallel=False) + graph = build_from_json(result, directed=True, root=tmp_path) + + documented = [ + (graph.nodes[src].get("knowledge_id"), graph.nodes[dst].get("source_file")) + for src, dst, data in graph.edges(data=True) + if data.get("relation") == "documents" + ] + assert ("security#Security", "src/service.py") in documented + diagnostics = validate_lattice(tmp_path) + assert diagnostics["valid"] is True, diagnostics["errors"] + + +def test_full_extract_links_at_lat_comment_to_knowledge_section(tmp_path): + spec = _write( + tmp_path / "lat.md" / "security.md", + "# Security\n\nSecurity constraints.\n\n## Tenant isolation\n\nAll reads require tenant_id.\n", + ) + source = _write( + tmp_path / "src" / "repository.py", + "# @lat: [[security#Security#Tenant isolation]]\n" + "def load_orders(tenant_id):\n" + " return tenant_id\n", + ) + + result = extract([spec, source], cache_root=tmp_path, root=tmp_path, parallel=False) + graph = build_from_json(result, directed=True, root=tmp_path) + + edges = [ + (src, dst, data) + for src, dst, data in graph.edges(data=True) + if data.get("relation") == "implemented_by" + ] + assert len(edges) == 1 + src, dst, data = edges[0] + assert graph.nodes[src]["knowledge_id"] == "security#Security#Tenant isolation" + assert graph.nodes[dst]["source_file"] == "src/repository.py" + assert data["source_location"] == "L1" + + +def test_full_extract_resolves_cross_file_short_wiki_reference(tmp_path): + overview = _write( + tmp_path / "lat.md" / "overview.md", + "# Overview\n\nArchitecture overview.\n\n## Runtime\n\nSee [[operations#Deployment]].\n", + ) + operations = _write( + tmp_path / "lat.md" / "operations.md", + "# Operations\n\nOperations summary.\n\n## Deployment\n\nDeployment constraints.\n", + ) + + result = extract([overview, operations], cache_root=tmp_path, root=tmp_path, parallel=False) + graph = build_from_json(result, directed=True, root=tmp_path) + + reference_edges = [ + (graph.nodes[src].get("knowledge_id"), graph.nodes[dst].get("knowledge_id")) + for src, dst, data in graph.edges(data=True) + if data.get("relation") == "references" + ] + assert ( + "overview#Overview#Runtime", + "operations#Operations#Deployment", + ) in reference_edges + + +def test_dotted_lattice_file_reference_is_not_misclassified_as_source(tmp_path): + overview = _write( + tmp_path / "lat.md" / "overview.md", + "# Overview\n\nSee [[operations.v2#Deployment]].\n", + ) + operations = _write( + tmp_path / "lat.md" / "operations.v2.md", + "# Operations\n\n## Deployment\n\nDeployment constraints.\n", + ) + + result = extract([overview, operations], cache_root=tmp_path, root=tmp_path, parallel=False) + graph = build_from_json(result, directed=True, root=tmp_path) + + assert any( + data.get("relation") == "references" + and graph.nodes[dst].get("knowledge_id") == "operations.v2#Operations#Deployment" + for _, dst, data in graph.edges(data=True) + ) + + +def test_lattice_change_rescans_unchanged_source_mentions(tmp_path): + _write( + tmp_path / "src" / "repository.py", + "# @lat: [[security#Security#Tenant isolation]]\ndef load():\n return True\n", + ) + spec = _write( + tmp_path / "lat.md" / "security.md", + "# Security\n\n## Tenant isolation\n\nAll reads require tenant_id.\n", + ) + + # An incremental update may pass only the changed lattice file. Graphify must + # still rediscover @lat mentions in unchanged source files. + result = extract([spec], cache_root=tmp_path, root=tmp_path, parallel=False) + + assert any( + edge.get("relation") == "implemented_by" + and edge.get("knowledge_id") == "security#Security#Tenant isolation" + and edge["source_file"] == "src/repository.py" + for edge in result["edges"] + ) + + +def test_source_reference_cannot_escape_project_root(tmp_path): + outside = _write(tmp_path.parent / "outside.py", "SECRET = True\n") + _write( + tmp_path / "lat.md" / "security.md", + "# Security\n\nNever index [[../outside.py#SECRET]].\n", + ) + + diagnostics = validate_lattice(tmp_path) + + assert diagnostics["valid"] is False + assert any(error["code"] == "unsafe-source-reference" for error in diagnostics["errors"]) + assert str(outside.resolve()) not in json.dumps(diagnostics) + + +def test_validation_respects_graphifyignore_when_scanning_code_mentions(tmp_path): + _write(tmp_path / ".graphifyignore", "ignored/\n") + _write( + tmp_path / "lat.md" / "security.md", + "---\nlat:\n require-code-mention: true\n---\n" + "# Security\n\n## Tenant isolation\n\nAll reads require tenant_id.\n", + ) + _write( + tmp_path / "ignored" / "fake.py", + "# @lat: [[security#Security#Tenant isolation]]\n", + ) + + diagnostics = validate_lattice(tmp_path) + + assert diagnostics["valid"] is False + assert any(error["code"] == "missing-code-mention" for error in diagnostics["errors"]) + + +def test_validation_reports_stale_and_ambiguous_code_mentions(tmp_path): + _write(tmp_path / "lat.md" / "a" / "rules.md", "# Rules\n\nA rules summary.\n") + _write(tmp_path / "lat.md" / "b" / "rules.md", "# Rules\n\nB rules summary.\n") + _write( + tmp_path / "src" / "repository.py", + "# @lat: [[missing#Section]]\n# @lat: [[rules#Rules]]\ndef load():\n return True\n", + ) + + diagnostics = validate_lattice(tmp_path) + codes = {error["code"] for error in diagnostics["errors"]} + + assert "broken-code-reference" in codes + assert "ambiguous-code-reference" in codes + + +def test_update_automatically_fails_after_rebuild_when_lattice_is_invalid( + tmp_path, monkeypatch, capsys +): + _write(tmp_path / "lat.md" / "index.md", "# Index\n\nSee [[missing#Section]].\n") + monkeypatch.setattr("graphify.watch._rebuild_code", lambda *args, **kwargs: True) + monkeypatch.setattr(sys, "argv", ["graphify", "update", str(tmp_path)]) + + with pytest.raises(SystemExit) as exc: + dispatch_command("update") + + captured = capsys.readouterr() + assert exc.value.code == 1 + assert "Knowledge lattice invalid" in captured.err + assert "broken-reference" in captured.out + + +def test_update_skips_knowledge_validation_for_projects_without_lattice( + tmp_path, monkeypatch, capsys +): + monkeypatch.setattr("graphify.watch._rebuild_code", lambda *args, **kwargs: True) + monkeypatch.setattr(sys, "argv", ["graphify", "update", str(tmp_path)]) + + dispatch_command("update") + + captured = capsys.readouterr() + assert "Code graph updated" in captured.out + assert "Knowledge lattice" not in captured.out + captured.err + + +def test_validate_lattice_reports_broken_ambiguous_and_unimplemented_required_sections(tmp_path): + _write(tmp_path / "lat.md" / "a" / "rules.md", "# Rules\n\nA rules summary.\n") + _write(tmp_path / "lat.md" / "b" / "rules.md", "# Rules\n\nB rules summary.\n") + _write( + tmp_path / "lat.md" / "index.md", + "---\nlat:\n require-code-mention: true\n---\n" + "# Index\n\nIndex summary.\n\n" + "## Required behavior\n\nMust be implemented. See [[rules#Rules]] and [[missing#Section]].\n", + ) + + diagnostics = validate_lattice(tmp_path) + codes = {item["code"] for item in diagnostics["errors"]} + + assert diagnostics["valid"] is False + assert "ambiguous-reference" in codes + assert "broken-reference" in codes + assert "missing-code-mention" in codes + + +def test_check_knowledge_cli_returns_json_and_nonzero_for_invalid_lattice(tmp_path): + _write(tmp_path / "lat.md" / "index.md", "# Index\n\nSee [[missing#Section]].\n") + env = os.environ.copy() + env["PYTHONPATH"] = str(Path(__file__).parents[1]) + + result = subprocess.run( + [sys.executable, "-m", "graphify", "check-knowledge", str(tmp_path), "--json"], + cwd=tmp_path, + env=env, + text=True, + capture_output=True, + ) + + assert result.returncode == 1 + payload = json.loads(result.stdout) + assert payload["valid"] is False + assert payload["errors"][0]["code"] == "broken-reference" + + +def test_query_retrieves_lattice_summary_after_normal_extraction(tmp_path): + spec = _write( + tmp_path / "lat.md" / "security.md", + "# Security\n\nAuthentication and authorization constraints.\n\n" + "## Tenant isolation\n\nEvery database query must include tenant_id.\n", + ) + result = extract([spec], cache_root=tmp_path, root=tmp_path, parallel=False) + graph = build_from_json(result, directed=True, root=tmp_path) + + output = _query_graph_text(graph, "tenant_id database query", token_budget=500) + + assert "Tenant isolation" in output + assert "Every database query must include tenant_id." in output diff --git a/uv.lock b/uv.lock index 8573a9e9d..44e05f4ab 100644 --- a/uv.lock +++ b/uv.lock @@ -1090,7 +1090,7 @@ wheels = [ [[package]] name = "graphifyy" -version = "0.9.31" +version = "0.9.37" source = { editable = "." } dependencies = [ { name = "networkx", version = "3.4.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" },