Skip to content

feat(yaml): extract Docker Compose services and GitHub Actions jobs - #2541

Open
bruno-growthsales wants to merge 1 commit into
Graphify-Labs:v8from
bruno-growthsales:feat/yaml-extractor
Open

feat(yaml): extract Docker Compose services and GitHub Actions jobs#2541
bruno-growthsales wants to merge 1 commit into
Graphify-Labs:v8from
bruno-growthsales:feat/yaml-extractor

Conversation

@bruno-growthsales

Copy link
Copy Markdown

Problem

.yaml/.yml have no AST extractor, so infrastructure-as-config contributes nothing structural to the graph. On a repo whose deploy path is docker-build.yml → Portainer/Swarm, the deploy topology was the one part of the architecture the graph could not answer questions about — the files are read by the semantic pass, but the service and job dependencies never become edges.

What this adds

extract_yaml, behind an optional [yaml] extra (tree-sitter-yaml), mirroring the sql / terraform pattern:

  • Docker Compose — service nodes; depends_on edges for both the list form and the long-form mapping ({db: {condition: ...}}); extends → base service.
  • GitHub Actions — job nodes; needsdepends_on (scalar and list forms); step-level and job-level uses → the action or reusable workflow.
  • Data YAML (k8s manifests, OpenAPI specs, fixtures) returns an empty result and is left to the semantic pass, the same way _is_config_json skips data JSON (AST pass explodes data .json into orphan key-nodes (CODE_EXTENSIONS includes .json) — 561 isolated nodes on a real repo #1224). A 1 MiB ceiling with the same bounded read keeps multi-MB lockfiles from being parsed for nothing.

On a real monorepo: docker-compose.yml → 5 services and 4 depends_on edges; ci.yml → 12 jobs, the build → lint-and-typecheck dependency, and 22 uses edges.

Two design decisions worth reviewing

1. The extensions stay in DOC_EXTENSIONS. collect_files keys off _DISPATCH, not CODE_EXTENSIONS, so registering the extractor is enough to get the structural pass without removing YAML from the semantic one — the same arrangement .md already has. It also avoids the #1689 "classified as code but no extractor" warning that moving the extensions would otherwise require handling.

2. External uses targets are marked type=module. Definitions are file-scoped with a contains edge; cross-file references are sourceless stubs so _rewire_unique_stub_nodes can collapse a Compose overlay's depends_on onto the base file's service (#2324, #1402). But an actions/checkout@v4 pinned by ten workflows is never defined anywhere in the corpus, so there is nothing to rewire onto, and _disambiguate_colliding_node_ids salts each stub with its own path — scattering one shared action into ten nodes. Marking them type=module takes the module-anchor exemption (#1327), whose rationale reads exactly like this case: "those are the same module, not distinct same-named symbols". Happy to take a different approach here if you would rather not overload type.

Tests

tests/test_yaml_config.py — 19 tests covering both shapes, the sequence-marker trap (- api vs api), forward references binding locally, data YAML staying empty, and the two cross-file cases (overlay rewire, shared-action merge) exercised through extract() and build_from_json.

Full suite: 4053 passed, 0 failed. ruff check clean.

Note on uv.lock

CI runs uv sync --frozen, so the lock had to be regenerated for the new extra. Regenerating also picked up a 0.9.31 → 0.9.35 project-version correction the lock had drifted from — unrelated to this change, but not separable from it.

🤖 Generated with Claude Code

https://claude.ai/code/session_013EeP6fFjQkaE2CxNz3RTnY

.yaml/.yml had no AST extractor, so infra-as-config contributed nothing
structural to the graph. On a repo whose deploy path is
docker-build.yml -> Portainer/Swarm, the deploy topology was the one part
of the architecture the graph could not answer questions about.

Adds extract_yaml behind an optional [yaml] extra (tree-sitter-yaml),
mirroring the sql/terraform pattern:

- Compose: service nodes; depends_on edges for both the list and the
  long-form mapping shape; extends -> base service.
- Actions: job nodes; needs -> depends_on (scalar and list forms);
  step-level and job-level `uses` -> the action / reusable workflow.
- Data YAML (k8s manifests, OpenAPI specs, fixtures) returns an empty
  result and is left to the semantic pass, the same way _is_config_json
  skips data JSON (Graphify-Labs#1224). A 1 MiB ceiling with the same bounded read
  keeps multi-MB lockfiles from being parsed for nothing.

Extensions stay in DOC_EXTENSIONS. collect_files keys off _DISPATCH, not
CODE_EXTENSIONS, so registering the extractor is enough to get the
structural pass without removing YAML from the semantic one -- the same
arrangement .md already has, and it avoids the Graphify-Labs#1689 "classified as code
but no extractor" warning.

Definitions are file-scoped and carry a contains edge; cross-file
references are sourceless stubs so _rewire_unique_stub_nodes can collapse
a Compose overlay's depends_on onto the base file's service (Graphify-Labs#2324,
Graphify-Labs#1402). An external `uses` target is marked type=module: the same action
pinned by ten workflows is ONE action, so it takes the module-anchor
exemption in _disambiguate_colliding_node_ids (Graphify-Labs#1327) instead of
scattering into ten path-salted nodes.

uv.lock also picks up the 0.9.31 -> 0.9.35 project-version correction it
had drifted from; CI runs --frozen, so the lock had to be regenerated for
the new extra regardless.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013EeP6fFjQkaE2CxNz3RTnY
@bruno-growthsales
bruno-growthsales changed the base branch from main to v8 August 7, 2026 18:32

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).


Graphify review — findings

This PR expands CI/CD and repository tooling for the graphify project. It adds several GitHub Actions workflows (a skillgen artifact-consistency check, PyPI trusted publishing, a self-graph release asset builder, and security scanning via bandit/pip-audit), switches the existing test workflow from pip to uv with full-history checkouts, and broadens the trigger branches. It also introduces supporting config files: .dockerignore, .gitattributes (Linguist HTML handling), .github/FUNDING.yml, a pre-commit hook running the skillgen check, and expanded .gitignore entries. The changed-symbols list additionally references many test, extractor, and skillgen-related files across the codebase, suggesting accompanying source/test/fixture changes not shown in the truncated diff. Reviewers should look at the full diff for those areas since only the CI/config portion is visible here.

No blocking issues surfaced.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 12643 functions depend on the 12624 functions this change touches.

Health — this change adds coupling hotspots:

  • worse: extract() — 373 callers, 39 callees
  • worse: _rebuild_code() — 93 callers, 52 callees
  • worse: build_from_json() — 153 callers, 17 callees
  • worse: detect() — 82 callers, 13 callees
  • new: deduplicate_entities() — 49 callers, 21 callees
  • new: build_merge() — 41 callers, 14 callees
  • worse: save_semantic_cache() — 50 callers, 9 callees
  • new: _extract_generic() — 18 callers, 20 callees
  • …and 198 more

Verification — 12643 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 12643 function(s) in the blast radius were not formally verified this run

@bruno-growthsales

Copy link
Copy Markdown
Author

Disclosing a limitation I hit while running this on a real monorepo, since a reviewer would reasonably ask about it.

The extractor is not reachable from graphify extract. That command builds its AST list from detect's code category:

code_files = [Path(p) for p in files_by_type.get("code", [])]

.yaml/.yml are DOC_EXTENSIONS, so on a fresh build they land in doc_files and — with --code-only — are skipped outright. The structural pass only sees them through collect_files, which keys off _DISPATCH and is used by update / watch.

In practice the build is two steps:

graphify extract . --code-only --out .   # no YAML
graphify update .                        # YAML appears: 43 nodes, 21 depends_on, 42 uses

This is pre-existing and not introduced here — .md has exactly the same shape today, with extract_markdown registered in _DISPATCH but unreachable from extract. I left it that way rather than widening the scope of this PR.

Two ways forward, whichever you prefer:

  1. Merge as-is. YAML lands via update, which is the documented steady-state command anyway.
  2. Follow-up PR widening extract's AST list to any extension with a registered extractor, instead of detect['files']['code']. Strictly additive — nothing currently extracted stops being extracted, and .md gains structural coverage in the initial build too. Happy to send it separately if you want it.

What I'd argue against is moving .yaml/.yml into CODE_EXTENSIONS: it would close the gap, but data YAML (k8s manifests, OpenAPI specs) would leave the semantic pass, and since extract_yaml deliberately returns empty for those, they would contribute nothing at all. That trades a gap for a regression.

Separately, while testing this I ran into an unrelated bug that made the results confusing at first — filed as #2543.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant