Collapse semantics to one artifact per namespace

CHANGELOG.md

@@ -7,7 +7,7 @@
 - Add `entropy-data sync --source <conn> --target <conn> --include <resources>` to copy portable organization state from one connection to another by exporting the source into a staged directory and importing it into the target. `sync` copies **nothing by default** — the resources to copy must be named with `--include`, and a run without it fails with a non-zero exit and the list of known resources. Supports `--exclude`, `--prune`, `--dry-run` (per-resource create/update/prune counts), and `--keep <dir>` to retain the staged export. Supported resources: `teams`, `tags`, `definitions`, `policies`, `sourcesystems`, `certifications`, `classification-schemes`, `assets`, `datacontracts`, `dataproducts`, `example-data`, `access`, `semantic-namespaces`, `semantic-ontology`, `organization-features`. Not synced: users & team members, API keys, git credentials, integration/connector credentials, usage, costs, test results, events, and lineage; organization customization, SCIM mapping, team-roles config, notification channels, connectors, and integrations are not supported yet. See the [Sync Environments guide](https://docs.entropy-data.com/howto/import-export) for a walkthrough.
 - Fix `apply`/`import` silently dropping `sourcesystems/` from an organization export — source systems are now applied in their correct dependency position (after policies, before assets).
 - `export`/`apply`/`sync` now also copy `certifications`, `classification-schemes`, `example-data`, and the `semantics` graph. All writes are idempotent PUT-by-id, team members are stripped on apply, asset tag assignments are replayed, and audit fields are stripped before PUT.
-- The `semantics` graph is copied as one OSI-compliant ontology YAML document per namespace (`semantic-ontology/<namespace>.yaml`) via the app's `GET`/`PUT /api/semantics/experimental/namespaces/{ns}/ontology.yaml` endpoint, which imports it in the correct internal order (groups before members, concepts before relationships). Requires that endpoint on the target instance; the namespace row itself is copied first as a normal resource.
+- The `semantics` graph is copied as one OSI-compliant ontology YAML document per namespace (`semantic-ontology/<namespace>.yaml`) via the app's `GET`/`PUT /api/semantics/experimental/namespaces/{ns}/ontology.yaml` endpoint, which imports it in the correct internal order (groups before members, concepts before relationships) and provisions the namespace from the metadata (`display_name`, `read_only`, `owner`) carried in the document's root `custom_properties`. There is no separate namespace artifact; a tree from an older version that still carries a `semantic-namespaces/` folder is applied as before. Requires that endpoint on the target instance.
 - `export`/`apply`/`sync` copy the organization's feature configuration as a singleton (`GET`/`PUT /api/organization/features`, artifact `organization-features/organization-features.yaml`), applied last so a restrictive policy it carries cannot reject earlier imports. Requires the app-side endpoint (entropy-data#1521); singletons are never pruned.
 
 ## [0.3.19]

README.md

@@ -196,8 +196,11 @@ Notes:
   keeps them so the artifact is a faithful snapshot.
 - The semantics graph is copied as one OSI ontology YAML document per namespace
   (`semantic-ontology/<namespace>.yaml`) via the app's `.../{ns}/ontology.yaml` endpoint,
-  which imports it in the correct internal dependency order. The namespace row is copied
-  first as a normal resource. Requires that endpoint on the target instance.
+  which imports it in the correct internal dependency order and provisions the namespace
+  from the metadata (display name, read-only flag, owning team) carried in the document's root
+  `custom_properties` — so there is no separate namespace artifact. Requires that endpoint
+  on the target instance. A tree exported by an older version that still carries a
+  `semantic-namespaces/` folder is applied as before.
 - The organization feature configuration is an org-level singleton
   (`organization-features/organization-features.yaml`), applied last and never pruned. It
   requires the app's `GET`/`PUT /api/organization/features` endpoint (entropy-data#1521),

src/entropy_data/resources.py

@@ -24,7 +24,12 @@
   listing the parent. Artifact: ``<name>/<parent_id>.yaml``. Not prunable. Used
   for the semantic ontology: the app imports the whole namespace document in the
   correct internal order (groups before members), so the client needs no
-  per-concept ordering of its own.
+  per-concept ordering of its own. Applying it also provisions the namespace from
+  the metadata in the document's root ``custom_properties``, so ``semantic-namespaces``
+  writes no artifact of its own (``artifact=False``): it is enumerated to drive the
+  ontology export and still pruned, but its identity is the ontology filenames. A
+  legacy tree that still carries a ``semantic-namespaces/`` directory is applied as a
+  flat resource, unchanged.
 * Singleton (``singleton`` set) — one org-level object read/written directly at
   ``/api/{api_path}`` (no id, no list). Artifact: ``<name>/<name>.yaml``. Not
   prunable. ``organization-features`` is applied last so a restrictive policy it
@@ -61,6 +66,12 @@ class Resource:
                 prunable) rather than a collection
     document    one raw-YAML document per parent, GET/PUT directly at the expanded
                 ``{parent}`` path (not prunable); requires ``parent``
+    artifact    whether this resource is written to / read from its own directory in
+                the artifact tree. ``False`` for a parent whose metadata now travels
+                inside its document child (``semantic-namespaces``): it is still listed
+                so the child can be enumerated and still pruned, but its identity comes
+                from the child's filenames rather than a directory of its own. A legacy
+                tree that still carries the directory is applied as before.
     """
 
     name: str
@@ -74,6 +85,7 @@ class Resource:
     parent: str | None = None
     singleton: bool = False
     document: bool = False
+    artifact: bool = True
 
     def path_for(self, parent_id: str | None = None) -> str:
         """Resolve ``api_path`` for a given parent id (identity for flat resources)."""
@@ -99,10 +111,12 @@ def path_for(self, parent_id: str | None = None) -> str:
     Resource("dataproducts", "dataproducts", detail=True),
     Resource("example-data", "example-data"),
     Resource("access", "access", paginated=True),
-    # Semantics: the namespace row carries the metadata; the whole ontology (concepts +
-    # relationships) is copied as one OSI YAML document per namespace via the app's
-    # ontology.yaml endpoint, which imports it in the correct internal dependency order.
-    Resource("semantic-namespaces", "semantics/experimental/namespaces", id_field="namespace"),
+    # Semantics: one artifact per namespace. The whole ontology (concepts + relationships) is copied
+    # as one OSI YAML document per namespace via the app's ontology.yaml endpoint, which imports it in
+    # the correct internal dependency order and provisions the namespace from the metadata carried in
+    # the document's root custom_properties. The namespace is still listed (to enumerate ontologies)
+    # and still pruned, but writes no artifact of its own — its identity is the ontology filenames.
+    Resource("semantic-namespaces", "semantics/experimental/namespaces", id_field="namespace", artifact=False),
     Resource(
         "semantic-ontology",
         "semantics/experimental/namespaces/{parent}/ontology.yaml",

src/entropy_data/sync.py

@@ -84,6 +84,23 @@ def _parent_ids(client: EntropyDataClient, resource: Resource) -> list[str]:
     return [str(item[parent.id_field]) for item in _list_all(client, parent) if item.get(parent.id_field)]
 
 
+def _child_document_ids(source: Path, resources: list[Resource], parent_name: str) -> set[str]:
+    """Identity of a parent whose own artifact was dropped: the stems of its document child's files.
+
+    ``semantic-namespaces`` writes no directory, so the set of namespaces an import carries is the set
+    of ``semantic-ontology/<ns>.yaml`` filenames — the same stems ``apply`` already PUTs as parent ids.
+    Used as the prune keep-set so pruning removes exactly the namespaces the import does not carry,
+    never all of them for lack of a directory to read.
+    """
+    child = next((r for r in resources if r.parent == parent_name and r.document), None)
+    if child is None:
+        return set()
+    child_dir = source / child.name
+    if not child_dir.is_dir():
+        return set()
+    return {f.stem for f in child_dir.glob("*.yaml")}
+
+
 # --- singleton (org-level object, no id) ---------------------------------------
 
 
@@ -265,6 +282,9 @@ def export_dir(client: EntropyDataClient, dest: Path, resources: list[Resource])
     """Enumerate the source and write one YAML file per resource under ``dest``."""
     total = SyncResult()
     for resource in resources:
+        # Enumerated (so its document child can be listed) but materialized only through that child.
+        if not resource.artifact:
+            continue
         console.print(f"\n[bold]{resource.name}[/bold]")
         if resource.singleton:
             total.add(_export_singleton(client, resource, dest / resource.name))
@@ -456,8 +476,12 @@ def plan_apply(
             n = len(list(resource_dir.glob("*.yaml"))) if resource_dir.is_dir() else 0
             counts = PlanCounts(update=n)
         else:
-            entries = _load_dir(resource_dir, resource) if resource_dir.is_dir() else []
-            imported = {rid for rid, _ in entries}
+            if not resource.artifact and not resource_dir.is_dir():
+                # Identity comes from the document child's files, not a directory of this resource's own.
+                imported = _child_document_ids(source, resources, resource.name)
+            else:
+                entries = _load_dir(resource_dir, resource) if resource_dir.is_dir() else []
+                imported = {rid for rid, _ in entries}
             try:
                 existing = _enumerate_ids(client, resource)
             except ApiError as e:
@@ -487,6 +511,14 @@ def apply_dir(
 
     for resource in resources:
         resource_dir = source / resource.name
+
+        # Artifact dropped and no legacy directory to fall back on: the document child provisions the
+        # resource, so issue no writes here — but still record its ids so a --prune keeps exactly the
+        # namespaces the import carries rather than deleting all of them.
+        if not resource.artifact and not resource_dir.is_dir():
+            imported_ids[resource.name] = _child_document_ids(source, resources, resource.name)
+            continue
+
         if not resource_dir.is_dir():
             continue
 

tests/commands/test_import_export.py

@@ -475,8 +475,9 @@ def test_export_semantics_ontology_document(monkeypatch, tmp_path):
     result = runner.invoke(app, ["export", "dir", str(dest), "--include", "semantic-namespaces,semantic-ontology"])
     assert result.exit_code == 0, result.output
 
-    assert yaml.safe_load((dest / "semantic-namespaces" / "core.yaml").read_text())["namespace"] == "core"
-    # The whole ontology is one document per namespace, written verbatim (no re-serialization).
+    # One artifact per namespace: the ontology document carries the namespace metadata, so no separate
+    # semantic-namespaces file is written. The document is verbatim (no re-serialization).
+    assert not (dest / "semantic-namespaces").exists()
     assert (dest / "semantic-ontology" / "core.yaml").read_text() == ontology
 
 
@@ -487,8 +488,6 @@ def test_import_semantics_ontology_document(monkeypatch, tmp_path):
 
     ontology = "version: 0.2.0.dev0\nname: core\nontology: []\n"
     src = tmp_path / "tree"
-    (src / "semantic-namespaces").mkdir(parents=True)
-    (src / "semantic-namespaces" / "core.yaml").write_text(yaml.safe_dump({"namespace": "core", "label": "Core"}))
     (src / "semantic-ontology").mkdir(parents=True)
     (src / "semantic-ontology" / "core.yaml").write_text(ontology)
 
@@ -500,7 +499,7 @@ def _record_ontology(request):
         captured["content_type"] = request.headers.get("Content-Type")
         return (200, {}, "")
 
-    responses.add(responses.PUT, f"{NS}/core", status=200)
+    # No PUT to the namespace endpoint: applying the ontology provisions the namespace.
     responses.add_callback(responses.PUT, f"{NS}/core/ontology.yaml", callback=_record_ontology)
 
     result = runner.invoke(app, ["apply", "dir", str(src), "--include", "semantic-namespaces,semantic-ontology"])
@@ -510,6 +509,56 @@ def _record_ontology(request):
     assert captured["content_type"] == "application/yaml"
 
 
+@responses.activate
+def test_apply_semantics_legacy_namespace_dir_still_applied(monkeypatch, tmp_path):
+    """A tree exported before the single-artifact change still carries semantic-namespaces/; apply it."""
+    monkeypatch.setattr(cfg, "CONFIG_FILE", tmp_path / "config.toml")
+    monkeypatch.setenv("ENTROPY_DATA_API_KEY", "test-key")
+
+    ontology = "version: 0.2.0.dev0\nname: core\nontology: []\n"
+    src = tmp_path / "tree"
+    (src / "semantic-namespaces").mkdir(parents=True)
+    (src / "semantic-namespaces" / "core.yaml").write_text(yaml.safe_dump({"namespace": "core", "label": "Core"}))
+    (src / "semantic-ontology").mkdir(parents=True)
+    (src / "semantic-ontology" / "core.yaml").write_text(ontology)
+
+    puts = []
+    responses.add_callback(responses.PUT, f"{NS}/core", callback=lambda r: (puts.append(r.url) or (200, {}, "")))
+    responses.add(responses.PUT, f"{NS}/core/ontology.yaml", status=200)
+
+    result = runner.invoke(app, ["apply", "dir", str(src), "--include", "semantic-namespaces,semantic-ontology"])
+    assert result.exit_code == 0, result.output
+    # The legacy namespace row is still PUT, preserving fidelity for pre-existing artifact trees.
+    assert puts == [f"{NS}/core"]
+
+
+@responses.activate
+def test_apply_semantics_prune_keeps_namespaces_from_ontology_filenames(monkeypatch, tmp_path):
+    """With no semantic-namespaces/ dir, the prune keep-set must come from the ontology filenames —
+    otherwise every namespace on the target would be deleted for lack of a directory to read."""
+    monkeypatch.setattr(cfg, "CONFIG_FILE", tmp_path / "config.toml")
+    monkeypatch.setenv("ENTROPY_DATA_API_KEY", "test-key")
+
+    src = tmp_path / "tree"
+    (src / "semantic-ontology").mkdir(parents=True)
+    (src / "semantic-ontology" / "core.yaml").write_text("version: 0.2.0.dev0\nname: core\nontology: []\n")
+
+    responses.add(responses.PUT, f"{NS}/core/ontology.yaml", status=200)
+    # Target has core (kept, present in the import) and legacy (absent -> pruned).
+    responses.add(responses.GET, NS, json=[{"namespace": "core"}, {"namespace": "legacy"}], status=200)
+    deletes = []
+    responses.add_callback(
+        responses.DELETE, f"{NS}/legacy", callback=lambda r: (deletes.append(r.url) or (200, {}, ""))
+    )
+
+    result = runner.invoke(
+        app,
+        ["apply", "dir", str(src), "--include", "semantic-namespaces,semantic-ontology", "--prune", "--yes"],
+    )
+    assert result.exit_code == 0, result.output
+    assert deletes == [f"{NS}/legacy"]
+
+
 # --- organization features (singleton) -----------------------------------------
 
 FEATURES = f"{BASE_URL}/api/organization/features"