diff --git a/CHANGELOG.md b/CHANGELOG.md index f36c492..4590347 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ - Add `entropy-data sync --source --target --include ` to copy portable organization state from one connection to another by exporting the source into a staged directory and importing it into the target. `sync` copies **nothing by default** — the resources to copy must be named with `--include`, and a run without it fails with a non-zero exit and the list of known resources. Supports `--exclude`, `--prune`, `--dry-run` (per-resource create/update/prune counts), and `--keep ` to retain the staged export. Supported resources: `teams`, `tags`, `definitions`, `policies`, `sourcesystems`, `certifications`, `classification-schemes`, `assets`, `datacontracts`, `dataproducts`, `example-data`, `access`, `semantic-namespaces`, `semantic-ontology`, `organization-features`. Not synced: users & team members, API keys, git credentials, integration/connector credentials, usage, costs, test results, events, and lineage; organization customization, SCIM mapping, team-roles config, notification channels, connectors, and integrations are not supported yet. See the [Sync Environments guide](https://docs.entropy-data.com/howto/import-export) for a walkthrough. - Fix `apply`/`import` silently dropping `sourcesystems/` from an organization export — source systems are now applied in their correct dependency position (after policies, before assets). - `export`/`apply`/`sync` now also copy `certifications`, `classification-schemes`, `example-data`, and the `semantics` graph. All writes are idempotent PUT-by-id, team members are stripped on apply, asset tag assignments are replayed, and audit fields are stripped before PUT. -- The `semantics` graph is copied as one OSI-compliant ontology YAML document per namespace (`semantic-ontology/.yaml`) via the app's `GET`/`PUT /api/semantics/experimental/namespaces/{ns}/ontology.yaml` endpoint, which imports it in the correct internal order (groups before members, concepts before relationships). Requires that endpoint on the target instance; the namespace row itself is copied first as a normal resource. +- The `semantics` graph is copied as one OSI-compliant ontology YAML document per namespace (`semantic-ontology/.yaml`) via the app's `GET`/`PUT /api/semantics/experimental/namespaces/{ns}/ontology.yaml` endpoint, which imports it in the correct internal order (groups before members, concepts before relationships) and provisions the namespace from the metadata (`display_name`, `read_only`, `owner`) carried in the document's root `custom_properties`. There is no separate namespace artifact; a tree from an older version that still carries a `semantic-namespaces/` folder is applied as before. Requires that endpoint on the target instance. - `export`/`apply`/`sync` copy the organization's feature configuration as a singleton (`GET`/`PUT /api/organization/features`, artifact `organization-features/organization-features.yaml`), applied last so a restrictive policy it carries cannot reject earlier imports. Requires the app-side endpoint (entropy-data#1521); singletons are never pruned. ## [0.3.19] diff --git a/README.md b/README.md index 23168e8..3028e72 100644 --- a/README.md +++ b/README.md @@ -196,8 +196,11 @@ Notes: keeps them so the artifact is a faithful snapshot. - The semantics graph is copied as one OSI ontology YAML document per namespace (`semantic-ontology/.yaml`) via the app's `.../{ns}/ontology.yaml` endpoint, - which imports it in the correct internal dependency order. The namespace row is copied - first as a normal resource. Requires that endpoint on the target instance. + which imports it in the correct internal dependency order and provisions the namespace + from the metadata (display name, read-only flag, owning team) carried in the document's root + `custom_properties` — so there is no separate namespace artifact. Requires that endpoint + on the target instance. A tree exported by an older version that still carries a + `semantic-namespaces/` folder is applied as before. - The organization feature configuration is an org-level singleton (`organization-features/organization-features.yaml`), applied last and never pruned. It requires the app's `GET`/`PUT /api/organization/features` endpoint (entropy-data#1521), diff --git a/src/entropy_data/resources.py b/src/entropy_data/resources.py index 3df0286..b2088d9 100644 --- a/src/entropy_data/resources.py +++ b/src/entropy_data/resources.py @@ -24,7 +24,12 @@ listing the parent. Artifact: ``/.yaml``. Not prunable. Used for the semantic ontology: the app imports the whole namespace document in the correct internal order (groups before members), so the client needs no - per-concept ordering of its own. + per-concept ordering of its own. Applying it also provisions the namespace from + the metadata in the document's root ``custom_properties``, so ``semantic-namespaces`` + writes no artifact of its own (``artifact=False``): it is enumerated to drive the + ontology export and still pruned, but its identity is the ontology filenames. A + legacy tree that still carries a ``semantic-namespaces/`` directory is applied as a + flat resource, unchanged. * Singleton (``singleton`` set) — one org-level object read/written directly at ``/api/{api_path}`` (no id, no list). Artifact: ``/.yaml``. Not prunable. ``organization-features`` is applied last so a restrictive policy it @@ -61,6 +66,12 @@ class Resource: prunable) rather than a collection document one raw-YAML document per parent, GET/PUT directly at the expanded ``{parent}`` path (not prunable); requires ``parent`` + artifact whether this resource is written to / read from its own directory in + the artifact tree. ``False`` for a parent whose metadata now travels + inside its document child (``semantic-namespaces``): it is still listed + so the child can be enumerated and still pruned, but its identity comes + from the child's filenames rather than a directory of its own. A legacy + tree that still carries the directory is applied as before. """ name: str @@ -74,6 +85,7 @@ class Resource: parent: str | None = None singleton: bool = False document: bool = False + artifact: bool = True def path_for(self, parent_id: str | None = None) -> str: """Resolve ``api_path`` for a given parent id (identity for flat resources).""" @@ -99,10 +111,12 @@ def path_for(self, parent_id: str | None = None) -> str: Resource("dataproducts", "dataproducts", detail=True), Resource("example-data", "example-data"), Resource("access", "access", paginated=True), - # Semantics: the namespace row carries the metadata; the whole ontology (concepts + - # relationships) is copied as one OSI YAML document per namespace via the app's - # ontology.yaml endpoint, which imports it in the correct internal dependency order. - Resource("semantic-namespaces", "semantics/experimental/namespaces", id_field="namespace"), + # Semantics: one artifact per namespace. The whole ontology (concepts + relationships) is copied + # as one OSI YAML document per namespace via the app's ontology.yaml endpoint, which imports it in + # the correct internal dependency order and provisions the namespace from the metadata carried in + # the document's root custom_properties. The namespace is still listed (to enumerate ontologies) + # and still pruned, but writes no artifact of its own — its identity is the ontology filenames. + Resource("semantic-namespaces", "semantics/experimental/namespaces", id_field="namespace", artifact=False), Resource( "semantic-ontology", "semantics/experimental/namespaces/{parent}/ontology.yaml", diff --git a/src/entropy_data/sync.py b/src/entropy_data/sync.py index a3a9ccf..1aab686 100644 --- a/src/entropy_data/sync.py +++ b/src/entropy_data/sync.py @@ -84,6 +84,23 @@ def _parent_ids(client: EntropyDataClient, resource: Resource) -> list[str]: return [str(item[parent.id_field]) for item in _list_all(client, parent) if item.get(parent.id_field)] +def _child_document_ids(source: Path, resources: list[Resource], parent_name: str) -> set[str]: + """Identity of a parent whose own artifact was dropped: the stems of its document child's files. + + ``semantic-namespaces`` writes no directory, so the set of namespaces an import carries is the set + of ``semantic-ontology/.yaml`` filenames — the same stems ``apply`` already PUTs as parent ids. + Used as the prune keep-set so pruning removes exactly the namespaces the import does not carry, + never all of them for lack of a directory to read. + """ + child = next((r for r in resources if r.parent == parent_name and r.document), None) + if child is None: + return set() + child_dir = source / child.name + if not child_dir.is_dir(): + return set() + return {f.stem for f in child_dir.glob("*.yaml")} + + # --- singleton (org-level object, no id) --------------------------------------- @@ -265,6 +282,9 @@ def export_dir(client: EntropyDataClient, dest: Path, resources: list[Resource]) """Enumerate the source and write one YAML file per resource under ``dest``.""" total = SyncResult() for resource in resources: + # Enumerated (so its document child can be listed) but materialized only through that child. + if not resource.artifact: + continue console.print(f"\n[bold]{resource.name}[/bold]") if resource.singleton: total.add(_export_singleton(client, resource, dest / resource.name)) @@ -456,8 +476,12 @@ def plan_apply( n = len(list(resource_dir.glob("*.yaml"))) if resource_dir.is_dir() else 0 counts = PlanCounts(update=n) else: - entries = _load_dir(resource_dir, resource) if resource_dir.is_dir() else [] - imported = {rid for rid, _ in entries} + if not resource.artifact and not resource_dir.is_dir(): + # Identity comes from the document child's files, not a directory of this resource's own. + imported = _child_document_ids(source, resources, resource.name) + else: + entries = _load_dir(resource_dir, resource) if resource_dir.is_dir() else [] + imported = {rid for rid, _ in entries} try: existing = _enumerate_ids(client, resource) except ApiError as e: @@ -487,6 +511,14 @@ def apply_dir( for resource in resources: resource_dir = source / resource.name + + # Artifact dropped and no legacy directory to fall back on: the document child provisions the + # resource, so issue no writes here — but still record its ids so a --prune keeps exactly the + # namespaces the import carries rather than deleting all of them. + if not resource.artifact and not resource_dir.is_dir(): + imported_ids[resource.name] = _child_document_ids(source, resources, resource.name) + continue + if not resource_dir.is_dir(): continue diff --git a/tests/commands/test_import_export.py b/tests/commands/test_import_export.py index f62d86a..535b0e0 100644 --- a/tests/commands/test_import_export.py +++ b/tests/commands/test_import_export.py @@ -475,8 +475,9 @@ def test_export_semantics_ontology_document(monkeypatch, tmp_path): result = runner.invoke(app, ["export", "dir", str(dest), "--include", "semantic-namespaces,semantic-ontology"]) assert result.exit_code == 0, result.output - assert yaml.safe_load((dest / "semantic-namespaces" / "core.yaml").read_text())["namespace"] == "core" - # The whole ontology is one document per namespace, written verbatim (no re-serialization). + # One artifact per namespace: the ontology document carries the namespace metadata, so no separate + # semantic-namespaces file is written. The document is verbatim (no re-serialization). + assert not (dest / "semantic-namespaces").exists() assert (dest / "semantic-ontology" / "core.yaml").read_text() == ontology @@ -487,8 +488,6 @@ def test_import_semantics_ontology_document(monkeypatch, tmp_path): ontology = "version: 0.2.0.dev0\nname: core\nontology: []\n" src = tmp_path / "tree" - (src / "semantic-namespaces").mkdir(parents=True) - (src / "semantic-namespaces" / "core.yaml").write_text(yaml.safe_dump({"namespace": "core", "label": "Core"})) (src / "semantic-ontology").mkdir(parents=True) (src / "semantic-ontology" / "core.yaml").write_text(ontology) @@ -500,7 +499,7 @@ def _record_ontology(request): captured["content_type"] = request.headers.get("Content-Type") return (200, {}, "") - responses.add(responses.PUT, f"{NS}/core", status=200) + # No PUT to the namespace endpoint: applying the ontology provisions the namespace. responses.add_callback(responses.PUT, f"{NS}/core/ontology.yaml", callback=_record_ontology) result = runner.invoke(app, ["apply", "dir", str(src), "--include", "semantic-namespaces,semantic-ontology"]) @@ -510,6 +509,56 @@ def _record_ontology(request): assert captured["content_type"] == "application/yaml" +@responses.activate +def test_apply_semantics_legacy_namespace_dir_still_applied(monkeypatch, tmp_path): + """A tree exported before the single-artifact change still carries semantic-namespaces/; apply it.""" + monkeypatch.setattr(cfg, "CONFIG_FILE", tmp_path / "config.toml") + monkeypatch.setenv("ENTROPY_DATA_API_KEY", "test-key") + + ontology = "version: 0.2.0.dev0\nname: core\nontology: []\n" + src = tmp_path / "tree" + (src / "semantic-namespaces").mkdir(parents=True) + (src / "semantic-namespaces" / "core.yaml").write_text(yaml.safe_dump({"namespace": "core", "label": "Core"})) + (src / "semantic-ontology").mkdir(parents=True) + (src / "semantic-ontology" / "core.yaml").write_text(ontology) + + puts = [] + responses.add_callback(responses.PUT, f"{NS}/core", callback=lambda r: (puts.append(r.url) or (200, {}, ""))) + responses.add(responses.PUT, f"{NS}/core/ontology.yaml", status=200) + + result = runner.invoke(app, ["apply", "dir", str(src), "--include", "semantic-namespaces,semantic-ontology"]) + assert result.exit_code == 0, result.output + # The legacy namespace row is still PUT, preserving fidelity for pre-existing artifact trees. + assert puts == [f"{NS}/core"] + + +@responses.activate +def test_apply_semantics_prune_keeps_namespaces_from_ontology_filenames(monkeypatch, tmp_path): + """With no semantic-namespaces/ dir, the prune keep-set must come from the ontology filenames — + otherwise every namespace on the target would be deleted for lack of a directory to read.""" + monkeypatch.setattr(cfg, "CONFIG_FILE", tmp_path / "config.toml") + monkeypatch.setenv("ENTROPY_DATA_API_KEY", "test-key") + + src = tmp_path / "tree" + (src / "semantic-ontology").mkdir(parents=True) + (src / "semantic-ontology" / "core.yaml").write_text("version: 0.2.0.dev0\nname: core\nontology: []\n") + + responses.add(responses.PUT, f"{NS}/core/ontology.yaml", status=200) + # Target has core (kept, present in the import) and legacy (absent -> pruned). + responses.add(responses.GET, NS, json=[{"namespace": "core"}, {"namespace": "legacy"}], status=200) + deletes = [] + responses.add_callback( + responses.DELETE, f"{NS}/legacy", callback=lambda r: (deletes.append(r.url) or (200, {}, "")) + ) + + result = runner.invoke( + app, + ["apply", "dir", str(src), "--include", "semantic-namespaces,semantic-ontology", "--prune", "--yes"], + ) + assert result.exit_code == 0, result.output + assert deletes == [f"{NS}/legacy"] + + # --- organization features (singleton) ----------------------------------------- FEATURES = f"{BASE_URL}/api/organization/features"