Add sync/export/apply-dir commands to copy portable org state between instances

CHANGELOG.md

@@ -2,6 +2,14 @@
 
 ## [Unreleased]
 
+- Add `entropy-data export dir <path>` to enumerate an organization's portable state into a local YAML tree (one directory per resource, one file per resource named by its id), mirroring the app's organization-export layout so the artifacts interchange.
+- Add `entropy-data apply dir <path>` to reconcile a local export directory tree into the connected instance, in the spirit of `kubectl apply -f <dir>`. The tree uses a folder-as-kind convention: the directory name is the resource kind and each YAML file below it is one resource (addressed by the id in its body); folders that are not a known kind are ignored. By default the whole tree is applied; `--include`/`--exclude` narrow it, `--dry-run` previews, and `--prune` also deletes instance resources absent from the tree (reverse dependency order, confirmation-gated unless `--yes`). `import zip` (unchanged) now extracts to a temp directory and applies it through the same engine.
+- Add `entropy-data sync --source <conn> --target <conn> --include <resources>` to copy portable organization state from one connection to another by exporting the source into a staged directory and importing it into the target. `sync` copies **nothing by default** — the resources to copy must be named with `--include`, and a run without it fails with a non-zero exit and the list of known resources. Supports `--exclude`, `--prune`, `--dry-run` (per-resource create/update/prune counts), and `--keep <dir>` to retain the staged export. Supported resources: `teams`, `tags`, `definitions`, `policies`, `sourcesystems`, `certifications`, `classification-schemes`, `assets`, `datacontracts`, `dataproducts`, `example-data`, `access`, `semantic-namespaces`, `semantic-ontology`, `organization-features`. Not synced: users & team members, API keys, git credentials, integration/connector credentials, usage, costs, test results, events, and lineage; organization customization, SCIM mapping, team-roles config, notification channels, connectors, and integrations are not supported yet. See the [Sync Environments guide](https://docs.entropy-data.com/howto/import-export) for a walkthrough.
+- Fix `apply`/`import` silently dropping `sourcesystems/` from an organization export — source systems are now applied in their correct dependency position (after policies, before assets).
+- `export`/`apply`/`sync` now also copy `certifications`, `classification-schemes`, `example-data`, and the `semantics` graph. All writes are idempotent PUT-by-id, team members are stripped on apply, asset tag assignments are replayed, and audit fields are stripped before PUT.
+- The `semantics` graph is copied as one OSI-compliant ontology YAML document per namespace (`semantic-ontology/<namespace>.yaml`) via the app's `GET`/`PUT /api/semantics/experimental/namespaces/{ns}/ontology.yaml` endpoint, which imports it in the correct internal order (groups before members, concepts before relationships). Requires that endpoint on the target instance; the namespace row itself is copied first as a normal resource.
+- `export`/`apply`/`sync` copy the organization's feature configuration as a singleton (`GET`/`PUT /api/organization/features`, artifact `organization-features/organization-features.yaml`), applied last so a restrictive policy it carries cannot reject earlier imports. Requires the app-side endpoint (entropy-data#1521); singletons are never pruned.
+
 ## [0.3.19]
 
 - Fix `entropy-data datacontracts yaml <id>` requesting `/api/datacontracts/{id}.yaml`, which does not exist on the server, resulting in a 404. It now requests the documented `/api/datacontracts/{id}/datacontract.yaml`.

README.md

@@ -111,9 +111,98 @@ entropy-data [--version] [--connection NAME] [--output table|json|yaml] [--debug
   search          query
   semantics       namespaces ... | concepts ... | relationships ... | search
   usage           list | submit | delete
+  export          dir
+  apply           dir [--include] [--exclude] [--prune] [--dry-run]
   import          zip
+  sync            --source SRC --target TGT --include a,b [--exclude] [--prune] [--dry-run] [--keep DIR]
 ```
 
+## Syncing organization state between instances
+
+`sync` copies the portable declarative state of an organization from one Entropy Data
+instance to another — for example to promote a test environment to prod. It exports the
+source and applies it to the target in one step (`export dir` + `apply dir` do the same in
+two). Only state reachable through the public `/api/**` API and portable across instances
+is copied (no secrets, telemetry, or environment-specific identity). Every write is an
+idempotent PUT-by-id, so runs converge and are safe to repeat.
+
+`sync` copies **nothing by default** — name the resources to sync with `--include`.
+
+**Supported resources** (in dependency order): `teams`, `tags`, `definitions`, `policies`,
+`sourcesystems`, `certifications`, `classification-schemes`, `assets`, `datacontracts`,
+`dataproducts`, `example-data`, `access`, `semantic-namespaces`, `semantic-ontology`,
+`organization-features`.
+
+**Not synced:** users & team members, API keys, git credentials, integration and connector
+credentials, usage, costs, test results, events, and lineage (per-instance identity,
+secrets, or telemetry). Organization customization, SCIM mapping, team-roles configuration,
+notification channels, connectors, and integrations are **not supported yet**.
+
+```bash
+# Preview a test -> prod sync of selected resources without writing anything.
+entropy-data sync --source test --target prod --include teams,policies,datacontracts,dataproducts --dry-run
+
+# Sync those resources.
+entropy-data sync --source test --target prod --include teams,policies,datacontracts,dataproducts
+
+# Mirror: also delete target resources that are absent from the source.
+entropy-data sync --source test --target prod --include datacontracts --prune
+
+# The two-step equivalent, with a reviewable YAML tree in between.
+entropy-data -c test export dir ./state
+entropy-data -c prod apply dir ./state
+```
+
+### `apply dir` — apply a local export tree
+
+`apply dir <path>` reconciles a local export directory into the connected instance, in the
+spirit of `kubectl apply -f <dir>`. The tree follows a **folder-as-kind** convention: the
+directory name is the resource kind and each YAML file below it is one resource, addressed
+by the id in its body.
+
+```
+state/
+  teams/                          # folder name = resource kind
+    marketing.yaml                # one file per resource (filename is cosmetic)
+  policies/
+    pii-policy.yaml
+  datacontracts/
+    orders-1-orders.yaml
+  semantic-ontology/
+    main.yaml                     # document resources: one YAML doc per namespace
+  organization-features/
+    organization-features.yaml    # singleton: <name>/<name>.yaml
+```
+
+Unlike `kubectl`, the files carry no `kind:` field — the enclosing folder is authoritative,
+so a file only means what its folder says. Folders that are not a known resource kind are
+ignored. Unlike `sync`, `apply dir` applies the **whole tree by default** (the directory is
+your selection); use `--include`/`--exclude` to narrow it, `--dry-run` to preview, and
+`--prune` to also delete instance resources absent from the tree. This layout matches the
+app's own organization export, so an `export dir` tree and an in-app export zip interchange.
+
+Useful options:
+
+- `--include a,b` — **required**; the resources to sync (names from the supported list above).
+- `--exclude a,b` — drop resources from the `--include` set.
+- `--prune` — after upserts, delete target resources absent from the source, in reverse
+  dependency order. Guarded by a confirmation prompt unless `--yes` is passed.
+- `--dry-run` — print per-resource create/update/(prune) counts; no writes.
+- `--keep DIR` — retain the staged export instead of using a temporary directory.
+
+Notes:
+
+- Team members are stripped on import (users are per-instance identities); the export
+  keeps them so the artifact is a faithful snapshot.
+- The semantics graph is copied as one OSI ontology YAML document per namespace
+  (`semantic-ontology/<namespace>.yaml`) via the app's `.../{ns}/ontology.yaml` endpoint,
+  which imports it in the correct internal dependency order. The namespace row is copied
+  first as a normal resource. Requires that endpoint on the target instance.
+- The organization feature configuration is an org-level singleton
+  (`organization-features/organization-features.yaml`), applied last and never pruned. It
+  requires the app's `GET`/`PUT /api/organization/features` endpoint (entropy-data#1521),
+  which must be merged and deployed to the target instance first.
+
 ## Connection Management
 
 Connections are stored in `~/.entropy-data/config.toml`:

src/entropy_data/cli.py

@@ -33,6 +33,16 @@ def get_client() -> EntropyDataClient:
     return EntropyDataClient(config)
 
 
+def client_for_connection(name: str) -> EntropyDataClient:
+    """Create an API client for a specific named connection (for `sync --source/--target`).
+
+    Resolves the connection by name only — the global --api-key/--host overrides
+    target the primary connection and must not bleed into a second endpoint.
+    """
+    config = resolve_connection(connection_name=name)
+    return EntropyDataClient(config)
+
+
 def get_output_format() -> OutputFormat:
     return _output_format
 
@@ -127,7 +137,7 @@ def main(
 from entropy_data.commands.definitions import definitions_app  # noqa: E402
 from entropy_data.commands.events import events_app  # noqa: E402
 from entropy_data.commands.example_data import example_data_app  # noqa: E402
-from entropy_data.commands.import_export import import_app  # noqa: E402
+from entropy_data.commands.import_export import apply_app, export_app, import_app  # noqa: E402
 from entropy_data.commands.integrations import integrations_app  # noqa: E402
 from entropy_data.commands.lineage import lineage_app  # noqa: E402
 from entropy_data.commands.organization import organization_app  # noqa: E402
@@ -137,6 +147,7 @@ def main(
 from entropy_data.commands.semantics import semantics_app  # noqa: E402
 from entropy_data.commands.settings import settings_app  # noqa: E402
 from entropy_data.commands.sourcesystems import sourcesystems_app  # noqa: E402
+from entropy_data.commands.sync import sync_command  # noqa: E402
 from entropy_data.commands.tags import tags_app  # noqa: E402
 from entropy_data.commands.teams import teams_app  # noqa: E402
 from entropy_data.commands.test_results import test_results_app  # noqa: E402
@@ -164,8 +175,31 @@ def main(
 app.add_typer(settings_app, name="settings", help="Manage organization settings.")
 app.add_typer(events_app, name="events", help="Poll events.")
 app.add_typer(lineage_app, name="lineage", help="Manage lineage (OpenLineage events).")
-app.add_typer(schemas_app, name="schemas", help="Get the JSON Schemas that data contracts (ODCS) and data products (ODPS) validate against.")
+app.add_typer(
+    schemas_app,
+    name="schemas",
+    help="Get the JSON Schemas that data contracts (ODCS) and data products (ODPS) validate against.",
+)
 app.add_typer(search_app, name="search", help="Search across resources.")
 app.add_typer(semantics_app, name="semantics", help="EXPERIMENTAL semantics API.")
 app.add_typer(usage_app, name="usage", help="Manage usage (OpenTelemetry traces).")
 app.add_typer(import_app, name="import", help="Import organization exports.")
+app.add_typer(export_app, name="export", help="Export organization state to a local YAML tree.")
+app.add_typer(
+    apply_app,
+    name="apply",
+    help="Apply a local export directory tree to an instance (folder name = resource kind).",
+)
+
+from entropy_data.resources import RESOURCE_ORDER as _RESOURCE_ORDER  # noqa: E402
+
+_SYNC_HELP = (
+    "Sync selected portable organization state from a source to a target connection.\n\n"
+    "Nothing is copied unless named with --include (sync copies nothing by default).\n\n"
+    "Supported resources: " + ", ".join(r.name for r in _RESOURCE_ORDER) + ".\n\n"
+    "Not synced — users & team members, API keys, git credentials, integration and connector "
+    "credentials, usage, costs, test results, events, and lineage (per-instance identity, secrets, "
+    "or telemetry); organization customization, SCIM mapping, team-roles config, notification "
+    "channels, connectors, and integrations are not supported yet."
+)
+app.command(name="sync", help=_SYNC_HELP)(sync_command)