[New Extension] Add chaos extension (Microsoft.Chaos 2026-05-01-preview) by kekivelez · Pull Request #9918 · Azure/azure-cli-extensions

kekivelez · 2026-06-05T02:06:29Z

Summary

New CLI extension for Azure Chaos Studio Microsoft.Chaos 2026-05-01-preview. Provides the az chaos command group for workspace lifecycle management, scenario browsing, scenario configuration with validation and execution, and run history with per-run cancellation.

Command surface

az chaos workspace {create, show, list, update, delete, refresh-recommendation, evaluate-scenarios, show-discovery, show-evaluation}
az chaos workspace identity {assign, remove, show}
az chaos scenario {create, show, list, update, delete}
az chaos scenario config {create, show, list, update, delete, validate, fix-permissions, execute, show-validation, show-permission-fix}
az chaos scenario run {start, show, list, cancel, wait}
az chaos discovered-resource {show, list}

Source-of-truth

The aaz/ subtree is generated via aaz-dev cli generate-by-swagger-tag from spec commit f228b86c in Azure/azure-rest-api-specs (swagger tag package-2026-05-01-preview).

This checklist is used to make sure that common guidelines for a pull request are followed.

Related command

az chaos

General Guidelines

Have you run azdev style <YOUR_EXT> locally? (pip install azdev required)
Have you run python scripts/ci/test_index.py -q locally? (pip install wheel==0.30.0 required) — N/A for the initial seed: this extension is not yet in src/index.json. The post-merge auto-PR will add the entry, after which this check applies to subsequent PRs.
My extension version conforms to the Extension version schema

For new extensions:

My extension description/summary conforms to the Extension Summary Guidelines.

About Extension Publish

There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update src/index.json automatically.
You only need to update the version information in file setup.py and historical information in file HISTORY.rst in your PR but do not modify src/index.json.

…view Initial codegen validation run — verify skill mechanics end-to-end against the latest Microsoft.Chaos spec; no upstream PR will be opened from this run. Generated via chaos-automation-codegen skill family (run-id 20260604-020139-f228b86). Spec pin: Azure/azure-rest-api-specs @ f228b86c API version: 2026-05-01-preview Swagger tag: package-2026-05-01-preview Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…g execute Mirrors the same shortening already applied to fix-permissions; resolves azdev-linter option_length_too_long HIGH finding (--scenario-configuration-name is 30 chars, threshold 22). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ions hook * custom.py: WorkspaceRefreshRecommendation(_RefreshRecommendation) overrides post_operations to call _check_inner_lro for discoveries/latest + evaluations/latest. Detects the silent-failure case the AAZ framework's final-state-via:location polling misses (status nested at properties.status, not root). Free function workspace_refresh_recommendations retired. _check_inner_lro + _build_arm_url refactored to accept cli_ctx directly (decoupled from cmd object). * WorkspaceEvaluateScenarios(WorkspaceRefreshRecommendation) registers the porcelain alias chaos workspace evaluate-scenarios at a name the spec doesn't define. Today behaves identically; will become a true composite of /discover + /evaluate when those ARM ops land in 2026-08-01-preview. * commands.py: register both subclasses via _register_aaz_subclass_overrides. g.custom_command for refresh-recommendations/evaluate-scenarios removed. * _help.py / _params.py: plural -> singular (refresh-recommendation), --name MyWorkspace examples retained (the new ARG_OPTIONS adds -n / --name aliases on the workspace_name arg, matching workspace create convention). * tests: integration test classes marked @live_only() (workspace lifecycle + discovered-resource browsing). 3 integration test files (scenario, scenario_config, scenario_run) pruned -- they exercised service behavior rather than extension behavior. test_custom_commands.py: TestRefreshRecommendationsInnerLRO rewritten to exercise _check_inner_lro directly (no free function to call anymore). test_command_registration.py: test_alias_registered_via_subclass replaces the old custom_command pattern assertion. Setup helpers fixed: sub kwarg injection, westus2 region, --mi-system-assigned identity flag (was --identity-type SystemAssigned), -w {ws} -> --workspace-name {ws}. * setup.py: version 0.1.4 -> 0.1.5 * HISTORY.rst: 0.1.5 release notes Generated aaz/ tree regenerated from spec commit f228b86c via the chaos-automation-codegen skill family (cli generate-by-swagger-tag). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…int strategy The 2 remaining @live_only() integration test classes (workspace lifecycle + discovered-resource browsing) added marginal value over the existing ~140 unit tests + azdev linter + spec-driven aaz-dev regen. They exercised service behavior that Microsoft.Chaos owns, not extension behavior. They were also flaky: ARG propagation lag after fresh Reader role assignment to the workspace MI triggers our (correct, production-desirable) inner-LRO diagnostic, making cassette recording non-deterministic without explicit mitigation in the test fixtures. Validation strategy is now: - Unit tests for extension code (custom.py, helpers, validators, formatters, AAZ subclass post_operations hooks) - azdev linter for help/param compliance - azdev style for code quality - Deterministic regen via chaos-automation-codegen skill - Developer smoke test before release - Live customer usage as canary Matches the validation approach of many other azure-cli-extensions packages that ship without ScenarioTest coverage. No recordings/ directory required. Folded the change description into the 0.1.5 HISTORY.rst entry (no version bump needed since no user-facing behavior changes; tests are not shipped to consumers). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The AAZ-generated WorkspacesRefreshRecommendations.__call__ passes None as the LRO success deserializer to build_lro_polling; the framework's base_polling._parse_resource later invokes that None from poll-result code paths, raising TypeError: 'NoneType' object is not callable. This is an aaz-dev codegen gap (final-state-via: location with a body-returning target should generate a real deserializer). Override _handler in WorkspaceRefreshRecommendation to provide a no-op deserializer (lambda _: None). post_operations (the inner-LRO diagnostic) is unaffected; it runs during _execute_operations, before .result() is called on the poller. WorkspaceEvaluateScenarios inherits the fix. Bump 0.1.5 -> 0.1.6. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Sibling fix to 0.1.6's refresh-recommendation fix. Same AAZ codegen gap: the AAZ-generated inner operation passes None as the LRO success deserializer; the framework's base_polling._parse_resource later invokes that None and raises TypeError: 'NoneType' object is not callable. New ScenarioConfigExecute subclass in custom.py overrides _handler to inject a no-op deserializer (lambda _: None). Registered in commands.py _register_aaz_subclass_overrides. Audit confirmed the remaining AAZ commands with the same pattern (fix- permissions, validate, cancel) are superseded by hand-written custom commands via g.custom_command and are not vulnerable. Bump 0.1.6 -> 0.1.7. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…idelines The 0.1.x version sequence (internal iteration during initial-seed development) was not compliant with the cli-extensions versioning guidelines, which require new extension preview versions to start at 1.0.0b1 (and new stable versions at 1.0.0). This extension is preview (azext.isPreview=True), never published, so the first public version must be 1.0.0b1. Reference: https://github.com/Azure/azure-cli/blob/release/doc/extensions/versioning_guidelines.md setup.py description rewritten to comply with the Extension Summary Guidelines (no 'Azure CLI' / 'command-line' / 'extension' language; specific and useful). Reference: https://github.com/Azure/azure-cli/blob/dev/doc/extensions/extension_summary_guidelines.md HISTORY.rst collapsed to a single 1.0.0b1 initial-preview-release entry summarizing the command surface and notable hand-written commands. Drops the 0.1.4-0.1.7 internal iteration entries (never published). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

azure-client-tools-bot-prd · 2026-06-05T02:06:32Z

Validation for Breaking Change Starting...

Thanks for your contribution!

azure-client-tools-bot-prd · 2026-06-05T02:06:35Z

Hi @kekivelez,
Please write the description of changes which can be perceived by customers into HISTORY.rst.
If you want to release a new extension version, please update the version in setup.py as well.

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds the initial chaos Azure CLI extension implementation (AAZ-generated command surface plus custom overrides), along with unit tests, packaging metadata, and documentation.

Changes:

Introduces the Chaos CLI extension loader, command registration, argument overrides, validators, and table output formatters.
Adds a hand-written chaos scenario run wait command and unit tests guarding command/help registration and “AAZ pristine” constraints.
Adds packaging files (setup.py, setup.cfg, extension metadata) and end-user docs (README.md, HISTORY.rst).

Reviewed changes

Copilot reviewed 67 out of 67 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/chaos/setup.py	Adds setuptools packaging entry point and metadata for the extension.
src/chaos/setup.cfg	Configures wheel build settings.
src/chaos/azext_chaos/tests/latest/test_validators.py	Unit tests for new validators (`--scopes`, `--parameters`).
src/chaos/azext_chaos/tests/latest/test_command_registration.py	Unit tests for help entries, command registration, and subclass override wiring.
src/chaos/azext_chaos/tests/latest/test_aaz_pristine.py	Enforces that AAZ-generated files remain unedited (pre/post hooks remain `pass`).
src/chaos/azext_chaos/tests/latest/init.py	Test package marker.
src/chaos/azext_chaos/tests/latest/README.md	Documents how to run tests and expected cmdcov coverage.
src/chaos/azext_chaos/tests/init.py	Test package marker.
src/chaos/azext_chaos/custom_wait.py	Adds custom AAZ-shaped `scenario run wait` command not produced by aaz-dev.
src/chaos/azext_chaos/commands.py	Registers custom commands and table transformers; installs AAZ subclass overrides/aliases.
src/chaos/azext_chaos/azext_metadata.json	Declares preview status and minimum CLI core version.
src/chaos/azext_chaos/aaz/latest/chaos/workspace/identity/_wait.py	AAZ-generated `chaos workspace identity wait` command.
src/chaos/azext_chaos/aaz/latest/chaos/workspace/identity/_show.py	AAZ-generated `chaos workspace identity show` command.
src/chaos/azext_chaos/aaz/latest/chaos/workspace/identity/_remove.py	AAZ-generated `chaos workspace identity remove` command.
src/chaos/azext_chaos/aaz/latest/chaos/workspace/identity/_assign.py	AAZ-generated `chaos workspace identity assign` command.
src/chaos/azext_chaos/aaz/latest/chaos/workspace/identity/init.py	Exports identity subgroup commands.
src/chaos/azext_chaos/aaz/latest/chaos/workspace/identity/__cmd_group.py	AAZ-generated identity command group.
src/chaos/azext_chaos/aaz/latest/chaos/workspace/_wait.py	AAZ-generated `chaos workspace wait` command.
src/chaos/azext_chaos/aaz/latest/chaos/workspace/_update.py	AAZ-generated `chaos workspace update` command.
src/chaos/azext_chaos/aaz/latest/chaos/workspace/_show.py	AAZ-generated `chaos workspace show` command.
src/chaos/azext_chaos/aaz/latest/chaos/workspace/_refresh_recommendation.py	AAZ-generated `chaos workspace refresh-recommendation` LRO command.
src/chaos/azext_chaos/aaz/latest/chaos/workspace/_list.py	AAZ-generated `chaos workspace list` command.
src/chaos/azext_chaos/aaz/latest/chaos/workspace/_delete.py	AAZ-generated `chaos workspace delete` command.
src/chaos/azext_chaos/aaz/latest/chaos/workspace/_create.py	AAZ-generated `chaos workspace create` command.
src/chaos/azext_chaos/aaz/latest/chaos/workspace/init.py	Exports workspace subgroup commands.
src/chaos/azext_chaos/aaz/latest/chaos/workspace/__cmd_group.py	AAZ-generated workspace command group.
src/chaos/azext_chaos/aaz/latest/chaos/scenario/run/_cancel.py	AAZ-generated `chaos scenario run cancel` command.
src/chaos/azext_chaos/aaz/latest/chaos/scenario/run/init.py	Exports scenario run subgroup commands.
src/chaos/azext_chaos/aaz/latest/chaos/scenario/run/__cmd_group.py	AAZ-generated run command group.
src/chaos/azext_chaos/aaz/latest/chaos/scenario/config/_wait.py	AAZ-generated `chaos scenario config wait` command.
src/chaos/azext_chaos/aaz/latest/chaos/scenario/config/_validate.py	AAZ-generated `chaos scenario config validate` command.
src/chaos/azext_chaos/aaz/latest/chaos/scenario/config/_show.py	AAZ-generated `chaos scenario config show` command.
src/chaos/azext_chaos/aaz/latest/chaos/scenario/config/_list.py	AAZ-generated `chaos scenario config list` command.
src/chaos/azext_chaos/aaz/latest/chaos/scenario/config/_fix_permissions.py	AAZ-generated `chaos scenario config fix-permissions` command.
src/chaos/azext_chaos/aaz/latest/chaos/scenario/config/_execute.py	AAZ-generated `chaos scenario config execute` command.
src/chaos/azext_chaos/aaz/latest/chaos/scenario/config/_delete.py	AAZ-generated `chaos scenario config delete` command.
src/chaos/azext_chaos/aaz/latest/chaos/scenario/config/init.py	Exports scenario config subgroup commands.
src/chaos/azext_chaos/aaz/latest/chaos/scenario/config/__cmd_group.py	AAZ-generated config command group.
src/chaos/azext_chaos/aaz/latest/chaos/scenario/_show.py	AAZ-generated `chaos scenario show` command.
src/chaos/azext_chaos/aaz/latest/chaos/scenario/_list.py	AAZ-generated `chaos scenario list` command.
src/chaos/azext_chaos/aaz/latest/chaos/scenario/_delete.py	AAZ-generated `chaos scenario delete` command.
src/chaos/azext_chaos/aaz/latest/chaos/scenario/init.py	Exports scenario subgroup commands.
src/chaos/azext_chaos/aaz/latest/chaos/scenario/__cmd_group.py	AAZ-generated scenario command group.
src/chaos/azext_chaos/aaz/latest/chaos/discovered_resource/_show.py	AAZ-generated `chaos discovered-resource show` command.
src/chaos/azext_chaos/aaz/latest/chaos/discovered_resource/_list.py	AAZ-generated `chaos discovered-resource list` command.
src/chaos/azext_chaos/aaz/latest/chaos/discovered_resource/init.py	Exports discovered-resource subgroup commands.
src/chaos/azext_chaos/aaz/latest/chaos/discovered_resource/__cmd_group.py	AAZ-generated discovered-resource command group.
src/chaos/azext_chaos/aaz/latest/chaos/init.py	Exports top-level `chaos` command group.
src/chaos/azext_chaos/aaz/latest/chaos/__cmd_group.py	AAZ-generated top-level `chaos` command group definition.
src/chaos/azext_chaos/aaz/latest/init.py	AAZ latest package marker.
src/chaos/azext_chaos/aaz/init.py	AAZ package marker.
src/chaos/azext_chaos/_validators.py	Adds validators for `--scopes` and `--parameters`.
src/chaos/azext_chaos/_table_format.py	Adds table formatters for key commands (`workspace`, `scenario`, `run`, etc.).
src/chaos/azext_chaos/_params.py	Adds argument aliases/overrides and hooks up validators.
src/chaos/azext_chaos/init.py	Adds extension command loader and AAZ command table loading.
src/chaos/README.md	Adds extension user documentation and guidance about AAZ regen workflow.
src/chaos/HISTORY.rst	Adds initial release notes and command surface overview.

Comments suppressed due to low confidence (3)

src/chaos/setup.py:1

from codecs import open is unnecessary on Python 3 and shadows the built-in open. Also, opening README.md / HISTORY.rst via relative paths can break when setup.py is invoked from a different working directory (common in build tooling). Use the built-in open and resolve file paths relative to __file__ (e.g., via pathlib.Path(__file__).resolve().parent).
src/chaos/setup.py:1
from codecs import open is unnecessary on Python 3 and shadows the built-in open. Also, opening README.md / HISTORY.rst via relative paths can break when setup.py is invoked from a different working directory (common in build tooling). Use the built-in open and resolve file paths relative to __file__ (e.g., via pathlib.Path(__file__).resolve().parent).
src/chaos/azext_chaos/commands.py:1
chaos workspace show-discovery and chaos workspace show-evaluation are registered below as custom commands, not by load_aaz_command_table(). Including them in _aaz_transformers (and the comment that everything is already registered) is misleading. Either remove these entries from _aaz_transformers or adjust the comment/split the transformer wiring so it's clear which commands are AAZ-generated vs custom-registered.

yonzhan · 2026-06-05T02:15:35Z

chaos

…verrides, tighten validate_scope, fix README JSON examples - _params.py: AAZ-generated commands expose the arg as `resource_group` (not `resource_group_name`); the argument_context overrides were no-ops and have been removed. AAZResourceGroupNameArg already provides -g/--resource-group with the standard completer. - _validators.py: validate_scope now enforces the subscription GUID segment via regex, matching the docstring (previously only checked the `/subscriptions/` prefix). - README.md: switched the `scenario config create` example to valid JSON for --parameters/--filters and added an `@parameters.json` file-form example. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The autogen'd default example for `chaos scenario config execute` referenced the spec-native URL path (`chaos workspace scenario configuration execute …`) rather than the renamed command. Added the leaf to EXAMPLE_REWRITES in the customize_workspace.py driver (Squall 55dcbe5d) and regenerated. No other aaz files changed. Addresses PR Azure#9918 reviewer comment on _execute.py:22. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…trap Introduce a top-level composite command that stands up a complete Chaos Studio environment in one step, inspired by `az containerapp up`: 1. Creates the resource group if it does not exist 2. Creates the workspace with a managed identity (user-assigned via --user-assigned, otherwise system-assigned) 3. Grants the workspace identity the built-in Reader role on each --scopes target (idempotent; skippable via --skip-permissions) so discovery and evaluation can enumerate resources 4. Runs the evaluate-scenarios workflow, then reports the discovered scenarios and suggested next commands `--scopes` is required (mirroring the portal's Create Workspace blade, which has no default scope); help text explains why and documents the ARM resource ID formats for scopes and user-assigned identities. The evaluate step routes through a single `_evaluate_scenarios_workflow` seam framed as the evaluate-scenarios workflow rather than the plumbing refresh-recommendation op, so it absorbs the upcoming spec split of refresh into discover + evaluate without changing setup. When a NEW Reader assignment is created (the case that lags in Azure Resource Graph), evaluation is retried up to 3 times with a delay before emitting a rerun hint; pre-existing no-op assignments and --skip-evaluation-wait take a single attempt. All changes are in hand-written supplementary files (custom.py, commands.py, _help.py, _params.py, _validators.py, _table_format.py) plus tests and README; the generated azext_chaos/aaz/ tree is untouched.

necusjz · 2026-06-17T07:29:42Z

/azp run

azure-pipelines · 2026-06-17T07:29:55Z

Azure Pipelines successfully started running 2 pipeline(s).

- run start: poll the execute LRO until terminal instead of failing healthy long runs at the 600s cap; warn that --no-wait returns immediately (F2) - run wait: raise the wait framework's swallowed timeout error so a wait on a nonexistent run exits non-zero instead of silently succeeding (F6) - scenario run show: surface errors/executionErrors in table output (F4 partial; full root-cause persistence is a BE concern) - scenario config --parameters: enforce [{key,value}] shape with targeted errors and a format hint; apply help to create and update (F7) - run show/cancel/wait: expose --run-id/--name/-n consistently (F9) - scenario config create: hide auto-derived --scenario-id (F10d) - discovered-resource list/show: lead with resourceName/type, not the GUID (F10e) - regenerate aaz tree from updated codegen driver (workspace create example now includes --scopes; config examples use --name; command groups carry real help) azdev style/linter/test all pass; 180 unit tests pass.

necusjz · 2026-06-18T00:41:00Z

/azp run

azure-pipelines · 2026-06-18T00:41:13Z

Azure Pipelines successfully started running 2 pipeline(s).

… include Chaos Studio

necusjz · 2026-06-18T03:59:22Z

/azp run

azure-pipelines · 2026-06-18T03:59:34Z

Azure Pipelines successfully started running 2 pipeline(s).

kekivelez · 2026-06-22T18:27:25Z

@necusjz looks like all CI is passing, anything else needed from me to merge?

necusjz · 2026-06-23T00:21:16Z

@kekivelez please provide the pr link under Azure/aaz, thanks.

kekivelez · 2026-06-23T00:26:57Z

@kekivelez please provide the pr link under Azure/aaz, thanks.

@necusjz sorry this is conflicting information. According to Alex Wang I was told that wasn’t necessary. Can you please confirm whether or not this is necessary as that would be quite the setback for us.

Screenshot from conversation.

kekivelez · 2026-06-24T04:34:47Z

Upstream command-model dependency: the Microsoft.Chaos command models for this extension are now submitted to Azure/aaz here: Azure/aaz#1034

kekivelez · 2026-06-25T05:14:51Z

@necusjz i have added the link to the aaz pr above

azclibot · 2026-06-26T01:12:20Z

[Release] Update index.json for extension [ chaos-1.0.0b1 ] : https://dev.azure.com/msazure/One/_build/results?buildId=169842269&view=results

Cristian Velez (from Dev Box) and others added 7 commits June 3, 2026 20:15

Copilot AI review requested due to automatic review settings June 5, 2026 02:06

azure-client-tools-bot-prd Bot added this to the July 2026 (2026-07-07) milestone Jun 5, 2026

Copilot AI reviewed Jun 5, 2026

View reviewed changes

yonzhan requested a review from necusjz June 5, 2026 02:14

yonzhan assigned necusjz Jun 5, 2026

Cristian Velez (from Dev Box) and others added 4 commits June 5, 2026 15:30

remove -w

de39211

Added missing optional location parameter and updated service name to…

8818d71

… include Chaos Studio

kekivelez mentioned this pull request Jun 24, 2026

[New Module] Add chaos command models (Microsoft.Chaos 2026-05-01-preview) Azure/aaz#1034

Merged

1 task

necusjz approved these changes Jun 26, 2026

View reviewed changes

necusjz merged commit aca9cf4 into Azure:main Jun 26, 2026
24 checks passed

Uh oh!

Conversation

kekivelez commented Jun 5, 2026

Summary

Command surface

Source-of-truth

Related command

General Guidelines

About Extension Publish

Uh oh!

azure-client-tools-bot-prd Bot commented Jun 5, 2026

Uh oh!

azure-client-tools-bot-prd Bot commented Jun 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yonzhan commented Jun 5, 2026

Uh oh!

necusjz commented Jun 17, 2026

Uh oh!

azure-pipelines Bot commented Jun 17, 2026

Uh oh!

necusjz commented Jun 18, 2026

Uh oh!

azure-pipelines Bot commented Jun 18, 2026

Uh oh!

necusjz commented Jun 18, 2026

Uh oh!

azure-pipelines Bot commented Jun 18, 2026

Uh oh!

kekivelez commented Jun 22, 2026

Uh oh!

necusjz commented Jun 23, 2026

Uh oh!

kekivelez commented Jun 23, 2026

Uh oh!

kekivelez commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kekivelez commented Jun 25, 2026

Uh oh!

Uh oh!

azclibot commented Jun 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kekivelez commented Jun 24, 2026 •

edited

Loading