Skip to content

Migrate CI/CD from GitHub Actions to ADO; remove GH Actions workflow #895

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
4gust merged 19 commits into from
May 21, 2026
Merged

Migrate CI/CD from GitHub Actions to ADO; remove GH Actions workflow #895

Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
ea16799
Add ADO pipeline hyperlinks to CI-AND-RELEASE-PIPELINES.md
RyAuld Mar 31, 2026
5eaf533
Add Python 3.8 to ADO CI matrix
RyAuld Mar 31, 2026
bd41633
Move benchmarks from GH Actions to ADO; remove cb job from GH workflow
RyAuld Mar 31, 2026
10814c4
Remove GH Actions CI/CD workflow — tests and publish fully covered by…
RyAuld Mar 31, 2026
a8154b4
Address Copilot review: fix benchmark branch condition, switch to Pub…
RyAuld Mar 31, 2026
6107946
Fix: move retries from job to retryCountOnTaskFailure on pytest step …
RyAuld Mar 31, 2026
1bd1f0a
Address Copilot review: wire tsaConfig.json to PublishSecurityAnalysi…
RyAuld Mar 31, 2026
a1a9cff
Merge origin/dev into RyAuld/update-pipeline-doc-links; keep deletion…
RyAuld May 8, 2026
de03197
Move cryptography version-gating tests to warning-only step
RyAuld May 8, 2026
358e010
Add Benchmark stage to pipeline documentation
RyAuld May 8, 2026
cfbd078
Address Copilot review: benchmark skip, pytest install, artifact guard
RyAuld May 8, 2026
30bd106
Fix: allow Build/Publish stages to proceed on SucceededWithIssues
RyAuld May 14, 2026
970bb02
Upgrade EsrpRelease task from v9 to v12
RyAuld May 15, 2026
02c6b07
updated for e2e tests
4gust May 21, 2026
180a6ff
merged dev
4gust May 21, 2026
faaa5fe
updated the pipeline runtime and updated the version for sku for rc
4gust May 21, 2026
8203f47
Update template-pipeline-stages.yml
4gust May 21, 2026
5cbdc58
Update template-pipeline-stages.yml
4gust May 21, 2026
6e3e6ad
Resolved comments
4gust May 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 35 additions & 21 deletions .Pipelines/CI-AND-RELEASE-PIPELINES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,45 +7,52 @@ including what each pipeline does, when it runs, and how to trigger a release.

## Pipeline Files

| File | Purpose |
|------|---------|
| [`azure-pipelines.yml`](../azure-pipelines.yml) | PR gate and post-merge CI — calls the shared template with `runPublish: false` |
| [`pipeline-publish.yml`](pipeline-publish.yml) | Release pipeline — manually queued, builds and publishes to PyPI |
| [`template-pipeline-stages.yml`](template-pipeline-stages.yml) | Shared stages template — PreBuildCheck, Validate, and CI stages reused by both pipelines |
| [`credscan-exclusion.json`](credscan-exclusion.json) | CredScan suppression file for known test fixtures |
| File | ADO Pipeline | Purpose |
|------|-------------|---------|
| [`azure-pipelines.yml`](../azure-pipelines.yml) | [MSAL.Python-PR-OneBranch-Official (3064)](https://dev.azure.com/IdentityDivision/IDDP/_build?definitionId=3064) | PR gate, post-merge CI, and performance benchmarks — calls the shared template with `runPublish: false`; runs benchmarks on post-merge pushes to `dev` |
| [`pipeline-publish.yml`](pipeline-publish.yml) | [MSAL.Python-Publish (3067)](https://dev.azure.com/IdentityDivision/IDDP/_build?definitionId=3067) | Release pipeline — manually queued, builds and publishes to PyPI |
Comment thread
RyAuld marked this conversation as resolved.
| [`template-pipeline-stages.yml`](template-pipeline-stages.yml) | — | Shared stages template — PreBuildCheck, Validate, UnitTests, and E2ETests stages reused by both pipelines |
| [`credscan-exclusion.json`](credscan-exclusion.json) | — | CredScan suppression file for known test fixtures |

---

## PR / CI Pipeline (`azure-pipelines.yml`)
## PR / CI Pipeline — [MSAL.Python-PR-OneBranch-Official (3064)](https://dev.azure.com/IdentityDivision/IDDP/_build?definitionId=3064)

### Triggers

| Event | Branches |
|-------|----------|
| Pull request opened / updated | all branches |
| Push / merge | `dev`, `azure-pipelines` |
| Pull request opened / updated | `dev` (PRs targeting `dev` only) |
| Push / merge | `dev` |
| Scheduled | Daily at 11:45 PM Pacific, `dev` branch (only when there are new changes) |

Fast unit-test feedback for PRs targeting **other** branches (e.g. `release-x.y.z`)
is provided separately by the GitHub Actions workflow
[`.github/workflows/python-package.yml`](../.github/workflows/python-package.yml),
which runs the package build and unit tests on every PR.
Comment thread
4gust marked this conversation as resolved.

### Stages

```
PreBuildCheck ─► CI
PreBuildCheck ─► UnitTests ─► E2ETests ─► Benchmark (post-merge to dev only)
```

| Stage | What it does |
|-------|-------------|
| **PreBuildCheck** | Runs SDL security scans: PoliCheck (policy/offensive content), CredScan (leaked credentials), and PostAnalysis (breaks the build on findings) |
| **CI** | Runs the full test suite on Python 3.9, 3.10, 3.11, 3.12, 3.13, and 3.14 |
| Stage | What it does | When it runs |
|-------|-------------|-------------|
| **PreBuildCheck** | Runs SDL security scans: PoliCheck (policy/offensive content), CredScan (leaked credentials), and PostAnalysis (breaks the build on findings) | Always |
| **UnitTests** | Runs the unit test suite on Python 3.9, 3.10, 3.11, 3.12, 3.13, and 3.14 (no Key Vault required) | After PreBuildCheck |
| **E2ETests** | Fetches the MSID Lab certificate from Key Vault and runs `tests/test_e2e.py` + `tests/test_fmi_e2e.py` on the same Python matrix. Skipped on forked PRs (no Key Vault access). | After UnitTests |
| **Benchmark** | Runs performance benchmarks on Python 3.9 and publishes `benchmark-results` artifact | Post-merge pushes to `dev` and manual runs only |
Comment thread
4gust marked this conversation as resolved.

The Validate stage is **skipped** on PR/CI runs (it only applies to release builds).
The `Validate` stage is **skipped** on PR/CI runs (it only applies to release builds).

> **SDL coverage:** The PreBuildCheck stage satisfies the OneBranch SDL requirement.
> It runs on every PR, every merge to `dev`, and on the daily schedule — ensuring
> continuous security scanning without a separate dedicated SDL pipeline.

---

## Release Pipeline (`pipeline-publish.yml`)
## Release Pipeline — [MSAL.Python-Publish (3067)](https://dev.azure.com/IdentityDivision/IDDP/_build?definitionId=3067)

### Triggers

Expand All @@ -62,18 +69,25 @@ with both parameters filled in.
### Stage Flow

```
PreBuildCheck ─► Validate ─► CI ─► Build ─┬─► PublishMSALPython (publishTarget == 'test.pypi.org (Preview / RC)')
└─► PublishPyPI (publishTarget == 'pypi.org (ESRP Production)')
PreBuildCheck ─► Validate ─► UnitTests ─► E2ETests ─► Build ─┬─► PublishMSALPython (publishTarget == 'test.pypi.org (Preview / RC)')
└─► PublishPyPI (publishTarget == 'pypi.org (ESRP Production)')
```

| Stage | What it does | Condition |
|-------|-------------|-----------|
| **PreBuildCheck** | PoliCheck + CredScan scans | Always |
| **Validate** | Asserts the `packageVersion` parameter matches `msal/sku.py __version__` | Always (release runs only) |
| **CI** | Full test matrix (Python 3.9–3.14) | After Validate passes |
| **Build** | Builds `sdist` and `wheel` via `python -m build`; publishes `python-dist` artifact | After CI passes |
| **UnitTests** | Unit test matrix (Python 3.9–3.14) | After Validate passes |
| **E2ETests** | E2E test matrix (Python 3.9–3.14) with MSID Lab cert from Key Vault | After UnitTests passes |
| **Build** | Builds `sdist` and `wheel` via `python -m build`; publishes `python-dist` artifact | After E2ETests passes |
| **PublishMSALPython** | Uploads to test.pypi.org | `publishTarget == test.pypi.org (Preview / RC)` |
| **PublishPyPI** | Uploads to PyPI via ESRP; requires manual approval | `publishTarget == pypi.org (ESRP Production)` |
| **PublishPyPI** | Uploads to PyPI via ESRP (`EsrpRelease@12`); requires manual approval | `publishTarget == pypi.org (ESRP Production)` |

> ⚠️ **TestPyPI publishing is currently a no-op.** The `MSAL-Test-Python-Upload`
> service connection has not yet been created (pending a test.pypi.org API
> token), so the `PublishMSALPython` stage prints a skip message rather than
> uploading. Until the SC exists, use the `pypi.org (ESRP Production)` path
> with an RC version (e.g. `1.36.0rc1`) for end-to-end validation.

---

Expand Down
22 changes: 11 additions & 11 deletions .Pipelines/pipeline-publish.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
# Publish targets:
# test.pypi.org (Preview / RC) — preview releases via MSAL-Test-Python-Upload SC
# (SC creation pending test.pypi.org API token)
# pypi.org (ESRP Production) — production releases via ESRP (EsrpRelease@9) using MSAL-ESRP-AME SC
# pypi.org (ESRP Production) — production releases via ESRP (EsrpRelease@12) using MSAL-ESRP-AME SC
#
# For pipeline documentation, see .Pipelines/CI-AND-RELEASE-PIPELINES.md.

Expand All @@ -27,12 +27,12 @@ pr: none

# Stage flow:
#
# PreBuildCheck ─► Validate ─► CI ─► Build ─► PublishMSALPython (publishTarget == Preview)
# └─► PublishPyPI (publishTarget == ESRP Production)
# PreBuildCheck ─► Validate ─► UnitTests ─► E2ETests ─► Build ─► PublishMSALPython (publishTarget == Preview)
# └─► PublishPyPI (publishTarget == ESRP Production)

stages:

# PreBuildCheck, Validate, and CI stages are defined in the shared template.
# PreBuildCheck, Validate, UnitTests, and E2ETests stages are defined in the shared template.
- template: template-pipeline-stages.yml
parameters:
packageVersion: ${{ parameters.packageVersion }}
Expand All @@ -43,8 +43,8 @@ stages:
# ══════════════════════════════════════════════════════════════════════════════
- stage: Build
displayName: 'Build package'
dependsOn: CI
condition: eq(dependencies.CI.result, 'Succeeded')
dependsOn: E2ETests
condition: in(dependencies.E2ETests.result, 'Succeeded', 'SucceededWithIssues')
jobs:
- job: BuildDist
displayName: 'Build sdist + wheel (Python 3.12)'
Expand Down Expand Up @@ -83,7 +83,7 @@ stages:
dependsOn: Build
condition: >
and(
eq(dependencies.Build.result, 'Succeeded'),
in(dependencies.Build.result, 'Succeeded', 'SucceededWithIssues'),
Comment thread
4gust marked this conversation as resolved.
Outdated
eq('${{ parameters.publishTarget }}', 'test.pypi.org (Preview / RC)')
)
jobs:
Expand Down Expand Up @@ -130,17 +130,17 @@ stages:

# ══════════════════════════════════════════════════════════════════════════════
# Stage 4b · Publish to PyPI (ESRP Production)
# Uses EsrpRelease@9 via the MSAL-ESRP-AME service connection.
# Uses EsrpRelease@12 via the MSAL-ESRP-AME service connection.
# IMPORTANT: configure a required manual approval on this environment in
# ADO → Pipelines → Environments → MSAL-Python-Release → Approvals and checks.
# IMPORTANT: EsrpRelease@9 requires a Windows agent.
# IMPORTANT: EsrpRelease@12 requires a Windows agent.
# ══════════════════════════════════════════════════════════════════════════════
- stage: PublishPyPI
displayName: 'Publish to PyPI (ESRP Production)'
dependsOn: Build
condition: >
and(
eq(dependencies.Build.result, 'Succeeded'),
in(dependencies.Build.result, 'Succeeded', 'SucceededWithIssues'),
Comment thread
4gust marked this conversation as resolved.
Outdated
eq('${{ parameters.publishTarget }}', 'pypi.org (ESRP Production)')
)
jobs:
Expand All @@ -159,7 +159,7 @@ stages:
artifactName: python-dist
targetPath: $(Pipeline.Workspace)/python-dist

- task: EsrpRelease@9
- task: EsrpRelease@12
displayName: 'Publish to PyPI via ESRP'
inputs:
connectedservicename: 'MSAL-ESRP-AME'
Expand Down
Loading
Loading