Skip to content

Implementing GitHub Actions for batch deploy pipeline notebook execution #4078

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Copilot wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

Implementing GitHub Actions for batch deploy pipeline notebook execution #4078

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
1e250b7
Add SDK sample for batch deployment from registry pipeline component
lavakumarrepala Jul 21, 2026
ce534a1
Update batch endpoints readme with registry pipeline deployment sample
lavakumarrepala Jul 21, 2026
f9e860d
Format registry batch deployment sample notebook for black
lavakumarrepala Jul 21, 2026
1e380ce
Add CI workflow for from-registry batch pipeline notebook
Copilot Jul 21, 2026
7300117
Fix notebook CI formatting and execution defaults
Copilot Jul 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 94 additions & 0 deletions .github/workflows/sdk-endpoints-batch-deploy-pipelines-from-registry-sdk-deploy-and-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# This code is autogenerated.
# Code is generated by running custom script: python3 readme.py
# Any manual changes to this file may cause incorrect behavior.
# Any manual changes will be overwritten if the code is regenerated.

name: sdk-endpoints-batch-deploy-pipelines-from-registry-sdk-deploy-and-test
# This file is created by sdk/python/readme.py.
# Please do not edit directly.
on:
workflow_dispatch:
schedule:
- cron: "29 5/12 * * *"
pull_request:
branches:
- main
paths:
- sdk/python/endpoints/batch/deploy-pipelines/from-registry/**
- .github/workflows/sdk-endpoints-batch-deploy-pipelines-from-registry-sdk-deploy-and-test.yml
- sdk/python/dev-requirements.txt
- infra/bootstrapping/**
- sdk/python/setup.sh

permissions:
id-token: write
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: check out repo
uses: actions/checkout@v2
- name: setup python
uses: actions/setup-python@v2
with:
python-version: "3.10"
- name: pip install notebook reqs
run: pip install -r sdk/python/dev-requirements.txt
- name: azure login
uses: azure/login@v1
with:
client-id: ${{ secrets.OIDC_AZURE_CLIENT_ID }}
tenant-id: ${{ secrets.OIDC_AZURE_TENANT_ID }}
subscription-id: ${{ secrets.OIDC_AZURE_SUBSCRIPTION_ID }}
- name: bootstrap resources
run: |
echo '${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}';
bash bootstrap.sh
working-directory: infra/bootstrapping
continue-on-error: false
- name: setup SDK
run: |
source "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh";
source "${{ github.workspace }}/infra/bootstrapping/init_environment.sh";
bash setup.sh
working-directory: sdk/python
continue-on-error: true
- name: validate readme
run: |
python check-readme.py "${{ github.workspace }}/sdk/python/endpoints/batch/deploy-pipelines/from-registry"
working-directory: infra/bootstrapping
continue-on-error: false
- name: setup-cli
run: |
source "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh";
source "${{ github.workspace }}/infra/bootstrapping/init_environment.sh";
bash setup.sh
working-directory: cli
continue-on-error: true
- name: Eagerly cache access tokens for required scopes
run: |
# Workaround for azure-cli's lack of support for ID token refresh
# Taken from: https://github.com/Azure/login/issues/372#issuecomment-2056289617

# Management
az account get-access-token --scope https://management.azure.com/.default --output none
# ML
az account get-access-token --scope https://ml.azure.com/.default --output none
- name: run endpoints/batch/deploy-pipelines/from-registry/sdk-deploy-and-test.ipynb
run: |
source "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh";
source "${{ github.workspace }}/infra/bootstrapping/init_environment.sh";
bash "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh" generate_workspace_config "../../.azureml/config.json";
bash "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh" replace_template_values "sdk-deploy-and-test.ipynb";
[ -f "../../.azureml/config" ] && cat "../../.azureml/config";
papermill -k python sdk-deploy-and-test.ipynb sdk-deploy-and-test.output.ipynb --log-output
working-directory: sdk/python/endpoints/batch/deploy-pipelines/from-registry
- name: upload notebook's working folder as an artifact
if: ${{ always() }}
uses: ./.github/actions/upload-artifact
with:
name: sdk-deploy-and-test
path: sdk/python/endpoints/batch/deploy-pipelines/from-registry
1 change: 1 addition & 0 deletions infra/bootstrapping/readme_validation_exclusions.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@
/home/runner/work/azureml-examples/azureml-examples/sdk/python/endpoints/batch/deploy-models/mnist-classifier
/home/runner/work/azureml-examples/azureml-examples/sdk/python/endpoints/batch/deploy-pipelines/batch-scoring-with-preprocessing
/home/runner/work/azureml-examples/azureml-examples/sdk/python/endpoints/batch/deploy-pipelines/hello-batch
/home/runner/work/azureml-examples/azureml-examples/sdk/python/endpoints/batch/deploy-pipelines/from-registry
/home/runner/work/azureml-examples/azureml-examples/sdk/python/endpoints/batch/deploy-pipelines/training-with-components
/home/runner/work/azureml-examples/azureml-examples/sdk/python/endpoints/online/custom-container
/home/runner/work/azureml-examples/azureml-examples/sdk/python/endpoints/online/custom-container/triton
Expand Down
260 changes: 260 additions & 0 deletions sdk/python/endpoints/batch/deploy-pipelines/from-registry/sdk-deploy-and-test.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,260 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Create a batch deployment from a pipeline component stored in a registry\n",
"\n",
"This example shows how to create a batch endpoint deployment from a pipeline component that is already registered in an Azure Machine Learning registry.\n",
"\n",
"> Important: when you retrieve a pipeline component from a registry and use it in `PipelineComponentBatchDeployment`, pass the component **ID** (`component.id`) instead of the component object. This avoids SDK-side re-registration/validation issues for registry-backed pipeline components.\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Connect to the workspace and registry\n",
"\n",
"In this section, we connect to the workspace that will host the batch endpoint and to the registry that stores the pipeline component.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%pip install azure-ai-ml==1.32.0"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"\n",
"from azure.ai.ml import MLClient, load_component\n",
"from azure.ai.ml.entities import (\n",
" AmlCompute,\n",
" BatchEndpoint,\n",
" PipelineComponentBatchDeployment,\n",
")\n",
"from azure.core.exceptions import ResourceNotFoundError\n",
"from azure.identity import DefaultAzureCredential"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Configure your workspace and registry details:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"subscription_id = \"<SUBSCRIPTION_ID>\"\n",
"resource_group = \"<RESOURCE_GROUP>\"\n",
"workspace = \"<AML_WORKSPACE_NAME>\"\n",
"registry_name = \"<REGISTRY_NAME>\"\n",
"component_name = \"train_pipeline_component\"\n",
"component_version = \"1\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"credential = DefaultAzureCredential()\n",
"workspace_ml_client = MLClient(credential, subscription_id, resource_group, workspace)\n",
"registry_ml_client = MLClient(credential, registry_name=registry_name)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"If you are running inside Azure Machine Learning and have a local config, you can connect to the workspace with `MLClient.from_config(DefaultAzureCredential())`.\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Ensure compute exists\n",
"\n",
"Batch deployments run on Azure Machine Learning compute attached to the workspace.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"compute_name = \"batch-cluster\"\n",
"if not any(\n",
" filter(lambda m: m.name == compute_name, workspace_ml_client.compute.list())\n",
"):\n",
" compute_cluster = AmlCompute(\n",
" name=compute_name,\n",
" description=\"Batch endpoints compute cluster\",\n",
" min_instances=0,\n",
" max_instances=5,\n",
" )\n",
" workspace_ml_client.begin_create_or_update(compute_cluster).result()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Get the pipeline component from the registry\n",
"\n",
"Retrieve the pipeline component from the registry.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"component_source = (\n",
" Path.cwd().parents[5]\n",
" / \"cli/jobs/pipelines-with-components/pipeline_with_pipeline_component/pipeline_with_train_eval_pipeline_component/components/train_pipeline_component.yml\"\n",
")\n",
"\n",
"try:\n",
" pipeline_component = registry_ml_client.components.get(\n",
" name=component_name, version=component_version\n",
" )\n",
"except ResourceNotFoundError:\n",
" pipeline_component = load_component(source=component_source)\n",
" pipeline_component.name = component_name\n",
" pipeline_component.version = component_version\n",
" pipeline_component = registry_ml_client.components.create_or_update(\n",
" pipeline_component\n",
" )\n",
"\n",
"print(pipeline_component.id)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Create the batch endpoint\n",
"\n",
"Enable component deployments on the endpoint by setting `ComponentDeployment.Enabled` to `True`.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import random\n",
"import string\n",
"\n",
"endpoint_name = \"registry-pipeline-batch\"\n",
"endpoint_suffix = \"\".join(\n",
" random.choice(string.ascii_lowercase + string.digits) for _ in range(5)\n",
")\n",
"endpoint_name = f\"{endpoint_name}-{endpoint_suffix}\"\n",
"print(endpoint_name)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"endpoint = BatchEndpoint(\n",
" name=endpoint_name,\n",
" description=\"A batch endpoint backed by a pipeline component from registry\",\n",
" properties={\"ComponentDeployment.Enabled\": True},\n",
")\n",
"workspace_ml_client.batch_endpoints.begin_create_or_update(endpoint).result()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Create the deployment\n",
"\n",
"Use the registry component **ID** when creating the deployment.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"deployment = PipelineComponentBatchDeployment(\n",
" name=\"registry-pipeline-deployment\",\n",
" description=\"Batch deployment from a registry pipeline component\",\n",
" endpoint_name=endpoint_name,\n",
" component=pipeline_component.id,\n",
" settings={\n",
" \"continue_on_step_failure\": False,\n",
" \"default_compute\": compute_name,\n",
" },\n",
")\n",
"workspace_ml_client.batch_deployments.begin_create_or_update(deployment).result()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Once created, configure the deployment as the default deployment for the endpoint.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"endpoint = workspace_ml_client.batch_endpoints.get(endpoint_name)\n",
"endpoint.defaults.deployment_name = deployment.name\n",
"workspace_ml_client.batch_endpoints.begin_create_or_update(endpoint).result()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
11 changes: 6 additions & 5 deletions sdk/python/endpoints/batch/readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ Example | Description | Input data type | Notebook
[Batch score an MLflow model for the Heart Disease Classification problem](deploy-models/heart-classifier-mlflow) | This example shows how you can deploy an MLflow model to a batch endpoint to perform batch predictions. This example uses an MLflow model based on the UCI Heart Disease Data Set. The database contains 76 attributes, but we are using a subset of 14 of them. The model tries to predict the presence of heart disease in a patient. It is integer valued from 0 (no presence) to 1 (presence). The model has been trained using an XGBBoost classifier and all the required preprocessing has been packaged as a scikit-learn pipeline, making this model an end-to-end pipeline that goes from raw data to predictions. | Tabular | [See notebook](deploy-models/heart-classifier-mlflow/mlflow-for-batch-tabular.ipynb)
[Batch score an XGBoost model for the Heart Disease Classification problem and write predictions on parquet files](deploy-models/custom-outputs-parquet) | This example shows how you can deploy a model to a batch endpoint to perform batch predictions. This example uses a model based on the UCI Heart Disease Data Set. The database contains 76 attributes, but we are using a subset of 14 of them. The model tries to predict the presence of heart disease in a patient. It is integer valued from 0 (no presence) to 1 (presence). The model has been trained using an XGBBoost classifier and all the required preprocessing has been packaged as a scikit-learn pipeline, making this model an end-to-end pipeline that goes from raw data to predictions. This example also customizes the way the endpoint write predictions. | Tabular | [See notebook](deploy-models/custom-outputs-parquet/custom-output-batch.ipynb)
[Batch score a model for MNIST classification with multiple deployments](deploy-models/mnist-classifier) | In this example, we're going to deploy a model to solve the classic MNIST ("Modified National Institute of Standards and Technology") digit recognition problem to perform batch inferencing over large amounts of data (image files). In the first section of this tutorial, we're going to create a batch deployment with a model created using Torch. Such deployment will become our default one in the endpoint. In the second half, we're going to see how we can create a second deployment using a model created with TensorFlow (Keras), test it out, and then switch the endpoint to start using the new deployment as default. | Images | [See notebook](deploy-models/mnist-classifier/mnist-batch.ipynb)
[Batch score and classify images using a ResNet50 model for the ImageNet dataset](deploy-models/imagenet-classifier) | The model we are going to work with was built using TensorFlow along with the RestNet architecture (Identity Mappings in Deep Residual Networks). This example shows also how to perform high performance inference over batches of images on GPU. | Images | [See notebook](deploy-models/imagenet-classifier/imagenet-classifier-batch.ipynb)
[Batch score and classify images using a ResNet50 model for the ImageNet dataset (MLflow)](deploy-models/imagenet-classifier) | The model we are going to work with was built using TensorFlow along with the RestNet architecture (Identity Mappings in Deep Residual Networks). This example shows you can package the model as MLflow and deploy later. | Images | [See notebook](deploy-models/imagenet-classifier/imagenet-classifier-mlflow.ipynb)
[Batch score a HuggingFace NLP model for text summarization](deploy-models/huggingface-text-summarization) | The model we are going to work with was built using the popular library transformers from HuggingFace along with a pre-trained model from Facebook with the BART architecture. It was introduced in the paper BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation. | Text | [See notebook](deploy-models/huggingface-text-summarization/text-summarization-batch.ipynb)
[Batch score and classify images using a ResNet50 model for the ImageNet dataset](deploy-models/imagenet-classifier) | The model we are going to work with was built using TensorFlow along with the [ResNet](https://arxiv.org/abs/1512.03385) architecture. | Images | [See notebook](deploy-models/imagenet-classifier/tf-image-classification.ipynb)
[Batch score and classify images using a ResNet50 model for the ImageNet dataset (MLflow)](deploy-models/imagenet-classifier) | The model we are going to work with was built using TensorFlow along with the [ResNet](https://arxiv.org/abs/1512.03385) architecture and packaged in MLflow format. | Images | [See notebook](deploy-models/imagenet-classifier/mlflow-image-classification.ipynb)
[Batch score a HuggingFace NLP model for text summarization](deploy-models/huggingface-text-summarization) | The model we are going to work with was built using the popular library transformers from HuggingFace. | Text | [See notebook](deploy-models/huggingface-text-summarization/text-summarization-batch.ipynb)


### Deploying pipeline components
Expand All @@ -30,5 +30,6 @@ The following section contains examples about how to deploy pipeline components
Example | Description | Input data type | Notebook
-|-|-|-
[Hello batch endpoints](deploy-pipelines/hello-batch) | This examples performs a simple Hello World example to ensure you can create batch endpoints with component deployments without issues. | None | [See notebook](deploy-pipelines/hello-batch/sdk-deploy-and-test.ipynb)
[Operationalize a training routine with Batch Endpoints](deploy-pipelines/training-with-components/) | Learn how to deploy a training pipeline under a batch endpoint to perform training over a tabular dataset. This pipeline multiple uses components (steps) defined in YAML and produces multiple outputs of the steps within, including models, transformations and evaluation results. It also use registered data assets as input data. | Tabular | [See notebook](deploy-pipelines/training-with-components/sdk-deploy-and-test.ipynb)
[Batch scoring with pre-processing](deploy-pipelines/batch-scoring-with-preprocessing/) | Learn how to deploy a pipeline under a batch endpoint that reuses a preprocessing component from the training routine to perform inference before running the model. This example not only reuses the code from the existing component but also pulls assets from the registry, including the model and the normalization parameters learnt during training. | Tabular and literal string | [See notebook](deploy-pipelines/batch-scoring-with-preprocessing/sdk-deploy-and-test.ipynb)
[Operationalize a training routine with Batch Endpoints](deploy-pipelines/training-with-components/) | Learn how to deploy a training pipeline under a batch endpoint to perform training over a tabular dataset. | Tabular | [See notebook](deploy-pipelines/training-with-components/sdk-deploy-and-test.ipynb)
[Batch scoring with pre-processing](deploy-pipelines/batch-scoring-with-preprocessing/) | Learn how to deploy a pipeline under a batch endpoint that reuses a preprocessing component from the training stage. | Tabular | [See notebook](deploy-pipelines/batch-scoring-with-preprocessing/sdk-deploy-and-test.ipynb)
[Create a batch deployment from a pipeline component in registry](deploy-pipelines/from-registry/) | Learn how to retrieve a pipeline component from a registry and create a batch deployment by passing the component ID to avoid registry-backed SDK validation issues. | Depends on registered component | [See notebook](deploy-pipelines/from-registry/sdk-deploy-and-test.ipynb)