feat(hubs): add Azure Resource Manager ingestion - #2277
Draft
Brett Wilson (MSBrett) wants to merge 2 commits into
Draft
feat(hubs): add Azure Resource Manager ingestion#2277Brett Wilson (MSBrett) wants to merge 2 commits into
Brett Wilson (MSBrett) wants to merge 2 commits into
Conversation
Add generic ARM query ingestion with quota and Savings Plan definitions, normalized KQL surfaces, catalog queries, documentation, and regression coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Brett Wilson (MSBrett)
had a problem deploying
to
ftk-pr
August 24, 2026 00:28 — with 
GitHub Actions
Error
GitHub Actions
Error
microsoft-github-policy-service
Bot
added
the
Needs: Review 👀
Contributor
|
Hubs + ADX (managed) deployment failed. View logs. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds reusable Azure Resource Manager ingestion to FinOps hubs, supporting savings-plan recommendations and quota/capacity datasets.
Changes:
- Adds manifest-driven ARM GET ingestion and deployment wiring.
- Adds normalized quota and savings-plan schemas and KQL queries.
- Extends portal configuration, documentation, and regression tests.
Reviewed changes
Copilot reviewed 45 out of 46 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
src/templates/finops-hub/test/main.test.bicep |
Adds feature deployment combinations. |
.../Recommendations/schemas/recommendations_1.1.json |
Defines savings-plan translation. |
.../Recommendations/queries/Recommendations-Microsoft-SavingsPlan-P3Y.json |
Adds three-year recommendations. |
.../Recommendations/queries/Recommendations-Microsoft-SavingsPlan-P1Y.json |
Adds one-year recommendations. |
.../Recommendations/app.bicep |
Registers new recommendation assets. |
.../Quota/schemas/quota_1.0-usage.json |
Defines usage translation. |
.../Quota/schemas/quota_1.0-sql.json |
Defines SQL quota translation. |
.../Quota/schemas/quota_1.0-disk.json |
Defines disk translation. |
.../Quota/schemas/quota_1.0-capacity-reservation.json |
Defines capacity translation. |
.../Quota/queries/Quota-Microsoft-StorageUsage.json |
Adds Storage usage ingestion. |
.../Quota/queries/Quota-Microsoft-SqlSubscriptionUsage.json |
Adds SQL usage ingestion. |
.../Quota/queries/Quota-Microsoft-PremiumSSDv2Disk.json |
Adds disk inventory ingestion. |
.../Quota/queries/Quota-Microsoft-ComputeUsage.json |
Adds compute usage ingestion. |
.../Quota/queries/Quota-Microsoft-CognitiveServicesUsage.json |
Adds AI quota ingestion. |
.../Quota/queries/Quota-Microsoft-CapacityReservation.json |
Adds capacity inventory ingestion. |
.../Quota/queries/Quota-Microsoft-AppServiceUsage.json |
Adds App Service usage ingestion. |
.../Quota/app.bicep |
Deploys quota queries and schemas. |
.../IngestionQueries/app.bicep |
Integrates ARM staging and finalization. |
.../AzureResourceManager/README.md |
Documents the ARM engine. |
.../AzureResourceManager/metadata.bicep |
Defines ARM app metadata. |
.../AzureResourceManager/app.bicep |
Implements ARM request pipelines. |
.../AzureResourceGraph/app.bicep |
Extends the shared engine contract. |
.../Analytics/scripts/IngestionSetup_v1_2.kql |
Normalizes savings-plan recommendations. |
.../Analytics/scripts/IngestionSetup_v1_0.kql |
Adds quota transforms and tables. |
.../Analytics/scripts/IngestionSetup_RawTables.kql |
Adds the quota raw table. |
.../Analytics/scripts/HubSetup_v1_0.kql |
Adds the versioned quota function. |
.../Analytics/scripts/HubSetup_Latest.kql |
Adds the latest quota alias. |
src/templates/finops-hub/modules/hub.bicep |
Wires ARM and quota apps. |
.../fx/hub-deploymentScript.bicep |
Forces deployment-script refreshes. |
src/templates/finops-hub/main.bicep |
Exposes quota configuration. |
src/templates/finops-hub/createUiDefinition.json |
Adds portal controls and guidance. |
src/scripts/Build-HubIngestionQueries.ps1 |
Generates manifests for both apps. |
src/queries/INDEX.md |
Catalogs the new queries. |
src/queries/finops-hub-database-guide.md |
Documents the quota contract. |
src/queries/catalog/savings-plan-recommendation-breakdown.kql |
Summarizes savings-plan recommendations. |
src/queries/catalog/quota-storage-usage.kql |
Queries Storage quota state. |
src/queries/catalog/quota-sql-subscription-usage.kql |
Queries SQL quota state. |
src/queries/catalog/quota-premium-ssd-v2-disks.kql |
Queries Premium SSD v2 inventory. |
src/queries/catalog/quota-headroom.kql |
Calculates quota headroom. |
src/queries/catalog/quota-current-usage.kql |
Queries current quota state. |
src/queries/catalog/quota-compute-usage.kql |
Queries compute quota state. |
src/queries/catalog/quota-cognitive-services-usage.kql |
Queries AI quota state. |
src/queries/catalog/quota-capacity-reservations.kql |
Queries capacity reservations. |
src/queries/catalog/quota-app-service-usage.kql |
Queries App Service quota state. |
src/powershell/Tests/Unit/HubsIngestionQueries.Tests.ps1 |
Extends ingestion regression coverage. |
.gitignore |
Ignores Copilot tracking artifacts. |
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+1029
to
+1031
| relativeUrl: { | ||
| value: '@concat(pipeline().parameters.queryScope, replace(pipeline().parameters.query, \'{location}\', pipeline().parameters.queryLocation))' | ||
| type: 'Expression' |
Comment on lines
+398
to
+400
| condition: { | ||
| value: '@and(not(empty(item().scope)), startswith(toLower(item().scope), \'/providers/microsoft.billing/\'))' | ||
| type: 'Expression' |
Comment on lines
+1005
to
+1007
| paginationRules: { | ||
| AbsoluteUrl: '$.nextLink' | ||
| } |
| name: 'queries_AzureResourceManager_CopyQuery' | ||
| parent: dataFactory | ||
| properties: { | ||
| concurrency: 1 |
Comment on lines
+1041
to
+1043
| blobPath: { | ||
| value: '@concat(\'_ftk-query-staging/\', replace(pipeline().parameters.ingestionPath, concat(pipeline().parameters.queryType, \'.parquet\'), \'\'), \'/SubAccountId=\', last(split(pipeline().parameters.queryScope, \'/\')), if(empty(pipeline().parameters.queryLocation), \'\', concat(\'/location=\', pipeline().parameters.queryLocation)), \'/x_SourceName=Azure Resource Manager/x_SourceProvider=Microsoft\', if(contains(string(pipeline().parameters.translator), \'x_SourceType\'), \'\', concat(\'/x_SourceType=\', pipeline().parameters.queryType)), \'/x_SourceVersion=\', pipeline().parameters.queryVersion, \'/\', pipeline().parameters.queryType, \'--\', pipeline().parameters.queryVersion, \'--\', replace(pipeline().parameters.queryScope, \'/\', \'_\'), \'--\', pipeline().parameters.queryLocation, \'.parquet\')' | ||
| type: 'Expression' |
| tmp_QueryLocation = replace_string(tostring(tmp_FileParts[3]), '.parquet', '') | ||
| | extend | ||
| ProviderName = coalesce(ProviderName, 'Microsoft'), | ||
| SubAccountId = tolower(coalesce(SubAccountId, tmp_QueryScope)), |
Comment on lines
+30
to
+37
| | summarize arg_max(x_RecommendationDate, *) by BenefitType, ArmSkuName, Term, LookBackPeriod, Scope | ||
| | project | ||
| BenefitType, | ||
| ArmSkuName, | ||
| Term, | ||
| LookBackPeriod, | ||
| Scope, | ||
| CurrencyCode, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🛠️ Description
This PR added a reusable Azure Resource Manager ingestion path for GET-only APIs and connected its output to the existing FinOps hub ingestion flow.
Managed Azure Resource Manager ingestion
Quota and recommendation data
Quota()functions and query-catalog entries for current usage, headroom, source-specific quota views, and savings plan recommendation breakdowns.Documentation and tests
.copilot-tracking/workflow artifacts.This work provides shared ingestion groundwork for MACC consumption. MACC Lots and MACC Events ingestion are not included in this PR. Quota and capacity support extends beyond the current iteration described in #1721.
Refs #1721
Related to #2120 and #2129
📷 Screenshots
Not applicable.
📋 Checklist
🔬 How did you test this change?
The full unit suite passed with 2,453 tests and no failures. The full lint suite passed with 3,439 tests and no failures. App Service ingestion completed through Data Factory and Azure Data Explorer, and all seven quota catalog projections returned the expected schema and key behavior.
📦 Deploy to test?
🙋♀️ Do any of the following that apply?
📑 Did you update
docs/changelog.md?A changelog entry is pending before this draft is ready for review.
📖 Did you update documentation?
Public documentation remains pending before this draft is ready for review.