feat: update promotion scripts to accommodate Airflow v1 and v2 (for Airflow 3 upgrade) by botanical · Pull Request #454 · NASA-IMPACT/veda-data

botanical · 2026-09-11T22:03:24Z

https://github.com/NASA-IMPACT/veda-architecture/issues/849

What Changed

I added scripts/airflow_api.py that triggers that DAGs according to the version it identifies based on env var settings
I updated the promotion scripts so that it would check the Airflow versions on Staging and Production (defaulting to use v2/ Airflow 3)
Notably, this refactor handles both Airflow 2 and Airflow 3 and allows staging and prod instances to be on different versions are we're staggering our rollout

anayeaye

These changes look good--was the main update just to the airflow base url?

botanical · 2026-09-18T21:43:56Z

+              else
+                echo "$collection_id failed to publish ❌"
+                exit 1
+              fi


I can delete this file once this PR is reviewed!

jjfrench · 2026-09-21T21:28:23Z

+        status_code = _extract_http_status_code(str(e))
+        print(json.dumps({"statusCode": status_code, "error": str(e)}))


should this raise?

I don't want it to raise necessarily because I don't want it to prematurely abort before it reports on the results of the files

maybe I'm confused then, should we catching a different exception? AirflowAPIError reads as if the trigger_dag_run request failed and the dag never started

Co-authored-by: Jamison French <50224594+jjfrench@users.noreply.github.com>

anayeaye

Looks good, I just realized that we have one more task to handle new dataset/collection publication and it looks like the existing promote_ scripts could be generalized as publish_ scripts to support both publication and promotion.

~~I approve of these changes if we want to make a second PR to handle both new dataset and collection publication and promotion~~

anayeaye · 2026-09-24T17:36:25Z

EDIT: I see that we are already using the promote_ scripts for new dataset publication. The language is confusing but if the functionality is working I guess we are OK. Maybe eventually we can touch up the readme to clarify that in this context promote does not only mean promote from staging to prod it also just means publish. Regardless, I am glad we don't have another PR to open :)

botanical added 3 commits September 11, 2026 14:40

feat: create airflow api functions based on version

cc23b0b

fix: functions and formatting

0794589

fix: update promotion scripts to use airflow functions

d03fdc2

botanical requested a review from smohiudd as a code owner September 11, 2026 22:03

fix: comment for airflows

e158b0b

anayeaye reviewed Sep 14, 2026

View reviewed changes

Comment thread scripts/airflow_api.py Outdated

Comment thread scripts/promote_collection.py

sandrahoang686 reviewed Sep 15, 2026

View reviewed changes

Comment thread scripts/promote_collection.py Outdated

Comment thread scripts/airflow_api.py

Comment thread scripts/airflow_api.py Outdated

Comment thread scripts/promote_dataset.py Outdated

fix: modify based on feedback

e891646

botanical requested review from anayeaye and sandrahoang686 September 15, 2026 20:47

botanical added 5 commits September 15, 2026 18:18

feat: create workflow for testing

eb49ffe

feat: update env vars to emphasize testing

fe5ccd5

fix: add pythonpath to env

eed6ca9

fix: add test collections

545ac9e

fix: add environment to pick up github envs

3bc9168

botanical deployed to staging September 16, 2026 01:44 — with GitHub Actions Active

fix: add oauth2 support for aiflow 3

14f98f6

botanical deployed to staging September 16, 2026 02:03 — with GitHub Actions Active

fix: add keycloak env vars and call to token endpoint

c9538bd

botanical deployed to staging September 16, 2026 02:22 — with GitHub Actions Active

fix: typos and fail gh jobs if status code not 200

31577a2

botanical had a problem deploying to staging September 16, 2026 02:27 — with GitHub Actions Failure

botanical had a problem deploying to staging September 16, 2026 02:29 — with GitHub Actions Failure

fix: attempt to log error message for login

9580f88

botanical had a problem deploying to staging September 16, 2026 02:30 — with GitHub Actions Failure

fix: token endpoint

1f38629

botanical had a problem deploying to staging September 16, 2026 02:35 — with GitHub Actions Failure

fix: update grant type, remove username and password for oauth2

75820d5

botanical had a problem deploying to staging September 16, 2026 02:42 — with GitHub Actions Failure

botanical had a problem deploying to staging September 16, 2026 22:10 — with GitHub Actions Failure

botanical had a problem deploying to staging September 18, 2026 19:09 — with GitHub Actions Failure

fix: pass in jwt sub

1bbb426

botanical had a problem deploying to staging September 18, 2026 19:11 — with GitHub Actions Failure

botanical had a problem deploying to staging September 18, 2026 19:13 — with GitHub Actions Failure

fix: logical date setting

11776b3

botanical had a problem deploying to staging September 18, 2026 20:54 — with GitHub Actions Failure

fix: add run id and fix call to build request body

b25f3e8

botanical deployed to staging September 18, 2026 21:02 — with GitHub Actions Active

botanical requested a review from a team September 18, 2026 21:16

fix: cleanup commented out code

3f50928

botanical deployed to staging September 18, 2026 21:43 — with GitHub Actions Active

botanical commented Sep 18, 2026

View reviewed changes

jjfrench reviewed Sep 21, 2026

View reviewed changes

jjfrench requested a review from a team September 21, 2026 21:30

fix: update based on feedback and add some docs

5087977

botanical had a problem deploying to staging September 22, 2026 00:08 — with GitHub Actions Failure

botanical had a problem deploying to staging September 22, 2026 00:13 — with GitHub Actions Failure

botanical deployed to staging September 22, 2026 00:19 — with GitHub Actions Active

botanical requested a review from jjfrench September 22, 2026 00:23

jjfrench reviewed Sep 22, 2026

View reviewed changes

Comment thread docs/README.md Outdated

jjfrench reviewed Sep 22, 2026

View reviewed changes

Comment thread docs/README.md Outdated

jjfrench reviewed Sep 22, 2026

View reviewed changes

Comment thread docs/README.md

jjfrench requested a review from a team September 22, 2026 14:38

sandrahoang686 reviewed Sep 22, 2026

View reviewed changes

Comment thread docs/README.md Outdated

Comment thread scripts/airflow_api.py

botanical and others added 2 commits September 22, 2026 10:09

Apply batched suggestions from code review

46588f4

Co-authored-by: Jamison French <50224594+jjfrench@users.noreply.github.com>

fix: update docs a bit

714e57f

ividito approved these changes Sep 24, 2026

View reviewed changes

anayeaye reviewed Sep 24, 2026

View reviewed changes

Comment thread docs/README.md

anayeaye approved these changes Sep 24, 2026

View reviewed changes

		status_code = _extract_http_status_code(str(e))
		print(json.dumps({"statusCode": status_code, "error": str(e)}))

Conversation

botanical commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Changed

Uh oh!

anayeaye left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

botanical Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jjfrench Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

botanical Sep 22, 2026

Choose a reason for hiding this comment

Uh oh!

jjfrench Sep 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anayeaye left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anayeaye commented Sep 24, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

botanical commented Sep 11, 2026 •

edited

Loading

anayeaye left a comment •

edited

Loading