Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 36 additions & 8 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,42 @@
# Environment Variables

# README
The workflows read config from two GitHub Environments (under Settings -> Environments): `staging` and `production`. A job only sees the secrets and vars of the environment it declares. The same secret name, therefore, can hold a different value in each stage.

## Which workflow uses which environment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could see a comprehensive environment table for each environment being useful for readability - could maybe consolidate the table below, then too


- [pr.yml](../.github/workflows/pr.yml) uses the `staging` environment. It runs when a PR is opened or updated and publishes to staging Airflow.
- [promote.yml](../.github/workflows/promote.yml) uses the `production` environment, which [promotion-checker.yml](../.github/workflows/promotion-checker.yml) triggers after the PR is approved. It publishes to production Airflow.

For the GitHub `staging` environment, the following config should be set:

- Secrets, for a staging instance using Airflow 2, `STAGING_SM2A_ADMIN_USERNAME` and `STAGING_SM2A_ADMIN_PASSWORD`.
- Secrets, for a staging instance using Airflow 3, `AIRFLOW_JWT_SECRET` and `AIRFLOW_JWT_SUB`.
- Secret for the MDX PR step: `APP_PEM`.
- Vars: `STAGING_SM2A_API_URL`, `STAGING_AIRFLOW_API_VERSION`, `DATASET_DAG_NAME`, and `ARTIFACT_RETENTION_DAYS`.
- Vars for the MDX step: `SKIP_MDX_PR_STEP`, `VEDA_CONFIG_REPO_ORG`, `VEDA_CONFIG_REPO_NAME`, `VEDA_CONFIG_APP_ID`, and `GH_ACTOR_EMAIL`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a specific reason why "GH_ACTOR_EMAIL" needs to be a var?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm maybe it doesn't need to be a var but we made it this way to be easily configurable (and I believe it's required for the git config setup)


For the GitHub `production` environment, the following config should be set:

- Secrets, for a production instance using Airflow 2, `SM2A_ADMIN_USERNAME` and `SM2A_ADMIN_PASSWORD`.
- Secrets, for a production instance using Airflow 3, `AIRFLOW_JWT_SECRET` and `AIRFLOW_JWT_SUB` *which should be different values from what is set in staging*.
- Vars: `SM2A_API_URL`, `PRODUCTION_AIRFLOW_API_VERSION` and `PROMOTION_DAG_NAME`.

> Note: `*_AIRFLOW_API_VERSION` is the Airflow major version (can be either 2 or 3) which defaults to 2 if unset. The JWT secrets **are only needed for Airflow 3**.

### Repo-level secrets

Set repo level secrets under `Settings` -> `Secrets and variables` -> `Actions`.

- `WORKFLOW_TRIGGER_TOKEN` which is a repo-level secret that isn't environment scoped. It is required for `promote.yml`.

## Environment Variables Reference for Collection and Dataset Promotion

> See the [terminology note](../README.md#step-4-promote-to-production) in the root README for what "promotion" implies.
>
>
| Variable | Used by | Default | Notes |
| -- | -- | -- | -- |
| STAGING_AIRFLOW_API_VERSION | Staging Publish/Promotion | 2 | The Airflow Version for the Staging Environment. For Airflow 2 = API v1 + Basic auth, Airflow 3 = API v2 + JWT |
| PRODUCTION_AIRFLOW_API_VERSION | Production Promotion | 2 | Same as above |
| AIRFLOW_JWT_SECRET | Any Publish/Promotion step that uses Airflow 3 | none | Conditionally required when the resolved API version is 3. Must match the secret that the Airflow 3 deployment signs with. Can be found in Secrets Manager |
| AIRFLOW_JWT_SUB | Airflow 3 only | none | Conditionally required when the resolved API verison is 3. The subject claim for the token (the Integer ID from Airflow's user table) |

| Variable | Set in | Used by | Default | Notes |
| -- | -- | -- | -- | -- |
| STAGING_AIRFLOW_API_VERSION | Staging GitHub Env | Staging Publish/Promotion | 2 | The Airflow Version for the Staging Environment. For Airflow 2 = API v1 + Basic auth, Airflow 3 = API v2 + JWT |
| PRODUCTION_AIRFLOW_API_VERSION | Production GitHub Env | Production Promotion | 2 | Same as above |
| AIRFLOW_JWT_SECRET | Both (optional) | Any Publish/Promotion step that uses Airflow 3 | none | Conditionally required when the resolved API version is 3. Must match the secret that the Airflow 3 deployment signs with. Can be found in Secrets Manager |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both - referring to Staging and Production GitHub Env?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes! should I make that more clear?

| AIRFLOW_JWT_SUB | Both (optional) | Airflow 3 only | none | Conditionally required when the resolved API version is 3. The subject claim for the token (the Integer ID from Airflow's user table) |
Loading