Skip to content
Open
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
11c9b5a
Add docker compose files
sitaowang1998 Aug 6, 2026
470f7e5
Fix network
sitaowang1998 Aug 6, 2026
139286e
Add image tag and worker image name
sitaowang1998 Aug 6, 2026
bb36fec
Add clean task
sitaowang1998 Aug 6, 2026
07baeee
Fix config
sitaowang1998 Aug 6, 2026
c6d4696
Merge branch 'main' into docker-compose
sitaowang1998 Aug 7, 2026
ce11907
Rename variables
sitaowang1998 Aug 7, 2026
4c7b4c0
Merge branch 'docker-compose' of github.com:sitaowang1998/spider into…
sitaowang1998 Aug 7, 2026
7f2d8f3
Merge branch 'main' into docker-compose
sitaowang1998 Aug 7, 2026
6b92262
Merge branch 'main' into docker-compose
sitaowang1998 Aug 11, 2026
bc35203
Expose user-facing config through env file
sitaowang1998 Aug 11, 2026
1bf2687
Merge branch 'docker-compose' of github.com:sitaowang1998/spider into…
sitaowang1998 Aug 11, 2026
d3dbf06
Fix generated storage config keys
sitaowang1998 Aug 11, 2026
0d6b169
Merge branch 'main' into docker-compose
sitaowang1998 Aug 11, 2026
b3eed39
Add defaults for Compose configuration
sitaowang1998 Aug 11, 2026
98ebc3b
Merge remote-tracking branch 'origin/main' into docker-compose
sitaowang1998 Aug 12, 2026
07f764e
Update Compose runtime configuration
sitaowang1998 Aug 12, 2026
ab5a157
Split configs out
sitaowang1998 Aug 12, 2026
d5c1a02
Merge branch 'main' into docker-compose
sitaowang1998 Aug 12, 2026
7807436
Rename env
sitaowang1998 Aug 12, 2026
95f30cf
Publish docker compose as oci artifact
sitaowang1998 Aug 13, 2026
0ccb692
Add 5 min wait timeout for docker compose up
sitaowang1998 Aug 13, 2026
c65ae75
Merge branch 'docker-compose' into docker-compose-oci
sitaowang1998 Aug 13, 2026
09ec079
Change artifact name
sitaowang1998 Aug 13, 2026
4916186
Merge branch 'main' into docker-compose-oci
sitaowang1998 Aug 13, 2026
9729ac7
Only push to tag on tag creation
sitaowang1998 Aug 13, 2026
152ab8e
Remove media type version
sitaowang1998 Aug 14, 2026
43ede1f
Apply suggestions from code review
sitaowang1998 Aug 14, 2026
fba8922
Merge branch 'docker-compose-oci' of github.com:sitaowang1998/spider …
sitaowang1998 Aug 14, 2026
a0dbcb5
Add go task install version; Fix bug
sitaowang1998 Aug 14, 2026
3c8f3b0
Add workflow trigger every night
sitaowang1998 Aug 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions .github/workflows/spider-compose.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
name: "spider-compose"

on:
pull_request:
Comment thread
20001020ycx marked this conversation as resolved.
paths: &monitored_paths
- ".github/workflows/spider-compose.yaml"
- "taskfile.yaml"
- "taskfiles/docker.yaml"
- "tools/deployment/spider-compose/**"
- "tools/yscope-dev-utils"
push:
branches: ["main"]
paths: *monitored_paths
tags: ["spider-huntsman-v*"]
workflow_dispatch:

permissions: {}

concurrency:
group: "${{github.workflow}}-${{github.ref}}"

# Cancel in-progress jobs for efficiency. Exclude the `main` branch to allow uninterrupted
# publishing of Docker Compose OCI artifacts.
cancel-in-progress: "${{github.ref != 'refs/heads/main'}}"

jobs:
validate:
runs-on: "ubuntu-latest"
steps:
- uses: "actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd" # v6.0.2
with:
submodules: "recursive"

- uses: "./tools/yscope-dev-utils/exports/github/actions/install-go-task"
Comment thread
sitaowang1998 marked this conversation as resolved.

- name: "Validate Docker Compose configuration"
shell: "bash"
run: "task docker:compose:validate"

publish:
if: "github.event_name != 'pull_request'"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if: "github.event_name != 'pull_request'"
# Publish from `main` and `spider-huntsman-vA.B.C` release branches.
if: >-
github.event_name != 'pull_request'
&& (github.ref == 'refs/heads/main'
|| startsWith(github.ref, 'refs/heads/spider-huntsman-v'))

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use tag instead of branch for release.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so say we have a tag called spider-huntsman-v0.0.1, does this workflow gets triggered?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Adding a tag triggers a push. Check https://github.com/sitaowang1998/spider/pkgs/container/spider%2Fcompose. I named the tag spider-huntsman-v0.0.1-test, and it triggers the workflow https://github.com/sitaowang1998/spider/actions/runs/31924593882/job/95110055471.

needs: "validate"
runs-on: "ubuntu-latest"
permissions:
contents: "read"
packages: "write"
steps:
- uses: "actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd" # v6.0.2
Comment thread
sitaowang1998 marked this conversation as resolved.
Outdated
with:
submodules: "recursive"

- uses: "./tools/yscope-dev-utils/exports/github/actions/install-go-task"
Comment thread
20001020ycx marked this conversation as resolved.

- uses: "oras-project/setup-oras@1d808f7d7f6995cc68b7bf507bfe5c5446e1dc9d" # v2.0.1
with:
version: "1.3.3"

- name: "Sanitize repository name"
id: "repo"
run: >-
echo "name=${GITHUB_REPOSITORY,,}" >> "$GITHUB_OUTPUT"

- name: "Get artifact tag"
id: "artifact"
shell: "bash"
run: |-
tag="nightly"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, are we doing nightly? if so should we add something like
schedule:
# Run daily at 00:15 UTC (the 15 is to avoid periods of high load)
- cron: "15 0 * * *"
in the triggering section of this workflow file?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see the need to trigger publication everyday. The nightly just means this is not a release.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

then I guess my question is why tagged with nightly. I don't think nightly just means this is not a release, It means a job runs once every 24 hours, usually overnight when server traffic and system usage are low, according to google.

Could you clarify for me how this term is defined in spider. In fact, Spider seldom uses this terminology - three instances only, but I do saw other worfklow used - cron: "15 0 * * *", that's why I suggested at the first place. Anyway, I dont see the need to trigger it everyday as well, just want to understand what this tag means.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added cron workflow trigger.

if [[ "${GITHUB_REF_TYPE}" == "tag" ]]; then
tag="${GITHUB_REF_NAME}"
fi
Comment thread
sitaowang1998 marked this conversation as resolved.
echo "tag=${tag}" >> "$GITHUB_OUTPUT"

- uses: "docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121" # v4.1.0
with:
registry: "ghcr.io"
username: "${{github.actor}}"
password: "${{secrets.GITHUB_TOKEN}}"

- name: "Publish Docker Compose OCI artifact"
shell: "bash"
run: >-
task docker:compose:publish
OCI_REFERENCE='ghcr.io/${{steps.repo.outputs.name}}/compose:${{steps.artifact.outputs.tag}}'
16 changes: 16 additions & 0 deletions taskfiles/docker.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,22 @@ tasks:
dir: "{{.G_SPIDER_COMPOSE_DIR}}"
cmd: "docker compose --env-file .env.example --file compose.yaml config --quiet"

# Publishes the complete Docker Compose deployment directory as an OCI artifact.
#
# @param {string} OCI_REFERENCE The destination OCI reference, including its tag.
Comment thread
sitaowang1998 marked this conversation as resolved.
Outdated
compose:publish:
requires:
vars: ["OCI_REFERENCE"]
deps: ["compose:validate"]
dir: "{{.G_SPIDER_COMPOSE_DIR}}/.."
cmd: >-
oras push
--artifact-type "application/vnd.yscope.spider.compose.v1"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we using v1? I thought we have moved on towards v2, docker compose was v2's syntax, docker-compose was v1's syntax.

@sitaowang1998 sitaowang1998 Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has nothing to do with docker compose version. It is a version we define ourselves. I'll remove it to avoid confusion.

'{{.OCI_REFERENCE}}'
"spider-compose/compose.yaml:application/yaml"
"spider-compose/configs/:application/vnd.yscope.spider.compose.configs.v1.tar"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto

"spider-compose/.env.example:text/plain"

compose:up:
dir: "{{.G_SPIDER_COMPOSE_DIR}}"
cmd: "docker compose --file compose.yaml up --detach --wait --wait-timeout 300"
Expand Down
Loading