Skip to content

Commit 0ee2dc7

Browse files
merge master into fix_comment_trigger_against_pr_branch
2 parents eac9d19 + 42cdd61 commit 0ee2dc7

115 files changed

Lines changed: 4460 additions & 360 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/REVIEWERS.yml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,6 @@ labels:
4949
- Abacn
5050
- ahmedabu98
5151
- bvolpato
52-
- manavgarg
5352
exclusionList: []
5453
- name: spanner
5554
reviewers:

.github/workflows/README.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -117,6 +117,7 @@ Please note that jobs with matrix need to have matrix element in the comment. Ex
117117
| Workflow name | Matrix | Trigger Phrase | Cron Status |
118118
|:-------------:|:------:|:--------------:|:-----------:|
119119
| [ PostCommit Go Dataflow ARM](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Go_Dataflow_ARM.yml) | N/A |`Run Go PostCommit Dataflow ARM`| [![.github/workflows/beam_PostCommit_Go_Dataflow_ARM](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Go_Dataflow_ARM.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Go_Dataflow_ARM.yml) |
120+
| [ PostCommit Java Examples Dataflow ARM ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Examples_Dataflow_ARM.yml) | N/A |`Run Java_Examples_Dataflow_ARM PostCommit`| [![PostCommit Java Examples Dataflow ARM](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Examples_Dataflow_ARM.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Examples_Dataflow_ARM.yml) |
120121
| [ PreCommit Community Metrics ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_CommunityMetrics.yml) | N/A |`Run CommunityMetrics PreCommit`| [![.github/workflows/beam_PreCommit_CommunityMetrics.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_CommunityMetrics.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_CommunityMetrics.yml) |
121122
| [ PreCommit Go ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Go.yml) | N/A |`Run Go PreCommit`| [![.github/workflows/beam_PreCommit_Go.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Go.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Go.yml) |
122123
| [ PreCommit Java Examples Dataflow ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Examples_Dataflow.yml) | N/A |`Run Java_Examples_Dataflow PreCommit`| [![.github/workflows/beam_PreCommit_Java_Examples_Dataflow.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Examples_Dataflow.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Examples_Dataflow.yml) |
@@ -137,4 +138,5 @@ Please note that jobs with matrix need to have matrix element in the comment. Ex
137138
| [ PreCommit Website ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website.yml) | N/A |`Run Website PreCommit`| [![.github/workflows/beam_PreCommit_Website.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website.yml) |
138139
| [ PreCommit Website Stage GCS ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website_Stage_GCS.yml) | N/A |`Run Website_Stage_GCS PreCommit`| [![PreCommit Website Stage GCS](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website_Stage_GCS.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website_Stage_GCS.yml) |
139140
| [ PreCommit Whitespace ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Whitespace.yml) | N/A |`Run Whitespace PreCommit`| [![.github/workflows/beam_PreCommit_Whitespace.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Whitespace.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Whitespace.yml) |
140-
| [ Python Validates Container Dataflow ARM ](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml) | ['3.8','3.9','3.10','3.11'] | `Run Python Validates Container Dataflow ARM (matrix_element)`| [![.github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml) |
141+
| [ Python Validates Container Dataflow ARM ](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml) | ['3.8','3.9','3.10','3.11'] | `Run Python ValidatesContainer Dataflow ARM (matrix_element)`| [![.github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml) |
142+
| [ PreCommit GoPortable ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_GoPortable.yml) | N/A |`Run GoPortable PreCommit`| [![.github/workflows/beam_PreCommit_GoPortable.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_GoPortable.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_GoPortable.yml) |

.github/workflows/beam_PostCommit_Go_Dataflow_ARM.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ permissions:
4444

4545
# This allows a subsequently queued workflow run to interrupt previous runs
4646
concurrency:
47-
group: '${{ github.workflow }} @ ${{ github.event.pull_request.head.label || github.head_ref || github.ref }}'
47+
group: '${{ github.workflow }} @ ${{ github.event.pull_request.head.label || github.sha || github.head_ref || github.ref }}-${{ github.event.sender.login }}-${{ github.event.schedule }}'
4848
cancel-in-progress: true
4949

5050
jobs:
Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
name: PostCommit Java Examples Dataflow ARM
19+
20+
on:
21+
push:
22+
tags: ['v*']
23+
branches: ['master', 'release-*']
24+
paths:
25+
- 'model/**'
26+
- 'sdks/java/**'
27+
- 'runners/google-cloud-dataflow-java/**'
28+
- 'examples/java/**'
29+
- 'examples/kotlin/**'
30+
- 'release/**'
31+
- '.github/workflows/beam_PostCommit_Java_Examples_Dataflow_ARM.yml'
32+
issue_comment:
33+
types: [created]
34+
schedule:
35+
- cron: '0 */6 * * *'
36+
workflow_dispatch:
37+
38+
# This allows a subsequently queued workflow run to interrupt previous runs
39+
concurrency:
40+
group: '${{ github.workflow }} @ ${{ github.event.pull_request.head.label || github.sha || github.head_ref || github.ref }}-${{ github.event.sender.login }}-${{ github.event.schedule }}'
41+
cancel-in-progress: true
42+
43+
#Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event
44+
permissions:
45+
actions: write
46+
pull-requests: read
47+
checks: read
48+
contents: read
49+
deployments: read
50+
id-token: none
51+
issues: read
52+
discussions: read
53+
packages: read
54+
pages: read
55+
repository-projects: read
56+
security-events: read
57+
statuses: read
58+
59+
env:
60+
TAG: $(date +'%Y%m%d-%H%M%S%N')
61+
62+
jobs:
63+
beam_PostCommit_Java_Examples__Dataflow_ARM:
64+
name: beam_PostCommit_Java_Examples__Dataflow_ARM (Run Java_Examples_Dataflow_ARM PostCommit)
65+
runs-on: [self-hosted, ubuntu-20.04, main]
66+
strategy:
67+
fail-fast: false
68+
if: |
69+
github.event_name == 'push' ||
70+
github.event_name == 'schedule' ||
71+
github.event.comment.body == 'Run Java_Examples_Dataflow_ARM PostCommit' ||
72+
github. event_name == 'workflow_dispatch'
73+
steps:
74+
- name: Check out repository code
75+
uses: actions/checkout@v3
76+
with:
77+
ref: ${{ github.event.pull_request.head.sha }}
78+
- name: Rerun on comment
79+
if: github.event.comment.body == 'Run Java_Examples_Dataflow_ARM PostCommit'
80+
uses: ./.github/actions/rerun-job-action
81+
with:
82+
pull_request_url: ${{ github.event.issue.pull_request.url }}
83+
github_repository: ${{ github.repository }}
84+
github_token: ${{ secrets.GITHUB_TOKEN }}
85+
github_job: "beam_PostCommit_Java_Examples__Dataflow_ARM (Run Java_Examples_Dataflow_ARM PostCommit)"
86+
github_current_run_id: ${{ github.run_id }}
87+
- name: Setup self-hosted
88+
uses: ./.github/actions/setup-self-hosted-action
89+
with:
90+
requires-py-38: false
91+
requires-py-39: false
92+
requires-go: false
93+
- name: Setup Gradle
94+
uses: gradle/gradle-build-action@v2
95+
with:
96+
cache-read-only: false
97+
- name: Set up Docker Buildx
98+
uses: docker/setup-buildx-action@v2
99+
- name: Authenticate on GCP
100+
id: auth
101+
uses: google-github-actions/setup-gcloud@v0
102+
with:
103+
service_account_email: ${{ secrets.GCP_SA_EMAIL }}
104+
service_account_key: ${{ secrets.GCP_SA_KEY }}
105+
project_id: ${{ secrets.GCP_PROJECT_ID }}
106+
export_default_credentials: true
107+
- name: GCloud Docker credential helper
108+
run: |
109+
gcloud auth configure-docker us.gcr.io
110+
- name: run javaExamplesDataflowARMPostCommit script for Java8
111+
uses: ./.github/actions/gradle-command-self-hosted-action
112+
with:
113+
gradle-command: :runners:google-cloud-dataflow-java:arm:examplesJavaRunnerV2IntegrationTestARM
114+
arguments: |
115+
-Pcontainer-architecture-list=arm64,amd64 \
116+
-Ppush-containers \
117+
-Pdocker-repository-root=us.gcr.io/apache-beam-testing/java-postcommit-it/arm \
118+
-Pdocker-tag=${{env.TAG}} \
119+
- name: run javaExamplesDataflowARMPostCommit script for Java11
120+
uses: ./.github/actions/gradle-command-self-hosted-action
121+
with:
122+
gradle-command: :runners:google-cloud-dataflow-java:arm:examplesJavaRunnerV2IntegrationTestARM
123+
arguments: |
124+
-PcompileAndRunTestsWithJava11 \
125+
-Pcontainer-architecture-list=arm64,amd64 \
126+
-Ppush-containers \
127+
-Pdocker-repository-root=us.gcr.io/apache-beam-testing/java-postcommit-it/arm \
128+
-Pdocker-tag=${{env.TAG}} \
129+
- name: run javaExamplesDataflowARMPostCommit script for Java17
130+
uses: ./.github/actions/gradle-command-self-hosted-action
131+
with:
132+
gradle-command: :runners:google-cloud-dataflow-java:arm:examplesJavaRunnerV2IntegrationTestARM
133+
arguments: |
134+
-PcompileAndRunTestsWithJava17 \
135+
-Pcontainer-architecture-list=arm64,amd64 \
136+
-Ppush-containers \
137+
-Pdocker-repository-root=us.gcr.io/apache-beam-testing/java-postcommit-it/arm \
138+
-Pdocker-tag=${{env.TAG}} \
139+
- name: Upload test report
140+
uses: actions/upload-artifact@v3
141+
with:
142+
name: java-code-coverage-report
143+
path: "**/build/test-results/**/*.xml"
Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
name: PreCommit GoPortable
17+
18+
on:
19+
push:
20+
tags: ['v*']
21+
branches: ['master', 'release-*']
22+
paths: ['model/**', 'sdks/go.**', 'release/**','.github/workflows/beam_PreCommit_GoPortable.yml']
23+
pull_request_target:
24+
branches: ['master', 'release-*']
25+
paths: ['model/**', 'sdks/go.**', 'release/**']
26+
issue_comment:
27+
types: [created]
28+
schedule:
29+
- cron: '0 */6 * * *'
30+
workflow_dispatch:
31+
32+
# This allows a subsequently queued workflow run to interrupt previous runs
33+
concurrency:
34+
group: '${{ github.workflow }} @ ${{ github.event.pull_request.head.label || github.sha || github.head_ref || github.ref }}-${{ github.event.sender.login }}-${{ github.event.schedule }}'
35+
cancel-in-progress: true
36+
37+
# Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event
38+
permissions:
39+
actions: write
40+
pull-requests: read
41+
checks: read
42+
contents: read
43+
deployments: read
44+
id-token: none
45+
issues: read
46+
discussions: read
47+
packages: read
48+
pages: read
49+
repository-projects: read
50+
security-events: read
51+
statuses: read
52+
53+
jobs:
54+
beam_PreCommit_GoPortable:
55+
timeout-minutes: 120
56+
if: |
57+
github.event_name == 'push' ||
58+
github.event_name == 'pull_request_target' ||
59+
github.event.comment.body == 'Run GoPortable PreCommit' ||
60+
github.event_name == 'schedule'
61+
runs-on: [self-hosted, ubuntu-20.04, main]
62+
name: beam_PreCommit_GoPortable (Run GoPortable PreCommit)
63+
steps:
64+
- name: Git checkout
65+
uses: actions/checkout@v3
66+
with:
67+
ref: ${{ github.event.pull_request.head.sha }}
68+
- name: Rerun on comment
69+
if: github.event.comment.body == 'Run GoPortable PreCommit'
70+
uses: ./.github/actions/rerun-job-action
71+
with:
72+
pull_request_url: ${{ github.event.issue.pull_request.url }}
73+
github_repository: ${{ github.repository }}
74+
github_token: ${{ secrets.GITHUB_TOKEN }}
75+
github_job: ${{ github.job }}
76+
github_current_run_id: ${{ github.run_id }}
77+
- name: Setup self-hosted
78+
uses: ./.github/actions/setup-self-hosted-action
79+
with:
80+
requires-py-39: false
81+
requires-go: false
82+
- name: Run goPortablePreCommit script
83+
uses: ./.github/actions/gradle-command-self-hosted-action
84+
with:
85+
gradle-command: :goPortablePreCommit

.github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -44,20 +44,21 @@ permissions:
4444

4545
# This allows a subsequently queued workflow run to interrupt previous runs
4646
concurrency:
47-
group: '${{ github.workflow }} @ ${{ github.event.pull_request.head.label || github.head_ref || github.ref }}'
47+
group: '${{ github.workflow }} @ ${{ github.event.pull_request.head.label || github.sha || github.head_ref || github.ref }}-${{ github.event.sender.login }}-${{ github.event.schedule }}'
4848
cancel-in-progress: true
4949

5050
jobs:
5151
beam_Python_ValidatesContainer_Dataflow_ARM:
52-
name: beam_Python_ValidatesContainer_Dataflow_ARM
52+
name: beam_Python_ValidatesContainer_Dataflow_ARM (Run Python ValidatesContainer Dataflow ARM 3.x)
5353
strategy:
5454
fail-fast: false
5555
matrix:
5656
python_version: ['3.8','3.9','3.10','3.11']
5757
if: |
5858
github.event_name == 'push' ||
5959
startsWith(github.event.comment.body, 'Run Python ValidatesContainer Dataflow ARM') ||
60-
github.event_name == 'schedule'
60+
github.event_name == 'schedule' ||
61+
github.event_name == 'workflow_dispatch'
6162
runs-on: [self-hosted, ubuntu-20.04, main]
6263
env:
6364
MULTIARCH_TAG: $(date +'%Y%m%d-%H%M%S%N')
@@ -72,7 +73,8 @@ jobs:
7273
with:
7374
comment_phrase: ${{ steps.set_comment_body.outputs.comment_body }}
7475
github_token: ${{ secrets.GITHUB_TOKEN }}
75-
github_job: "${{ github.job }} (${{ matrix.python_version }})"
76+
github_job: "beam_Python_ValidatesContainer_Dataflow_ARM (Run Python ValidatesContainer Dataflow ARM 3.x) (${{ matrix.python_version }})"
77+
github_current_run_id: ${{ github.run_id }}
7678
- name: Install Python
7779
uses: actions/setup-python@v4
7880
with:
@@ -101,19 +103,20 @@ jobs:
101103
echo "py_ver_clean=$PY_VER_CLEAN" >> $GITHUB_OUTPUT
102104
- name: run Python validatesContainerARM script
103105
id: run_script
104-
run: |
106+
uses: ./.github/actions/gradle-command-self-hosted-action
107+
with:
105108
gradle-command: :sdks:python:test-suites:dataflow:py${{steps.set_py_ver_clean.outputs.py_ver_clean}}:validatesContainerARM
106109
arguments: |
107110
-PpythonVersion=${{ matrix.python_version }} \
108111
-Pcontainer-architecture-list=arm64,amd64 \
109112
-Pdocker-repository=us.gcr.io/apache-beam-testing/github-actions \
110113
-Pdocker-tag=${{ env.MULTIARCH_TAG }} \
111-
-Ppush-containers
112-
env:
113-
MULTIARCH_TAG: ${{ env.MULTIARCH_TAG }}
114-
USER: github-actions
114+
-Ppush-containers \
115+
env:
116+
MULTIARCH_TAG: ${{ env.MULTIARCH_TAG }}
117+
USER: github-actions
115118
- name: Archive code coverage results
116119
uses: actions/upload-artifact@v3
117120
with:
118121
name: python-code-coverage-report
119-
path: "**/pytest*.xml"
122+
path: "**/pytest*.xml"

.test-infra/tools/stale_dataflow_prebuilt_image_cleaner.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ set -euo pipefail
2323
# Images more than 5 day old and not the latest (either has latest label or newest)
2424

2525
PUBLIC_REPOSITORIES=(beam-sdk beam_portability)
26-
PRIVATE_REPOSITORIES=(java-postcommit-it python-postcommit-it jenkins)
26+
PRIVATE_REPOSITORIES=(java-postcommit-it python-postcommit-it jenkins github-actions)
2727
# set as the same as 6-week release period
2828
DELETE_BEFORE_DAY=$(date --iso-8601=s -d '6 weeks ago')
2929

0 commit comments

Comments
 (0)