Skip to content

Commit a4c7944

Browse files
liferoadliferoad
authored
Add PreCommit Java Dataflow workflow (#38094) (#38124)
* Add PreCommit Java Dataflow workflow (#38094) Split Dataflow worker tests into a separate PreCommit job to reduce Java PreCommit runtime. The StreamingDataflowWorkerTest suite takes >1h, causing the main Java PreCommit to exceed typical CI timeouts. New workflow: beam_PreCommit_Java_Dataflow - Runs only on changes to runners/google-cloud-dataflow-java/worker/** - Triggers on push, PR, schedule (every 6h), workflow_dispatch, and issue comment 'Run Java Dataflow PreCommit' - Runs :runners:google-cloud-dataflow-java:worker:test - Timeout: 240 minutes Fixes #38094 * Add release/trigger_all_tests.json to trigger path per GHA validation * Exclude Dataflow worker from main Java PreCommit (#38094) Add !runners/google-cloud-dataflow-java/worker/** to both push and pull_request_target path filters in beam_PreCommit_Java.yml. Without this, the main Java PreCommit still runs the full javaPreCommit (including slow StreamingDataflowWorkerTest) on worker changes, which defeats the purpose of splitting to a separate PreCommit job. * Add beam_PreCommit_Java_Dataflow.yml to workflows/README.md * Address PR review comments: rename, ubuntu-24.04, drop trigger file - Rename workflow to 'PreCommit Java Dataflow Non-portable Worker' - Update job_name and job_phrase to match - Change runs-on from ubuntu-20.04 to ubuntu-24.04 (20.04 pool being shutdown) - Remove trigger file reference (not needed per @Abacn) - Update workflows/README.md entry * Restore trigger file path reference in pull_request_target * Add release/trigger_all_tests.json to pull_request_target trigger path * Simplify gradle-command to 'build' with -p flag per Abacn review * Remove worker/windmill from javaPreCommit, add to new PreCommit per Abacn review
1 parent efd2abf commit a4c7944

4 files changed

Lines changed: 141 additions & 2 deletions

File tree

‎.github/workflows/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -228,6 +228,7 @@ PreCommit Jobs run in a schedule and also get triggered in a PR if relevant sour
228228
| [ PreCommit Go ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Go.yml) | N/A |`Run Go PreCommit`| [![.github/workflows/beam_PreCommit_Go.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Go.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Go.yml?query=event%3Aschedule) |
229229
| [ PreCommit GoPortable ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_GoPortable.yml) | N/A |`Run GoPortable PreCommit`| [![.github/workflows/beam_PreCommit_GoPortable.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_GoPortable.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_GoPortable.yml?query=event%3Aschedule) |
230230
| [ PreCommit Java ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java.yml) | N/A |`Run Java PreCommit`| [![.github/workflows/beam_PreCommit_Java.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java.yml?query=event%3Aschedule) |
231+
| [ PreCommit Java Dataflow Non-portable Worker ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Dataflow.yml) | N/A |`Run Java Dataflow Non-portable Worker PreCommit`| N/A |
231232
| [ PreCommit Java Amazon Web Services2 IO Direct ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Amazon-Web-Services2_IO_Direct.yml) | N/A |`Run Java_Amazon-Web-Services2_IO_Direct PreCommit`| [![.github/workflows/beam_PreCommit_Java_Amazon-Web-Services2_IO_Direct.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Amazon-Web-Services2_IO_Direct.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Amazon-Web-Services2_IO_Direct.yml?query=event%3Aschedule) |
232233
| [ PreCommit Java Amqp IO Direct ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Amqp_IO_Direct.yml) | N/A |`Run Java_Amqp_IO_Direct PreCommit`| [![.github/workflows/beam_PreCommit_Java_Amqp_IO_Direct.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Amqp_IO_Direct.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Amqp_IO_Direct.yml?query=event%3Aschedule) |
233234
| [ PreCommit Java Azure IO Direct ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Azure_IO_Direct.yml) | N/A |`Run Java_Azure_IO_Direct PreCommit`| [![.github/workflows/beam_PreCommit_Java_Azure_IO_Direct.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Azure_IO_Direct.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Azure_IO_Direct.yml?query=event%3Aschedule) |

‎.github/workflows/beam_PreCommit_Java.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ on:
2323
- 'model/**'
2424
- 'sdks/java/**'
2525
- 'runners/**'
26+
- '!runners/google-cloud-dataflow-java/worker/**'
2627
- 'examples/java/**'
2728
- 'examples/kotlin/**'
2829
- 'release/**'
@@ -74,6 +75,7 @@ on:
7475
- 'model/**'
7576
- 'sdks/java/**'
7677
- 'runners/**'
78+
- '!runners/google-cloud-dataflow-java/worker/**'
7779
- 'examples/java/**'
7880
- 'examples/kotlin/**'
7981
- 'release/**'

‎.github/workflows/beam_PreCommit_Java_Dataflow.yml‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or or in writing, software
12+
# distributed under the License is distributed on an "AS IS"
13+
# BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
14+
# express or implied. See the License for the specific language
15+
# governing permissions and limitations under the License.
16+
17+
name: PreCommit Java Dataflow Non-portable Worker
18+
19+
on:
20+
push:
21+
tags: ['v*']
22+
branches: ['master', 'release-*']
23+
paths:
24+
- 'runners/google-cloud-dataflow-java/worker/**'
25+
- '.github/workflows/beam_PreCommit_Java_Dataflow.yml'
26+
pull_request_target:
27+
branches: ['master', 'release-*']
28+
paths:
29+
- 'runners/google-cloud-dataflow-java/worker/**'
30+
- '.github/trigger_files/beam_PreCommit_Java_Dataflow.json'
31+
- '.github/workflows/beam_PreCommit_Java_Dataflow.yml'
32+
- 'release/trigger_all_tests.json'
33+
issue_comment:
34+
types: [created]
35+
schedule:
36+
- cron: '30 2/6 * * *'
37+
workflow_dispatch:
38+
39+
concurrency:
40+
group: '${{ github.workflow }} @ ${{ github.event.pull_request.number || github.event.pull_request.head.label || github.sha || github.head_ref || github.ref }}-${{ github.event.schedule || github.event.comment.id || github.event.sender.login }}'
41+
cancel-in-progress: true
42+
43+
permissions:
44+
actions: write
45+
pull-requests: write
46+
checks: write
47+
contents: read
48+
deployments: read
49+
id-token: none
50+
issues: write
51+
discussions: read
52+
packages: read
53+
pages: read
54+
repository-projects: read
55+
security-events: read
56+
statuses: read
57+
58+
env:
59+
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
60+
GRADLE_ENTERPRISE_CACHE_USERNAME: ${{ secrets.GE_CACHE_USERNAME }}
61+
GRADLE_ENTERPRISE_CACHE_PASSWORD: ${{ secrets.GE_CACHE_PASSWORD }}
62+
63+
jobs:
64+
beam_PreCommit_Java_Dataflow:
65+
name: ${{ matrix.job_name }} (${{ matrix.job_phrase }})
66+
runs-on: [self-hosted, ubuntu-24.04, main]
67+
strategy:
68+
fail-fast: false
69+
matrix:
70+
job_name: [beam_PreCommit_Java_Dataflow_NonportableWorker]
71+
job_phrase: [Run Java Dataflow Non-portable Worker PreCommit]
72+
timeout-minutes: 240
73+
if: |
74+
github.event_name == 'push' ||
75+
github.event_name == 'pull_request_target' ||
76+
(github.event_name == 'schedule' && github.repository == 'apache/beam') ||
77+
github.event_name == 'workflow_dispatch' ||
78+
github.event.comment.body == 'Run Java Dataflow Non-portable Worker PreCommit'
79+
steps:
80+
- uses: actions/checkout@v4
81+
- name: Setup repository
82+
uses: ./.github/actions/setup-action
83+
with:
84+
comment_phrase: ${{ matrix.job_phrase }}
85+
github_token: ${{ secrets.GITHUB_TOKEN }}
86+
github_job: ${{ matrix.job_name }} (${{ matrix.job_phrase }})
87+
- name: Setup environment
88+
uses: ./.github/actions/setup-environment-action
89+
with:
90+
python-version: default
91+
disable-cache: true
92+
- name: Run Dataflow Worker tests
93+
uses: ./.github/actions/gradle-command-self-hosted-action
94+
with:
95+
gradle-command: build
96+
arguments: |
97+
-p runners/google-cloud-dataflow-java/worker
98+
-PdisableSpotlessCheck=true \
99+
-PdisableCheckStyle=true \
100+
-PenableJacocoReport \
101+
:runners:google-cloud-dataflow-java:worker:build \
102+
:runners:google-cloud-dataflow-java:worker:windmill:build \
103+
:runners:google-cloud-dataflow-java:worker:test
104+
- name: Archive JUnit Test Results
105+
uses: actions/upload-artifact@v4
106+
if: ${{ !success() }}
107+
with:
108+
name: JUnit Test Results
109+
path: "**/build/reports/tests/"
110+
- name: Publish JUnit Test Results
111+
uses: EnricoMi/publish-unit-test-result-action@v2
112+
if: always()
113+
with:
114+
commit: '${{ env.prsha || env.GITHUB_SHA }}'
115+
comment_mode: ${{ github.event_name == 'issue_comment' && 'always' || 'off' }}
116+
files: '**/build/test-results/**/*.xml'
117+
large_files: true
118+
- name: Archive SpotBugs Results
119+
uses: actions/upload-artifact@v4
120+
if: always()
121+
with:
122+
name: SpotBugs Results
123+
path: '**/build/reports/spotbugs/*.html'
124+
- name: Publish SpotBugs Results
125+
uses: jwgmeligmeyling/spotbugs-github-action@v1.2
126+
if: always()
127+
with:
128+
name: Publish SpotBugs
129+
path: '**/build/reports/spotbugs/*.html'
130+
- name: Archive Jacoco Results
131+
uses: actions/upload-artifact@v4
132+
with:
133+
name: Jacoco Results
134+
path: '**/build/jacoco/report/**'
135+
- name: Upload coverage reports to Codecov
136+
uses: codecov/codecov-action@v3
137+
with:
138+
flags: java

‎build.gradle.kts‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -275,8 +275,6 @@ tasks.register("javaPreCommit") {
275275
dependsOn(":runners:google-cloud-dataflow-java:build")
276276
dependsOn(":runners:google-cloud-dataflow-java:examples-streaming:build")
277277
dependsOn(":runners:google-cloud-dataflow-java:examples:build")
278-
dependsOn(":runners:google-cloud-dataflow-java:worker:build")
279-
dependsOn(":runners:google-cloud-dataflow-java:worker:windmill:build")
280278
dependsOn(":runners:java-fn-execution:build")
281279
dependsOn(":runners:java-job-service:build")
282280
dependsOn(":runners:jet:build")

0 commit comments

Comments
 (0)