Skip to content

chore(hadoop): switch to hadoop shaded client #2767

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wForget wants to merge 5 commits into
base: master
Choose a base branch
from
Open

chore(hadoop): switch to hadoop shaded client #2767

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3ee9cb7
chore(test): switch to hadoop-client-minicluster for Hadoop 3
wForget Jul 16, 2026
1c7bf02
fix
wForget Jul 17, 2026
6e344db
fix
wForget Jul 17, 2026
63ffab9
fix
wForget Jul 17, 2026
31afd36
fix
wForget Jul 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion client-mr/core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<artifactId>${hadoop-client-api.artifactId}</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion>
Expand Down
2 changes: 1 addition & 1 deletion client-mr/hadoop2.8/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<artifactId>${hadoop-client-api.artifactId}</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion>
Expand Down
2 changes: 1 addition & 1 deletion client-mr/hadoop3.2/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<artifactId>${hadoop-client-api.artifactId}</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion>
Expand Down
2 changes: 1 addition & 1 deletion client-spark/common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<artifactId>${hadoop-minicluster.artifactId}</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
Expand Down
2 changes: 1 addition & 1 deletion client-spark/spark2/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<artifactId>${hadoop-minicluster.artifactId}</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
Expand Down
2 changes: 1 addition & 1 deletion client-spark/spark3/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<artifactId>${hadoop-minicluster.artifactId}</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
Expand Down
2 changes: 1 addition & 1 deletion client-spark/spark4/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<artifactId>${hadoop-minicluster.artifactId}</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
Expand Down
4 changes: 2 additions & 2 deletions client-tez/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<artifactId>${hadoop-client-api.artifactId}</artifactId>
<exclusions>
<exclusion>
<groupId>com.sun.jersey</groupId>
Expand Down Expand Up @@ -130,7 +130,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<artifactId>${hadoop-minicluster.artifactId}</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
Expand Down
9 changes: 7 additions & 2 deletions client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,14 @@
<artifactId>mockito-inline</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<scope>test</scope>

@LuciferYang LuciferYang Jul 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test-scoped commons-io added to the client module isn't referenced anywhere under client/src (neither main nor test). Is it covering a transitive test dependency after the hadoop artifact switch, or just added along the way? Drop it if it's not needed; add a one-line comment if it is.

</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<artifactId>${hadoop-client-api.artifactId}</artifactId>
<exclusions>
<exclusion>
<groupId>com.sun.jersey</groupId>
Expand All @@ -68,7 +73,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<artifactId>${hadoop-minicluster.artifactId}</artifactId>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
Expand Down
4 changes: 2 additions & 2 deletions common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<artifactId>${hadoop-client-api.artifactId}</artifactId>
<exclusions>
<exclusion>
<groupId>com.google.protobuf</groupId>
Expand Down Expand Up @@ -136,7 +136,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<artifactId>${hadoop-minicluster.artifactId}</artifactId>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
Expand Down
6 changes: 5 additions & 1 deletion coordinator/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,10 @@
<groupId>org.apache.uniffle</groupId>
<artifactId>rss-internal-client</artifactId>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java-util</artifactId>
Expand Down Expand Up @@ -78,7 +82,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<artifactId>${hadoop-client-api.artifactId}</artifactId>
<exclusions>
<exclusion>
<groupId>com.sun.jersey</groupId>
Expand Down
2 changes: 1 addition & 1 deletion integration-test/common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<artifactId>${hadoop-minicluster.artifactId}</artifactId>
<exclusions>
<exclusion>
<groupId>com.sun.jersey</groupId>
Expand Down
4 changes: 2 additions & 2 deletions integration-test/mr/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<artifactId>${hadoop-minicluster.artifactId}</artifactId>
<version>${hadoop.version}</version>
<scope>test</scope>
<exclusions>
Expand Down Expand Up @@ -107,7 +107,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<artifactId>${hadoop-client-api.artifactId}</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion>
Expand Down
2 changes: 1 addition & 1 deletion integration-test/spark-common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<artifactId>${hadoop-minicluster.artifactId}</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
Expand Down
2 changes: 1 addition & 1 deletion integration-test/spark2/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<artifactId>${hadoop-minicluster.artifactId}</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
Expand Down
2 changes: 1 addition & 1 deletion integration-test/spark3/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<artifactId>${hadoop-minicluster.artifactId}</artifactId>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
Expand Down
2 changes: 1 addition & 1 deletion integration-test/spark4/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<artifactId>${hadoop-minicluster.artifactId}</artifactId>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
Expand Down
4 changes: 2 additions & 2 deletions integration-test/tez/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<artifactId>${hadoop-minicluster.artifactId}</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
Expand All @@ -116,7 +116,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<artifactId>${hadoop-client-api.artifactId}</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
Expand Down
51 changes: 47 additions & 4 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@
<awaitility.version>4.2.0</awaitility.version>
<checkstyle.version>9.3</checkstyle.version>
<commons-collections4.version>4.4</commons-collections4.version>
<commons-io.version>2.16.1</commons-io.version>
<commons-logging.version>1.2</commons-logging.version>
<commons-lang3.version>3.10</commons-lang3.version>
<commons-codec.version>1.9</commons-codec.version>
Expand All @@ -69,6 +70,9 @@
<guava.version>32.1.3-jre</guava.version>
<hadoop.scope>provided</hadoop.scope>
<hadoop.version>2.8.5</hadoop.version>
<hadoop-client-api.artifactId>hadoop-common</hadoop-client-api.artifactId>
<hadoop-client-runtime.artifactId>hadoop-client</hadoop-client-runtime.artifactId>
<hadoop-minicluster.artifactId>hadoop-minicluster</hadoop-minicluster.artifactId>
<httpclient.version>4.5.3</httpclient.version>
<httpcore.version>4.4.4</httpcore.version>
<java.version>1.8</java.version>
Expand Down Expand Up @@ -339,6 +343,13 @@
<artifactId>commons-collections4</artifactId>
<version>${commons-collections4.version}</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>${commons-io.version}</version>
<!-- commonis-io introduced by hadoop-client, so mark it as provided -->
<scope>provided</scope>

@LuciferYang LuciferYang Jul 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pinning commons-io to provided in dependencyManagement breaks the standalone services (server/coordinator/storage). Their main code uses commons-io (FileUtils in LocalStorageChecker, IOUtils in coordinator), but the new declarations carry no scope and inherit provided. Since copy-dependencies only collects runtime scope, the -Phadoop-dependencies-included distribution no longer bundles commons-io. That distribution is meant to run without HADOOP_HOME, so the server throws NoClassDefFoundError: org/apache/commons/io/FileUtils at startup. CI misses it because the test classpath still has the provided jar. Declaring commons-io as <scope>${hadoop.scope}</scope> in those three modules aligns it with hadoop: provided by default, compile (bundled) under -Phadoop-dependencies-included.

</dependency>
<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
Expand Down Expand Up @@ -408,7 +419,7 @@

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<artifactId>${hadoop-client-api.artifactId}</artifactId>
<version>${hadoop.version}</version>
<scope>${hadoop.scope}</scope>
<exclusions>
Expand Down Expand Up @@ -438,7 +449,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<artifactId>${hadoop-client-runtime.artifactId}</artifactId>
<version>${hadoop.version}</version>
<scope>${hadoop.scope}</scope>
<exclusions>
Expand Down Expand Up @@ -641,7 +652,7 @@

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<artifactId>${hadoop-minicluster.artifactId}</artifactId>
<version>${hadoop.version}</version>
<scope>test</scope>
<exclusions>
Expand Down Expand Up @@ -1973,6 +1984,9 @@
<spark.version>4.0.2</spark.version>
<client.type>4</client.type>
<hadoop.version>3.4.1</hadoop.version>
<hadoop-client-api.artifactId>hadoop-client-api</hadoop-client-api.artifactId>
<hadoop-client-runtime.artifactId>hadoop-client-runtime</hadoop-client-runtime.artifactId>
<hadoop-minicluster.artifactId>hadoop-client-minicluster</hadoop-minicluster.artifactId>

@LuciferYang LuciferYang Jul 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After spark4 switches to hadoop-client-minicluster, that shaded jar has no BouncyCastle, and spark4 (unlike hadoop3.2) adds no bcprov. It passes today only because -Duniffle.test.skip.kerberos=true skips all Kerberos tests. The @DisabledIfSystemProperty disabledReason only says "MiniKdc is not compatible with JDK 17+", which hides the second reason: spark4 also lacks BouncyCastle. If someone enables these tests under spark4 or drops the flag, KerberizedHadoop will fail the HADOOP-17324 way. Noting this dependency in the spark4 profile or that comment would save the next person the trip.

<fasterxml.jackson.version>2.12.7</fasterxml.jackson.version>
<jetty.version>9.4.53.v20231009</jetty.version>
<enforcer.skip>true</enforcer.skip>
Expand Down Expand Up @@ -2083,6 +2097,16 @@
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>${hadoop-client-api.artifactId}</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>${hadoop-client-runtime.artifactId}</artifactId>
<scope>provided</scope>
</dependency>
<!-- Remove log4j-slf4j-impl dependency -->
<dependency>
<groupId>org.apache.logging.log4j</groupId>
Expand Down Expand Up @@ -2124,6 +2148,9 @@
<netty.version>4.2.7.Final</netty.version>
<client.type>4</client.type>
<hadoop.version>3.4.1</hadoop.version>
<hadoop-client-api.artifactId>hadoop-client-api</hadoop-client-api.artifactId>
<hadoop-client-runtime.artifactId>hadoop-client-runtime</hadoop-client-runtime.artifactId>
<hadoop-minicluster.artifactId>hadoop-client-minicluster</hadoop-minicluster.artifactId>
<fasterxml.jackson.version>2.12.7</fasterxml.jackson.version>
<jetty.version>9.4.53.v20231009</jetty.version>
<enforcer.skip>true</enforcer.skip>
Expand Down Expand Up @@ -2240,6 +2267,16 @@
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>${hadoop-client-api.artifactId}</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>${hadoop-client-runtime.artifactId}</artifactId>
<scope>provided</scope>
</dependency>
<!-- Remove log4j-slf4j-impl dependency -->
<dependency>
<groupId>org.apache.logging.log4j</groupId>
Expand Down Expand Up @@ -2617,7 +2654,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<artifactId>${hadoop-client-api.artifactId}</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion>
Expand Down Expand Up @@ -2687,13 +2724,19 @@
<properties>
<hadoop.version>2.8.5</hadoop.version>
<hadoop.short.version>2.8</hadoop.short.version>
<hadoop-minicluster.artifactId>hadoop-minicluster</hadoop-minicluster.artifactId>
</properties>
</profile>
<profile>
<id>hadoop3.2</id>
<properties>
<hadoop.version>3.2.1</hadoop.version>
<hadoop.short.version>3.2</hadoop.short.version>
<!--
HADOOP-17324: Hadoop 3.2.x hadoop-client-minicluster does not include the shaded
Bouncy Castle classes required by the Kerberos MiniDFS tests.
-->
<hadoop-minicluster.artifactId>hadoop-minicluster</hadoop-minicluster.artifactId>

@LuciferYang LuciferYang Jul 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description says "Hadoop 3.2 and Spark 4 profiles use hadoop-client-minicluster", which contradicts the code. The hadoop3.2 profile actually pins back to the non-shaded hadoop-minicluster, with a HADOOP-17324 comment: the 3.2.x shaded jar lacks the BouncyCastle needed by the Kerberos MiniDFS tests. The code is correct; just fix the description, e.g. "only the Spark 4 profile switches to hadoop-client-minicluster; hadoop3.2 keeps hadoop-minicluster because the shaded jar lacks BouncyCastle".

<commons-codec.version>1.11</commons-codec.version>
<metrics.version>3.2.4</metrics.version>
</properties>
Expand Down
8 changes: 6 additions & 2 deletions server/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,10 @@
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
</dependency>

<dependency>
<groupId>io.grpc</groupId>
Expand All @@ -89,7 +93,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<artifactId>${hadoop-client-api.artifactId}</artifactId>
<scope>${hadoop.scope}</scope>
<exclusions>
<exclusion>
Expand All @@ -100,7 +104,7 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<artifactId>${hadoop-minicluster.artifactId}</artifactId>
<exclusions>
<exclusion>
<groupId>com.sun.jersey</groupId>
Expand Down
Loading
Loading