chore(hadoop): switch to hadoop shaded client#2767
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #2767 +/- ##
============================================
+ Coverage 51.10% 51.75% +0.64%
- Complexity 3360 3978 +618
============================================
Files 534 600 +66
Lines 26055 33228 +7173
Branches 2442 3142 +700
============================================
+ Hits 13316 17197 +3881
- Misses 11880 14917 +3037
- Partials 859 1114 +255 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
wForget
changed the title
| <artifactId>commons-io</artifactId> | ||
| <version>${commons-io.version}</version> | ||
| <!-- commonis-io introduced by hadoop-client, so mark it as provided --> | ||
| <scope>provided</scope> |
There was a problem hiding this comment.
Pinning commons-io to provided in dependencyManagement breaks the standalone services (server/coordinator/storage). Their main code uses commons-io (FileUtils in LocalStorageChecker, IOUtils in coordinator), but the new declarations carry no scope and inherit provided. Since copy-dependencies only collects runtime scope, the -Phadoop-dependencies-included distribution no longer bundles commons-io. That distribution is meant to run without HADOOP_HOME, so the server throws NoClassDefFoundError: org/apache/commons/io/FileUtils at startup. CI misses it because the test classpath still has the provided jar. Declaring commons-io as <scope>${hadoop.scope}</scope> in those three modules aligns it with hadoop: provided by default, compile (bundled) under -Phadoop-dependencies-included.
| HADOOP-17324: Hadoop 3.2.x hadoop-client-minicluster does not include the shaded | ||
| Bouncy Castle classes required by the Kerberos MiniDFS tests. | ||
| --> | ||
| <hadoop-minicluster.artifactId>hadoop-minicluster</hadoop-minicluster.artifactId> |
There was a problem hiding this comment.
The PR description says "Hadoop 3.2 and Spark 4 profiles use hadoop-client-minicluster", which contradicts the code. The hadoop3.2 profile actually pins back to the non-shaded hadoop-minicluster, with a HADOOP-17324 comment: the 3.2.x shaded jar lacks the BouncyCastle needed by the Kerberos MiniDFS tests. The code is correct; just fix the description, e.g. "only the Spark 4 profile switches to hadoop-client-minicluster; hadoop3.2 keeps hadoop-minicluster because the shaded jar lacks BouncyCastle".
| <hadoop.version>3.4.1</hadoop.version> | ||
| <hadoop-client-api.artifactId>hadoop-client-api</hadoop-client-api.artifactId> | ||
| <hadoop-client-runtime.artifactId>hadoop-client-runtime</hadoop-client-runtime.artifactId> | ||
| <hadoop-minicluster.artifactId>hadoop-client-minicluster</hadoop-minicluster.artifactId> |
There was a problem hiding this comment.
After spark4 switches to hadoop-client-minicluster, that shaded jar has no BouncyCastle, and spark4 (unlike hadoop3.2) adds no bcprov. It passes today only because -Duniffle.test.skip.kerberos=true skips all Kerberos tests. The @DisabledIfSystemProperty disabledReason only says "MiniKdc is not compatible with JDK 17+", which hides the second reason: spark4 also lacks BouncyCastle. If someone enables these tests under spark4 or drops the flag, KerberizedHadoop will fail the HADOOP-17324 way. Noting this dependency in the spark4 profile or that comment would save the next person the trip.
| <dependency> | ||
| <groupId>commons-io</groupId> | ||
| <artifactId>commons-io</artifactId> | ||
| <scope>test</scope> |
There was a problem hiding this comment.
The test-scoped commons-io added to the client module isn't referenced anywhere under client/src (neither main nor test). Is it covering a transitive test dependency after the hadoop artifact switch, or just added along the way? Drop it if it's not needed; add a one-line comment if it is.
|
Are you also working on adding support for Spark 4.2? If so, I’ll hold off on my changes, haha. |
3 participants
What changes were proposed in this pull request?
Switch the Hadoop 3.x test dependency from
hadoop-miniclustertohadoop-client-minicluster.This PR also introduces the
hadoop-minicluster.artifactIdproperty so that:hadoop-minicluster, becausehadoop-client-miniclusteris unavailable for Hadoop 2.8.5.hadoop-client-minicluster.Why are the changes needed?
While upgrading to Spark 4.2, I encountered a test classpath conflict between Jersey 3 used by Spark and Jersey 2 introduced by Hadoop MiniDFS through
hadoop-minicluster.Compared with
hadoop-minicluster,hadoop-client-miniclusterhas a cleaner dependency set and is therefore used for Hadoop 3.x test profiles.The property-based selection preserves compatibility with Hadoop 2.8, which still requires the original
hadoop-miniclusterartifact.How was this patch tested?
Existing test cases.