[FLINK-29549]- Flink Glue Catalog integration by fmorillo7694 · Pull Request #206 · apache/flink-connector-aws

fmorillo7694 · 2025-05-07T10:12:17Z

Purpose of the change

For example: Implements the Table API for the Kinesis Source.

Verifying this change

Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

Added integration tests for end-to-end deployment
Added unit tests
Manually verified by running the Kinesis connector on a local Flink cluster.

Significant changes

(Please check any boxes [x] if the answer is "yes". You can first publish the PR and check them afterwards, for convenience.)

Dependencies have been added or upgraded
Public API has been changed (Public API is any class annotated with @Public(Evolving))
Serializers have been changed
New feature has been introduced
- If yes, how is this documented? (not applicable / docs / JavaDocs / not documented)

Co-Authored-By: Anthony Pounds-Cornish <antpc@amazon.co.uk>

refactoring directory Adding Parent Pom

Samrat002

Thanks for the contribution. A few points to note:

It looks like most of the code has already been reviewed across multiple PRs.

Concerns:

The current implementation exposes low-level Glue details (e.g., support for lowercase table names) directly through the Flink Catalog. Is this acceptable? I recommend raising this topic with the community. In my opinion, it would be better to encapsulate such Glue-specific behavior and avoid exposing it directly via the Flink Catalog interface. The catalog's behavior should remain consistent with other catalogs, with differences controlled via configuration only.
Please consider implementing the basic configuration options defined in FLIP-277. If that’s not feasible in this PR, a fast follow-up would be valuable, especially for users relying on different credential modes.
It seems that some .idea folder files have been committed. Please remove them from the PR.

Cheers,
Samrat

fmorillo7694 · 2025-05-14T15:24:39Z

Thanks for the contribution. A few points to note:

It looks like most of the code has already been reviewed across multiple PRs.

Concerns:

1. The current implementation exposes low-level Glue details (e.g., support for lowercase table names) directly through the Flink Catalog. Is this acceptable? I recommend raising this topic with the community. In my opinion, it would be better to encapsulate such Glue-specific behavior and avoid exposing it directly via the Flink Catalog interface. The catalog's behavior should remain consistent with other catalogs, with differences controlled via configuration only.

2. Please consider implementing the basic configuration options defined in [FLIP-277](https://cwiki.apache.org/confluence/display/FLINK/FLIP-277). If that’s not feasible in this PR, a fast follow-up would be valuable, especially for users relying on different credential modes.

3. It seems that some `.idea` folder files have been committed. Please remove them from the PR.

Cheers, Samrat

Hey Samrat. How could we encapsulate this specific issues? From user perspective we are already limiting them creating tables and databases with uppercase. In regards of the schema, we encapsulate storing the actual original columnNames in column parameters in Glue, so even though user sees their schema in glue with lower case (default for glue). we actually leverage the original column name.

Yes right after we get the first release delivered we will add the remainder authentication, and partitions support and alter support
Fixed!

Samrat002 · 2025-05-16T08:36:52Z

Hey Samrat. How could we encapsulate this specific issues? From user perspective we are already limiting them creating tables and databases with uppercase. In regards of the schema, we encapsulate storing the actual original columnNames in column parameters in Glue, so even though user sees their schema in glue with lower case (default for glue). we actually leverage the original column name.

Thanks for the detailed response.

Can you help me understand the technical complexity to support CaseSensitivity from GlueCatalog?

This is a deviation.
I believe understanding how other connectors or catalogs handle mismatches, there can be a couple of ways to handle it

Surfacing the mismatch to the end user with constraints
Handling it in FlinkCatalog to provide consistent behaviour.

We can start a thread in the community to discuss this approach.

I am fine with either of the approaches as long as the community agrees on it.

leekeiabstraction · 2025-05-19T15:14:11Z

+     */
+    @Override
+    public void close() throws CatalogException {
+        if (glueClient != null) {


Is there a scenario where glueClient would be null?

glueClient is created lazily.

i think in the following scenario client can be null

Create a catalog

Never fired any DDL or DML SQL command

Close the catalog or shell

this guard unexpected failure NullPointerException

leekeiabstraction · 2025-05-19T15:25:59Z

+            int maxRetries = 3;
+            int retryCount = 0;
+            long retryDelayMs = 200;
+            while (retryCount < maxRetries) {


Curious on what is the motivation for retries here? Is there a specific RuntimeException that we have in mind?

I had a look at AWS sdk, the close is best-effort and logs in case of any exceptions: https://github.com/aws/aws-sdk-java-v2/blob/master/utils/src/main/java/software/amazon/awssdk/utils/IoUtils.java#L76-L85

We can simplify code here by removing retry logic as sdk does not seem to be surfacing exceptions during close().

nicusX · 2025-05-27T16:57:12Z

+                    .tableList();
+
+            // Filter tables to only include those that are of type VIEW
+            List<String> viewNames = allTables.stream()


You can return here

nicusX · 2025-05-28T07:46:22Z

@Samrat002 @FranMorilloAWS about case sensitivity, these are the logical rules I suggest:

the actual case should preserved for all supported objects and used consistently in Flink interface, e.g. inCREATE TABLE... but also SHOW TABLE, SHOW COLUMNS ... etc. From a user's perspective, the way object names are actually stored in Glue should not be a concern.
The additional metadata should be stored in Glue in a consistent way for all object types. Parameters seems to be available for Database, Tables, and Columns in Glue. A problem arises for Functions which don't have any Parameters if I am not mistaken.
The additional metadata should not leak into Flink user interface. The "originalName" should not pop up in a SHOW TABLES or SHOW COLUMNS
The fact Glue UI is limited and only shows lowercase names is not a concern of this component's

I would add that the way the additional metadata (the original name and any additional info required) is stored in Glue should be clearly documented, in case a user want's to build their own external interface to extract or update information in the Glue Catalog.

fmorillo7694 · 2025-06-03T15:02:12Z

Thanks for the contribution. A few points to note:

It looks like most of the code has already been reviewed across multiple PRs.

Concerns:

1. The current implementation exposes low-level Glue details (e.g., support for lowercase table names) directly through the Flink Catalog. Is this acceptable? I recommend raising this topic with the community. In my opinion, it would be better to encapsulate such Glue-specific behavior and avoid exposing it directly via the Flink Catalog interface. The catalog's behavior should remain consistent with other catalogs, with differences controlled via configuration only.

2. Please consider implementing the basic configuration options defined in [FLIP-277](https://cwiki.apache.org/confluence/display/FLINK/FLIP-277). If that’s not feasible in this PR, a fast follow-up would be valuable, especially for users relying on different credential modes.

3. It seems that some `.idea` folder files have been committed. Please remove them from the PR.

Cheers, Samrat

Hello @Samrat002 @nicusX. Using Database Parameters and Table Parameters we are now able to use lower/upper case for defining the Database and Table Name. By using the Show Tables/Show Databases, Describe table commands, we will show the original Flink Definition, even though in Glue UI it will be all in lower case

Samrat002 · 2025-06-06T09:29:53Z

Thanks for incorporating the changes. I will review the pr in next couple of days .

Samrat002

Patch looks great. Thank you for making the change.
minor comments
Cheers,
Samrat

Samrat002 · 2025-08-05T05:39:10Z

+# AWS Glue Catalog
+
+The AWS Glue Catalog provides a way to use [AWS Glue](https://aws.amazon.com/glue) as a catalog for Apache Flink. 
+This allows users to access Glue's metadata store directly from Flink SQL and Table API.


In my understanding, the DataStream API can also utilize the Glue Catalog. Is that not true?

Samrat002 · 2025-08-05T05:40:38Z

+Before getting started, ensure you have the following:
+
+- **AWS account** with appropriate permissions for AWS Glue and other required services
+- **AWS credentials** properly configured


Can you add a simlink to How to configure from AWS documentation ?

Samrat002 · 2025-08-12T09:18:06Z

+        String region = config.get(REGION.key());
+        String defaultDatabase = config.getOrDefault(DEFAULT_DATABASE.key(), DEFAULT_DATABASE.defaultValue());
+
+        // Ensure required properties are present


unwanted comment

Samrat002 · 2025-08-12T09:21:29Z

+        GetUserDefinedFunctionsRequest.Builder functionsRequest =
+                GetUserDefinedFunctionsRequest.builder()
+                        .databaseName(databaseName);
+        List<String> glueFunctions;


If there is no function present, then glueFunctions will be a null object. This might be an issue.
can you add

Suggested change

List<String> glueFunctions;

List<String> glueFunctions = new ArrayList<>();

Samrat002 · 2025-08-12T09:26:07Z

+     * @param glueClient The Glue client used for interacting with the AWS Glue service.
+     * @param catalogName The catalog name associated with the Glue operations.
+     */
+    protected GlueOperator(GlueClient glueClient, String catalogName) {


Add a null check on input params glueClient

melin · 2025-10-09T01:53:06Z

Can we refer to Paimon's solution for integrating Glue?
https://github.com/apache/paimon/pull/4283/files#diff-534e1f558f31481abfd234e3b9c56c17a0988acf360a2264b0a090fa3a6aedd5

https://issues.apache.org/jira/browse/FLINK-38457

@FranMorilloAWS

fmorillo7694 · 2025-12-11T10:06:43Z

@melin. To me knowledge Paimon integration with Glue Data Catalog is to store Paimon Tables in Glue, however it wouldnt work for storing Kinesis, MSK, and other streaming sources. They have different implementations.

…entation

- Remove scala.binary.version property (Flink 2.0 dropped Scala deps) - Replace flink-table-planner_2.12 with flink-table-planner-loader - Migrate CatalogTable.of() to CatalogTable.newBuilder() API (removed in Flink 2.0) - All 120 tests pass

…entation

…t partition operations (G1) Fixes three of the four HIGH findings from the Glue Data Catalog review (B1 constructor client wiring was already fixed on this branch): - B2: buildTableInput now persists partitionKeys and the table comment (description) on the Glue TableInput; createRegularTable splits data columns (storage descriptor) from partition columns (TableInput), so partition metadata declared in DDL survives the round-trip. - B3: column names are no longer force-lowercased with an originalName side-channel parameter; the declared case is stored directly in Glue. Legacy tables carrying the originalName parameter are still honored on read for backwards compatibility. - G1: partition operations are implemented against the Glue partition APIs (GetPartitions with pagination, GetPartition, CreatePartition, UpdatePartition, DeletePartition) via a new GluePartitionOperator, following the existing operator pattern and the Flink Catalog contract for exceptions (TableNotPartitionedException, PartitionNotExist, PartitionAlreadyExists, PartitionSpecInvalid). listPartitionsByFilter intentionally remains unsupported; Flink's planner falls back to listPartitions. Read path: getSchemaFromGlueTable now appends partition columns (stored at the Glue table level) to the schema so CatalogTable partition keys resolve, and tolerates a null storage descriptor. Tests: FakeGlueClient gains an in-memory partition store and persists TableInput.partitionKeys; new regression tests cover the partitioned round-trip (B2), case preservation + legacy-compat read (B3), and the full partition CRUD lifecycle incl. contract exceptions (G1). 126 tests pass; spotless clean.

fix(catalog): Persist partition metadata, preserve column case, implement partition ops

…terTable Real-AWS integration testing (us-west-2) proved that Glue lowercases column names on CreateTable, breaking the assumption introduced when the originalName column parameter write was removed: DESCRIBE lost declared case, SELECT on mixed-case columns failed to resolve, and streaming jobs against catalog-registered tables could not plan. Changes, each validated against a real Glue Data Catalog: - Store columns lowercase with the declared name in the originalName column parameter; restore it on read (round-trip now exact). - Partition columns: Glue rejects column-level parameters on partition keys (400: Parameters not supported for partition columns), so their declared case is preserved in an order-preserving table-level parameter (flink.original-partition-keys) instead. - Implement alterTable via Glue UpdateTable (was UnsupportedOperationException), preserving the original table name. - Filter the new internal parameter from user-visible table options. Verified with a 25-step real-AWS integration suite: full DDL surface (mixed-case databases/tables/columns, describe, alter, partitions add/show/drop) plus a real streaming job writing to and reading from Kinesis through source/sink tables registered in the Glue catalog, including JSON field-name case verification on the wire. 25/25 PASS. Unit tests: 126/126.

…ixes fix(catalog): Preserve column case against real Glue; implement alterTable

fmorillo7694 · 2026-08-10T11:43:31Z

Real AWS Glue integration test results (design rationale for `originalName`)

The latest head (b1a05d3) includes fixes driven by an integration test run against a real Glue Data Catalog (us-west-2), not Moto/Localstack. Posting the evidence here since it settles a design question a reviewer might reasonably raise: why store lowercase names plus an originalName parameter instead of writing mixed-case names directly?

Empirical facts about real Glue (reproducible via the AWS CLI)

Glue lowercases column names on CreateTable. Creating a table with columns userId / EventTime stores and returns userid / eventtime. Glue does not preserve mixed-case column names, so writing them as-is silently loses case.
Glue rejects column-level Parameters on partition columns with 400: Parameters not supported for partition columns. The originalName column-parameter mechanism therefore cannot be used for partition keys — their declared case is preserved in an order-preserving table-level parameter (flink.original-partition-keys) instead.

What case loss breaks (observed before the fix, same suite)

DESCRIBE returned [userid, eventtime, username] for a table declared [userId, EventTime, userName]
SELECT userId FROM t → Column 'userId' not found in any table; did you mean 'userid'?
Partition ops failed to match: spec {eventDate=...} vs stored keys [eventdate]
Streaming INSERT INTO / SELECT against catalog-registered Kinesis tables failed to plan, and JSON field names on the wire would be lowercased (silent data corruption for consumers)

Validation on the current head

25-step integration suite against real AWS, all passing:

DDL: mixed-case databases, tables, and columns round-trip exactly (create / show / describe / drop); ALTER TABLE ... SET (now implemented via UpdateTable) with a case-preservation regression check; partitioned tables with ADD/SHOW/DROP PARTITION on mixed-case partition keys
Streaming: unbounded job writing to and reading from a real Kinesis stream through source/sink tables registered in this catalog; JSON field names verified case-intact on the wire via GetRecords; PARTITIONED BY on a Kinesis table verified to set the record partition key from the (case-restored) column
Unit tests: 126/126

One known behavior worth documenting: partition columns are appended after data columns on read-back (Glue models partition keys outside the storage descriptor, like Hive), so declaring partition columns last keeps the schema order stable across the round-trip.

Samrat002 and others added 3 commits August 18, 2024 23:09

[FLINK-30481][FLIP-277] GlueCatalog Implementation

7cb8122

Co-Authored-By: Anthony Pounds-Cornish <antpc@amazon.co.uk>

AWS Glue Catalog implementation for Apache Flink

ee2b385

refactoring directory Adding Parent Pom

merged commits

e449cd6

fmorillo7694 marked this pull request as ready for review May 7, 2025 10:18

fixing conflicts with AWSGeneralUtil

3f0c943

fmorillo7694 changed the title ~~[FLINK-29549]- Updated Flink Glue Catalog integration~~ [FLINK-29549] Updated Flink Glue Catalog integration May 7, 2025

fmorillo7694 changed the title ~~[FLINK-29549] Updated Flink Glue Catalog integration~~ [FLINK-29549]- Flink Glue Catalog integration May 7, 2025

Samrat002 suggested changes May 13, 2025

View reviewed changes

fmorillo7694 added 5 commits May 14, 2025 12:45

fixing build

af636c0

removing idea

ae1d761

removing gitignore

059a57a

Adding Docs

3d78f41

Add AWS Glue Catalog documentation and update POM

17eaa1c

boring-cyborg Bot added the component=Documentation label May 14, 2025

Update AWS Glue Catalog documentation to match README

7bea413

leekeiabstraction suggested changes May 19, 2025

View reviewed changes

nicusX reviewed May 27, 2025

View reviewed changes

support for uppercase for database and table name

1da04f0

fmorillo7694 added 3 commits June 3, 2025 13:55

adding lowercase support for database name

8b24573

adding upper case support for table names

9df0e25

updating tests

97ed6bf

Samrat002 reviewed Sep 19, 2025

View reviewed changes

Merge remote-tracking branch 'upstream/main' into new-combined-implem…

59a97eb

…entation

fmorillo7694 force-pushed the new-combined-implementation branch from ee90672 to a7e1c06 Compare May 30, 2026 19:14

FCAgent and others added 6 commits June 1, 2026 10:15

Merge remote-tracking branch 'upstream/main' into new-combined-implem…

f102054

…entation

[hotfix] Fix spotless formatting violations in flink-catalog-aws-glue

46e9f73

Merge pull request #4 from fmorillo7694/glue-catalog-high-fixes

7e4a11f

fix(catalog): Persist partition metadata, preserve column case, implement partition ops

Merge pull request #5 from fmorillo7694/glue-catalog-real-glue-case-f…

b1a05d3

…ixes fix(catalog): Preserve column case against real Glue; implement alterTable

	List<String> glueFunctions;
	List<String> glueFunctions = new ArrayList<>();

Uh oh!

Conversation

fmorillo7694 commented May 7, 2025

Purpose of the change

Verifying this change

Significant changes

Uh oh!

Samrat002 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fmorillo7694 commented May 14, 2025

Uh oh!

Samrat002 commented May 16, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nicusX commented May 28, 2025

Uh oh!

fmorillo7694 commented Jun 3, 2025

Uh oh!

Samrat002 commented Jun 6, 2025

Uh oh!

Samrat002 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

melin commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fmorillo7694 commented Dec 11, 2025

Uh oh!

fmorillo7694 commented Aug 10, 2026

Real AWS Glue integration test results (design rationale for originalName)

Empirical facts about real Glue (reproducible via the AWS CLI)

What case loss breaks (observed before the fix, same suite)

Validation on the current head

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

melin commented Oct 9, 2025 •

edited

Loading

Real AWS Glue integration test results (design rationale for `originalName`)