Skip to content

[spark] Support custom Paimon lake table paths #3832

Description

@wzx140

Search before asking

  • I searched in the issues and found nothing similar.

Description

Motivation

Fluss tables can use table.datalake.database-name and
table.datalake.table-name to map a Fluss table to a Paimon table with a
different database or table name. #3476

Spark lake reads currently reject tables whose resolved Paimon path differs
from the Fluss table path. This issue tracks adding full Spark support for
custom Paimon lake table paths.

Scope

  • Resolve and use the mapped Paimon database and table name when creating the
    Spark lake source.
  • Support custom Paimon paths in Fluss/Paimon union reads.
  • Ensure predicate pushdown continues to work when the lake source uses a
    mapped database or table name.
  • Add coverage for custom database mapping, custom table mapping, and combined
    database/table mapping.
  • Cover both direct lake reads and union reads.

Current behavior

Spark throws the following exception when a lake read resolves to a different
Paimon table path:

Custom lake table path is not supported for Spark lake reads yet.

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions