Search before asking
Description
Motivation
Fluss tables can use table.datalake.database-name and
table.datalake.table-name to map a Fluss table to a Paimon table with a
different database or table name. #3476
Spark lake reads currently reject tables whose resolved Paimon path differs
from the Fluss table path. This issue tracks adding full Spark support for
custom Paimon lake table paths.
Scope
- Resolve and use the mapped Paimon database and table name when creating the
Spark lake source.
- Support custom Paimon paths in Fluss/Paimon union reads.
- Ensure predicate pushdown continues to work when the lake source uses a
mapped database or table name.
- Add coverage for custom database mapping, custom table mapping, and combined
database/table mapping.
- Cover both direct lake reads and union reads.
Current behavior
Spark throws the following exception when a lake read resolves to a different
Paimon table path:
Custom lake table path is not supported for Spark lake reads yet.
### Willingness to contribute
- [ ] I'm willing to submit a PR!
Search before asking
Description
Motivation
Fluss tables can use
table.datalake.database-nameandtable.datalake.table-nameto map a Fluss table to a Paimon table with adifferent database or table name. #3476
Spark lake reads currently reject tables whose resolved Paimon path differs
from the Fluss table path. This issue tracks adding full Spark support for
custom Paimon lake table paths.
Scope
Spark lake source.
mapped database or table name.
database/table mapping.
Current behavior
Spark throws the following exception when a lake read resolves to a different
Paimon table path: