Repository navigation
Spec: Add shared pagination definitions for Polaris APIs - #6028
sririshindra wants to merge 4 commits into
Conversation
New Polaris list operations either reference the Iceberg REST Catalog pagination definitions or define their own. The Iceberg definitions require a server to return all results when a request has no pageToken, which defeats the purpose of paginating, and per-operation copies drift apart. Add spec/polaris-pagination.yaml with one pageToken/pageSize parameter pair and a page token schema, usable from both the catalog and the management APIs. Its rules: any response may be paginated, neither parameter is needed to start, the server decides the page size, and a server may return everything in one response. Each response names the token field in its own API's style. No operation references the file yet. Existing operations keep their current definitions.
There was a problem hiding this comment.
LGTM 👍 Thanks, @sririshindra !
I assume you're planning to use the these definitions in your storage config PRs, right?
That's correct @dimas-b . I will rebase my storage config pr on top of this once this is merged. Thanks for the review! |
|
Hi @sririshindra , having one shared pagination definition makes sense. The main thing I'd like settled before operations start referencing this file is the 'any response may be paginated' rule. Is it true only for the endpoint with pagination? For the Iceberg endpoints, the dev list went fail-closed in #5553: an oversized unpaged listing gets rejected instead of truncated. This file goes the other way for Polaris-native APIs. That may be the right call, but can we get an explicit dev-list sign-off? |
|
@flyrain : Re: I think consensus on this is best reached on the As for me, the current spec wording proposed in this PR is reasonable. Actual usage depends on endpoints that reference the common definitions. |
The first rule now says that it applies to the responses of operations that use these definitions, so it cannot be read as a statement about every Polaris list operation. Operations that reference the Iceberg REST Catalog pagination definitions keep those rules.
|
@flyrain Yes, the rules only apply to operations that reference this file. I pushed a small commit so the first rule now says that explicitly: "Any response of an operation that uses these definitions may be paginated." Nothing on main references the file yet, so this PR doesn't change any existing behavior. Operations that use the Iceberg pagination definitions keep the Iceberg rules and the #5553 behavior. That includes the Iceberg listings and also the Polaris catalog listings that reuse those definitions (policies, generic tables, semantic models). I don't think the two conflict. Fail-closed is there for clients written against the Iceberg contract, which promises the full result when there's no The Shares API on main already works this way ( I agree it should be signed off explicitly, so I started a thread on dev@: https://lists.apache.org/thread/2d3v9ob15w9fcpkb9qs65tmvhkmm1dcf. I'll wait for that to settle before this gets merged. cc @dimas-b |
flyrain
left a comment
There was a problem hiding this comment.
Thanks @sririshindra for working on it. LGTM overall. Left some comments.
| required: false | ||
| description: > | ||
| A hint for the largest number of results the client wants in one page. The server decides | ||
| the actual page size, which may be smaller. |
There was a problem hiding this comment.
Could we say that the server configures the maximum page size and a client can ask for a smaller page? For example, if the server maximum is 100, pageSize=10 should return at most 10 results, and pageSize=500 should return at most 100. The spec can define that rule and leave the actual maximum to each deployment.
There was a problem hiding this comment.
Yes, that's the intent. I changed pageSize from a hint to an upper bound: a server that paginates returns at most pageSize results in a page, and fewer when its own maximum is smaller. With a maximum of 100, pageSize=10 gets at most 10 and pageSize=500 at most 100. Polaris already does this (PageTokenUtil.boundPageSize). The maximum itself stays a deployment setting; on dev@ Dmitri suggested the new operations get their own limit, on by default, which the implementation PRs will pick up.
I kept the qualifier "a server that paginates", because the last rule still lets a server that doesn't paginate return everything in one response, as in the Iceberg spec.
| results. | ||
| - A client does not need to send `pageToken` or `pageSize` to start paginating. The server | ||
| may paginate a response to a request that has neither. | ||
| - The server decides the page size. A page may hold fewer results than `pageSize`. |
There was a problem hiding this comment.
Minor: I think we've explain the rule already in pageSize, do we need to duplicate it here?
There was a problem hiding this comment.
Agreed. With pageSize now stating the bound, I removed this rule. The first rule already says that only a missing token ends the listing, so a short page can't be read as the last one.
| The page token from the previous response of the same list operation, sent unchanged. Omit it | ||
| to get the first page. |
There was a problem hiding this comment.
Could we specify that the server returns 400 Bad Request when the client sends an invalid pageToken?
There was a problem hiding this comment.
Could we require clients to keep the parent, filters, and sort order the same, while allowing pageSize to change? Should the server return 400 if those other parameters change?
There was a problem hiding this comment.
Done. The pageToken description now says the server returns 400 Bad Request for a page token it cannot use. Polaris already does that for a token it can't decode (PageTokenUtil.deserializePageToken). It also says that an empty pageToken means the same as no pageToken, which JB suggested on dev@ because Java Iceberg clients send one to start.
There was a problem hiding this comment.
Agreed, and JB raised the same point on dev@. The description now says that with a page token the client sends the same path and query parameters as in the previous request, except pageSize, which it may change.
For the server side I wrote "may" return 400 when the other parameters differ, not "must": today's tokens only carry the page size and the position, so the server can't tell that a filter changed without a change to the token format.
| # Pagination definitions shared by the Polaris-specific APIs: the Polaris catalog APIs and the | ||
| # management APIs. A list operation references the two parameters and uses the PageToken schema for |
There was a problem hiding this comment.
Minor: we don't have to enumerate them.
| # Pagination definitions shared by the Polaris-specific APIs: the Polaris catalog APIs and the | |
| # management APIs. A list operation references the two parameters and uses the PageToken schema for | |
| # Pagination definitions shared by the Polaris-specific APIs. A list operation references the two parameters and uses the PageToken schema for |
| # the token in its response. The response field is named in the style of the API it belongs to: | ||
| # `next-page-token` in the catalog APIs, `nextPageToken` in the management APIs. |
There was a problem hiding this comment.
Could we leave the response field names to the individual API specs? This file can focus on the pagination rules they share.
| # the token in its response. The response field is named in the style of the API it belongs to: | |
| # `next-page-token` in the catalog APIs, `nextPageToken` in the management APIs. | |
| # the token in its response. |
There was a problem hiding this comment.
Done, each API spec names its own field.
pageSize is now an upper bound for a server that paginates, capped by the server's own maximum, instead of a hint. That makes the separate "the server decides the page size" rule redundant, so it is removed. pageToken now says that an empty token means the first page, that a client sends the same parameters on every page except pageSize, and that the server returns 400 for a token it cannot use and may return it when the other parameters change. The header comment no longer lists the APIs or the response field names; each API spec names its own field.
|
@flyrain Thanks for the review. I pushed d4df9ed with your suggestions and the spec points from the dev@ thread: The dev@ thread has +1s from JB, Vignesh, Ayush, Dmitri and Yufei, with no objections: https://lists.apache.org/thread/2d3v9ob15w9fcpkb9qs65tmvhkmm1dcf @dimas-b the push dismissed your approval. Could you take another look? |
dimas-b
left a comment
There was a problem hiding this comment.
LGTM in general. A could of clarifying comments below 🙂
| required: false | ||
| description: > | ||
| The largest number of results the client wants in one page. A server that paginates returns | ||
| at most this many results in a page, and fewer when its own maximum page size is smaller. |
There was a problem hiding this comment.
maximum page size is a fuzzy term. In certain situations the size may be the number of entries, in others the number of bytes.
I'd prefer to avoid this complication in the spec and just say "The server may return fewer items, even zero items. If zero items are returned the client should request yet another page using the fresh page token from the empty response". (feel free to rephrase / edit).
The zero case is actualized in #5544
There was a problem hiding this comment.
Agreed, I dropped "maximum page size". pageSize now says the server returns at most that many results and may return fewer, even none, in a page that is not the last; a client that gets an empty page with a page token requests the next page with it. The first rule already makes a missing token the only end-of-results signal, so the zero-results case from #5544 is covered.
| - Any response of an operation that uses these definitions may be paginated. A client must | ||
| always check for a page token in the response and, when one is present, send it unchanged | ||
| as the `pageToken` query parameter to get the next page. The server omits the page token | ||
| from the last page. A client must treat a missing or `null` page token as the end of the |
There was a problem hiding this comment.
Technically an empty token fits here too since the doc above (line 35) equates empty tokens with missing tokens.
There was a problem hiding this comment.
Good catch. The rule now reads "a missing, null, or empty page token", matching the pageToken parameter.
flyrain
left a comment
There was a problem hiding this comment.
LGTM. Thanks @sririshindra !
pageSize no longer refers to a server's "maximum page size", which can mean a number of entries or a number of bytes. It now says that a server may return fewer results than pageSize, even none, in a page that is not the last, and that a client follows the page token of an empty page. Since an empty pageToken in a request means no token, an empty page token in a response now also marks the end of the results, along with a missing or null one.
New Polaris list operations either reference the Iceberg REST Catalog pagination definitions or define their own (#5927, #5943). The Iceberg definition requires a server that supports pagination to return all results when the request has no
pageToken. That defeats the purpose of paginating, and Polaris already works around it for its Iceberg endpoints (#5282, #5553). Separate copies per operation also drift apart.This adds
spec/polaris-pagination.yaml, one definition that the catalog and the management APIs can both reference:PolarisPageTokenandPolarisPageSizeparameters (query namespageTokenandpageSize);PolarisPageTokenschema for the token in the response. Each response names the field in its API's style (next-page-tokenin the catalog APIs,nextPageTokenin the management APIs).The rules: any response may be paginated; a client needs neither parameter to start; the server decides the page size, which may be smaller than
pageSize; a server may return everything in one response. A missing ornulltoken ends the listing.No operation references the file yet, so the bundled spec and the generated code don't change. I checked locally that a management operation and a catalog operation can reference it: the catalog bundle inlines it and the generated API and model modules compile. The
Polarisprefix avoids a name clash with the IcebergPageTokenin the bundled spec.Existing operations (generic tables, policies, semantic models, the Shares API) keep their current definitions. The generic table and policy handlers enforce the Iceberg rule today, so moving them would also change server behavior; that can follow separately.
Checklist
CHANGELOG.md(if needed) (not needed: no behavior change)site/content/in-dev/unreleased(if needed) (not needed)