Skip to content

Spec: Add shared pagination definitions for Polaris APIs - #6028

Open
sririshindra wants to merge 4 commits into
apache:mainfrom
sririshindra:spec-shared-pagination
Open

sririshindra wants to merge 4 commits into
apache:mainfrom
sririshindra:spec-shared-pagination

Conversation

@sririshindra

Copy link
Copy Markdown
Contributor

New Polaris list operations either reference the Iceberg REST Catalog pagination definitions or define their own (#5927, #5943). The Iceberg definition requires a server that supports pagination to return all results when the request has no pageToken. That defeats the purpose of paginating, and Polaris already works around it for its Iceberg endpoints (#5282, #5553). Separate copies per operation also drift apart.

This adds spec/polaris-pagination.yaml, one definition that the catalog and the management APIs can both reference:

  • PolarisPageToken and PolarisPageSize parameters (query names pageToken and pageSize);
  • a PolarisPageToken schema for the token in the response. Each response names the field in its API's style (next-page-token in the catalog APIs, nextPageToken in the management APIs).

The rules: any response may be paginated; a client needs neither parameter to start; the server decides the page size, which may be smaller than pageSize; a server may return everything in one response. A missing or null token ends the listing.

No operation references the file yet, so the bundled spec and the generated code don't change. I checked locally that a management operation and a catalog operation can reference it: the catalog bundle inlines it and the generated API and model modules compile. The Polaris prefix avoids a name clash with the Iceberg PageToken in the bundled spec.

Existing operations (generic tables, policies, semantic models, the Shares API) keep their current definitions. The generic table and policy handlers enforce the Iceberg rule today, so moving them would also change server behavior; that can follow separately.

Checklist

New Polaris list operations either reference the Iceberg REST Catalog
pagination definitions or define their own. The Iceberg definitions
require a server to return all results when a request has no pageToken,
which defeats the purpose of paginating, and per-operation copies drift
apart.

Add spec/polaris-pagination.yaml with one pageToken/pageSize parameter
pair and a page token schema, usable from both the catalog and the
management APIs. Its rules: any response may be paginated, neither
parameter is needed to start, the server decides the page size, and a
server may return everything in one response. Each response names the
token field in its own API's style.

No operation references the file yet. Existing operations keep their
current definitions.
@github-project-automation github-project-automation Bot moved this to PRs In Progress in Basic Kanban Board Nov 9, 2026
@sririshindra
sririshindra marked this pull request as ready for review November 9, 2026 23:20
dimas-b
dimas-b previously approved these changes Nov 9, 2026

@dimas-b dimas-b left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍 Thanks, @sririshindra !

I assume you're planning to use the these definitions in your storage config PRs, right?

@sririshindra

sririshindra commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

LGTM 👍 Thanks, @sririshindra !

I assume you're planning to use the these definitions in your storage config PRs, right?

That's correct @dimas-b . I will rebase my storage config pr on top of this once this is merged. Thanks for the review!

@flyrain

flyrain commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

Hi @sririshindra , having one shared pagination definition makes sense. The main thing I'd like settled before operations start referencing this file is the 'any response may be paginated' rule. Is it true only for the endpoint with pagination? For the Iceberg endpoints, the dev list went fail-closed in #5553: an oversized unpaged listing gets rejected instead of truncated. This file goes the other way for Polaris-native APIs. That may be the right call, but can we get an explicit dev-list sign-off?

@dimas-b

dimas-b commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

@flyrain : Re: I'd like settled before operations start referencing this file is the 'any response may be paginated' rule.

I think consensus on this is best reached on the dev ML. Would you like to open a discussion there?

As for me, the current spec wording proposed in this PR is reasonable. Actual usage depends on endpoints that reference the common definitions.

The first rule now says that it applies to the responses of operations
that use these definitions, so it cannot be read as a statement about
every Polaris list operation. Operations that reference the Iceberg REST
Catalog pagination definitions keep those rules.
@sririshindra

Copy link
Copy Markdown
Contributor Author

@flyrain Yes, the rules only apply to operations that reference this file. I pushed a small commit so the first rule now says that explicitly: "Any response of an operation that uses these definitions may be paginated."

Nothing on main references the file yet, so this PR doesn't change any existing behavior. Operations that use the Iceberg pagination definitions keep the Iceberg rules and the #5553 behavior. That includes the Iceberg listings and also the Polaris catalog listings that reuse those definitions (policies, generic tables, semantic models).

I don't think the two conflict. Fail-closed is there for clients written against the Iceberg contract, which promises the full result when there's no pageToken. Those clients may never look for a token, so truncating would silently drop results. An operation that uses this file tells clients from the start that they must always check for a token and follow it, so a first page can't be mistaken for the whole result.

The Shares API on main already works this way (spec/polaris-shares-api.yaml): if pageSize is absent the service's maximum is used, and nextPageToken is present when more results exist. This file just writes that down once so new list operations can share it.

I agree it should be signed off explicitly, so I started a thread on dev@: https://lists.apache.org/thread/2d3v9ob15w9fcpkb9qs65tmvhkmm1dcf. I'll wait for that to settle before this gets merged.

cc @dimas-b

@sririshindra
sririshindra requested a review from dimas-b November 14, 2026 02:16
dimas-b
dimas-b previously approved these changes Nov 17, 2026

@flyrain flyrain left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sririshindra for working on it. LGTM overall. Left some comments.

Comment thread spec/polaris-pagination.yaml Outdated
required: false
description: >
A hint for the largest number of results the client wants in one page. The server decides
the actual page size, which may be smaller.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we say that the server configures the maximum page size and a client can ask for a smaller page? For example, if the server maximum is 100, pageSize=10 should return at most 10 results, and pageSize=500 should return at most 100. The spec can define that rule and leave the actual maximum to each deployment.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's the intent. I changed pageSize from a hint to an upper bound: a server that paginates returns at most pageSize results in a page, and fewer when its own maximum is smaller. With a maximum of 100, pageSize=10 gets at most 10 and pageSize=500 at most 100. Polaris already does this (PageTokenUtil.boundPageSize). The maximum itself stays a deployment setting; on dev@ Dmitri suggested the new operations get their own limit, on by default, which the implementation PRs will pick up.

I kept the qualifier "a server that paginates", because the last rule still lets a server that doesn't paginate return everything in one response, as in the Iceberg spec.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"at most" SGTM

Comment thread spec/polaris-pagination.yaml Outdated
results.
- A client does not need to send `pageToken` or `pageSize` to start paginating. The server
may paginate a response to a request that has neither.
- The server decides the page size. A page may hold fewer results than `pageSize`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: I think we've explain the rule already in pageSize, do we need to duplicate it here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. With pageSize now stating the bound, I removed this rule. The first rule already says that only a missing token ends the listing, so a short page can't be read as the last one.

Comment thread spec/polaris-pagination.yaml Outdated
Comment on lines +36 to +37
The page token from the previous response of the same list operation, sent unchanged. Omit it
to get the first page.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we specify that the server returns 400 Bad Request when the client sends an invalid pageToken?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we require clients to keep the parent, filters, and sort order the same, while allowing pageSize to change? Should the server return 400 if those other parameters change?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. The pageToken description now says the server returns 400 Bad Request for a page token it cannot use. Polaris already does that for a token it can't decode (PageTokenUtil.deserializePageToken). It also says that an empty pageToken means the same as no pageToken, which JB suggested on dev@ because Java Iceberg clients send one to start.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and JB raised the same point on dev@. The description now says that with a page token the client sends the same path and query parameters as in the previous request, except pageSize, which it may change.

For the server side I wrote "may" return 400 when the other parameters differ, not "must": today's tokens only carry the page size and the position, so the server can't tell that a filter changed without a change to the token format.

Comment thread spec/polaris-pagination.yaml Outdated
Comment on lines +20 to +21
# Pagination definitions shared by the Polaris-specific APIs: the Polaris catalog APIs and the
# management APIs. A list operation references the two parameters and uses the PageToken schema for

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: we don't have to enumerate them.

Suggested change
# Pagination definitions shared by the Polaris-specific APIs: the Polaris catalog APIs and the
# management APIs. A list operation references the two parameters and uses the PageToken schema for
# Pagination definitions shared by the Polaris-specific APIs. A list operation references the two parameters and uses the PageToken schema for

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment thread spec/polaris-pagination.yaml Outdated
Comment on lines +22 to +23
# the token in its response. The response field is named in the style of the API it belongs to:
# `next-page-token` in the catalog APIs, `nextPageToken` in the management APIs.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we leave the response field names to the individual API specs? This file can focus on the pagination rules they share.

Suggested change
# the token in its response. The response field is named in the style of the API it belongs to:
# `next-page-token` in the catalog APIs, `nextPageToken` in the management APIs.
# the token in its response.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, each API spec names its own field.

pageSize is now an upper bound for a server that paginates, capped by the
server's own maximum, instead of a hint. That makes the separate "the
server decides the page size" rule redundant, so it is removed.

pageToken now says that an empty token means the first page, that a
client sends the same parameters on every page except pageSize, and that
the server returns 400 for a token it cannot use and may return it when
the other parameters change.

The header comment no longer lists the APIs or the response field names;
each API spec names its own field.
@sririshindra

Copy link
Copy Markdown
Contributor Author

@flyrain Thanks for the review. I pushed d4df9ed with your suggestions and the spec points from the dev@ thread: pageSize is now an upper bound for a server that paginates, an empty pageToken means the first page, a client sends the same parameters on every page except pageSize, and the server returns 400 for a token it cannot use. Replies are inline.

The dev@ thread has +1s from JB, Vignesh, Ayush, Dmitri and Yufei, with no objections: https://lists.apache.org/thread/2d3v9ob15w9fcpkb9qs65tmvhkmm1dcf

@dimas-b the push dismissed your approval. Could you take another look?

@sririshindra
sririshindra requested a review from flyrain November 18, 2026 23:27
dimas-b
dimas-b previously approved these changes Nov 18, 2026

@dimas-b dimas-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM in general. A could of clarifying comments below 🙂

Comment thread spec/polaris-pagination.yaml Outdated
required: false
description: >
The largest number of results the client wants in one page. A server that paginates returns
at most this many results in a page, and fewer when its own maximum page size is smaller.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maximum page size is a fuzzy term. In certain situations the size may be the number of entries, in others the number of bytes.

I'd prefer to avoid this complication in the spec and just say "The server may return fewer items, even zero items. If zero items are returned the client should request yet another page using the fresh page token from the empty response". (feel free to rephrase / edit).

The zero case is actualized in #5544

#5544 (comment)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, I dropped "maximum page size". pageSize now says the server returns at most that many results and may return fewer, even none, in a page that is not the last; a client that gets an empty page with a page token requests the next page with it. The first rule already makes a missing token the only end-of-results signal, so the zero-results case from #5544 is covered.

Comment thread spec/polaris-pagination.yaml Outdated
- Any response of an operation that uses these definitions may be paginated. A client must
always check for a page token in the response and, when one is present, send it unchanged
as the `pageToken` query parameter to get the next page. The server omits the page token
from the last page. A client must treat a missing or `null` page token as the end of the

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically an empty token fits here too since the doc above (line 35) equates empty tokens with missing tokens.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. The rule now reads "a missing, null, or empty page token", matching the pageToken parameter.

flyrain
flyrain previously approved these changes Nov 18, 2026

@flyrain flyrain left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @sririshindra !

pageSize no longer refers to a server's "maximum page size", which can
mean a number of entries or a number of bytes. It now says that a server
may return fewer results than pageSize, even none, in a page that is not
the last, and that a client follows the page token of an empty page.

Since an empty pageToken in a request means no token, an empty page token
in a response now also marks the end of the results, along with a missing
or null one.
@sririshindra
sririshindra dismissed stale reviews from flyrain and dimas-b via cf08c5a November 18, 2026 23:41

@ayushtkn ayushtkn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants