Skip to content
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions spec/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,9 @@ Apache Polaris provides the following OpenAPI specifications:
- [oauth-tokens-api.yaml](polaris-catalog-apis/oauth-tokens-api.yaml) - Contains the specification for the
internal OAuth Token endpoint, extracted from the Apache Iceberg REST Catalog API.

- [polaris-pagination.yaml](polaris-pagination.yaml) - Defines the pagination parameters and page token
shared by the Polaris-specific list operations, in both the catalog and the management APIs.


## Generated Specification Files
The specification files in the generated folder are automatically created using OpenAPI bundling tools such as
Expand Down
66 changes: 66 additions & 0 deletions spec/polaris-pagination.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#

# Pagination definitions shared by the Polaris-specific APIs: the Polaris catalog APIs and the
# management APIs. A list operation references the two parameters and uses the PageToken schema for

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: we don't have to enumerate them.

Suggested change
# Pagination definitions shared by the Polaris-specific APIs: the Polaris catalog APIs and the
# management APIs. A list operation references the two parameters and uses the PageToken schema for
# Pagination definitions shared by the Polaris-specific APIs. A list operation references the two parameters and uses the PageToken schema for

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

# the token in its response. The response field is named in the style of the API it belongs to:
# `next-page-token` in the catalog APIs, `nextPageToken` in the management APIs.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we leave the response field names to the individual API specs? This file can focus on the pagination rules they share.

Suggested change
# the token in its response. The response field is named in the style of the API it belongs to:
# `next-page-token` in the catalog APIs, `nextPageToken` in the management APIs.
# the token in its response.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, each API spec names its own field.

#
# These rules differ from the Iceberg REST Catalog API, where a server must return all results when
# the request has no `pageToken`. Operations defined by the Iceberg REST Catalog API keep its rules.

---
components:
parameters:
PolarisPageToken:
name: pageToken
in: query
required: false
description: >
The page token from the previous response of the same list operation, sent unchanged. Omit it
to get the first page.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we specify that the server returns 400 Bad Request when the client sends an invalid pageToken?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we require clients to keep the parent, filters, and sort order the same, while allowing pageSize to change? Should the server return 400 if those other parameters change?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. The pageToken description now says the server returns 400 Bad Request for a page token it cannot use. Polaris already does that for a token it can't decode (PageTokenUtil.deserializePageToken). It also says that an empty pageToken means the same as no pageToken, which JB suggested on dev@ because Java Iceberg clients send one to start.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and JB raised the same point on dev@. The description now says that with a page token the client sends the same path and query parameters as in the previous request, except pageSize, which it may change.

For the server side I wrote "may" return 400 when the other parameters differ, not "must": today's tokens only carry the page size and the position, so the server can't tell that a filter changed without a change to the token format.

schema:
type: string

PolarisPageSize:
name: pageSize
in: query
required: false
description: >
A hint for the largest number of results the client wants in one page. The server decides
the actual page size, which may be smaller.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we say that the server configures the maximum page size and a client can ask for a smaller page? For example, if the server maximum is 100, pageSize=10 should return at most 10 results, and pageSize=500 should return at most 100. The spec can define that rule and leave the actual maximum to each deployment.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's the intent. I changed pageSize from a hint to an upper bound: a server that paginates returns at most pageSize results in a page, and fewer when its own maximum is smaller. With a maximum of 100, pageSize=10 gets at most 10 and pageSize=500 at most 100. Polaris already does this (PageTokenUtil.boundPageSize). The maximum itself stays a deployment setting; on dev@ Dmitri suggested the new operations get their own limit, on by default, which the implementation PRs will pick up.

I kept the qualifier "a server that paginates", because the last rule still lets a server that doesn't paginate return everything in one response, as in the Iceberg spec.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"at most" SGTM

schema:
type: integer
minimum: 1

schemas:
PolarisPageToken:
description: |
An opaque token that continues a list operation. Clients must not interpret it.
- Any response of an operation that uses these definitions may be paginated. A client must
always check for a page token in the response and, when one is present, send it unchanged
as the `pageToken` query parameter to get the next page. The server omits the page token
from the last page. A client must treat a missing or `null` page token as the end of the

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically an empty token fits here too since the doc above (line 35) equates empty tokens with missing tokens.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. The rule now reads "a missing, null, or empty page token", matching the pageToken parameter.

results.
- A client does not need to send `pageToken` or `pageSize` to start paginating. The server
may paginate a response to a request that has neither.
- The server decides the page size. A page may hold fewer results than `pageSize`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: I think we've explain the rule already in pageSize, do we need to duplicate it here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. With pageSize now stating the bound, I removed this rule. The first rule already says that only a missing token ends the listing, so a short page can't be read as the last one.

- A server that does not paginate returns all results in one response, with no page token.
type: string
Loading