Skip to content

IPIP-431: Opt-in Extensible CAR Metadata on Trustless Gateway #431

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bajtos wants to merge 18 commits into from
Closed

IPIP-431: Opt-in Extensible CAR Metadata on Trustless Gateway #431

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
2eb7b9e
IPIP: CAR `meta` (content type parameter)
bajtos Aug 8, 2023
3814b6a
add IPIP/0431-gateway-car-trailer.md
bajtos Aug 8, 2023
cb1d8b3
Merge branch 'main' into gw-car-metadata-trailer
bajtos Aug 8, 2023
1c0fbaa
chore: editora fixes, prep for HTML preview
lidel Aug 8, 2023
a7e75d7
chore: enable HTML preview
lidel Aug 8, 2023
68715c4
ipip-431: add upside to one of alternatives
lidel Aug 8, 2023
ed86a0f
ipip-431: add CARv3 to Alternatives
lidel Aug 10, 2023
5056bde
meta=eof+data update
Oct 3, 2023
9170c29
metadata schema update including json schema usage
Oct 4, 2023
65ffcfc
Update content path and query params to retrieval params
geomatrick Oct 4, 2023
9d1b61f
fixes from Miros feedback
Oct 4, 2023
93c3c28
json schema wrapper around top level object
Oct 4, 2023
eacf51a
dag and car params
Oct 13, 2023
62fb207
more alternatvies from discussion added
Oct 13, 2023
72ed04c
Merge pull request #1 from bajtos/review-1-fixes
bajtos Oct 18, 2023
e1fc296
add Patrick as a co-editor
bajtos Oct 18, 2023
152f4a6
formatting cleanup, remove duplicate schema, describe attack vectors
bajtos Oct 18, 2023
b6069bf
Merge branch 'main' into gw-car-metadata-trailer
bajtos Oct 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 94 additions & 0 deletions IPIP/0431-gateway-car-trailer.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
---
# IPIP number should match its pull request number. After you open a PR,
# please update title and update the filename to `ipip0000`.
title: "IPIP-0431: CAR metadata trailer in Gateway responses"
date: 2023-08-08
ipip: proposal
editors:
- name: Miroslav Bajtoš
# relatedIssues:
# - link to issue
order: 0000
tags: ['ipips', 'httpGateways']
---

## Summary

Define an optional enhancement of the CARv1 stream that allows a Gateway server to provide
additional metadata about the CARv1 response. Introduce a new content type that allows the client
and the server to signal or negotiate the inclusion of extra metadata.

## Motivation

SPARK is a Filecoin Station module that measures the reputation of Storage Providers by periodically
retrieving a random CID. Since both SPs and SPARK nodes are permissionless, and Proof of Retrieval
is an unsolved problem, we need a way to verify that a SPARK node retrieved the given CID from the
given SP. To enable that, we need the Trustless Gateway serving the retrieval request to include a
retrieval attestation after the entire response was sent to the client.

We currently have no mechanism to signal that a CAR file transmission over HTTP completed
successfully. However, we need this in order to be able to use CARs as a way of serving streaming
responses for queries. One way of solving this problem is to append an extra block at the end of the
CAR stream with information that clients can use to check whether all CAR blocks have been received.

## Detailed design

CAR content type
(`[application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car)`)
already supports multiple parameters like `version` and `order`, which allows gateways to indicate
which CAR flavors is returned with the response.

The proposed solution introduces a new parameters for the content type headers in HTTP requests
and responses: `meta`.

When the content type parameter `meta` is set to `eof`, the Gateway will write one additional CAR
block with metadata to the response, after it sent all CAR blocks.

The metadata format is DAG-CBOR and open to extension.

## Design rationale

The proposal introduces a minimal change allowing Gateways and retrieval clients to explicitly opt
into receiving additional metadata block at the end of the CAR response stream.

The metadata block is designed to be very flexible and able to support new use-cases that may arise
in the future.

### User benefit

- Clients of trustless gateways can use the fields from the metadata as an attestation that they
performed the retrieval from the given server.

- The `len` field in the metadata block allows clients to verify whether they received all CAR
bytes.

### Compatibility

The new feature requires clients to explicitly ask the server to include the extra block,
therefore the change is fully backwards-compatible for all existing gateway clients.

Gateways receiving requests for the new content type can ignore the `meta` parameter they don't
support and return back a response with one of the content types they support. This makes the
proposed change backwards-compatible for existing gateways too.


### Security

The proposed specification change does not introduce any negative security implications.
Comment thread
bajtos marked this conversation as resolved.
Outdated

### Alternatives

Instead of adding a new content type argument, we were considering sending the additional metadata
in HTTP response trailers. Unfortunately, HTTP trailers are not widely supported by the ecosystem.
Nginx proxy module discards them, browser `Fetch API` do not allow clients to access trailer
headers, neither does the Rust `reqwest` client.

## Test fixtures

TBD

Using one CID, request the CAR data using various combinations of content type parameters.

### Copyright

Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
28 changes: 27 additions & 1 deletion src/http-gateways/trustless-gateway.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,8 @@ Below response types SHOULD be supported:
- [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car)
- Disables IPLD/IPFS deserialization, requests a verifiable CAR stream to be
returned, implementations MAY support optional CAR content type parameters
(:cite[ipip-0412]) and the explicit [CAR format signaling in HTTP Request](#car-format-signaling-in-request).
(:cite[ipip-0412]), the explicit [CAR format signaling in HTTP Request](#car-format-signaling-in-request)
and the optional [CAR metadata block](#car-meta-content-type-parameter).

- [application/vnd.ipfs.ipns-record](https://www.iana.org/assignments/media-types/application/vnd.ipfs.ipns-record)
- A verifiable :cite[ipns-record] (multicodec `0x0300`).
Expand Down Expand Up @@ -301,6 +302,31 @@ of their presence in the DAG or the value assigned to the "dups" parameter, as
the raw data is already present in the parent block that links to the identity
CID.

## CAR `meta` (content type parameter)

The `meta` parameter allows clients to request the server to include additional metadata about the
CAR to be included at the end of the response body.

This parameter can be used with `version=1` only.

When the parameter is not set, the server must not add any extra CAR blocks to the response.

The metadata block is a regular CAR block with the following properties:

- CID specifies multicodec `car-metada` (0x04ff), see
[multicodec#334](https://github.com/multiformats/multicodec/pull/334).

- The payload contains metadata encoded as DAG-CBOR.

The metadata MUST include the following fields:

- `len` - byte length of the CAR data (excluding the metadata block)
Comment thread
bajtos marked this conversation as resolved.
Outdated
- `b3h` - Blake3 hash (checksum) of the CAR data (excluding the metadata block).
- `b3h_sig` - A signature over `<len><b3h><request>` using server's Ed2559 identity.
- `len` is encoded as `varint`,
- `b3h` is encoded as 32 bytes,
- The effective query as executed by the gateway. This query is the request url - path and query string arguments.
Copy link
Copy Markdown
Member

@lidel lidel Aug 8, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These b3* fields are specific to SPARK retrieval attestation and should not be listed in the trustless gateway spec as a MUST. These may be mandatory for SPARK, but are optional for the rest of IPFS ecosystem.

Please move them to "User benefit" section of the IPIP document and explain how meta=eof enables SPARK use case by allowing for these custom signatures to be passed along with the data. It makes a good example of extensibility that does not require PL's permission.

ps. I know other services like dagHouse use different hash functions for getting "CAR CID", putting all bets on Blake3 feels like an unnecessary divergence.

Perhaps this could be made bit more future-proof and generic if blake3 is represented as Multihash wrapped in CIDv1+car codec (0x0202)? Just an idea, fine to ignore, given these are specific to SPARK.

Either way, this belongs to the "userland benefiting from metadata extensibility" story.

Suggested change
- `b3h` - Blake3 hash (checksum) of the CAR data (excluding the metadata block).
- `b3h_sig` - A signature over `<len><b3h><request>` using server's Ed2559 identity.
- `len` is encoded as `varint`,
- `b3h` is encoded as 32 bytes,
- The effective query as executed by the gateway. This query is the request url - path and query string arguments.

Copy link
Copy Markdown

@geomatrick geomatrick Sep 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that the Spark use case belongs in Userland section. However, the individual keys of the metadata section, and what the servers must do to implement them, feels like something that should be in the trustless gateway spec.

Keys like car_bytes, data_bytes, block_count will be used by Spark but also may be used by others, and the definition of a key (i.e. what does the server actually return as the value for each key) must be the same for each use case. E.g. if one use case sets data_bytes to be the total byte length of blocks and another use case sets it to be the total byte length of the CAR stream then trustless gateway implementers will need to implement different logic for each different use case.

What's more, for the Spark use case, we do not want gateway operators to know that they are serving a Spark request and not some other request. Since the Spark ones will be incentivised and other request may not be, servers may simply provide a good retrieval service to Spark clients and a poor service to other clients.

car_bytes, data_bytes, block_count seem generic enough. The troublesome one is then the Blake3Hash and signature.

Copy link
Copy Markdown

@geomatrick geomatrick Sep 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps what we need to do is leave this IPIP to concern the metadata block being appended without any constraints on what can be included in it. Then in a separate place, we define a canonical way to include a key value object in the metadata block and how the server should implement certain useful keys such as car_bytes et al

Copy link
Copy Markdown
Member

@lidel lidel Sep 15, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be ok to suggest a key name convention for generic things like car_bytes data_bytes and block_count in the section that described meta parameter, as long it is

  • scoped to JSON (perhaps list under explicit meta=eof[+json]?)
  • change requirement from MUST to SHOULD (convention, not a hard requirement)

I think it would be also ok to have a documented convention for passing a hash of the CAR stream (aka CAR CID) – maybe name it car_cid and use CIDv1 with 0x0202 codec – this convention is already used by .storage folks, no need to invent anything new.

Copy link
Copy Markdown
Author

@bajtos bajtos Oct 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be also ok to have a documented convention for passing a hash of the CAR stream (aka CAR CID) – maybe name it car_cid and use CIDv1 with 0x0202 codec – this convention is already used by .storage folks, no need to invent anything new.

+1 to document car_cid.

For SPARK, we specifically want a Blake3 hash so that we can use inclusion proofs. That's why we want to use a dedicated field b3checksum instead of a more generic car_cid.

Copy link
Copy Markdown
Member

@lidel lidel Aug 8, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Including content path and CAR export parameters feels generic enough to keep in the spec, but we should not mix content path with car and url parameters as it leads to bugs around things like percent-encoding especially where ? or / is involved (cc ipfs/gateway-conformance#115).

These should be three separate fields:

  • content_path - requested content path
  • dag_params - map with DAG params like dag-scope, entity-bytes from IPIP-402
  • car_params - map with CAR content type params like order and dups from IPIP-412

Copy link
Copy Markdown

@geomatrick geomatrick Sep 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lidel what is the reasoning behind splitting up dag_params and car_params? Could we instead go for content_path and query_params to keep it simple and generic (allowing for other query params)?

Copy link
Copy Markdown
Member

@lidel lidel Sep 15, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that was a conscious design choice, to avoid mixing data selector with details of the transport format (not everything is in URL query params):

  • dag_params are about what user data was selected, not tied to any specific transport (could be applied to something other than CAR)
    • these are things that land in URL query
  • car_params are specific to CAR container format, they do not change the user data that was selected, only the way it is represented when sent as CAR
    • these are things that land in Accept/Content-Type headers
  • people read "query" and assume URL query :)

@patrickwoodhead that being said, if you want to simplify, this IPIP could go with a single dag-json map named response_params (to avoid confusion with URL query params, and account for the fact that server may ignore some of request params when producing a response)

Copy link
Copy Markdown
Author

@bajtos bajtos Oct 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about using retrieval_params instead of response_params? In my mind, response parameters don't describe "what user data was selected".

What is our motivation in SPARK:
We want the gateway to describe what exactly the client is retrieving (CID, subpath, dag params, car params) and provide a signature over that.

If two SPARK checker clients submit a metadata block with the same retrieval parameters (CID, subpath, dag params, car params) then we want:

  • To be able to verify that the clients retrieved the DAG (sub)tree they were expected to retrieve.
  • Confidence that both clients made the same request to the gateway (semantically?) and were supposed to receive exactly the same response from the gateway.

content_path - requested content path

When the client requests GET /ipfs/bafy1234/cat.jpg, what is content_path?

  • /ipfs/bafy1234/cat.jpg
  • bafy1234/cat.jpg
  • /cat.jpg
  • something else?

We need the metadata block to describe both the CID requested (bafy1234) and the resource subpath if specified (/cat.jpg).


## CAR format parameters and determinism

The default header and block order in a CAR format is not specified by IPLD specifications.
Expand Down