Skip to content

IPIP-402: Partial CAR Support on Trustless Gateways #402

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lidel merged 21 commits into from
Jul 27, 2023
Merged

IPIP-402: Partial CAR Support on Trustless Gateways #402

Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
dd84771
feat(http-gateways): add trustless parameters
hannahhoward Apr 17, 2023
6f7f7b3
style(lint): fix markdown errors
hannahhoward Apr 17, 2023
aeff21c
chore: apply suggestions
lidel Apr 27, 2023
b5eca27
Merge main into feat/improving-trustless-gateways
lidel May 10, 2023
c68f06a
ipip-0402: add IPIP document, first pass
lidel May 11, 2023
1fc8915
ipip-0402: car-scope=file → dag-scope=entity & bytes → entity-bytes
lidel May 11, 2023
98191b5
ipip-402: remove mentions of ordered CARs for now
lidel May 11, 2023
a6bfb2c
docs: small editorial fixes
hacdias May 17, 2023
78e81e0
Merge branch 'main' into feat/improving-trustless-gateways
hacdias May 17, 2023
bfb5d3c
ipip-402: wip clarify car roots expectations
lidel Jun 6, 2023
0639715
ipip-402: clarify CarV1Header.roots
lidel Jun 6, 2023
56786ff
ipip-402: relax CAR roots requirement
lidel Jun 7, 2023
61d1088
ipip-404: descope car roots and determinism
lidel Jun 20, 2023
e1af6e4
ipip-402: clarify error scenarios
lidel Jul 3, 2023
f5f33ff
Merge branch 'main' into feat/improving-trustless-gateways
lidel Jul 4, 2023
4ebbb96
ipip-402: final test fixtures section
lidel Jul 4, 2023
e3c36f3
chore: fix typos
lidel Jul 5, 2023
0953cb6
ipip-402: clarify entity-bytes for sharded unixfs file
lidel Jul 12, 2023
ffb1550
ipip-402: clarify status code in errors when streaming car
lidel Jul 12, 2023
584098d
ipip-402: clarify unixfs offset
lidel Jul 21, 2023
917efb9
chore: fix typo
lidel Jul 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 73 additions & 54 deletions src/ipips/ipip-0402.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,11 +138,6 @@ Gateways ignore unknown URL parameters. A client sending them to a
gateway that does not implement this IPIP will get all blocks for the requested
DAG.

<!-- TODO
mention feature detection via OPTIONS -- a separate IPIP?
OR suggest executing test request and client-side detection the first time a gateway is used.
-->

#### CAR roots and determinism

As of 2023-06-20, the behavior of the `roots` CAR field remains an [unresolved item within the CARv1 specification](https://web.archive.org/web/20230328013837/https://ipld.io/specs/transport/car/carv1/#unresolved-items):
Expand Down Expand Up @@ -235,9 +230,14 @@ single request.

## Test fixtures
Comment thread
lidel marked this conversation as resolved.
Copy link
Copy Markdown
Member

@lidel lidel Jul 4, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ A detailed list of compliance checks for dag-scope and entity-bytes can be found in gateway-conformance v0.2.0 → trustless_gateway_car_test.go. I've made another pass and double-checked them + pushed 4ebbb96 which adds CIDs, CARs, and short summary of each fixture to this IPIP.

This should be enough as a go-to resource for people who want to test with some reference data, or understand what the compliance tests do and where to find them.


<!-- TODO: describe what and how should be tested
- [ ] add relevant tests to https://github.com/ipfs/gateway-conformance → and reference fixtures here
-->
Relevant tests were added to
[gateway-conformance](https://github.com/ipfs/gateway-conformance) test suite
in [#56](https://github.com/ipfs/gateway-conformance/pull/56) and
[#85](https://github.com/ipfs/gateway-conformance/issues/85).
Detailed list of compliance checks for `dag-scope` and `entity-bytes` can be found in
[`v0.2.0/trustless_gateway_car_test.go`](https://github.com/ipfs/gateway-conformance/blob/v0.2.0/tests/trustless_gateway_car_test.go) or later.

Below are CIDs, CARs, and short summary of each fixture.

### Testing pathing

Expand All @@ -249,15 +249,24 @@ returned CAR MUST include both the block with the file data and all blocks
necessary for traversing from the root CID to the terminating element (all
parents, root CID and a subdirectory below it).

Fixtures:

:::example

- TODO(gateway-conformance): `/ipfs/dag-pb-cid/parent/file?format=car` (UnixFS file in a subdirectory)
Sample fixtures:

- TODO(gateway-conformance): `/ipfs/dag-pb-cid/hamt-parent1/file?format=car` (UnixFS file on a path within HAMT-sharded parent directory)
- `bafybeietjm63oynimmv5yyqay33nui4y4wx6u3peezwetxgiwvfmelutzu`
from [`subdir-with-two-single-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/subdir-with-two-single-block-files.car)
for testing `/ipfs/dag-pb-cid/subdir/ascii.txt?format=car`
(UnixFS file in a subdirectory)

- TODO(gateway-conformance): `/ipfs/dag-cbor-cid/file?format=car` (UnixFS file on a path with DAG-CBOR root CID)
- `bafybeidbclfqleg2uojchspzd4bob56dqetqjsj27gy2cq3klkkgxtpn4i`
from [`single-layer-hamt-with-multi-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/single-layer-hamt-with-multi-block-files.car)
for testing `/ipfs/dag-pb-hamt-cid/686.txt?format=car`
(UnixFS file on a path within HAMT-sharded parent directory)

- `bafybeia264q44a3kmfc2otctzu4egp2k235o3t7mslz2yjraymp4nv6asi`
from [`dir-with-dag-cbor-with-links.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/dir-with-dag-cbor-with-links.car)
for testing `/ipfs/dag-cbor-cid/document?format=car`
(UnixFS file on a path with DAG-CBOR root CID)

:::

Expand All @@ -271,13 +280,23 @@ To test real world use, request UnixFS `file` or a `directory` from a sub-path.
The returned CAR MUST include blocks required for path traversal and ONLY the
root block of the terminating entity.

Fixtures:

:::example

- TODO(gateway-conformance): `/ipfs/cid/parent/directory?format=car&dag-scope=block` (UnixFS directory on a path)
Sample fixtures:

- `bafybeietjm63oynimmv5yyqay33nui4y4wx6u3peezwetxgiwvfmelutzu`
from [`subdir-with-two-single-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/subdir-with-two-single-block-files.car)
for testing
- `/ipfs/dag-pb-cid/subdir/ascii.txt?format=car&dag-scope=block` (UnixFS file in a subdirectory)
- `/ipfs/dag-pb-cid?format=car&dag-scope=block` (UnixFS directory)

- TODO(gateway-conformance): `/ipfs/cid/parent1/parent2/file?format=car&dag-scope=block` (UnixFS file on a path)
- `bafybeidbclfqleg2uojchspzd4bob56dqetqjsj27gy2cq3klkkgxtpn4i`
from [`single-layer-hamt-with-multi-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/single-layer-hamt-with-multi-block-files.car)
for testing:
- `/ipfs/dag-pb-hamt-cid/1.txt?format=car&dag-scope=block`
(UnixFS multi-block file on a path within HAMT-sharded parent directory)
- `/ipfs/dag-pb-hamt-cid?format=car&dag-scope=block`
(UnixFS HAMT-sharded directory)

:::

Expand All @@ -296,48 +315,45 @@ IPLD entity. Currently, the most popular entity types are:
- `raw` / `dag-cbor`
(block with raw data or DAG-CBOR document, potentially linking to other CIDs)

Fixtures:

:::example

- TODO(gateway-conformance): `/ipfs/cid/chunked-dag-pb-file?format=car&dag-scope=entity`
- Request a `chunked-dag-pb-file` (UnixFS file encoded with `dag-pb` with
more than one chunk). Returned blocks MUST be enough to deserialize the file.
Sample fixtures:

- TODO(gateway-conformance): `/ipfs/cid/dag-cbor-with-link?format=car&dag-scope=entity`
- Request a `dag-cbor-with-link` (DAG-CBOR document with CBOR Tag 42 pointing
at a third-party CID). The response MUST include the terminating entity (DAG-CBOR)
and MUST NOT include the CID from the Tag 42 (IPLD Link).
- `bafybeidh6k2vzukelqtrjsmd4p52cpmltd2ufqrdtdg6yigi73in672fwu`
from [`subdir-with-mixed-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/subdir-with-mixed-block-files.car)
for testing:
- `/ipfs/dag-pb-cid/subdir/multiblock.txt?format=car&dag-scope=entity` (UnixFS multi-block file in a subdirectory)
- `/ipfs/dag-pb-cid/subdir?format=car&dag-scope=entity` (UnixFS directory)

- TODO(gateway-conformance): `/ipfs/cid/flat-directory/file?format=car&dag-scope=entity`
- Request UnixFS `flat-directory`. The response MUST include the minimal set of
blocks required for enumeration of directory contents, and no blocks that
belong to child entities.
- `bafybeidbclfqleg2uojchspzd4bob56dqetqjsj27gy2cq3klkkgxtpn4i`
from [`single-layer-hamt-with-multi-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/single-layer-hamt-with-multi-block-files.car)
for testing:
- `/ipfs/dag-pb-hamt-cid/1.txt?format=car&dag-scope=entity`
(UnixFS multi-block file on a path within HAMT-sharded parent directory, returned blocks MUST be enough to deserialize the file)
- `/ipfs/dag-pb-hamt-cid?format=car&dag-scope=entity`
(UnixFS HAMT-sharded directory, response MUST include the minimal set of blocks required for enumeration of directory contents, and no blocks that belong to child entities)

- TODO(gateway-conformance): `/ipfs/cid/hamt-directory/file?format=car&dag-scope=entity`
- Request UnixFS `hamt-directory`. The response MUST include the minimal set of
blocks required for enumeration of directory contents, and no blocks that
belong to child entities.
- `bafybeia264q44a3kmfc2otctzu4egp2k235o3t7mslz2yjraymp4nv6asi`
from [`dir-with-dag-cbor-with-links.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/dir-with-dag-cbor-with-links.car)
for testing `/ipfs/dag-cbor-cid/document?format=car&dag-scope=entity`
(DAG-CBOR document with IPLD Links must return all necessary blocks to verify the path, the document itself, but not the content behind any of the child entity IPLD Links)

:::

### Testing `dag-scope=all`

This is the implicit default used when `dag-scope` is not present,
and explicitly used in the context of proxy gateway supporting :cite[ipip-0288].

Fixtures:
and used in the context of deserialized UnixFS TAR responses from :cite[ipip-0288].

:::example

- TODO(gateway-conformance): `/ipfs/cid-of-a-directory?format=car&dag-scope=all`
- Request a CID of UnixFS `directory` which contains two files. The response MUST
contain all blocks that can be accessed by recursively traversing all IPLD
Links from the root CID.
Sample fixtures:

- TODO(gateway-conformance): `/ipfs/cid/chunked-dag-pb-file?format=car&dag-scope=all`
- Request a CID of UnixFS `file` encoded with `dag-pb` codec and more than
one chunk. The response MUST contain blocks for all `file` chunks.
- `bafybeidh6k2vzukelqtrjsmd4p52cpmltd2ufqrdtdg6yigi73in672fwu`
from [`subdir-with-mixed-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/subdir-with-mixed-block-files.car)
for testing:
- `/ipfs/dag-pb-cid/subdir?format=car&dag-scope=all` (path parents and the entire UnixFS subdirectory, returned recursively)
- `/ipfs/dag-pb-cid/subdir/multiblock.txt?format=car&dag-scope=all` (path parents and the entire UnixFS multi-block file)

:::

Expand All @@ -358,16 +374,19 @@ Use of the below fixture is highly recommended:

:::example

- TODO(gateway-conformance): `/ipfs/dag-pb-file?format=car&entity-bytes=40000000000-40000000002`

- Request a byte range from the middle of a big UnixFS `file`. The response MUST
contain only the minimal set of blocks necessary for fullfilling the range
request.

- TODO(gateway-conformance): `/ipfs/10-bytes-cid?format=car&entity-bytes=4:-2`

- Request a byte range from the middle of a small file, to -2 bytes from the end.
- (TODO confirm we want keep this -- added since it was explicitly stated as a supported thing in path-gateway.md)
- `bafybeidh6k2vzukelqtrjsmd4p52cpmltd2ufqrdtdg6yigi73in672fwu`
from [`subdir-with-mixed-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/subdir-with-mixed-block-files.car)
for testing:
- `/ipfs/dag-pb-cid/subdir/multiblock.txt?format=car&dag-scope=entity&entity-bytes=0:*` (path blocks and all the blocks for the multi-block UnixFS file)
- `multiblock.txt?format=car&dag-scope=entity&entity-bytes=512:1023` (path blocks and all the blocks for the the range request within multi-block UnixFS file)
- `multiblock.txt?format=car&dag-scope=entity&entity-bytes=512:-256` (path blocks and all the blocks for the the range request within multi-block UnixFS file)
- `/ipfs/dag-pb-cid/subdir?format=car&dag-scope=entity&entity-bytes=0:*` (path blocks and all the blocks to enumerate UnixFS directory)

- `QmYhmPjhFjYFyaoiuNzYv8WGavpSRDwdHWe5B4M5du5Rtk`
from [`file-3k-and-3-blocks-missing-block.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/file-3k-and-3-blocks-missing-block.car)
for testing:
- `/ipfs/dag-pb-cid?format=car&dag-scope=entity&entity-bytes=0:1000` (only the blocks needed to fullfill the request, MUST succeed despite the fact that a block after the range is not retrievable)
- `/ipfs/dag-pb-cid?format=car&dag-scope=entity&entity-bytes=2200:*` (only the blocks needed to fullfill the request, MUST succeed despite the fact that a block before the range is not retrievable)

:::

Expand Down