Skip to content
Closed
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 144 additions & 0 deletions http/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
# libp2p over HTTP <!-- omit in toc -->

| Lifecycle Stage | Maturity | Status | Latest Revision |
| --------------- | ------------- | ------ | --------------- |
| 1A | Working Draft | Active | r0, 2022-11-10 |

Authors: [@marcopolo]

Interest Group: [@marcopolo], [@mxinden], [@marten-seemann]

[@marcopolo]: https://github.com/mxinden
[@mxinden]: https://github.com/mxinden
[@marten-seemann]: https://github.com/marten-seemann

# Table of Contents <!-- omit in toc -->
- [Context](#context)
- [Why not two separate stacks?](#why-not-two-separate-stacks)
- [Why HTTP rather than a custom request/response protocol?](#why-http-rather-than-a-custom-requestresponse-protocol)
- [Implementation](#implementation)
- [HTTP over libp2p streams](#http-over-libp2p-streams)
- [libp2p over plain HTTPS](#libp2p-over-plain-https)
- [Choosing between libp2p streams vs plain HTTPS](#choosing-between-libp2p-streams-vs-plain-https)
- [Implementation recommendations](#implementation-recommendations)
- [Example – Go](#example--go)
- [Prior art](#prior-art)

# Context

HTTP is everywhere. Especially in CDNs, cloud offerings, and caches.

HTTP on libp2p and libp2p on HTTP are both commonly requested features. This has
come up recently at IPFS Camp and especially in the [data transfer track]. One
Comment thread
MarcoPolo marked this conversation as resolved.
Outdated
aspect of the discussion makes it seem like you can use HTTP _OR_ use libp2p,
but that isn't the case. Before this spec you could use the HTTP protocol on top
of a libp2p stream (with little to no extra cost). And this spec outlines how to use libp2p _on top of_ HTTP.

This spec defines a new libp2p abstraction for stateless request/response
protocols. This abstraction is notably nothing new, it is simply HTTP. Being
HTTP, This abstraction can run over a plain TCP+TLS HTTP (henceforth referred to
as _plain https_) or on top of a libp2p stream.

## Why not two separate stacks?

Having libp2p as the abstraction over _how_ the HTTP request gets sent gives developers a lot of benefits for free, such as:

1. NAT traversal: You can make an HTTP request to a peer that's behind a NAT.
1. Fewer connections: If you already have a libp2p connection, we can use use that to create a stream for the HTTP request. So that HTTP request will be much faster (You don't have to pay the two round trips to establish the connection)
Comment thread
MarcoPolo marked this conversation as resolved.
Outdated
1. Allows JS clients to make HTTPS requests to _any_ peer via WebTransport.
1. Allows more reuse of the protocol logic, just like how applications can integrate GossipSub, bitswap, graphsync, and Kademlia.
1. You get mutual authentication of peer IDs automatically.

Get the benefits of libp2p (webtransport, fewer connections, webrtc, nat traversal)
Comment thread
MarcoPolo marked this conversation as resolved.
Outdated

## Why HTTP rather than a custom request/response protocol?

HTTP has been around for 30+ years, and it isn't going anywhere. Developers are already very familiar with it. There's is no need to reinvent the wheel here.

# Implementation

## HTTP over libp2p streams

If we have an existing libp2p connection that supports streams, we can run the HTTP protocol as follows:

Client:
1. Open a new stream to the target peer.
1. Negotiate the `/libp2p-http` protocol.
1. Use this stream for HTTP. (i.e. start sending the request)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this HTTP 1.1?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://datatracker.ietf.org/doc/rfc9292/ could be useful as the wire format.

1. Close the write side when finished uploading the HTTP request.
1. Close the stream when the response is received.

Server:
1. Register the `/libp2p-http` protocol.
1. One a new stream speaking `/libp2p-http` pass the stream to an HTTP handler.
Comment thread
MarcoPolo marked this conversation as resolved.
Outdated
1. Close the stream when finished uploading the request.
Comment thread
MarcoPolo marked this conversation as resolved.
Outdated

## libp2p over plain HTTPS

This is nothing more than a thin wrapper over standard HTTP. The only thing
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be slightly OT but has the Upgrade header been considered for running libp2p on top of HTTP?

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Upgrade

GET /index.html HTTP/1.1
Host: www.example.com
Connection: upgrade
Upgrade: multistream-select/1.0.0

Once confirmed, we can then negotiate any other protocol on top, i.e. yamux, noise, etc.

This could be a neat way for establishing a libp2p connection from the browser, assuming the peer we want to reach exposes an HTTP endpoint we can use to trigger the upgrade.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming we define a corresponding "libp2p over http" multiaddress protocol, we can build a Transport that makes a GET request with the above upgrade, waiting for 101 Switching Protocols and then using the resulting stream for libp2p.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe: /http(s)/dns4/example.com/tcp/80

Note that http at the front means we run all of the following protocols on top of it, i.e. the very bottom transport is HTTP.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds like reinventing WebSocket.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WebSockets have a framing overhead though whereas unless I am missing something, using Upgrade would just hand us the stream.

libp2p should do here is ensure that we verify the peer's TLS certificate as
defined by the [tls spec](../tls/tls.md). This SHOULD be interoperable with standard HTTP clients who pass a correct TLS cert. For example curl should work fine:

```
$ curl --insecure --cert ./client.cert --key ./client.key https://127.0.0.1:9561/echo -d "Hello World"

Hello World
```

## Choosing between libp2p streams vs plain HTTPS

Implementations SHOULD choose a libp2p stream if an existing libp2p connection
is available. If there is an existing HTTP connection, then implementations
SHOULD use that connection rather than starting a new libp2p connection. If
there is no connection implementations may choose either to create a new HTTP
connection or a libp2p connection or expose this as an option to users.

# Implementation recommendations

Each implementation should decide how this works, but the general recommendations are:

1. Make this look and feel like a normal HTTP client and server. There's no
benefit of doing things differently here, and the familiarity will let people
build things faster.

1. Aim to make the returned libp2p+HTTP objects interop with the general HTTP ecosystem of the language.

## Example – Go

We create a host as normal, but enable HTTP:
```
h, err := libp2p.New(libp2p.WithHTTP(
HTTPConfig: HTTPConfig{
EnableHTTP: true,
// Enable
HTTPServerAddr: multiaddr.StringCast("/ip4/127.0.0.1/tcp/9561/tls/http"),
}))
```

We can define HTTP Handlers using standard types:
```
h1.SetHTTPHandler("/echo", func(peer peer.ID, w http.ResponseWriter, r *http.Request) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be really nice if we could reuse the http.ServeMux. We could make the peer ID available as a header field.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this idea!

w.WriteHeader(200)
io.Copy(w, r.Body)
})
```

We can create a client that accepts standard types
```
// client is a standard http.Client
client, err := h2.NewHTTPClient(h1.ID())
require.NoError(t, err)

resp, err := client.Post("/echo", "application/octet-stream", bytes.NewReader([]byte("Hello World")))
```

For more details see the implementation [PR](https://github.com/libp2p/go-libp2p/pull/1874).

# Prior art

- rust-libp2p's request-response protocol: https://github.com/libp2p/rust-libp2p/tree/master/protocols/request-response
- go-libp2p's [go-libp2p-http]

[data transfer track]: (https://youtube.com/watch?v=VRn_U8ytvok&feature=share&si=EMSIkaIECMiOmarE6JChQQ)
[rust-libp2p request-response protocol]: (https://github.com/libp2p/rust-libp2p/tree/master/protocols/request-response)
[go-libp2p-http]: (https://github.com/libp2p/go-libp2p-http)