rfc: mutability & encryption for forge#84
Conversation
hannahhoward
left a comment
hannahhoward
left a comment
There was a problem hiding this comment.
Overall, this is good, but there are some critical missing bits, that I think will emerge from working closely with @Peeja , @alanshaw and the existing go devs.
Specifically,
-
"Service layer" -- is this a server? a secondary process process on the dev machine? I don't either of these are a good idea, and the server approach would break a number of design principles about our system (namely that all CIDs should be generated on the client). Personally I think a language port is gonna be WAY faster (especially with AI aided dev) and product a way less complex system, so I'd argue strongly for a full port. These aren't complex libraries and I think we could have them ported in a week or two with the AI helping. And then we have a single process for a single machine, way simpler to maintain and reason about.
-
There's a bit of unspecified confusion about how Pail works in the Forge context, that I think you might need to embed with @Peeja on guppy to really grok. So Guppy has a notion of "sources" -- i.e. data sources (usually large, deep directories) that get uploaded within a space. Each space has 1..n sources, and when you upload within a space, after the first upload of a source, only the "delta" gets updated-- Guppy knows how to upload just blocks to make a new updated UnixFS root. So with mutabiltiy:
- You have the list of sources which get updated, and you DEFINITELY want that to be represented by Pail + UCN.
- You have the directory tree structure within the sources itself. This is currently UnixFS and is updated properly each incremental upload.
- So the real question is about whether to use Pail for the whole directory tree, and I think that's a complicated question that merits further examination
Reasons not to use Pail:
- These are extremely big complicated directories and Pail hasn't been tested at a scale even remotely close to working with these directories
- The retrieval patters and general usage for Pail is totally different than for UnixFS -- so the downstream change implications of using Pail for the whole directory tree structure are unknown.
Reasons to use Pail:
- Much more fine grained "multi-writer" capabilities are unlocked if you use Pail for everything. If you used pail for just the sources list, then you'd essentially have a last-writer-wins on a per-source level -- if source X is in state A, and two different guppies make several changes to the directory tree structure, written as UnixFS, then the directory structure would by default ONLY get the changes of the last client to write. Note: we could apply a smarter merge outside of PAIL, similar to the way I merge Markdown files in Clawracha. I actually believe this wouldn't be TOO hard.
Final sidebar: Current Guppy is also smart enough to only upload diff blocks for Files when they change. Encryption will kill that ability I believe, unless there's some useful way to encode only changes that works for encrypted data. Worth a google.
|
After the feedback received and the new POC completed by @alanshaw, I decided to break it into 2 RFCs: |
There was a problem hiding this comment.
I'm pretty set on using Pail, especially since the go library exists, but if @alanshaw disagrees I'd be open to using the catalog approach.
|
@hannahhoward & @alanshaw - I've updated the Mutability RFC and addressed Hannah's suggestions. |
| 1. User requests decryption with delegation containing `nb.prefix` | ||
| 2. KMS extracts `path` from encrypted metadata block | ||
| 3. KMS validates: `path.startsWith(delegation.nb.prefix)` | ||
| 4. If valid → unwrap DEK; if invalid → reject |
There was a problem hiding this comment.
How does the service know what the wrapped decryption key is? Does it have to fetch the content? Does it need a retrieval delegation?
There was a problem hiding this comment.
We could combine the path with the DEK before wrapping it, and then use the wrapped DEK in KMS. WDYT?
| 1. **Mutable References**: Stable pointers that update to the latest version of uploaded content | ||
| 2. **Content Catalog**: Track uploaded content (path to CID mappings) across clients | ||
|
|
||
| **Related:** [forge-encryption.md](./forge-encryption.md) - Encryption key rotation depends on mutability to track metadata CID changes. |
There was a problem hiding this comment.
Perhaps worth being a bit more explicit about this in the encryption doc. Is it just this? It does not depend on mutability otherwise?
|
@alanshaw & @hannahhoward - we talked about starting v1 storing only the root path in Pail to handle mutability, I think that makes sense for the non-encrypted files. However, for the encryption, we kind of need the full path to handle the key rotation process. For encrypted uploads, we need
Proposal
Does this dual strategy make sense, or should we unify? |
RFC: An alternate interface for mutable content in GuppyMy main concern with adding a "name" to a "source" is that it's not an explicit action. It's a passive attribute of the source that we've decided will be used after the data is uploaded to record a name for the uploaded DAG. The name is attached to a collection of static files AND it is being used as a mutable pointer that MAY point to the DAG created by said source. Lets walk though the proposed interface: $ guppy upload source add did:key:zSpace "My Stuff" /path/to/alans/stuffWhen I type $ guppy upload did:key:zSpace
bafyAlansStuff
$ guppy ls did:key:zSpace
My Stuff bafyAlansStuffI delegate access to Felipe runs: $ guppy upload source add did:key:zSpace "My Stuff" /path/to/felipes/stuffThen Felipe runs $ guppy upload did:key:zSpace
bafyFelipesStuff
$ guppy ls did:key:zSpace
My Stuff bafyFelipesStuffWhat happens when if I want to update "My Stuff" to point back at my stuff not Felipe's stuff? Can I run This is an odd situation. We could make One possible solution to prevent issues like this is for However at this point I find myself questioning the exposure of sources in the CLI UX at all. Why do I need to create a source before uploading it? If I am always going to reference the file path in the upload command, why doesn't it just create or use the source for the directory I provide to ...but if we don't have sources where does the name go? Well, what if we provide a name to $ guppy upload did:key:zSpace "My Stuff" /path/to/alans/stuffThis feels better, upload is an action and it looks like we're asking the name of this upload to be "My Stuff" and I feel it's more concievable that this will overwrite another upload with the same name. Note that this also allows the same source to be put to multiple paths. e.g. $ guppy upload did:key:zSpace "Backups/My Stuff/2026-04-20" /path/to/alans/stuff
IMHO it's better, and I could probably get behind this. However, the name for this upload is not an attribute, it's more like an ID, a path or a key i.e. if you upload with the same key it'll overwrite. The problem is we've mixed uploading files and managing buckets. The word "upload" makes it look like the name is an attribute not an identity. Also calling it "name" and not "key". One way we could easily rectify this would be to switch to using bucket semantics for our CLI commands. i.e. instead of The magic is that when you provide a file path the CLI will automatically create a source and upload that data for you. This is expected, and is no different from say, the AWS CLI. So then we can distil our command to simply: $ guppy put <space> <key> <path>
Side note, this also allows users to construct buckets that contain data that they didn't directly upload to a space e.g. we might in the future support Proposal SummaryTo Summarize, my proposal would be:
|
@alanshaw - I really liked the idea of having new commands to handle the bucket operations, that looks more natural and easy to reason about, plus the |
Like Felipe, I agree this makes the DX more intuitive and clears up the confusion around name as an identity. That said, I think there are a couple of things worth thinking about before moving forward with this change. The current two-step approach lets users add sources over time and then run uploads cumulatively, instead of just doing one-off uploads. Guppy also supports different configs per source (like shard size). That might not be a big deal to drop, but it really depends on whether users actually use different shard sizes per source, I’m not sure if they do. |
4 participants
Forge Mutability & Encryption - Summary
What We're Building
Two complementary features for Forge enterprise customers:
Mutability (📖 Forge Mutability RFC)
Problem: Content on IPFS is immutable (CIDs change when content changes). Enterprise backup workflows need stable references that update to the latest version.
Solution: Extend existing Guppy commands (
upload source add,upload,retrieve) with Pail + UCN integration.How it works:
Commands:
Encryption (📖 Forge Encryption RFC)
Problem: Enterprise customers need data protection before content leaves their systems.
Solution: Client-side encryption with UCAN-gated access control.
How it works:
Encryption modes
How They Work Together
Encryption is an opt-in flag on the existing
upload source addcommand:Key rotation requires mutability because rotating keys creates new metadata CIDs that need to be tracked in Pail.
Implementation Status
Why This Approach?
Closes storacha/project-tracking#663