fix: make gRPC max recv message size configurable and distinguish ResourceExhausted errors - #427
Open
fraseragain wants to merge 3 commits into
Open
fix: make gRPC max recv message size configurable and distinguish ResourceExhausted errors#427fraseragain wants to merge 3 commits into
fraseragain wants to merge 3 commits into
Conversation
…ourceExhausted errors DID Documents with resource collections exceeding gRPC's default 4MB receive-message limit (e.g. did:cheqd:testnet:a6af96a5-ec0c-4b66-b4cd-6e5916cadc8b, which has grown to 9,008 linked resources) fail resolution with a gRPC ResourceExhausted error that was previously indistinguishable from a genuine not-found DID, since every ledger query mapped any error to notFound. Adds a configurable GRPC_MAX_RECV_MSG_SIZE setting (default 16MB, threaded through the existing Config/RawConfig pipeline alongside the other ledger connection settings) so operators can raise the limit without a code change, and distinguishes ResourceExhausted from other gRPC errors so a future overflow surfaces as internalError (500) instead of silently misreporting notFound (404).
The linked page moved as part of a Cosmos SDK docs site reorganization; the URL now 404s. Points to the current location.
3 tasks
The Test workflow has been failing on every run since 2026-02-25 (develop and every subsequent PR, ~6 months) because both the unit-tests and integration-tests jobs installed the ginkgo CLI via `go install .../ginkgo@latest`, which ignores go.mod entirely and always fetches the newest published release. ginkgo v2.32.1 now requires Go >= 1.25.0, but CI's actions/setup-go resolves Go 1.24.0 from go.mod, so the install step itself failed before any tests ran. go.mod already pins github.com/onsi/ginkgo/v2 v2.28.1 as a real dependency, compatible with the project's own Go 1.24.0 requirement. Dropping the @latest suffix makes `go install` resolve the CLI version from the current module's go.mod/go.sum instead, keeping it permanently in lockstep with the library version the project actually depends on rather than drifting again on ginkgo's next release. Also fixes the integration-tests job's ginkgo install running from `working-directory: ./..` (needed for @latest, which ignores the local module regardless of cwd; breaks the module-relative install once @latest is dropped, since it points outside the repo), and the "Show logs" step's working-directory: ./docker/localnet, which doesn't exist in this repo and was failing outright whenever triggered, compounding earlier failures in the same job.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DID Documents whose resource collections exceed gRPC's default 4MB receive-message limit fail resolution — e.g.
did:cheqd:testnet:a6af96a5-ec0c-4b66-b4cd-6e5916cadc8b, whose collection has grown to 9,008 linked resources (~4.2MB), returnsnotFoundfrom resolver.cheqd.net despite existing and resolving correctly via direct chain queries.Root cause:
openGRPCConnectionWithTimeoutopens its gRPC connection with no message-size override, soQueryCollectionResources(called during normal DID resolution to buildmetadata.resources) fails client-side withcodes.ResourceExhaustedonce a collection's response exceeds 4MB. EveryLedgerService.Query*method then maps any gRPC error, includingResourceExhausted, straight toNewNotFoundError, so the real cause was invisible — the resolver reportednotFoundfor a DID that genuinely exists.codes.ResourceExhaustedis a safe, unambiguous signal to special-case: confirmed cheqd-node returnscodes.Unknown(wrapped Cosmos SDK error) for a genuinely nonexistent DID, nevercodes.NotFound, so it can't collide with a real not-found response.did:cheqd:testnet:bf2d7da8-0f79-43cc-8126-cb176e589c5c) from the same publishing pattern at 3,746 resources (~1.8MB) and still actively growing — this isn't a one-off, other DIDs are on the same trajectory.Changes
GRPC_MAX_RECV_MSG_SIZEas a configurable setting (default 16MB), threaded through the existingRawConfig/Config→ constructor-injection pipeline the same way other ledger connection settings (TESTNET_ENDPOINTtimeout/TLS) already work — not hardcoded, since an operator being unable to raise this without a code change is what caused the original incident to go unresolved.mapGrpcErrorhelper (mirroring thestatus.FromError/codespattern already used inendpoint_manager.go) that mapsResourceExhaustedtointernalError(500) instead ofnotFound(404), so a future overflow is diagnosable rather than silently misreported. All other gRPC errors keep today'snotFoundbehaviour unchanged.docker-compose.ymlandREADME.mdalongside the existing ones.No automated test added for
mapGrpcError— the existingtests/unit/tests/integrationsuites use aMockLedgerServicefake that fabricates*types.IdentityErrordirectly and never exercises the real gRPC call path this touches; a meaningful test would need a colocatedservices/*_test.gofile, a first for this codebase. Verified manually instead (see test plan).Test plan
go build ./...— cleangofmt -l ./go vet ./...— cleanmake unit-tests— all 7 suites, 227 specs, 0 failuresgrpc.cheqd.networkbackend:did:cheqd:testnet:a6af96a5-ec0c-4b66-b4cd-6e5916cadc8b(9,008 resources) →200with full DID Document, previously404404 notFounddid:cheqd:testnet:bf2d7da8-0f79-43cc-8126-cb176e589c5c(3,746 resources, still growing) → unchanged200GRPC_MAX_RECV_MSG_SIZEset below the DID's actual size →500 internalErrorwith a clear log line, confirming the error-mapping fires correctly rather than falling back tonotFound