[tesseract]: retry missing-state RPC errors, and park GET requests whose proofs fail #1441

Summary

Even on v2.4.18, a single "missing state" reply from an RPC still drops relayer work. Two gaps:

1. No retry when a chain has only one RPC URL. With a single entry in rpc_urls, the EVM client is a plain HTTP RootProvider (tesseract/messaging/evm/src/lib.rs ~361–379): no fallback_service, no ErrorResponseAsFailure ([tesseract]: don't drop an inbound range when its proof query fails, and don't let an RPC error beat another endpoint's answer #1435), and no retry. A load balancer such as dRPC routes requests across a mix of archive and pruned nodes. One request that lands on a pruned node fails the call, although an immediate retry would probably succeed.
2. GET requests are dropped when a proof query fails. tesseract/messaging/messaging/src/get_requests.rs (~95–112 and the storage-proof branch right after) logs Failed to fetch proofs for get requests: … and continues. The GET request is never parked or retried, so [tesseract]: don't drop an inbound range when its proof query fails, and don't let an RPC error beat another endpoint's answer #1435's park-and-retry does not cover it.

What happened

The SDK live CI job (run 40040673957, job 120712315122, PR #1440) hung on Run SDK test - get request, a GET self-delivery from BSC Chapel to Polygon Amoy. The testnet relayer (v2.4.18, BSC Chapel on a single dRPC URL) logged:

23:57:44  INFO messaging-inbound: Got 1 get_requests from EVM-97
23:57:44 TRACE messaging-inbound: Fetching source proofs for 1 get_requests from EVM-97
23:57:45 ERROR messaging-inbound: Failed to fetch proofs for get requests: server returned an error response: error code -32000: historical state … is not available

After that it logged nothing more for that request. In the same run, a POST request whose proof failed was parked by #1435 and delivered about two minutes later:

23:50:45 ERROR messaging-inbound: Could not build proofs for this range, parked its requests to be proved at a newer height source=Evm(97) … parked=1 err=deserialization error: invalid type: null, …
23:52:34  INFO messaging-inbound: Evm(97)->EVM-80002: 1 messages,

Provider behaviour (eth_getProof on BSC Chapel, 6 tries per depth)

Alchemy, like Dwellir, keeps only ~128 blocks of state and fails fast beyond that. dRPC can serve older proofs, but only some of the time: the outcome depends on which upstream node the request reaches. It reports misses either as -32000 historical state … is not available (HTTP 200) or as HTTP 500 Temporary internal error. Hyperbridge proves BSC Chapel heights that are often more than 128 blocks old (about 57 s at 0.45 s per block), so these misses happen regularly.

Suggested fixes

1. Retry transient RPC errors with backoff, for single-URL and multi-URL providers alike. Add a retry layer with a backoff of 250 ms, 500 ms, 1 s, 2 s: up to 4 retries after the first attempt.
   • Single URL: wrap the plain HTTP provider.
   • Multiple URLs: wrap fallback_service from the outside, so each retry runs a fresh parallel round across every endpoint. A load balancer such as dRPC then routes each round to a different upstream node. Today FallbackService makes exactly one round, with no retry and no backoff: when every endpoint misses, the call fails.
   • Retry only on transient errors:
     • -32000 messages containing historical state, missing trie node or header not found;
     • HTTP 5xx, including dRPC's Temporary internal error;
     • HTTP 429 and request timeouts.
   • Never retry deterministic errors: reverts from eth_call / eth_estimateGas, invalid params, nonce too low. For eth_sendRawTransaction, treat already known as success, and do not blindly resend after an ambiguous failure.
   • Bound the whole call by an overall time budget. A slow upstream already takes up to ~30 s per round for deep proofs, so the retries must not multiply that into minutes.
2. Park and retry GET requests the same way [tesseract]: don't drop an inbound range when its proof query fails, and don't let an RPC error beat another endpoint's answer #1435 does for POST requests. When query_state_proof fails for the source proof or the storage proof in get_requests.rs, keep the GET request and re-prove it at the next StateMachineUpdated height instead of dropping it.

Workaround applied on our testnet relayer

[bsc].rpc_urls lists the same dRPC URL three times. That forces the multi-URL fallback_service path, so each call becomes three parallel dRPC requests, and with #1435 an error no longer beats a success. This is a stopgap: it also makes the per-URL byzantine quorum check meaningless for that chain (identical providers), and it does not cover the GET path.