proof: Bound deserialize against wire-length OOM

[nervana21]

Description and Notes

Proof::deserialize previously used untrusted u64 length prefixes to size Vec::with_capacity for targets and hashes. A hostile payload could claim a huge count and OOM the process before any proof bytes were validated.

This PR rejects oversized prefixes before allocation. Target and hash counts above MAX_PROOF_DESERIALIZE_COUNT ((4 * 1024³) / 32 == 1 << 27, ~4 GiB of 32-byte hashes) return ProofError::OversizedAllocation. Cap is deserialize-only. Proof::serialize and in-memory construction are unchanged.

Changelog

proof: Bound deserialize against wire-length OOM
 - reject target/hash counts above MAX_PROOF_DESERIALIZE_COUNT before allocating
 - return ProofError::OversizedAllocation for oversized length prefixes
 - add unit tests for excessive target and hash caps

To verify changes

• cargo test test_deserialize_rejects
• cargo test --lib

[Davidson-Souza]

I thought about this a little more. Ideally this library won't use Bitcoin specific values — we do have other consumers that aren't doing the default Bitcoin stuff we do in Floresta.

Floresta doesn't even use this deserializer internally. I like the patch, but I would just pick a big number — like 4GB — and stick with it, rather than computing Bitcoin-specific upper bounds.

Edit: and probably make this explicit in the docs.

[nervana21]

Updates:

• Addressed @Davidson-Souza's feedback.