bound the request body size read by RPCServletUtils - #10384
bound the request body size read by RPCServletUtils#10384alphacharlie-dev wants to merge 1 commit into
Conversation
readContent copies the request InputStream into a ByteArrayOutputStream in a loop that only stops at end-of-stream, then materialises the result as a String, so peak memory is roughly twice the body size. No Content-Length ceiling and no running byte total are applied anywhere on the path, and the method is reached without authentication from both GWT-RPC (readContentAsGwtRpc) and RequestFactory (RequestFactoryServlet), so a single large POST can exhaust the heap. Servlet container max-post-size settings do not usually cover text/x-gwt-rpc or application/json. Reject up front when Content-Length already exceeds the limit, and keep a running total inside the loop so chunked transfers and an understated header are bounded as well. The limit defaults to 1 MiB and can be raised with the gwt.rpc.maxRequestBodyBytes system property; an unparseable value falls back to the default rather than disabling the check. Applications that send payloads larger than 1 MiB will need to set that property.
|
Can you discuss more exactly how you confirmed this? Unless I'm very much mistaken, the servlet container should be managing the content-length header (or chunked encoding, which means no header, or h2, which means the size of the h2 DATA frame is part of the binary format and not a proper header) and should be transparent to the application (or in our case, the toolkit code that decodes the stream into objects)? Heap exhaustion shouldn't be necessary to confirm, but the ServletRequest.getInputStream() should be correctly bounded to the actual request size. If you discover that it isn't, that seems like it must be a bug in the servlet container, not GWT, as keep-alive, pipelining, etc cannot work correctly with incorrect content-length. I suspect that if there is a content vs |
|
Hi Colin — let me take your points one at a time. 1. "Can you discuss more exactly how you confirmed this?" You're right that heap exhaustion isn't required to confirm it — so I didn't actually OOM anything. The test is a deliberately bounded 8 MB probe. It calls On unpatched That proves the read loop has no break conditioned on accumulated size: while (true) {
int byteCount = in.read(buffer);
if (byteCount == -1) break; // only EOF stops it — never size
out.write(buffer, 0, byteCount);
}
return new String(out.toByteArray(), charset);Whatever the stream yields, GWT buffers — so a multi-GB body scales straight to OOM (recorded analytically; no need to actually crash a box). After the patch, the same probe is rejected: And the regression suite is clean — Worth noting: the test isolates GWT's own code path — it isn't claiming a container misbehaves, it just shows that GWT itself has no bound. Whether the container supplies one is points 3–5 below. 2. "The container manages content-length / chunked / h2, and that should be transparent to the app." You're right that the container handles the transport framing faithfully — I'm not disputing that. The disagreement is about what it bounds. The container guarantees it delivers exactly the bytes the client framed; it does not guarantee those bytes are small. The size is chosen by whoever sends the request. "Bounded to the actual request size" is true, but the actual request size is attacker-controlled, and GWT buffers all of it. HTTP/2 is the same story — the container reassembles all the DATA frames and hands the total over; the total is still attacker-controlled. 3. " It is — but that size is whatever the client declared or streamed, and two attacker paths need no container bug:
4. "If it isn't bounded, that's a container bug — keep-alive and pipelining can't work with an incorrect content-length." I agree — and that's exactly the point: those mechanisms require a correct content-length, so a malicious client has every reason to send a correct one, just very large. The attack doesn't use an incorrect length, so the keep-alive/pipelining argument doesn't apply to it. The client follows the transport rules and still fills the heap, because nothing on the GWT side says "a request this big isn't legitimate." 5. "A content vs content-length mismatch would be a bug in the servlet container, not gwt-rpc." Agreed — but I'm not relying on a mismatch. The probe uses the no-Content-Length (chunked) case, and the realistic attack uses either that or an honest large Content-Length. In both, the bytes delivered match what was framed; the container is doing its job. The only problem is that GWT buffers an unbounded amount before it ever looks at the payload. (And for completeness, the only existing size check in this file — So why fix it in GWT rather than rely on the container? Container body limits exist (Tomcat |
|
So this is just "If an attacker sends 1gb, the server receives 1gb, and that's bad"? You have described (but not provided?) a test case where a ServletInputStream instance is created that provides too much data, but no real examples (though you offered, and I asked).
Integer.MAX_VALUE is only 2gb, that's going to crash a server? BAOS will definitely throw OOM if you ask for that much (real limit is about Integer.MAX_VALUE - 2 or so), and asking for a little less might take some time (getting into DoS territory). Compression enters the picture and lets the client send less to use more bytes - but I'm still not sure how we can comprehensively solve both usability and security sides here: 1mb is a "medium sized" message to send over gwt-rpc, I definitely wouldn't feel comfortably imposing limits here until we had some kind of super-linear growth bug. I think there are several appropriate places to define a limit like this without baking assumptions like this into GWT itself:
I think a reasonable case could be made both for and against just about any arbitrary size you could specify here (100k? 10mb? 100mb?), and arguably it would need to be defined per-servlet, possibly subject to some access controls (rather than a single system property to rule them all). I don't think we can reasonably take this change with some specific, concrete issues. |
|
Colin — fair, and I think you're right on the substance. Let me concede the points rather than keep defending the change as-is:
The one thing I'd still flag — not to relitigate, but because it informs what would be useful — is that the "the container already bounds it" assumption has a real gap for this content type: Tomcat's To that end: I'll skip the end-to-end PoC. I'd been about to build one, but you've correctly placed linear DoS outside GWT's scope, so the PoC would only demonstrate the thing you've already said isn't GWT's job — not worth either of our time. What I'd like to do instead is gut the PR down to something purely opt-in, with no default and no behavior change unless an app asks for it:
No system property, no default cap, nothing breaks. If even that's more than you want in-framework, I'm happy to close this and drop the pattern as a wiki/gist writeup instead — let me know which you'd prefer and I'll rework (or close) accordingly. |
In case you're using GWT yourself, what problem does it solve for you, that you can't solve with the container settings mentioned above? Increasing complexity of the system for something that can be easily achieved in other ways is IMHO not worth it. It might make sense to make a documentation PR mentioning this concern and the Jetty / Tomcat properties that allow to mitigate it, possibly as a paragraph in https://www.gwtproject.org/doc/latest/DevGuideServerCommunication.html |
RPCServletUtils.readContentreads the request body into aByteArrayOutputStreamin a loop thatonly terminates at end-of-stream, then materialises the result as a
String:Nothing consults
Content-Lengthand no running total is kept, so the peak footprint is roughlytwice the body size with no ceiling. The method is reached without authentication from both
protocols —
AbstractRemoteServiceServlet:182→readContentAsGwtRpc, andRequestFactoryServlet:129— so one large POST can drive the JVM toOutOfMemoryError.Servlet container
max-post-sizesettings generally don't help, because they apply toform-encoded bodies rather than
text/x-gwt-rpcorapplication/json. The existingexceedsUncompressedContentLengthLimithelper gates response compression and has no bearing here.The change
Reject up front when
Content-Lengthalready exceeds the limit, and keep a running total insidethe loop so chunked transfers and an understated header are bounded too. Default 1 MiB, raisable
via the
gwt.rpc.maxRequestBodyBytessystem property; an unparseable value falls back to thedefault rather than disabling the check.
Compatibility
Applications sending payloads above 1 MiB will need that property set. 1 MiB is a proposal, not a
measured threshold — you know real-world GWT-RPC payload sizes far better than I do, and I'd
rather take a maintainer's number than defend mine.
Testing
RPCServletUtilsTestpasses unchanged, including all sixtestContentLength*buffer-boundarycases that exercise this exact method. I also checked the boundary directly: a body of exactly
1 MiB is accepted, one byte over is rejected.
Full JRE suite (
ant -f user/build.xml test.nongwt): 25 suites / 2701 tests / 0 failures,identical to the
mainbaseline. I did not run the GWT-mode/browser suites.Provenance
Found during a security review of GWT 2.13.1, confirmed still present on
main. I have a standalonereproduction (a generator
ServletInputStreamfeeding 8 MB, asserting it is buffered in full) anda test asserting the secure behaviour, both runnable against a plain checkout — happy to attach or
open a matching issue if that's useful. I have not attempted to demonstrate actual heap exhaustion;
the scaling argument is analytical.
Reported to the Google OSS VRP, which declined it on repository-tier grounds rather than on the
merits and suggested bringing it here.