`json.load` should document that it uses `json.loads`

[godlygeek]

Documentation

json.load is implemented in terms of json.loads. This is surprising: typically APIs that accept file objects work incrementally and don't require buffering the entire contents of the file in memory (and in fact, json.dump does work incrementally and doesn't buffer the entire contents of the file in memory).

The documentation doesn't mention that json.load reads the entire file into memory. It only says that the file-like passed to load needs to support .read(), but of course all readable file-likes support .read(). It doesn't say that .read() will be called only once and with no arguments.

If anything, the current documentation heavily implies that loads is implemented in terms of load, rather than the reverse. Currently all of the parameters are documented for json.load, and json.loads only refers to that documentation by saying:

Identical to load(), but instead of a file-like object, deserialize s (a str, bytes or bytearray instance containing a JSON document)

To me, this implies that loads behaves as though by load(StringIO(s)).

This documentation should be reversed: all of the documentation that currently lives on json.load should be moved to json.loads, and the docs for json.load should be updated to say:

Identical to loads, passing fp.read() as the string to decode.

[dagangtj]

I'll work on this documentation fix. Assigning to myself.

[sergey-miryanov]

This documentation should be reversed: all of the documentation that currently lives on json.load should be moved to json.loads, and the docs for json.load should be updated to say:

I don't think it's worth changing. It doesn't add any clarity. Even if json.load loads the entire file into memory, that's just an implementation detail that could change in the future.

[godlygeek]

It doesn't add any clarity.

It clarifies the current behavior. Users will be able to learn about this sharp edge by reading the docs rather than the source code.

Things are currently documented as though load is the main function and loads is a thin wrapper, when in fact the opposite is true. I found out about this while investigating why a program's peak memory usage was almost double what I expected it to be, and the answer turned out to be that an API that appears to perform incremental reads actually buffered an entire file in memory. Everyone else I've pointed this out to has been surprised as well, so I think it ought to be documented.

Even if json.load loads the entire file into memory, that's just an implementation detail that could change in the future.

Sure, but of course the docs could change in the future as well.

If this is an implementation detail, I think it would be fine to document it as such, though. But, I still think it's important to document, because I believe it's something important to know when choosing which of the two load methods to use (and indeed, possibly when choosing whether to use the json module at all or to reach outside the stdlib).

[sergey-miryanov]

If we document implementation details, they become part of the public interface. We can't change a public interface without a deprecation period, see https://peps.python.org/pep-0387/.

[picnixz]

I agree that this is an implementation detail. What can be done however is to make an incremental decoder but this would be a separate issue (which may already exists). I also do not wish to document this detail for now.

[godlygeek]

The docs currently say that json.loads is

Identical to load(), but instead of a file-like object, deserialize s (a str, bytes or bytearray instance containing a JSON document) to a Python object using this conversion table.

Surely we could instead say that json.load is

Identical to loads(), but instead of a str, bytes or bytearray instance, deserialize the contents of fp (a .read()-supporting text file or binary file containing the JSON document to be deserialized).

and move the documentation of the parameters from json.load to json.loads.

That corrects the implication, without actually stating outright that there will be a single call to read, even as a CPython implementation detail. It will correct the misconception that people reading the docs currently have, without committing to anything.

[picnixz]

I do not know whether it is really necessaey. It will not help you know about the fact that you might have a higher-than-expected memory consumption.

[picnixz]

FTR, I think we moved around the docs of those functions in the past (or maybe wanted to) but I do not remember whether there was a final decision about which function to use for documenting the parameters.