Skip to content

Decoder and encoder limits that need a maintainer decision #1191

Open

Description

@slachiewicz

The hardening PRs linked from #969 leave a few questions open. They are choices about the intended contract rather than bugs with one obvious fix, so they are collected here instead of being decided inside a PR.

  1. Largest var-data length and numInGroup an encoder may write. The generated encoders and the Go struct encoder ([Go] Return an error when a struct-codec var-data length or group count exceeds its type's maximum #1158) stop at applicableMaxValue, which for an unsigned type without a schema maxValue is one below the type maximum (254 for uint8), because the type maximum is the null value. The C++ OTF encoder in [C++] Reject a numInGroup or var data length that does not fit its wire field in the OTF encoder #1159 allows the full type range (255 for uint8). Which limit should all encoders use?

  2. Schema names that are not valid identifiers. XmlSchemaParser.checkForValidName reports such names only as warnings, and names read from a .sbeir file are not checked. [Java] Reject generated file names that would leave the output directory #1181 stops a name from leaving the output directory, but a name like a b or x;y still reaches the generators. Should a name that is invalid in every target language be an error by default?

  3. Fields newer than the acting version in OTF. decodeFields in the Java and C++ OTF decoders hands every field to the listener, including fields added after the acting version, so a listener that reads them without checking the version reads beyond an older, shorter block. Should the decoders skip those fields, or is checking the version the listener's job?

  4. Groups whose fields are all constant. Such a group legitimately has blockLength 0, so numInGroup is not bounded by the remaining buffer and the Java DTO, Java OTF and C++ OTF decoders iterate up to the count type's maximum on a few bytes of input. Should numInGroup be capped, for example by the remaining buffer length when blockLength is 0?

  5. Decoding with bounds checks turned off. With agrona.disable.bounds.checks=true (Java) or SBE_NO_BOUNDS_CHECK (C++), several of the new length checks disappear along with the existing ones, including the DTO var-data sizing in [Java] Size DTO var-data byte arrays from a bounds-checked wrap #1151. Is that the intended contract for those switches, or should checks that prevent large allocations stay on?

  6. Composite members with their own sinceVersion. IrGenerator gives a primitive member of a composite used by a field the version max(field.sinceVersion, composite.sinceVersion) and drops the member's own sinceVersion; enum, set and nested composite members keep theirs. For <composite><type name="mantissa" primitiveType="int64"/><type name="exp" primitiveType="int8" sinceVersion="1"/></composite> on a version 0 field, a version 0 producer writes 8 bytes, but the generated accessors read exp unguarded at acting version 0, and the blockLength minimum in [Java] Reject a group blockLength below the fields of the acting version #1153, [Java] Check var-data length and group blockLength in the OTF decoder #1154, [C++] Reject a wire blockLength shorter than the fixed fields of the acting version #1176, [C] Reject a wire blockLength shorter than the fixed fields of the acting version #1177, [C++] Reject a wire blockLength shorter than the fixed fields of the acting version in the OTF decoder #1179, [Java] Reject a message blockLength shorter than the fields of the acting version #1183, [Go] Reject OTF group entries whose blockLength is below the fixed fields #1187 and [Go] Reject generated flyweight group headers whose blockLength is below the fixed fields #1188 is 9, so that valid version 0 block is rejected. Should the IR keep the member's sinceVersion (which also guards the accessor), or is a member sinceVersion on a primitive not meant to be supported?

  7. Decoding a length above maxValue. No decoder rejects a var-data length above the length type's maxValue today, while [Go] Return an error when a struct-codec var-data length or group count exceeds its type's maximum #1158 makes the Go encoder refuse to write one and the C++ put<X>(const char *, length) does not check it. Should decoders reject such lengths, and if so only when range checks are enabled, as the Go struct codec's other range checks are?

  8. E108 in the C++ codecs. It is used for "index >= count" and "buffer too short for next group index", and [C++] Reject a wire blockLength shorter than the fixed fields of the acting version #1176 adds "blockLength too short for group". Is one code for all three fine, or should the blockLength check get its own?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions