Behavior of substructure endianness

[DenisD3D]

Hello,

Is it possible to have some more information on the inheritance of endianness across sub structures ?
I've made the following test:

from caterpillar.py import Dynamic, struct, uint32, LittleEndian, BigEndian, unpack, f, getstruct

@struct
class MyStruct:
    my_uint: uint32

@struct(order=BigEndian)
class Response:
    little: LittleEndian + getstruct(MyStruct) # or f[MyStruct, MyStruct, LittleEndian]
    big: BigEndian + getstruct(MyStruct) # or f[MyStruct, MyStruct, BigEndian]
    inherit: MyStruct

my_uint = 0x01020304
raw = my_uint.to_bytes(4, "little") + my_uint.to_bytes(4, "big") + my_uint.to_bytes(4, "big")

print("Raw: 0x" + raw.hex())     # Raw: 0x040302010102030401020304     (my_uint8 in little, big & big)

result = unpack(Response, raw)

print(f"Little-endian: 0x{result.little.my_uint:02X}")
print(f"Big-endian: 0x{result.big.my_uint:02X}")
print(f"Inherit (Little): 0x{result.inherit.my_uint:02X}")

# Little-endian: 0x1020304     <== Expected output: my_uint
# Big-endian: 0x4030201        <== Decoded as little != my_uint ==> How to decode as Big without changing MyStruct ?
# Inherit (Little): 0x4030201    <== Should decode with same endianness as Response (BigEndian) ==> Here != my_uint

As you can see, MyStruct never inherit the endianness from the parent context. I also tried using @struct(order=Dynamic) on MyStruct. The only way I found to have MyStruct decode in BigEndian is to annotate it with @struct(order=BigEndian).

How can I make a structure inherit the endianness ?

My use case: I have some structures that are sometimes transmitted in little endian and sometimes in big. They contain the same data and change only by endianness. My current workaround is to duplicated the structure as ReversedMyStruct but it isn't pretty.
I believe that if order=None / not provided, a structure should inherit the parent context endianness pretty much like uint32 behave.

Best regards,
Denis

[MatrixEditor]

Hi @DenisD3D ,

thanks for reaching out! Endianess is kind of tricky with this library:

1. You can only apply Endianess to fields in a struct, not the struct itself. When you use @struct with a custom byteorder, that endianess is simply applied to all defined fields at eval time. In your case all you did was creating a new Field object in the Response class that had no effect on the MyStruct.my_uint field.
2. Currently, it is not possible to change the endianess of a struct via a simple BigEndian + <type> due to the internal model. It would require creating a copy of that struct (essentially what your were doing manually)

However, there is a quick solution here: the Dynamic byteorder.

from caterpillar.py import (
    CTX_ORDER,
    Action,
    BigEndian,
    Dynamic,
    Invisible,
    LittleEndian,
    StructDefMixin,
    f,
    struct,
    uint32,
)
from caterpillar.types import uint32_t


# You can also create a custom class taking the ByteOrder as an argument
# (refer to Action docstring for more explanation)
def apply_be(context):
    context._root[CTX_ORDER] = BigEndian


@struct(order=Dynamic)
class MyStruct(StructDefMixin):
    my_uint: uint32_t
    # the next field will ALWAYS decode as little endian
    my_uint_le: f[int, LittleEndian + uint32]


@struct(order=BigEndian, kw_only=True)
class Response(StructDefMixin):
    # fmt: off
    little: MyStruct
    # in order to change the endianess for integer fields, we have to set
    # it globally. Fields (not struct types) that are combined with
    # -------------------------------------------------------------------------# every field below that is annotated with Dynamic
    _order_big: f[None, Action(both=apply_be)] = Invisible()                   # will be using the BigEndian byteorder
    big: MyStruct                                                              #
    inherit: MyStruct                                                          #
    # -------------------------------------------------------------------------#


my_uint = 0x01020304
raw = (
    my_uint.to_bytes(4, "little")
    + my_uint.to_bytes(4, "little")
    # ---
    + my_uint.to_bytes(4, "big")
    + my_uint.to_bytes(4, "little")
    # ---
    + my_uint.to_bytes(4, "big")
    + my_uint.to_bytes(4, "big") # my_uint_le should be 0x4030201 then
)

# Raw: 0x040302010102030401020304     (my_uint8 in little, big & big)
print("Raw: 0x" + raw.hex())
result = Response.from_bytes(raw)
# fmt: off
print(f"Little-endian: 0x{result.little.my_uint:02X}, le_int: 0x{result.little.my_uint_le:02X}")
print(f"Big-endian: 0x{result.big.my_uint:02X}, le_int: 0x{result.big.my_uint_le:02X}")
print(f"Inherit (Little): 0x{result.inherit.my_uint:02X}, le_int: 0x{result.inherit.my_uint_le:02X}")

# BEFORE
# Little-endian: 0x1020304     <== Expected output: my_uint
# Big-endian: 0x4030201        <== Decoded as little != my_uint ==> How to decode as Big without changing MyStruct ?
# Inherit (Little): 0x4030201  <== Should decode with same endianness as Response (BigEndian) ==> Here != my_uint

# NOW
# Little-endian: 0x1020304, le_int: 0x1020304
# Big-endian: 0x1020304, le_int: 0x1020304
# Inherit (Little): 0x1020304, le_int: 0x4030201

assert result.to_bytes() == raw

I hope that helps a bit.

[DenisD3D]

Hello @MatrixEditor,
Thanks for the quick reply and the working demo.

While the result is working as expected, the syntax isn't really straight forward.
One of the keypoint of your library was that the syntax was really easy to understand, even without knowing the library.
I have some complex nested structure and I believe _order_big: f[None, Action(both=apply_be)] = Invisible() will have unexpected side effect, with new user not understanding the reasons.

It could be awesome to have structure inherit parent endianness in a future version of the library. Either with a direct inheritance like fields or with a custom endianness (VariableEndianness ?). Bellow are some configuration I would be using if it was available.

@struct(order=None) # or VariableEndianness
class MyStruct:
    my_uint: uint32

@struct(order=BigEndian)
class Response_V1():
    a: MyStruct # my_uint decoded in big

@struct(order=LittleEndian)
class Response_V2():
    c: MyStruct # my_uint decoded in little

@struct
class Response_V3():
    c: MyStruct # my_uint decoded as unpack(.... order=XXX)

@struct(order=BigEndian)
class Response_V4():
    c: LittleEndian + MyStruct # my_uint decoded as little

@struct(order=BigEndian)
class Response_V4():
    c: LittleEndian + MyStruct[10] # my_uint decoded as little

For everything else, I really enjoy using your library.

Best,
Denis

[MatrixEditor]

You're right, action syntax should only be used in edge cases when nothing else is applicable.

Adding a new byteorder without breaking previous versions is a bit tricky: I pushed a dev branch with a poc for an Inherit byteorder that uses the defined endian from the parent context.

For instance,

# Outer(BE) -> Middle(LE, explicit) -> Inner(Inherit)
# Inner will use Middle's LE
@struct(order=Inherit)
class Inner:
    v: uint32_t

@struct(order=LittleEndian)
class Middle:
    v: Inner

@struct(order=BigEndian)
class Outer:
    v: Middle

data = bytes.fromhex("11223344")
obj = unpack(Outer, data)
assert obj.v.v.v == 0x44332211
assert pack(obj) == data

The <endian> + <type> works too with this new byteorder type. Maybe this is what you were looking for - if you have more edge cases for the endian, please share them so I can try to enhance the poc even further.

[DenisD3D]

Hello @MatrixEditor,
Thanks you very much for the POC and your reactivity, it looks awesome.

I can't try on the my big codebase right now, but I've written some test code and it's working really well.

The only thing I could see missing in the current POC is the ability to set Inherit as the default behavior.
While I agree on preserving the existing behavior by default, I would love a way to override it if needed.

@struct  # Order not specified
class Inner:
    v: uint32


@struct(order=LittleEndian)
class Middle1:
    v: Inner


@struct(order=BigEndian)
class Outer:
    v: Middle1


data = bytes.fromhex("11223344")
obj = unpack(
    Outer,
    data,
    order=BigEndian,
    default_struct_order=Inherit,  # Specify that by default structure order is inherited
)
print(obj)
assert obj.v.v.v == 0x44332211
assert pack(obj) == data


# Another option is O_DEFAULT_STRUCT_ORDER.value = Inherit 

Maybe it's possible with what is already available but I haven't succeeded.

Denis

[MatrixEditor]

HI @DenisD3D ,

while this feature is now somehow needed, I won't be able to get back to this in the next few days. I've already tested some approaches -this feature requires a lot more careful considerations before shipping it to the library. I will post whenever an implementation is ready.

[MatrixEditor]

Hi @DenisD3D,

I've pushed some changes that introduce a global "default" struct endianess when packing and unpacking. It can be applied using the default_struct_order extra argument or via O_DEFAULT_STRUCT_ENDIAN.value directly. However, I had to normalize the byteorder resolution in some places which also introduced some performance overhead - I'm still evaluating whether this trade-off can be overlooked.

The default_struct_order now also accepts little and big endian directly. These are then applied on "struct"-level and take precedence over the custom field-byteorder. Resolution is as follows:

field.order                                # 1. Endian + <struct> direct assignment 
if field and field.has_order()
else (
    context.get(CTX_ORDER)                 # 2. set via default_struct_order=... and enclosed in a struct with order=None 
    or getattr(obj, ATTR_BYTEORDER, None)  # 3. set via self.__byteorder__ = ...
    or O_DEFAULT_ENDIAN.value              # 4. set via order=... in pack() or unpack()
    or LittleEndian                        # 5. fallback
)

[DenisD3D]

Hello @MatrixEditor,
I've made some more testing and everything is working well for me in my codebase.
I only noticed a small typing issue:

decoded = unpack(
    LittleEndian + MyStruct,
    payload,
    order=BigEndian,
    default_struct_order=Inherit,
)
Operator "+" not supported for types "ByteOrder" and "type[MyStruct]"Pylance[reportOperatorIssue](https://github.com/microsoft/pylance-release/blob/main/docs/diagnostics/reportOperatorIssue.md)

Do you know when you might merge the POC ?
Best regards,
Denis