Profiles generated with the ebpf-profiler emitted from the OTEL pipeline have an invalid value at the 0-th position in the attribute table map. The conformance check tool is flagging the issue correctly.
Edit: seems to be due to grpc (un-)marshalling.
Error:
setup_3_grpc_file/out/profiles_1776370937817061840.pb: conformance checks failed: dictionary: attribute_table: must have zero value &v1development.KeyValueAndUnit
When debugging I saw the following:
Expected new(T):
v1development.KeyValueAndUnit {state: impl.MessageState {NoUnkeyedLiterals: pragma.NoUnkeyedLiterals {}, DoNotCompare: pragma.DoNotCompare [], DoNotCopy: pragma.DoNotCopy [], atomicMessageInfo: *impl.MessageInfo nil}, sizeCache: 0, unknownFields: []uint8 len: 0, cap: 0, nil, KeyStrindex: 0,
Value: *v1.AnyValue nil, UnitStrindex: 0}
Actual table[0]:
*v1development.KeyValueAndUnit {state: impl.MessageState {NoUnkeyedLiterals: pragma.NoUnkeyedLiterals {}, DoNotCompare: pragma.DoNotCompare [], DoNotCopy: pragma.DoNotCopy [], atomicMessageInfo: *impl.MessageInfo nil}, sizeCache: 0, unknownFields: []uint8 len: 0, cap: 0, nil, KeyStrindex: 0,
Value: *v1.AnyValue {state: (*"impl.MessageState")(0x3d1efaa5040), sizeCache: 0, unknownFields: []uint8 len: 0, cap: 0, nil, Value: v1.isAnyValue_Value nil}, UnitStrindex: 0}
Value looks empty, but is not in fact nil.
The particular failing check: https://github.com/open-telemetry/sig-profiling/blob/main/profcheck/check.go#L376
Writing to json file nets the following:
json: "attributeTable":[{"value":{}}
But I also tested capturing the grpc request and unmarshalling - same json representation:
pb -> inspect -> out.json: "attributeTable":[{"value": {}},
where inspect is something I quickly cobbled together to print the .pb files.
Reproduced here: https://github.com/bmarinov/ebpf-profiler-pipeline-dbg/tree/main/repro/setup_3_grpc_file
Sample file (otel grpc exporter -> .pb -> inspect command from the linked repo.
profiles_1776376300858502811.pb.json
I managed to get the same error with all four scenarios. Most of this is unrelated to this particular issue, but all four setups and OTEL pipelines end up with the same invalid profiles:
- setup_1: profiler -> collector (filter) -> json file
- setup_2: profiler -> json file
- setup_3:
- profiler -> grpc endpoint -> pb file -> conformance tool
- profiler -> grpc endpoint -> pb file -> json (main.go -> inspect)
- setup_4: profiler -> filter -> pb file
Profiles
generated with the ebpf-profileremitted from the OTEL pipeline have an invalid value at the 0-th position in the attribute table map. The conformance check tool is flagging the issue correctly.Error:
When debugging I saw the following:
Expected
new(T):Actual
table[0]:Value looks empty, but is not in fact nil.
The particular failing check: https://github.com/open-telemetry/sig-profiling/blob/main/profcheck/check.go#L376
Writing to json file nets the following:
json:
"attributeTable":[{"value":{}}But I also tested capturing the grpc request and unmarshalling - same json representation:
pb -> inspect -> out.json:
"attributeTable":[{"value": {}},where inspect is something I quickly cobbled together to print the .pb files.
Reproduced here: https://github.com/bmarinov/ebpf-profiler-pipeline-dbg/tree/main/repro/setup_3_grpc_file
Sample file (otel grpc exporter -> .pb -> inspect command from the linked repo.
profiles_1776376300858502811.pb.json
I managed to get the same error with all four scenarios. Most of this is unrelated to this particular issue, but all four setups and OTEL pipelines end up with the same invalid profiles: