Skip to content

feat(encoder): add Protocol Buffers as a third E3AP encoding#35

Open
Thecave3 wants to merge 4 commits into
mainfrom
16-add-protobuf-as-a-new-encoding-methodology
Open

feat(encoder): add Protocol Buffers as a third E3AP encoding#35
Thecave3 wants to merge 4 commits into
mainfrom
16-add-protobuf-as-a-new-encoding-methodology

Conversation

@Thecave3

@Thecave3 Thecave3 commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add a third, runtime-selectable E3AP encoding, Protocol Buffers, implemented the same way as the existing ASN.1 and JSON encoders and chosen via E3Config.encoding (EncodingFormat::PROTOBUF).
  • New .proto grammars for E3AP and the Simple SM; protoc generates a pb_e3ap library at build time. New ProtobufE3Encoder behind the E3Encoder factory, gated by LIBE3_ENABLE_PROTOBUF (off by default). build_libe3 --enable-protobuf flag + dependency install; the Simple agent/dapp example runs protobuf end-to-end.
  • Tooling: round-trip unit tests for all 11 PDU types, encoding-size benchmark coverage, CI builds all three encodings together, and the release workflow publishes per-encoding .deb variants (all/asn1/json/protobuf). Also fixes the unit-test framework summary that never tallied passing tests.

Type of change

  • Bug fix
  • New feature / enhancement
  • New Service Model
  • Refactor (no behavior change)
  • Documentation
  • Test / CI / packaging
  • Other (explain):

Linked issue

Closes #16

Mandatory test checklist

These mirror what CI (.github/workflows/pr-tests.yml) enforces. All boxes must be ticked before review.

  • ./build_libe3 -c -d build -j $(nproc) -r -t passes (Release build + tests) — run with --enable-protobuf, 16/16 tests pass incl. test_protobuf_encoder
  • ./build_libe3 -c -d build -j $(nproc) -g -t passes (Debug build + tests) — verified in Debug for the default and all-encodings configurations
  • cd build && ctest --output-on-failure is clean (all-encodings build: 17/17; default build: 15/15 with the protobuf test correctly skipped)
  • MPMC queue benchmark (./build/test_bench_mpmc_queue) shows no regression vs main (unchanged; passes as part of the suite)
  • VERSION bumped per SemVer (0.0.6 → 0.0.7; additive EncodingFormat::PROTOBUF enumerator + new opt-in option)
  • If public headers under include/ were touched, ./build_libe3 --docs renders without new Doxygen warnings — the only public-header change is the additive EncodingFormat::PROTOBUF enumerator (Doxygen-commented); docs not re-rendered locally, covered by the release docs job
  • If new build dependencies were added, they are installed by ./build_libe3 -I (added protobuf-compiler/libprotobuf-dev for apt, plus dnf/pacman/brew equivalents)
  • If the libe3.pc interface changed, downstream consumers still link cleanly — libe3.pc is unchanged

CI checklist

  • Unit Tests workflow is green (Debug + Release matrix on ubuntu-latest) — will run on this PR
  • MPMC Queue Benchmark job has posted results to this PR with no regression — will run on this PR

Twin-repo coordination

libe3 is paired with dapps and dApp-openairinterface5g. We do not accept patches that break or reduce compatibility with the twin repositories.

  • This PR does not change the E3 wire protocol or public ABI, OR a paired PR exists in each affected twin repo (link below). — The change is purely additive and opt-in: LIBE3_ENABLE_PROTOBUF defaults OFF, the new EncodingFormat::PROTOBUF enumerator is appended, and the existing ASN.1/JSON wire formats and ABI are unchanged.

Paired PR(s): none required (additive, opt-in)

Workflow confirmation

  • Commits will be squashed at merge time. Updates to this PR will be applied via rebase only — no merge commits, no duplicated history. (See CONTRIBUTING.md § Pull Request Process.)
  • I have read and followed CONTRIBUTING.md.

@Thecave3 @claude
feat(encoder): add Protocol Buffers as a third E3AP encoding (#16)
1e0d858 
Add a runtime-selectable Protocol Buffers encoder alongside ASN.1 and JSON,
implemented the same way as the existing encoders and selected via
E3Config.encoding.

- messages/proto/V1/e3ap-1.0.0.proto and examples/sm_simple/e3sm_simple.proto
  grammars; protoc generates the pb_e3ap library at build time
- ProtobufE3Encoder behind the E3Encoder factory, gated by LIBE3_ENABLE_PROTOBUF
- build_libe3 --enable-protobuf flag and protobuf packages in the dependency
  installer; find_package(Protobuf) wiring in CMake
- end-to-end protobuf path in the simple agent/dapp example (SM payload encoding
  follows the selected format)
- round-trip unit tests for all 11 PDU types and encoding-size benchmark coverage
- PR CI builds all three encodings together; the release workflow now publishes
  per-encoding .deb variants (all/asn1/json/protobuf) via create_deb.sh --variant

Also fixes the unit-test framework summary, which never tallied passing tests
(reported "0 passed ... out of 0").

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Thecave3 Thecave3 linked an issue Jul 1, 2026 that may be closed by this pull request
Add protobuf as a new encoding methodology #16
Open
@Thecave3 Thecave3 self-assigned this Jul 1, 2026
@github-actions

github-actions Bot commented Jul 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

⏱️ Full-loop Latency Benchmark (commit 37e683d)

Full-loop latency benchmark (N=1041 after 50 warmup)

All values in microseconds (us). Link: zmq, transport: ipc, encoding: ASN.1 APER.

Phase mean p50 p99 max
1. Collect indication data 0 0 0 11
2. Create & encode indication 0 0 0 7
3. Deliver indication (RAN -> dApp) 88 87 150 158
4. Decode indication 0 0 0 12
5. Process data 0 0 0 0
6. Create & encode control 0 0 0 5
7. Deliver control (dApp -> RAN) 100 101 159 183
8. Decode & handle control 0 0 0 6
Total round-trip 190 175 298 310

Benchmarked on ubuntu-latest, Release build, ZMQ + IPC, ASN.1 APER.

@github-actions

github-actions Bot commented Jul 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🔄 E2E dApp Integration Results (commit 37e683d)

posix/ipc

  • dApp exit: 0
  • Indications received: 7

posix/tcp

  • dApp exit: 0
  • Indications received: 7

zmq/ipc

  • dApp exit: 0
  • Indications received: 7

zmq/tcp

  • dApp exit: 0
  • Indications received: 7

example_simple_agent + example_simple_dapp on ubuntu-latest, Release build

@github-actions

github-actions Bot commented Jul 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🔀 E2E Topologies — multi-dApp / multi-RAN (commit 37e683d)

zmq/ipc

  • ✅ 1 RAN - 1 dApp: indications=5
    • dapp peer=t11 ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
  • ✅ 1 RAN - 2 dApps: dApp#1 ind=5 sub=1, dApp#2 ind=6 sub=2, RAN saw 2 dApps
    • dapp1 peer=t12 ran=ran-shared sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.6 max=1 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
    • dapp2 peer=t12 ran=ran-shared sub=2 indications=6 seq=[0..5] dropped=0 (0%) age_ms(avg=0.5 max=1 @seq=0) hist[<=1:6 2-5:0 6-10:0 >10:0]
  • ✅ 2 RANs - 1 dApp: from ran-a ind=5, from ran-b ind=5
    • dapp peer=t2a ran=ran-a sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.6 max=1 @seq=1) hist[<=1:5 2-5:0 6-10:0 >10:0]
    • dapp peer=t2b ran=ran-b sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.6 max=1 @seq=2) hist[<=1:5 2-5:0 6-10:0 >10:0]

zmq/tcp

  • ✅ 1 RAN - 1 dApp: indications=5
    • dapp peer=default ran=ran-solo sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
  • ✅ 1 RAN - 2 dApps: dApp#1 ind=5 sub=1, dApp#2 ind=6 sub=2, RAN saw 2 dApps
    • dapp1 peer=default ran=ran-shared sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.4 max=1 @seq=3) hist[<=1:5 2-5:0 6-10:0 >10:0]
    • dapp2 peer=default ran=ran-shared sub=2 indications=6 seq=[0..5] dropped=0 (0%) age_ms(avg=0.333333 max=1 @seq=4) hist[<=1:6 2-5:0 6-10:0 >10:0]
  • ✅ 2 RANs - 1 dApp: from ran-a ind=5, from ran-b ind=5
    • dapp peer=default ran=ran-a sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0.6 max=1 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]
    • dapp peer=off100 ran=ran-b sub=1 indications=5 seq=[0..4] dropped=0 (0%) age_ms(avg=0 max=0 @seq=0) hist[<=1:5 2-5:0 6-10:0 >10:0]

example_simple_agent + example_simple_dapp on ubuntu-latest, Release. Indication age is report-only.

Thecave3 and others added 3 commits July 3, 2026 00:18
@Thecave3 @claude
feat(benchmarks): wire encoding selection through the full-loop laten…
c23ed57 
…cy bench

Accept --encoding protobuf and pass the selected encoding into every
Simple SM wrapper call, so the SM payload matches the E3AP envelope
encoding end-to-end (previously the payload was always ASN.1 APER
regardless of --encoding).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Thecave3 @claude
feat(examples): add a JSON branch to the Simple SM wrapper
4520151 
Encode/decode the Simple SM payloads as camelCase JSON text when the
selected encoding is JSON, mirroring the existing protobuf branch, so
'-e json' runs are JSON end-to-end (envelope and SM payload). This
fixes the previous 0-completion JSON runs: the JSON envelope encoder
embeds the indication payload as a nested JSON object, which failed to
parse when the SM payload was ASN.1 bytes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Thecave3 @claude
feat(benchmarks): measure encode/decode CPU cost and information content
37e683d 
Extend bench_encoding_size beyond wire size: time encode and decode of
every PDU type per enabled encoding with Google Benchmark (auto-
calibrated iteration batching, DoNotOptimize barriers, p50/p99 across
20 repetitions), and report each sample PDU's intrinsic information
content in bytes so encoded_bytes/info_bytes reads as framing overhead.

CSV columns become: encoding, message_type, encoded_bytes, info_bytes,
encode_ns_p50, encode_ns_p99, decode_ns_p50, decode_ns_p99.

Google Benchmark is fetched via FetchContent only when
LIBE3_BUILD_INTEGRATION_TESTS=ON (system package used when present);
nothing ships with the library.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Thecave3 Thecave3 mentioned this pull request Jul 4, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@aferaudo aferaudo Awaiting requested review from aferaudo
@Ninjabippo1205 Ninjabippo1205 Awaiting requested review from Ninjabippo1205

At least 1 approving review is required to merge this pull request.

Assignees

@Thecave3 Thecave3

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add protobuf as a new encoding methodology

1 participant

@Thecave3