feat(kao): add support for bao kao testing framework by danielRep · Pull Request #363 · bao-project/bao-hypervisor

danielRep · 2026-05-15T10:01:46Z

Overview

This PR integrates Bao Kao (abbreviated as bkao, "to put to the test"), an end-to-end testing (and benchmarking) framework for the Bao Hypervisor. Bao Kao automates the full pipeline from source build to on-target execution: it fetches toolchains, builds guest workloads, generates Bao configuration files based on targeted setup from YAML descriptors, assembles a bootable image (with optional firmware), launches it on the target platform (emulated or physical), and captures serial output to determine pass/fail.

Two new submodules are introduced:

tests/bkao: submodule containing the Bao Kao framework
tests/benchs: OSYX close-source benchmarks

Platforms supported:

Virtual:
- qemu-aarch64-virt
- qemu-riscv64-virt
- fvp-a
- fvp-r
Physical
- zcu104
- s32z270
- tc4dx
- rh850 (partially)

Note

This PR supersedes #154. We have refactored completely the framework, removing nix support and leveraging only python to implement bao kao.

Source Tree Structure

  tests/
  ├── bkao/                   # Bao Kao framework (submodule)
  │   └── src/
  │       ├── bkao.py         # Main runner and orchestration entry point
  │       ├── platforms/      # Per-platform build and launch logic
  │       ├── firmware/       # ATF, OpenSBI, U-Boot constructors
  │       ├── guests/         # Guest workload builders (baremetal, ...)
  │       ├── hypervisor/     # Hypervisor builders including Bao and a config generator
  │       └── utils/          # Helpers
  ├── benchs/                 # Benchmarks workload repository (submodule)
  │   ├── src/benchmarks/
  │   │   ├── ctx-switch/     # Context-switch benchmark
  │   │   └── irq-lat/        # IRQ latency benchmark
  │   └── configs/            # Per-benchmark YAML platform configs
  └── tests/
      ├── configs/
      │   ├── baremetal/      # Baremetal guest configs (YAML + platform BSPs)
      │   ├── freertos/       # FreeRTOS guest configs
      │   └── linux/          # Linux guest configs
      └── src/
          ├── inc/            # Test framework headers (testf.h, asserts, commands)
          ├── boot.c          # Boot test
          ├── irq.c           # IRQ test 
          └── bao-test.mk     # Test build integration makefile

Tests Available

┌─────┬────────────┬──────────┬───────────┬──────────────────────────────────────────────┐
│ ID  │   Suite    │   Test   │   Setup   │                 Description                  │
├─────┼────────────┼──────────┼───────────┼──────────────────────────────────────────────┤
│ 100 │ BOOT_CHECK │ VM_BOOT  │ baremetal │ Check that baremetal guest boots             │
│     │            │          │           │ successfully                                 │
├─────┼────────────┼──────────┼───────────┼──────────────────────────────────────────────┤
│ 101 │ BOOT_CHECK │ CPU_BOOT │ baremetal │ Check that all CPUs on the baremetal guest   │
│     │            │          │           │ boot successfully                            │
├─────┼────────────┼──────────┼───────────┼──────────────────────────────────────────────┤
│ 200 │ IRQ_CHECK  │ TIMER    │ baremetal │ Check that timer interrupt is triggered and  │
│     │            │          │           │ handled successfully                         │
├─────┼────────────┼──────────┼───────────┼──────────────────────────────────────────────┤
│ 201 │ IRQ_CHECK  │ UART     │ baremetal │ Check that UART interrupt is triggered and   │
│     │            │          │           │ handled successfully                         │
└─────┴────────────┴──────────┴───────────┴──────────────────────────────────────────────┘

Tests are defined using the BAO_TEST(suite, test, setup, description) macro and discovered automatically at runtime by scanning tests/tests/src/*.c. Each test is tagged with a suite, a setup (which selects the VM configuration to use), and an ID used for selective execution.

IDs are computed as (file_index × 100) + test_index_within_file, where:

file_index — 1-based position of the .c file in alphabetical order within
tests/tests/src/
test_index — 0-based order of BAO_TEST(...) appearances within that file

So with the current files (boot.c, irq.c):

┌────────┬────────────┬─────────┬─────┐
│  File  │ file_index │ test_nr │ ID  │
├────────┼────────────┼─────────┼─────┤
│ boot.c │ 1          │ 0       │ 100 │
├────────┼────────────┼─────────┼─────┤
│ boot.c │ 1          │ 1       │ 101 │
├────────┼────────────┼─────────┼─────┤
│ irq.c  │ 2          │ 0       │ 200 │
├────────┼────────────┼─────────┼─────┤
│ irq.c  │ 2          │ 1       │ 201 │
└────────┴────────────┴─────────┴─────┘

Guest types supported in test configs: baremetal, freertos, linux. For now, we only target a baremetal setup.

How to Run (TLDR)

Via bao Makefile targets:
Run all tests for a platform:
make tests PLATFORM=<platform>

Run all benchmarks for a platform:
make benchs PLATFORM=<platform>

Other options only via calling bkao-py directly:

Run specific tests or benchmarks by ID:
python3 tests/bkao/src/bkao.py -t 100 101 -p <platform>
python3 tests/bkao/src/bkao.py -b 100 -p <platform>

Run all tests except some:
python3 bkao.py -t -x 200,201 -p <platform>

Skip firmware and toolchain rebuild (faster iteration):
python3 bkao.py -t -p <platform> --no-firmware-build --no-toolchain-build

Use custom hypervisor sources:
python3 bkao.py -t -p <platform> --hyp-srcs /path/to/bao

Set verbosity (0=final report only, 1=failures, 2=all):
python3 bkao.py -t -p <platform> -l 2

Pass platform-specific args - only virtual platforms (e.g. GIC version):
python3 bkao.py -t -p qemu-aarch64-virt --plat-virt-args "GICV3"

Bao Kao Process Pipeline

For each test or benchmark, bkao executes the following pipeline:

Discover: scans tests/tests/src/*.c for BAO_TEST macros, benchs/src/benchmarks/ for benchmark dirs, and platforms/*.py for platform builders.
Parse args: reads CLI arguments, resolves which tests/benchmarks to run, and builds the runtime config.
Platform setup: instantiates the requested platform class and calls setup_platform() (emulator paths, serial config, etc.).
Toolchain: builds or fetches the cross-compilation toolchain; skipped with --no-toolchain-build, expecting the prefix to be in PATH.
launch_tests(): for each group of tests/benchmarks sharing the same setup:

Renders config.c from the YAML VM descriptor via Jinja2 templates.
Builds each guest workload (baremetal, FreeRTOS, Linux, or benchmark binary).
Builds the Bao hypervisor against the generated config and guest images.
Builds firmware (ATF/OpenSBI/U-Boot); skipped with --no-firmware-build.
Launches the platform and captures serial output, parsing [TESTF-C] tokens for pass/fail.

Final cleanup: removes wrkdir/ build artifacts.

josecm · 2026-08-06T21:49:58Z

+
+tests:
+	@echo "Running bao-kao tests for $(PLATFORM)..."
+	@python3 $(kao_dir)/kao.py -t -p $(PLATFORM)


We're not passing --hyp-srcs, so kao clones bao-hypervisor at tag v2.0.0 and tests that instead of the branch. You can see it in the last CI run: [INFO] Fetching hypervisor sources..., which only prints when it actually clones. So the green ticks here are on the release, not on this PR.

The flag already exists and is wired up, so I think it's just:

Suggested change

@python3 $(kao_dir)/kao.py -t -p $(PLATFORM)

@python3 $(kao_dir)/kao.py -t -p $(PLATFORM) --hyp-srcs $(cur_dir)

I'd also make the no-arg case not silently clone, otherwise anyone calling kao.py directly hits the same thing.

Note this has to go together with removing the make clean kao does on the hypervisor sources, otherwise every make tests will wipe your build/ and bin/.

Fixed in ff5c6e2.

Aditionally, this required changes in bao-kao (bao-project/bao-kao@322e337)

josecm · 2026-08-06T21:49:58Z

+
+.PHONY: tests
+
+tests:


There's no way to pass anything else to kao from here (test ids, -l, --no-firmware-build). Can we forward these?

The old tests.mk did this already:

LOG_LEVEL?=2 ECHO?=tf SETUP?=baremetal TEST?=boot

and passed them all through, so you could do make ... TEST=irq LOG_LEVEL=1. I'd keep that, either the same way or with a single KAO_ARGS.

Also if tests/kao isn't initialized this fails with a bare python3: can't open file.

fixed in f69f2ce and 1a690a5.

josecm · 2026-08-06T21:49:58Z


 endif
+
+-include $(tests_dir)/tests.mk


This includes a file that isn't in the tree and then we define the targets inline anyway. It's a leftover: tests/tests.mk existed on feat/test-framework (67b6c7f) and was dropped in the rewrite, but the include stayed.

Either remove the include, or put the targets back in tests/tests.mk where they were. I'd lean to the second. This is bao's build system and these are orchestrating an external runner, and -include also degrades nicely when the submodule isn't initialized. If you do that the include has to move up near line 73, because non_build_targets is consumed at line 82.

As it is now there's also a trap: if someone re-adds tests/tests.mk with a tests: recipe, make will warn about overriding and the inline one silently wins.

Fixed in d47ac0f.

josecm · 2026-08-06T21:49:58Z

+
+benchs:
+	@echo "Running bao-kao benchmarks for $(PLATFORM)..."
+	@python3 $(kao_dir)/kao.py -b -p $(PLATFORM)


If the benchmarks submodule goes away from this repo (see my comment in .gitmodules), this target should probably go with it.

Right now with no benchs checked out kao finds no benchmarks and exits 0 without doing anything, so make benchs looks like it worked.

Fixed in 664d043.

josecm · 2026-08-06T21:49:58Z

+
+  run-tests:
+    runs-on: ubuntu-latest
+    container:


Every other workflow goes through ./.github/actions/bao-container-run now. It was added in a97c302 because the image is too big to pull under container: on the free runners (bao-ci#94). This one predates it, so it just needs to move over. The branch is 35 commits behind main anyway, so it needs a rebase first.

After that --user root and the safe.directory step below aren't needed, the action does both.

Fixed in 3ec913a.

josecm · 2026-08-06T21:49:58Z

+        }                         \
+    } while (0)
+
+#define TESTF_PASS(message) printf("    Message: %s\n", message);


This one isn't guarded by TESTF_LOG_LEVEL like the rest of the pass-side output, so it prints even at level 0. I ran with level 1 and everything went quiet except this line.

Fixed in 54bfa12.

josecm · 2026-08-06T21:49:58Z

+# Bao Hypervisor VM Configuration
+# This YAML describes the fields needed to generate a C config file (config.c)
+
+vms:


Question, not a change request: what do we get from the yaml + jinja over just writing config.c?

One yaml per (setup, platform) is one config.c per (setup, platform), so it's not fewer files, and nothing in the output is actually dynamic. Every value comes from the yaml, and BAO_WRKDIR_IMGS is resolved by the preprocessor, not the template. It's also missing things we'd want to test: cpu_affinity, colors, ipcs, remio_devs, mmu. And s32z270 already ships a hand-written plat.c next to its yaml.

Is the idea (a) to describe VMs separately and compose them for multi-guest setups, or (b) to be able to generate configs for another hypervisor? args[2].split("+") already builds a guest list so (a) looks intended, but the config lookup is still per-combination. If it is (a), does it also do the allocation (non-overlapping regions, cpu partitioning)? Just concatenating two VM configs doesn't really save anything.

Either way it should be in the README, because from outside it isn't obvious what it's for.

This was a design choice; its main purpose is to make a test scenario a single, declarative unit: the Bao configuration together with the guest/build parameters needed to reproduce that scenario. In that sense, a YAML file describes more than the contents of config.c. It is intended to be the input consumed by the test/build tooling, while the generated config.c is one target-specific artifact of that description. This also gives us a path to emit equivalent scenario configurations for other hypervisors, rather than making the benchmark/test definitions inherently Bao-specific.

That said, the current implementation does not yet fully realize the compositional model you describe:

args[2].split("+") was introduced with multi-guest scenarios in mind, but configuration selection is still effectively per (setup, platform) combination.

It does not currently compose independently defined VM descriptions and perform resource allocation or validation (for example, CPU partitioning, non-overlapping memory regions, cache colors, IPCs, remote-I/O devices, or MMU settings).

So the intended answer is closer to (a) as a longer-term scenario-description direction, plus (b) as an eventual portability benefit, rather than a claim that the current YAML files already provide full VM composition or allocation. Today, they mainly centralize the scenario metadata/configuration used by the test infrastructure and generate a Bao-specific configuration from it.

josecm · 2026-08-06T21:49:58Z

+
+vms:
+  - vm1:
+    name: "baremetal"


This is a byte-for-byte copy of the baremetal one, including name: "baremetal", so the freertos config declares a baremetal VM. Same for the linux one. Nothing can select them either, build_guests only knows baremetal.

Can we drop both directories from this PR and bring them back with the guest builders behind them?

Fixed in ab070b9.

josecm · 2026-08-06T21:49:58Z

+
+vms:
+  - vm1:
+    name: "baremetal"


Same as the freertos one.

Fixed in ab070b9.

josecm · 2026-08-06T21:49:58Z

@@ -0,0 +1,1241 @@
+/*
+ * Copyright 2021-2024 NXP


This BSP is 79 files and 33.5k lines, which is around 63% of the PR, for a board that isn't in the CI matrix and needs a T32 probe anyway. It's also a different license from ours and tests/ is outside license-check's scope, so nothing verifies it.

Can we leave s32z270 for a follow-up, or move the BSP to its own repo as a submodule like benchs? It would make this PR much easier to review.

Fixed in f2f822f.

josecm · 2026-08-07T10:02:11Z

+        irq_set_prio(UART_IRQ_ID, UART_IRQ_PRIO);
+        COMMAND_SEND_CHAR("a");
+
+        timer_wait(TEST_TIME_WAIT);


The TIMER test above spins on the flag with the watchdog as backstop, but this one sleeps 200ms and asserts. The injected char goes through the logger thread and a pty, so on a loaded runner 200ms can be missed and the test goes red on timing alone. Can we make it spin on irq_en_uart like TIMER does? The watchdog already bounds it.

Fixed in 10aeae7.

Signed-off-by: Daniel Oliveira <drawnpoetry@gmail.com> Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com> Signed-off-by: Miguel Silva <miguelafsilva5@gmail.com>

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

Signed-off-by: Daniel Oliveira <drawnpoetry@gmail.com>

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

josecm · 2026-08-23T22:44:28Z

@Diogo21Costa can you go comment by comment and say which points of the review are addressed by the latest push, and if relevant, provide futher information. For those not addressed and we which you don't plan to address, please explain we don't think they should be.

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

Diogo21Costa · 2026-08-31T11:02:49Z

@Diogo21Costa can you go comment by comment and say which points of the review are addressed by the latest push, and if relevant, provide futher information. For those not addressed and we which you don't plan to address, please explain we don't think they should be.

@josecm Matched each of the comments with the specific fixes. While validating the review changes, make tests PLATFORM=qemu-riscv64-virt exposed additional integration problems. These were addressed by:
cbcf574 fix(riscv): update test configuration
0f69f50 update(kao): add boot timeout
0fcda4b update(kao): use supported RISC-V QEMU
e17c3ab update(kao): report setup failures
80c2f98 update(kao): enable RISC-V Svpbmt
dea359b fix(riscv): build AIA test guest

with corresponding Bao Kao commits:
e2f3659 fix(riscv): enable AIA in QEMU
1334fec fix(logger): time out stalled boots
71d0579 fix(riscv): use supported QEMU version
f72572a fix(setup): report command failures
ad50039 fix(riscv): enable Svpbmt in QEMU

danielRep requested review from DavidMCerdeira, josecm, miguelafsilva5 and sandro2pinto as code owners May 15, 2026 10:01

danielRep mentioned this pull request May 15, 2026

feat(test-framework): Add a test-framework to bao #154

Closed

danielRep force-pushed the feat/bao-kao branch 8 times, most recently from 6f43b3a to 2ef3a2a Compare May 15, 2026 16:17

danielRep changed the title ~~feat(bkao): add support for bao kao testing framework~~ feat(kao): add support for bao kao testing framework May 15, 2026

danielRep force-pushed the feat/bao-kao branch 2 times, most recently from 010c9b8 to fe6b4ea Compare May 18, 2026 09:11

miguelafsilva5 reviewed May 19, 2026

View reviewed changes

Comment thread tests/tests/src/irq.c Outdated

Comment thread tests/tests/src/irq.c Outdated

Comment thread tests/tests/configs/baremetal/tc4dx.yaml Outdated

Comment thread tests/tests/configs/baremetal/tc4dx.yaml Outdated

Comment thread tests/tests/configs/baremetal/tc4dx.yaml

danielRep force-pushed the feat/bao-kao branch from b1606ac to 8922c08 Compare May 22, 2026 09:53

josecm force-pushed the main branch from 6be6124 to 8daa11c Compare July 20, 2026 08:28

josecm self-assigned this Aug 6, 2026

josecm requested changes Aug 6, 2026

View reviewed changes

josecm mentioned this pull request Aug 7, 2026

fix(mem): fix hypervisor self-coloring boot hang #391

Open

2 tasks

josecm reviewed Aug 7, 2026

View reviewed changes

danielRep and others added 7 commits August 11, 2026 10:42

feat(kao): add support for bao kao testing framework

224b439

Signed-off-by: Daniel Oliveira <drawnpoetry@gmail.com> Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com> Signed-off-by: Miguel Silva <miguelafsilva5@gmail.com>

fix(irq_test): fix IRQ priorities

5931c67

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

fix(tc4dx_cfg): update baremetal test config

7d5849b

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

update(kao): update submodule

ce57353

Signed-off-by: Daniel Oliveira <drawnpoetry@gmail.com>

fix(kao): test current sources

ff5c6e2

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

ref(tests): move kao make targets

d47ac0f

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

feat(tests): forward kao arguments

f69f2ce

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

Diogo21Costa added 17 commits August 11, 2026 11:29

fix(tests): check kao submodule

1a690a5

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

ref(benchs): remove private benchmarks

664d043

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

feat(kao): add wrkdir support

707e9e4

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

fix(ci): use bao container action

3ec913a

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

update(ci): cache fvp models

3769011

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

update(ci): reuse toolchains

902ebe4

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

fix(testf): respect log level

54bfa12

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

fix(irq_test): wait for uart interrupt

10aeae7

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

ref(configs): remove unsupported guests

ab070b9

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

ref(s32z270): defer test support

f2f822f

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

fix(riscv): update test configuration

cbcf574

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

update(kao): add boot timeout

0f69f50

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

update(kao): use supported RISC-V QEMU

0fcda4b

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

update(kao): report setup failures

e17c3ab

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

update(kao): enable RISC-V Svpbmt

80c2f98

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

fix(riscv): build AIA test guest

dea359b

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

update(kao): forward log level

fce6be8

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

Diogo21Costa force-pushed the feat/bao-kao branch from 8922c08 to fce6be8 Compare August 14, 2026 09:31

danielRep requested a review from miguelafsilva5 August 28, 2026 10:23

update(kao): fix FVP toolchains

3b5464f

Signed-off-by: Diogo Costa <diogoandreveigacosta@gmail.com>

	@python3 $(kao_dir)/kao.py -t -p $(PLATFORM)
	@python3 $(kao_dir)/kao.py -t -p $(PLATFORM) --hyp-srcs $(cur_dir)


		.PHONY: tests

		tests:

Uh oh!

Conversation

danielRep commented May 15, 2026

Overview

Source Tree Structure

Tests Available

How to Run (TLDR)

Bao Kao Process Pipeline

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

josecm commented Aug 23, 2026

Uh oh!

Diogo21Costa commented Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants