New Mir testing framework high-level description

[matejpavlovic]

What are, on the high level (no need to write a lot), the ideas behind the new testing framework?
In particular:

• Is there a difference in basic approach to the old (patchy) testing code, or is it similar in principle, just done properly and cleanly?
  • If it is fundamentally different, what is the idea behind the new approach?
• How does the logical time simulation work?
  • What is the high-level idea behind it?
  • Based on what are time events injected in the system and how?

[sergefdrv]

@matejpavlovic

difference in basic approach

No significant change so far, I'm focused on implementing logical time simulation.

How does the logical time simulation work?

I'm still shaping it. I'm trying to (1) use an abstract interface for low-level timers in Mir's timer module, (2) wrap Mir modules to control passage of events. Everything would appear instantaneous outside of the logical time simulation; the system would step from one scheduled event to another. The simulation would provide a way to block and wait for a specified duration of simulated time, so that the module wrappers would be able to insert pseudo-random delays, e.g. for simulating communication delays.

[matejpavlovic]

Not quite sure I understand.

• What will (approximately) the abstract interface for low-level timers look like and how will it be used?
  • Will it expose a similar interface as Go's default time package, but do something else under the hood?
  • If yes, what's the main idea of how it will schedule the timers?
• For the wrappers, what exactly will they do?
  • Buffer the stream of events sent to the wrapped module and modify that stream?
  • If yes, what modifications will they perform and based on what do they decide which modifications are legitimate?
  • Or is it about buffering events at the output of a module and inserting those events in the main event loop only later, after events produced by other modules have been inserted?

As for the timer, the default timer module currently produces (delayed) events based on the "Delay" and "Repeat" events it consumes. Those events are inserted into Mir's input event stream at some point (namely after a real-time delay). When this is used in a test case (as is currently done), there are periods where all nodes already processed all events, their queues are empty, and everybody is waiting for some (real-time) timer to fire that inserts new events in the event loop. I guess you want to get rid of these waiting periods, right?

It also looks like you want to simulate communication delays (very useful, btw!), but I'm not sure I understand how this is done with a module wrapper. By communication, you mean event-based communication between modules of the same node, or the exchange of messages between different nodes? (If it is the former, this question will probably be at least partially answered by answering the questions from above).

I guess what would really help me understand is an example (as concrete as possible) of what a hypothetical test run would look like. Something like "The testing framework instantiates 4 Mir nodes with such and such modules (the timer module being replaced by a special one), calls Debug (or Run) on the Nodes, when such module at some node produces an event for such module, this happens, when the timer receives a Delay event, that happens, etc..., and all that is to simulate a scenario where XYZ". No need to talk only about what's implemented, it's just to have a glimpse of what the result should approximately look like.

[matejpavlovic]

In a way, the "everything appearing instantaneous" is implemented (quite rudimentarily) in the mircat debugger, but only for a single node with a saved trace of events. It feeds (including timer events) one by one to the node and (by default) waits for user input before submitting the next event. But it has serious drawbacks in the sense that it requires a pre-recorded event trace and it only works with a single node configured for ISS, and thus is not very general. Is what you have in mind something similar, but more general (e.g. taking the event trace not from disk, but directly from node output and potentially shuffling it around where appropriate)?

Sorry, these are really many questions, I know. We can also have a sync call to discuss them if that's more convenient.

[sergefdrv]

@matejpavlovic I am fully concentrated on the task figuring out details through explorative coding, not sure if dumping my mind would be particularly efficient at this point. I'd better show and explain you everything once I have a working draft of the code.

[matejpavlovic]

Ok let's discuss later then, but let's make it this week, as next week I can't. Still, it would be great to have a few words about the goal of the testing infrastructure (not the design or a complete mind dump, just a more concrete view of the problem it is solving). I think it's a good exercise to clarify (for both oneself and for the others) in which direction the effort is going. And I'm always open for a little brainstorm - if that can be useful to you, let me know.

[sergefdrv]

Sure, the main idea is in (1) getting rid of real time by replacing it with a simulated clock, and (2) testing against randomized, but reproducible, schedules by adding pseudo-random delays measured in the simulated time. I'll ping you once I have something worth looking at in my wip branch.