record the click history

[joyful2024]

Does anyone know how to record the click history? that will enable us to do learning by demonstration.

[gasse]

This is something we though of a while ago but never implemented.
#28

What you could do is add Javascript event listeners on the playwright context (will be registered for all visited pages), which records the list of all JS events somewhere in the JS main document. Then you just have to extract this list when you extract the observation.
All of that can be done either by patching the environment class (reset() and _get_obs()), or by creating a custom task that can wrap around any existing task (setup() and validate()).

One challenge if you want to learn by demonstration is that you also want to extract the observation right before the action happens, which is not trivial at all. This is moistly why we never implemented this feature.