Add parallel tuning on multiple remote GPUs using Ray #328
Conversation
|
stijnh
changed the title
|
The current parallel runner works. I've been able to run on multiple GPUs on DAS6-VU and DAS6-Leiden. There are several remaining problems:
|
|
Sometimes with Python you run into an error and think: 'How on Earth has this error not surfaced years ago?' It seems that observers has been None by default instead of an empty list since forever and miraculously it was never an issue. It seems that the code responsible for replacing a None value with an empty list is currently hidden in (and duplicated across) the backends, and because there is now code that (rightfully so) assumes observers is a list just before the backends are created this is suddenly an issue. The real issue is of course that the backends have somehow become responsible for sanitizing user input, which is not what a backend should do. |
|
… instead of `__init__`
…` instead of `__init__`
…d of `runner.dev.name`
stijnh
force-pushed
the
parallel_runner
branch
3 times, most recently
from
ee74137 to
a769aea
Compare
|
|
I have been tested this on Snellius and it appears to work fine with multiple GPUs across multiple nodes. Remaining issues:
|
|
3 participants
Working on a simple parallel runner that uses Ray to distribute the benchmarking of different configurations to remote Ray workers.