Unable to use the 2GPUs on my 2 nvidia RTX A6000

[kokarl]

Hello,

I have setup a new fresh install of DSA on a new Ubuntu Linux 22.04 LTS box, with 2 nvidia RTX A6000 graphic cards.
I have set up the latest nvidia drivers and nvidia CUDA drivers. I have else setup nvidia-docker2. Everything seems to be set up fine.
All the GPU of the 2 cards are working fine, when I test with GPU_BURN docker image.

I am trying to use all the GPU inside Monailabel and PTCsegmentation plugins. When I am inside the docker image of Monailabel or PTCsegmentation, I can see the 2 cards. But When I launch a job, only 1 GPU of 1 card (always the same) is used. I never succeed to run a job using all the GPU in the same time.

It is pretty strange, because I can see that the nvidia runtime is well used in the jobs logs output.

What could be wrong with my set up ?
How can I manage DSA to use all the GPU for the plugins ?

Best regards,
Karl

[manthey]

Sorry for the delay in answering. Did you figure this out? When we run a job, we specify docker.types.DeviceRequest(count=-1, capabilities=[['gpu']])] if we detect the docker image should use gpu, which is supposed to give permission to all gpus.