I am struggling with a CI/CD pipeline that trains Rasa models with the GitLab docker executor. I used an existing Rasa 2.8 (full) image for it but it seems that it does not support CUDA out-of-the-box. Is that correct?
If so, what is the best practice to build an image that supports CUDA in the case of CI/CD?
@ScienceGuy can you share some code or error?
Well, there is no error message. I cannot see that the training process uses the GPU (no load on the GPU, no nvidia-smi installed, no cuda/version.txt). So I am blind the container seeing if it works or not. I can say that the training time did not decrease at all - therefore, I assume that it does not work.
My current experiments include building a new image that uses nvidia/cuda:10.1-base-ubuntu18.04 as base image.
I tried it further with a tensorflow base image 2.6.1-gpu and installed the latest Rasa version. I receive this error now:
E tensorflow/stream_executor/cuda/cuda_diagnostics.cc:313] kernel version 470.57.2 does not match DSO version 460.91.3 -- cannot find working devices in this configuration
I assume this indicates where the error comes from. Any thoughts?
Maybe anyone can provide a Dockerfile for a tensorflow image that works with the recent Rasa version? I am stuck for 2 days figuring out how to get the runner working. A pre-built image with would be great and I think there is even a related issue open in the Rasa github repo.
@ScienceGuy No Idea bro, sorry to disappoint you, I’m currently working on CPU not on GPU, else I should faced this issue. But, please do provide me the steps you have done so far, staring installation or rasa or rasa x?
There’s an open issue here.