Rasa train (rasa 1.9.x | TensorFlow 2) on GPU?

Rasa Open Source
1

Version: dev-CC Hi all, Is there anyone successfully trained a bot/model using Rasa 1.9.X on GPU? I was using Rasa 1.7 with gpu and training was fast and straight forward. But we recently switched to Rasa 1.9 to be able using DIET classifier improving entity extraction.

From performance (accuracy) aspect, DIET classifier is great but quite slow and always running on CPUs. We did a deep dive in the codes and there was/is nothing preventing DIET from GPU, even it seems it tries to load data on GPU but no processing at all and then switches back to CPUs!

Does anyone experiencing same situation, and have you find any solution?

We did some hacks and got the following error:

image
image1342×138 11 KB

I searched for this error, and it looks kind of TensorFlow 2.1 bug (I am not 100% sure that this TF bug is the source our problem but somehow explains weird behavior of our GPUs).

Meanwhile, we have setup and tested the environment variables for gpu run: TF_FORCE_GPU_ALLOW_GROWTH=True

tf.config.experimental.list_physical_devices(‘GPU’)

tf.config.set_visible_devices(, ‘GPU’)

Versions: Rasa 1.9.5 , TensorFlow 2.1.0 , Ubuntu: 16.04.6 LTS , CUDA: 10.1

2

It seems to be a CUDA issue. Can you run normal tensorflow/keras code with CUDA?

3

@koaning, yes I can run simple tensorflow code with no problem.

4

But are these tensorflow programs using tensorflow? Putting the error into google suggests this result; tensorflow2.0 - Could not load dynamic library 'cudart64_101.dll' on tensorflow CPU-only installation - Stack Overflow.

5

@amvas I have almost no exposure to tensorflow and cuda. Can you perhaps guide me a little bit on how to run the NLU and core both on GPU. Also did you see any performance gains. We are working on some latency critical app and need to optimize the latency as much as possible. So did it improve the overall response time on GPU?

6

I had a similar issue (but with CUDA 11.1 and TF 2.1.2) and I found the solution here: ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory · Issue #26182 · tensorflow/tensorflow · GitHub

Basically my linux was pointing to the wrong CUDA lib, so I fixed it just adding the correct version to LD_LIBRARY_PATH

LD_LIBRARY_PATH=/usr/local/cuda-10.2/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

I hope it helps! Paolo

1 Like
7

@amvas How much time it took to train your model within 1 min or it it took around 2-3 minutes.Can you please let me know I can try