Version: dev-CC Hi all, Is there anyone successfully trained a bot/model using Rasa 1.9.X on GPU? I was using Rasa 1.7 with gpu and training was fast and straight forward. But we recently switched to Rasa 1.9 to be able using DIET classifier improving entity extraction.
From performance (accuracy) aspect, DIET classifier is great but quite slow and always running on CPUs. We did a deep dive in the codes and there was/is nothing preventing DIET from GPU, even it seems it tries to load data on GPU but no processing at all and then switches back to CPUs!
Does anyone experiencing same situation, and have you find any solution?
We did some hacks and got the following error:
I searched for this error, and it looks kind of TensorFlow 2.1 bug (I am not 100% sure that this TF bug is the source our problem but somehow explains weird behavior of our GPUs).
Meanwhile, we have setup and tested the environment variables for gpu run: TF_FORCE_GPU_ALLOW_GROWTH=True
Versions: Rasa 1.9.5 , TensorFlow 2.1.0 , Ubuntu: 16.04.6 LTS , CUDA: 10.1