How can I check that Rasa using my GPU? GPU core just get to 46%

Getting Started with Rasa
1

I’m really new to Rasa and TensorFlow, but I would like to know if it is using my GPU to train the model. I’m in doubt because when I’m training the CPUs go till 100%, but the GPU core max is 46%. I’m checking with OpenHardware Monitor in a Windows 10 machine.

Is it a usual behavior? Can I do anything to check that the GPU is being used?

I installed the CUDA libraries and I have a virtual environment with tensorflow-gpu installed. If I open a python shell and run the commands below it displays that the CUDA dll is correctly loaded (some trouble to make it work):

In [1]: import tensorflow as tf
   ...: if tf.test.gpu_device_name():
   ...:     print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))
   ...: else:
   ...:     print("Please install GPU version of TF")
   ...:
2020-04-04 01:35:27.599799: I tensorflow/stream_executor/platform/default/dso_lo
ader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-04-04 01:35:29.522705: I tensorflow/core/platform/cpu_feature_guard.cc:142]
 Your CPU supports instructions that this TensorFlow binary was not compiled to
use: AVX2
2020-04-04 01:35:29.528037: I tensorflow/stream_executor/platform/default/dso_lo
ader.cc:44] Successfully opened dynamic library nvcuda.dll
2020-04-04 01:35:29.556458: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1
555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2060 computeCapability: 7.5
coreClock: 1.68GHz coreCount: 30 deviceMemorySize: 6.00GiB deviceMemoryBandwidth
: 312.97GiB/s
2020-04-04 01:35:29.562886: I tensorflow/stream_executor/platform/default/dso_lo
ader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-04-04 01:35:29.568748: I tensorflow/stream_executor/platform/default/dso_lo
ader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-04-04 01:35:29.574943: I tensorflow/stream_executor/platform/default/dso_lo
ader.cc:44] Successfully opened dynamic library cufft64_10.dll
2020-04-04 01:35:29.579120: I tensorflow/stream_executor/platform/default/dso_lo
ader.cc:44] Successfully opened dynamic library curand64_10.dll
2020-04-04 01:35:29.585310: I tensorflow/stream_executor/platform/default/dso_lo
ader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2020-04-04 01:35:29.591044: I tensorflow/stream_executor/platform/default/dso_lo
ader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2020-04-04 01:35:29.601424: I tensorflow/stream_executor/platform/default/dso_lo
ader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-04-04 01:35:29.605307: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1
697] Adding visible gpu devices: 0
2020-04-04 01:35:30.143904: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1
096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-04-04 01:35:30.147119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1
102]      0
2020-04-04 01:35:30.148477: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1
115] 0:   N
2020-04-04 01:35:30.150273: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1
241] Created TensorFlow device (/device:GPU:0 with 4604 MB memory) -> physical G
PU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capabil
ity: 7.5)
2020-04-04 01:35:30.155254: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1
555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2060 computeCapability: 7.5
coreClock: 1.68GHz coreCount: 30 deviceMemorySize: 6.00GiB deviceMemoryBandwidth
: 312.97GiB/s
2020-04-04 01:35:30.160500: I tensorflow/stream_executor/platform/default/dso_lo
ader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-04-04 01:35:30.162734: I tensorflow/stream_executor/platform/default/dso_lo
ader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-04-04 01:35:30.164797: I tensorflow/stream_executor/platform/default/dso_lo
ader.cc:44] Successfully opened dynamic library cufft64_10.dll
2020-04-04 01:35:30.166825: I tensorflow/stream_executor/platform/default/dso_lo
ader.cc:44] Successfully opened dynamic library curand64_10.dll
2020-04-04 01:35:30.168830: I tensorflow/stream_executor/platform/default/dso_lo
ader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2020-04-04 01:35:30.171060: I tensorflow/stream_executor/platform/default/dso_lo
ader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2020-04-04 01:35:30.173093: I tensorflow/stream_executor/platform/default/dso_lo
ader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-04-04 01:35:30.176974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1
697] Adding visible gpu devices: 0
2020-04-04 01:35:30.179210: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1
096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-04-04 01:35:30.181703: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1
102]      0
2020-04-04 01:35:30.183077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1
115] 0:   N
2020-04-04 01:35:30.184702: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1
241] Created TensorFlow device (/device:GPU:0 with 4604 MB memory) -> physical G
PU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capabil
ity: 7.5)
Default GPU Device: /device:GPU:0
1 Like
2

I can’t help you on Windows 10 but your nvidia card has probably a utility to monitor which process use it. On linux, nvidia-smi does this. When training a rasa model, I am not above 35% GPU use on an RTX2070. I am at 3% without training anything. So your values are probably fine.

1 Like