GPU over 2X slower than CPU

akela

(Garry Paxinos) #1

I just tried using a GPU for my training and it is a lot slower than CPU. I’m using a i7 8700 and a Nvidia 2070 gpu. The GPU is over 2X slower - and I’m just trying to figure out why. This happens both on NLU and on Core training.

I have the appropriate Tensorflow-gpu version, Cuda, and CUDDN installed.

While anaconda shows keras-gpu being available, ‘pip’ doesn’t seem to be able to find it… So I’ve only installed a GPU version of tensorflow.

core training shows -

CPU: 734/8734 [==============================] - 1s 110us/step - loss: 2.4849 - acc: 0.3525 Epoch 2/100 8734/8734 [==============================] - 1s 75us/step - loss: 1.7924 - acc: 0.4560 Epoch 3/100 8734/8734 [==============================] - 1s 75us/step - loss: 1.2614 - acc: 0.6503 Epoch 4/100 8734/8734 [==============================] - 1s 75us/step - loss: 0.8581 - acc: 0.7908 Epoch 5/100 8734/8734 [==============================] - 1s 75us/step - loss: 0.6043 - acc: 0.8576

GPU: Epoch 1/100 9114/9114 [==============================] - 2s 224us/step - loss: 2.4962 - acc: 0.3366 Epoch 2/100 9114/9114 [==============================] - 2s 185us/step - loss: 1.7185 - acc: 0.5086 Epoch 3/100 9114/9114 [==============================] - 2s 186us/step - loss: 1.2080 - acc: 0.6572 Epoch 4/100 9114/9114 [==============================] - 2s 185us/step - loss: 0.8227 - acc: 0.8024

I have verified that the GPU is being used by watching ‘nvtop’ - training uses about 39% of the GPU according to nvtop.

Any recommendations on what I can look into and/or try?


(Akela Drissner) #2

Is this for NLU or Core?


(Garry Paxinos) #3

The timing shown is for core. But I also see NLU slower as well - perhaps not as much as Core, but still slower


(Akela Drissner) #4

Alright, so this is because LSTMs aren’t really optimised for GPUs. There is a GPU implementation for the LSTM in tensorflow, but we don’t have that in our repo at the moment. I don’t think it would speed up much on the amount of data you’re using anyways