Tensorflow driver : UNKNOWN ERROR (303)

[zerotrope (zero)]

Hello,

Since last requirements.txt update on RASAHQ/rasa repo, I systematically get the following messaging in terminal when starting rasa x interface :

E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: UNKNOWN ERROR (303)

It does not prevent any functionality of rasa to fail though. My researches towards cuda driver problems didn’t bring any satisfactory answers so far, any idea ?

Thanks,

z.

[hari (Hari Teneti)]

I am also seeing the same issue when I start the RASA Core server.

ds_botserver | 2019-11-27 19:22:49 INFO root - Starting Rasa server on http://localhost:5005 gds_botserver | 2019-11-27 19:22:54.942209: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: UNKNOWN ERROR (303)

As mentioned in the previous post it does not prevent any functionality. The server works fine but the message will give false impression to the deployment team that the servers are not up correctly.

[sudo-ranjith (Ranjith kumar)]

I am also getting the same error, but i would like to know the solution

tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: UNKNOWN ERROR (303)

[rslater (Robert Slater)]

It seems to come from a fallback from GPU to CPU processing. I am wondering if somehow the tensorflow-gpu package is getting installed versus the “regular” tensorflow.

[micha]

Did anyone of you solve it? Experiencing the same issue

[zerotrope (zero)]

Hi @rslater

Could you elaborate ? Might be the gpu package installed when installing the rasa framework itself, instead of the cpu one ?

[priya359 (priyadarshani priya)]

hi all

I am also facing the same error when i try to run rasa server on docker Did anyone get the solution of this error ?

[harieharaan]

Hi,

I am also facing the same error when using rasa with docker. Any solution?

[ioncoa]

Same problem

OS Ubuntu 18.04 (virtual environment, vmware) Python version: 3.6.9

installed packages: pip freeze tensor2tensor==1.14.1 tensorboard==1.15.0 tensorflow==1.15.0 tensorflow-datasets==1.3.2 tensorflow-estimator==1.15.1 tensorflow-gan==2.0.0 tensorflow-hub==0.7.0 tensorflow-metadata==0.15.2 tensorflow-probability==0.7.0

[d7x (Paul)]

Same problem with

OS Ubuntu 18.04.3 (virtual environment, vmware) Python version: 3.6.9

tensorflow packages: tensor2tensor==1.14.1 tensorboard==1.15.0 tensorflow==1.15.0 tensorflow-datasets==1.3.2 tensorflow-estimator==1.15.1 tensorflow-gan==2.0.0 tensorflow-hub==0.7.0 tensorflow-metadata==0.15.2 tensorflow-probability==0.7.0

Could the issue be related to cuda not being installed? I checked my Ubuntu folders and cuda isn’t installed. Two questions:

1. Can Rasa run with the tensorflow-gpu version?
2. Should cuda be installed if running a cuda-compatible Nvidia graphics card?

[memorizeracks (onstacks)]

Getting the same error when running Rasa Server in Docker.

[stephens (Greg Stephens)]

This error has to do with the type of CPU you are running on and can be ignored because the training continues and produces a model.

[memorizeracks (onstacks)]

Localhost:5005 still wont connect though.

2020-01-07 16:57:06 INFO     root  - Starting Rasa server on http://localhost:5005
2020-01-07 16:57:13.234759: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] 
failed call to cuInit: UNKNOWN ERROR (303)

[priya359 (priyadarshani priya)]

@stephens yes it produces a model…but it is creating problem while running rasa server on docker

[stephens (Greg Stephens)]

Can you attach a log of the runtime errors?

[nmstoker (Neil Stoker)]

Just a side point, but regarding your question about tensorflow-gpu: for version 1.15.0 of TensorFlow (which is the one referenced in requirements.txt currently) there is no tensorflow-gpu package, it’s a single package for CPU and GPU (see here for details: Install TensorFlow with pip  |  TensorFlow)

[Tushar (Tushar Rana)]

I am also facing the same issue with the latest Rasa 1.6.1

Earlier it was working with the older version Rasa 1.5.3

But in new version 1.6.1 this is issue and i can’t use rasa.

Please fix this issue ASAP so that we can use rasa.

[gaponeni]

Looks that you don’t have permission to use gpu. I’m not familiar with Rasa, but try something like ‘sudo’ to run your code with privileges.

[rasafan]

I have the same error and localhost:5000 won’t connect, did you solve this?

[wink94 (wink)]

i get the same error with tensorflow version 2.1.0

2020-03-04 17:57:56.088639: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library ‘libnvinfer.so.6’; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory

2020-03-04 17:57:56.088723: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library ‘libnvinfer_plugin.so.6’; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory

2020-03-04 17:57:56.088739: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.

2020-03-04 17:57:57.396943: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)

2020-03-04 17:58:04 INFO rasa.nlu.selectors.response_selector - Retrieval intent parameter was left to its default value. This response selector will be trained on training examples combining all retrieval intents.