Hi @stephens,
Happens to me that every I run a RASA server (e.g. with cli commands: rasa train or rasa run ...) I got the (apparently not-blockingbut suspicious) error message:
E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
See below below the logs, when running: (venv) $ rasa run --debug --enable-api -m models:
2020-05-02 20:29:20 DEBUG rasa.cli.utils - Parameter 'endpoints' not set. Using default location 'endpoints.yml' instead.
No chat connector configured, falling back to the REST input channel. To connect your bot to another channel, read the docs here: https://rasa.com/docs/rasa/user-guideessaging-and-voice-channels
2020-05-02 20:29:20 DEBUG rasa.core.utils - Available web server routes:
/conversations/<conversation_id>/messages POST add_message
/conversations/<conversation_id>/tracker/events POST append_events
/webhooks/rest GET custom_webhook_RestInput.health
/webhooks/rest/webhook POST custom_webhook_RestInput.receive
/model/test/intents POST evaluate_intents
/model/test/stories POST evaluate_stories
/conversations/<conversation_id>/execute POST execute_action
/domain GET get_domain
/ GET hello
/model PUT load_model
/model/parse POST parse
/conversations/<conversation_id>/predict POST predict
/conversations/<conversation_id>/tracker/events PUT replace_events
/conversations/<conversation_id>/story GET retrieve_story
/conversations/<conversation_id>/tracker GET retrieve_tracker
/status GET status
/model/predict POST tracker_predict
/model/train POST train
/conversations/<conversation_id>/trigger_intent POST trigger_intent
/model DELETE unload_model
/version GET version
2020-05-02 20:29:20 INFO root - Starting Rasa server on http://localhost:5005
2020-05-02 20:29:20 DEBUG rasa.core.utils - Using the default number of Sanic workers (1).
2020-05-02 20:29:20 INFO root - Enabling coroutine debugging. Loop id 95987480.
2020-05-02 20:29:21 DEBUG rasa.model - Extracted model to '/tmp/tmp8se_uv6g'.
/home/solyaris/venv/lib/python3.6/site-packages/thinc/neural/_custom_kernels.py:36: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/solyaris/venv/lib/pytn3.6/site-packages/thinc/neural/_custom_kernels.cu' mode='r' encoding='utf8'>
SRC = (PWD / "_custom_kernels.cu").open("r", encoding="utf8").read()
/home/solyaris/venv/lib/python3.6/site-packages/thinc/neural/_custom_kernels.py:39: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/solyaris/venv/lib/pytn3.6/site-packages/thinc/neural/_murmur3.cu' mode='r' encoding='utf8'>
MMH_SRC = (PWD / "_murmur3.cu").open("r", encoding="utf8").read()
2020-05-02 20:29:34 INFO rasa.nlu.components - Added 'SpacyNLP' to component cache. Key 'SpacyNLP-en'.
2020-05-02 20:29:34 DEBUG rasa.utils.tensorflow.models - Loading the model ...
2020-05-02 20:29:34.403298: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
2020-05-02 20:29:34 DEBUG rasa.utils.tensorflow.models - Finished loading the model.
2020-05-02 20:29:34 DEBUG rasa.utils.tensorflow.models - Building tensorflow prediction graph...
2020-05-02 20:29:38 DEBUG rasa.utils.tensorflow.models - Finished building tensorflow prediction graph.
2020-05-02 20:29:38 DEBUG rasa.core.tracker_store - Connected to InMemoryTrackerStore.
2020-05-02 20:29:38 DEBUG rasa.core.lock_store - Connected to lock store 'InMemoryLockStore'.
2020-05-02 20:29:38 DEBUG rasa.model - Extracted model to '/tmp/tmp8yc0gxg1'.
2020-05-02 20:29:38 DEBUG pykwalify.compat - Using yaml library: /home/solyaris/venv/lib/python3.6/site-packages/ruamel/yaml/__init__.py
2020-05-02 20:29:38 DEBUG rasa.utils.tensorflow.models - Loading the model ...
2020-05-02 20:29:38 DEBUG rasa.utils.tensorflow.models - Finished loading the model.
2020-05-02 20:29:38 DEBUG rasa.utils.tensorflow.models - Building tensorflow prediction graph...
2020-05-02 20:29:39 DEBUG rasa.utils.tensorflow.models - Finished building tensorflow prediction graph.
2020-05-02 20:29:39 DEBUG rasa.core.nlg.generator - Instantiated NLG to 'TemplatedNaturalLanguageGenerator'.
Here my PC configuration:
(venv) $ pip list | grep tensorflow
tensorflow 2.1.0
tensorflow-addons 0.9.1
tensorflow-estimator 2.1.0
tensorflow-hub 0.7.0
tensorflow-probability 0.7.0
WARNING: You are using pip version 20.0.2; however, version 20.1 is available.
You should consider upgrading via the '/home/solyaris/venv/bin/python3 -m pip install --upgrade pip' command.
(venv) $ pip3 --version
pip 20.0.2 from /home/solyaris/venv/lib/python3.6/site-packages/pip (python 3.6)
(venv) $ rasa --version
Rasa 1.9.7
(venv) $ inxi -GC
CPU: Quad core Intel Core i7-8565U (-MT-MCP-) cache: 8192 KB
clock speeds: max: 4600 MHz 1: 1151 MHz 2: 841 MHz 3: 958 MHz 4: 848 MHz 5: 802 MHz 6: 838 MHz
7: 802 MHz 8: 838 MHz
Graphics: Card-1: Intel Device 3ea0
Card-2: Advanced Micro Devices [AMD/ATI] Topaz XT [Radeon R7 M260/M265 / M340/M360 / M440/M445]
Display Server: N/A drivers: modesetting,ati,amdgpu (unloaded: fbdev,vesa)
tty size: 169x47 Advanced Data: N/A out of X
Any idea about the origin of the issue?
Thanks/giorgio