For 3 days I had changed everything even I had installed the system and still same error without any clue what is the error, just Training failed does anyone in the world have an answer
You can always check the logs to see what the error is.
Are you using a local or server installation? Does your bot successfully train if you do rasa train?
You should give more information if you want an answer. So many things could have caused this error. Check out How to ask a great question.
2 Likes
Thanks for the response
I’m using vps from hosting company.
I searched everywhere for logs nothing thier only F12 and log says internal server error.
Even I’m asking for specialist to solve this meaningless errors
1 Like
Sorry, it’s hard to help you if you cannot find the logs. What Rasa and Rasa-X versions do you have? How are you running Rasa-X? What is your operating system?
Thanks for the response, If anyone can help me out how to get the logos,
Even I can provide you with with all vps credentials.
Server us linux Ubuntu
I highly recommend that you DON’T provide anyone online with your credentials.
@snek I agree with you, but I’m tired of this issues, I couldn’t find the logos and no one could give me an answer
I understand your frustration but it’s difficult to help if you are not providing information.
How did you install Rasa-X? Did you use the quick installation script?
Have you read this? It’s a tutorial for getting logs. 
Dear @snek, I had install it using Docker Compose Installation, for Rasa X, if thier any way to export logos I appreciate sharing it with me
As I said, please check this link which explains how to access logs.:
So you do have access to your logs.
Try the command kubectl get pods --namespace rasa. This will give you a list of pods and their status.
To see the logs of a pod, enter kubectl logs <pod_name>.
Example:
user@server:~$ kubectl get pods --namespace rasa
NAME READY STATUS RESTARTS AGE
rasa-postgresql-0 1/1 Running 0 3d4h
rasa-redis-master-0 1/1 Running 0 3d4h
rasa-nginx-8665d4ccb-qk6pz 1/1 Running 0 3d4h
rasa-rabbit-0 1/1 Running 0 6h18m
rasa-rasa-production-87cb4bc4d-6jgzb 1/1 Running 0 6h19m
rasa-rasa-worker-6785f597f-k6gw6 1/1 Running 0 6h19m
rasa-rasa-x-866d9bb4df-bnhdk 1/1 Running 0 6h18m
rasa-db-migration-service-0 1/1 Running 1 3d4h
rasa-event-service-6965f9b784-skgmh 1/1 Running 6 3d4h
rasa-app-84c444875d-6h97k 1/1 Running 0 6h19m
and then
user@server:~$ kubectl logs rasa-app-84c444875d-6h97k
2021-03-01 08:22:35 INFO rasa_sdk.endpoint - Starting action endpoint server...
2021-03-01 08:22:36 INFO rasa_sdk.executor - Registered function for 'action_change_language'.
2021-03-01 08:22:36 INFO rasa_sdk.executor - Registered function for 'action_fetch_quota'.
2021-03-01 08:22:36 INFO rasa_sdk.executor - Registered function for 'action_check_weather'.
2021-03-01 08:22:36 INFO rasa_sdk.executor - Registered function for 'action_out_of_scope'.
2021-03-01 08:22:36 INFO rasa_sdk.endpoint - Action endpoint is up and running on http://localhost:5055
rasa-x_1 | [2021-03-01 14:49:22 +0000] - (sanic.access)[INFO][172.18.0.5:59482]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160 rasa-x_1 | [2021-03-01 14:49:22 +0000] - (sanic.access)[INFO][172.18.0.4:40274]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160 rasa-x_1 | [2021-03-01 14:49:32 +0000] - (sanic.access)[INFO][172.18.0.4:40282]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160 rasa-x_1 | [2021-03-01 14:49:32 +0000] - (sanic.access)[INFO][172.18.0.5:59490]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160 rasa-x_1 | [2021-03-01 14:49:42 +0000] - (sanic.access)[INFO][172.18.0.4:40290]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160 rasa-x_1 | [2021-03-01 14:49:42 +0000] - (sanic.access)[INFO][172.18.0.5:59498]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160 rasa-x_1 | [2021-03-01 14:49:52 +0000] - (sanic.access)[INFO][172.18.0.5:59506]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160 rasa-x_1 | [2021-03-01 14:49:52 +0000] - (sanic.access)[INFO][172.18.0.4:40298]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160 rasa-x_1 | [2021-03-01 14:50:02 +0000] - (sanic.access)[INFO][172.18.0.5:59516]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160 rasa-x_1 | [2021-03-01 14:50:02 +0000] - (sanic.access)[INFO][172.18.0.4:40308]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160 rasa-x_1 | [2021-03-01 14:50:12 +0000] - (sanic.access)[INFO][172.18.0.4:40316]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160 rasa-x_1 | [2021-03-01 14:50:12 +0000] - (sanic.access)[INFO][172.18.0.5:59520]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160 rasa-x_1 | [2021-03-01 14:50:22 +0000] - (sanic.access)[INFO][172.18.0.5:59530]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160 rasa-x_1 | [2021-03-01 14:50:22 +0000] - (sanic.access)[INFO][172.18.0.4:40322]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160 rasa-x_1 | [2021-03-01 14:50:32 +0000] - (sanic.access)[INFO][172.18.0.4:40330]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160 rasa-x_1 | [2021-03-01 14:50:32 +0000] - (sanic.access)[INFO][172.18.0.5:59538]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160 rasa-x_1 | [2021-03-01 14:50:42 +0000] - (sanic.access)[INFO][172.18.0.5:59544]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160 rasa-x_1 | [2021-03-01 14:50:42 +0000] - (sanic.access)[INFO][172.18.0.4:40340]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160 rasa-x_1 | [2021-03-01 14:50:52 +0000] - (sanic.access)[INFO][172.18.0.4:40346]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160 rasa-x_1 | [2021-03-01 14:50:52 +0000] - (sanic.access)[INFO][172.18.0.5:59550]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160 rasa-x_1 | [2021-03-01 14:51:02 +0000] - (sanic.access)[INFO][172.18.0.4:40354]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160 rasa-x_1 | [2021-03-01 14:51:02 +0000] - (sanic.access)[INFO][172.18.0.5:59562]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=YGAgSwF72apLbMz 404 160
@snek thank you for helping
Train your model, let it fail, then search the logs of the pods for somewhere where it says ERROR.
Don’t just search the logs of the rasa-rasa-x pod. Try rasa-rasa-production, rasa-rasa-worker, and rasa-app. Look if there are errors.
Also, after doing kubectl get pods --namespace rasa, make sure all your pods have STATUS running.
rasa-production_1 | 2021-02-28 14:47:46.335572: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
rasa-production_1 | 2021-02-28 14:47:51.922907: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
rasa-production_1 | 2021-02-28 14:47:51.922983: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)
rasa-production_1 | 2021-02-28 14:47:51.923033: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (2de3d499970e): /proc/driver/nvidia/version does not exist
root@vmi375351:/etc/rasa# Attaching to rasa_rasa-production_1
rasa-production_1 | Starting Rasa X in production mode... 🚀
rasa-production_1 | /opt/venv/lib/python3.8/site-packages/tzlocal/unix.py:158: UserWarning: Can not find any timezone configuration, defaulting to UTC.
rasa-production_1 | warnings.warn('Can not find any timezone configuration, defaulting to UTC.')
rasa-production_1 | 2021-02-28 18:19:31.591269: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
rasa-production_1 | 2021-02-28 18:19:31.592737: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
rasa-production_1 | 2021-02-28 18:20:27.162756: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
rasa-production_1 | 2021-02-28 18:20:27.162882: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)
rasa-production_1 | 2021-02-28 18:20:27.163868: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (2de3d499970e): /proc/driver/nvidia/version does not exist
rasa-production_1 | /opt/venv/lib/python3.8/site-packages/tzlocal/unix.py:158: UserWarning: Can not find any timezone configuration, defaulting to UTC.
rasa-production_1 | warnings.warn('Can not find any timezone configuration, defaulting to UTC.')
rasa-production_1 | Starting Rasa X in production mode... 🚀
rasa-production_1 | 2021-03-01 14:57:12.175961: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
rasa-production_1 | 2021-03-01 14:57:12.176253: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
rasa-production_1 | 2021-03-01 14:57:54.776976: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
rasa-production_1 | 2021-03-01 14:57:54.777273: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)
rasa-production_1 | 2021-03-01 14:57:54.777630: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (2de3d499970e): /proc/driver/nvidia/version does not exist
@ChrisRahme thanks, that logo I got
Hmm nothing seems wrong there. This is just because you don’t have NVIDIA CUDA, nothing to be worried about.
Did you train and let it fail before checking the logs? Try other pods.
exactly nothing wrong, but still fail
/etc/rasa# docker-compose logs rasa-x pod ERROR: No such service: pod
also upload filed
Replace rasa-x with the complete name of the pod you found by doing kubectl get pods.
For example, rasa-rasa-x-866d9bb4df-bnhdk.