Rasa-X not responding on EC2 (with docker) after upgrading from 0.29 to 0.34

Hi,

We have been using Rasa-X v0.29 on EC2 (with a docker) over the past few month, and it worked perfectly.
We upgraded for Rasa-X v0.34 today (deployed on EC2, with a docker), but Rasa-X doesn’t respond back.
It does not identify the intent/story, and does not utter back.
We couldn’t spot any error in the docker-compose logs, and the docker images (+ps -a) seems OK.
Can you please support us?
What might cause this erroneous behavior? How can we debug it?

Thanks ahead,
Newt Team

Hey @sgino209, how did you upgrade from 0.29 to 0.34?

Hi Akelad,

Thanks for your quick response.

We were following your formal webpage guidelines, and launched the install.sh script via curl, from the EC2 machine where RasaX is deployed on.

We then built and pushed the docker image, following by docker-compose down+up.
This flow worked well so far on version 0.29.

Please note that RasaX at r0.34 works well locally, and the above issue only refers for the EC2 deployment (with a Docker).

Can you please support us with that issue?

Rasa-X UI snapshot:

Docker status:

Thanks,
Shahar

I think I am having this problem too:

could you share the logs of both the rasa x and rasa production containers when you’re having a conversation with the bot?

actually, if you edit the containers to start in debug mode that would also be great. For the production one that would be done by adding --debug to the rasa command, and for Rasa X it would be done by adding the environment variable DEBUG_MODE: "true"

Hi Akelad,

We are in a slightly different position now, differs from the originally stated problem.

We tried to make a fresh start on our EC2 machine, removed all docker images (sudo docker system prune -a) and uninstalled Rasa (curl -s get-rasa-x.rasa.com | sudo bash -s -- --uninstall).

We then performed the following actions:

% sudo curl -sSL -o install.sh 'https://storage.googleapis.com/rasa-x-releases/0.34.0/install.sh'
% sudo bash install.sh
% sudo docker build . -t newt/rasabot_app:newt_actions
% sudo docker push newt/rasabot_app:newt_actions
% sudo docker-compose down --remove-orphans
% sudo docker-compose up -d

We ended up with the following Docker images (sudo docker images):

REPOSITORY             TAG                 IMAGE ID            CREATED             SIZE
newt/rasabot_app       newt_actions        d6ec9dcaea31        8 minutes ago       281MB
rasa/rasa-sdk          latest              1864878bf5a9        53 minutes ago      151MB
bitnami/rabbitmq       3.8.9               d4c196273f9a        2 hours ago         244MB
rasa/rasa-x            0.34.0              1bd0f7d7c23c        12 days ago         2.25GB
rasa/nginx             0.34.0              ea905db30635        12 days ago         90.2MB
rasa/rasa              2.1.2-full          1ae20eafdcbd        2 weeks ago         1.91GB
bitnami/postgresql     11.9.0              0964f45e6b89        5 weeks ago         250MB
bitnami/redis          6.0.8               da471e740b1c        7 weeks ago         103MB
rasa/duckling          0.1.6.4             1253a54501d5        2 months ago        147MB

And with the following Docker status (sudo docker ps -a):

CONTAINER ID        IMAGE                               COMMAND                  CREATED             STATUS                          PORTS                                         NAMES
c79290a72269        rasa/nginx:0.34.0                   "/opt/bitnami/entryp…"   5 minutes ago       Up 5 minutes                    0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp   rasa_nginx_1
c07f6a993fdc        newt/rasabot_app:newt_actions       "./entrypoint.sh sta…"   5 minutes ago       Up 5 minutes                    5055/tcp                                      rasa_app_1
072cb07b8033        rasa/rasa:2.1.2-full                "rasa x --no-prompt …"   5 minutes ago       Up 26 seconds                   5005/tcp                                      rasa_rasa-production_1
33feea12c737        rasa/rasa:2.1.2-full                "rasa x --no-prompt …"   5 minutes ago       Up 26 seconds                   5005/tcp                                      rasa_rasa-worker_1
b39b14ab26bb        rasa/rasa-x:0.34.0                  "sh -c 'user_id=$(id…"   5 minutes ago       Up 5 minutes                    5002/tcp                                      rasa_rasa-x_1
c3b1a7c46a05        rasa/rasa-x:0.34.0                  "python -m rasax.com…"   5 minutes ago       Restarting (0) 36 seconds ago                                                 rasa_db-migration_1
f2174f6e4158        bitnami/postgresql:11.9.0           "/opt/bitnami/script…"   5 minutes ago       Up 5 minutes                    5432/tcp                                      rasa_db_1
6fa10883dddd        bitnami/redis:6.0.8                 "/opt/bitnami/script…"   5 minutes ago       Up 5 minutes                    6379/tcp                                      rasa_redis_1
c3873815a7f7        bitnami/rabbitmq:3.8.9              "/opt/bitnami/script…"   5 minutes ago       Up 5 minutes                    4369/tcp, 5672/tcp, 15672/tcp, 25672/tcp      rasa_rabbit_1
06b3d597e0ae        rasa/duckling:0.1.6.4               "duckling-example-ex…"   5 minutes ago       Up 5 minutes                    8000/tcp                                      rasa_duckling_1

Now, when entering the web URL, we get the following error message (from /api/health):

{"database_migration":{"status":"pending","current_revision":[],"target_revision":["052172fdb4b3"],"progress_in_percent":0.0}}

We see many error messages in sudo docker-compose logs app, mainly from db-migration_1.

Can you please support us?

Thanks ahead,
Newt team

Can you show the error messages in rasa_db-migration_1?

Hi Akelad,

Thanks for your quick reply.

We’ve managed to make some progress, by conducting the following actions:

Backup our old RASA (r0.29) in /etc/rasa.old:

% cd ~
% sudo mv /etc/rasa /etc/rasa.old

Fresh Start (r0.34):

curl -sSL -o install.sh https://storage.googleapis.com/rasa-x-releases/0.34.0/install.sh
sudo bash ./install.sh
cd /etc/rasa
wget -qO docker-compose.yml https://storage.googleapis.com/rasa-x-releases/0.34.0/docker-compose.ce.yml
wget -qO rasa_x_commands.py https://storage.googleapis.com/rasa-x-releases/0.34.0/rasa_x_commands.py

Restore custom settings from our previous revision:

sudo cp -r /etc/rasa.old/actions ./
sudo cp /etc/rasa.old/Dockerfile ./
sudo cp /etc/rasa.old/docker-compose* ./
sudo cp /etc/rasa.old/credentials.yml ./

Build and Push docker images:

sudo docker build . -t newt/rasabot_app:newt_actions
sudo docker push newt/rasabot_app:newt_actions

Launch:

sudo docker-compose up -d

Now, we manage to see Rasa-X login page, but we cannot figure out what credentials shall be applied (password).

We tried to apply the rasa_x_commands.py script, but face same issue as reported here --> Error when I try to resolve set admin password for rasa x

We tried either of the below commands:

% sudo python rasa_x_commands.py create --update admin me q1w2e3r4
% docker-compose exec rasa-x bash -c 'python3 /app/scripts/manage_users.py create --update admin me q1w2e3r4'

Both have ended up with:

ERROR: Couldn't connect to Docker daemon at http+docker://localhost - is it running?

If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.

Can you please guide us on that?

Thanks ahead,
Newt Team

If you started from scratch, I assume that your password salt environment variable got overwritten (it lives in the .env file), which is causing these errors.

As for the error messages you’re getting now, I’m not sure, this seems to be an issue with your docker setup

Thanks Akelad.
Issue has been solved.
We’ve published our happy-path guide at this link, for assisting others who might struggle with version update