RASA-X deploy (EC2) with docker ends-up with Nginx 504 gateway time-out

Hi,

We’re trying to deploy RASA-X v0.41.0 with docker on EC2 machine.
We performed the following steps, which ended up eventually with an Nginx 504 gateway time-out error.

We first installed RASA-X v0.41.1, and ramped-up docker images:

% curl -sSL -o install.sh https://storage.googleapis.com/rasa-x-releases/0.41.1/install.sh
%  sudo bash ./install.sh
% cd /etc/rasa
% sudo docker-compose up -d

Then we registered the admin user:

% wget -qO docker-compose.yml https://storage.googleapis.com/rasa-x-releases/0.41.1/docker-compose.ce.
% wget -qO rasa_x_commands.py https://storage.googleapis.com/rasa-x-releases/0.41.1/rasa_x_commands.py
% sudo python rasa_x_commands.py create admin me <PASSWORD>
% sudo python rasa_x_commands.py create --update admin me <PASSWORD>

Next we’ve imported the following files, from another (working) environment we maintain:

% sudo cp /etc/rasa.old/Dockerfile ./Dockerfile
% sudo cp -r /etc/rasa.old/actions ./
% sudo cp /etc/rasa.old/docker-compose.override.yml ./

Finally, we added action server data, through DockerHub, and refreshed docker accordingly:

% sudo docker login
% sudo docker build . -t <USER>/<REPO>:<TAG>
% sudo docker push <USER>/<REPO>:<TAG>
% sudo docker-compose up -d

Following is the .env content:

PASSWORD_SALT=uDjDlZWKGmxKzDV
RASA_X_VERSION=0.41.1
RASA_VERSION=2.6.2
RASA_X_DEMO_VERSION=0.40.0
RASA_TELEMETRY_ENABLED=true
REDIS_PASSWORD=LDPTuvIHwLJUON.
RABBITMQ_PASSWORD=UlhDKeqJmwNIsSv
RASA_X_TOKEN=kPJdW1WXSfTilgP
JWT_SECRET=VwkZAb6xVtFemVEXcepWKIH3L
RASA_TOKEN=tmBf7JOJkY6ffD9
DB_PASSWORD=yfdzxljNqSChiuW

Following is the docker-compose.override.yml content:

version: '3.4'
services:
  app:
    image: <USER>/<REPO>:<TAG>
  rasa-x:
    environment:
      - DEFAULT_STREAM_READING_TIMEOUT_IN_SECONDS=25

Following is the Dockerfile content:

# Extend the official Rasa SDK image
FROM rasa/rasa-sdk:latest

# Add environment variables
ENV DB_AWS_ACCESS_KEY_ID=<OUR_ACCESS_KEY_ID>
ENV DB_AWS_SECRET_ACCESS_KEY=<OUR_SECRET_ACCESS_KEY>

# Use subdirectory as working directory
WORKDIR /app

# Copy any additional custom requirements, if necessary (uncomment next line)
COPY actions/requirements-actions.txt ./

# Change back to root user to install dependencies
USER root

# Install extra requirements for actions code, if necessary (uncomment next line)
RUN pip install -r requirements-actions.txt

# Copy actions folder to working directory
COPY ./actions /app/actions

# By best practices, don't run the code with root user
USER 1001

Following is the logs tail (sudo docker-compose logs --tail=100):

db_1               | postgresql 06:39:01.79
db_1               | postgresql 06:39:01.79 Welcome to the Bitnami postgresql container
db_1               | postgresql 06:39:01.80 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql
db_1               | postgresql 06:39:01.80 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql/issues
db_1               | postgresql 06:39:01.80
db_1               | postgresql 06:39:01.83 INFO  ==> ** Starting PostgreSQL setup **
db_1               | postgresql 06:39:01.85 INFO  ==> Validating settings in POSTGRESQL_* env vars..
db_1               | postgresql 06:39:01.86 INFO  ==> Loading custom pre-init scripts...
db_1               | postgresql 06:39:01.86 INFO  ==> Initializing PostgreSQL database...
db_1               | postgresql 06:39:01.88 INFO  ==> pg_hba.conf file not detected. Generating it...
db_1               | postgresql 06:39:01.88 INFO  ==> Generating local authentication configuration
db_1               | postgresql 06:39:03.18 INFO  ==> Starting PostgreSQL in background...
db_1               | postgresql 06:39:03.57 INFO  ==> Creating user admin
db_1               | postgresql 06:39:03.60 INFO  ==> Granting access to "admin" to the database "rasa"
db_1               | postgresql 06:39:03.65 INFO  ==> Configuring replication parameters
db_1               | postgresql 06:39:03.69 INFO  ==> Configuring fsync
db_1               | postgresql 06:39:03.73 INFO  ==> Loading custom scripts...
db_1               | postgresql 06:39:03.74 INFO  ==> Enabling remote connections
db_1               | postgresql 06:39:03.77 INFO  ==> Stopping PostgreSQL...
db_1               | waiting for server to shut down.... done
db_1               | server stopped
db_1               |
db_1               | postgresql 06:39:03.88 INFO  ==> ** PostgreSQL setup finished! **
db_1               | postgresql 06:39:03.91 INFO  ==> ** Starting PostgreSQL **
db_1               | 2021-08-12 06:39:03.948 GMT [1] LOG:  pgaudit extension initialized
db_1               | 2021-08-12 06:39:03.949 GMT [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
db_1               | 2021-08-12 06:39:03.949 GMT [1] LOG:  listening on IPv6 address "::", port 5432
db_1               | 2021-08-12 06:39:03.959 GMT [1] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
db_1               | 2021-08-12 06:39:03.976 GMT [136] LOG:  database system was shut down at 2021-08-12 06:39:03 GMT
db_1               | 2021-08-12 06:39:03.986 GMT [1] LOG:  database system is ready to accept connections
db_1               | 2021-08-12 06:39:40.918 GMT [189] ERROR:  column rasa_x_user.default_workspace_id does not exist at character 392
db_1               | 2021-08-12 06:39:40.918 GMT [189] STATEMENT:  SELECT rasa_x_user.username AS rasa_x_user_username, rasa_x_user.name_id AS rasa_x_user_name_id, rasa_x_user.password_hash AS rasa_x_user_password_hash, rasa_x_user.team AS rasa_x_user_team, rasa_x_user.api_token AS rasa_x_user_api_token, rasa_x_user.project AS rasa_x_user_project, rasa_x_user.data AS rasa_x_user_data, rasa_x_user.username_is_assigned AS rasa_x_user_username_is_assigned, rasa_x_user.default_workspace_id AS rasa_x_user_default_workspace_id, rasa_x_user.authentication_mechanism AS rasa_x_user_authentication_mechanism
db_1               | 	FROM rasa_x_user
db_1               | 	WHERE rasa_x_user.username = 'me'
db_1               | 	 LIMIT 1
rasa-x_1           | [2021-08-12 07:03:13 +0000] - (sanic.access)[INFO][172.20.0.9:53936]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=kPJdW1WXSfTilgP  404 160
rasa-x_1           | [2021-08-12 07:03:14 +0000] - (sanic.access)[INFO][172.20.0.8:52500]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=kPJdW1WXSfTilgP  404 160
rasa-x_1           | [2021-08-12 07:03:23 +0000] - (sanic.access)[INFO][172.20.0.9:54006]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=kPJdW1WXSfTilgP  404 160
rasa-x_1           | [2021-08-12 07:03:24 +0000] - (sanic.access)[INFO][172.20.0.8:52570]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=kPJdW1WXSfTilgP  404 160
...
x:5002/api/projects/default/models/tags/production?token=kPJdW1WXSfTilgP  404 160
rasa-x_1           | [2021-08-12 07:06:44 +0000] - (sanic.access)[INFO][172.20.0.8:53914]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=kPJdW1WXSfTilgP  404 160
redis_1            | redis 06:39:00.80
redis_1            | redis 06:39:00.80 Welcome to the Bitnami redis container
redis_1            | redis 06:39:00.80 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-redis
redis_1            | redis 06:39:00.80 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-redis/issues
redis_1            | redis 06:39:00.80
redis_1            | redis 06:39:00.80 INFO  ==> ** Starting Redis setup **
redis_1            | redis 06:39:00.83 INFO  ==> Initializing Redis
redis_1            | redis 06:39:00.85 INFO  ==> Setting Redis config file
redis_1            | redis 06:39:00.87 INFO  ==> ** Redis setup finished! **
redis_1            |
redis_1            | redis 06:39:00.89 INFO  ==> ** Starting Redis **
redis_1            | 1:C 12 Aug 2021 06:39:00.903 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1            | 1:C 12 Aug 2021 06:39:00.903 # Redis version=6.2.4, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1            | 1:C 12 Aug 2021 06:39:00.903 # Configuration loaded
redis_1            | 1:M 12 Aug 2021 06:39:00.904 * monotonic clock: POSIX clock_gettime
redis_1            | 1:M 12 Aug 2021 06:39:00.907 * Running mode=standalone, port=6379.
redis_1            | 1:M 12 Aug 2021 06:39:00.907 # Server initialized
redis_1            | 1:M 12 Aug 2021 06:39:00.907 * Ready to accept connections
rasa-x_1           | [2021-08-12 07:06:53 +0000] - (sanic.access)[INFO][172.20.0.9:55418]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=kPJdW1WXSfTilgP  404 160
rasa-x_1           | [2021-08-12 07:06:54 +0000] - (sanic.access)[INFO][172.20.0.8:53982]: GET http://rasa-x:5002/api/projects/default/models/tags/production?token=kPJdW1WXSfTilgP  404 160

Can you please support us?

Thanks ahead,

Newt Team

We’ve managed to overcome this issue, by starting a new EC2 machine on Amazon.
We do not understand what was causing this behavior, and hope not to encounter it again at the future.