RasaX cert issue connecting to cloud DB

We’re trying to connect Rasa X to a could instance of postgres DB but getting this error. (The postgres instance is v10)

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) 
could not open certificate file "/root/.postgresql/postgresql.crt": 
Permission denied FATAL:  no pg_hba.conf entry for host "xxx.xxx.xxx.xxx", user 
".......", database ".....", SSL off

We don’t see a place to add the cert in helm chart config though.

1 Like

FYI I’ve same issue (with rasa x 0.28.1), I followed “manual docker install”, except having postgresql db on external server (so excluded postgresql container from docker compose config, of course all environment variables that configures db are present and are ok). Seems to be some SQLAlchemy-related issue but have no idea what’s going on. Hey rasa team, could you look at this?

UPDATE: seems to be regression, it works with rasa-x images 0.27.x… hope will be fixed soon :wink:

Thanks for jumping in on this @krebs, we’ll take a look. Looks like @tatianaf used helm charts to deploy, is that the same for you @krebs?

No helm, as I wrote above, I’m using manual docker install procedure ;p Here is my docker compose, slightly modified version of “yours docker-compose.ce.yml” - I’m using custom, existing rasa bot instance as “rasa-production” (rasa-bot service in my config), custom actions service, external cloud database (no db container) and own web serwer (no nginx container). It “works”* (no above exceptions) with rasax version 0.27.6 image, but when changing it to later one it explodes with same exception as Tatiana presented here.

(*) actually rasax is not working at all with this config (probably problem is caused by custom rasa bot & actions I’m trying to “resue” with rasax - seems not supporting that properly) - after entering RasaX UI there are no stories, no training data, cannot talk with bot there (no response, nothing in logs), cannot connect to git repository (exceptions “AttributeError: ‘NoneType’ object has no attribute ‘git’” in rasax logs after repo connection attempt)… but this topic covers only a external db connection issue :wink:

Additional info: I’m not deploying via “docker-compose up” but using docker swarm “stack deploy” (on docker engine v18.09 @ ubuntu server v16.04).

UPDATE: after few days spent on it I know little more, so I’m sharing this knowledge with others, who fights with similar problems…

Firstly - official rasa-x docker compose file linked on the current version of (really poor ;p) documentation (https://storage.googleapis.com/rasa-x-releases/0.28.3/docker-compose.ce.yml) has a bug - there is missing LOCAL_MODE variable (it was present in 0.27.x composes but for unknown reason was removed later, without any mention in documentation about it and how to set it up properly). This miss makes rasa-x entering in “local mode” (whatever it is) and this is a cause why there are exceptions when trying connect a git repository. When rasa-x runs “not in local mode” connecting repos works properly.

Secondly - unfortunately seems that rasa-x does not work at all with external databases, when not in “local mode”. Previously I was unaware, that I’m running “in local mode” (I took & use such compose form official docs without LOCAL_MODE) so I thought “everything is ok” as there were no db exceptions during service start @v0.27.6 image, but as soon I set this LOCAL_MODE to false in my compose, rasa-x crashes when trying to connect with external database with db error discussed here - no matter what version of rasa image is used - looks like it works only with “docker db container” (as presented in official docker composes) and I’m waiting for any proof that I’m wrong on it ;p

Hey rasa team, could you start testing & documenting your products more seriously? Starting versions “from zero” is not an explanation to providing such crappy stuff… I’m really disappointed and give up with rasa-x for few weeks or months, until all such problems will be solved :frowning:

version: "3.7"

x-database-credentials: &database-credentials
  DB_HOST: "xxxxxxxxxxxxxxxxxxxxxx"
  DB_PORT: "xxx"
  DB_USER: "xxxxxxx"
  DB_PASSWORD: "${DB_PASSWORD}"
  DB_LOGIN_DB: "${DB_LOGIN_DB:-rasa}"

x-rabbitmq-credentials: &rabbitmq-credentials
  RABBITMQ_HOST: "rabbit"
  RABBITMQ_USERNAME: "user"
  RABBITMQ_PASSWORD: ${RABBITMQ_PASSWORD}

x-redis-credentials: &redis-credentials
  REDIS_HOST: "redis"
  REDIS_PORT: "6379"
  REDIS_PASSWORD: ${REDIS_PASSWORD}
  REDIS_DB: "1"

x-duckling-credentials: &duckling-credentials
  RASA_DUCKLING_HTTP_URL: "http://duckling:9000"

x-rasax-credentials: &rasax-credentials
  RASA_X_HOST: "http://rasa-x:5002"
  RASA_X_USERNAME: ${RASA_X_USERNAME}
  RASA_X_PASSWORD: ${RASA_X_PASSWORD}
  RASA_X_TOKEN: ${RASA_X_TOKEN}
  JWT_SECRET: ${JWT_SECRET}
  RASA_USER_APP: "http://rasa-actions:5055"
  RASA_PRODUCTION_HOST: "http://rasa-bot:5005"
  RASA_WORKER_HOST: "http://rasa-worker:5005"
  RASA_TOKEN: ${RASA_TOKEN}

x-rasa-credentials: &rasa-credentials
  <<: *rabbitmq-credentials
  <<: *rasax-credentials
  <<: *database-credentials
  <<: *redis-credentials
  <<: *duckling-credentials
  RASA_TOKEN: ${RASA_TOKEN}
  RASA_MODEL_PULL_INTERVAL: 10
  RABBITMQ_QUEUE: "rasa_production_events"

x-rasa-services: &default-rasa-service
  restart: always
  image: "rasa/rasa:${RASA_VERSION}-full"
  expose:
    - "5005"
  command: >
    x
    --debug
    --no-prompt
    --production
    --config-endpoint http://rasa-x:5002/api/config?token=${RASA_X_TOKEN}
    --port 5005
    --jwt-method HS256
    --jwt-secret ${JWT_SECRET}
    --auth-token '${RASA_TOKEN}'
    --cors "*"
  depends_on:
    - rasa-x
    - rabbit
    - redis

services:
  rasa-x:
    restart: always
    image: "rasa/rasa-x:${RASA_X_VERSION}"
    ports:
      - 5100:5002
    volumes:
      - ./models:/app/models
      - ./environments.yml:/app/environments.yml
      - ./credentials.yml:/app/credentials.yml
      - ./endpoints.yml:/app/endpoints.yml
      - ./logs:/logs
      - ./auth:/app/auth
      - ./connectors:/app/connectors
      - ./utils:/app/utils
    environment:
      <<: *database-credentials
      <<: *rasa-credentials
      SELF_PORT: "5002"
      DB_DATABASE: "${DB_DATABASE:-rasa}"
      RASA_MODEL_DIR: "/app/models"
      PASSWORD_SALT: ${PASSWORD_SALT}
      RABBITMQ_QUEUE: "rasa_production_events"
      RASA_X_USER_ANALYTICS: "0"
      SANIC_RESPONSE_TIMEOUT: "3600"

  rasa-bot:
    <<: *default-rasa-service
    ports:
     - 5101:5005
    environment:
      <<: *rasa-credentials
      RASA_ENVIRONMENT: "production"
      DB_DATABASE: "tracker"
      RASA_MODEL_SERVER: "http://rasa-x:5002/api/projects/default/models/tags/production"
    env_file:
      - .env
    volumes:
      - ./models:/app/models
      - ./environments.yml:/app/environments.yml
      - ./credentials.yml:/app/credentials.yml
      - ./endpoints.yml:/app/endpoints.yml
      - ./logs:/logs
      - ./auth:/app/auth
      - ./connectors:/app/connectors
      - ./utils:/app/utils

  rasa-worker:
    <<: *default-rasa-service
    environment:
      <<: *rasa-credentials
      RASA_ENVIRONMENT: "worker"
      DB_DATABASE: "worker_tracker"
      RASA_MODEL_SERVER: "http://rasa-x:5002/api/projects/default/models/tags/production"
    env_file:
      - .env
    volumes:
      - ./models:/app/models
      - ./environments.yml:/app/environments.yml
      - ./credentials.yml:/app/credentials.yml
      - ./endpoints.yml:/app/endpoints.yml
      - ./logs:/logs
      - ./auth:/app/auth
      - ./connectors:/app/connectors
      - ./utils:/app/utils

  rasa-actions:
    image: custom_actions:latest
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    volumes:
      - ./db:/app/db
      - ./actions:/app/actions
      - ./connectors:/app/connectors
      - ./utils:/app/utils
      - ./credentials.yml:/app/credentials.yml
      - ./logs:/app/logs
    expose:
      - "5055"
    command:
      - start
      - --debug
      - --actions
      - actions
    env_file:
      - .env
    depends_on:
      - rasa-bot

  rabbit:
    restart: always
    image: "bitnami/rabbitmq:3.8.3"
    environment:
      RABBITMQ_HOST: "rabbit"
      RABBITMQ_USERNAME: "user"
      RABBITMQ_PASSWORD: ${RABBITMQ_PASSWORD}
      RABBITMQ_DISK_FREE_LIMIT: "{mem_relative, 0.1}"
    expose:
      - "5672"

  duckling:
    restart: always
    image: "rasa/duckling:0.1.6.3"
    expose:
      - "9000"
    command: ["duckling-example-exe", "-p 9000", "--no-access-log", "--no-error-log"]

  redis:
    restart: always
    image: "bitnami/redis:5.0.8"
    environment:
      REDIS_PASSWORD: ${REDIS_PASSWORD}
    expose:
      - "6379"
1 Like

Hi,

The environment variable LOCAL_MODE got removed with 0.28 (see https://rasa.com/docs/rasa-x/changelog/rasa-x-changelog/#id19), that’s why it is not set anymore in the image. Internally, everything stays the same though. The setting is done automatically whenever Rasa X is started via the docker file. There is no need and no option to set this up (that’s why it’s not in the documentation). So starting from a docker image without LOCAL_MODE in 0.28 is still the same as starting from one with LOCAL_MODE = False in 0.27.

We will address the issue with connecting to external databases.

What do you mean when you say “as soon I set this LOCAL_MODE to false in my compose”? In 0.28 setting this won’t do anything, and in 0.27 it is already set to false in the image. What version were you using and what modifications did you make to the file when it worked vs. when it didn’t work?

Sure, no doubt that default config described in docs is working properly, but that’s not my case. My setup is different, I’m even not using “docker-compose up” to deploy containers (use “docker stack” for swarm deploys instead), I’m deploying on quite old (but still commonly used) system with somehow “outdated” docker engine etc - maybe this also matters. Anyway, despite I have “–production” parameter for rasa-x in my compose, rasa-x starts in local mode unless I explicitly add LOCAL_MODE=false to the docker compose in the same place, from where was removed. Actually have no idea on what “mode” rasa-x started, but side effects of missing LOCAL_MODE=false variable was unable to setup git repository connection (excepions as described occured) and “no exceptions” with configured external databae (I have no idea if rasa really used that external database in this case, but at least no exceptions happens). With LOCAL_MODE=false in docker compose, connecting git repository worked (tested with temporally configured docker db container) but as soon I remove docker db container and configured external db, rasa-x crashes.

I’ve spend few days on “fighting” with it, made a lot of attempts to solve that problem by changing copmoses/environment variables etc but sorry, did not wrote down every step I did and even don’t remember most of them, maybe I missed sth or did sth wrong, maybe it’s a “rasa-x fault”. I have no more spare time to spend on it, so could not assist you on that. I just wanted to share my felling that rasa-x is still too unstable and not well tested in anything else than “default blessed environment”. I believe it’s only a matter of time when all problems will be solved, but for me current rasa-x is a powerful toy rather than production-grade tool. Even if external databases starts working as expected, there is too much “issues” for taking rasa-x into my dev stack:

  • Unable to split nlu/stories/congigs to different structure than “default project layout” - totally wrong “assumption”, as this is quite simple to code (rasa open source supports at least passing “data directory” parameter and splitting files inside them)
  • No support for “retrieval actions” (IMO one of most useful rasa experimental features and my bots uses them a lot),
  • Not preserving existing comments in nlu/domain files (rasa-x overwrites them in repository by “cleaned & sanitized” version, generated from model - why not working as “git rebase” to allow merge new one with existing file somehow, even manually?),
  • domain.yml cannot be splitted (actually it’s also rasa open source problem too)- with lot of defined utters (and other stuff) there, it’s hard to maintain such big domain files - again, providing “domain directory” param with splitted files inside & merging them somehow is really not a rocked science…
  • Too flat UI structure for nlu / responses / stories - (I know, that’s a feature for “non devs” to make it simple;) - i.e. when stories files are splitted, why not make a tree-like structure and keep all stories from appropriate file on proper tree leaf/tab? I cannot imagine big bots development without that and it’s not only about “too much scrolling on long lists” :frowning:

So sure, rasa-x is a great tool for “non-devs”, but not fits well to development of more complicated bots with custom structure etc due to some strange “limitations & assumptions” that authors made on it :frowning: But I’ll keep my eye on rasa-x further development as it looks really promising, so keep on working on making it better and better & good luck with that :slight_smile:

Hi @krebs, thanks for all of your feedback. It’s really useful and i’ve made sure it gets back to the product team. Some of the things you’ve addressed are things we are already working on improving and others are great new ideas.

A few things that might be helpful:

No support for “retrieval actions” (IMO one of most useful rasa experimental features and my bots uses them a lot),

Support for training with retrieval intents was already introduced on 0.28.0, but I know you’ve been having trouble upgrading.

but as soon I remove docker db container and configured external db, rasa-x crashes.

Yes, rasa-x relies on an external DB. Without it, there is no place to store the information and it will crash.