Rasa X CE Kubernetes Issue

I am trying to setup rasa x on a self-hosted kubernetes cluster. I am having issues with the talk page. I can type and send a message but I never see an intent listed under the message. If I refresh the page I can see the response from the bot above my message and the intent is then listed. When I open the conversations page the conversations do not load. The models and training pages work without any issues.

Here is my configmap.

kubectl describe configmap configuration-files
Name:         configuration-files
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
environments:
----
rasa:
    production:
      url: http://rasa-production:5005
      token: ${RASA_TOKEN}
    worker:
      url: http://rasa-worker:5005
      token: ${RASA_TOKEN}

rasa-credentials:
----
rasa:
    url: ${RASA_X_HOST}/api

rasa-endpoints:
----
models:
    url: ${RASA_MODEL_SERVER}
    token: ${RASA_X_TOKEN}
    wait_time_between_pulls: ${RASA_MODEL_PULL_INTERVAL}
tracker_store:
    type: sql
    dialect: "postgresql"
    url: ${DB_HOST}
    port: ${DB_PORT}
    username: ${DB_USER}
    password: ${DB_PASSWORD}
    db: ${DB_DATABASE}
    login_db: ${DB_LOGIN_DB}
event_broker:
    type: "pika"
    url: ${RABBITMQ_HOST}
    username: ${RABBITMQ_USERNAME}
    password: ${RABBITMQ_PASSWORD}
    queue: ${RABBITMQ_QUEUE}
action_endpoint:
    url: ${RASA_USER_APP}/webhook
    token:  ""

Events:  <none>

All pods are running.

kubectl get pods
NAME                                                         READY   STATUS    RESTARTS   AGE
api-69bd97474d-mfd6c                                         1/1     Running   1          20h
app-5bd5dcfb4b-7m9rv                                         1/1     Running   0          35m
db-7ffb94ccf9-tz48c                                          1/1     Running   0          35m
duckling-685df689dd-989jb                                    1/1     Running   0          28m
nginx-858c89d576-xrdjj                                       1/1     Running   0          27m
rabbit-698f496497-8nq99                                      1/1     Running   0          27m
rasa-production-98cd95644-txwdq                              1/1     Running   0          27m
rasa-worker-7c85984764-wn9rx                                 1/1     Running   0          27m
rasa-x-77594cb8f5-dzzhn                                      1/1     Running   0          27m
wayfaring-arachnid-nfs-client-provisioner-7bdf6bcf8f-ccf8l   1/1     Running   14         6d23h

There does appear to be an issue in the db pod logs.

kubectl logs db-7ffb94ccf9-tz48c

Welcome to the Bitnami postgresql container
Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql
Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql/issues
Send us your feedback at containers@bitnami.com

INFO  ==> ** Starting PostgreSQL setup **
INFO  ==> Validating settings in POSTGRESQL_* env vars..
INFO  ==> Initializing PostgreSQL database...
INFO  ==> postgresql.conf file not detected. Generating it...
INFO  ==> pg_hba.conf file not detected. Generating it...
INFO  ==> Starting PostgreSQL in background...
/tmp:5432 - accepting connections
INFO  ==> Creating user admin
INFO  ==> Grating access to "admin" to the database "rasa"
INFO  ==> Configuring replication parameters
INFO  ==> Loading custom scripts...
INFO  ==> Enabling remote connections
INFO  ==> Stopping PostgreSQL...
INFO  ==> ** PostgreSQL setup finished! **

INFO  ==> ** Starting PostgreSQL **
2019-07-22 03:19:27.253 GMT [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2019-07-22 03:19:27.253 GMT [1] LOG:  listening on IPv6 address "::", port 5432
2019-07-22 03:19:27.276 GMT [1] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2019-07-22 03:19:27.414 GMT [186] LOG:  database system was shut down at 2019-07-22 03:19:26 GMT
2019-07-22 03:19:27.452 GMT [1] LOG:  database system is ready to accept connections
2019-07-22 03:20:01.545 GMT [193] LOG:  invalid length of startup packet
2019-07-22 03:21:01.545 GMT [195] LOG:  invalid length of startup packet
2019-07-22 03:22:01.546 GMT [197] LOG:  invalid length of startup packet
2019-07-22 03:23:01.547 GMT [199] LOG:  invalid length of startup packet
2019-07-22 03:24:01.548 GMT [201] LOG:  invalid length of startup packet
2019-07-22 03:24:51.917 GMT [203] WARNING:  there is no transaction in progress
2019-07-22 03:25:01.548 GMT [206] LOG:  invalid length of startup packet
2019-07-22 03:25:04.136 GMT [204] WARNING:  there is no transaction in progress
2019-07-22 03:26:01.551 GMT [215] LOG:  invalid length of startup packet
2019-07-22 03:27:01.554 GMT [221] LOG:  invalid length of startup packet
2019-07-22 03:28:01.557 GMT [226] LOG:  invalid length of startup packet
2019-07-22 03:29:01.564 GMT [232] LOG:  invalid length of startup packet
2019-07-22 03:30:01.571 GMT [237] LOG:  invalid length of startup packet
2019-07-22 03:31:01.578 GMT [242] LOG:  invalid length of startup packet
2019-07-22 03:32:01.584 GMT [248] LOG:  invalid length of startup packet
2019-07-22 03:33:01.589 GMT [253] LOG:  invalid length of startup packet
2019-07-22 03:34:01.594 GMT [258] LOG:  invalid length of startup packet
2019-07-22 03:35:01.599 GMT [263] LOG:  invalid length of startup packet
2019-07-22 03:36:01.607 GMT [269] LOG:  invalid length of startup packet
2019-07-22 03:37:01.607 GMT [274] LOG:  invalid length of startup packet
2019-07-22 03:38:01.611 GMT [282] LOG:  invalid length of startup packet
2019-07-22 03:39:01.614 GMT [287] LOG:  invalid length of startup packet
2019-07-22 03:40:01.617 GMT [292] LOG:  invalid length of startup packet
2019-07-22 03:41:01.620 GMT [297] LOG:  invalid length of startup packet
2019-07-22 03:42:01.622 GMT [302] LOG:  invalid length of startup packet
2019-07-22 03:43:01.625 GMT [307] LOG:  invalid length of startup packet
2019-07-22 03:44:01.627 GMT [312] LOG:  invalid length of startup packet
2019-07-22 03:45:01.634 GMT [317] LOG:  invalid length of startup packet
2019-07-22 03:46:01.632 GMT [322] LOG:  invalid length of startup packet
2019-07-22 03:47:01.634 GMT [327] LOG:  invalid length of startup packet
2019-07-22 03:48:01.636 GMT [332] LOG:  invalid length of startup packet
2019-07-22 03:49:01.637 GMT [337] LOG:  invalid length of startup packet
2019-07-22 03:50:01.639 GMT [342] LOG:  invalid length of startup packet
2019-07-22 03:51:01.642 GMT [347] LOG:  invalid length of startup packet
2019-07-22 03:52:01.643 GMT [352] LOG:  invalid length of startup packet

Hi @nkane2898, welcome to the forum! Which version of rasa X are you running?

Ricwo,

I am using the 0.19.5 container image.

Name:           app-5bd5dcfb4b-w5lbr
Image:          rasa/rasa-x-demo:latest

Name:           db-7ffb94ccf9-wbqm7
Image:          bitnami/postgresql:11.2.0

Name:           duckling-685df689dd-989jb
Image:         rasa/duckling:latest

Name:           nginx-858c89d576-lzqpb
Image:          rasa/nginx:0.19.5

Name:           rabbit-698f496497-7wl6j
Image:          bitnami/rabbitmq:3.7.15

Name:           rasa-production-98cd95644-fsfvr
Image:         rasa/rasa:1.1.7-full

Name:           rasa-worker-7c85984764-gv65c
Image:         rasa/rasa:1.1.7-full

Name:           rasa-x-77594cb8f5-qvsfv
Image:          rasa/rasa-x:0.19.5

@nkane2898 how did you generate your deployment files? Would you mind sharing the contents of each of your deployment configs?

I followed the steps here Openshift-kubernetes. I used the compose file from here Docker Compose CE. Attached is a zip folder of the configmap, deployments, services, and persistent volume claims generated by kompose. I have removed the passwords from the files. rasa-kubernetes.zip (12.7 KB)

Hi @nkane2898, the instructions you found are mainly geared towards openshift and we’re aware of some issues that may come up if they’re applied literally to kubernetes.

I cannot see anything obviously wrong with your config files. We’re working on redesigning the cluster deployment process using helm charts at the moment. I will try to ping you here once that’s available

Thank you for the update. I’ve been running the rasa/rasa-x:0.20.0, rasa/nginx:0.20.0, and rasa/rasa:1.1.7-full images for the past couple of days which appears to have resolved the issue.