Kubernetes deployment failing with not a lot of debug logs

[Deprecated] Rasa X Community Edition
1

I’m installing for the first time. After deployment, most of the pods are having issues.

`   NAME                                         READY   STATUS                       RESTARTS   AGE`
    pod/rasax-app-5964fb46fc-fmkz4               0/1     CreateContainerConfigError   0          16m
    pod/rasax-duckling-79db85f94d-p7s5w          0/1     CreateContainerConfigError   0          16m
    pod/rasax-event-service-579ff894d6-s9l6l     0/1     CrashLoopBackOff             7          16m
    pod/rasax-nginx-56bbfb78ff-5nbkz             0/1     CrashLoopBackOff             7          16m
    pod/rasax-rabbit-0                           0/1     Pending                      0          15m
    pod/rasax-rasa-production-5d5485574d-lqm6z   1/1     Running                      6          16m
    pod/rasax-rasa-worker-795bb7445f-g6sz6       1/1     Running                      6          15m
    pod/rasax-rasa-x-5c9b98f4bf-c85pf            0/1     CrashLoopBackOff             7          15m
    pod/rasax-redis-master-0                     0/1     Pending                      0          15m

I do have debugging turned on in values.yml, but I’m not seeing a lot of log message :

debugMode: "true"

Also, when I try to get the logs for rasa-x, I get the following error message :

kubectl -n $NS logs rasax-rasa-x-5c9b98f4bf-c85pf
sh: 1: cannot create /etc/passwd: Permission denied

Any tips on debugging ?

2

I did get some logs from nginx and rasa-production …

$ stern --namespace $NS rasax.* -t --since 10m

+ rasax-event-service-579ff894d6-67x87 › rasa-x
+ rasax-nginx-56bbfb78ff-vrpvr › rasa-x
rasax-event-service-579ff894d6-67x87 rasa-x 2020-04-13T15:54:44.293830000Z INFO:__main__:Waiting until database migrations have been executed...
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:44.676543000Z /opt/bitnami/entrypoint.sh: line 11: /opt/bitnami/nginx/conf/conf.d/rasax.nginx.tmp: Permission denied
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:44.679891000Z mv: cannot stat '/opt/bitnami/nginx/conf/conf.d/rasax.nginx.tmp': No such file or directory
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:44.680319000Z SSL encryption is not used since no certificates were provided.
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:44.682267000Z sed: couldn't open temporary file /opt/bitnami/nginx/conf/conf.d/sedgnbZjo: Permission denied
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:44.690667000Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:44.690894000Z                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: app
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:44.720224000Z 6
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:45.730254000Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:45.730554000Z                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: app
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:45.855361000Z 6
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:46.864744000Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:46.865136000Z                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: app
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:46.869731000Z 6
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:47.879012000Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:47.882384000Z                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: app
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:47.884116000Z 6
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:48.902433000Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:48.905689000Z                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: app
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:48.906115000Z 6
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:49.917100000Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:49.920478000Z                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: app
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:49.920884000Z 6
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:50.930666000Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:50.934304000Z                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: app
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:50.934727000Z 6
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:51.946633000Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:51.949756000Z                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: app
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:51.950197000Z 6
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:52.960283000Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:52.963527000Z                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: app
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:52.963946000Z 6
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:53.974987000Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:53.978184000Z                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: app
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:53.978643000Z 6
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:54.982042000Z Custom action container not found at host 'app'. Excluding it from the Nginx configuration.
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:54.985306000Z sed: couldn't open temporary file /opt/bitnami/nginx/conf/conf.d/sed7YGFma: Permission denied
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:54.985535000Z sed: couldn't open temporary file /opt/bitnami/nginx/conf/conf.d/sedMtkpca: Permission denied
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:54.985825000Z >> exec docker CMD
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:54.986123000Z nginx
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:54.987291000Z 2020/04/13 15:54:54 [emerg] 1#0: host not found in upstream "${RASA_PRODUCTION_HOST}" in /opt/bitnami/nginx/conf/conf.d/rasax.nginx:2
rasax-nginx-56bbfb78ff-vrpvr rasa-x 2020-04-13T15:54:54.987544000Z nginx: [emerg] host not found in upstream "${RASA_PRODUCTION_HOST}" in /opt/bitnami/nginx/conf/conf.d/rasax.nginx:2
+ rasax-rasa-worker-795bb7445f-rkjv9 › rasa-x
+ rasax-rasa-production-5d5485574d-4b9wb › rasa-x
rasax-rasa-production-5d5485574d-4b9wb rasa-x 2020-04-13T15:57:51.661686000Z 2020-04-13 15:57:51 DEBUG    rasa.cli.x  - Failed to connect to server. Retrying. Cannot connect to host rasax-rasa-x-rasa-x:5002 ssl:default [Connection timed out]
3

Hi John,

What platform are you running on?

If you’re running on Windows or Mac try installing multipass first. Then run:

multipass launch --name k3s --mem 4G --disk 50G
multipass shell k3s

And then do the one-line deploy in the multipass shell:

curl -s get-rasa-x.rasa.com | sudo bash

After you installed it, wait for Kubernetes to spin on all the instances and execute the following on your local machine to get the IP address of the running Multipass Ubuntu instance:

multipass info k3s
4

Unfortunately I get the same errors. I am using an Ubuntu machine and even reinstalled k3s. But the Cluster is loading for 45 min now and the pods keep crashing with the same status

5

My previous post was actually from my company’s internal cloud’s kubernetes cluster. I generated the deployment files from values.yml using helm.

I did also tried it as well on my Ubuntu laptop at home using the one-line deploy command. This worked flawlessly. Although, I did get CrashLoopBackOff for rasa-production and rasa-worker. I’m running version 18.04.4 LTS of Ubuntu.

6

Take a look at the logs for the pods. Something like this:

kubectl logs -l app.kubernetes.io/component=rasa-x