Rasa X quick install script stuck in a loop: Kubernetes Pods in CrashLoopBackOff state

Hello, I have been using the quick install script to deploy Rasa X on a server. But the script starts looping at the “Deploying Rasa X …” stage. This is caused by the kube pods looping this way :

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                     READY   STATUS             RESTARTS   AGE
kube-system   helm-install-traefik-49587               0/1     Completed          0          4d3h
kube-system   local-path-provisioner-6d59f47c7-q6bnh   1/1     Running            8          8d
kube-system   metrics-server-7566d596c8-nb7zn          1/1     Running            8          8d
kube-system   coredns-8655855d6-88cct                  1/1     Running            10         8d
rasa          rasa-app-5794444455-zzsb4                1/1     Running            0          94m
rasa          rasa-redis-master-0                      1/1     Running            0          94m
rasa          rasa-postgresql-0                        1/1     Running            0          94m
rasa          rasa-nginx-7b6464c486-v7kpk              1/1     Running            0          94m
rasa          rasa-event-service-6598867d8c-kmd56      0/1     CrashLoopBackOff   29         94m
rasa          rasa-rasa-production-57b8b945b9-dctb7    0/1     CrashLoopBackOff   22         94m
rasa          rasa-rasa-x-74cb6ff89c-q2vwh             0/1     CrashLoopBackOff   23         94m
rasa          rasa-rabbit-0                            1/1     Running            17         94m
rasa          rasa-rasa-worker-794d868b4b-pspdk        1/1     Running            22         94m

The describe pods have the following describe states :

Name:         rasa-app-5794444455-zzsb4
Namespace:    rasa
Priority:     0
Node:         helpdesk-system-product-name/172.18.3.100
Start Time:   Tue, 04 Aug 2020 15:06:57 +0200
Labels:       app.kubernetes.io/component=app
              app.kubernetes.io/instance=rasa
              app.kubernetes.io/name=rasa-x
              pod-template-hash=5794444455
Annotations:  <none>
Status:       Running
IP:           10.42.0.32
IPs:
  IP:           10.42.0.32
Controlled By:  ReplicaSet/rasa-app-5794444455
Containers:
  rasa-x:
    Container ID:   containerd://cd3477ff24d7a10fc4f84c23d739a6569f6f71e56c2e1ff6b3b558b7b786377e
    Image:          rasa/rasa-x-demo:0.31.2
    Image ID:       docker.io/rasa/rasa-x-demo@sha256:d28416cec74f64285600e8c27e715f46d6353dfefb06421ed2c0bbf3cc5b60e8
    Port:           5055/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 04 Aug 2020 15:06:59 +0200
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:http/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:http/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-525zl (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-525zl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-525zl
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>


Name:         rasa-redis-master-0
Namespace:    rasa
Priority:     0
Node:         helpdesk-system-product-name/172.18.3.100
Start Time:   Tue, 04 Aug 2020 15:06:57 +0200
Labels:       app=redis
              chart=redis-10.5.14
              controller-revision-hash=rasa-redis-master-588b548b96
              release=rasa
              role=master
              statefulset.kubernetes.io/pod-name=rasa-redis-master-0
Annotations:  checksum/configmap: d2119f115a890d516244b8223192b72c390badd3e08a274f7dec3a864b99f88d
              checksum/health: 450d124d4e1f71dbd31b162b39c284742b6c866e7e1c16e146e97d24f361a12a
              checksum/secret: 2c8f163950adce25d51f1e1cbb096f1340194cd6c650c19599ebe8669c8608e7
Status:       Running
IP:           10.42.0.39
IPs:
  IP:           10.42.0.39
Controlled By:  StatefulSet/rasa-redis-master
Containers:
  rasa-redis:
    Container ID:  containerd://9db3b668fce1cfbddb8de0f74e595b52a7dee41168bbefb11cca78c13cbfdee1
    Image:         docker.io/bitnami/redis:5.0.8-debian-10-r16
    Image ID:      docker.io/bitnami/redis@sha256:ad0c65962bdd18bbeae28666538c6a29b621f2a380ea55d484954ffb1618f62e
    Port:          6379/TCP
    Host Port:     0/TCP
    Command:
      /bin/bash
      -c
      if [[ -n $REDIS_PASSWORD_FILE ]]; then
        password_aux=`cat ${REDIS_PASSWORD_FILE}`
        export REDIS_PASSWORD=$password_aux
      fi
      if [[ ! -f /opt/bitnami/redis/etc/master.conf ]];then
        cp /opt/bitnami/redis/mounted-etc/master.conf /opt/bitnami/redis/etc/master.conf
      fi
      if [[ ! -f /opt/bitnami/redis/etc/redis.conf ]];then
        cp /opt/bitnami/redis/mounted-etc/redis.conf /opt/bitnami/redis/etc/redis.conf
      fi
      ARGS=("--port" "${REDIS_PORT}")
      ARGS+=("--requirepass" "${REDIS_PASSWORD}")
      ARGS+=("--masterauth" "${REDIS_PASSWORD}")
      ARGS+=("--include" "/opt/bitnami/redis/etc/redis.conf")
      ARGS+=("--include" "/opt/bitnami/redis/etc/master.conf")
      /run.sh ${ARGS[@]}
      
    State:          Running
      Started:      Tue, 04 Aug 2020 15:07:00 +0200
    Ready:          True
    Restart Count:  0
    Liveness:       exec [sh -c /health/ping_liveness_local.sh 5] delay=5s timeout=5s period=5s #success=1 #failure=5
    Readiness:      exec [sh -c /health/ping_readiness_local.sh 5] delay=5s timeout=1s period=5s #success=1 #failure=5
    Environment:
      REDIS_REPLICATION_MODE:  master
      REDIS_PASSWORD:          <set to the key 'redis-password' in secret 'rasa-redis'>  Optional: false
      REDIS_PORT:              6379
    Mounts:
      /data from redis-data (rw)
      /health from health (rw)
      /opt/bitnami/redis/etc/ from redis-tmp-conf (rw)
      /opt/bitnami/redis/mounted-etc from config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-525zl (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  redis-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  redis-data-rasa-redis-master-0
    ReadOnly:   false
  health:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rasa-redis-health
    Optional:  false
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rasa-redis
    Optional:  false
  redis-tmp-conf:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  default-token-525zl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-525zl
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>


Name:         rasa-postgresql-0
Namespace:    rasa
Priority:     0
Node:         helpdesk-system-product-name/172.18.3.100
Start Time:   Tue, 04 Aug 2020 15:06:57 +0200
Labels:       app=postgresql
              chart=postgresql-8.6.13
              controller-revision-hash=rasa-postgresql-845f7db444
              heritage=Helm
              release=rasa
              role=master
              statefulset.kubernetes.io/pod-name=rasa-postgresql-0
Annotations:  <none>
Status:       Running
IP:           10.42.0.36
IPs:
  IP:           10.42.0.36
Controlled By:  StatefulSet/rasa-postgresql
Containers:
  rasa-postgresql:
    Container ID:   containerd://2c6bc75c8f040b6f3787fccff8c2bd02306954048d2eb4f19980cd55a14c3513
    Image:          docker.io/bitnami/postgresql:11.7.0-debian-10-r55
    Image ID:       docker.io/bitnami/postgresql@sha256:ec16eb9ff2e7bf0669cfc52e595f17d9c52efd864c3f943f404d525dafaaaf96
    Port:           5432/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 04 Aug 2020 15:06:59 +0200
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      250m
      memory:   256Mi
    Liveness:   exec [/bin/sh -c exec pg_isready -U "postgres" -d "rasa" -h 127.0.0.1 -p 5432] delay=30s timeout=5s period=10s #success=1 #failure=6
    Readiness:  exec [/bin/sh -c -e exec pg_isready -U "postgres" -d "rasa" -h 127.0.0.1 -p 5432
[ -f /opt/bitnami/postgresql/tmp/.initialized ] || [ -f /bitnami/postgresql/.initialized ]
] delay=5s timeout=5s period=10s #success=1 #failure=6
    Environment:
      BITNAMI_DEBUG:           false
      POSTGRESQL_PORT_NUMBER:  5432
      POSTGRESQL_VOLUME_DIR:   /bitnami/postgresql
      PGDATA:                  /bitnami/postgresql/data
      POSTGRES_USER:           postgres
      POSTGRES_PASSWORD:       <set to the key 'postgresql-password' in secret 'rasa-postgresql'>  Optional: false
      POSTGRES_DB:             rasa
      POSTGRESQL_ENABLE_LDAP:  no
    Mounts:
      /bitnami/postgresql from data (rw)
      /dev/shm from dshm (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-525zl (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-rasa-postgresql-0
    ReadOnly:   false
  dshm:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  1Gi
  default-token-525zl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-525zl
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>


Name:         rasa-nginx-7b6464c486-v7kpk
Namespace:    rasa
Priority:     0
Node:         helpdesk-system-product-name/172.18.3.100
Start Time:   Tue, 04 Aug 2020 15:06:57 +0200
Labels:       app.kubernetes.io/component=nginx
              app.kubernetes.io/instance=rasa
              app.kubernetes.io/name=rasa-x
              pod-template-hash=7b6464c486
Annotations:  <none>
Status:       Running
IP:           10.42.0.35
IPs:
  IP:           10.42.0.35
Controlled By:  ReplicaSet/rasa-nginx-7b6464c486
Containers:
  rasa-x:
    Container ID:   containerd://a131e92cdac132969ea03e8c57cec9f55b8ed106fb4465f5fee6fb8827322b55
    Image:          rasa/nginx:0.31.2
    Image ID:       docker.io/rasa/nginx@sha256:d817798cea15487d6aaccb35176e6e0262c36ce8a0c651758eb2ee1f60e4f0fd
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 04 Aug 2020 15:06:59 +0200
    Ready:          True
    Restart Count:  0
    Liveness:       exec [curl localhost:8080/nginx_status] delay=10s timeout=1s period=10s #success=1 #failure=10
    Readiness:      exec [curl localhost:8080/nginx_status] delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      RASA_X_HOST:           rasa-rasa-x-rasa-x:5002
      RASA_PRODUCTION_HOST:  rasa-rasa-x-rasa-production:5005
      CUSTOM_ACTION_HOST:    rasa-rasa-x-app:5055
    Mounts:
      /opt/bitnami/nginx/conf/bitnami/terms from agreement (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-525zl (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  agreement:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rasa-agreement
    Optional:  false
  default-token-525zl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-525zl
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>


Name:         rasa-rasa-production-57b8b945b9-dctb7
Namespace:    rasa
Priority:     0
Node:         helpdesk-system-product-name/172.18.3.100
Start Time:   Tue, 04 Aug 2020 15:06:57 +0200
Labels:       app.kubernetes.io/component=rasa-production
              app.kubernetes.io/instance=rasa
              app.kubernetes.io/name=rasa-x
              pod-template-hash=57b8b945b9
Annotations:  checksum/rasa: 18fc95b82b1880a5818ab448a0be737b9313e26062e450a2c4a72bc7eed5c134
Status:       Running
IP:           10.42.0.33
IPs:
  IP:           10.42.0.33
Controlled By:  ReplicaSet/rasa-rasa-production-57b8b945b9
Containers:
  rasa-x:
    Container ID:  containerd://a1a02a6f0c3d8b4d6518bef03add9c96a79752da554af655be2f482b00bddd45
    Image:         rasa/rasa:1.10.8-full
    Image ID:      docker.io/rasa/rasa@sha256:9911909a5b8ac48fcf1dd707ac37d0a0047b011991f912d80e800c6a27b07695
    Port:          5005/TCP
    Host Port:     0/TCP
    Args:
      x
      --no-prompt
      --production
      --config-endpoint
      http://rasa-rasa-x-rasa-x:5002/api/config?token=$(RASA_X_TOKEN)
      --port
      5005
      --jwt-method
      HS256
      --jwt-secret
      $(JWT_SECRET)
      --auth-token
      $(RASA_TOKEN)
      --cors
      *
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Tue, 04 Aug 2020 16:17:09 +0200
      Finished:     Tue, 04 Aug 2020 16:19:28 +0200
    Ready:          False
    Restart Count:  18
    Liveness:       http-get http://:http/ delay=10s timeout=1s period=10s #success=1 #failure=10
    Environment:
      DB_PASSWORD:             <set to the key 'postgresql-password' in secret 'rasa-postgresql'>  Optional: false
      DB_DATABASE:             tracker
      RASA_X_TOKEN:            <set to the key 'rasaXToken' in secret 'rasa-rasa'>  Optional: false
      RASA_TOKEN:              <set to the key 'rasaToken' in secret 'rasa-rasa'>   Optional: false
      RASA_X_USERNAME:         me
      RASA_X_PASSWORD:         <set to the key 'initialPassword' in secret 'rasa-rasa'>  Optional: false
      RABBITMQ_QUEUE:          rasa_production_events
      JWT_SECRET:              <set to the key 'jwtSecret' in secret 'rasa-rasa'>            Optional: false
      REDIS_PASSWORD:          <set to the key 'redis-password' in secret 'rasa-redis'>      Optional: false
      RABBITMQ_PASSWORD:       <set to the key 'rabbitmq-password' in secret 'rasa-rabbit'>  Optional: false
      RABBITMQ_QUEUE:          rasa_production_events
      RASA_ENVIRONMENT:        production
      RASA_MODEL_SERVER:       http://rasa-rasa-x-rasa-x:5002/api/projects/default/models/tags/production
      RASA_DUCKLING_HTTP_URL:  http://rasa-rasa-x-duckling:8000
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-525zl (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-525zl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-525zl
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                   From                                   Message
  ----     ------     ----                  ----                                   -------
  Warning  Unhealthy  26m (x139 over 76m)   kubelet, helpdesk-system-product-name  Liveness probe failed: Get http://10.42.0.33:5005/: dial tcp 10.42.0.33:5005: connect: connection refused
  Normal   Pulling    6m15s (x19 over 76m)  kubelet, helpdesk-system-product-name  Pulling image "rasa/rasa:1.10.8-full"
  Warning  BackOff    79s (x152 over 60m)   kubelet, helpdesk-system-product-name  Back-off restarting failed container




Name:         rasa-rasa-x-74cb6ff89c-q2vwh
Namespace:    rasa
Priority:     0
Node:         helpdesk-system-product-name/172.18.3.100
Start Time:   Tue, 04 Aug 2020 15:07:06 +0200
Labels:       app.kubernetes.io/component=rasa-x
              app.kubernetes.io/instance=rasa
              app.kubernetes.io/name=rasa-x
              pod-template-hash=74cb6ff89c
Annotations:  checksum/rasa: 18fc95b82b1880a5818ab448a0be737b9313e26062e450a2c4a72bc7eed5c134
              checksum/rasax: 19edd294c171755a84e7d4dc3e7a818d59b0d32ace94483a3cad6ccc1937c531
Status:       Running
IP:           10.42.0.41
IPs:
  IP:           10.42.0.41
Controlled By:  ReplicaSet/rasa-rasa-x-74cb6ff89c
Containers:
  rasa-x:
    Container ID:   containerd://38a96dfb90138a6628bcb5d80686bf25fc7adc32096841474119efc40b79f9f8
    Image:          rasa/rasa-x:0.31.2
    Image ID:       docker.io/rasa/rasa-x@sha256:44c289f807afe686b6696cbc577ed925ed1e844ab9a4850fa7be0e4b032d4fce
    Port:           5002/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 04 Aug 2020 16:19:48 +0200
      Finished:     Tue, 04 Aug 2020 16:19:49 +0200
    Ready:          False
    Restart Count:  19
    Liveness:       http-get http://:http/ delay=10s timeout=1s period=10s #success=1 #failure=10
    Readiness:      http-get http://:http/ delay=10s timeout=1s period=10s #success=1 #failure=10
    Environment:
      SELF_PORT:                               5002
      LOCAL_MODE:                              false
      RASA_X_HOST:                             http://rasa-rasa-x-rasa-x:5002
      RASA_MODEL_DIR:                          /app/models
      RUN_EVENT_CONSUMER_AS_SEPARATE_SERVICE:  true
      RABBITMQ_QUEUE:                          rasa_production_events
      RABBITMQ_PASSWORD:                       <set to the key 'rabbitmq-password' in secret 'rasa-rabbit'>  Optional: false
      PASSWORD_SALT:                           <set to the key 'passwordSalt' in secret 'rasa-rasa'>         Optional: false
      RASA_X_USER_ANALYTICS:                   0
      SANIC_RESPONSE_TIMEOUT:                  3600
      JWT_SECRET:                              <set to the key 'jwtSecret' in secret 'rasa-rasa'>   Optional: false
      RASA_TOKEN:                              <set to the key 'rasaToken' in secret 'rasa-rasa'>   Optional: false
      RASA_X_TOKEN:                            <set to the key 'rasaXToken' in secret 'rasa-rasa'>  Optional: false
      RASA_X_USERNAME:                         me
      RASA_X_PASSWORD:                         <set to the key 'initialPassword' in secret 'rasa-rasa'>  Optional: false
      DB_USER:                                 postgres
      DB_HOST:                                 rasa-postgresql
      DB_PORT:                                 5432
      DB_DATABASE:                             rasa
      DB_PASSWORD:                             <set to the key 'postgresql-password' in secret 'rasa-postgresql'>  Optional: false
      QUICK_INSTALL:                           true
    Mounts:
      /app/auth from rasa-x-claim (rw,path="auth")
      /app/credentials.yml from rasa-configuration (rw,path="credentials.yml")
      /app/endpoints.yml from rasa-configuration (rw,path="endpoints.yml")
      /app/environments.yml from environments (rw,path="environments.yml")
      /app/git from rasa-x-claim (rw,path="git")
      /app/logs from rasa-x-claim (rw,path="logs")
      /app/models from rasa-x-claim (rw,path="models")
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-525zl (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  environments:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rasa-rasa-x-configuration-files
    Optional:  false
  rasa-configuration:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rasa-rasa-configuration-files
    Optional:  false
  rasa-x-claim:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  rasa-rasa-x-claim
    ReadOnly:   false
  default-token-525zl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-525zl
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason   Age                  From                                   Message
  ----     ------   ----                 ----                                   -------
  Warning  BackOff  66s (x361 over 76m)  kubelet, helpdesk-system-product-name  Back-off restarting failed container


Name:         rasa-rasa-worker-794d868b4b-pspdk
Namespace:    rasa
Priority:     0
Node:         helpdesk-system-product-name/172.18.3.100
Start Time:   Tue, 04 Aug 2020 15:06:57 +0200
Labels:       app.kubernetes.io/component=rasa-worker
              app.kubernetes.io/instance=rasa
              app.kubernetes.io/name=rasa-x
              pod-template-hash=794d868b4b
Annotations:  checksum/rasa: 18fc95b82b1880a5818ab448a0be737b9313e26062e450a2c4a72bc7eed5c134
Status:       Running
IP:           10.42.0.40
IPs:
  IP:           10.42.0.40
Controlled By:  ReplicaSet/rasa-rasa-worker-794d868b4b
Containers:
  rasa-x:
    Container ID:  containerd://d74de6669f24ddb209679e5487ca2166aa767275d727861fad2b071b277ee6f5
    Image:         rasa/rasa:1.10.8-full
    Image ID:      docker.io/rasa/rasa@sha256:9911909a5b8ac48fcf1dd707ac37d0a0047b011991f912d80e800c6a27b07695
    Port:          5005/TCP
    Host Port:     0/TCP
    Args:
      x
      --no-prompt
      --production
      --config-endpoint
      http://rasa-rasa-x-rasa-x:5002/api/config?token=$(RASA_X_TOKEN)
      --port
      5005
      --jwt-method
      HS256
      --jwt-secret
      $(JWT_SECRET)
      --auth-token
      $(RASA_TOKEN)
      --cors
      *
    State:          Running
      Started:      Tue, 04 Aug 2020 16:21:48 +0200
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Tue, 04 Aug 2020 16:14:25 +0200
      Finished:     Tue, 04 Aug 2020 16:16:44 +0200
    Ready:          True
    Restart Count:  18
    Liveness:       http-get http://:http/ delay=10s timeout=1s period=10s #success=1 #failure=10
    Environment:
      DB_PASSWORD:             <set to the key 'postgresql-password' in secret 'rasa-postgresql'>  Optional: false
      DB_DATABASE:             worker_tracker
      RASA_X_TOKEN:            <set to the key 'rasaXToken' in secret 'rasa-rasa'>  Optional: false
      RASA_TOKEN:              <set to the key 'rasaToken' in secret 'rasa-rasa'>   Optional: false
      RASA_X_USERNAME:         me
      RASA_X_PASSWORD:         <set to the key 'initialPassword' in secret 'rasa-rasa'>  Optional: false
      RABBITMQ_QUEUE:          rasa_production_events
      JWT_SECRET:              <set to the key 'jwtSecret' in secret 'rasa-rasa'>            Optional: false
      REDIS_PASSWORD:          <set to the key 'redis-password' in secret 'rasa-redis'>      Optional: false
      RABBITMQ_PASSWORD:       <set to the key 'rabbitmq-password' in secret 'rasa-rabbit'>  Optional: false
      RABBITMQ_QUEUE:          rasa_production_events
      RASA_ENVIRONMENT:        worker
      RASA_MODEL_SERVER:       http://rasa-rasa-x-rasa-x:5002/api/projects/default/models/tags/production
      RASA_DUCKLING_HTTP_URL:  http://rasa-rasa-x-duckling:8000
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-525zl (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-525zl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-525zl
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                    From                                   Message
  ----     ------     ----                   ----                                   -------
  Normal   Pulling    11m (x17 over 76m)     kubelet, helpdesk-system-product-name  Pulling image "rasa/rasa:1.10.8-full"
  Warning  BackOff    6m11s (x141 over 60m)  kubelet, helpdesk-system-product-name  Back-off restarting failed container
  Warning  Unhealthy  79s (x181 over 76m)    kubelet, helpdesk-system-product-name  Liveness probe failed: Get http://10.42.0.40:5005/: dial tcp 10.42.0.40:5005: connect: connection refused


Name:         rasa-rabbit-0
Namespace:    rasa
Priority:     0
Node:         helpdesk-system-product-name/172.18.3.100
Start Time:   Tue, 04 Aug 2020 15:06:57 +0200
Labels:       app=rabbitmq
              chart=rabbitmq-6.19.2
              controller-revision-hash=rasa-rabbit-d84cc789f
              release=rasa
              statefulset.kubernetes.io/pod-name=rasa-rabbit-0
Annotations:  checksum/secret: 30d1860d7362cc19405b94b5cc3486901fb74653ac6cbf9db4b1fe519cf8e82f
Status:       Running
IP:           10.42.0.37
IPs:
  IP:           10.42.0.37
Controlled By:  StatefulSet/rasa-rabbit
Containers:
  rabbitmq:
    Container ID:  containerd://72095bf8223bcd9f8d3f17604bfc579e7c43952280caeefef815a7a9590ecebd
    Image:         docker.io/bitnami/rabbitmq:3.8.3-debian-10-r11
    Image ID:      docker.io/bitnami/rabbitmq@sha256:e7daf1f57f15902f83cc9e1346a0f29c6a2de4d0212efb8363e2d365ef1dc295
    Ports:         4369/TCP, 5672/TCP, 25672/TCP, 15672/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Command:
      bash
      -ec
      mkdir -p /opt/bitnami/rabbitmq/.rabbitmq/
      mkdir -p /opt/bitnami/rabbitmq/etc/rabbitmq/
      touch /opt/bitnami/rabbitmq/var/lib/rabbitmq/.start
      #persist the erlang cookie in both places for server and cli tools
      echo $RABBITMQ_ERL_COOKIE > /opt/bitnami/rabbitmq/var/lib/rabbitmq/.erlang.cookie
      cp /opt/bitnami/rabbitmq/var/lib/rabbitmq/.erlang.cookie /opt/bitnami/rabbitmq/.rabbitmq/
      #change permission so only the user has access to the cookie file
      chmod 600 /opt/bitnami/rabbitmq/.rabbitmq/.erlang.cookie /opt/bitnami/rabbitmq/var/lib/rabbitmq/.erlang.cookie
      #copy the mounted configuration to both places
      cp  /opt/bitnami/rabbitmq/conf/* /opt/bitnami/rabbitmq/etc/rabbitmq
      # Apply resources limits
      ulimit -n "${RABBITMQ_ULIMIT_NOFILES}"
      #replace the default password that is generated
      sed -i "/CHANGEME/cdefault_pass=${RABBITMQ_PASSWORD//\\/\\\\}" /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq.conf
      exec rabbitmq-server
      
    State:          Running
      Started:      Tue, 04 Aug 2020 16:21:12 +0200
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 04 Aug 2020 16:16:42 +0200
      Finished:     Tue, 04 Aug 2020 16:21:12 +0200
    Ready:          False
    Restart Count:  14
    Liveness:       exec [sh -c rabbitmq-api-check "http://user:$RABBITMQ_PASSWORD@127.0.0.1:15672/api/healthchecks/node" '{"status":"ok"}'] delay=120s timeout=20s period=30s #success=1 #failure=6
    Readiness:      exec [sh -c rabbitmq-health-check "http://user:$RABBITMQ_PASSWORD@127.0.0.1:15672/api/healthchecks/node" '{"status":"ok"}'] delay=10s timeout=20s period=30s #success=1 #failure=3
    Environment:
      BITNAMI_DEBUG:                        false
      MY_POD_IP:                             (v1:status.podIP)
      MY_POD_NAME:                          rasa-rabbit-0 (v1:metadata.name)
      MY_POD_NAMESPACE:                     rasa (v1:metadata.namespace)
      K8S_SERVICE_NAME:                     rasa-rabbit-headless
      K8S_ADDRESS_TYPE:                     hostname
      RABBITMQ_NODENAME:                    rabbit@$(MY_POD_NAME).$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local
      K8S_HOSTNAME_SUFFIX:                  .$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local
      RABBITMQ_LOGS:                        -
      RABBITMQ_ULIMIT_NOFILES:              65536
      RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS:  +S 2:1
      RABBITMQ_USE_LONGNAME:                true
      RABBITMQ_ERL_COOKIE:                  <set to the key 'rabbitmq-erlang-cookie' in secret 'rasa-rabbit'>  Optional: false
      RABBITMQ_PASSWORD:                    <set to the key 'rabbitmq-password' in secret 'rasa-rabbit'>       Optional: false
    Mounts:
      /opt/bitnami/rabbitmq/conf from config-volume (rw)
      /opt/bitnami/rabbitmq/var/lib/rabbitmq from data (rw)
      /usr/local/sbin/rabbitmq-api-check from healthchecks (rw,path="rabbitmq-api-check")
      /usr/local/sbin/rabbitmq-health-check from healthchecks (rw,path="rabbitmq-health-check")
      /var/run/secrets/kubernetes.io/serviceaccount from rasa-rabbit-token-f9rrp (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-rasa-rabbit-0
    ReadOnly:   false
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rasa-rabbit-config
    Optional:  false
  healthchecks:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rasa-rabbit-healthchecks
    Optional:  false
  rasa-rabbit-token-f9rrp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rasa-rabbit-token-f9rrp
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                  From                                   Message
  ----     ------     ----                 ----                                   -------
  Warning  BackOff    31m (x4 over 31m)    kubelet, helpdesk-system-product-name  Back-off restarting failed container
  Warning  Unhealthy  73s (x126 over 75m)  kubelet, helpdesk-system-product-name  Readiness probe failed: curl: (22) The requested URL returned error: 401


Name:         rasa-event-service-6598867d8c-kmd56
Namespace:    rasa
Priority:     0
Node:         helpdesk-system-product-name/172.18.3.100
Start Time:   Tue, 04 Aug 2020 15:06:57 +0200
Labels:       app.kubernetes.io/component=event-service
              app.kubernetes.io/instance=rasa
              app.kubernetes.io/name=rasa-x
              pod-template-hash=6598867d8c
Annotations:  <none>
Status:       Running
IP:           10.42.0.34
IPs:
  IP:           10.42.0.34
Controlled By:  ReplicaSet/rasa-event-service-6598867d8c
Containers:
  rasa-x:
    Container ID:  containerd://4b11cc6f0887cec732c69e5604ec9d6518e54b83ba6a2238954d62aade268d63
    Image:         rasa/rasa-x:0.31.2
    Image ID:      docker.io/rasa/rasa-x@sha256:44c289f807afe686b6696cbc577ed925ed1e844ab9a4850fa7be0e4b032d4fce
    Port:          5673/TCP
    Host Port:     0/TCP
    Args:
      python
      -m
      rasax.community.services.event_service
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Tue, 04 Aug 2020 16:22:13 +0200
      Finished:     Tue, 04 Aug 2020 16:23:12 +0200
    Ready:          False
    Restart Count:  25
    Liveness:       http-get http://:http/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:http/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      SELF_PORT:              5673
      RABBITMQ_QUEUE:         rasa_production_events
      RABBITMQ_PASSWORD:      <set to the key 'rabbitmq-password' in secret 'rasa-rabbit'>  Optional: false
      RASA_X_USER_ANALYTICS:  0
      LOCAL_MODE:             false
      DB_USER:                postgres
      DB_HOST:                rasa-postgresql
      DB_PORT:                5432
      DB_DATABASE:            rasa
      DB_PASSWORD:            <set to the key 'postgresql-password' in secret 'rasa-postgresql'>  Optional: false
    Mounts:
      /app/endpoints.yml from rasa-configuration (rw,path="endpoints.yml")
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-525zl (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  rasa-configuration:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rasa-rasa-configuration-files
    Optional:  false
  default-token-525zl:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-525zl
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                    From                                   Message
  ----     ------     ----                   ----                                   -------
  Normal   Pulling    16m (x21 over 76m)     kubelet, helpdesk-system-product-name  Pulling image "rasa/rasa-x:0.31.2"
  Warning  BackOff    6m13s (x225 over 70m)  kubelet, helpdesk-system-product-name  Back-off restarting failed container
  Warning  Unhealthy  82s (x148 over 76m)    kubelet, helpdesk-system-product-name  Readiness probe failed: Get http://10.42.0.34:5673/health: dial tcp 10.42.0.34:5673: connect: connection refused

I feel like the problem comes from this pod : rasa-rasa-x-74cb6ff89c-q2vwh which have the following problem :

$ kubectl logs -f rasa-rasa-x-74cb6ff89c-q2vwh -n rasa
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2339, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 304, in unique_connection
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get
    self._dec_overflow()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
    exc_value, with_traceback=exc_tb,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
    self.__connect(first_connect_check=True)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
    pool.logger.debug("Error on connect(): %s", e)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
    exc_value, with_traceback=exc_tb,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
    connection = pool._invoke_creator(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 493, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL:  password authentication failed for user "postgres"


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/site-packages/rasax/community/__main__.py", line 19, in <module>
    main()
  File "/usr/local/lib/python3.7/site-packages/rasax/community/server.py", line 31, in main
    sql_migrations.run_migrations(session)
  File "/usr/local/lib/python3.7/site-packages/rasax/community/sql_migrations.py", line 25, in run_migrations
    _run_schema_migrations(session)
  File "/usr/local/lib/python3.7/site-packages/rasax/community/sql_migrations.py", line 41, in _run_schema_migrations
    _run_alembic_migration(alembic_config)
  File "/usr/local/lib/python3.7/site-packages/rasax/community/sql_migrations.py", line 65, in _run_alembic_migration
    command.upgrade(alembic_config, target_revision)
  File "/usr/local/lib/python3.7/site-packages/alembic/command.py", line 298, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.7/site-packages/alembic/script/base.py", line 489, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 98, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.7/site-packages/alembic/util/compat.py", line 184, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.7/site-packages/rasax/community/database/schema_migrations/alembic/env.py", line 96, in <module>
    run_migrations_online()
  File "/usr/local/lib/python3.7/site-packages/rasax/community/database/schema_migrations/alembic/env.py", line 68, in run_migrations_online
    with connectable.connect() as connection:
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2266, in connect
    return self._connection_cls(self, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 104, in __init__
    else engine.raw_connection()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2373, in raw_connection
    self.pool.unique_connection, _connection
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2343, in _wrap_pool_connect
    e, dialect, self
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1585, in _handle_dbapi_exception_noconnection
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2339, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 304, in unique_connection
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get
    self._dec_overflow()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
    exc_value, with_traceback=exc_tb,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
    self.__connect(first_connect_check=True)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
    pool.logger.debug("Error on connect(): %s", e)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
    exc_value, with_traceback=exc_tb,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
    connection = pool._invoke_creator(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 493, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  password authentication failed for user "postgres"

(Background on this error at: http://sqlalche.me/e/13/e3q8)

I’m a bit stuck does someone have any lead?

OS NAME=“Ubuntu” VERSION=“18.04.3 LTS (Bionic Beaver)”

Rasa versions RASA_X_VERSION=0.31.1 RASA_VERSION=1.10.8

Take a look at the log for the event service. Both Rasa X and rasa-production depend on this service.

kubectl logs rasa-event-service-6598867d8c-kmd56

Hey @stephens.

The rasa Event service is looping with this message :

$ kubectl logs rasa-event-service-6598867d8c-kmd56
INFO:__main__:Starting event service (standalone: True).
INFO:__main__:Waiting until database migrations have been executed...
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:rasax.community.database.utils:Unable to get database revision heads.
WARNING:rasax.community.database.utils:DB revision(s) do not match migration scripts revision(s):
WARNING:rasax.community.database.utils:DB revision: None
WARNING:rasax.community.database.utils:Migration scripts revision: ['473b15141809']
WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.

It looks like a data migration is failing. Unless you have a lot of data to migrate I would delete the namespace and re-install.

1 Like

Going through this as well. Tried uninstalling and reinstalling at least 5 times now… no resolution