Hello, I have been using the quick install script to deploy Rasa X on a server. But the script starts looping at the “Deploying Rasa X …” stage. This is caused by the kube pods looping this way :
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system helm-install-traefik-49587 0/1 Completed 0 4d3h
kube-system local-path-provisioner-6d59f47c7-q6bnh 1/1 Running 8 8d
kube-system metrics-server-7566d596c8-nb7zn 1/1 Running 8 8d
kube-system coredns-8655855d6-88cct 1/1 Running 10 8d
rasa rasa-app-5794444455-zzsb4 1/1 Running 0 94m
rasa rasa-redis-master-0 1/1 Running 0 94m
rasa rasa-postgresql-0 1/1 Running 0 94m
rasa rasa-nginx-7b6464c486-v7kpk 1/1 Running 0 94m
rasa rasa-event-service-6598867d8c-kmd56 0/1 CrashLoopBackOff 29 94m
rasa rasa-rasa-production-57b8b945b9-dctb7 0/1 CrashLoopBackOff 22 94m
rasa rasa-rasa-x-74cb6ff89c-q2vwh 0/1 CrashLoopBackOff 23 94m
rasa rasa-rabbit-0 1/1 Running 17 94m
rasa rasa-rasa-worker-794d868b4b-pspdk 1/1 Running 22 94m
The describe pods have the following describe states :
Name: rasa-app-5794444455-zzsb4
Namespace: rasa
Priority: 0
Node: helpdesk-system-product-name/172.18.3.100
Start Time: Tue, 04 Aug 2020 15:06:57 +0200
Labels: app.kubernetes.io/component=app
app.kubernetes.io/instance=rasa
app.kubernetes.io/name=rasa-x
pod-template-hash=5794444455
Annotations: <none>
Status: Running
IP: 10.42.0.32
IPs:
IP: 10.42.0.32
Controlled By: ReplicaSet/rasa-app-5794444455
Containers:
rasa-x:
Container ID: containerd://cd3477ff24d7a10fc4f84c23d739a6569f6f71e56c2e1ff6b3b558b7b786377e
Image: rasa/rasa-x-demo:0.31.2
Image ID: docker.io/rasa/rasa-x-demo@sha256:d28416cec74f64285600e8c27e715f46d6353dfefb06421ed2c0bbf3cc5b60e8
Port: 5055/TCP
Host Port: 0/TCP
State: Running
Started: Tue, 04 Aug 2020 15:06:59 +0200
Ready: True
Restart Count: 0
Liveness: http-get http://:http/health delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:http/health delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-525zl (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-525zl:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-525zl
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
Name: rasa-redis-master-0
Namespace: rasa
Priority: 0
Node: helpdesk-system-product-name/172.18.3.100
Start Time: Tue, 04 Aug 2020 15:06:57 +0200
Labels: app=redis
chart=redis-10.5.14
controller-revision-hash=rasa-redis-master-588b548b96
release=rasa
role=master
statefulset.kubernetes.io/pod-name=rasa-redis-master-0
Annotations: checksum/configmap: d2119f115a890d516244b8223192b72c390badd3e08a274f7dec3a864b99f88d
checksum/health: 450d124d4e1f71dbd31b162b39c284742b6c866e7e1c16e146e97d24f361a12a
checksum/secret: 2c8f163950adce25d51f1e1cbb096f1340194cd6c650c19599ebe8669c8608e7
Status: Running
IP: 10.42.0.39
IPs:
IP: 10.42.0.39
Controlled By: StatefulSet/rasa-redis-master
Containers:
rasa-redis:
Container ID: containerd://9db3b668fce1cfbddb8de0f74e595b52a7dee41168bbefb11cca78c13cbfdee1
Image: docker.io/bitnami/redis:5.0.8-debian-10-r16
Image ID: docker.io/bitnami/redis@sha256:ad0c65962bdd18bbeae28666538c6a29b621f2a380ea55d484954ffb1618f62e
Port: 6379/TCP
Host Port: 0/TCP
Command:
/bin/bash
-c
if [[ -n $REDIS_PASSWORD_FILE ]]; then
password_aux=`cat ${REDIS_PASSWORD_FILE}`
export REDIS_PASSWORD=$password_aux
fi
if [[ ! -f /opt/bitnami/redis/etc/master.conf ]];then
cp /opt/bitnami/redis/mounted-etc/master.conf /opt/bitnami/redis/etc/master.conf
fi
if [[ ! -f /opt/bitnami/redis/etc/redis.conf ]];then
cp /opt/bitnami/redis/mounted-etc/redis.conf /opt/bitnami/redis/etc/redis.conf
fi
ARGS=("--port" "${REDIS_PORT}")
ARGS+=("--requirepass" "${REDIS_PASSWORD}")
ARGS+=("--masterauth" "${REDIS_PASSWORD}")
ARGS+=("--include" "/opt/bitnami/redis/etc/redis.conf")
ARGS+=("--include" "/opt/bitnami/redis/etc/master.conf")
/run.sh ${ARGS[@]}
State: Running
Started: Tue, 04 Aug 2020 15:07:00 +0200
Ready: True
Restart Count: 0
Liveness: exec [sh -c /health/ping_liveness_local.sh 5] delay=5s timeout=5s period=5s #success=1 #failure=5
Readiness: exec [sh -c /health/ping_readiness_local.sh 5] delay=5s timeout=1s period=5s #success=1 #failure=5
Environment:
REDIS_REPLICATION_MODE: master
REDIS_PASSWORD: <set to the key 'redis-password' in secret 'rasa-redis'> Optional: false
REDIS_PORT: 6379
Mounts:
/data from redis-data (rw)
/health from health (rw)
/opt/bitnami/redis/etc/ from redis-tmp-conf (rw)
/opt/bitnami/redis/mounted-etc from config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-525zl (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
redis-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: redis-data-rasa-redis-master-0
ReadOnly: false
health:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: rasa-redis-health
Optional: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: rasa-redis
Optional: false
redis-tmp-conf:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
default-token-525zl:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-525zl
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
Name: rasa-postgresql-0
Namespace: rasa
Priority: 0
Node: helpdesk-system-product-name/172.18.3.100
Start Time: Tue, 04 Aug 2020 15:06:57 +0200
Labels: app=postgresql
chart=postgresql-8.6.13
controller-revision-hash=rasa-postgresql-845f7db444
heritage=Helm
release=rasa
role=master
statefulset.kubernetes.io/pod-name=rasa-postgresql-0
Annotations: <none>
Status: Running
IP: 10.42.0.36
IPs:
IP: 10.42.0.36
Controlled By: StatefulSet/rasa-postgresql
Containers:
rasa-postgresql:
Container ID: containerd://2c6bc75c8f040b6f3787fccff8c2bd02306954048d2eb4f19980cd55a14c3513
Image: docker.io/bitnami/postgresql:11.7.0-debian-10-r55
Image ID: docker.io/bitnami/postgresql@sha256:ec16eb9ff2e7bf0669cfc52e595f17d9c52efd864c3f943f404d525dafaaaf96
Port: 5432/TCP
Host Port: 0/TCP
State: Running
Started: Tue, 04 Aug 2020 15:06:59 +0200
Ready: True
Restart Count: 0
Requests:
cpu: 250m
memory: 256Mi
Liveness: exec [/bin/sh -c exec pg_isready -U "postgres" -d "rasa" -h 127.0.0.1 -p 5432] delay=30s timeout=5s period=10s #success=1 #failure=6
Readiness: exec [/bin/sh -c -e exec pg_isready -U "postgres" -d "rasa" -h 127.0.0.1 -p 5432
[ -f /opt/bitnami/postgresql/tmp/.initialized ] || [ -f /bitnami/postgresql/.initialized ]
] delay=5s timeout=5s period=10s #success=1 #failure=6
Environment:
BITNAMI_DEBUG: false
POSTGRESQL_PORT_NUMBER: 5432
POSTGRESQL_VOLUME_DIR: /bitnami/postgresql
PGDATA: /bitnami/postgresql/data
POSTGRES_USER: postgres
POSTGRES_PASSWORD: <set to the key 'postgresql-password' in secret 'rasa-postgresql'> Optional: false
POSTGRES_DB: rasa
POSTGRESQL_ENABLE_LDAP: no
Mounts:
/bitnami/postgresql from data (rw)
/dev/shm from dshm (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-525zl (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-rasa-postgresql-0
ReadOnly: false
dshm:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: 1Gi
default-token-525zl:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-525zl
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
Name: rasa-nginx-7b6464c486-v7kpk
Namespace: rasa
Priority: 0
Node: helpdesk-system-product-name/172.18.3.100
Start Time: Tue, 04 Aug 2020 15:06:57 +0200
Labels: app.kubernetes.io/component=nginx
app.kubernetes.io/instance=rasa
app.kubernetes.io/name=rasa-x
pod-template-hash=7b6464c486
Annotations: <none>
Status: Running
IP: 10.42.0.35
IPs:
IP: 10.42.0.35
Controlled By: ReplicaSet/rasa-nginx-7b6464c486
Containers:
rasa-x:
Container ID: containerd://a131e92cdac132969ea03e8c57cec9f55b8ed106fb4465f5fee6fb8827322b55
Image: rasa/nginx:0.31.2
Image ID: docker.io/rasa/nginx@sha256:d817798cea15487d6aaccb35176e6e0262c36ce8a0c651758eb2ee1f60e4f0fd
Port: 8080/TCP
Host Port: 0/TCP
State: Running
Started: Tue, 04 Aug 2020 15:06:59 +0200
Ready: True
Restart Count: 0
Liveness: exec [curl localhost:8080/nginx_status] delay=10s timeout=1s period=10s #success=1 #failure=10
Readiness: exec [curl localhost:8080/nginx_status] delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
RASA_X_HOST: rasa-rasa-x-rasa-x:5002
RASA_PRODUCTION_HOST: rasa-rasa-x-rasa-production:5005
CUSTOM_ACTION_HOST: rasa-rasa-x-app:5055
Mounts:
/opt/bitnami/nginx/conf/bitnami/terms from agreement (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-525zl (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
agreement:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: rasa-agreement
Optional: false
default-token-525zl:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-525zl
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
Name: rasa-rasa-production-57b8b945b9-dctb7
Namespace: rasa
Priority: 0
Node: helpdesk-system-product-name/172.18.3.100
Start Time: Tue, 04 Aug 2020 15:06:57 +0200
Labels: app.kubernetes.io/component=rasa-production
app.kubernetes.io/instance=rasa
app.kubernetes.io/name=rasa-x
pod-template-hash=57b8b945b9
Annotations: checksum/rasa: 18fc95b82b1880a5818ab448a0be737b9313e26062e450a2c4a72bc7eed5c134
Status: Running
IP: 10.42.0.33
IPs:
IP: 10.42.0.33
Controlled By: ReplicaSet/rasa-rasa-production-57b8b945b9
Containers:
rasa-x:
Container ID: containerd://a1a02a6f0c3d8b4d6518bef03add9c96a79752da554af655be2f482b00bddd45
Image: rasa/rasa:1.10.8-full
Image ID: docker.io/rasa/rasa@sha256:9911909a5b8ac48fcf1dd707ac37d0a0047b011991f912d80e800c6a27b07695
Port: 5005/TCP
Host Port: 0/TCP
Args:
x
--no-prompt
--production
--config-endpoint
http://rasa-rasa-x-rasa-x:5002/api/config?token=$(RASA_X_TOKEN)
--port
5005
--jwt-method
HS256
--jwt-secret
$(JWT_SECRET)
--auth-token
$(RASA_TOKEN)
--cors
*
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Tue, 04 Aug 2020 16:17:09 +0200
Finished: Tue, 04 Aug 2020 16:19:28 +0200
Ready: False
Restart Count: 18
Liveness: http-get http://:http/ delay=10s timeout=1s period=10s #success=1 #failure=10
Environment:
DB_PASSWORD: <set to the key 'postgresql-password' in secret 'rasa-postgresql'> Optional: false
DB_DATABASE: tracker
RASA_X_TOKEN: <set to the key 'rasaXToken' in secret 'rasa-rasa'> Optional: false
RASA_TOKEN: <set to the key 'rasaToken' in secret 'rasa-rasa'> Optional: false
RASA_X_USERNAME: me
RASA_X_PASSWORD: <set to the key 'initialPassword' in secret 'rasa-rasa'> Optional: false
RABBITMQ_QUEUE: rasa_production_events
JWT_SECRET: <set to the key 'jwtSecret' in secret 'rasa-rasa'> Optional: false
REDIS_PASSWORD: <set to the key 'redis-password' in secret 'rasa-redis'> Optional: false
RABBITMQ_PASSWORD: <set to the key 'rabbitmq-password' in secret 'rasa-rabbit'> Optional: false
RABBITMQ_QUEUE: rasa_production_events
RASA_ENVIRONMENT: production
RASA_MODEL_SERVER: http://rasa-rasa-x-rasa-x:5002/api/projects/default/models/tags/production
RASA_DUCKLING_HTTP_URL: http://rasa-rasa-x-duckling:8000
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-525zl (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-525zl:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-525zl
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 26m (x139 over 76m) kubelet, helpdesk-system-product-name Liveness probe failed: Get http://10.42.0.33:5005/: dial tcp 10.42.0.33:5005: connect: connection refused
Normal Pulling 6m15s (x19 over 76m) kubelet, helpdesk-system-product-name Pulling image "rasa/rasa:1.10.8-full"
Warning BackOff 79s (x152 over 60m) kubelet, helpdesk-system-product-name Back-off restarting failed container
Name: rasa-rasa-x-74cb6ff89c-q2vwh
Namespace: rasa
Priority: 0
Node: helpdesk-system-product-name/172.18.3.100
Start Time: Tue, 04 Aug 2020 15:07:06 +0200
Labels: app.kubernetes.io/component=rasa-x
app.kubernetes.io/instance=rasa
app.kubernetes.io/name=rasa-x
pod-template-hash=74cb6ff89c
Annotations: checksum/rasa: 18fc95b82b1880a5818ab448a0be737b9313e26062e450a2c4a72bc7eed5c134
checksum/rasax: 19edd294c171755a84e7d4dc3e7a818d59b0d32ace94483a3cad6ccc1937c531
Status: Running
IP: 10.42.0.41
IPs:
IP: 10.42.0.41
Controlled By: ReplicaSet/rasa-rasa-x-74cb6ff89c
Containers:
rasa-x:
Container ID: containerd://38a96dfb90138a6628bcb5d80686bf25fc7adc32096841474119efc40b79f9f8
Image: rasa/rasa-x:0.31.2
Image ID: docker.io/rasa/rasa-x@sha256:44c289f807afe686b6696cbc577ed925ed1e844ab9a4850fa7be0e4b032d4fce
Port: 5002/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 04 Aug 2020 16:19:48 +0200
Finished: Tue, 04 Aug 2020 16:19:49 +0200
Ready: False
Restart Count: 19
Liveness: http-get http://:http/ delay=10s timeout=1s period=10s #success=1 #failure=10
Readiness: http-get http://:http/ delay=10s timeout=1s period=10s #success=1 #failure=10
Environment:
SELF_PORT: 5002
LOCAL_MODE: false
RASA_X_HOST: http://rasa-rasa-x-rasa-x:5002
RASA_MODEL_DIR: /app/models
RUN_EVENT_CONSUMER_AS_SEPARATE_SERVICE: true
RABBITMQ_QUEUE: rasa_production_events
RABBITMQ_PASSWORD: <set to the key 'rabbitmq-password' in secret 'rasa-rabbit'> Optional: false
PASSWORD_SALT: <set to the key 'passwordSalt' in secret 'rasa-rasa'> Optional: false
RASA_X_USER_ANALYTICS: 0
SANIC_RESPONSE_TIMEOUT: 3600
JWT_SECRET: <set to the key 'jwtSecret' in secret 'rasa-rasa'> Optional: false
RASA_TOKEN: <set to the key 'rasaToken' in secret 'rasa-rasa'> Optional: false
RASA_X_TOKEN: <set to the key 'rasaXToken' in secret 'rasa-rasa'> Optional: false
RASA_X_USERNAME: me
RASA_X_PASSWORD: <set to the key 'initialPassword' in secret 'rasa-rasa'> Optional: false
DB_USER: postgres
DB_HOST: rasa-postgresql
DB_PORT: 5432
DB_DATABASE: rasa
DB_PASSWORD: <set to the key 'postgresql-password' in secret 'rasa-postgresql'> Optional: false
QUICK_INSTALL: true
Mounts:
/app/auth from rasa-x-claim (rw,path="auth")
/app/credentials.yml from rasa-configuration (rw,path="credentials.yml")
/app/endpoints.yml from rasa-configuration (rw,path="endpoints.yml")
/app/environments.yml from environments (rw,path="environments.yml")
/app/git from rasa-x-claim (rw,path="git")
/app/logs from rasa-x-claim (rw,path="logs")
/app/models from rasa-x-claim (rw,path="models")
/var/run/secrets/kubernetes.io/serviceaccount from default-token-525zl (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
environments:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: rasa-rasa-x-configuration-files
Optional: false
rasa-configuration:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: rasa-rasa-configuration-files
Optional: false
rasa-x-claim:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: rasa-rasa-x-claim
ReadOnly: false
default-token-525zl:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-525zl
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 66s (x361 over 76m) kubelet, helpdesk-system-product-name Back-off restarting failed container
Name: rasa-rasa-worker-794d868b4b-pspdk
Namespace: rasa
Priority: 0
Node: helpdesk-system-product-name/172.18.3.100
Start Time: Tue, 04 Aug 2020 15:06:57 +0200
Labels: app.kubernetes.io/component=rasa-worker
app.kubernetes.io/instance=rasa
app.kubernetes.io/name=rasa-x
pod-template-hash=794d868b4b
Annotations: checksum/rasa: 18fc95b82b1880a5818ab448a0be737b9313e26062e450a2c4a72bc7eed5c134
Status: Running
IP: 10.42.0.40
IPs:
IP: 10.42.0.40
Controlled By: ReplicaSet/rasa-rasa-worker-794d868b4b
Containers:
rasa-x:
Container ID: containerd://d74de6669f24ddb209679e5487ca2166aa767275d727861fad2b071b277ee6f5
Image: rasa/rasa:1.10.8-full
Image ID: docker.io/rasa/rasa@sha256:9911909a5b8ac48fcf1dd707ac37d0a0047b011991f912d80e800c6a27b07695
Port: 5005/TCP
Host Port: 0/TCP
Args:
x
--no-prompt
--production
--config-endpoint
http://rasa-rasa-x-rasa-x:5002/api/config?token=$(RASA_X_TOKEN)
--port
5005
--jwt-method
HS256
--jwt-secret
$(JWT_SECRET)
--auth-token
$(RASA_TOKEN)
--cors
*
State: Running
Started: Tue, 04 Aug 2020 16:21:48 +0200
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Tue, 04 Aug 2020 16:14:25 +0200
Finished: Tue, 04 Aug 2020 16:16:44 +0200
Ready: True
Restart Count: 18
Liveness: http-get http://:http/ delay=10s timeout=1s period=10s #success=1 #failure=10
Environment:
DB_PASSWORD: <set to the key 'postgresql-password' in secret 'rasa-postgresql'> Optional: false
DB_DATABASE: worker_tracker
RASA_X_TOKEN: <set to the key 'rasaXToken' in secret 'rasa-rasa'> Optional: false
RASA_TOKEN: <set to the key 'rasaToken' in secret 'rasa-rasa'> Optional: false
RASA_X_USERNAME: me
RASA_X_PASSWORD: <set to the key 'initialPassword' in secret 'rasa-rasa'> Optional: false
RABBITMQ_QUEUE: rasa_production_events
JWT_SECRET: <set to the key 'jwtSecret' in secret 'rasa-rasa'> Optional: false
REDIS_PASSWORD: <set to the key 'redis-password' in secret 'rasa-redis'> Optional: false
RABBITMQ_PASSWORD: <set to the key 'rabbitmq-password' in secret 'rasa-rabbit'> Optional: false
RABBITMQ_QUEUE: rasa_production_events
RASA_ENVIRONMENT: worker
RASA_MODEL_SERVER: http://rasa-rasa-x-rasa-x:5002/api/projects/default/models/tags/production
RASA_DUCKLING_HTTP_URL: http://rasa-rasa-x-duckling:8000
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-525zl (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-525zl:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-525zl
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulling 11m (x17 over 76m) kubelet, helpdesk-system-product-name Pulling image "rasa/rasa:1.10.8-full"
Warning BackOff 6m11s (x141 over 60m) kubelet, helpdesk-system-product-name Back-off restarting failed container
Warning Unhealthy 79s (x181 over 76m) kubelet, helpdesk-system-product-name Liveness probe failed: Get http://10.42.0.40:5005/: dial tcp 10.42.0.40:5005: connect: connection refused
Name: rasa-rabbit-0
Namespace: rasa
Priority: 0
Node: helpdesk-system-product-name/172.18.3.100
Start Time: Tue, 04 Aug 2020 15:06:57 +0200
Labels: app=rabbitmq
chart=rabbitmq-6.19.2
controller-revision-hash=rasa-rabbit-d84cc789f
release=rasa
statefulset.kubernetes.io/pod-name=rasa-rabbit-0
Annotations: checksum/secret: 30d1860d7362cc19405b94b5cc3486901fb74653ac6cbf9db4b1fe519cf8e82f
Status: Running
IP: 10.42.0.37
IPs:
IP: 10.42.0.37
Controlled By: StatefulSet/rasa-rabbit
Containers:
rabbitmq:
Container ID: containerd://72095bf8223bcd9f8d3f17604bfc579e7c43952280caeefef815a7a9590ecebd
Image: docker.io/bitnami/rabbitmq:3.8.3-debian-10-r11
Image ID: docker.io/bitnami/rabbitmq@sha256:e7daf1f57f15902f83cc9e1346a0f29c6a2de4d0212efb8363e2d365ef1dc295
Ports: 4369/TCP, 5672/TCP, 25672/TCP, 15672/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
Command:
bash
-ec
mkdir -p /opt/bitnami/rabbitmq/.rabbitmq/
mkdir -p /opt/bitnami/rabbitmq/etc/rabbitmq/
touch /opt/bitnami/rabbitmq/var/lib/rabbitmq/.start
#persist the erlang cookie in both places for server and cli tools
echo $RABBITMQ_ERL_COOKIE > /opt/bitnami/rabbitmq/var/lib/rabbitmq/.erlang.cookie
cp /opt/bitnami/rabbitmq/var/lib/rabbitmq/.erlang.cookie /opt/bitnami/rabbitmq/.rabbitmq/
#change permission so only the user has access to the cookie file
chmod 600 /opt/bitnami/rabbitmq/.rabbitmq/.erlang.cookie /opt/bitnami/rabbitmq/var/lib/rabbitmq/.erlang.cookie
#copy the mounted configuration to both places
cp /opt/bitnami/rabbitmq/conf/* /opt/bitnami/rabbitmq/etc/rabbitmq
# Apply resources limits
ulimit -n "${RABBITMQ_ULIMIT_NOFILES}"
#replace the default password that is generated
sed -i "/CHANGEME/cdefault_pass=${RABBITMQ_PASSWORD//\\/\\\\}" /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq.conf
exec rabbitmq-server
State: Running
Started: Tue, 04 Aug 2020 16:21:12 +0200
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 04 Aug 2020 16:16:42 +0200
Finished: Tue, 04 Aug 2020 16:21:12 +0200
Ready: False
Restart Count: 14
Liveness: exec [sh -c rabbitmq-api-check "http://user:$RABBITMQ_PASSWORD@127.0.0.1:15672/api/healthchecks/node" '{"status":"ok"}'] delay=120s timeout=20s period=30s #success=1 #failure=6
Readiness: exec [sh -c rabbitmq-health-check "http://user:$RABBITMQ_PASSWORD@127.0.0.1:15672/api/healthchecks/node" '{"status":"ok"}'] delay=10s timeout=20s period=30s #success=1 #failure=3
Environment:
BITNAMI_DEBUG: false
MY_POD_IP: (v1:status.podIP)
MY_POD_NAME: rasa-rabbit-0 (v1:metadata.name)
MY_POD_NAMESPACE: rasa (v1:metadata.namespace)
K8S_SERVICE_NAME: rasa-rabbit-headless
K8S_ADDRESS_TYPE: hostname
RABBITMQ_NODENAME: rabbit@$(MY_POD_NAME).$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local
K8S_HOSTNAME_SUFFIX: .$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local
RABBITMQ_LOGS: -
RABBITMQ_ULIMIT_NOFILES: 65536
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS: +S 2:1
RABBITMQ_USE_LONGNAME: true
RABBITMQ_ERL_COOKIE: <set to the key 'rabbitmq-erlang-cookie' in secret 'rasa-rabbit'> Optional: false
RABBITMQ_PASSWORD: <set to the key 'rabbitmq-password' in secret 'rasa-rabbit'> Optional: false
Mounts:
/opt/bitnami/rabbitmq/conf from config-volume (rw)
/opt/bitnami/rabbitmq/var/lib/rabbitmq from data (rw)
/usr/local/sbin/rabbitmq-api-check from healthchecks (rw,path="rabbitmq-api-check")
/usr/local/sbin/rabbitmq-health-check from healthchecks (rw,path="rabbitmq-health-check")
/var/run/secrets/kubernetes.io/serviceaccount from rasa-rabbit-token-f9rrp (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-rasa-rabbit-0
ReadOnly: false
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: rasa-rabbit-config
Optional: false
healthchecks:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: rasa-rabbit-healthchecks
Optional: false
rasa-rabbit-token-f9rrp:
Type: Secret (a volume populated by a Secret)
SecretName: rasa-rabbit-token-f9rrp
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 31m (x4 over 31m) kubelet, helpdesk-system-product-name Back-off restarting failed container
Warning Unhealthy 73s (x126 over 75m) kubelet, helpdesk-system-product-name Readiness probe failed: curl: (22) The requested URL returned error: 401
Name: rasa-event-service-6598867d8c-kmd56
Namespace: rasa
Priority: 0
Node: helpdesk-system-product-name/172.18.3.100
Start Time: Tue, 04 Aug 2020 15:06:57 +0200
Labels: app.kubernetes.io/component=event-service
app.kubernetes.io/instance=rasa
app.kubernetes.io/name=rasa-x
pod-template-hash=6598867d8c
Annotations: <none>
Status: Running
IP: 10.42.0.34
IPs:
IP: 10.42.0.34
Controlled By: ReplicaSet/rasa-event-service-6598867d8c
Containers:
rasa-x:
Container ID: containerd://4b11cc6f0887cec732c69e5604ec9d6518e54b83ba6a2238954d62aade268d63
Image: rasa/rasa-x:0.31.2
Image ID: docker.io/rasa/rasa-x@sha256:44c289f807afe686b6696cbc577ed925ed1e844ab9a4850fa7be0e4b032d4fce
Port: 5673/TCP
Host Port: 0/TCP
Args:
python
-m
rasax.community.services.event_service
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Tue, 04 Aug 2020 16:22:13 +0200
Finished: Tue, 04 Aug 2020 16:23:12 +0200
Ready: False
Restart Count: 25
Liveness: http-get http://:http/health delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:http/health delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
SELF_PORT: 5673
RABBITMQ_QUEUE: rasa_production_events
RABBITMQ_PASSWORD: <set to the key 'rabbitmq-password' in secret 'rasa-rabbit'> Optional: false
RASA_X_USER_ANALYTICS: 0
LOCAL_MODE: false
DB_USER: postgres
DB_HOST: rasa-postgresql
DB_PORT: 5432
DB_DATABASE: rasa
DB_PASSWORD: <set to the key 'postgresql-password' in secret 'rasa-postgresql'> Optional: false
Mounts:
/app/endpoints.yml from rasa-configuration (rw,path="endpoints.yml")
/var/run/secrets/kubernetes.io/serviceaccount from default-token-525zl (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
rasa-configuration:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: rasa-rasa-configuration-files
Optional: false
default-token-525zl:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-525zl
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulling 16m (x21 over 76m) kubelet, helpdesk-system-product-name Pulling image "rasa/rasa-x:0.31.2"
Warning BackOff 6m13s (x225 over 70m) kubelet, helpdesk-system-product-name Back-off restarting failed container
Warning Unhealthy 82s (x148 over 76m) kubelet, helpdesk-system-product-name Readiness probe failed: Get http://10.42.0.34:5673/health: dial tcp 10.42.0.34:5673: connect: connection refused
I feel like the problem comes from this pod : rasa-rasa-x-74cb6ff89c-q2vwh which have the following problem :
$ kubectl logs -f rasa-rasa-x-74cb6ff89c-q2vwh -n rasa
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2339, in _wrap_pool_connect
return fn()
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 304, in unique_connection
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
rec = pool._do_get()
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get
self._dec_overflow()
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
exc_value, with_traceback=exc_tb,
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get
return self._create_connection()
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
self.__connect(first_connect_check=True)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
pool.logger.debug("Error on connect(): %s", e)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
exc_value, with_traceback=exc_tb,
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
connection = pool._invoke_creator(self)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 493, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py", line 127, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL: password authentication failed for user "postgres"
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.7/site-packages/rasax/community/__main__.py", line 19, in <module>
main()
File "/usr/local/lib/python3.7/site-packages/rasax/community/server.py", line 31, in main
sql_migrations.run_migrations(session)
File "/usr/local/lib/python3.7/site-packages/rasax/community/sql_migrations.py", line 25, in run_migrations
_run_schema_migrations(session)
File "/usr/local/lib/python3.7/site-packages/rasax/community/sql_migrations.py", line 41, in _run_schema_migrations
_run_alembic_migration(alembic_config)
File "/usr/local/lib/python3.7/site-packages/rasax/community/sql_migrations.py", line 65, in _run_alembic_migration
command.upgrade(alembic_config, target_revision)
File "/usr/local/lib/python3.7/site-packages/alembic/command.py", line 298, in upgrade
script.run_env()
File "/usr/local/lib/python3.7/site-packages/alembic/script/base.py", line 489, in run_env
util.load_python_file(self.dir, "env.py")
File "/usr/local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 98, in load_python_file
module = load_module_py(module_id, path)
File "/usr/local/lib/python3.7/site-packages/alembic/util/compat.py", line 184, in load_module_py
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/lib/python3.7/site-packages/rasax/community/database/schema_migrations/alembic/env.py", line 96, in <module>
run_migrations_online()
File "/usr/local/lib/python3.7/site-packages/rasax/community/database/schema_migrations/alembic/env.py", line 68, in run_migrations_online
with connectable.connect() as connection:
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2266, in connect
return self._connection_cls(self, **kwargs)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 104, in __init__
else engine.raw_connection()
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2373, in raw_connection
self.pool.unique_connection, _connection
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2343, in _wrap_pool_connect
e, dialect, self
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1585, in _handle_dbapi_exception_noconnection
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2339, in _wrap_pool_connect
return fn()
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 304, in unique_connection
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
rec = pool._do_get()
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get
self._dec_overflow()
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
exc_value, with_traceback=exc_tb,
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get
return self._create_connection()
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
self.__connect(first_connect_check=True)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
pool.logger.debug("Error on connect(): %s", e)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
exc_value, with_traceback=exc_tb,
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
connection = pool._invoke_creator(self)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 493, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py", line 127, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: password authentication failed for user "postgres"
(Background on this error at: http://sqlalche.me/e/13/e3q8)
I’m a bit stuck does someone have any lead?
OS NAME=“Ubuntu” VERSION=“18.04.3 LTS (Bionic Beaver)”
Rasa versions RASA_X_VERSION=0.31.1 RASA_VERSION=1.10.8
