Rasa Killed by Kubernetes

Hey guys, im trying to put my rasa chatbot on aks. I build a image with miniconda and in docker-compose works very well, but when i put in kubernetes my rasa application doesnt stabilizes, the kubernetes says “Killed” in logs container. Bellow ar the docker-compose and his logs, the same of kubernetes deployment.

rasa-core:0.10.4 rasa-nlu:0.13.3

docker-compose.yaml

version: '3.1'
services:
rasa:
image: myrasaimageonazure
command: bash app.sh
ports:
- 5005:5005

logs of docker-compose:

rasa_1 | 2018-09-17 15:30:44.711591: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
rasa_1 | Using TensorFlow backend.
rasa_1 | 2018-09-17 15:30:44 WARNING py.warnings - /opt/conda/envs/rasa-env/lib/python3.5/site-packages/pykwalify/core.py:99: UnsafeLoaderWarning:
rasa_1 | The default 'Loader' for 'load(stream)' without further arguments can be unsafe.
rasa_1 | Use 'load(stream, Loader=ruamel.yaml.Loader)' explicitly if that is OK.
rasa_1 | Alternatively include the following in your code:
rasa_1 |
rasa_1 | import warnings
rasa_1 | warnings.simplefilter('ignore', ruamel.yaml.error.UnsafeLoaderWarning)
rasa_1 |
rasa_1 | In most other cases you should consider using 'safe_load(stream)'
rasa_1 | data = yaml.load(stream)
rasa_1 |
rasa_1 | 2018-09-17 15:33:02 WARNING py.warnings - /opt/conda/envs/rasa-env/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.
rasa_1 | if diff:
rasa_1 |
rasa_1 | [192.168.99.1](https://l.facebook.com/l.php?u=http%3A%2F%2F192.168.99.1%2F&h=AT2id_igU4MtwW35DskH8nm8KeJcqFVwbnk2TNHv-_nh9x_kad_Cv0Rp4G1Rslk6b-JmnExZH9Caco_RlePUHW-PeJ8SE9-tyAoIUU1Vph15YTiZFZZXICdlWg) - - [2018-09-17 15:33:03] "POST /conversations/asdfasdfasdf/respond HTTP/1.1" 200 442 0.205809
rasa_1 | 2018-09-17 15:33:09 WARNING py.warnings - /opt/conda/envs/rasa-env/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.
rasa_1 | if diff:
rasa_1 |
rasa_1 | [192.168.99.1](https://l.facebook.com/l.php?u=http%3A%2F%2F192.168.99.1%2F&h=AT2id_igU4MtwW35DskH8nm8KeJcqFVwbnk2TNHv-_nh9x_kad_Cv0Rp4G1Rslk6b-JmnExZH9Caco_RlePUHW-PeJ8SE9-tyAoIUU1Vph15YTiZFZZXICdlWg) - - [2018-09-17 15:33:09] "POST /conversations/asdfasdfasdf/respond HTTP/1.1" 200 378 0.020443
rasa_1 | 2018-09-17 15:33:20 WARNING py.warnings - /opt/conda/envs/rasa-env/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.
rasa_1 | if diff:
rasa_1 |
rasa_1 | [192.168.99.1](http://192.168.99.1/) - - [2018-09-17 15:33:20] "POST /conversations/asdfasdfasdf/respond HTTP/1.1" 200 178 0.021819
rasa_1 | 2018-09-17 15:37:12 WARNING py.warnings - /opt/conda/envs/rasa-env/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.
rasa_1 | if diff:
rasa_1 |
rasa_1 | [192.168.99.1](https://l.facebook.com/l.php?u=http%3A%2F%2F192.168.99.1%2F&h=AT2id_igU4MtwW35DskH8nm8KeJcqFVwbnk2TNHv-_nh9x_kad_Cv0Rp4G1Rslk6b-JmnExZH9Caco_RlePUHW-PeJ8SE9-tyAoIUU1Vph15YTiZFZZXICdlWg) - - [2018-09-17 15:37:12] "POST /conversations/asdfasdfasdf/respond HTTP/1.1" 200 301 0.023151
rasa_1 | 2018-09-17 15:37:29 WARNING py.warnings - /opt/conda/envs/rasa-env/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.
rasa_1 | if diff:
rasa_1 |
rasa_1 | [192.168.99.1](http://192.168.99.1/) - - [2018-09-17 15:37:29] "POST /conversations/asdfasdfasdf/respond HTTP/1.1" 200 301 0.022728
rasa_1 | 2018-09-17 15:37:42 WARNING py.warnings - /opt/conda/envs/rasa-env/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.
rasa_1 | if diff:
rasa_1 |
rasa_1 | [192.168.99.1](https://l.facebook.com/l.php?u=http%3A%2F%2F192.168.99.1%2F&h=AT2id_igU4MtwW35DskH8nm8KeJcqFVwbnk2TNHv-_nh9x_kad_Cv0Rp4G1Rslk6b-JmnExZH9Caco_RlePUHW-PeJ8SE9-tyAoIUU1Vph15YTiZFZZXICdlWg) - - [2018-09-17 15:37:42] "POST /conversations/asdfasdfasdf/respond HTTP/1.1" 200 438 0.022223
rasa_1 | 2018-09-17 15:37:55 WARNING py.warnings - /opt/conda/envs/rasa-env/lib/python3.5/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.
rasa_1 | if diff:
rasa_1 |

kubernetes deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: rasa-deployment
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: rasa-pod
    spec:  
      containers:
        - name: container-rasa
          image: myrasaimageonazure
          ports:
          - containerPort: 5005
            name: rasa
          resources:
            requests:
              cpu: 200m # 0.1 CPU
              memory: 100Mi # ~104MB
            limits:
              cpu: 250m # 0.3 CPU
              memory: 300Mi # ~314MB
      restartPolicy: Always
      imagePullSecrets:
      - name: azure-auth

kubernetes logs of start container:

42104286_2203664793000928_7636622766216577024_n

Rasa needs a lot more memory than that though it depends on your config…

If you are using spaCy and duckling, probably you will need more memory than the limits you have put

1 Like