Kubernetes/Quick install deployment not working on SW >=2.8.0

Rasa Open Source
1

Hi,

Currently, i m using Rasa 2.7.1. When i try to upgrade Rasa to versions >=2.8.0, it works properly on local mode, but it doesn’t on Rasa server’s deployment using Kubernetes/quick install script and exact new versions of Rasa/Rasa X/Rasa SDK as for local (the deployment is getting ready but after i upload the training model, it sucks in bot typing and no response received from the bot). Below are the current requirements for the actions server. What m i missing?

Rasa Version      :         2.7.1
Minimum Compatible Version: 2.6.0
Rasa SDK Version  :         2.7.0
Rasa X Version    :         0.41.2
Python Version    :         3.8.10
Operating System  :         Linux-5.11.0-38-generic-x86_64-with-glibc2.29
Python Path       :         /usr/bin/python3

cx-Oracle
google
mysql-connector-python
pycountry
requests
spacy
pyodbc
jsonrpcclient==3.3.6
pysimplesoap
pyTelegramBotAPI
pytz
2

also, the command: kubectl get pods --namespace rasa doesn’t show any error (all pods are running).

3

Note that i delete Kubernetes instance and docker images before any new deployment:

- kubectl delete namespaces rasa
- docker rmi -f $(docker images -a -q)
4

Do you see any error when you do those?

kubectl logs --namespace rasa <pod_name>

kubectl logs --previous --namespace rasa <pod_name>
5

Chris, no, everything looks fine, no errors or any clues. Just the bot doesn’t answer.

6

Typically, you would update the image tags in your values.yml and do a helm upgrade. What error are you seeing? Are the pods still running the old version? A describe on your pods would be good to see.

7

@stephens

We are using the below quick-install script without values.yml. Kindly find below the script used for deployment:

# ALIASES
function mkcd() { mkdir -p -- "$1" && cd -P -- "$1"; }
alias cls="clear"
alias py="python3.8"
alias py3="python3.8"
alias py37="python3.8"
alias py3.7="python3.8"
alias python="python3.8"
alias python3="python3.8"
alias python37="python3.8"

alias pip="py -m pip3.8"
alias pip3="py -m pip3.8"
alias pip37="py -m pip3.8"

alias desktop="mkcd /home"
alias chatbot="mkcd /home/fyp-chatbot"

alias kube="kubectl --namespace rasa"
alias kp="kube get pods"
alias ks="kube get services"
alias ki="kube get ingress"
alias kd="kube get deployments"
alias kss="kube get statefulsets"
alias krs="kube get replicasets"
alias ksx="kube get secrets"
alias kdel="kubectl delete namespaces rasa"
function kl() { kube logs "$1"; }

function octant-dashboard() { sudo OCTANT_LISTENER_ADDR=0.0.0.0:7777 octant --disable-open-browser --kubeconfig $KUBECONFIG; }

alias api_token_json="curl -X POST -H \"Content-Type: application/json\"  -d '{\"username\":\"me\", \"password\":\"IP@eco@2021\"}' http://10.126.15.10/api/auth"
alias api_token_value="api_token_json | python3 -c 'import json,sys;obj=json.load(sys.stdin);print(obj[\"access_token\"])'"

#GIT
git-pull() 
{
	chatbot && git stash --include-untracked && git reset --hard && git pull local master
}

#Docker
docker-push() 
{
        chatbot && kubectl delete secret docker-hub --ignore-not-found && \
	kubectl create secret docker-registry docker-hub --docker-username=xx --docker-password=x --docker-email=y && \
        export ACTION_SERVER_IMAGE="rasachat/rasa" && \
        export ACTION_SERVER_TAG="chatbot" && \
        export ACTION_SERVER_FULL=$ACTION_SERVER_IMAGE":"$ACTION_SERVER_TAG && \
        sudo docker login && sudo docker build . -t $ACTION_SERVER_FULL && \
        sudo docker push $ACTION_SERVER_FULL
}


# Modify Rasa X deployment
rasa-deploy()
{
        export RASA_VERSION="2.7.1" && \
		export RASA_X_VERSION="0.41.2" && \
		export RASA_SDK_VERSION="2.7.0" && \
		export DEBUG_MODE=true && \
        export INITIAL_USER_PASSWORD="zzz" && \
        export ADDITIONAL_CHANNEL_CREDENTIALS="rest=''" && \
        curl -s get-rasa-x.rasa.com | sudo -E bash
}

# Disable intent insights calculation beacuse it makes the server very slow
# To disable: https://rasa.com/docs/rasa-x/user-guide/annotate-nlu-examples#changing-the-schedule
# To enable: https://rasa.com/docs/rasa-x/pages/http-api#operation/putInsightConfig
disable-insights() 
{
	TOKEN=$(api_token_value) \
	curl --request PUT \
		 --url 'localhost/api/insights/config' \
		 --header "Authorization: Bearer "$1 \
		 --header 'Content-Type: application/json' \
		 --data '{"schedule": null, "cross_validation_folds": 4, "calculator_configuration": null}' && \
	printf "\n"
}

# Deploy action server
deploy-actions() 
{
	cd /home/fyp-chatbot && sudo systemctl start docker && sudo docker system prune && \
	printf ">>> Pulling from local Rasa GIT...\n" && git-pull && \
	printf "\n\n>>> Building and pushing Docker image...\n" && docker-push && \
	printf "\n\n>>> Modifying Rasa X deployment...\n" && rasa-deploy && \
	printf "\n\n>>> Rasa X is now using image " && echo $ACTION_SERVER_FULL"." && \
	printf "\n\n>>> Deployment completed successfully." && kubectl expose pod rasa-postgresql-0 --port=5432 --target-port=5432 --external-ip=zz --namespace=rasa
}

When we perform an upgrade, we just change the below version values of Rasa/Rasa X/Rasa SDK based on version matrix compatibility:

export RASA_VERSION="2.7.1" && \
		export RASA_X_VERSION="0.41.2" && \
		export RASA_SDK_VERSION="2.7.0"
8

What does the describe show on your pods, does it confirm the upgrade has happened? What’s your pullPolicy for the images?

Since you’re using the Rasa Helm Chart via the quick install script, you should be using the associated values.yml, not the environment variables.

9

can u assist us more about how to do it? an instructions link will be great.

10

There’s a new Rasa deployment series here and the Rasa Advanced Deployment workshop.

Also, the O’Reilly Kubernetes: Up and Running is very good.