New Rasa-X-Helm commit breaks quick-install and helm upgrade

Hello. Suddenly, I cannot modify my deployment anymore:

$ curl -s get-rasa-x.rasa.com | sudo -E bash

Found existing cluster.
Installing Helm command line interface  ...
Upgrading Rasa X ...
Error: UPGRADE FAILED: error validating "": error validating data: [ValidationError(Deployment.spec.template.spec.containers[0].livenessProbe): unknown field "exec:command" in io.k8s.api.core.v1.Probe, ValidationError(Deployment.spec.template.spec.containers[0].readinessProbe): unknown field "exec:command" in io.k8s.api.core.v1.Probe]

I get the same error when I do

$ helm upgrade --values values.yml --namespace rasa --reuse-values rasa rasa-x/rasa-x
1 Like

It might be due to this commit considering the chart was working yesterday and that it works with previous versions of the chart. Merge pull request #168 from RasaHQ/nginx_probes · RasaHQ/rasa-x-helm@4d1a1f1 · GitHub

here it is working with 1.8.2

$ helm install rasa rasa-x/rasa-x -n ***************** --version 1.8.2
NAME: rasa
LAST DEPLOYED: Tue Apr  6 19:28:46 2021
NAMESPACE: *******************************
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thanks for installing Rasa X  !
Creating a Rasa X user:

- Rasa X CE: go to the terminal of the `rasa-x` pod and then
  execute `python scripts/manage_users.py create --update me <your password> admin`
  to set your password
- Rasa X EE: go to the terminal of the `rasa-x` pod and then
  execute `python scripts/manage_users.py create <your username> <your password> <role of your user>`.
  You can then log in using these credentials.

Check out the Rasa X docs here for more help:
https://rasa.com/docs/rasa-x/

latest version (1.8.4)

$ helm install rasa rasa-x/rasa-x -n ***************** --version 1.8.4
Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: [ValidationError(Deployment.spec.template.spec.containers[0].livenessProbe): unknown field "exec:command" in io.k8s.api.core.v1.Probe, ValidationError(Deployment.spec.template.spec.containers[0].readinessProbe): unknown field "exec:command" in io.k8s.api.core.v1.Probe]
I think it begs the question, how did it pass the ci tests?
2 Likes

Oh wow, looks indeed like this might be the cause. This needs more attention then… @mloubser, any ideas since you did the merge?

Meanwhile, is there a way to downgrade using the quick-install command?

1 Like

We had to temporarily modify our pipeline to install the 1.8.2 version. The quick install command would look like this:

helm install <your release name> rasa-x/rasa-x --version 1.8.2

Edit:

It also applies to helm upgrade just in case:

helm upgrade <your release name> rasa-x/rasa-x --version 1.8.2
2 Likes

Hi @jsolis

Where can i find the version “1.8.2”. I used the Quick install for my deployment.

curl -s get-rasa-x.rasa.com | sudo bash

2 Likes

I am sorry, I’m not familiar with that way of installing RasaX.

I was referring to this method Helm Chart Installation. The version “1.8.2” is the latest helm chart version that worked for me.

A quick look at curl -s get-rasa-x.rasa.com would indicate that it executes a helm install somewhere. Installing a specific version of the chart could be done by saving the script, modifying the script somewhere around line 250 and then executing it.

Going from:

  if [[ $1 == "install" ]]
  then
    command=("install")
  else
    command=("upgrade" "--reuse-values")
  fi

  command=("${command[@]}"
    "--set rasax.tag=${RASA_X_VERSION}"
    "--set eventService.tag=${RASA_X_VERSION}"
    "--set nginx.tag=${RASA_X_VERSION}"
    "--set rasa.tag=${RASA_VERSION}-full"
  )

  if [[ $1 == "install" ]]
  then
    command=("${command[@]}"
      "--set rasax.initialUser.password=${INITIAL_USER_PASSWORD}"
      "--set global.postgresql.postgresqlPassword=${POSTGRES_PASSWORD}"
      "--set global.redis.password=${RABBITMQ_PASSWORD}"
      "--set rabbitmq.rabbitmq.password=${REDIS_PASSWORD}")
  fi

  if $IS_EMBEDDED_CLUSTER
  then
    command=("${command[@]}" "--set nginx.service.type=ClusterIP")
    command=("${command[@]}" "--set ingress.hosts[0].host=,ingress.hosts[0].paths={/}")
  else
    command=("${command[@]}" "--set nginx.service.type=${NGINX_SERVICE_TYPE}")
    # gcloud needs another path for the ingress than K3s
    command=("${command[@]}" "--set ingress.hosts[0].host=,ingress.hosts[0].paths={/*}")
  fi

  if  [[ $1 == "install" ]] || [[ -n "${DISABLE_TELEMETRY}" ]]
  then
    command=("${command[@]}" "--set rasax.disableTelemetry=${DISABLE_TELEMETRY:-false}")
  fi

  if  [[ $1 == "install" ]] || [[ -n "${ENABLE_DUCKLING}" ]]
  then
    command=("${command[@]}" "--set duckling.enabled=${ENABLE_DUCKLING:-False}")
  fi

  if  [[ $1 == "install" ]] || [[ -n "${ACTION_SERVER_IMAGE}" ]]
  then
    command=("${command[@]}" "--set app.name=${ACTION_SERVER_IMAGE:-rasa/rasa-x-demo}")
  fi

  if  [[ $1 == "install" ]] || [[ -n "${ACTION_SERVER_TAG}" ]]
  then
    command=("${command[@]}" "--set app.tag=${ACTION_SERVER_TAG:-${RASA_X_DEMO_VERSION}}")
  fi

  if  [[ $1 == "install" ]] || [[ -n "${DEBUG_MODE}" ]]
  then
    command=("${command[@]}" "--set global.debugMode=${DEBUG_MODE:-False}")
  fi

  if [[ -n "${ADDITIONAL_CHANNEL_CREDENTIALS}" ]]
  then
    # additional credentials may be passed in comma separated, e.g.
    # facebook.verify="dasda",facebook.test="dasd"
    IFS=',' read -ra channels <<< "$ADDITIONAL_CHANNEL_CREDENTIALS"
    for channel_setting in "${channels[@]}"
    do
     command=("${command[@]}" "--set rasa.additionalChannelCredentials.$channel_setting")
    done
  fi

  command=("${command[@]}"
    "--set rasax.extraEnvs[0].name=QUICK_INSTALL"
    "--set-string rasax.extraEnvs[0].value=true"
    "--namespace ${DEPLOYMENT_NAMESPACE}" "${DEPLOYMENT_NAME}" "rasa-x/rasa-x"
  )

  no_root_helm "${command[@]}" > ${REDIRECT}
}

To:

  if [[ $1 == "install" ]]
  then
    command=("install")
  else
    command=("upgrade" "--reuse-values")
  fi

  command=("${command[@]}"
    "--set rasax.tag=${RASA_X_VERSION}"
    "--set eventService.tag=${RASA_X_VERSION}"
    "--set nginx.tag=${RASA_X_VERSION}"
    "--set rasa.tag=${RASA_VERSION}-full"
  )

  if [[ $1 == "install" ]]
  then
    command=("${command[@]}"
      "--set rasax.initialUser.password=${INITIAL_USER_PASSWORD}"
      "--set global.postgresql.postgresqlPassword=${POSTGRES_PASSWORD}"
      "--set global.redis.password=${RABBITMQ_PASSWORD}"
      "--set rabbitmq.rabbitmq.password=${REDIS_PASSWORD}")
  fi

  if $IS_EMBEDDED_CLUSTER
  then
    command=("${command[@]}" "--set nginx.service.type=ClusterIP")
    command=("${command[@]}" "--set ingress.hosts[0].host=,ingress.hosts[0].paths={/}")
  else
    command=("${command[@]}" "--set nginx.service.type=${NGINX_SERVICE_TYPE}")
    # gcloud needs another path for the ingress than K3s
    command=("${command[@]}" "--set ingress.hosts[0].host=,ingress.hosts[0].paths={/*}")
  fi

  if  [[ $1 == "install" ]] || [[ -n "${DISABLE_TELEMETRY}" ]]
  then
    command=("${command[@]}" "--set rasax.disableTelemetry=${DISABLE_TELEMETRY:-false}")
  fi

  if  [[ $1 == "install" ]] || [[ -n "${ENABLE_DUCKLING}" ]]
  then
    command=("${command[@]}" "--set duckling.enabled=${ENABLE_DUCKLING:-False}")
  fi

  if  [[ $1 == "install" ]] || [[ -n "${ACTION_SERVER_IMAGE}" ]]
  then
    command=("${command[@]}" "--set app.name=${ACTION_SERVER_IMAGE:-rasa/rasa-x-demo}")
  fi

  if  [[ $1 == "install" ]] || [[ -n "${ACTION_SERVER_TAG}" ]]
  then
    command=("${command[@]}" "--set app.tag=${ACTION_SERVER_TAG:-${RASA_X_DEMO_VERSION}}")
  fi

  if  [[ $1 == "install" ]] || [[ -n "${DEBUG_MODE}" ]]
  then
    command=("${command[@]}" "--set global.debugMode=${DEBUG_MODE:-False}")
  fi

  if [[ -n "${ADDITIONAL_CHANNEL_CREDENTIALS}" ]]
  then
    # additional credentials may be passed in comma separated, e.g.
    # facebook.verify="dasda",facebook.test="dasd"
    IFS=',' read -ra channels <<< "$ADDITIONAL_CHANNEL_CREDENTIALS"
    for channel_setting in "${channels[@]}"
    do
     command=("${command[@]}" "--set rasa.additionalChannelCredentials.$channel_setting")
    done
  fi

  command=("${command[@]}"
    "--set rasax.extraEnvs[0].name=QUICK_INSTALL"
    "--set-string rasax.extraEnvs[0].value=true"
    "--namespace ${DEPLOYMENT_NAMESPACE}" "${DEPLOYMENT_NAME}" "rasa-x/rasa-x"
    "--version 1.8.2"
  )

  no_root_helm "${command[@]}" > ${REDIRECT}
}

That little change to this bit

  command=("${command[@]}"
    "--set rasax.extraEnvs[0].name=QUICK_INSTALL"
    "--set-string rasax.extraEnvs[0].value=true"
    "--namespace ${DEPLOYMENT_NAMESPACE}" "${DEPLOYMENT_NAME}" "rasa-x/rasa-x"
  )

should do the trick for this specific situation.

It’s not as pretty as just curl -s get-rasa-x.rasa.com | sudo bash but it should get you out of this situation you are in.

2 Likes

That’s awesome! Thanks a lot for the help!

I want to mark your answer as a solution but I’m afraid the Rasa team will not check the thread if they see a solution has been found.

I will update the title.

Also, someone just opened an issue on GitHub: Default nginx liveness and readiness probes broken · Issue #173 · RasaHQ/rasa-x-helm · GitHub

1 Like

I want to mark your answer as a solution but I’m afraid the Rasa team will not check the thread if they see a solution has been found.

No worries, we are on it! Please mark the solution in the meantime so that anyone running into this can use the workaround, and I can come back to mark it when upgraded!

2 Likes

Awesome, thanks!

1 Like

Thank you @jsolis I managed to reinstall using helm method.

2 Likes

Hi y’all – should work again with the quick install script and the Helm charts (make sure to do a helm repo update to get chart version 1.8.5)

3 Likes

Thank you!

1 Like