We have a staging rasa pipeline using GitLab-ci. It currently builds our app image, deploys rasa to our staging k8s cluster using that custom image as the app server, and cleans the environment whenever we change the actions (meaning a new action would trigger cleanup and deployment automatically).
It works great but it has a major downfall. The cleanup step is basically a helm uninstall. That means that for testing actions we have to retrain the bot every single time (since helm uninstall deletes the PV). This obviously limits the amount of debugging we can get done in a day.
We’ve thought of two solutions for this.
Modify the ci in a way that making changes to the actions dir, triggers an image build and push and then redeploy only the app server using the new image.
Modify the ci (and maybe helm chart) and change the RetainPolicy of the PersistentVolumes to RETAIN instead of DELETE. Also using a simple if statement to check for existing volumes in the namespace and if there are none do a full deploy and if there are some, mount them for the rasax pod.
What are your thought on the two options? Do you have a similar experience?
This is how our project structure looks like
. ├── actions │ ├── actions │ │ ├── actions.py │ │ ├── __init__.py │ │ └── requirements.txt │ └── Dockerfile ├── config.yml ├── data │ ├── nlu.yml │ ├── rules.yml │ └── stories.yml ├── domain.yml ├── extra-values.yaml ├── __pycache__ │ ├── ... ├── rabbitmq │ └── values.yml ├── README.md ├── results │ ├──... └── tests └── conversation_tests.md
We appreciate any input that you may have