Train a model inside a Kubernetes pod

Hi @ChrisRahme following the Rasa Open source deployment tutorial in Kubernetes, I’ve been successful in deploying my bot with custom actions in my cluster. For reference, I’ve followed this tutorial - Understanding Rasa Deployments - Local Kubernetes Demo - YouTube

Now, the approach provided here is that you have an already trained rasa model outside the Kubernetes cluster and it is simply put in the pod container. This works fine for me too. What I really want to do now is that I just want model training to happen inside the container itself i.e. based on the nlu and core data present in the pod, model training should happen via the ‘rasa train’ command and then rasa server should run this model via the ‘rasa run’ command as soon as the pod is deployed.

What I have and what I want can be more clearly depicted by this image -

You should look at setting up an init-container if you want to use this approach. You could also look at the trainInitialModel option.

The recommended approach is to do training and testing in a CI/CD pipeline (see the Rasa github actions), push the model to block storage somewhere and use initialModel: in the values.yaml to load this model.

1 Like

Hi @stephens thanks for your reply. The thing is that I am not using helm chart approach, so I can’t use the ‘trainInitialModel’ option in values.yaml. I’ve created my own yaml as described in the video link attached in my post. In the video it is shown that in the container spec, we can specify rasa commands under command parameter like this - command: [‘rasa’, ‘run’, ‘port’, ‘5005’, ‘–enable-api’], so I thought if I could put the ‘train’ command among these options, then the line could be - command: [‘rasa’, ‘train’, ‘run’, ‘port’, ‘5005’, ‘–enable-api’]. But, this is not working for me. Although, I will take a look at the initContainer approach.

The helm charts work fine with kind (shown in that video).

Yes, I know. But the thing is using helm is not advised in my current organization and so I need to find an alternative. I’m currently looking to see the initContainer approach. Will let you know if it works for me or not.

In that case, you can use the helm template command on your local computer to generate the kubernetes yml files.

Okay @stephens, will try that and get the required yml files.

I’m able to train the model inside the pod using the suggested init Container approach. My init Container trains the model using the nlu and core data. Then, the actual app containers take care of starting the rasa server and the action server. Thanks!

1 Like