Perhaps you’ve found your answer by now – but you can take the Rasa base image, modify it, and then use that new image in your deployment.
Here’s a starter on how to commit changes to a docker image, there may be a rasa-specific version out there too: