HTTP endpoints becoming unresponsive while training model

Rasa Open Source
1

Hello everyone, I’m building a custom training loop for RASA chatbot model and I have encountered some error with custom channel endpoints becoming unresponsive when the model is training.

Some information about my system:

  • Os: Ubuntu 18.04
  • RASA version: 2.8.1
  • RASA sdk: 2.8.4

Detailed description: I have created a custom model training function:

from rasa.model_training import train_async
### Other endpoints ###
@custom_webhook.route("/update_flow", methods=["POST"])
        async def update_flow(request: Request) -> HTTPResponse:

            ### Updating training data files ###

            training_result = await train_async(
                domain="data", config="config.yml", training_files="data")
            return response.json("Updated", status=200)

The function run fine but when awaiting the training result, other api endpoints like health check stop responding. They only works again when the model finished training.

I see that there is a github issue for this but I don’t know how to apply it to my situation. https://github.com/RasaHQ/rasa/pull/4774

Thanks for reading.

2

Hello everyone, I’m sorry for bumping this but I still need help with this problem.

3

You shouldn’t use the same Rasa instance for both training and production. The recommended architecture is to have a separate instance to train the model.

It’s best to do training as part of a CI/CD pipeline. There’s a description here and Rasa has implemented some really nice GitHub Actions to support this workflow.

After training and testing the new model, you would push the model to the production instance.