Training multiple agents dynamically

Hey, I’m Ruben from the Jovo team, happy to be here. We are currently trying to compensate for the removal of projects, in essence we need a different model per user, so high scalability is key.

Currently, we have a Flask server, which dynamically generates a model and spins up an agent for it. Since an agent can take up memory of up to 240MB, we can’t/don’t want to run them in parallel, as this would be too costly with 20+ agents. Dynamically loading an agent upon request takes up to 12s, not an alternative. What we were trying to do now was to spin up agents asynchronously and then use shelve to serialize and persist the agent instance and load that on request, which should be much faster and way better to scale. Unfortunately, we get this error when trying to serialize the agent:

Serializing it did work neither with dill nor with json.dumps. I’m not a Python expert, in fact this is the first Python code I wrote in years, but a bit of debugging led me to believe that this error only comes up when using the DIETClassifier component. Is there a solution to that problem or do I need to look into other classifier components?

1 Like

@rubenaeg Heya! Rubin, Nice to meet you. I guess this issue is not related to rasa, its more related to flask.

Nice to meet you as well :slight_smile: Unfortunately cannot confirm. Tried it outside of the Flask scope and it still fails with the same error. Serialization fails for a trained/loaded DIETClassifier.

@rubenaeg Try flush the memory of the server, or share logs of rasa .

Seems like this could be an issue with tf.keras, as mentioned here.

@rubenaeg Meanwhile, please share your Python Version and Rasa Version, please

rasa --version

@rubenaeg Please try to update the flask app to the latest.

$ rasa --version
$ Rasa Version      :         2.8.1
$ Minimum Compatible Version: 2.8.0
$ Rasa SDK Version  :         2.8.1
$ Rasa X Version    :         None
$ Python Version    :         3.8.11
$ Operating System  :         Linux-5.10.52-1-MANJARO-x86_64-with-glibc2.2.5
$ Python Path       :         /*/python/bin/python3.8
$ python --version
$ Python 3.8.11