Currently, we have a Flask server, which dynamically generates a model and spins up an agent for it. Since an agent can take up memory of up to 240MB, we can’t/don’t want to run them in parallel, as this would be too costly with 20+ agents. Dynamically loading an agent upon request takes up to 12s, not an alternative. What we were trying to do now was to spin up agents asynchronously and then use shelve to serialize and persist the agent instance and load that on request, which should be much faster and way better to scale. Unfortunately, we get this error when trying to serialize the agent:
Serializing it did work neither with
dill nor with
json.dumps. I’m not a Python expert, in fact this is the first Python code I wrote in years, but a bit of debugging led me to believe that this error only comes up when using the
DIETClassifier component. Is there a solution to that problem or do I need to look into other classifier components?