Memory accumulation on train and reload agent with updated models

We are currently on Rasa - 1.3.2 version. We have few concerns related to the memory consumption by Rasa in the context of the way we use it.

Use case

We have a user interface which allows the user to define utterances/dialog flow etc. After changes, we allow the user to publish the changes which will update the models on the Rasa instance that is already running and serving requests. On publish, we run APIs to train the models and reload the agent with a new Interpreter instance.

We observed that the memory keeps on accumulating each time

  • We try to train the models on the fly
  • Reload the agent on the fly i.e. Recreate the Agent with the updated models.

Note: we spawn a new thread each time we try to train and reload the models.

NLU Pipeline Configuration


  • name: “WhitespaceTokenizer”

intent_split_symbol: “_”

intent_tokenization_flag: true

  • name: “RegexFeaturizer”

  • name: “CRFEntityExtractor”

  • name: “EntitySynonymMapper”

  • name: “CountVectorsFeaturizer”

  • name: “EmbeddingIntentClassifier”

  • name: DucklingHTTPExtractor

dimensions: [“time”, “number”, “email”]

locale: “en_US”

timezone: “UTC”

Dialog Policy Configurations

  1. {“name”: “KerasPolicy”,“priority”:1,“epochs”: 300,“batch_size”: 10,“validation_split”: 0.2,“featurizer”: [{“name”: “MaxHistoryTrackerFeaturizer”,“max_history”: 5,“state_featurizer”: [{“name”: “BinarySingleStateFeaturizer”}]}]}

  2. (“name”: “AugmentedMemoizationPolicy”,“priority”:2,“max_history”: 6}

  3. {“name”:“FallbackPolicy”,“priority”:3,“nlu_threshold”:0.3,“core_threshold”:0.3,“ambiguity_threshold”:0,“fallback_action_name”:“action_default_fallback”}

  4. {“name”:“FormPolicy”,“priority”:4}

  5. {“name”:“MappingPolicy”,“priority”:5}

Any help/inputs/suggestions on how to address the memory issue is appreciated.

1 Like

we’re planning to release an update 1.6 where memory consumption of nlu models is drastically reduced

@Ghostvv - Thanks for the prompt reply. May I know when 1.6 will be released?

Also, we have observed the memory accumulation while building core models as well. In fact, core modes generation has more accumulation compared to nlu models and it does not get release after the model generation. Any insights on this?

what do you mean by memory accumulation?

1.6 is planned for Wednesday, 18.12.2019

Thanks for the update !

When the Core models are built, the memory gets occupied and does not get released after the models are generated. As we keep generating the models one after the other memory gets occupied and does not get released.

As @ksoneji mentioned in the original post, we have observed the same behavior when the agent is reloaded on the fly (i.e. Recreate the Agent with new Interpreter instance) with the updated models

trained policies keep a reference to the training data they got trained with. you could try persisting the agent and loading it afterwards