Reducing Memory Usage for Rasa Open Source

Rasa Open Source
1

Hi all,

I’ve been deploying Rasa Open Source on GKE of late and am wondering whether the memory usage I am seeing is normal. I see Rasa core using over 2 GB of memory on average, and this seems like quite a lot. I have 105 intents with 1800 training samples. Does this sound like normal memory usage?

I am using Spacy so am guessing this might be one of the causes, but am wondering what other people do to reduce memory. At the moment we are putting one rasa source instance + one action server on each node using 1 e2-medium machines (2 vcpus and 4 GB memory), which means each prod env is costing us ~$40. We have staging env using preemptible nodes so they cost nearer ~$15.

So it seems like it will cost us ~$55 per bot per month to host a workable Rasa set up (with both dev and prod envs). Is this what other people are paying?

Are there any tricks to reduce memory usage?

Thanks!

2

I am also seeing same issue. As compared to rasa1.10, rasa2 consumes lot of memory (almost 2 GB). Tagging @Juste

3

Hello @botmatic && prakashdale,

over the last years of tensorflow releases some memory leak where spotted.

this is one way to bypass some of the problems.

EDIT1 : Also it is known that Tensorflow has a tendency to try to allocate all available RAM which makes the process killed by OS. To solve the issue you could use tf.GPUOptions to limit Tensorflow 's RAM usage.

this is one way.

the other way is to circumvent the data spike by making regular memory metrics during your trainings. https://www.tensorflow.org/api_docs/python/tf/config/experimental/get_memory_usage So you will find the crulprit spike data which is generating/consuming the most memory.

@botmatic you have by chance any monitoring running with some system metrics ? ( prometheus, datadog, nagios ? ) so there is some graph with the overall memory consumption over time ? would be intresting.

and last but not least. Regular Restarts after training a training is finished.

2 Likes
4

Thanks @RASADSA I’ll have a look at your link and monitoring. I actually moved from medium to small spacy model and now I’m seeing 700 MB memory usage :slight_smile:

1 Like