Is there some way to reduce or limiting the memory usage during rasa train. I have a pretty big bot with around 13k action examples. Training has used up all 32GB of RAM that I have and my machine freezes after that.
I’m using Rasa 2.2.0.
I’ve taken to doing the training on AWS EC2 instances instead. I was wondering if anyone has an idea of which sort of instance might be best suited to doing the training? CPU intensive / Memory intensive / GPU intensive ?
Just to check, you have 13K action examples or 13K training examples?
This is a screenshot of the training. So it’s 13k action examples.
The initial estimate shows around 8 hours training, but that eventually drops to around 3-4 hours.
However, it seems to require around 2-3 hours to load the yml files before it even starts training and once the training has been completed, it take an additional 4-5 hours for it to output the final tar model file.
I’ve noticed that when loading the files, it can use up to around 38GB RAM, but after that, the CPU and memory usage drops to quite low, though GPU usage remains rather high.
Interestingly, I upgraded to version rasa 2.4.0 and it helped a lot with training time. Loading the yml files previously took 2-3 hours, it now takes a couple of mins. Writing the model file used to take 4-5 hours, it now takes a couple of mins. Training of the model took a consistent amount of time but that is something that I can live with.