Training Rasa on an Ubuntu 22.04 eats up all the memory

We are running Rasa on an Ubuntu 22.04 (AWS EC2 server). This is for a personal project that I’m working on. I see that, when I hit ‘rasa train’,the training starts and at around 66%, I see the ‘# trackers’ is at 16384 and it’s eating up all the memory. From this point on wards, the memory consumption starts to spike and the training percentage doesn’t move forward.

I need tips to impose memory consumption limits. Run the training with some memory boundaries.

Please help me to understand these trackers that I see in the status. What can I do to bring that number down?

I’ve attached a screenshot of the status that I see.

Update: The number of trackers that I mentioned, just increased to 1048576. Is there a way to limit them? What are these trackers?

I’m a newbie to Rasa.