Any way to to check Training process

Rasa Open Source
1

Hi Rasa Now I’m training story with script generation. They’re about 260G including 45 files and 5.8G for each. I’m running with “rasa train --debug” and find it’s pending on “pykwalify.compat - Using yaml library: /usr/local/lib64/python3.6/site-packages/ruamel/yaml/init.py” which kept for hours.

I’m wonder if there is way to check more details and estimate how long should I wait. My bot was running on VM with 20GB memory ,8 Vcpus and I’m using supervised_embeddings for pipeline.

BTW is there any limit for rasa training or any suggestion for improvement. Thanks in advance

2

@linyisen you are most probably running into memory issues when loading the whole of your data. Currently Rasa stack loads up all of data into memory at once and hence 260 GB of data would cause a problem. I would suggest training on smaller subsets of data which can fit in memory.

3

hi Daksh Thanks for reply, yes, it’s killed by OOM yesterday evening, it leads to almost the same size tmp directory under /tmp and crash out of swap. I will decrease it below 10G and increase swap.