Hello, I have around 1374 stories 150000 line of codes, and currently training core with 12GB of RAM and it returns process killed, any idea how to train this whole lot story? or train it separately? thanks in advance.
note : we have separated the stories into 4 files and trained it together.
I have the same problem here.
I think it occurs because of np.array() need to load features to the memory. Maybe dev can use something like batch processing to optimize and resolve this problem. @akelad @Juste @tmbo . References : python - MemoryError for np.array - Data Science Stack Exchange
and by the way. here is the policy that we use
policies:
- name: KerasPolicy
epochs: 50
max_history: 6
- name: AugmentedMemoizationPolicy
max_history: 6
any suggestion for lighter policy so the training can be done without memory error ?
have you tried using --augmentation 0?
I just tried using --augmentation 0, however, our training accuracy became lower (around 0.5 only) and our core is now messed up. Now, the core can’t even predict the next utterances and actions. We directly point to the intent using “/intent” (we did not use the nlu), so the problem is definitely the core. Do you happen to know why this happened? Is this because of the ‘-augmentation 0’
From what I read on rasa core docs. It’s important to concatenate stories and create longer stories. But, it turns out that the lower augmentation we set, the stories will become more messed up (maybe this occured because I have so much stories). @akelad do you have any other suggestion ?
-- augmentation 0 stops artificial augmentation of the data, this was implemented for the keras policy mainly. It’s important to concatenate/create longer stories mainly for the keras policy, not so much for the memo policy. I’m not sure what you mean about your stories getting messed up if you have lower augmentation? Surely this should be the opposite