Hi guys! We have a project with 280k stories and we want to train a basic Keras policy in Rasa Core with it. It takes two hours to load the stories and around 150 GB of RAM. We noticed that the loading is performed on a single core. Is there any plan to move to multiple cores to accelerate this process? Is it a python limitation with the use of await async? Thanks, L-P and the Dialogue Team

Sorry to answer with an unrelated question, but how did you get 280,000 annotated stories?

Hi Leo! We have these stories because we are trying to move from our actual rule-based system to an LSTM policy. It is thus easy for us to create “stories” from our rule-based system and then convert them to Rasa format so we can use them as training data.

hey That is interesting. How did you do that.

@amn41 @Ghostvv his is a use-case of training LSTM based on a rule-based system. We discussed it earlier, we were wondering if this is something you’ve seen other people face and if you have any suggestions of how to train the model with a large training set. The reason why this training is so large is that we observe that the model keeps learning with more data. We do have early-stopping in place, but it still quite slow at this scale. Any help on what to investigate would be appreciated. Thanks, Alexis

I’m guessing Rasa was never designed to work with this many stories. I think the Embedding policy claims to work well in adapting to deviations from the happy path with 30 stories or so and the Keras Policy with about a 100. Isn’t the whole point to build a model that can generalize well from a few hundred cases instead of feeding it every possible permutation? I’m not very familiar with the ML aspect of this, so if someone could explain how you avoid overfitting and losing the benefit of the models “artificial intelligence”, I would really appreciate it.

Hi L-P, Thanks for the post. This is super interesting, we’ve never trained on that many stories before so I’m not surprised it’s taking a while. I suspect that a great deal of your stories are implementing some business logic. Did you consider wrapping some of that logic up into a form? I suspect you could dramatically reduce the amount of stories you generate that way. The ideal case (as I see it) would be that you implement the business logic inside of forms, and use ML to learn the deviations from the happy path. I’m not sure how those are handled in your rule-based system (if at all) but maybe you could post a couple of example stories and we could have a look.

Hi Alan, Thanks for your answer. Yes, we are trying to mimic a business logic, but it’s a pretty complicated one. We have around 800 possible questions and 1200 possible outcomes. We are trying to learn the best questions to ask based on the results of the previous ones to maximize the confidence in the outcome and minimize the number of questions asked. We do not want to implement a form, since our whole point is to get away from an “hard-coded” system.

Also, with our entire dataset (325k stories), we are not able to train as we get a sig kill right after Rasa is done loading the data…

Hi, thanks for your post, that’s very interesting idea to train ML policy to learn overly comlicated rule-based system. How do you load and featurize the stories? I’m afraid with this amount of stories you’d be better feeding them directly from the file using tf.keras during training. I’m not sure about tf.keras, but TensorFlow has special tf.data.Dataset object to handle data: Importing Data | TensorFlow Core | TensorFlow I think they somehow integrated it into tf.keras

@Ghostvv That would mean by-passing the normal Rasa Core training? As of now, we use the provided agent.load_data( ) method.

Story Loading on multiple cores

Rasa Open Source

Ghostvv (Vladimir Vlasov) April 16, 2019, 2:08pm 18

I’m not sure, because you don’t give old stories to the algorithm, so algorithm will slowly start to override old behavior with the new one. Exact details depend on how varied your stories are

Topic		Replies	Views
[Ask] process killed when training stories on RASA Core Rasa Open Source	8	1846	March 6, 2019
Running out of memory with a large number of stories Rasa Open Source	3	1301	March 11, 2019
More information on rasa core model training process Rasa Open Source	9	1947	August 17, 2018
[SOLVED]RASA story misbehaving Rasa Open Source	23	4206	May 22, 2019
Story Blocks never process before Interactive Training with many Checkpoints Rasa Open Source	2	579	March 26, 2022

Story Loading on multiple cores

Related topics