Rasa Model taking alot of time to train

Rasa Open Source
1

Hi,

So bascially i am trying to train Rasa_NLU model with 868000 records, 6 distinct entities and 85 distinct intents. It is taking alot of time to train the model more than 36 hrs

System Specification : 8 Core Processor, 64 GB Ram plus 32 GB swap memory and Ubuntu 16.04 OS.

Config.json : { “pipeline”: “spacy_sklearn”, “path” : “models_1/”, “data” : “Chatito”, “emulate” : “luis”, “num_threads”:6 }

Could you please help me with the same?

1 Like
2

Hello @tejas. We are looking into some memory related issues. However, 10k examples per intent looks like a very big amount of data. As of now, I would look into ways to cut the size of the training data you have excluding examples which don’t contribute much to the performance of the model.

3

Your specification is good enough to train your data, I think its weird when it took 36hours. Try watch your resources, maybe RAM is full?

4

Hi @tejas @Juste:- I am also facing the same issue. Can you please suggest the required configuration to reduce the training time.

5

Hey @prasgaut. One thing you could try is setting the augmentation to 0 to prevent Rasa from augmenting stories from the ones you have in your stories.md file. You can do that by using the augmentation flag as follows: rasa train --augmentation 0

1 Like
6

JEY @Juste Thanks for providing this solution. I have tried this argument and it speed up the rasa core training but my main concern is regarding Rasa NLU team. Its taking too much time to train the model. Can you please provide any solution to speed up NLU training.

7

@Juste @Tanja:- Can you please suggest configuration that can help me to reduce NLU training time. Currently i have 20 intent and 20K conversation.

Server configuration:- 8core 16 GB ram

8

Hey @prasgaut. 20 intents shouldn’t take long to train. How many examples for each intent do you have? I would argue that the amount of examples you have could be reduced without sacrificing the performance of your model.

9

Hi Team,

I am facing same issue, is there any solution …

10

Hi @Juste,

Now I have intent count to 59 and I have reduced the conversation up to 8514, which makes 150 conversations per intent. I have used lookup to reduce the conversation and 80 epoch as per early stopping. Now my training time is approx to 2 hours. Can I reduce this training time further

1 Like