I have about 800 intents, each with around 20 training phrases on average, exported from Dialogflow.
I used the rasa tools to convert that into a rasa file.
This is only NLU training.
I am attempting to now train this model. I have tested the following GCP virtual machine configurations using Debian:
224 vCPUs + 224 GB Ram => ~26 hours of training.
96 vCPUs + 86 GB Ram + 8 GPU Nvidia Tesla [with installed tensorflow-gpu] => ~38 hours of training.
Obviously this is not sustainable, especially since we have to train our model daily. I mean, even the most expensive and most powerful Google VM cannot handle this, yet the same exact dataset takes less than 1 hr to train in Dialogflow.
I also tested removing all entities and synonyms - it made no difference in the training phase. I am using the default configuration yml values… aka 100 epochs, etc, etc.
So, I would love to get some input, insights, ideas, whatever… how do I get Rasa to train this set? And no, more RAM, GPU and CPU is NOT THE ANSWER