My team recently tried to migrate to Rasa 2.0. We’ve had several issues with the TED and especially that it doesn’t have the same metrics (t-loss, loss, accuracy) with the ones we got through training in Rasa 1.10.8.
We’ve tried to train TED for various hyperparameter settings (changed dense dimensions, epochs (30-200), batch sizes) but still got mediocre results (Rasa 1.10: 99% accuracy, Rasa 2.0: 84% accuracy).
Also, I would like to point out that theoretically the network would have to overfit. We’ve tried training with 40,100,300 epochs but the accuracy is still the same.
The policy used:
I will attach some snipets from the training procedure.
I’ve also tried everything in the following thread: TED classifier has lower accuracy after migration to Rasa 2.0 .
Thus, I don’t think it’s a hyperparameter tuning problem. Maybe the embedding space is not that large but I’ve tried changing it also.
Is something wrong with the implementation of TED in Rasa 2.0?
PS. We’ve secured a higher accuracy (93.4%) by setting the max_history to 10 or 13 but this exponentially increases the training time. Also, as far as I’m concerned, for shallow conversations (with a depth of 5-7 dialogue turns) this shouldn’t produce better results. Our tests (either automatically from rasa test or from manual shell+actions testing) are usually 5-7 turns.