How big of a difference between rasa 1.x.x and 0.x.x components?

Rasa Open Source
1

Hello! I have a problem migrating from Rasa 0.14.5 (nlu) and 0.15.1 (core) to Rasa 1.10.10. Details:

  • data is almost the same
  • I reused as much settings from the old config as I could
  • The performance is way worse

So I thought I should as a question: How much the components internally differ between new and old versions of Rasa?

KerasPolicy and AugmentedMemoizationPolicy components in Rasa 1.x.x are not as effective as they were in the old Rasa. E.g. Rasa core 0.15.1 showed great performance with these configs:

  • name: KerasPolicy max_history: 4-5 batch_size: [128, 512] epochs: 300
  • name: AugmentedMemoizationPolicy: max_history: 4-5
  • name FallbackPolicy nlu_threshold: 0.7 core_threshold: 0.5

With Rasa 1.10.10 these configs perform rather poorly. And also I think I don’t fully understand what max_history does exactly: does it contain only user_intent-bot_action pairs, or default internal actions also (action_listen, etc.)?

It is the same with DIETClassifier: it also performs worse than EmbeddingIntentClassifier with the same configs.

2

Hi @rhododendron, there is a significant difference under the hood between Rasa 0.X and 1.X. The TEDPolicy builds on top of the Keras Policy by adding an attention mechanism that improves prediction accuracy. You configure the DIETClassifier with the same behavior as the EmbeddingIntentClassifier using this guide.

1 Like
3

Hi! Thank you for the answer, I’ll definitely check the configs. I also have a question about TEDPolicy: I’ve tried to run it with different batch sizes (also with the default one), but it always throws OOM error, it needs more than 100Gb! Is there something I can do about it without reducing the training data? Thanks!