Upgrading from 1.2.* to 1.3 highly decreased classifier's confidence

Rasa Open Source
1

I have default config

  • name: WhitespaceTokenizer case_sensitive: false
  • name: RegexFeaturizer
  • name: CRFEntityExtractor
  • name: CountVectorsFeaturizer
  • name: EmbeddingIntentClassifier epochs: 60

I think this is about hyperparametres and I’m bad at tuning. Can you advice me what to do and what changed in the classifier after 1.2?

When i train on 1.2.* i get following report:

Epochs: 100/100 [00:11<00:00, 8.43it/s, loss=0.195, acc=1.000]

rasa.nlu.classifiers.embedding_intent_classifier - Finished training embedding classifier, loss=0.195, train accuracy=1.000

hist
hist866×855 20.5 KB

1.3.* and 1.4.*: Epochs: 100/100 [00:07<00:00, 12.99it/s, loss=1.959, acc=0.999] rasa.utils.train_utils - Finished training embedding policy, train loss=1.959, train accuracy=0.999

As you can see loss is higher

hist
hist857×855 19.6 KB

2

There are too many things that have changed between 1.2 and 1.3. As mentioned in change logs, embedding_intent_classifier has definitely changed in default parameters as well as architecture.

So if it is breaking in your particular case, there could be more than one issue. Just to start you can play with evaluate_every_num_epochs which decides how often to calculate training accuracy, evaluate_on_num_examples which decide number of examples to use for training accuracy calculation. Both have changed in these versions