Hello! Recently I’ve decided to try using rasa data validate --debug. The results were quite interesting.
There were a lot of messages about my utterances not being used in any story, like that:
DEBUG rasa.validator - The utterance ‘utter_perevod-na-karty’ is not used in any story.
But I do have a story with this utter and intent:
story_perevod-na-karty
perevod-na-karty
utter_perevod-na-karty
And it looks like all my utterance actions are listed like that in debug mode. What could it be? Some details:
every story is separated from other with a newline
encoding: UTF-8
2 spaces are used for indentation
I have 2 level stories and their utterances also listed like the others
What could it be? I don’t understand: although the training performs bad, it still can sometimes follow stories right. Thanks for the help.
Hi! I am using Rasa v1.10.10. I was able to find errors in stories.md (incorrect formatting), which caused Rasa to count the whole file as corrupted.
But now there is another problem, @tyd!
This is my config file. I am trying to recreate behaviour of EmbeddingIntentClassifier from Rasa 0.14.5. The performance of the DIETClassifier is still worse than it was in the old version, but strangely, the Rasa core module performs worse too, although policies are the same as they were in the old version. Maybe someone could help me out? Thanks!
Today I tried “EmbeddingIntentClassifier”, as it is still present in Rasa v1.10.10. It was better than using DIET with the config from above, BUT now, when I use “cosine” similarity, the NLU model gives almost identical probabilities for all the intents - about 3.33 ("ranking_length" is set to 3).
When I use “inner” for similarity, top intent’s confidence of the model grows noticeably (0.4 and higher), but still overall accuracy is not as good as it was in the old Rasa (0.14.5), although I use the same exact configs. What could it be?
How many training stories, intents, entities, and examples per intent/entity do you have in your training data?
What is the output of rasa data validate and rasa data validate stories --max-history X, where X is the max-history of your policy (X = 4 according to your config)?
By how much do results change when you change the random seed a few times? If they change a lot, you simply never had enough / good training data and just got lucky with the old version.
If you really want to reproduce old EmbeddingIntentClassifier behaviour, the closest config should be