Stories do not behave as expected

Rasa Open Source
1

Hello! Recently I’ve decided to try using rasa data validate --debug. The results were quite interesting. There were a lot of messages about my utterances not being used in any story, like that:

DEBUG rasa.validator - The utterance ‘utter_perevod-na-karty’ is not used in any story.

But I do have a story with this utter and intent:

story_perevod-na-karty

  • perevod-na-karty
    • utter_perevod-na-karty

And it looks like all my utterance actions are listed like that in debug mode. What could it be? Some details:

  • every story is separated from other with a newline
  • encoding: UTF-8
  • 2 spaces are used for indentation
  • I have 2 level stories and their utterances also listed like the others

What could it be? I don’t understand: although the training performs bad, it still can sometimes follow stories right. Thanks for the help.

2

Hi @rhododendron. What version fo Rasa Open Source are you using?

3

Hi! I am using Rasa v1.10.10. I was able to find errors in stories.md (incorrect formatting), which caused Rasa to count the whole file as corrupted. But now there is another problem, @tyd! This is my config file. I am trying to recreate behaviour of EmbeddingIntentClassifier from Rasa 0.14.5. The performance of the DIETClassifier is still worse than it was in the old version, but strangely, the Rasa core module performs worse too, although policies are the same as they were in the old version. Maybe someone could help me out? Thanks!

pipeline:

  • name: “WhitespaceTokenizer”
  • name: “CountVectorsFeaturizer” analyzer: “word” min_ngram: 1 max_ngram: 2
  • name: “DIETClassifier” hidden_layers_sizes: text: [256, 128] number_of_transformer_layers: 0 weight_sparsity: 0 intent_classification: True entity_recognition: False use_masked_language_model: False BILOU_flag: False epochs: 60 random_seed: 550 drop_rate: 0.1 similarity_type: “cosine”

policies:

  • name: “KerasPolicy” epochs: 300 batch_size: 64 max_history: 4 random_seed: 2
  • name: “AugmentedMemoizationPolicy” max_history: 4
  • name: “FallbackPolicy” nlu_threshold: 0.7 core_threshold: 0.5 fallback_action_name: “action_default_fallback”
4

Hello @rhododendron

Have you tried using the default settings for the pipeline instead?

1 Like
5

Hey, @j.mosig Yes, it was pretty bad too.

Today I tried “EmbeddingIntentClassifier”, as it is still present in Rasa v1.10.10. It was better than using DIET with the config from above, BUT now, when I use “cosine” similarity, the NLU model gives almost identical probabilities for all the intents - about 3.33 ("ranking_length" is set to 3).

When I use “inner” for similarity, top intent’s confidence of the model grows noticeably (0.4 and higher), but still overall accuracy is not as good as it was in the old Rasa (0.14.5), although I use the same exact configs. What could it be?

6

Ok. A few more questions: :slight_smile:

  1. How many training stories, intents, entities, and examples per intent/entity do you have in your training data?
  2. What is the output of rasa data validate and rasa data validate stories --max-history X, where X is the max-history of your policy (X = 4 according to your config)?
  3. By how much do results change when you change the random seed a few times? If they change a lot, you simply never had enough / good training data and just got lucky with the old version.

If you really want to reproduce old EmbeddingIntentClassifier behaviour, the closest config should be

- name: DIETClassifier
  hidden_layers_sizes:
    text: [256, 128]
  number_of_transformer_layers: 0
  weight_sparsity: 0
  intent_classification: True
  entity_recognition: False
  use_masked_language_model: False
  BILOU_flag: False

which is what you are using it seems. So I’d count on the problem being what I describe in my third question.