NLU_Fallback is responding when it shouldn't

Rasa Open Source
1

Hey there,

I am having a bit of an issue. I get “nlu_fallback” as primary response even if the correct respons is getting better confidence.

NLU model loaded. Type a message and press enter to parse it. Next message: hej { “text”: “hej”, “intent”: { “name”: “nlu_fallback”, “confidence”: 0.45817649364471436 }, “entities”: [], “intent_ranking”: [ { “name”: “nlu_fallback”, “confidence”: 0.45817649364471436 }, { “id”: 8107020950297172155, “name”: “greet”, “confidence”: 0.5418235063552856 },

I know that my threshold is probably too high (0.2). But I wonder how I could get this simple task so wrong. Any ideas?

Thanks, /Magnus

=====

My config file below.

language: “sv”

pipeline:

  • name: SpacyNLP model: sv_pipeline
  • name: SpacyTokenizer intent_tokenization_flag: False token_pattern: None
  • name: SpacyFeaturizer
  • name: RegexFeaturizer
  • name: LexicalSyntacticFeaturizer features: - [low, title, upper, suffix2] # features for the word preceding the word being evaluated - [EOS, title, suffix5] # features for the word being evaluated - [prefix2] # features for the word following the word being evaluated
  • name: CountVectorsFeaturizer
  • name: CountVectorsFeaturizer analyzer: “char_wb” min_ngram: 1 max_ngram: 4
  • name: RegexEntityExtractor
  • name: DIETClassifier epochs: 100 embedding_dimension: 30 number_of_transformer_layers: 4 entity_recognition: False model_confidence: linear_norm constrain_similarities: True use_masked_language_model: True ranking_length: 5
  • name: EntitySynonymMapper
  • name: ResponseSelector epochs: 200 retrieval_intent: faq
  • name: ResponseSelector retrieval_intent: chitchat scale_loss: false epochs: 100
  • name: FallbackClassifier threshold: 0.7

policies:

  • name: MemoizationPolicy max_history: 3
  • name: TEDPolicy epochs: 100 max_history: 5 constrain_similarities: True model_confidence : linear_norm
  • name: RulePolicy core_fallback_threshold: 0.2 core_fallback_action_name: action_default_fallback enable_fallback_prediction: True
2

Hello there,

What I found useful while determining the fallback threshold was to run cross-validation, get misclassified examples and look at their confidences, and find a sweet spot. Also think tf-idf, make your intents distinct so that you get rid of the ambiguity. On the other hand, anything classified below the fallback threshold (which is 0.7 in your case) will fall into fallback, no matter what.

1 Like
4

I use to have intent “greet” as rule before but moved it to a story… not sure it was a good idea but cross validation I should try. Thanks.

5

As long as it’s in a story and your bot can classify it right, it should work, from what I see, the problem is your fallback threshold being too high.

6

any suggestion on what I should have? 0.3?

7

It depends on your data, you should run a cross-validation to see what is the confidence that your model misclassifies the intents, but if you don’t want it, default value is 0.3, you can go with that.

1 Like