Error in intent classification - Without slots or entities

Rasa Open Source
1

Hey. My query is wrt incorrect intent classification.

My stories.md file is

story_unmarried couple

  • greet
    • utter_greet
  • unmarried_couple
    • utter_unmarried_couple
  • goodbye
    • utter_goodbye

story_swimming pool

  • greet
    • utter_greet
  • swimming_pool
    • utter_swimming_pool
  • goodbye
    • utter_goodbye

story_checkin time

  • greet
    • utter_greet
  • checkin_time
    • utter_checkin_time
  • goodbye
    • utter_goodbye

story_greet

  • greet
    • utter_greet
  • goodbye
    • utter_goodbye

story_goodbye

  • greet
    • utter_greet
  • goodbye
    • utter_goodbye

and my nlu.md is

intent:unmarried_couple

  • Do you allow unmarried couples?
  • Are unmarried couples allowed?
  • Do you allow couples with valid ID?
  • Is your hotel couple friendly?
  • Can I stay with my girlfriend if we are not married?

intent:swimming_pool

  • Do you have a swimming pool?
  • Is there a swimming pool in the hotel?
  • Is there a pool in your hotel?
  • Swimming pool?
  • Can I use swimming pool in your hotel?

intent:greet

  • hey
  • hi
  • hello

intent:goodbye

  • bye
  • goodbye
  • haveaniceday
  • seeyou

My config.yml is

language: en

pipeline:

  • name: WhitespaceTokenizer
  • name: LexicalSyntacticFeaturizer
  • name: CountVectorsFeaturizer
  • name: CountVectorsFeaturizer analyzer: “char_wb” min_ngram: 1 max_ngram: 4
  • name: DIETClassifier epochs: 100

policies:

  • name: MemoizationPolicy
  • name: TEDPolicy
  • name: MappingPolicy
  • name: FallbackPolicy nlu_threshold: 0.8 fallback_action_name: ‘utter_default_fallback’

Now, if I start the conversation by asking Smoking?, it gets detected as unmarried_couple intent with a confidence of 0.998. How is that possible?

P.S. I used to have a separate intent and corresponding for smoking but I removed it and then retrained the model. The intent and story have been removed

Smoking intent :

story_smoking

  • greet
    • utter_greet
  • smoking
    • utter_smoking
  • goodbye
    • utter_goodbye

Smoking story :

story_smoking

  • greet
    • utter_greet
  • smoking
    • utter_smoking
  • goodbye
    • utter_goodbye

To reproduce it, add the smoking intent and story, retrain the model and then remove it and retrain again. The idea behaviour should be that entering the text “Smoking” triggers the fallback but thats not the case here.

PPS. I am initiating a rasa shell after each retraining so the tracker state is cleared.

BTW @Juste I am a huge fan :slight_smile:

2

Hi @dociledevil. Thank you so much and it’s great to have you in Rasa community! :slight_smile:

Thats sounds like an issue with NLU for sure. Just to be clear - intents like checkin_time and smoking are in your training data, right?

3

Yeah. They were in my training data. When I trained my model again after removing them and then retried entering the intent, I was expecting rasa going to fallback. But, that did not happen. What did happen was that Rasa’s NLU predicted another intent(totally unrelated - swimming pool) with a confidence of 0.99.

Also, I would like to add that rasa shell is picking the latest trained model (as it should) from models/

4

Hey @Juste. Can you please look into this. I am stuck horribly and no matter what classifier I am using, I am getting bad predictions for this training data. Whats more alarming is the fact that if I train on the same dataset and config multiple times, I get confidence scores that vary greatly.