Predicted intent is always chitchat

I added few intents and chitchat in my training data and trained the bot. When I started testing the bot, other than the defined intents in nlu.yml file, any other input whether unknown or gibberish or some random words will get predicted as chitchat intent with confidence as 1.0 or 0.99. I even added two stage fall back logic in config.yml, rules.yml and custom actions. For fallback to happen, prediction confidence has to be lower than 1.0 or 0.99.

Never encountered such situation before. Even with the smaller training data I have never got this result.

Hi, could you please share your Rasa version and YAML files (config.yml, domain.yml, nlu.yml, rules.yml) that lead to the issue?

Please post the actual code/logs/stacktrace text into the channel instead of the screenshots. That way we can more easily copy/paste the code and help you debug. You can use markdown syntax ``` to create code blocks.


version: "2.0"
- story: chitchat
  - intent: chitchat
  - action: utter_chitchat

- story: more chitchat
  - intent: greet
  - action: utter_greet
  - intent: chitchat
  - action: utter_chitchat
  - intent: chitchat
  - action: utter_chitchat
- story: greet chitchat
  - intent: greet
  - action: utter_greet
  - intent: chitchat
  - action: utter_chitchat
- story: greet only
  - intent: greet
  - action: utter_greet


version: "2.0"
- intent: greet
  examples: |
    - Hi
    - Hello
    - Hi
    - hello 
    - hi 
    - heeeeeeellllllllooooooooooooo
    - hai
    - Hai 
    - Hi How are you?
    - how are you?
    - how are you
    - How are you? 
    - How are you  
    - I am doing good, how are you? 
    - I am doing good how are you? 
    - I am doing good how are you
    - I am doing good, How are you? 


language: en
  - name: SpacyNLP
    model: "en_core_web_md"
  - name: SpacyTokenizer
  - name: SpacyEntityExtractor
  - name: SpacyFeaturizer
    pooling: mean
  - name: CountVectorsFeaturizer
    analyzer: char_wb
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 64
    constrain_similarities: true
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 64
    constrain_similarities: true
  - name: FallbackClassifier
    threshold: 0.3
    ambiguity_threshold: 0.1

- name: MemoizationPolicy
- name: RulePolicy
- name: TEDPolicy
  max_history: 5
  epochs: 20
  - 32
  - 64


version: "2.0"
- rule: to greet the customer
  - intent: greet
  - action: utter_greet

- rule: Implemtation of Two Stage FallBack 
  - intent: nlu_fallback
  - action: action_two_stage_fallback
  - active_loop: action_two_stage_fallback


version: '2.0'


store_entities_as_slots: true


session_expiration_time: 480

carry_over_slots_to_new_session: true


- affirm

- greet

- ask_how_contribute

- ask_question_in_forum

- ask_when_next_event

- ask_which_events

- ask_why_contribute

- book_demo

- broken

- bye

- canthelp

- chitchat:

use_entities: []

- contact_sales

- deny

- enter_data:

use_entities: []

- explain

- faq:


- product

- nlu_fallback

- trigger_response_selector:

use_entities: false

  - text: "Hi, How are you doing?"
  - text: "Its good to hear from you"
  - text: "Always looking forward to say you a Hi"
  - buttons:
    - payload: '/feedback{{"feedback_value": "positive"}}'
      title: πŸ‘
    - payload: '/feedback{{"feedback_value": "negative"}}'
      title: πŸ‘Ž
    text: How is this conversation going?
  - text: I'm sorry I couldn't help you. Please let me know if you have any suggestions for how I can improve.
  - text: Anything in particular you'd like to know about, e.g. intent classification or entity recognition?
  - text: If you're new to Rasa, I can help you get started! If you've already started building an assistant, you can also ask me about the different parts of Rasa 🐦
  - text: Did you encounter any problems during installation?
  - text: What's your job? πŸ•΄
  - text: Are you trying to install Rasa X on your local machine or on a server?
  - text: Are you migrating from another tool?
  - text: What's your name?
  - text: What's your name?

- action_default_ask_affirmation
- utter_chitchat
- utter_faq
- utter_out_of_scope
- utter_greet

I am using rasa V2.8.10. I wanted to start with a only one intent in the bot as there are more work in the custom actions at this point of the project. Basically this is for MVP.

Thanks for sharing the files. Is this your complete nlu.yml or did something accidentally get cut when copying it? It only contains training examples for one intent (greet), which for me leads to an error when training the pipeline:

2021-11-25 11:32:53 ERROR    rasa.nlu.classifiers.diet_classifier  - Cannot train 'DIETClassifier'. Need at least 2 different intent classes. Skipping training of classifier.

@MatthiasLeimeister Thanks for your reply. I also included chitchat.yml ( from demo bot ) in my training data.

Let me give a screenshot of the training:

Hi, I tried training with the chitchat.yml data and the nlu.yml you provided and saw a similar behaviour. The classifier looks very skewed towards the chitchat class. The reason could be that the training data contains far more chitchat examples, which cover a very diverse set of topics and the examples for greet are partly overlapping with the chitchat/ask_howdoing section. So anything that is not clearly greet will be classified as chitchat with high confidence.

This skew in performance can also be seen in the test output, running

rasa test nlu --nlu data/nlu/ --cross-validation --runs 1 --folds 2

I got the following intent results:

  "greet": {
    "precision": 0.5714285714285714,
    "recall": 0.75,
    "f1-score": 0.6486486486486486,
    "support": 16,
    "confused_with": {
      "chitchat": 4
  "chitchat": {
    "precision": 0.9950433705080545,
    "recall": 0.9889162561576355,
    "f1-score": 0.9919703520691785,
    "support": 812,
    "confused_with": {
      "greet": 9

Could you try using more balanced training data for the two intents and avoid overlapping examples?

was a solution ever found for this? I am facing a similar problem, where the fallback action won’t trigger because the confidence score is always 0.9+