Mood happy, mood unhappy stay in the system even when they're not in domain/nlu

I am working on Rasa 2.4. The use case is a chatbot that is handling intents and sub intents. I’ve removed the default rasa stories, responses and intents. Yet the system is classifying it as mood unhappy sometimes. I’ve tried working on different systems and deleted all temp files yet the issue persists. Please find the output, nlu, config etc below.

I need help removing those intents from the system and instead triggering fallback.

Output:

Your input> → Hello Hi. I am Sevak at your service. May I know your name please? Your input → Rahul How may I help you today Rahul? Your input → I want food I’m sorry, I didn’t quite understand that. Could you rephrase? Your input → what is cdc? Change Data Capture; It is a tool by IBM. Replicate data from source to target and vice-versa. Your input → what is nested jobplan Jobplan ID which is mapped in Nested Jobplan ID field against the task, which has to be performed by the user. Your input → what colour dress are you wearing? > d:\anaconda\envs\sevakiaf\lib\site-packages\rasa\shared\utils\io.py:96: UserWarning: Interpreter parsed an intent ‘mood_unhappy’ which is not defined in the domain. Please make sure all intents are listed in the domain. More info at Domain Bye, Have a nice day

Output: <rasa shell --debug>

> Your input ->  how is the weather today?
> 2021-04-27 13:19:38 DEBUG    rasa.core.lock_store  - Issuing ticket for conversation 'fa39d6f36a2e4101b8633a12ba6a3adb'.
> 2021-04-27 13:19:38 DEBUG    rasa.core.lock_store  - Acquiring lock for conversation 'fa39d6f36a2e4101b8633a12ba6a3adb'.
> 2021-04-27 13:19:38 DEBUG    rasa.core.lock_store  - Acquired lock for conversation 'fa39d6f36a2e4101b8633a12ba6a3adb'.
> 2021-04-27 13:19:38 DEBUG    rasa.core.tracker_store  - Recreating tracker for id 'fa39d6f36a2e4101b8633a12ba6a3adb'
> 2021-04-27 13:19:38 DEBUG    rasa.nlu.selectors.response_selector  - Adding following selector key to message property: faq
> 2021-04-27 13:19:38 DEBUG    ***rasa.core.processor  - Received user message 'how is the weather today?' with intent '{'id': 1248384363710024904, 'name': 'mood_unhappy', 'confidence': 0.30179139971733093}' and entities '[]'***
> d:\anaconda\envs\sevakiaf\lib\site-packages\rasa\shared\utils\io.py:96: UserWarning: Interpreter parsed an intent 'mood_unhappy' which is not defined in the domain. Please make sure all intents are listed in the domain.
>   More info at https://rasa.com/docs/rasa/domain
> 2021-04-27 13:19:38 DEBUG    rasa.core.processor  - Logged UserUtterance - tracker now has 14 events.
> 2021-04-27 13:19:38 DEBUG    rasa.core.policies.memoization  - Current tracker state:
> [state 0] user intent: faq | previous action name: action_listen
> [state 1] user intent: faq | previous action name: utter_faq
> [state 2] user intent: deny | previous action name: action_listen
> [state 3] user intent: deny | previous action name: utter_goodbye
> [state 4] user intent: mood_unhappy | previous action name: action_listen
> 2021-04-27 13:19:38 DEBUG    rasa.core.policies.memoization  - There is no memorised next action
> 2021-04-27 13:19:38 DEBUG    rasa.core.policies.rule_policy  - Current tracker state:
> [state 1] user intent: faq | previous action name: action_listen
> [state 2] user intent: faq | previous action name: utter_faq
> [state 3] user intent: deny | previous action name: action_listen
> [state 4] user intent: deny | previous action name: utter_goodbye
> [state 5] user text: how is the weather today? | previous action name: action_listen
> 2021-04-27 13:19:38 DEBUG    rasa.core.policies.rule_policy  - There is no applicable rule.
> 2021-04-27 13:19:38 DEBUG    rasa.core.policies.rule_policy  - Current tracker state:
> [state 1] user intent: faq | previous action name: action_listen
> [state 2] user intent: faq | previous action name: utter_faq
> [state 3] user intent: deny | previous action name: action_listen
> [state 4] user intent: deny | previous action name: utter_goodbye
> [state 5] user intent: mood_unhappy | previous action name: action_listen
> 2021-04-27 13:19:38 DEBUG    rasa.core.policies.rule_policy  - There is no applicable rule.
> 2021-04-27 13:19:38 DEBUG    rasa.core.policies.ensemble  - Made prediction using user intent.
> 2021-04-27 13:19:38 DEBUG    rasa.core.policies.ensemble  - Added `DefinePrevUserUtteredFeaturization(False)` event.
> 2021-04-27 13:19:38 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_1_TEDPolicy.
> 2021-04-27 13:19:38 DEBUG    rasa.core.processor  - Predicted next action 'utter_goodbye' with confidence 0.61.
> 2021-04-27 13:19:38 DEBUG    rasa.core.processor  - Policy prediction ended with events '[<rasa.shared.core.events.DefinePrevUserUtteredFeaturization object at 0x000001FD0894C948>]'.
> 2021-04-27 13:19:38 DEBUG    rasa.core.processor  - Action 'utter_goodbye' ended with events '[BotUttered('Bye, Have a nice day', {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, {"utter_action": "utter_goodbye"}, 1619509778.8535676)]'.
> 2021-04-27 13:19:38 DEBUG    rasa.core.policies.memoization  - Current tracker state:
> [state 0] user intent: faq | previous action name: utter_faq
> [state 1] user intent: deny | previous action name: action_listen
> [state 2] user intent: deny | previous action name: utter_goodbye
> [state 3] user intent: mood_unhappy | previous action name: action_listen
> [state 4] user intent: mood_unhappy | previous action name: utter_goodbye
> 2021-04-27 13:19:38 DEBUG    rasa.core.policies.memoization  - There is no memorised next action
> 2021-04-27 13:19:38 DEBUG    rasa.core.policies.rule_policy  - Current tracker state:
> [state 1] user intent: faq | previous action name: action_listen
> [state 2] user intent: faq | previous action name: utter_faq
> [state 3] user intent: deny | previous action name: action_listen
> [state 4] user intent: deny | previous action name: utter_goodbye
> [state 5] user intent: mood_unhappy | previous action name: action_listen
> [state 6] user intent: mood_unhappy | previous action name: utter_goodbye
> 2021-04-27 13:19:38 DEBUG    rasa.core.policies.rule_policy  - There is no applicable rule.
> 2021-04-27 13:19:38 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_1_TEDPolicy.
> 2021-04-27 13:19:38 DEBUG    rasa.core.processor  - Predicted next action 'action_listen' with confidence 0.99.
> 2021-04-27 13:19:38 DEBUG    rasa.core.processor  - Policy prediction ended with events '[]'.
> 2021-04-27 13:19:38 DEBUG    rasa.core.processor  - Action 'action_listen' ended with events '[]'.
> 2021-04-27 13:19:38 DEBUG    rasa.core.lock_store  - Deleted lock for conversation 'fa39d6f36a2e4101b8633a12ba6a3adb'.
> Bye, Have a nice day

Domain:

> intents:
> 
>   - greet
> 
>   - goodbye
> 
>   - affirm
> 
>   - deny
> 
>   - bot_challenge
> 
>   - got_solution
> 
>   - help_categories
> 
>   - introduction:
> 
>      use_entities:
> 
>        - name
> 
>   - faq
> 
> entities:
> 
>   - name
> 
> slots:
> 
>   name:
> 
>     type: text
> 
>     # this slot will not influence the predictions
> 
>     # of the dialogue policies
> 
>     influence_conversation: false    
> 
> actions:
> 
>   - utter_greet
> 
>   - utter_how_can_i_help
> 
>   - utter_how_can_i_help_no_name
> 
>   - utter_please_rephrase
> 
>   - utter_did_that_help
> 
>   - utter_can_i_do_anything_more
> 
>   - utter_ask_query
> 
>   - utter_goodbye
> 
>   - utter_iamabot
> 
>   - utter_introduction
> 
>   - utter_not_got_solution
> 
>  
> 
> forms:
> 
>   intro_form:
> 
>     name:
> 
>     - type: from_entity
> 
>       entity: name
> 
>       intent: introduction
> 
>  
> 
> responses:
> 
>   utter_default:
> 
>   - text: "Sorry I didn't get that. Can you put it in different words?"
> 
>   utter_greet:
> 
>   - text: "Hi. I am Sevak at your service. May I know your name please?"
> 
> # How can I help
> 
>   utter_how_can_i_help:
> 
>   - text: "How may I help you today {name}?"
> 
>     utter_how_can_i_help_no_name:
> 
>   - text: "Very well. How may I help you today?"
> 
>   utter_please_rephrase:
> 
>   - text: I'm sorry, I didn't quite understand that. Could you rephrase?
> 
>   
> 
>   utter_did_that_help:
> 
>   - text: "Did that help you?"
> 
> # Anything more to assist
> 
>   utter_can_i_do_anything_more:
> 
>   - text: Can I help you with something else?
> 
>   - text: Is there any other query that I can help you with?
> 
>   utter_ask_query:
> 
>   - text: "Please, state your query"
> 
>   - text: "Please, state your question"
> 
>   utter_goodbye:
> 
>   - text: "Bye, Have a nice day"
> 
>   utter_iamabot:
> 
>   - text: "I am a bot, powered by Rasa."
> 
>  
> 
>   utter_help_categories:
> 
>   - text: "I can help you in over hundreds of queries. They can be related to Issues & Returns, Common Data Management, Item information, tools, assets, DC RDC queries, extension and anticipation, work order concession, flight log book, dr/pwr, jobplan, snags, reports, model (cm), weekly training plans, preventive maintainance among others."
> 
> # greeting and progress to the person's purpose
> 
>   utter_introduction:
> 
>   - text: "Pleased to be acquainted with you {name}."
> 
>   utter_not_got_solution:
> 
>   - text: "I am sorry, that is the extent of my knowledge. If you need more assistance please contact a human representative."
> 
> + around 220 sub intents responses of faq

nlu:

version: “2.0”

nlu:

  • intent: greet

    examples: |

    • hey

    • hello

    • hi

    • hello there

    • good morning

    • good evening

    • moin

    • hey there

    • let’s go

    • hey dude

    • goodmorning

    • goodevening

    • good afternoon

  • intent: goodbye

    examples: |

    • good afternoon

    • cu

    • good by

    • cee you later

    • good night

    • bye

    • goodbye

    • have a nice day

    • see you around

    • bye bye

    • see you later

  • intent: affirm

    examples: |

    • yes

    • y

    • indeed

    • of course

    • that sounds good

    • correct

  • intent: deny

    examples: |

    • no

    • n

    • never

    • I don’t think so

    • don’t like that

    • no way

    • not really

    • i would rather not disclose

    • not really

  • intent: bot_challenge

    examples: |

    • are you a bot?

    • are you a human?

    • am I talking to a bot?

    • am I talking to a human?

  • intent: got_solution

    examples: |

    • Sure… Got it.

    • I understand.

    • Thanks.

    • Ok

    • okay thanks

    • cool

    • that was helpful

  • intent: not_got_solution

    examples: |

    • That does not answer my question.

    • It is still not clear

    • I don’t understand

    • Can you elaborate more?

    • What do you mean by that?

  • intent: help_categories

    examples: |

    • Please help me understand, what all can you tell me about?

    • What are the things I can ask you?

    • Can you tell me what topics you are informed about?

    • What are the ways in which you can help me?

    • How can you help me?

  • intent: introduction

    examples: |

  • around 220 sub intents of FAQ

config: > # Configuration for Rasa NLU.

    > 
    > # https://rasa.com/docs/rasa/nlu/components/
    > 
    > language: en
    > 
    > pipeline:
    > 
    > # No configuration for the NLU pipeline was provided. The following default pipeline was used to train your model.
    > 
    > # If you'd like to customize it, uncomment and adjust the pipeline.
    > 
    > # See https://rasa.com/docs/rasa/tuning-your-model for more information.
    > 
    >   - name: WhitespaceTokenizer
    > 
    >   # - name: RegexFeaturizer
    > 
    >   # - name: LexicalSyntacticFeaturizer
    > 
    >   - name: LanguageModelFeaturizer
    > 
    >     model_name: "bert"
    > 
    >     model_weights: "rasa/LaBSE"
    > 
    >     cache_dir: null
    > 
    >   # - name: CountVectorsFeaturizer
    > 
    >   # - name: CountVectorsFeaturizer
    > 
    >     # analyzer: char_wb
    > 
    >     # min_ngram: 1
    > 
    >     # max_ngram: 4
    > 
    >   - name: DIETClassifier
    > 
    >     epochs: 150
    > 
    >     constrain_similarities: true
    > 
    >   - name: EntitySynonymMapper
    > 
    >   - name: ResponseSelector
    > 
    >     epochs: 100
    > 
    >     retrieval_intent: faq
    > 
    >     constrain_similarities: true
    > 
    >   - name: FallbackClassifier
    > 
    >     threshold: 0.3
    > 
    >     ambiguity_threshold: 0.3
    > 
    > # Configuration for Rasa Core.
    > 
    > # https://rasa.com/docs/rasa/core/policies/
    > 
    > policies:
    > 
    > # No configuration for policies was provided. The following default policies were used to train your model.
    > 
    > # If you'd like to customize them, uncomment and adjust the policies.
    > 
    > # See https://rasa.com/docs/rasa/policies for more information.
    > 
    >   - name: MemoizationPolicy
    > 
    >   - name: TEDPolicy
    > 
    >     max_history: 5
    > 
    >     epochs: 150
    > 
    >     constrain_similarities: true
    > 
    >   - name: RulePolicy
    > 
    >     core_fallback_threshold: 0.25
    > 
    >     core_fallback_action_name: "action_default_fallback"
    > 
    >     enable_fallback_prediction: True

rules

> 
> version: "2.0"
> 
> rules:
> 
> # - rule: Say goodbye anytime the user says goodbye
> 
>   # steps:
> 
>   # - intent: goodbye
> 
>   # - action: utter_goodbye
> 
> - rule: Say 'I am a bot' anytime the user challenges
> 
>   steps:
> 
>   - intent: bot_challenge
> 
>   - action: utter_iamabot
> 
> - rule: what are the things you can help me with
> 
>   steps:
> 
>   - intent: help_categories
> 
>   - action: utter_help_categories
> 
> # - rule: respond to FAQs
> 
>   # steps:
> 
>   # - intent: faq
> 
>   # - action: utter_faq
> 
> # - rule: when someone tells their name
> 
>   # steps:
> 
>   # - intent: introduction
> 
>   # - action: intro_form
> 
>   # - action: utter_how_can_i_help
> 
> - rule: Ask the user to rephrase whenever they send a message with low NLU confidence
> 
>   steps:
> 
>   - intent: nlu_fallback
> 
>   - action: utter_please_rephrase
> 
>   # some other fallback option
> 
>   # additional rule in intent

stories:

stories
version: "2.0"
  • story: faq with chit chat

    steps:

    • intent: greet

    • action: utter_greet

    • intent: introduction

      entities:

      • name : “Douglas”
    • action: intro_form

    • action: utter_how_can_i_help

    • intent: faq

    • action: utter_faq

    • intent: got_solution

    • action: utter_can_i_do_anything_more

    • intent: goodbye

    • action: utter_goodbye

  • story: faq with chit chat name deny

    steps:

    • intent: greet

    • action: utter_greet

    • intent: deny

    • action: utter_how_can_i_help_no_name

    • intent: faq

    • action: utter_faq

    • intent: got_solution

    • action: utter_can_i_do_anything_more

    • intent: deny

    • action: utter_goodbye

  • story: faq with chit chat name deny not got solution

    steps:

    • intent: greet

    • action: utter_greet

    • intent: deny

    • action: utter_how_can_i_help_no_name

    • intent: faq

    • action: utter_faq

    • intent: not_got_solution

    • action: utter_not_got_solution

    • action: utter_can_i_do_anything_more

    • intent: deny

    • action: utter_goodbye

  • story: faq without greet

    steps:

    • intent: introduction

      entities:

      • name : “Douglas”
    • action: intro_form

    • action: utter_how_can_i_help

    • intent: faq

    • action: utter_faq

    • intent: got_solution

    • action: utter_can_i_do_anything_more

    • intent: goodbye

    • action: utter_goodbye

  • story: faq without greet intent affirm

    steps:

    • intent: introduction

      entities:

      • name : “Douglas”
    • action: intro_form

    • action: utter_how_can_i_help

    • intent: faq

    • action: utter_faq

    • intent: got_solution

    • action: utter_can_i_do_anything_more

    • intent: affirm

    • intent: faq

    • action: utter_faq

    • intent: goodbye

    • action: utter_goodbye

  • story: faq without name

    steps:

    • intent: greet

    • action: utter_greet

    • intent: faq

    • action: utter_faq

    • intent: got_solution

    • action: utter_can_i_do_anything_more

    • intent: affirm

    • intent: goodbye

    • action: utter_goodbye

  • story: faq without name and utter_can_i_do_anything_more

    steps:

    • intent: greet

    • action: utter_greet

    • intent: faq

    • action: utter_faq

    • intent: goodbye

    • action: utter_goodbye

  • story: faq without utter_can_i_do_anything_more

    steps:

    • intent: greet

    • action: utter_greet

    • intent: introduction

      entities:

      • name : “Douglas”
    • action: intro_form

    • action: utter_how_can_i_help

    • intent: faq

    • action: utter_faq

    • intent: goodbye

    • action: utter_goodbye

  • story: faq with introduction

    steps:

    • intent: greet

    • action: utter_greet

    • intent: introduction

      entities:

      • name : “Douglas”
    • action: intro_form

    • action: utter_how_can_i_help

  • story: faq without bye

    steps:

    • intent: greet

    • action: utter_greet

    • intent: introduction

      entities:

      • name : “Douglas”
    • action: intro_form

    • action: utter_how_can_i_help

    • intent: faq

    • action: utter_faq

    • intent: got_solution

    • action: utter_can_i_do_anything_more

  • story: faq with introduction only

    steps:

    • intent: introduction

      entities:

      • name : “Douglas”
    • action: intro_form

    • action: utter_how_can_i_help

  • story: faq with introduction and bye

    steps:

    • intent: introduction

      entities:

      • name : “Douglas”
    • action: intro_form

    • action: utter_how_can_i_help

    • intent: faq

    • action: utter_faq

    • intent: goodbye

    • action: utter_goodbye

  • story: faq with got_solution and affirm

    steps:

    • intent: got_solution

    • action: utter_can_i_do_anything_more

    • intent: affirm

    • action: utter_ask_query

    • intent: faq

    • action: utter_faq

    • intent: goodbye

    • action: utter_goodbye

  • story: faq with got_solution and deny

    steps:

    • intent: got_solution

    • action: utter_can_i_do_anything_more

    • intent: deny

    • action: utter_goodbye

  • story: faq questions

    steps:

    • intent: faq

    • action: utter_faq

  • story: faq questions with bye

    steps:

    • intent: faq

    • action: utter_faq

    • intent: goodbye

    • action: utter_goodbye

  • story: faq questions with bye help categories

    steps:

    • intent: help_categories

    • action: utter_help_categories

    • intent: faq

    • action: utter_faq

    • intent: goodbye

    • action: utter_goodbye

  • story: faq with chit chat help categories

    steps:

    • intent: greet

    • action: utter_greet

    • intent: introduction

      entities:

      • name : “Douglas”
    • action: intro_form

    • action: utter_how_can_i_help

    • intent: help_categories

    • action: utter_help_categories

    • intent: faq

    • action: utter_faq

    • intent: got_solution

    • action: utter_can_i_do_anything_more

    • intent: goodbye

    • action: utter_goodbye

Hi @Snorlaxcode

Have you retrained the model? The domain is stored with a trained model, so changing the file won’t have an affect until retraining.

Hi Sorry for the late response. I’ve retrained the model multiple times, created new workspace and even deleted all temp files (windows app data). It still shows the same issues. I started a rasa x instance just now and I got the error again.

> Starting Rasa X in local mode... 🚀
> The following intents were found in your stories and will be added to the domain: nlu_fallback
> c:\users\ayush\.conda\envs\rasa1\lib\site-packages\rasa\shared\utils\io.py:96: UserWarning: Issue found in 'C:\Users\Ayush\AppData\Local\Temp\tmp4l9fqzey': 
> Found intent 'mood_great' in stories which is not part of the domain.
>   More info at https://rasa.com/docs/rasa/stories
> c:\users\ayush\.conda\envs\rasa1\lib\site-packages\rasa\shared\utils\io.py:96: UserWarning: Issue found in 'C:\Users\Ayush\AppData\Local\Temp\tmp4l9fqzey': 
> Found intent 'mood_unhappy' in stories which is not part of the domain.
>   More info at https://rasa.com/docs/rasa/stories
> The following intents were found in your NLU training data and will be added to the domain: mood_unhappy, mood_great
> 2021-05-05 13:23:35 WARNING  rasax.community.services.insights.insight_service  - No local evaluation result files found.
> [2021-05-05 13:23:46 +0530] [12204] [INFO] Goin' Fast @ http://0.0.0.0:5002
> 2021-05-05 13:23:46 INFO     sanic.root  - Goin' Fast @ http://0.0.0.0:5002
> 
> The server is running at http://localhost:5002/login?username=me&password=tOPnxmE3iJ0j
> 
> [2021-05-05 13:23:46 +0530] [12204] [INFO] Starting worker [12204]
> 2021-05-05 13:23:46 INFO     sanic.root  - Starting worker [12204]
> c:\users\ayush\.conda\envs\rasa1\lib\site-packages\rasa\utils\train_utils.py:535: UserWarning: model_confidence is set to `softmax`. It is recommended to try using `model_confidence=linear_norm` to make it easier to tune fallback thresholds.
>   category=UserWarning,
> WARNING:rasax.community.services.event_consumers.event_consumer:Saving event failed due to an 'IntegrityError'. This means that the event is already stored in the database. The 
> event data was '{"sender_id": "3c29c21103f94eb39b6ca51bcbc2c70c", "event": "action", "timestamp": 1620201334.0573633, "name": "action_listen", "policy": null, "confidence": null, "action_text": null}'. (sqlite3.IntegrityError) UNIQUE constraint failed: conversation_event.conversation_id, conversation_event.timestamp, conversation_event.type_name       
> [SQL: INSERT INTO conversation_event (conversation_id, type_name, timestamp, intent_name, action_name, slot_name, slot_value, policy, is_flagged, data, evaluation, rasa_environment) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]
> [parameters: ('3c29c21103f94eb39b6ca51bcbc2c70c', 'action', 1620201334.0573633, None, 'action_listen', None, None, None, 0, '{"sender_id": "3c29c21103f94eb39b6ca51bcbc2c70c", "event": "action", "timestamp": 1620201334.0573633, "name": "action_listen", "policy": null, "confidence": null, "action_text": null}', None, 'production')]
> (Background on this error at: http://sqlalche.me/e/13/gkpj)