Hello. I am creating my first rasa chatbot. It is a chatbot for teachers of a certain LMS. This bot handles about 150 faq intentions and was trained using more than 1500 examples. Also handles some chitchat.
I have designed the intentions names using ‘faq/…’ as described here
So I have these 2 main groups of intentions and some others. One faq intention can look like this:
- intent: faq/teacher__scores_history__int_0080
examples: |
- ¿Cómo puedo verificar los cambios que he realizado en las planillas de actividades?
- ¿Cómo validar que cambios he realizado?
- ¿Cómo miro mis cambios?
- ¿Por dondé miro mis cambios?
- ¿Dónde encuentro el historial de cambios?
- existe un módulo que guarde el historial de cambios?
- la plataforma guarda el historial de cambios de la planilla?
- el sistema almacena un log de modificaciones en las calificaciones?
In my rules.yml file I have these 2
- rule: Handle faq
steps:
- intent: faq
- action: utter_faq
- action: utter_did_that_help
- rule: Handle chitchat
steps:
- intent: chitchat
- action: utter_chitchat
Everything works just fine. But, sometimes, the user’s question may not be very clear and I would like to implement a customized fallback action which gives the user the 4 most similar questions or the possibility to rephrase. I have named this action ‘show_higest_confidence_intents’ and I have added it to the actions section in the domain
version: "3.1"
session_config:
session_expiration_time: 60
carry_over_slots_to_new_session: true
actions:
- greet
- goodbye
- show_higest_confidence_intents
- utter_faq
- utter_chitchat
and I have also added show_higest_confidence_intents to the rules file
- rule: "Ask the user to rephrase or give best options when there is low NLU confidence"
steps:
- intent: nlu_fallback
- action: show_higest_confidence_intents
Then, I added the fallback classifier in the config.yml file as described here:
# The config recipe.
# https://rasa.com/docs/rasa/model-configuration/
recipe: default.v1
# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/
language: es # es
pipeline:
# # No configuration for the NLU pipeline was provided. The following default pipeline was used to train your model.
# # If you'd like to customize it, uncomment and adjust the pipeline.
# # See https://rasa.com/docs/rasa/tuning-your-model for more information.
- name: WhitespaceTokenizer
- name: RegexFeaturizer
- name: LexicalSyntacticFeaturizer
- name: CountVectorsFeaturizer
- name: CountVectorsFeaturizer
analyzer: char_wb
min_ngram: 1
max_ngram: 4
- name: DIETClassifier
epochs: 100
constrain_similarities: true
- name: EntitySynonymMapper
- name: ResponseSelector
epochs: 100
constrain_similarities: true
- name: FallbackClassifier
threshold: 0.7
core_threshold: 0.8
ambiguity_threshold: 0.05
fallback_action_name: show_higest_confidence_intents
# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
# # No configuration for policies was provided. The following default policies were used to train your model.
# # If you'd like to customize them, uncomment and adjust the policies.
# # See https://rasa.com/docs/rasa/policies for more information.
- name: MemoizationPolicy
- name: UnexpecTEDIntentPolicy
max_history: 5
epochs: 100
- name: TEDPolicy
max_history: 5
epochs: 100
constrain_similarities: true
- name: RulePolicy
core_fallback_threshold: 0.3
core_fallback_action_name: "show_higest_confidence_intents"
enable_fallback_prediction: True
The problem: Every time I shot a weird question, rasa identifies it is a ‘faq’ intent with 100% of confidence (1.0) which is right BUT… inside, the choosen specific FAQ intention has a very low confidence (< 30%) and thus, the custom fallback action function never triggers.
What I mean is… identifying certain question as a FAQ, may be correct, but RASA should have not uttered the answer because the specific intention had a very low confidence and was not what the user wanted.
"intent_ranking": [
{
"name": "faq",
"confidence": 1.0
},
{
"name": "goodbye",
"confidence": 1.8199880202278335e-25
},
{
"name": "greet",
"confidence": 9.676360764584161e-26
}
],
"response_selector": {
"all_retrieval_intents": [
"chitchat",
"faq"
],
"default": {
"response": {
"responses": [
{
"text": "hahahaha =)"
},
],
"confidence": 0.2529321610927582,
"intent_response_key": "chitchat/int_laughing",
"utter_action": "utter_chitchat/int_laughing"
},
"ranking": [
{
"confidence": 0.2529321610927582,
"intent_response_key": "chitchat/int_laughing"
},
{
"confidence": 0.11360147595405579,
"intent_response_key": "chitchat/int_how_are_you"
},
{
"confidence": 0.05717119574546814,
"intent_response_key": "chitchat/int_feeling_neutral"
},
{
"confidence": 0.03046223893761635,
"intent_response_key": "chitchat/int_where_do_you_work"
}
]
}
}
In addition, I am not sure if I am mistaking anything in the config, domain or rules files above.
What I need is similar to this contribution from @koaning.
How can I implement fallback classifier inside the faq ‘sub intentions’ so when faq confidence is very high, but the specific intention confidence is low, the fallback action code is triggered?
Any help would be much appreciated.
Thanks.
