I am currently working on a chatbot with four requirements for the FallbackPolicy:
- Make it buttonless (customer has to answer yes or no, buttons are not supported by the GUI).
- Policy suggests only one possible intent for the user. If he/she says it is not the correct intent, another form is triggered to ask whether the client would contact customer support.
- Suggested intent is “translated” (modified to a normal phrase instead of outputting the intent name in Rasa to the user).
- If the confidence of the prediction is extremely low, no possible intents is suggested. Instead the form for asking if he/she wants to contact customer support is triggered. This avoids extremely stupid situations.
Here I will explain how did I do it. Perhaps it gives you some ideas for your bot. I think it is generally good to share the knowledge. It is also great for me to come back one day when I am trying to figure out what did I do. This is done in Rasa version 3.0.6 and Python 3.8.10.
Examples of the use cases I covered are given at the end of this post.
Step 1. Add FallbackPolicy to the config.yml file.
pipeline:
- name: FallbackClassifier
threshold: 0.85
policies:
- name: RulePolicy
core_fallback_threshold: 0.8
The threshold in FallbackClassifier means that if the DIETClassifier is unsure of its prediction and the highest confidence is lower than 0.85, Rasa predicts intent “nlu_fallback” with confidence 0.85 to the user’s utterance. That is needed for triggering our next actions. I suggest you decide on the threshold by analyzing the confidence histogram.
Step 2. Add a rule for confidence check-up
When I have detected that confidence is lower than my threshold, I want to:
A) direct the customer to the customer support if the confidence is extremely low.
B) ask the customer if he/she meant the highest by confidence intent.
For both situations I have to trigger a form. So let’s add a rule that if nlu_fallback is detected (prediction score is lower than 0.85) we use an action that decides what’s next.
in rules.yml:
- rule: Low confidence is detected, activate next action
steps:
- intent: nlu_fallback
- action: action_check_confidence
in domain.yml:
actions:
- action_check_confidence
responses:
utter_not_confident:
- text: "Hmm... I am not sure I understood correctly."
utter_did_not_predict_correctly:
- text: "Hmm... I think I still have to study the language."
in folder actions/ a new file action_check_confidence.py is created:
from typing import Dict, Text, List, Any
from rasa_sdk import Tracker
from rasa_sdk.events import EventType, SlotSet, FollowupAction
from rasa_sdk.executor import CollectingDispatcher
from rasa_sdk import Action
class AskForSlotAction(Action):
"""Checks the real value of the confidence and triggers a next action"""
def name(self) -> Text:
return "action_check_confidence"
def __init__(self) -> None:
import csv
self.LOWEST_CONFIDENCE = 0.4
def run(self,
dispatcher: CollectingDispatcher,
tracker: Tracker,
domain: Dict[Text, Any]
) -> List['Event']:
intent_ranking = tracker.latest_message.get('intent_ranking', [])
if intent_ranking[0].get('name', '') == 'nlu_fallback':
intent_ranking = [intent_ranking[1]]
else:
intent_ranking = [intent_ranking[0]]
if intent_ranking[0].get("confidence", 0) < self.LOWEST_CONFIDENCE:
return [FollowupAction(name = "utter_did_not_predict_correctly"), SlotSet("affirm_deny",None)]
else:
return [FollowupAction(name = "utter_not_confident"), SlotSet("affirm_deny",None)]
return []
Here I have hard-coded the self.LOWEST_CONFIDENCE that should be chosen by analysing the confidence histogram. We take the latest message from the Tracker and take out the ranking of the prediction with highest confidence (they are in order in the Tracker).
If the confidence is lower than 0.4, we use a FollowupAction “utter_did_not_predict_correctly”. If it is higher (that means between 0.4-0.85) we use a FollowupAction “utter_not_confident”. I also make sure my slot “affirm_deny” is empty.
It’s important to use a FollowupAction, because regular utter_message is not considered as action and therefore we cannot use them in rules.
I am using these FolloupActions to trigger the suitable forms. Let’s continue with the situation B and come back to A later.
Step 3. Create a custom buttonless FallbackPolicy form
Creating Forms is nicely described in the documentation. I create a custom_fallback_form that asks to fill a slot “affirm_deny”. The slot is detected from a text and it has a condition that it is only used in the custom_fallback_form. Since I want to modify the question each time I have added an action for asking the value of the slot. It is triggered the same way as utter_ask_<form_name>_<slot_name> or utter_ask_<slot_name>. I add form name in the action name also, because later I want to use the same slot in another form.
slots:
affirm_deny:
type: text
mappings:
- type: from_text
conditions:
- active_loop: custom_fallback_form
requested_slot: affirm_deny
forms:
custom_fallback_form:
required_slots:
- affirm_deny
actions:
- action_ask_custom_fallback_form_affirm_deny
responses:
- utter_what_was_the_question_about:
- text: "Did you ask about the following topic: {intent}?"
The form is triggered when FollowupAction “utter_not_confident” is used in the previous action. rule.yml:
- rule: Activate custom fallback form
steps:
- action: utter_not_confident
- action: custom_fallback_form
- active_loop: custom_fallback_form
The action itself is as follows. I use a CSV file to translate the intent names into better format if it is represented there. I keep the file in a subfolder in actions folder. The action utters the response “utter_what_was_the_question_about” and fills it’s variable “intent”.
from typing import Dict, Text, List, Any
from rasa_sdk import Tracker
from rasa_sdk.events import EventType, SlotSet, FollowupAction
from rasa_sdk.executor import CollectingDispatcher
from rasa_sdk import Action
class AskForSlotAction(Action):
"""Asks for an affirmation of the intent if NLU threshold is not met."""
def name(self) -> Text:
return "action_ask_custom_fallback_form_affirm_deny"
def __init__(self) -> None:
import csv
self.intent_mappings = {}
with open('actions/action_files/intent_description_mapping.csv',
newline='',
encoding='utf-8') as file:
csv_reader = csv.reader(file)
for row in csv_reader:
self.intent_mappings[row[0]] = row[1]
def run(self,
dispatcher: CollectingDispatcher,
tracker: Tracker,
domain: Dict[Text, Any]
) -> List['Event']:
intent_ranking = tracker.latest_message.get('intent_ranking', [])
if intent_ranking[0].get('name', '') == 'nlu_fallback':
intent_ranking = [intent_ranking[1]]
else:
intent_ranking = [intent_ranking[0]]
first_intent_names = [intent.get('name', '')
for intent in intent_ranking
if intent.get('name', '') not in ['out_of_scope', 'nlu_fallback']]
mapped_intents = [(name, self.intent_mappings.get(name, name))
for name in first_intent_names]
dispatcher.utter_message(response="utter_what_was_the_question_about", intent=mapped_intents[0][1])
return []
example of the CSV file:
intent,button
greeting,I wanted to say hello
weather,asking about the weather
Great. Now the asks our fallback question in a suitable way. But what will we do with the answer (that is filled slot)?
Step 4. Check the content of the custom_fallback_policy slot
So far:
- User said something
- Rasa intent prediction confidence is between 0.4 and 0.85
- Rasa asks user if he/she meant this intent
- User answers (slot is filled and form is deactivated)
Let’s create the form’s deactivation rule and put an action in the end. This action will validate the user’s answer and choose to utter a response (user says yes) or activate a form for directing the user to the customer support (user doesn’t agree with the bot’s suggested intent).
rules.yml:
- rule: Deactivate custom fallback form
condition:
# Condition that form is active.
- active_loop: custom_fallback_form
steps:
# Form is deactivated
- action: custom_fallback_form
- active_loop: null
- slot_was_set:
- requested_slot: null
# The actions we want to run when the form is submitted.
- action: action_react_to_affirm_deny_in_custom_fallback_form
domain.yml:
actions:
- action_react_to_affirm_deny_in_custom_fallback_form
A new action in the actions folder is as follows. It takes examples from an existing intent “affirm” and matches them with the slot value. I have not shown the affirm example here. It’s a regular intent defined in domain.yml and with examples in a nlu.yml file.
If the value matches an example in the intent “affirm” list, the action utters “utter_”+intent name. It is important to notice that this action only works when the good intent creation is followed (response equals to “utter_”+intent name not to something else). This action also fails if user agrees, but the agreement is not matched to the example list. Another idea is to try to see if the intent “affirm” is predicted, but let’s keep that for some other time. This needs a thorough analysis of the intent prediction results.
If user doesn’t affirm, the FollowupAction utter_did_not_predict_correctly is triggered. Is this familiar? This is also triggered in our action_check_confidence action when the highest confidence is too low. We use this to trigger the directing to user customer form.
Another important thing is to notice that I am setting my slot back to None. If it stays filled, next time the Form won’t be activated. Form is only activated if the required slot is empty.
from typing import Dict, Text, List, Any
from rasa_sdk import Tracker
from rasa_sdk.events import EventType, SlotSet, FollowupAction
from rasa_sdk.executor import CollectingDispatcher
from rasa_sdk import Action
from .utils import get_intent_examples
class ActionDealWithAffirmationAnswer(Action):
"""Previously bot asked for an affirmation of the intent if NLU threshold is not met. Now we deal with it"""
def name(self) -> Text:
return "action_react_to_affirm_deny_in_custom_fallback_form"
def __init__(self) -> None:
self.affirm_list = get_intent_examples("affirm")
def run(self,
dispatcher: CollectingDispatcher,
tracker: Tracker,
domain: Dict[Text, Any]
) -> List['Event']:
affirmdeny_value = tracker.get_slot("affirm_deny")
events = tracker.events_after_latest_restart()[-25:]
events_with_utterances = list()
for event in events:
if "parse_data" in event.keys():
events_with_utterances.append(event)
if affirmdeny_value.lower() in self.affirm_list:
if events_with_utterances[-2]["parse_data"]["intent_ranking"][0]["name"] == "nlu_fallback":
response = "utter_"+events_with_utterances[-2]["parse_data"]["intent_ranking"][1]["name"]
else:
response = "utter_"+events_with_utterances[-2]["parse_data"]["intent_ranking"][0]["name"]
dispatcher.utter_message(response=response)
return [SlotSet("affirm_deny",None)]
else:
return [FollowupAction(name = "utter_did_not_predict_correctly"), SlotSet("affirm_deny",None)]
return [SlotSet("affirm_deny",None)]
Now we have used the FollowupAction “utter_did_not_predict_correctly” with the promise to trigger a form to direct the user to the customer support. Let’s do that form and then we’re done!
Step 5. Create a direct_to_customer_support form
rules.yml:
- rule: Activate directing to customer support form
steps:
- action: utter_did_not_predict_correctly
- action: direct_to_customer_support_form
- active_loop: direct_to_customer_support_form
domain.yml. I added a new condition to the slot, the form and (since I don’t need to have a customized utterance this time) the question for asking the value of the slot.
slots:
affirm_deny:
type: text
mappings:
- type: from_text
conditions:
- active_loop: custom_fallback_form
requested_slot: affirm_deny
- active_loop: direct_to_customer_support_form
requested_slot: affirm_deny
forms:
direct_to_customer_support_form:
required_slots:
- affirm_deny
responses:
utter_ask_direct_to_customer_support_form_affirm_deny:
- text: "I'm sorry I cannot answer. Would you like me to direct you to the customer support?"
Step 6. Check the content of the direct_to_customer_support_form slot
This is quite similar to step 4.
rules.yml:
- rule: Deactivate directing to customer support form
condition:
# Condition that form is active.
- active_loop: direct_to_customer_support_form
steps:
# Form is deactivated
- action: direct_to_customer_support_form
- active_loop: null
- slot_was_set:
- requested_slot: null
# The actions we want to run when the form is submitted.
- action: action_react_to_affirm_deny_in_direct_to_customer_support_form
domain.yml:
actions:
- action_react_to_affirm_deny_in_direct_to_customer_support_form
responses:
utter_direct_to_customer_support:
- text: "I am directing you to the customer support. Please wait a little."
utter_do_not_direct_to_customer_support:
- text: "Okay then. Have a nice day!"
utter_default:
- text: "Sorry, I cannot help."
new action:
from typing import Dict, Text, List, Any
from rasa_sdk import Tracker
from rasa_sdk.events import EventType, SlotSet, FollowupAction
from rasa_sdk.executor import CollectingDispatcher
from rasa_sdk import Action
from .utils import get_intent_examples
class ActionDealWithAffirmationAnswer2(Action):
"""Previously bot asked for an affirmation of the intent if NLU threshold is not met. Now we deal with it"""
def name(self) -> Text:
return "action_react_to_affirm_deny_in_direct_to_customer_support_form"
def __init__(self) -> None:
self.affirm_list = get_intent_examples("kinnitamine")
def run(self,
dispatcher: CollectingDispatcher,
tracker: Tracker,
domain: Dict[Text, Any]
) -> List['Event']:
affirmdeny_value = tracker.get_slot("affirm_deny")
if affirmdeny_value in self.affirm_list: #This is where you are directed to the customer support. Currently I just print out the text.
dispatcher.utter_message(response="utter_direct_to_customer_support")
return [SlotSet("affirm_deny",None)]
else:
dispatcher.utter_message(response="utter_do_not_direct_to_customer_support")
return [SlotSet("affirm_deny",None)]
return [SlotSet("affirm_deny",None)]
EXAMPLES
Case 1.
- User utters
- Rasa predicts intent X with confidence .86
- The response defined by rule or story is uttered
Case 2.
- User utters
- Rasa predicts intent X with confidence .65 and assigns “nlu_fallback” with confidence 0.85 to it instead
- action_check_confidence is triggered
- Action action_check_confidence triggers action “utter_not_confident”.
- Action “utter_not_confident” triggers custom_fallback_form.
- This form uses action “action_ask_custom_fallback_form_affirm_deny” to ask whether the user meant intent X.
This continues in two ways.
Case 2a.
- User affirms
- custom_fallback_form ends
- Action “action_react_to_affirm_deny_in_custom_fallback_form” is triggered.
- Action “action_react_to_affirm_deny_in_custom_fallback_form” utters the response for intent X
- Slots are cleared
Case 2b.
- User denies
- custom_fallback_form ends
- Action “action_react_to_affirm_deny_in_custom_fallback_form” is triggered.
- Action “action_react_to_affirm_deny_in_custom_fallback_form” triggers action “utter_did_not_predict_correctly”.
- Slots are cleared.
- Case 2b continues with case 4.
Case 3.
- User utters
- Rasa predicts intent X with confidence .35 and assigns “nlu_fallback” with confidence 0.85 to it instead
- action_check_confidence is triggered
- Action action_check_confidence triggers action “utter_did_not_predict_correctly”.
- Slots are cleared.
- Case 3 continues with case 4
Case 4.
- Action “utter_did_not_predict_correctly” has been uttered (in case 3 or 2b).
- This triggers direct_to_customer_form.
- Action “utter_ask_direct_to_customer_support_form_affirm_deny” wether the user likes to contact the customer support.
This case continues in two possible ways
Case 4a.
- User affirms.
- direct_to_customer_form ends
- Action “action_react_to_affirm_deny_in_direct_to_customer_form” is triggered.
- Action “action_react_to_affirm_deny_in_direct_to_customer_form” utters “utter_direct_to_customer_support”
- Slots are cleared
Case 4b.
- User affirms.
- direct_to_customer_form ends
- Action “action_react_to_affirm_deny_in_direct_to_customer_form” is triggered.
- Action “action_react_to_affirm_deny_in_direct_to_customer_form” utters “utter_do_not_direct_to_customer_support”
- Slots are cleared
Case 2, 2b, 4, 4a in chat:
User > Terribly sorry to bother you, but I cannot find the SAve button...
Bot > Did you ask about the following topic: pension savings?
User > Nope
Bot > Hmm... I think I still have to study the language.
Bot > I'm sorry I cannot answer. Would you like me to direct you to the customer support?
User > yesh
Bot > I am directing you to the customer support. Please wait a little.
Future work
It’s not an ideal solution in the sense that the bot utters two utterances in a row (unnecessary from the story design point of view). But it does work. When this issue is solved, one can do proper test stories as well (although good luck with coming up utterances with prediction confidence between 0.4-0.85).