Failing Gracefully with the TwoStageFallbackPolicy

I have a question about the TwoStageFallbackPolicy: I used the Rasa demo as a template for the defaultAskAffirmation action: rasa-demo/actions.py at master · RasaHQ/rasa-demo · GitHub, and extracted present entities in the user message to tailor button titles and payloads.

When the NLU does not reach the treshold, the buttons are correctly filled, but upon selecting a button, my fallback bot utterance gets triggered. I assume this happens in the ‘action_revert_fallback’, where UserUttered is filled with my button payload, but seems to get reverted afterwards? How to prevent this behaviour, do I need to override action_revert_fallback?

Action 'action_default_ask_affirmation' ended with events '[]'
Bot utterance 'BotUttered(text: Sorry, I'm not sure I've understood you correctly. Do you mean ... , data: {
  "elements": null,
  "buttons": [
    {
      "payload": "/get_help{\"course\": \"programming\"}",
      "title": "I want to get in touch with the helpdesk."
    },
    {
      "payload": "/get_description{\"course\": \"programming\"}",
      "title": "What is programming about?"
    },
    {
      "payload": "/affirm{\"course\": \"programming\"}",
      "title": "Agreed."
    },
    {
      "payload": "/out_of_scope",
      "title": "Something else"
    }
  ],
  "attachment": null
})'
Current tracker state [None, None, None, None, None, None, None, {}, {'intent_get_help': 1.0, 'prev_action_listen': 1.0, 'slot_course_0': 1.0, 'entity_course': 1.0}, {'intent_get_help': 1.0, 'prev_action_default_ask_affirmation': 1.0, 'slot_course_0': 1.0, 'entity_course': 1.0}]
There is no memorised next action
There is no active form
Predicted next action using policy_1_TwoStageFallbackPolicy
Predicted next action 'action_listen' with prob 1.00.
Action 'action_listen' ended with events '[]'
Recreating tracker for id 'default'
Received user message '/get_description{"course":"programming"}' with intent '{'name': 'get_description', 'confidence': 1.0}' and entities '[{'entity': 'course', 'start': 16, 'end': 40, 'value': 'programming'}]'
Logged UserUtterance - tracker now has 18 events
Current slot values: 
	booking-type: None
	booking_confirmed: None
	cancellation_confirmed: None
	categories: None
	course: programming
	date: None
	date-period: None
	dates: None
	display-option: None
	employee_id: None
	given-name: None
	last-name: None
	location: None
	locations: None
	other_date: None
	other_location: None
	requested_slot: None
	seminar_id: None
	time: None
	title: None
	user-level: None
	user_verified: None
User 'default' affirmed intent 'get_description'
Current tracker state [None, None, None, None, None, None, {}, {'intent_get_help': 1.0, 'prev_action_listen': 1.0, 'slot_course_0': 1.0, 'entity_course': 1.0}, {'intent_get_help': 1.0, 'prev_action_default_ask_affirmation': 1.0, 'slot_course_0': 1.0, 'entity_course': 1.0}, {'prev_action_listen': 1.0, 'slot_course_0': 1.0, 'intent_get_description': 1.0, 'entity_course': 1.0}]
There is no memorised next action
There is no active form
Predicted next action using policy_1_TwoStageFallbackPolicy
Predicted next action 'action_revert_fallback_events' with prob 1.00.
Action 'action_revert_fallback_events' ended with events '['UserUtteranceReverted()', 'UserUtteranceReverted()', 'ActionExecuted(action: action_listen, policy: None, confidence: None)', 'UserUttered(text: /get_description{"course":"programming"}, intent: {\'name\': \'get_description\', \'confidence\': 1.2}, entities: [{\'entity\': \'course\', \'start\': 16, \'end\': 40, \'value\': \'programming\'}])']'
Current tracker state [None, None, None, None, None, None, None, None, {}, {'prev_action_listen': 1.0, 'intent_get_description': 1.0, 'entity_course': 1.0}]
There is no memorised next action
There is no active form
Predicted next action using policy_0_EmbeddingPolicy
Predicted next action 'action_provide_description' with prob 0.50.
Calling action endpoint to run action 'action_provide_description'.
Action 'action_provide_description' ended with events '[]'
Bot utterance 'BotUttered(text: You need to specify a course for your request, data: {

This seems like a Core fallback. Does your button trigger an intent which is not in a story (or in this story flow)?

action_revert_fallback reverts all events caused by the fallback policy so that you don’t have to add extra stories. You should not need to overwrite it.

No, the intent that gets triggered is the right one according to the story flow.

After the buttons are presented according to ActionDefaultAskAffirmation TwoStageFallback predicts action_listen – > I clickt the button /get_description{\"course\": \"programming\"} --> action_revert_fallback is run --> the right action action_provide_description is run. It should use the slot “course” to give a description of that course. However the course slot is empty, so in my custom code the fallback “you need to specify a course” is run. That’s why I thought the user utterance (or button click) is reverted and slot emptied. Now I found the workaround to use course = next(tracker.get_latest_entity_values('course'), None) in my action instead of tracker.get_slot('course') and it works.

I specified the FallbackPolicy as follows in the policy config:

fallback_core_action_name: "action_default_fallback"
fallback_nlu_action_name: "action_default_fallback"
deny_suggestion_intent_name: "out_of_scope"

you can create a slot for the course entity and it should work fine and then it should automatically be set as slot

Can anyone link me the new legacy-docs link for the twostagefallback policy please? Can’t find it anywhere!

Sorry, but better late than never :smiley: Training and Policies

I got “Did you mean ‘None’?” in my testing and it turned out that’s when the bot’s prediction list is all 0.0 for all intents. Leaving this here as n FYI for others.

1 Like

I’m using two stage fallback policy. When the intent have low confidence, the bot ask if the user is talking about the most confidence socre intent. That works fine. But when I select the “no” button, its go to action_revert_fallback_events but I want the bot to ask the user to rephrase it. What am I missing?

1 Like

Hi ,@huberrom thanks for the question , exactly this is wat my question ,… if u did this concept, pls share to me … i struggling in a months , …

Thanks in advance…

Which intent does the no button trigger?

@murugan could you please elaborate a bit on your question?

could u pls share your code …

hi @Tobias_Wochinger , thanks for reply , actually i working with single stage fallback policy , i want to work with two stage fallback policy what are the configuration i need , and my training file is .md file , not in .csv file , so how can i map my intent with two stage fall back policy ("did u mean {} ?, ") , could u pls share any sample code , or send how can i create custom action for two stage fallback policy using my .md file

here with i attached my jupyter notebook file .intent_twoStageFallbackPolicy.ipynb (155.9 KB) domain.yml (7.1 KB) nlu.md (10.8 KB) policy.yml (249 Bytes)config.yml (310 Bytes)

Hi @murugan,

sorry I don’t have a bot set up on my currently, but check out the rasa-demo bot. You can see how we did the action_default_ask_affirmation there (rasa-demo/actions.py at ef2bd0a75a10756cec5afbd770945999c128d970 · RasaHQ/rasa-demo · GitHub) and also how we configured it (rasa-demo/config.yml at master · RasaHQ/rasa-demo · GitHub) . hope that helps you :slight_smile:

ok thanks …

Hi @Tobias_Wochinger I am very sorry for sounding ignorant… I am still very new to Rasa (just got front end to work though! so that’s a plus! :slight_smile: ) I read your blog post several times and I am at a loss with what to do with the CSV file… does it serve as some sort of the database or look up?

Thank you so much for helping me and guiding me through my Rasa journey!! - Eli

Hi @Eli,

does it serve as some sort of the database or look up?

Yes, it’s exactly doing that. It maps the plain intent names (e.g. inform, greet) to nicer prompts (e.g. say hello). Does that make sense?

1 Like

Is there anyway I could take a look at the CSV file in your blog as an example and are you able to elaborate more on what it does in the custom action?

@Eli I think the examples of the rasa-demo I sent you in the posts should cover this. Which parts of the implementation are hard to understand for you?

1 Like

Hi @Tobias_Wochinger! Sorry for the late response! I was off work for a few days! So to confirm my understanding of the Action Defualt Ask Affirmation action in your blog…

from rasa_core_sdk import Action
import csv

^ This is importing the essentials to make the action work…

class ActionDefaultAskAffirmation(Action):
   """Asks for an affirmation of the intent if NLU threshold is not met."""

   def name(self):
       return "action_default_ask_affirmation"

   def __init__(self):
       self.intent_mappings = {}
       # read the mapping from a csv and store it in a dictionary
       with open('intent_mapping.csv', newline='', encoding='utf-8') as file:
           csv_reader = csv.reader(file)
           for row in csv_reader:
               self.intent_mappings[row[0]] = row[1]

^ I have found the CSV file on Github and it looks like it beautifully maps your intents to a button and an entity? - Did I understand this correctly?

  • With the bot I am building, it is a searchbot, so I am wondering if the Rasa could just work off with a few examples on for the mapping, since my bot will be indexing thousands of search terms and documents.

  • How is this different from having examples in the NLU?

      def run(self, dispatcher, tracker, domain):
             # get the most likely intent
             last_intent_name = tracker.latest_message['intent']['name']
    

^ This line here is as you described, it will get the most likely intent, through training and confidence level, correct?

   # get the prompt for the intent
   intent_prompt = self.intent_mappings[last_intent_name]

^ Here is where I got confused… not sure what a prompt is in this case.

 # Create the affirmation message and add two buttons to it.
   # Use '/<intent_name>' as payload to directly trigger '<intent_name>'
   # when the button is clicked.
   message = "Did you mean '{}'?".format(intent_prompt)
   buttons = [{'title': 'Yes',
               'payload': '/{}'.format(last_intent_name)},
              {'title': 'No',
               'payload': '/out_of_scope'}]
   dispatcher.utter_button_message(message, buttons=buttons)

   return []

^ Lastly, for the buttons here, is there anyway to maybe list 3 to 5 search terms based on the bot’s confidence level instead of a yes or no… I have been just hardcoding buttons for the the past bot exercises I have done, so I am not sure how to do anything dynamic yet.

Thank you in advance for helping me out and staying with me, Tobias!

@Tobias_Wochinger - sorry for spamming this thread. I was reviewing more files in the Rasa Demo git repository and I noticed this custom action: Is is the action that triggers the Ask Affirmation action? And also to confirm my understanding here:

class ActionDefaultFallback(Action):
def name(self) -> Text:
    return "action_default_fallback"

def run(
    self,
    dispatcher: CollectingDispatcher,
    tracker: Tracker,
    domain: Dict[Text, Any],
) -> List["Event"]:

^ This look to be the default code to make the action work…?

    # Fallback caused by TwoStageFallbackPolicy
    if (
        len(tracker.events) >= 4
        and tracker.events[-4].get("name") == "action_default_ask_affirmation"
    ):

^ Unsure what this does…?

        dispatcher.utter_template("utter_restart_with_button", tracker)

        return [SlotSet("feedback_value", "negative"), ConversationPaused()]

^ after passing the if conditions of the above code it will show a set of buttons? Unsure why the feedback value is set to negative though… and would it help improve the chatbot to have a feedback value.

    # Fallback caused by Core
    else:
        dispatcher.utter_template("utter_default", tracker)
        return [UserUtteranceReverted()] 

^ If none of the conditions in the if statement is passed, the bot will return a default error utterance…?

Again, thanks so much for baring with me, Tobias!