Failing Gracefully with the TwoStageFallbackPolicy

(Tobias Wochinger) #1

Check out our blog post on the TwoStageFallbackPolicy which came with Rasa Core 0.13. It provides an intuitive and user-friendly conversation flow in case of NLU messages with low classification confidence. Have you tried it already? Share your experiences with us in the thread below :rocket:

2 Likes

(Romain Huber) #2

Hey @Tobias_Wochinger, we are going to replace the new fallback policy in our project, and looking at the code it seems that the default configuration is only saying something like “Do you mean [name_intent]”, but in your blog, you are proposing a choice between (I think) the two highest confident intent.

Could you share the code for this custom action ?

I would like to personalise it with something like propose two intent if the difference in confidence between then is lower than 10, or only propose the highest confident one.

But it’s a really nice feature, everyone was looking forward to have it in our bot :rocket:

0 Likes

(Tobias Wochinger) #3

@huberrom Great to your that you happy with the new policy :rocket: This is our current implementation which we are using inour bot on the documentation: rasa-demo/actions.py at 096568e64df4c25902f91fcf7b409fa2fcd05b16 · RasaHQ/rasa-demo · GitHub .

We are currently also working on an improved version of it, which is currently a pr include entities in button titles by wochinge · Pull Request #277 · RasaHQ/rasa-demo · GitHub (might be merged till you read this :smile:).

The basic idea of the implementation is that we read the button titles from a csv file. We saw that it is important to add potentially recognised entities to the payload of the button and to the button message title. We first put the entities in brackets after the intent description, but now we want to actually use different button titles depending on the intent and the extracted entities.

Let me know if you have any more questions :rocket:

1 Like

(Akarsh) #4

Hey i am having an issue when i tried to use the csv file to read the meaning of intent just like in the demo bot…it shows up like did you mean user_ask_bot_name? It shows intent name instead of the meaning in csv file why is this?

0 Likes

(Romain Huber) #5

Can you share your csv file to be sure it has the right format ?

It seems to work for me

0 Likes

(Romain Huber) #6

Hey @Tobias_Wochinger, finally have the time to try the policy, look really great so far but I have a small problem :

I override both action_default_ask_rephrase and action_default_ask_affirmation because I want to change the text from english to french, so it shouldn’t change the general behaviour, but I have the following problem :

(The second picture if following the first one, after asking to rephrase, I’m just saying “Vitale” again, and the end of the second picture is the bot asking to rephrase again)

I’m mind, I thought that after the second “Something else”, the bot would answer “Sorry I don’t provide this skill”, but it’s only running “action_default_ask_rephrase”

Any idea on this ?

0 Likes

(Akarsh) #7

@huberrom hey thanks for the reply, i used the same format and code in the demo bot and its not executing the right intent when i press the button i have raised the issue on the github page #3133. intent_description_mapping.csv (4.1 KB)

0 Likes

(Akarsh) #8

If anyone could help. Thanks in advance

0 Likes

(Romain Huber) #9

Yes sounds good to me.

Don’t you have a warning saying that it can’t find the file ?

Maybe it’ll help, I override both action_default_ask_rephrase and action_default_ask_affirmation with the code I found here : rasa-demo/actions.py at 096568e64df4c25902f91fcf7b409fa2fcd05b16 · RasaHQ/rasa-demo · GitHub and here rasa_core/action.py at 431d4c89c5025bec2be51c065add2ab272089bdf · RasaHQ/rasa_core · GitHub .

Add some logs to see if the csv is correctly read. Don’t forget to add both of them in your domain.yml so that they are called instead of the default one

0 Likes

(Akarsh) #10

No warnings or anything!..the error i got is in issue #3133 I didn’t add it in the domain file ! Thank you for the help, i ll get back after checking! …

0 Likes

(Romain Huber) #11

Oook sorry for the question before, but after reading the policy, I noticed this :

Args:
     nlu_threshold: minimum threshold for NLU confidence.
          If intent prediction confidence is lower than this,
          predict fallback action with confidence 1.0.
      core_threshold: if NLU confidence threshold is met,
          predict fallback action with confidence
          `core_threshold`. If this is the highest confidence in
          the ensemble, the fallback action will be executed.
      fallback_core_action_name: This action is executed if the Core
          threshold is not met.
      fallback_nlu_action_name: This action is executed if the user
          denies the recognised intent for the second time.
      deny_suggestion_intent_name: The name of the intent which is used
           to detect that the user denies the suggested intents.

So I changed my policy to this :

policies:
  - name: KerasPolicy
    epochs: 200
    max_history: 10
  - name: TwoStageFallbackPolicy
    nlu_threshold: 0.7
    core_threshold: 0.3
    fallback_nlu_action_name: "action_hors_perimetre"
    fallback_core_action_name: "action_default_ask_affirmation"
    deny_suggestion_intent_name: "hors_perimetre"
  - name: AugmentedMemoizationPolicy
    max_history: 10

And now it’s working. So fallback_nlu_action_name is the action called when the user denies the second time, and the fallback_core_action_name is the action called to propose the recognized intents. Wasn’t clear to me with the doc, but working like a charm now

1 Like

(Mohamed Hassan Kadri) #12

i just overrided the class ActionDefaultAskAffirmation so i can customize the default message but that didnt work i followed the example in rasa demo can any one help me

0 Likes

(Romain Huber) #13

Did you put ActionDefaultAskAffirmation in your domain.yml ?

0 Likes

(Mohamed Hassan Kadri) #14

When i addded action_default_ask_affirmation to the domain.yml i start getting this error

Failed to run custom action ‘action_default_ask_affirmation’. Action server responded with a non 200 status code of 500. Make sure your action server properly runs actions and returns a 200 once the action is executed. Error: 500 Server Error: INTERNAL SERVER ERROR for url: http://localhost:5055/webhook

0 Likes

(Romain Huber) #15

Well, that means that either your action server is not started (or correctly set up) or you didn’t define the action_default_ask_affirmation

0 Likes

(Mohamed Hassan Kadri) #16
Well the server is working fine and here is the code that i puted in my action.py i think its fine too 

> class ActionDefaultAskAffirmation(Action):
>    def name(self):
>         return "action_default_ask_affirmation"
>    def __init__(self):
>        import pandas as pd
>        self.intent_mappings = {}
>        # read the mapping from a csv and store it in a dictionary
>        with open('data\intent_description_mapping.csv', newline='', encoding='utf-8') as file:
>            csv_reader = csv.reader(file)
>            for row in csv_reader:
>                
>                self.intent_mappings[row[0]] = row[1]
>    def run(self, dispatcher, tracker, domain):
>        # get the most likely intent
>        last_intent_name = tracker.latest_message['intent']['name']
> 
>        # get the prompt for the intent
>        intent_prompt = self.intent_mappings[last_intent_name]
> 
>        # Create the affirmation message and add two buttons to it.
>        # Use '/<intent_name>' as payload to directly trigger '<intent_name>'
>        # when the button is clicked.
>        message_title = ("Sorry, I'm not sure I've understood "
>                          "you correctly 🤔 is your Question...")
>        print('heyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy')
>        message = "Hahahahah '{}'?".format(intent_prompt)
>        buttons = [{'title': 'yes',
>                    'payload': '/{}'.format(last_intent_name)},
>                   {'title': 'No',
>                    'payload': '/out_of_scope'}]
>        dispatcher.utter_button_message(message, buttons=buttons)
> 
>        return []
0 Likes

(Mohamed Hassan Kadri) #17

I should point too that i have other actions and they are running without any problem

0 Likes

(Romain Huber) #18

Do you have the logs of the action server when the action is called ?

0 Likes

(Mohamed Hassan Kadri) #20
2019-04-01 10:36:30 DEBUG    rasa_core.processor  - Received user message 'how can we ge tthis all' with intent '{'name': 'ask_whatspossible', 'confidence': 0.2748849016662266}' and entities '[]'
2019-04-01 10:36:30 DEBUG    rasa_core.processor  - Logged UserUtterance - tracker now has 32 events
2019-04-01 10:36:30 DEBUG    rasa_core.processor  - Current slot values:
        age: None
        category: None
        month: None
        nationality: None
        requested_slot: nationality
        ssc: None
        ssc_level: None
        year: None
2019-04-01 10:36:30 DEBUG    rasa_core.policies.memoization  - Current tracker state [{'intent_inform_about_internship_nationality': 1.0, 'prev_utter_start_scholarship': 1.0, 'active_form_scholarship_form': 1.0},
{'prev_scholarship_form': 1.0, 'intent_inform_about_internship_nationality': 1.0, 'active_form_scholarship_form': 1.0}, {'prev_action_listen': 1.0, 'intent_inform_about_internship_nationality': 1.0, 'active_form_scholarship_form': 1.0}, {'intent_inform_about_internship_nationality': 1.0, 'active_form_scholarship_form': 1.0, 'prev_utter_noworries': 1.0}, {'prev_action_listen': 1.0, 'intent_ask_whatspossible': 1.0, 'active_form_scholarship_form': 1.0}]
2019-04-01 10:36:30 DEBUG    rasa_core.policies.memoization  - There is no memorised next action
2019-04-01 10:36:30 DEBUG    rasa_core.policies.form_policy  - There is an active form 'scholarship_form'
2019-04-01 10:36:30 DEBUG    rasa_core.policies.two_stage_fallback  - User '321652a109204396829239e948e08a4d' has to affirm intent 'ask_whatspossible'.
2019-04-01 10:36:30 DEBUG    rasa_core.policies.ensemble  - Predicted next action using policy_2_FormPolicy
2019-04-01 10:36:30 DEBUG    rasa_core.processor  - Predicted next action 'scholarship_form' with prob 1.00.
2019-04-01 10:36:30 DEBUG    rasa_core.actions.action  - Calling action endpoint to run action 'scholarship_form'.
2019-04-01 10:36:31 DEBUG    rasa_core.actions.action  - Failed to validate slot nationality with action scholarship_form
2019-04-01 10:36:31 DEBUG    rasa_core.policies.memoization  - Current tracker state [{'intent_inform_about_internship_nationality': 1.0, 'prev_utter_start_scholarship': 1.0, 'active_form_scholarship_form': 1.0},
{'prev_scholarship_form': 1.0, 'intent_inform_about_internship_nationality': 1.0, 'active_form_scholarship_form': 1.0}, {'prev_action_listen': 1.0, 'intent_inform_about_internship_nationality': 1.0, 'active_form_scholarship_form': 1.0}, {'intent_inform_about_internship_nationality': 1.0, 'active_form_scholarship_form': 1.0, 'prev_utter_noworries': 1.0}, {'prev_action_listen': 1.0, 'intent_ask_whatspossible': 1.0, 'active_form_scholarship_form': 1.0}]
2019-04-01 10:36:31 DEBUG    rasa_core.policies.memoization  - There is no memorised next action
2019-04-01 10:36:31 DEBUG    rasa_core.policies.form_policy  - There is an active form 'scholarship_form'
2019-04-01 10:36:31 DEBUG    rasa_core.policies.two_stage_fallback  - User '321652a109204396829239e948e08a4d' has to affirm intent 'ask_whatspossible'.
2019-04-01 10:36:31 DEBUG    rasa_core.policies.ensemble  - Predicted next action using policy_3_TwoStageFallbackPolicy
2019-04-01 10:36:31 DEBUG    rasa_core.processor  - Predicted next action 'action_default_ask_affirmation' with prob 1.00.
2019-04-01 10:36:31 DEBUG    rasa_core.actions.action  - Calling action endpoint to run action 'action_default_ask_affirmation'.
2019-04-01 10:36:32 ERROR    rasa_core.actions.action  - Failed to run custom action 'action_default_ask_affirmation'. Action server responded with a non 200 status code of 500. Make sure your action server properly runs actions and returns a 200 once the action is executed. Error: 500 Server Error: INTERNAL SERVER ERROR for url: http://localhost:5055/webhook
2019-04-01 10:36:32 ERROR    rasa_core.processor  - Encountered an exception while running action 'action_default_ask_affirmation'. Bot will continue, but the actions events are lost. Make sure to fix the exception in your custom code.
2019-04-01 10:36:33 DEBUG    rasa_core.processor  - Failed to execute custom action.
Traceback (most recent call last):
  File "C:\Users\mohamed-hassan.kadri\AppData\Local\Programs\Python\Python36\lib\site-packages\rasa_core\actions\action.py", line 342, in run
    response.raise_for_status()
  File "C:\Users\mohamed-hassan.kadri\AppData\Local\Programs\Python\Python36\lib\site-packages\requests\models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url: http://localhost:5055/webhook

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\mohamed-hassan.kadri\AppData\Local\Programs\Python\Python36\lib\site-packages\rasa_core\processor.py", line 346, in _run_action
    events = action.run(dispatcher, tracker, self.domain)
  File "C:\Users\mohamed-hassan.kadri\AppData\Local\Programs\Python\Python36\lib\site-packages\rasa_core\actions\action.py", line 362, in run
    raise Exception("Failed to execute custom action.")
Exception: Failed to execute custom action.
2019-04-01 10:36:33 DEBUG    rasa_core.processor  - Action 'action_default_ask_affirmation' ended with events '[]'
2019-04-01 10:36:33 DEBUG    rasa_core.policies.memoization  - Current tracker state [{'prev_scholarship_form': 1.0, 'intent_inform_about_internship_nationality': 1.0, 'active_form_scholarship_form': 1.0}, {'prev_action_listen': 1.0, 'intent_inform_about_internship_nationality': 1.0, 'active_form_scholarship_form': 1.0}, {'intent_inform_about_internship_nationality': 1.0, 'active_form_scholarship_form': 1.0, 'prev_utter_noworries': 1.0}, {'prev_action_listen': 1.0, 'intent_ask_whatspossible': 1.0, 'active_form_scholarship_form': 1.0}, {'intent_ask_whatspossible': 1.0, 'active_form_scholarship_form': 1.0, 'prev_action_default_ask_affirmation': 1.0}]
2019-04-01 10:36:33 DEBUG    rasa_core.policies.memoization  - There is no memorised next action
2019-04-01 10:36:33 DEBUG    rasa_core.policies.form_policy  - There is an active form 'scholarship_form'
2019-04-01 10:36:33 DEBUG    rasa_core.policies.ensemble  - Predicted next action using policy_3_TwoStageFallbackPolicy
2019-04-01 10:36:33 DEBUG    rasa_core.processor  - Predicted next action 'action_listen' with prob 1.00.
2019-04-01 10:36:33 DEBUG    rasa_core.processor  - Action 'action_listen' ended with events '[]
0 Likes

(Mohamed Hassan Kadri) #21

Any help guys

0 Likes