TED predicted 'action_listen' based on user intent after action fails

Rasa Open Source
1

Hi, I’m trying to understand why rasa is trying to predict after a failure. My story is like:

- story: user info path

  steps:

    - intent: user_details

    - action: user_details_form

    - active_loop: user_details_form

    - active_loop: null

    - action: action_store_details

While running rasa shell with debug mode, I observed that the action was failing due to some issue with the code in actions.py file. But rasa instead of just throwing that error and stopping there, proceeds to predict the next action. So TEDpolicy which previously predicted action_store_details now predicts action_listen as next intent. In such cases of failure, I might want to tell the user that something has gone wrong, instead of listening to the user using the action_listen intent (User might not even understand that something is wrong, or if they do, they might not know what to enter after this). How should I solve this?

P.S:

Rasa Version      :         2.8.13
Minimum Compatible Version: 2.8.9
Rasa SDK Version  :         2.8.4
Rasa X Version    :         None 
Python Version    :         3.8.8
Operating System  :         Windows-10-10.0.19041-SP0
2

@souvikg10 any thoughts?

3

@lis - you have to handle it in your actions.py in order to return an utterance for failure. Like an action_fallback

1 Like
4

Alright, thanks. Is there any config setup I can do instead? And how to identify the right threshold values for action_fallback?

5

Also, @souvikg10,

DEBUG    rasa.core.policies.memoization  - There is no memorised next action
DEBUG    rasa.core.policies.rule_policy  - There is no applicable rule.
DEBUG    rasa.core.policies.ted_policy  - TED predicted 'action_user_details' based on user intent.
DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_1_RulePolicy
DEBUG    rasa.core.processor  - Predicted next action 'action_default_fallback' with confidence 0.30

This is exactly what is happening when I run rasa shell with debug mode. Is this because of the policy priority?

6

@nik202 would also like to hear your thoughts :slight_smile:

7

@lis So, your use-case is that if you encountered any custom action issue, the bot is waiting for action_listen (it trigger) and you want to overcome this issue? even as you shared the debug screen above, I can see the action_default_fallback with a confidence score of 0.30?

In summary: If there is an issue, the user will not get any message i.e simply will not get a response from the input query and you want to tell the user, that the input you enter is not valid.

Is that right?

8

@nik202, My first problem was when custom actions was failing action_listen was triggered by default. Like Souvik said, I could handle that in the custom actions. I wanted to know if there is any other configuration change I could make instead. My second question is - Now my actions file is not throwing an error, but I see that action_default_fallback is triggered because Rule Policy has a higher priority than TED policy. But I don’t have any rule there. My guess is that TED predicted an action with some confidence which is lesser than the threshold value that is used to trigger action_default_fallback. But how should I improve this?

9

@lis

For 1: Yes, default fallback is the right choice.

For 2: Please provide a good amount of examples, and try to delete older trained models. If you don’t have a good example, then the model will get confused and ended up with a fallback.

I hope this will help you!

10

@nik202 The dialogue is so much like in the story path. If the max_history parameter is set correctly, why is it still not able to get predict the action with higher confidence? TED identifies action_user_details but action_default_fallback is identified as next action based on rule policy. The following is the conversation:

User: Store user details
Bot: What is the user's name?
User: Rick
Bot: What is the user's age?
User: 34
Bot: Sorry I didn't get that

Bot should have predicted action_user_details and saved this into a db. But that did not happen. When you say I have to provide more examples - do you mean more story paths? if so, in this case what kind of story paths will you provide?

11

@lis share with me your config.yml file and stories or rules.yml file and even share me --debug output for particular example which you are using.

12

hi @nik202, thanks for offering to help. before i share those, i have one question for you -

i’m trying to check the channel the user sends. I want this because in case the user sends from one channel (channel A) i don’t want to trigger a form. for the same intent, but for messages originating from another channel (channel B) i want the form to be activated. so in the previous case, im actually also checking for the channel the user is asking the questions from using a custom action like:

intent: user_details
action: action_check_channel
slot_was_set:
  - channel_A : true
action: action_post_message

intent: order_details
action: action_check_channel
slot_was_set: 
  - channel_A : false
action: user_details_form
active_loop: user_details_form
active_loop: null
action: action_user_details

could this be the reason why it is not working as expected? if so, what is the other way to check channel before activating a form?

13

@lis can be I am not sure, please investigate this as I did not implement this personally, I’d encourage to follow single channel at a time.

1 Like