Goes to fallback after form is deactivated

Rasa Open Source
1

My story looks like this:

raise claim path

  • greet
    • utter_greet
  • raise_claim
    • utter_affirm_raise_claim
    • group_claim_form
    • form{“name”: “group_claim_form”}
    • form{“name”: null}
    • utter_thanks_confirmation_support_reachout

Submit in my actions.py looks like this:

def submit(self, dispatcher: CollectingDispatcher, tracker: Tracker, domain: Dict[Text, Any]) -> List[Dict]:

dispatcher.utter_template('utter_submit', tracker)
return []

After collecting the required (free text) slots, the submit method is called. After utter_submit is dispatched, it falls back to default instead of executing utter_thanks_confirmation_support_reachout that I have mentioned in stories.md. Why does this happen?

I tried returning (from submit in actions) a SlotSet event that sets a bool slot called FormSubmitted. And I checked for the same in the story (like - slot{…: true}) after the - form{“name”: null} line. This didnt work either.

Can someone please help?

Debug Log of rasa_core.run:

2019-01-20 00:41:27 DEBUG rasa_core.processor - Action ‘group_claim_form’ ended with events ‘[‘SlotSet(key: employee_id, value: A123)’, ‘Form(None)’, ‘SlotSet(key: requested_slot, value: None)’]’

2019-01-20 00:41:27 DEBUG rasa_core.processor - Bot utterance ‘BotUttered(text: Thank you for providing your employeed id (None) and company name (MGM)!, data: { “elements”: null, “buttons”: null, “attachment”: null })’

2019-01-20 00:41:27 DEBUG rasa_core.policies.fallback - NLU confidence 0.0 is lower than NLU threshold 0.3. Predicting fallback action: action_default_fallback

Debug Log of actions:

DEBUG:rasa_core_sdk.forms:Trying to extract requested slot ‘employee_id’ … DEBUG:rasa_core_sdk.forms:Got mapping ‘{‘type’: ‘from_text’, ‘intent’: [], ‘not_intent’: []}’ DEBUG:rasa_core_sdk.forms:Successfully extracted ‘A123’ for requested slot ‘employee_id’ DEBUG:rasa_core_sdk.forms:No slots left to request DEBUG:rasa_core_sdk.forms:Deactivating the form ‘group_claim_form’ DEBUG:rasa_core_sdk.executor:Successfully ran ‘group_claim_form’ 127.0.0.1 - - [2019-01-20 00:41:25] “POST /webhook HTTP/1.1” 200 383 0.001707

1 Like
2

The reason is in the log - last intent confidence was 0 and the threshold of NLU is set to 0.3 thus it went to fallback. if last input by user’s NLU confidence is higher than threshold then this should work as expected and there is nothing wrong with your story. Although it is strange that entity was extracted correctly (meaning it is in utterance example) but intent detection went south. I solved the similar problem by setting threshold to be negative. Downside of this approach is that when user said something unexpected which really should go “default action” it won’t go there anymore. So I trained “user frustration” intent and checking that intent first in the FormAction.

3

Thank you so much. Glad to know the story looks fine. I will try setting the threshold to a lower value (probably negative). However, I do want to understand why the last intent confidence was zero, when I am merely executing a formaction that deals with free text. And the log talks about zero confidence, after the submit of the formaction was successful, which looks strange to me.

I would like to have a solution for this, without playing with threshold value.

4

This is the entire ActionClass (to check if there is any issue here)…

class GroupClaimForm(FormAction):

def name(self):

   # type: () -> Text
   return "group_claim_form"

@staticmethod

def required_slots(tracker):

   # type: () -> List[Text]
   return ["company_name", "employee_id"]

def slot_mappings(self):

    # type: () -> Dict[Text: Union[Dict, List[Dict]]]
    return {"company_name": [self.from_text()], "employee_id": [self.from_text()]}

def submit(self, dispatcher: CollectingDispatcher, tracker: Tracker, domain: Dict[Text, Any]) -> List[Dict]:

   # type: (CollectingDispatcher, Tracker, Dict[Text, Any]) -> List[Dict]

   # utter submit template
   dispatcher.utter_template('utter_submit', tracker)
   return []
5

2 things. Looks like fix was merged 4 days ago to ignore NLU confidence while still in the form so with latest, you shouldn’t have to play with threshold. NLU confidence is zero because your NLU model didn’t think user input was similar to something in your utterance examples. I would try to review your utterance examples and maybe increase the number of training data?

6

Thats wonderful.

I had the following versions installed:

rasa-core 0.12.3
rasa-core-sdk 0.12.1
rasa-nlu 0.13.8

I uninstalled those and reinstalled just now and got the below. But the fix doesnt seem to be there. Should I specify any particular version number to get the latest fix? Thanks.

rasa-core 0.12.4
rasa-core-sdk 0.12.1
rasa-nlu 0.13.8

7

I dont use from_intent or from_entity in actions, instead I use from_text (like below).

def slot_mappings(self):

    # type: () -> Dict[Text: Union[Dict, List[Dict]]]
    return {"company_name": [self.from_text()], "employee_id": [self.from_text()]}

So, I would expect the core to not try to match with any intent or entity in the training data. Am I wrong?

8

@RamkumarManavalan, 0.12.4 was released 14 days ago per Releases · RasaHQ/rasa_core · GitHub so the fix: https://github.com/RasaHQ/rasa_core/issues/1533 is not on the release but instead you can just install master.

and you are correct that from_text should just user’s next utterance. so scratch my statement about you need training data.

That says, once u get latest loaded (not released so you will need to clone master and install via pip install -e . with setup.py) my guess is you should be good!

9

Wonderful. I pulled the latest and the fix works awesome! Thanks much. I am able to use from_text() and dont have to adjust threshold at all.

Log (Before Fix):

2019-01-23 10:45:42 DEBUG rasa_core.actions.action - Calling action endpoint to run action ‘group_claim_form’. 2019-01-23 10:45:42 DEBUG rasa_core.actions.action - Failed to validate slot employee_id with action group_claim_form 2019-01-23 10:45:42 DEBUG rasa_core.featurizers - Feature ‘entity_EmployeeId’ (value: ‘1.0’) could not be found in feature map. Make sure you added all intents and entities to the domain 2019-01-23 10:45:42 DEBUG rasa_core.policies.fallback - NLU confidence 0.0 is lower than NLU threshold 0.3.

Log (After Fix):

rasa_core.actions.action - Calling action endpoint to run action ‘group_claim_form’. 2019-01-23 10:47:23 DEBUG rasa_core.processor - Action ‘group_claim_form’ ended with events ‘[‘SlotSet(key: employee_id, value: 123Z)’, ‘Form(None)’, ‘SlotSet(key: requested_slot, value: None)’]’

Thank you so much naoko for your continuous support!!!

1 Like
10

@akelad @erohmensing @ricwo @Ghostvv , Can anyone of you please help on this issue?

Versions (Old): rasa-core (0.14.0a1) , rasa-core-sdk (0.12.2), rasa-nlu (0.14.3)

After RASA-X migration:

Version ( New) : Rasa 1.2.8

stories.md

## blackout_1
* blackout+host_or_db+ducklingtime{"duration": 10, "hostname": "ciec-ho-a1s.sys.net"}
    - slot{"duration": 10}
    - slot{"hostname": "ciec-ho-a1s.sys.comcast.net"}
	- action_blackout_comments
    - form{"name": "action_blackout_comments"}
    - form{"name": null}
    - action_blackout_notify
	- action_blackout

actions.py:

class BlackoutComments(FormAction):

def name(self):
    return "action_blackout_comments"

@staticmethod
def required_slots(tracker):
    return ["blackout_comments"]

def slot_mappings(self):
    return {"blackout_comments": self.from_text()}

def submit(self, dispatcher, tracker, domain):
    dispatcher.utter_template('utter_bc_submit', tracker)
    return [ ]

Interactive learning:

? Next user input (Ctr-c to abort): blackout ciec-ho-a1s.sys for 2 hours ? Is the NLU classification for ‘blackout for ciec-ho-a1s.sys.net 2 hours’ with intent ‘blackout+host_or_db+ducklingtime’ correct? Yes

Chat History

Bot You

──────────────────────────────────────────────────────────────── 1 action_listen ──────────────────────────────────────────────────────────────── 2 blackout for ciec-ho-a1s.sys…net 2 hours intent: blackout+host_or_db+ducklingtime 0.89 ──────────────────────────────────────────────────────────────── 3 slot{“duration”: 2} slot{“hostname”: “ciecdb-ho-a1s.sys…net”}

Current slots: dbname: None, blackout_comments: None, time: None, application_name: None, hostname: ciecdb-ho-a1s.sys…net, start_date: None, duration: 2, location: None, requested_slot: None, no_of_hours: None, end_date: None

? The bot wants to run ‘utter_awr_timeframe’, correct? No

Chat History

Bot You

──────────────────────────────────────────────────────────────── 1 action_listen ──────────────────────────────────────────────────────────────── 2 blackout for ciec-ho-a1s.sys…net 2 hours intent: blackout+host_or_db+ducklingtime 0.89 ──────────────────────────────────────────────────────────────── 3 slot{“duration”: 2} slot{“hostname”: “ciecdb-ho-a1s.sys…net”}

Current slots: dbname: None, blackout_comments: None, time: None, application_name: None, hostname: ciecdb-ho-a1s.sys…net, start_date: None, duration: 2, location: None, requested_slot: None, no_of_hours: None, end_date: None

? What is the next action of the bot? 0.00 action_blackout_comments Thanks! The bot will now run action_blackout_comments.

Chat History

Bot You

──────────────────────────────────────────────────────────────── 1 action_listen ──────────────────────────────────────────────────────────────── 2 blackout for ciec-ho-a1s.sys.net 2 hours intent: blackout+host_or_db+ducklingtime 0.89 ──────────────────────────────────────────────────────────────── 3 slot{“duration”: 2} slot{“hostname”: “ciecdb-ho-a1s.sys…net”} action_blackout_comments form{“name”: “action_blackout_comments”} slot{“requested_slot”: “blackout_comments”} Please provide a reason for blackout

Current slots: dbname: None, blackout_comments: None, time: None, application_name: None, hostname: ciecdb-ho-a1s.sys…net, start_date: None, duration: 2, location: None, requested_slot: blackout_comments, no_of_hours: None, end_date: None

? The bot wants to run ‘action_listen’, correct? Yes ? Next user input (Ctr-c to abort): this is for test ? Is the NLU classification for ‘this is for test’ with intent ‘host_or_db’ correct? (Y/n)

When i try to fill the blackout_comments slot, instead of taking the inputs as it is, nlu trying to predict the intent. Could you guys please help me to understand what am doing wrong here?

11

When i try to fill the blackout_comments slot, instead of taking the inputs as it is, nlu trying to predict the intent. Could you guys please help me to understand what am doing wrong here?

Even if you are inside a form, NLU prediction still happens (this is required for e.g. from_intent and not_intent in slot mappings). So it makes sense for interactive learning to be asking you for the intent

12

@erohmensing thanks for your reply. So i just needs to continue with the flow by accepting the whatever intent it predicts? Will it not affect my next action?

13

If your slot mapping is from_text, then yes the intent will not matter, so long as the form action is predicted like it should be, it will fill the slot with the input regardless of intent

14

Hi, I know I’m kinda bumping a dead thread, but I’m having the reverse problem: when an user enters some junk, it unfortunately gets predicted to the intent defined in my from_intent, albeit with abysmal probability (<0.3). How do I customize so that it only triggers when an intent is of at least a certain confidence value?