FormAction Setting Slot Value with Failed intent / entity detection end with FallBack

Rasa Open Source
1

In my FormAction, I have slot name ‘customer_name’ to be filled. I often see extracting name is not trivial to train and I see rasa demo bot Sara also handle by (a) try extract (b) if not get from tracker.latest_message['text']. here: rasa-demo/actions.py at master · RasaHQ/rasa-demo · GitHub

So I took a similar approach in validate method but when Entity Extraction fails as well as user intent (in this case intent: inform) detected as None, although I set slot value manually (by returning SlotSet) next action is predicted as FallBack because NLU confidence is zero.

I have 2 logs attached. First one is a failed case and the Second one is success case (I carefully typed the customer name shown in training nlu data).

Any suggestion as to the best way to solve this issue?

Thank you

===[Log 1: Predicted FallBack as Next Action]===
Please enter your name - First Last
2019-01-10 18:33:20 DEBUG    rasa_core.processor  - Predicted next action 'action_listen' with prob 1.00.
2019-01-10 18:33:20 DEBUG    rasa_core.processor  - Action 'action_listen' ended with events '[]'
127.0.0.1 - - [2019-01-10 18:33:20] "POST /webhooks/rest/webhook?stream=true&token= HTTP/1.1" 200 235 0.084590
naoko reeves

2019-01-10 18:33:22 DEBUG    rasa_core.tracker_store  - Recreating tracker for id 'default'
2019-01-10 18:33:22 DEBUG    rasa_core.processor  - Received user message 'naoko reeves' with intent '{'name': None, 'confidence': 0.0}' and entities '[]'
2019-01-10 18:33:22 DEBUG    rasa_core.processor  - Logged UserUtterance - tracker now has 23 events
2019-01-10 18:33:22 DEBUG    rasa_core.processor  - Current slot values: 
	customer_name: None
	nextbot.main_menu: My case statuses
	pin_number: 1234
	requested_slot: customer_name
2019-01-10 18:33:22 DEBUG    rasa_core.policies.memoization  - Current tracker state [None, {}, {'intent_nextbot.select_main_menu_item': 1.0, 'entity_nextbot.main_menu': 1.0, 'prev_action_listen': 1.0, 'slot_nextbot.main_menu_0': 1.0}]
2019-01-10 18:33:22 DEBUG    rasa_core.policies.memoization  - There is a memorised next action '13'
2019-01-10 18:33:22 DEBUG    rasa_core.policies.form_policy  - There is an active form 'nextiva_crm.auth_form'
@@@ Ah look, because NLU confidence is 0.0 although I set Slot to tell FormAction all the slots has been set@@@
2019-01-10 18:33:22 DEBUG    rasa_core.policies.fallback  - NLU confidence 0.0 is lower than NLU threshold 0.2. Predicting fallback action: action_default_fallback
2019-01-10 18:33:22 DEBUG    rasa_core.policies.ensemble  - Predicted next action using policy_1_FormPolicy
2019-01-10 18:33:22 DEBUG    rasa_core.processor  - Predicted next action 'nextiva_crm.auth_form' with prob 1.00.
2019-01-10 18:33:22 DEBUG    rasa_core.actions.action  - Calling action endpoint to run action 'nextiva_crm.auth_form'.
@@@ This is where I'm setting customer_name Slot @@@
2019-01-10 18:33:22 DEBUG    rasa_core.processor  - Action 'nextiva_crm.auth_form' ended with events '['SlotSet(key: customer_name, value: naoko reeves)', 'Form(None)', 'SlotSet(key: requested_slot, value: None)']'

2019-01-10 18:33:22 DEBUG    rasa_core.policies.memoization  - Current tracker state [{}, {'intent_nextbot.select_main_menu_item': 1.0, 'entity_nextbot.main_menu': 1.0, 'prev_action_listen': 1.0, 'slot_nextbot.main_menu_0': 1.0}, {'intent_nextbot.select_main_menu_item': 1.0, 'entity_nextbot.main_menu': 1.0, 'prev_nextiva_crm.auth_form': 1.0, 'slot_nextbot.main_menu_0': 1.0}]
2019-01-10 18:33:22 DEBUG    rasa_core.policies.memoization  - There is a memorised next action '12'
2019-01-10 18:33:22 DEBUG    rasa_core.policies.form_policy  - There is no active form

@@@ but because nlu confidence is 0 it goes follback... @@@
2019-01-10 18:33:22 DEBUG    rasa_core.policies.fallback  - NLU confidence 0.0 is lower than NLU threshold 0.2. Predicting fallback action: action_default_fallback
2019-01-10 18:33:22 DEBUG    rasa_core.policies.ensemble  - Predicted next action using policy_3_FallbackPolicy
2019-01-10 18:33:22 DEBUG    rasa_core.processor  - Predicted next action 'action_default_fallback' with prob 1.00.
2019-01-10 18:33:22 DEBUG    rasa_core.processor  - Action 'action_default_fallback' ended with events '['UserUtteranceReverted()']'
2019-01-10 18:33:22 DEBUG    rasa_core.processor  - Bot utterance 'BotUttered(text: Sorry, I am unable to help you with that yet., data: {
  "elements": null,
  "buttons": null,
  "attachment": null
})'

===[Log 2: Correctly predicted next action]===

Please enter your name - First Last
2019-01-10 18:42:51 DEBUG    rasa_core.policies.ensemble  - Predicted next action using policy_1_FormPolicy
2019-01-10 18:42:51 DEBUG    rasa_core.processor  - Predicted next action 'action_listen' with prob 1.00.
2019-01-10 18:42:51 DEBUG    rasa_core.processor  - Action 'action_listen' ended with events '[]'
127.0.0.1 - - [2019-01-10 18:42:51] "POST /webhooks/rest/webhook?stream=true&token= HTTP/1.1" 200 235 0.119477
test test

2019-01-10 18:42:53 DEBUG    rasa_core.tracker_store  - Recreating tracker for id 'default'
2019-01-10 18:42:53 DEBUG    rasa_core.processor  - Received user message 'test test' with intent '{'name': 'nextbot.inform', 'confidence': 0.9531433582305908}' and entities '[{'start': 0, 'end': 9, 'value': 'test test', 'entity': 'customer_name', 'confidence': 0.9090301154837648, 'extractor': 'ner_crf'}]'
2019-01-10 18:42:53 DEBUG    rasa_core.processor  - Logged UserUtterance - tracker now has 84 events
2019-01-10 18:42:53 DEBUG    rasa_core.processor  - Current slot values: 
	customer_name: None
	nextbot.main_menu: My case statuses
	pin_number: 3423
	requested_slot: customer_name
2019-01-10 18:42:53 DEBUG    rasa_core.policies.memoization  - Current tracker state [None, {}, {'intent_nextbot.select_main_menu_item': 1.0, 'entity_nextbot.main_menu': 1.0, 'prev_action_listen': 1.0, 'slot_nextbot.main_menu_0': 1.0}]
2019-01-10 18:42:53 DEBUG    rasa_core.policies.memoization  - There is a memorised next action '13'
2019-01-10 18:42:53 DEBUG    rasa_core.policies.form_policy  - There is an active form 'nextiva_crm.auth_form'
2019-01-10 18:42:53 DEBUG    rasa_core.policies.ensemble  - Predicted next action using policy_1_FormPolicy
@@@ Great, next action predicted correctly @@@
2019-01-10 18:42:53 DEBUG    rasa_core.processor  - Predicted next action 'nextiva_crm.auth_form' with prob 1.00.
2

Solved this by setting fallback_threshold to negative number so it never fallback. This works okay if all you have is just Forms and Actions cause you can do fine control. For instance, what I have is to train “hand off to human” intent. within the form, I check the last intent if the intent of “provide info” is not detected correctly. if “hand off to human” intent, will go ahead set follow up action to do that otherwise, go ahead set last utterance to the slot (of course the slot require entity extraction this might not work well)

3

in the first log, FormAction is also predicted

FormPolicy overrides FallbackPolicy prediction, it is just the log is misleading. The fact that Fallback is triggered after Form is finished is a bug. See the issue here: FallbackPolicy shouldn't act on NLU confidence if the last user message was part of a form. · Issue #2869 · RasaHQ/rasa · GitHub

4

I fixed the log and Fallback prediction when the From is finished in this PR: fix fallback by Ghostvv · Pull Request #1591 · RasaHQ/rasa_core · GitHub

5

Thank you

6

The PR is merged, should be working in master correctly