Rasa not following story

Rasa Open Source
1

Hi everyone,

I run into an issue where Rasa is not stick with the storyline.

Here are two stories I wrote:

%E5%9B%BE%E7%89%87
%E5%9B%BE%E7%89%87546×653 33 KB

I have been testing this with a happy path along storyline 2, it starts to behave unexpectedly when it reaches ‘deny’. The ‘deny’ intent was recognized but rasa did not execute the corresponding action, which is ‘utter_ask_wo’ in this case. Instead, it returned ‘action_ask_recent_wo’ followed by ‘action_default_fallback’. Here is the log:

No 2018-12-06 17:17:07 DEBUG rasa_core.tracker_store - Recreating tracker for id ‘default’ 2018-12-06 17:17:07 WARNING py.warnings - /home/neilx/Develop/venv_fu/lib/python3.6/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use array.size > 0 to check that an array is not empty. if diff:

2018-12-06 17:17:07 DEBUG rasa_core.processor - Received user message ‘No’ with intent ‘{‘name’: ‘deny’, ‘confidence’: 0.7905183810527504}’ and entities ‘’ 2018-12-06 17:17:07 DEBUG rasa_core.processor - Logged UserUtterance - tracker now has 66 events 2018-12-06 17:17:07 DEBUG rasa_core.processor - Current slot values: _id: 1234 _name: neil _position: Assistant _wonum: B2345678 2018-12-06 17:17:07 DEBUG rasa_core.policies.memoization - Current tracker state [{‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘entity_name’: 1.0, ‘prev_action_get_name’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘prev_utter_ask_position’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘entity_name’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_listen’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_get_position’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘slot__position_0’: 1.0, ‘prev_utter_confirm_all’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_utter_thanks’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_utter_thanks’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘slot__position_0’: 1.0, ‘prev_utter_confirm_all’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_ask_recent_wo’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_listen’: 1.0, ‘slot__position_0’: 1.0, ‘intent_deny’: 1.0, ‘slot__id_0’: 1.0}] 2018-12-06 17:17:07 DEBUG rasa_core.policies.memoization - There is no memorised next action 2018-12-06 17:17:07 DEBUG rasa_core.policies.ensemble - Predicted next action using policy_1_KerasPolicy 2018-12-06 17:17:07 DEBUG rasa_core.processor - Predicted next action ‘action_ask_recent_wo’ with prob 0.43. 2018-12-06 17:17:07 DEBUG rasa_core.actions.action - Calling action endpoint to run action ‘action_ask_recent_wo’. 2018-12-06 17:17:07 DEBUG rasa_core.processor - Action ‘action_ask_recent_wo’ ended with events ‘[‘SlotSet(key: _wonum, value: B2345678)’]’ 2018-12-06 17:17:07 DEBUG rasa_core.processor - Bot utterance ‘BotUttered(text: Are you contacting us in regard to the work order about Air Conditioner (B2345678) created on 2018-12-05 15:00:56?, data: { “elements”: null, “buttons”: null, “attachment”: null })’ 2018-12-06 17:17:07 DEBUG rasa_core.policies.memoization - Current tracker state [{‘prev_utter_ask_position’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘entity_name’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_listen’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_get_position’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘slot__position_0’: 1.0, ‘prev_utter_confirm_all’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_utter_thanks’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_utter_thanks’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘slot__position_0’: 1.0, ‘prev_utter_confirm_all’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_ask_recent_wo’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_listen’: 1.0, ‘slot__position_0’: 1.0, ‘intent_deny’: 1.0, ‘slot__id_0’: 1.0}, {‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_ask_recent_wo’: 1.0, ‘slot__position_0’: 1.0, ‘intent_deny’: 1.0, ‘slot__id_0’: 1.0}] 2018-12-06 17:17:07 DEBUG rasa_core.policies.memoization - There is no memorised next action Are you contacting us in regard to the work order about Air Conditioner (B2345678) created on 2018-12-05 15:00:56? 2018-12-06 17:17:07 DEBUG rasa_core.policies.ensemble - Predicted next action using policy_2_FallbackPolicy 2018-12-06 17:17:07 DEBUG rasa_core.processor - Predicted next action ‘action_default_fallback’ with prob 1.00. 2018-12-06 17:17:07 DEBUG rasa_core.processor - Action ‘action_default_fallback’ ended with events ‘[‘UserUtteranceReverted()’]’ 2018-12-06 17:17:07 DEBUG rasa_core.processor - Bot utterance ‘BotUttered(text: Sorry, I cannot understand that. Please try something else., data: { “elements”: null, “buttons”: null, “attachment”: null })’ 2018-12-06 17:17:07 DEBUG rasa_core.policies.memoization - Current tracker state [{‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘entity_name’: 1.0, ‘prev_action_listen’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘entity_name’: 1.0, ‘prev_action_get_name’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘prev_utter_ask_position’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘entity_name’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_listen’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_get_position’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘slot__position_0’: 1.0, ‘prev_utter_confirm_all’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_utter_thanks’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_utter_thanks’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘slot__position_0’: 1.0, ‘prev_utter_confirm_all’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_ask_recent_wo’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}] 2018-12-06 17:17:07 DEBUG rasa_core.policies.memoization - There is no memorised next action Sorry, I cannot understand that. Please try something else. 2018-12-06 17:17:07 DEBUG rasa_core.policies.ensemble - Predicted next action using policy_1_KerasPolicy 2018-12-06 17:17:07 DEBUG rasa_core.processor - Predicted next action ‘action_listen’ with prob 0.54. 2018-12-06 17:17:07 DEBUG rasa_core.processor - Action ‘action_listen’ ended with events ‘’ 127.0.0.1 - - [2018-12-06 17:17:07] “POST /webhooks/rest/webhook?stream=true&token= HTTP/1.1” 200 387 0.053924

Thanks in advance, Neil

2

you can evaluate your bot just follow the below link their you can find evaluation matrix + failed stories too

3

I m also facing the same issue and I believe it is because Ur next action is being predicted by keras policy. How about memorization policy? Import memorization policy,set max_history to 5, Set augmentation factor to 0. And give it a try. It may work.

4

Thanks for the reply. I did try the evaluation and got 100% correct rate, but the problem stil occurs somehow.

5

I have been experimenting with different combinations of policies including memorization and keras. I have also tried different value for max_history, but the problem still remains.

6

Try with max history for memorization= 30 And for keras keep it 10. Actually I changed my max history parameters and it is working now.

7

I have provided a detailed solution.Kindly have a look.