Rasa not following story

neilx4 · December 6, 2018, 9:56am

Hi everyone,

I run into an issue where Rasa is not stick with the storyline.

Here are two stories I wrote:

I have been testing this with a happy path along storyline 2, it starts to behave unexpectedly when it reaches ‘deny’. The ‘deny’ intent was recognized but rasa did not execute the corresponding action, which is ‘utter_ask_wo’ in this case. Instead, it returned ‘action_ask_recent_wo’ followed by ‘action_default_fallback’. Here is the log:

No 2018-12-06 17:17:07 DEBUG rasa_core.tracker_store - Recreating tracker for id ‘default’ 2018-12-06 17:17:07 WARNING py.warnings - /home/neilx/Develop/venv_fu/lib/python3.6/site-packages/sklearn/preprocessing/label.py:151: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use array.size > 0 to check that an array is not empty. if diff:

2018-12-06 17:17:07 DEBUG 2018-12-06 17:17:07 DEBUG 2018-12-06 17:17:07 DEBUG _id: 1234 _name: neil _position: Assistant _wonum: B2345678 2018-12-06 17:17:07 DEBUG 2018-12-06 17:17:07 DEBUG 2018-12-06 17:17:07 DEBUG 2018-12-06 17:17:07 DEBUG 2018-12-06 17:17:07 DEBUG 2018-12-06 17:17:07 DEBUG 2018-12-06 17:17:07 DEBUG “elements”: null, “buttons”: null, “attachment”: null })’ 2018-12-06 17:17:07 DEBUG 2018-12-06 17:17:07 DEBUG Are you contacting 2018-12-06 17:17:07 DEBUG 2018-12-06 17:17:07 DEBUG 2018-12-06 17:17:07 DEBUG 2018-12-06 17:17:07 DEBUG “elements”: null, “buttons”: null, “attachment”: null })’ 2018-12-06 17:17:07 DEBUG 2018-12-06 17:17:07 DEBUG Sorry, I cannot 2018-12-06 17:17:07 DEBUG 2018-12-06 17:17:07 DEBUG 2018-12-06 17:17:07 DEBUG 127.0.0.1 - - [2018-12-06 rasa_core.processor - Received user message ‘No’ with intent ‘{‘name’: ‘deny’, ‘confidence’: 0.7905183810527504}’ and entities ‘[]’ rasa_core.processor - Logged UserUtterance - tracker now has 66 events rasa_core.processor - Current slot values: rasa_core.policies.memoization - Current tracker state [{‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘entity_name’: 1.0, ‘prev_action_get_name’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘prev_utter_ask_position’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘entity_name’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_listen’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_get_position’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘slot__position_0’: 1.0, ‘prev_utter_confirm_all’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_utter_thanks’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_utter_thanks’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘slot__position_0’: 1.0, ‘prev_utter_confirm_all’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_ask_recent_wo’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_listen’: 1.0, ‘slot__position_0’: 1.0, ‘intent_deny’: 1.0, ‘slot__id_0’: 1.0}] rasa_core.policies.memoization - There is no memorised next action rasa_core.policies.ensemble - Predicted next action using policy_1_KerasPolicy rasa_core.processor - Predicted next action ‘action_ask_recent_wo’ with prob 0.43. rasa_core.actions.action - Calling action endpoint to run action ‘action_ask_recent_wo’. rasa_core.processor - Action ‘action_ask_recent_wo’ ended with events ‘[‘SlotSet(key: _wonum, value: B2345678)’]’ rasa_core.processor - Bot utterance ‘BotUttered(text: Are you contacting us in regard to the work order about Air Conditioner (B2345678) created on 2018-12-05 15:00:56?, data: { rasa_core.policies.memoization - Current tracker state [{‘prev_utter_ask_position’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘entity_name’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_listen’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_get_position’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘slot__position_0’: 1.0, ‘prev_utter_confirm_all’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_utter_thanks’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_utter_thanks’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘slot__position_0’: 1.0, ‘prev_utter_confirm_all’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_ask_recent_wo’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_listen’: 1.0, ‘slot__position_0’: 1.0, ‘intent_deny’: 1.0, ‘slot__id_0’: 1.0}, {‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_ask_recent_wo’: 1.0, ‘slot__position_0’: 1.0, ‘intent_deny’: 1.0, ‘slot__id_0’: 1.0}] rasa_core.policies.memoization - There is no memorised next action us in regard to the work order about Air Conditioner (B2345678) created on 2018-12-05 15:00:56? rasa_core.policies.ensemble - Predicted next action using policy_2_FallbackPolicy rasa_core.processor - Predicted next action ‘action_default_fallback’ with prob 1.00. rasa_core.processor - Action ‘action_default_fallback’ ended with events ‘[‘UserUtteranceReverted()’]’ rasa_core.processor - Bot utterance ‘BotUttered(text: Sorry, I cannot understand that. Please try something else., data: { rasa_core.policies.memoization - Current tracker state [{‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘entity_name’: 1.0, ‘prev_action_listen’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘entity_name’: 1.0, ‘prev_action_get_name’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘prev_utter_ask_position’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘entity_name’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_listen’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_get_position’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘slot__position_0’: 1.0, ‘prev_utter_confirm_all’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_utter_thanks’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_utter_thanks’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘slot__position_0’: 1.0, ‘prev_utter_confirm_all’: 1.0, ‘slot__id_0’: 1.0}, {‘entity_wonum’: 1.0, ‘intent_enter_name’: 1.0, ‘slot__wonum_0’: 1.0, ‘slot__name_0’: 1.0, ‘prev_action_ask_recent_wo’: 1.0, ‘slot__position_0’: 1.0, ‘slot__id_0’: 1.0}] rasa_core.policies.memoization - There is no memorised next action understand that. Please try something else. rasa_core.policies.ensemble - Predicted next action using policy_1_KerasPolicy rasa_core.processor - Predicted next action ‘action_listen’ with prob 0.54. rasa_core.processor - Action ‘action_listen’ ended with events ‘[]’ 17:17:07] “POST /webhooks/rest/webhook?stream=true&token= HTTP/1.1” 200 387 0.053924

Thanks in advance, Neil

NikhilBansal21 · December 6, 2018, 12:00pm

you can evaluate your bot just follow the below link their you can find evaluation matrix + failed stories too

ashukrishna100 · December 8, 2018, 1:41am

I m also facing the same issue and I believe it is because Ur next action is being predicted by keras policy. How about memorization policy? Import memorization policy,set max_history to 5, Set augmentation factor to 0. And give it a try. It may work.

neilx4 · December 11, 2018, 1:05am

Thanks for the reply. I did try the evaluation and got 100% correct rate, but the problem stil occurs somehow.

neilx4 · December 11, 2018, 1:07am

I have been experimenting with different combinations of policies including memorization and keras. I have also tried different value for max_history, but the problem still remains.

ashukrishna100 · December 11, 2018, 1:31am

Try with max history for memorization= 30 And for keras keep it 10. Actually I changed my max history parameters and it is working now.

ashukrishna100 · December 12, 2018, 6:49am

I have provided a detailed solution.Kindly have a look.