AugmentedMemoization breaking on entity detection

samscudder · December 23, 2019, 5:00pm

I have a bot which uses a form to interact with an issue API using custom actions. The bot has both “show me details” and “show me details of issue 123xyz” examples in the training data, and picks up the intent correctly.

When I type in “show me details”, since issue is a requested slot, I get a question asking for the issue, and that get’s recognized OK.

Problem is, if I type int “show me details of issue 123xyz”, the entity is extracted, but I get a “There is no memoized next action” error from the AugmentedMemoizationPolicy, and get pushed back to fallback, even if there is 100% confidence.

The memoization seems to be picking up an “extract_issue_number” intent in the middle and is getting muddled up. When it fails, I see this:

Current tracker state [{'prev_action_listen': 1.0, 'entity_issue_number': 1.0, 'intent_show_issue_details': 1.0}]

If I omit the issue number and it get’s asked by the form, I see this:

Current tracker state [{'prev_action_listen': 1.0, 'intent_show_issue_details': 1.0, 'slot_profile_0': 1.0}]

I’m using max history of 1. I’ve tried 0 and 2 and with these other values, the stories aren’t picked up correctly.

amn41 · January 3, 2020, 1:14pm

does ignoring entities for this intent work as a solution?

samscudder · January 3, 2020, 4:49pm

The slots are unfeaturized, so I thought they wouldn’t interfere. I trained with use_entities: [], but that didn’t help. Here’s the debug output:

Your input ->  show me details of issue 20010385879595
2020-01-03 13:34:26 DEBUG    rasa.core.processor  - Received user message 'show me details of issue 20010385879595' with intent '{'name': 'detail_issue', 'confidence': 0.9917369484901428}' and entities '[{'start': 20, 'end': 34, 'value': '20010385879595', 'entity': 'issue_number', 'confidence': 0.9939148859997056, 'extractor': 'CRFEntityExtractor'}]'
2020-01-03 13:34:26 DEBUG    rasa.core.processor  - Current slot values:
        issue_number: 20010385879595
        profile: corretor
        user_name: None
2020-01-03 13:34:26 DEBUG    rasa.core.processor  - Logged UserUtterance - tracker now has 325 events
2020-01-03 13:34:26 DEBUG    rasa.core.policies.form_policy  - There is no active form
2020-01-03 13:34:26 DEBUG    rasa.core.policies.memoization  - Current tracker state [{'prev_action_listen': 1.0, 'entity_issue_number': 1.0, 'intent_detail_issue': 1.0, 'slot_profile_0': 1.0}]
2020-01-03 13:34:26 DEBUG    rasa.core.policies.memoization  - Launch DeLorean...
2020-01-03 13:34:26 DEBUG    rasa.core.policies.memoization  - Current tracker state [{'prev_action_listen': 1.0, 'entity_issue_number': 1.0, 'intent_detail_issue': 1.0}]
2020-01-03 13:34:26 DEBUG    rasa.core.policies.memoization  - There is no memorised next action
2020-01-03 13:34:26 DEBUG    rasa.core.policies.fallback  - NLU confidence threshold met, confidence of fallback action set to core threshold (0.7).
2020-01-03 13:34:26 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_4_FallbackPolicy
2020-01-03 13:34:26 DEBUG    rasa.core.processor  - Predicted next action 'utter_default' with confidence 0.70.
2020-01-03 13:34:26 DEBUG    rasa.core.processor  - Action 'utter_default' ended with events '['BotUttered(text: Sorry, I can't understand that. Could you rephrase it?, data: {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, metadata: {})']'
2020-01-03 13:34:26 DEBUG    rasa.core.policies.form_policy  - There is no active form
2020-01-03 13:34:26 DEBUG    rasa.core.policies.memoization  - Current tracker state [{'entity_issue_number': 1.0, 'intent_detail_issue': 1.0, 'slot_profile_0': 1.0, 'prev_utter_default': 1.0}]
2020-01-03 13:34:26 DEBUG    rasa.core.policies.memoization  - Launch DeLorean...
2020-01-03 13:34:26 DEBUG    rasa.core.policies.memoization  - Current tracker state [{'prev_utter_default': 1.0}]
2020-01-03 13:34:26 DEBUG    rasa.core.policies.memoization  - There is no memorised next action
2020-01-03 13:34:26 DEBUG    rasa.core.policies.mapping_policy  - There is no mapped action for the predicted intent, 'detail_issue'.
2020-01-03 13:34:26 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_4_FallbackPolicy
2020-01-03 13:34:26 DEBUG    rasa.core.processor  - Predicted next action 'action_listen' with confidence 1.00.
2020-01-03 13:34:26 DEBUG    rasa.core.processor  - Action 'action_listen' ended with events '[]'
2020-01-03 13:34:26 DEBUG    rasa.core.lock_store  - Deleted lock for conversation 'default'.
Sorry, I can't understand that. Could you rephrase it?
Your input ->

My fallback thresholds are set at 0.7, and I’m using 1.4.6 at the moment.

samscudder · January 3, 2020, 6:13pm

Here’s the story:

## detail_issue_corretor_story
* detail_issue
    - slot{"profile" : "corretor"}
    - show_issue_form
    - form{"name": "show_issue_form"}
    - form{"name": null }
    - action_detail_issue
    - action_clear_issue_number

The form isn’t ever called though. In my NLU I have examples with and without an issue number. If I try and use a phrase that doesn’t have the issue number embedded in it, it works, requesting the issue number:

Your input ->  show me issue details
2020-01-03 15:02:21 DEBUG    rasa.core.processor  - Received user message 'show me issue details' with intent '{'name': 'detail_issue', 'confidence': 0.781658947467804}' and entities '[]'
2020-01-03 15:02:21 DEBUG    rasa.core.processor  - Logged UserUtterance - tracker now has 429 events
2020-01-03 15:02:21 DEBUG    rasa.core.policies.form_policy  - There is no active form
2020-01-03 15:02:21 DEBUG    rasa.core.policies.memoization  - Current tracker state [{'prev_action_listen': 1.0, 'slot_profile_0': 1.0, 'intent_detail_issue': 1.0}]
2020-01-03 15:02:21 DEBUG    rasa.core.policies.memoization  - There is a memorised next action '350'
2020-01-03 15:02:21 DEBUG    rasa.core.policies.fallback  - NLU confidence threshold met, confidence of fallback action set to core threshold (0.7).
2020-01-03 15:02:21 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_1_AugmentedMemoizationPolicy
2020-01-03 15:02:21 DEBUG    rasa.core.processor  - Predicted next action 'detail_issue_form' with confidence 1.00.
2020-01-03 15:02:21 DEBUG    rasa.core.actions.action  - Calling action endpoint to run action 'detail_issue_form'.
2020-01-03 15:02:21 DEBUG    rasa.core.processor  - Action 'detail_issue_form' ended with events '['BotUttered(text: What is the issue number?, data: {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, metadata: {"issue_number": null, "profile": "corretor", "requested_slot": null, "user_name": null})', 'Form(detail_issue_form)', 'SlotSet(key: requested_slot, value: issue_number)']'
2020-01-03 15:02:21 DEBUG    rasa.core.processor  - Current slot values:
        issue_number: None
        profile: corretor
        requested_slot: issue_number
        user_name: None
2020-01-03 15:02:21 DEBUG    rasa.core.policies.form_policy  - There is an active form 'detail_issue_form'
2020-01-03 15:02:21 DEBUG    rasa.core.policies.memoization  - Current tracker state [{'prev_action_listen': 1.0, 'slot_profile_0': 1.0, 'intent_detail_issue': 1.0}]
2020-01-03 15:02:21 DEBUG    rasa.core.policies.memoization  - There is a memorised next action '350'
2020-01-03 15:02:21 DEBUG    rasa.core.policies.mapping_policy  - There is no mapped action for the predicted intent, 'detail_issue'.
2020-01-03 15:02:21 DEBUG    rasa.core.policies.fallback  - NLU confidence threshold met, confidence of fallback action set to core threshold (0.7).
2020-01-03 15:02:21 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_0_FormPolicy
2020-01-03 15:02:21 DEBUG    rasa.core.processor  - Predicted next action 'action_listen' with confidence 1.00.
2020-01-03 15:02:21 DEBUG    rasa.core.processor  - Action 'action_listen' ended with events '[]'
2020-01-03 15:02:21 DEBUG    rasa.core.lock_store  - Deleted lock for conversation 'default'.
What is the issue number?
Your input ->  20010385879595
2020-01-03 15:02:22 DEBUG    rasa.core.processor  - Received user message '20010385879595' with intent '{'name': 'inform_issue', 'confidence': 0.8992024064064026}' and entities '[{'start': 0, 'end': 14, 'value': '20010385879595', 'entity': 'issue_number', 'confidence': 0.9870757511758435, 'extractor': 'CRFEntityExtractor'}]'
2020-01-03 15:02:22 DEBUG    rasa.core.processor  - Current slot values:
        issue_number: 20010385879595
        profile: corretor
        requested_slot: issue_number
        user_name: None
2020-01-03 15:02:22 DEBUG    rasa.core.processor  - Logged UserUtterance - tracker now has 436 events
2020-01-03 15:02:22 DEBUG    rasa.core.policies.form_policy  - There is an active form 'detail_issue_form'
2020-01-03 15:02:22 DEBUG    rasa.core.policies.memoization  - Current tracker state [{'prev_action_listen': 1.0, 'slot_profile_0': 1.0, 'intent_detail_issue': 1.0}]
2020-01-03 15:02:22 DEBUG    rasa.core.policies.memoization  - There is a memorised next action '350'
2020-01-03 15:02:22 DEBUG    rasa.core.policies.fallback  - NLU confidence threshold met, confidence of fallback action set to core threshold (0.7).
2020-01-03 15:02:22 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_0_FormPolicy
2020-01-03 15:02:22 DEBUG    rasa.core.processor  - Predicted next action 'detail_issue_form' with confidence 1.00.
2020-01-03 15:02:22 DEBUG    rasa.core.actions.action  - Calling action endpoint to run action 'detail_issue_form'.
2020-01-03 15:02:22 DEBUG    rasa.core.processor  - Action 'detail_issue_form' ended with events '['SlotSet(key: issue_number, value: 20010385879595)', 'FollowupAction(action: action_detail_issue)', 'Form(None)', 'SlotSet(key: requested_slot, value: None)']'
2020-01-03 15:02:22 DEBUG    rasa.core.processor  - Current slot values:
        issue_number: 20010385879595
        profile: corretor
        requested_slot: None
        user_name: None
2020-01-03 15:02:22 DEBUG    rasa.core.processor  - Predicted next action 'action_detail_issue' with confidence 1.00.
2020-01-03 15:02:22 DEBUG    rasa.core.actions.action  - Calling action endpoint to run action 'action_detail_issue'.
2020-01-03 15:02:23 DEBUG    rasa.core.processor  - Action 'action_detail_issue' ended with events '['BotUttered(text: Change Password Issue - opened 03/01/2020 15:15:45, data: {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, metadata: {})', 'BotUttered(text: Description: Open Issue test with password change subject, data: {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, metadata: {})']'
2020-01-03 15:02:23 DEBUG    rasa.core.processor  - Current slot values:
        issue_number: 20010385879595
        profile: corretor
        requested_slot: None
        user_name: None
2020-01-03 15:02:23 DEBUG    rasa.core.policies.form_policy  - There is no active form
2020-01-03 15:02:23 DEBUG    rasa.core.policies.memoization  - Current tracker state [{'prev_action_detail_issue': 1.0, 'slot_profile_0': 1.0, 'intent_detail_issue': 1.0}]
2020-01-03 15:02:23 DEBUG    rasa.core.policies.memoization  - There is a memorised next action '14'
2020-01-03 15:02:23 DEBUG    rasa.core.policies.mapping_policy  - There is no mapped action for the predicted intent, 'inform_issue'.
2020-01-03 15:02:23 DEBUG    rasa.core.policies.fallback  - NLU confidence threshold met, confidence of fallback action set to core threshold (0.7).
2020-01-03 15:02:23 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_1_AugmentedMemoizationPolicy
2020-01-03 15:02:23 DEBUG    rasa.core.processor  - Predicted next action 'action_clear_issue_number' with confidence 1.00.
2020-01-03 15:02:23 DEBUG    rasa.core.actions.action  - Calling action endpoint to run action 'action_clear_issue_number'.
Change Password Issue - opened 03/01/2020 15:15:45
Description: Open Issue test with password change subject
2020-01-03 15:02:23 DEBUG    rasa.core.processor  - Action 'action_clear_issue_number' ended with events '['SlotSet(key: issue_number, value: None)']'
2020-01-03 15:02:23 DEBUG    rasa.core.processor  - Current slot values:
        issue_number: None
        profile: corretor
        requested_slot: None
        user_name: None
2020-01-03 15:02:23 DEBUG    rasa.core.policies.form_policy  - There is no active form
2020-01-03 15:02:23 DEBUG    rasa.core.policies.memoization  - Current tracker state [{'prev_action_clear_issue_number': 1.0, 'slot_profile_0': 1.0, 'intent_detail_issue': 1.0}]
2020-01-03 15:02:23 DEBUG    rasa.core.policies.memoization  - There is a memorised next action '0'
2020-01-03 15:02:23 DEBUG    rasa.core.policies.mapping_policy  - There is no mapped action for the predicted intent, 'inform_issue'.
2020-01-03 15:02:23 DEBUG    rasa.core.policies.fallback  - NLU confidence threshold met, confidence of fallback action set to core threshold (0.7).
2020-01-03 15:02:23 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_1_AugmentedMemoizationPolicy
2020-01-03 15:02:23 DEBUG    rasa.core.processor  - Predicted next action 'action_listen' with confidence 1.00.
2020-01-03 15:02:23 DEBUG    rasa.core.processor  - Action 'action_listen' ended with events '[]'
2020-01-03 15:02:23 DEBUG    rasa.core.lock_store  - Deleted lock for conversation 'default'.
Your input ->

samscudder · January 6, 2020, 2:31pm

Managed to fix this by separating into two intents. One with the issue number specified in the text and one where the issue number is picked up with the form.

In the story with the issue number specified, I added a {“issue_number”: “123456abc123e4”} after the intent, so that the nlu picked up that the issue_number comes from the intent. After this, the forms started working OK.

Topic		Replies	Views
Newbie problem intent and entity Rasa Open Source	0	166	March 20, 2024
Overlapping Entity Issue Feedback on Rasa Open Source	9	757	February 6, 2023
Policies and stories problem Rasa Open Source	4	1390	July 13, 2019
Failed to use AugmentedMemoizationPolicy to predict actions correctly Rasa Open Source	3	326	August 7, 2020
Name entity is not extracting Rasa Open Source	7	777	November 21, 2023

AugmentedMemoization breaking on entity detection

Related topics