I had a problem when the input was correctly recognized as alarm.activate intent and the slot x_selection_all (text type) was filled, other than alarm.activate action was predicted, or fallback policy was applied. I do not have any other stories like this with intent alarm.activate and filled slot x_selection_all. According to documentation
The MemoizationPolicy just memorizes the conversations in your training data. It predicts the next action with confidence 1.0 if this exact conversation exists in the training data, otherwise it predicts None with confidence 0.0 .
I would expect that MemoizationPolicy will be used and predict alarm.activate action because the input fits this exact story.
What helped was setting augmentation_factor to 0 in TEDPolicy in config file:
Where are you setting the slot? Can you post the debug output (use the --debug switch)?
Every single time I had intent accuracy > fallback threshold, and had a fallback action, I had a “there is no memorized next action” message in the debug output. It was always due to a situation not predicted in the stories.
and input “activate all alarms” I get this debug output:
Your input -> activate all alarms
2020-07-17 16:45:11 DEBUG rasa.core.tracker_store - Creating a new tracker for id '3ace16de573042f985236f6d1d8e3189'.
2020-07-17 16:45:11 DEBUG rasa.core.processor - Starting a new session for conversation ID '3ace16de573042f985236f6d1d8e3189'.
2020-07-17 16:45:11 DEBUG rasa.core.processor - Action 'action_session_start' ended with events '[<rasa.core.events.SessionStarted object at 0x7f57a016bdd8>, <rasa.core.events.ActionExecuted object at 0x7f57a016b400>]'.
2020-07-17 16:45:11 DEBUG rasa.core.processor - Current slot values:
x_selection_all: None
2020-07-17 16:45:12 DEBUG rasa.nlu.classifiers.diet_classifier - There is no trained model for 'ResponseSelector': The component is either not trained or didn't receive enough training data.
2020-07-17 16:45:12 DEBUG rasa.nlu.selectors.response_selector - Adding following selector key to message property: default
2020-07-17 16:45:12 DEBUG rasa.core.processor - Received user message 'activate all alarms' with intent '{'name': 'alarm.activate', 'confidence': 0.9985135197639465}' and entities '[{'entity': 'x_selection_all', 'start': 9, 'end': 12, 'value': 'all', 'extractor': 'DIETClassifier'}]'
2020-07-17 16:45:12 DEBUG rasa.core.processor - Current slot values:
x_selection_all: all
2020-07-17 16:45:12 DEBUG rasa.core.processor - Logged UserUtterance - tracker now has 5 events.
2020-07-17 16:45:12 DEBUG rasa.core.policies.memoization - Current tracker state [None, None, None, {}, {'slot_x_selection_all_0': 1.0, 'prev_action_listen': 1.0, 'intent_alarm.activate': 1.0, 'entity_x_selection_all': 1.0}]
2020-07-17 16:45:12 DEBUG rasa.core.policies.memoization - There is no memorised next action
2020-07-17 16:45:12 DEBUG rasa.core.policies.form_policy - There is no active form
2020-07-17 16:45:12 DEBUG rasa.core.policies.fallback - NLU confidence threshold met, confidence of fallback action set to core threshold (0.4).
2020-07-17 16:45:12 DEBUG rasa.core.policies.ensemble - Predicted next action using policy_1_TEDPolicy
2020-07-17 16:45:12 DEBUG rasa.core.processor - Predicted next action 'alarm.activate' with confidence 0.74.
2020-07-17 16:45:12 DEBUG rasa.core.actions.action - Calling action endpoint to run action 'alarm.activate'.
Which is okay but I’m still wondering why MemoizationPolicy wasn’t used to predict next action when the “conversation” exactly match my scenario?
2020-07-17 16:45:12 DEBUG rasa.nlu.classifiers.diet_classifier - There is no trained model for 'ResponseSelector': The component is either not trained or didn't receive enough training data.
Yep, I saw it. But I think it’s not an error, at least not related to my question. According to the documentation, I assume that ResponseSelector is used to handle similar intents with responses.md file.
The ResponseSelector NLU component is designed to make it easier to handle dialogue elements like Small Talk and FAQ messages in a simple manner. By using the ResponseSelector, you only need one story to handle all FAQs, instead of adding new stories every time you want to increase your bot’s scope.
I used to have this kind of intents but I’ve removed them and left ResponseSelector in the pipeline. When I put back these intents (with responses.md file) or remove ResponseSelector from the pipeline, it doesn’t have any effect on choosing the next action by TEDPolicy and this DEBUG output is gone.
So I still don’t understand why MemoizationPolicy is not used to predict next action when conversation exactly matches that one scenario.
Like I mentioned, when you get there is no memorised next action, it couldn’t find a matching story, so it’s not matching you story for some reason.
The memoization policy also has a max_history, and checks the tracker state. Looking at your log:
2020-07-17 16:45:12 DEBUG rasa.core.policies.memoization - Current tracker state [None, None, None, {}, {'slot_x_selection_all_0': 1.0, 'prev_action_listen': 1.0, 'intent_alarm.activate': 1.0, 'entity_x_selection_all': 1.0}]
2020-07-17 16:45:12 DEBUG rasa.core.policies.memoization - There is no memorised next action
I’m not sure why you’ve got None, None, None, {} in the tracker history as this could be the problem. It could be that an action (or actions) you have are returning None instead of returning an empty list. I suggest checking to see if they return an event or [].
P.S.
It shouldn’t be makiing any difference in this case, but (can’t recall where exactly), one of the rasa components used to split the intent name on _. It was hard coded, and I don’t know if they changed it (this was around version 1.4.6), so I’d use alarm_activate instead of alarm.activate.
I don’t know why tracker history contains None, None, None, {} when the conversation was started as a fresh session and no custom action was called. When the conversation continues, tracker history list is filled with regular state’s dictionaries such as {'slot_x_selection_all_0': 1.0, 'prev_action_listen': 1.0, 'intent_alarm.activate': 1.0, 'entity_x_selection_all': 1.0}. I will try to look closely what’s going on and if it’s a problem.
However, I probably figured out why the next action wasn’t predicted by the MemoizationPolicy even if the story structure matches the conversation. I changed story syntax to use intent with the entity:
## alarm activate all
* alarm.activate{"x_selection_all": "all"}
- alarm.activate
instead of SlotSet event -slot{"x_selection_all": "all"} as in my first post and it works fine now. Although the slot x_selection_all is automatically filled from the entity of the intent, it’s probably not the same as SlotSet event and using a slot to distinguish story flow is not appropriate here.
