Memoization Policy with max history

How does Memoization Policy with max history argument chooses its next action? Does it need to be up to max histroy identical to stories or just a sub part? I recognize that in my case MemoizationPolicy picts wrong next action although this is not part of story. When using max history = 2, in my case just the prev action might be encountered in stories.

I’m not sure i entirely understand your question, but if a history of 2 (or whatever your max history value is) is available, it takes that amount of history into consideration

Yea so the last 2 states have to coincide with one part of one of the stories or with one whole story?

from what i noticed is one full story.

* welcome
- utter_welcome
* greet
- utter_greet

if your max history is 2, this story needs to have an exact match.

that’s my presumption

1 Like

Must this be part of any story or one whole story itself with an ID?

@datistiquo Having a question regarding this as well. And maybe we could collaboratively work on a more extensive documentation of the entire policy part.

To my understanding after reading the featurizer and policies docs, it sounds to me like every single event

e.g. running a bot action, receiving a user message, setting slots

creates a new state. However, the memoization policy only takes into account the last max_history states at prediction time. An example: Let’s take the following story

* welcome
  - utter_welcome
* greet 
  - utter_greet
  - utter_how_can_i_help

My tracker event history overall shows 11 events for this example: (only showing event types for length reasons)

1. user (user's welcome utterance)
2. action (utter_welcome action)
3. bot (bot's welcome utterance)
4. user (user's greet utterance)
5. action (utter_greet action)
6. bot (bot's greet utterance)
7. action (utter_how_can_i_help action)
8. bot (bot's how_can_i_help utterance)
9. action (listen)

However, we only care about the last (let’s say 3) states at prediction, e.g. when predicting the action after utter_greet the featurized states will look something like:

[
    {'prev_action_utter_welcome': 1.0, 'intent_welcome': 1.0},  # oldest featurized prediction state
    {'prev_action_listen': 1.0, 'intent_greet': 1.0},
    {'prev_utter_greet': 1.0, 'intent_greet': 1.0}  # last featurized prediction state
]

These are essentially pairs of (last_user_message, last_bot_action) and could be read like:

  1. Initially, I responded with utter_welcome to user’s intent_welcome (beginning of conversation)
  2. Then, I listened and and received intent_greet
  3. Just now, I responded with utter_greet to user’s intent_greet

Additionally, the set slots and entities are also included and in these states and considered during prediction.

So for this example, you’d already need a max_history=3 even though there are just 2 turns. Is this correct so far @akelad @souvikg10?

However, I’m still unclear about a few other memoization related things:

Augmentation Factor

I see that data is augmented generally before training and seems to not be treated differently depending on the policy that is being trained. If a story is only 3 prediction events long, but max_history is 5, then it will only be memorized if I randomly walked any of the augmented paths that include my story, correct?

This would essentially mean my augmentation factor should be so large, that almost any combination of story appears in the training data at least once. Shouldn’t I use no augmentation for Memoization though? Which would cause other ML policies to generalize worse though, right?

AugmentedMemoizationPolicy

Shouldn’t I almost always use this one instead? Since memoization takes slots into account, I typically don’t want any previously set slots to mess up my memoization, do I?

Maybe I’m getting it wrong, but let’s say I have this story:

* thanks
  - utter_thanks

But if the first thing my bot does is ask the user for its name and store it in a slot, then my training story (with slot name = None) and the real context (name = some name) will always differ, causing the memoization to never match this story, correct?

Inconsistent behavior due to max_history

If my max_history is small, what happens if the same last events are followed by multiple, different events? E.g. I could have

* intent1
  - utter2
* intent3
  - utter4

and

* intent5
  - utter2
* intent3
  - utter6

If max_history is so short that it only spans across utter2 & intent3, what would it do? Not predict anything? Predict utter4 or utter6 randomly?

Thanks a lot @akelad! Sorry for this very long post. Just saw many questions around this topic on the forum and lots of unclarity that I thought would be useful to address. Hope you’re the right person for this.

2 Likes

@smn-snkl I have provided a detailed solution for this, Kindly have a look.

@ashukrishna100 : Thanks for sharing your insights.

I was, however, rather looking for the algorithmic explanation so I can make sure I choose and configure my policies appropriately.

@smn-snkl That are questions I asked myself too in the past. Like: Need the whole story block matches the story or just a part of it to be used by MemoPolicy?

I think a state used for prediction is this featurized tracker state at one time which consists of intents, last actions, entities…And max_history means you have max_history of them using for prediction.

1.) memoization policy- this policy is used to remember the events as per story.

2.)keras policy: this policy is used to predict next action if the user asks something different from stories.

However,tracker depends on max_history parameters in case path of two/more stories are very similar. If we pass max_history parameter to memoization policy it compares the user path with story path and if it is same,predict the next action correctly.

If we pass max_history parameter to keras policy,it tunes our model for unknown events/not exactly same as stories.

Augmentation factor stitches stories and then the policies(keras and memoization) act on those stitched stories. You can find those sticthed stories in models/dialogue.

1 Like

[ {‘prev_action_utter_welcome’: 1.0, ‘intent_welcome’: 1.0}, # oldest featurized prediction state {‘prev_action_listen’: 1.0, ‘intent_greet’: 1.0}, {‘prev_utter_greet’: 1.0, ‘intent_greet’: 1.0} # last featurized prediction state ]

Your list of tracker contains 3 elements i.e. your max history is 3, if u pass max history 30 the list will contain 30 elements

@datistiquo @ashukrishna100: The issue is that the exact story needs to match. Thus, if you increase the max_history to 30 as you @ashukrishna100 for example did, you’ll run into quite some issues.

Let me show you a test I just performed. Let’s say I only have this one story:

* greet
  - utter_greet

With max_history=30 at the beginning of the conversation my tracker will be

[None, ... 28x None ..., None]

If I now say ‘Hi’ it will answer with ‘Hi’ as well because the featurized tracker state matches the training state:

[
	None, ... 27x None ..., None,
	{'intent_greet': 1.0, 'prev_action_listen': 1.0}
]

However, if I say ‘Hi’ again my tracker will look like this:

[
	None, ... 26x None ..., None,
    {'intent_greet': 1.0, 'prev_action_listen': 1.0},
	{'intent_greet': 1.0, 'prev_utter_greet': 1.0}
]

This however will not match the memorized training state from above, thus leading to:

2018-12-13 15:05:22 DEBUG rasa_core.policies.memoization - There is no memorised next action

Therefore, I believe I only have two options in this scenario:

  1. Reduce max_history=1 will make sure I will always correctly classify this exact story. However, other longer stories potentially won’t work anymore
  2. I use augmentation to make sure my short story from above is augmented in a way that stories like the second one (see below) will be memorized as well. However, the tricky thing is I have no real control over augmentation. Thus, I would need a really high augmentation factor to make sure my story appears at any position in a tracker state with max_history=30 the exact same way.

[None, … 26x None …, None, {‘intent_greet’: 1.0, ‘prev_action_listen’: 1.0}, {‘intent_greet’: 1.0, ‘prev_utter_greet’: 1.0}]

So both options seem unpromising individually. Therefore, I believe the way to go is:

  1. Combine both: Use a small max_history, e.g. 2-5 to match very simple stories like the ones above. That will cover all your question - answer use cases. However, the higher your max_history, the larger your augmentation_factor should be. E.g. if I have two stories like the one above (let’s say we also have *goodbye - utter_goodbye) and I set max_history=2 even though both stories have length 1, then I will definitely need augmentation to cover greet -> goodbye as well as goodbye -> greet appearing anywhere in the conversation. You know what I mean?
  2. Rely on an ML-based policy for everything else

What 1. allows you to do at least is that if augmentation is sufficiently large I detect all combinations of my stories so I’ll definitely find any combination of greet and goodbye, e.g. with max_history=2 and augmentation=50 no matter what I say greet -> greet, greet -> goodbye, goodbye -> goodbye, or goodbye -> greet all of them will match a training example.

That is also what my test with ~30 smalltalk stories showed:

  • greet -> goodbye : There is a memorised next action ‘45’
  • goodbye -> goodbye : There is a memorised next action ‘0’
  • goodbye -> greet : There is a memorised next action ‘46’

However, I also already ran into the issue that greet -> greet was apparently not trained through augmentation, thus greet -> greet was predicted using the KerasPolicy …

2018-12-13 15:46:51 DEBUG    rasa_core.policies.memoization  - Current tracker state [{'prev_action_listen': 1.0, 'intent_greet': 1.0}, {'prev_utter_greet': 1.0, 'intent_greet': 1.0}]
2018-12-13 15:46:51 DEBUG    rasa_core.policies.memoization  - There is no memorised next action
2018-12-13 15:46:51 DEBUG    rasa_core.policies.form_policy  - There is no active form
2018-12-13 15:46:51 DEBUG    rasa_core.policies.ensemble  - Predicted next action using policy_0_KerasPolicy

Which already shows the fallacy of the memoization policy. So for 30 simple stories, augmentation needs to be >> 50 already, potentially slowing down training a lot, especially when combined with a KerasPolicy that has max_history >> 3 as well.

1 Like

That makes sense. And I also have been in entanglement with this too. Because when you have your happy paths of your story and user writes hiin between or you just create one additoinal new intent you don’t want to write all possible variations with this new intent? This would blow up exponentially…I hope Keras will learn that somehow by itself? But this I found difficult. If feel that like eg slots don’t have a much impact for prediction? @smn-snkl the augmentaion factor is shared by all policies, right?

It would be nice if the devs like @akelad or @Juste clarify. I also miss in the docs the explanations of all basic components for Rasa. Many underlying stuff is very fuzzy.

@smn-snkl Do you mind chatting somehow privately (mail)?

@datistiquo

I do agree on augmentation factor part. For bigger stories we can keep augmentation factor high which will further help in better prediction of next action but one thing is very clear,if we have similar stories then predicted actions become too ambiguous with probabilities with very little difference. For longer and similar stories we can keep augmentation factor and max_history both to be high.

Sorry about not getting back to you sooner, December has been a busy month.

Most of what you (@smn-snkl and @datistiquo) said makes sense! Increasing max_history to 30 would indeed not be ideal, because then if the conversation goes even slightly off course and you don’t have a story for it, it will derail.

The ideal max_history is of course up to what your stories look like, but 3-6 is generally enough. You can experiment with the evaluation script to see whether you need to increase it. @smn-snkl so a max history of 2 might be ok for a bit, but as soon as you have a bit more complicated stories you’ll probably need to increase it. E.g. if you have these two stories:

## example1
* greet
  - utter_greet
* greet
  - utter_already_said_hi

## example2
* greet
  - utter_greet

You’d need a max history of 3 here to take into account the previous utter_greet.

As for augmentation, it doesn’t need to grow exponentially large, the ML policies will take care of a lot of stuff. E.g. if you go off track from your stories, the KerasPolicy may jump in and then eventually when your conversation state matches with a conversation again, then the Memo policy can jump back in.

I hope that clarifies some things? I’m off for the holidays now, but will be happy to discuss this further when I’m back. Merry christmas!

1 Like

Thanks @akelad, that’s confirming my thoughts!

Sure. Let me know how I can reach out to you.

Hi all, just reopening this as we are tearing our hair out a bit with memoization. Specifically, an issue @smn-snkl raised earlier but seems to have dropped out - how memoization handles slot setting.

We have a fairly simple story, as an example:

  • take_action
    • utter_actions_menu
  • create_livewire
    • utter_confirm_livewire_intent
    • utter_ask_subject
  • select{“subject”: “Climate Change”}
    • utter_ask_livewire_content

The problem we are facing is that try whatever we can, no matter the combination of max history, aug factor, nothing, we just cannot get Memoization to handle that last step - actions are predicted right up until utter_ask_subject, then the user enters something, and it fails.

We’ve tried including explicitly the “- slot{“subject”: “Climate Change”}” step, or removing it, and it makes no difference. We had a formAction action in there, made no difference. Have tried max histories from 2 to 30, no difference. Have reordered the story, and memoization works right up until it hits a slot fill, and then it fails, hence we think this is very likely to do with how the policy interacts with slot filling.

Anyone else run into same thing and solved it? @akelad Is this a known issue at all?

Thanks a ton,

Luke

Edit - what we have just realized is that it seems if the user types the exact same thing as in the training policy, then memoization (sometimes) recognizes it. But of course in almost all cases the user is not going to type the same content.

Anyone know if there’s a way to tell memoization to look at only what slot was filled, not the specific content that was filled - am wondering if maybe by passing a different state_featurizer? At present we’re just passing none. Is there a featurizer that ignores the content of slots?

Do you mean sometimes a different intent gets recognised?

@luke-jordan what kind of slots are you using? For some slot types the value matters, for others it only matters whether the slot is filled or not, see this