Policies and stories problem

Rasa Open Source
1

Hello all I am pretty new to rasa, was reading up on policies lately because I met some problems with my stories. My stories ended half way through a series of form actions and am not sure why is that happening. It was able to detect the correct NLU but it did not predict the right actions.

Which was why I decided to read up on policies since it’s the core of rasa. I am trying to understand the following policies:

  • KerasPolicy
  • MemoizationPolicy
  • AugmentedMemoizationPolicy

1) How do people derive the values of epochs needed for KerasPolicy?

2) How do people derived the max history for MemoizationPolicy/ KerasPolicy/ AugmentedMemoizationPolicy? Ex: Do they count the number of intents a story have or count the number of events that it has?

3) When do you use AugmentedMemoizationPolicy vs MemoizationPolicy.

I have a few categorical slots where my stories would depend on therefore I assume/ think that AugmentedMemoizationPolicy is not suited for me but am still unsure about it.

Lastly, I read that augmentation factor would glue the stories randomly together. My question would be:

4) How do I view the glued stories and how does setting it to 0 
affect my stories, compare to setting it at the default value of
20? Does it make my stories more accurate?

I am unsure about all this.

Thanks!

1 Like
2

Hi @leleland welcome to the community! Thanks for your great questions, I think this is valuable feedback for what we can improve in our documentation.

  1. There’s no real guidelines to this. Basically, when KerasPolicy is training, pay attention to the accuracy values while it’s training. When it hits a relatively stable level, that means it doesn’t need to train any further. So for the next round you can set the number of epochs to that number you observed
  2. This is very dependent on your stories. It basically defines the maximum amount of past events in a conversation (this includes UserUttered and ActionExecuted events, not SlotSet events) Rasa should pay attention to when making a prediction. In general, the default value of that (6) is more than enough, it’s only when you have longer stories that you may have to increase that. To illustrate this, here’s two examples. For the following stories, the history that’s required would be 2 for Rasa to predict the last action:
  - utter_ask_existing_account
* affirm
  - utter_enter_password

  - utter_ask_user_happy
* affirm
  - utter_great

For a stories like this, it would be 3:

  - utter_confirm_input
  - utter_ask_current_user
* affirm
  - utter_enter_password

  - utter_deny_input
  - utter_ask_current_user
* affirm
  - utter_contact_service

  1. AugmentedMemoizationPolicy is for when you have your stories in shorter blocks, and also want to be able to jump from segment to segment (for example, the user changing his mind about what he wants to do). It has a “forgetting mechanism” that will forget a certain amount of steps in it’s history and try to find a match in your stories with a reduced history. In general it’s a very good policy, and I would strongly recommend it. I don’t think having categorical slots should be a problem

  2. The glued stories I think get dumped to your model after training. They’re only relevant to the KerasPolicy, it needs some augmented data to be able to train accurately, so if you’re using that policy I would suggest using the agumentation factor.

Let me know if all that makes sense!

5 Likes
3

Hello @akelad thanks for your reply!! It really did help me to understand better now. However I might have some extra questions, hope you don’t mind!

  1. So now that you have said it, I went to take a look at the accuracy and yes at 50 epochs, the accuracy is close to 1. So I would assume I would just need 50 epochs for the next round of training!

  2. So for this story:

* greet
- RestaurantForm
- form{"name": "restaurant_form"}
- form{"name": null}
- CuisineForm
- form{"name": "cuisine_form"}
- form{"name": null}

I would assume that the value would be (2) since SlotSet events does not affect it? Also, when do I start counting the number of events? From the first intent all the way to the last action or half way through the stories?

  1. Oh my, that is very interesting. Without you explaining this to me, I wouldn’t have know what it actually does! So in other words, MemoizationPolicy is to remember a portion of the events base on the max_histories while AugmentedMemorizationPolicy is to forget the number of events base on max_histories so that it can go to another part of the story.

  2. I guess I have to research more on what KerasPolicy actually does! So how do you determine what value of the augmentation factor it should be? I have read that in order to strictly follow the stories, augmentation 0 is being use to train the model. (Tried this but didn’t get to even run my story, it just went to fallback action all the way!)

In summary, I thank you for your time and patience to read and answer my questions. It was indeed helpful although it left me with many many other doubts!

1 Like
4

Hey @akelad don’t mind me intruding in but this is really very helpful for beginners like me to understand about policies. Would be great if such explaination was found in the Docs. Will probably try to ask you some questions too but am interested in your reply to @leleland.

Thanks.

2 Likes
5

  1. For that Story, the value would be 2 when predicting the action “CuisineForm”, yes. The way you count, is from the last action that is being predicted upwards

  2. Kind of yeah, MemoizationPolicy will always pay attention to the full max_history value, AugmentedMemoizationPolicy may use values smaller than that

  3. Yeah KerasPolicy is to be used in combination for one of the other ones. And since it’s an LSTM training, you will need to use augmentation (probably higher than 20) for it to learn the correct behaviour

1 Like