[Solved]Predicted Action not following the story

I have two stories which shares some of the intents. The action is not being executed as per both stories. The story is getting stuck at following point

story 1(with intent 1)

  • Inform

-slot{‘id’=100}

-action_validate

story 2 (with intent2)

*Inform

-slot{‘id’=100}

-action_register

It is executing action_validate in place of action_execute. I have set max_history in memoization policy to 10, augmentation factor to 0. But the prediction for next action is not coming correct. I checked in debug mode and probability for next action is action_validate is 0.52 action_execute is 0.46 And the next action is being predicted by keras policy when we see in debug mode. Is that causing such a small margin or for this particular step keras policy is overriding memorization policy as we are passing union of both the policies to Agent.

@souvikg10 @akelad

the slot id - what type of slot it is?

what is making the difference between action_validate and action_register, it is not quite clear

So I have just attached the snippet of the story. The story goes like following

##story 1

*Intent1

-utter_validate

*Inform

-slot{‘id’=100}

-action_validate

##story 2

*Intent2

-utter_execute

*Inform

-slot{‘id’=100}

-action_execute

So the intent is different initially. Afterwards slots are filled which is same for both the cases. Then the action is executed. But it is not predicting the correct action as per the story.

can you set max_history to 2 instead, 10 is too much

@souvikg10 Not able to get it. Do i need to change max_history in keraspolicy as well? Also do i need to change the augmentation factor?

can you show some debug

You can also check my blog to understand the policies better

@souvikg10 . Thank you so much for you support.Though debug I am able to solve the problem.The current tracker list prints current tracker state which matches with the story and predicts next action. I fixed it using doing some changes in keras and memoization policy’s max history parameters. Also i nullified augmentation factor parameter.

1 Like

@akelad @souvikg10

Many people are facing the same issue.So I will write the complete solution for this.

python -m rasa_core.run -d models/dialogue -u models/nlu/default/chatter --debug --endpoint endpoints.yml

On debugging, I reached up to the point where action was not executed correctly as probability for the other action was little high( action_validate:0.52,action_execute:0.46 ) even though it has to predict action_execute as per the story. I checked the current tracker state at the point it was failing. It returns a list as follows

Current tracker state =[{'prev_action_save_userID': 1.0, 'intent_inform': 1.0, 'entity_param': 1.0}, {'intent_inform': 1.0, 'entity_param': 1.0, 'prev_utter_orgID': 1.0}, {'prev_action_listen': 1.0, 'intent_inform': 1.0, 'entity_param': 1.0}, {'intent_inform': 1.0, 'entity_param': 1.0, 'prev_action_save_orgID': 1.0}, {'intent_inform': 1.0, 'entity_param': 1.0, 'prev_utter_trnID': 1.0}, {'prev_action_listen': 1.0, 'intent_inform': 1.0, 'entity_param': 1.0}, {'intent_inform': 1.0, 'entity_param': 1.0, 'prev_action_save_trnID': 1.0}, {'intent_inform': 1.0, 'entity_param': 1.0, 'prev_utter_areaID': 1.0}, {'prev_action_listen': 1.0, 'intent_inform': 1.0, 'entity_param': 1.0}, {'intent_inform': 1.0, 'entity_param': 1.0, 'prev_action_save_areaID': 1.0}]

which is a list of length 10(my max_history was also 10) and it starts from prev_action_save_userID which is same in both stories. It meant that my story is not getting the first step i.e the intent in the tracker state list. I checked the core training file and changed my max_history to 30 so that it can capture full story. For better prediction of unknown, I also changed max_history in keras. Also,set augmentation factor to 0. And the bot was behaving perfectly fine.

from rasa_core.policies.memoization import MemoizationPolicy
from rasa_core.policies.keras_policy import KerasPolicy
from rasa_core.featurizers import MaxHistoryTrackerFeaturizer, BinarySingleStateFeaturizer, SingleStateFeaturizer
fallback = FallbackPolicy(fallback_action_name="utter_unclear",core_threshold=0.2, nlu_threshold=0.8)
agent = Agent(Domain.load(domain_file) , policies=[MemoizationPolicy(max_history=30),KerasPolicy(MaxHistoryTrackerFeaturizer(BinarySingleStateFeaturizer(),
                                            max_history=10)),fallback])
data = agent.load_data(training_data_file,augmentation_factor=0)
	agent.train(
				data,
				epochs = 700,
				batch_size = 50,
				validation_split = 0.0)
1 Like

@ashukrishna100 : Did you try it with just setting augmentation to 0 and max_history=2?

Essentially, I believe the max_history needs to be set to # of (bot) actions in your longest story at most. In your example above max_history=2 should be sufficient.

But it also seems that the example above

Is not the full example, right?

The real story is a bit long with many inform steps in between, I would say 4 intents for ‘inform’. However, I checked the list of events in current tracker state list and its length was 10. And it started from some intermediate ‘inform’ intent. Tht’s why the prediction was not going accurate as the tracker was getting confused in predicting next action/utterance.

however, if there was some different step in both the stories in somewhere middle,the max_history=2 would have worked. The prediction of next action depends on the tracker state and stories completely.

1 Like

@souvikg10 @ashukrishna100 @smn-snkl @Juste can you clear my doubt here.see below code of stories with max_history=3(Memoization) and max_history=5(Keras) ,I am using augmentation =20 as I have added max stories 20. so my input hi->intent1->intent2 is working fine and following story 3(memoization) but when input is hi->intent1->intent3 it still doing action 2(following memoization) as past history is matching to story3. So my doubt is Why not on based on “entity”:“value” it is predicting correct action i.e. action 3?

#story1
* greet - utter greet

#story 2                                                                                             
*intent1{"entity":"value"}
 - action 1

#story 3                                                                                               
* greet
 - utter greet
*intent1{"entity":"value"}
 - action 1    
*intent2{"entity":"value"}
- action 2

#story4                                                                                              
*intent3{"entity":"value"}
- action 3

if I change story4 like this

#story 4

  • greet
  • utter greet
  • intent1{“entity”:“value”}
  • action 1
  • intent3{“entity”:“value”}
  • action 3

then hi->intent1 is not working correctly,as it is predicitng based on Keras and prediciting wrong action, What should I do so that hi->intent1->intent2 and hi->intent1->intent3 both would work fine?

and when I use --augmentaion 0 then hi->intent1 is not following any story(no memoization) and predicting entirly based on keras.I am really not following these stories’s story :confused: .

as per my understanding augmentation factor and augmentation is different …Augmentation factor was being used in earlier version . With Setting augmentation = 0 in current setup it strictly follows the story …Also it gives priority error between memoization and augmentation policy for Priority 3

hey @ashukrishna100 can you show me your config.yml in which u code ur policy??? It’ll really helpful to me