[Solved]Predicted Action not following the story

Rasa Open Source
1

I have two stories which shares some of the intents. The action is not being executed as per both stories. The story is getting stuck at following point

story 1(with intent 1)

  • Inform

-slot{‘id’=100}

-action_validate

story 2 (with intent2)

*Inform

-slot{‘id’=100}

-action_register

It is executing action_validate in place of action_execute. I have set max_history in memoization policy to 10, augmentation factor to 0. But the prediction for next action is not coming correct. I checked in debug mode and probability for next action is action_validate is 0.52 action_execute is 0.46 And the next action is being predicted by keras policy when we see in debug mode. Is that causing such a small margin or for this particular step keras policy is overriding memorization policy as we are passing union of both the policies to Agent.

2

@souvikg10 @akelad

3

the slot id - what type of slot it is?

what is making the difference between action_validate and action_register, it is not quite clear

4

So I have just attached the snippet of the story. The story goes like following

##story 1

*Intent1

-utter_validate

*Inform

-slot{‘id’=100}

-action_validate

##story 2

*Intent2

-utter_execute

*Inform

-slot{‘id’=100}

-action_execute

So the intent is different initially. Afterwards slots are filled which is same for both the cases. Then the action is executed. But it is not predicting the correct action as per the story.

5

can you set max_history to 2 instead, 10 is too much

6

@souvikg10 Not able to get it. Do i need to change max_history in keraspolicy as well? Also do i need to change the augmentation factor?

7

can you show some debug

You can also check my blog to understand the policies better

8

@souvikg10 . Thank you so much for you support.Though debug I am able to solve the problem.The current tracker list prints current tracker state which matches with the story and predicts next action. I fixed it using doing some changes in keras and memoization policy’s max history parameters. Also i nullified augmentation factor parameter.

1 Like
9

@akelad @souvikg10

Many people are facing the same issue.So I will write the complete solution for this.

python -m rasa_core.run -d models/dialogue -u models/nlu/default/chatter --debug --endpoint endpoints.yml

On debugging, I reached up to the point where action was not executed correctly as probability for the other action was little high( action_validate:0.52,action_execute:0.46 ) even though it has to predict action_execute as per the story. I checked the current tracker state at the point it was failing. It returns a list as follows

Current tracker state =[{'prev_action_save_userID': 1.0, 'intent_inform': 1.0, 'entity_param': 1.0}, {'intent_inform': 1.0, 'entity_param': 1.0, 'prev_utter_orgID': 1.0}, {'prev_action_listen': 1.0, 'intent_inform': 1.0, 'entity_param': 1.0}, {'intent_inform': 1.0, 'entity_param': 1.0, 'prev_action_save_orgID': 1.0}, {'intent_inform': 1.0, 'entity_param': 1.0, 'prev_utter_trnID': 1.0}, {'prev_action_listen': 1.0, 'intent_inform': 1.0, 'entity_param': 1.0}, {'intent_inform': 1.0, 'entity_param': 1.0, 'prev_action_save_trnID': 1.0}, {'intent_inform': 1.0, 'entity_param': 1.0, 'prev_utter_areaID': 1.0}, {'prev_action_listen': 1.0, 'intent_inform': 1.0, 'entity_param': 1.0}, {'intent_inform': 1.0, 'entity_param': 1.0, 'prev_action_save_areaID': 1.0}]

which is a list of length 10(my max_history was also 10) and it starts from prev_action_save_userID which is same in both stories. It meant that my story is not getting the first step i.e the intent in the tracker state list. I checked the core training file and changed my max_history to 30 so that it can capture full story. For better prediction of unknown, I also changed max_history in keras. Also,set augmentation factor to 0. And the bot was behaving perfectly fine.

from rasa_core.policies.memoization import MemoizationPolicy
from rasa_core.policies.keras_policy import KerasPolicy
from rasa_core.featurizers import MaxHistoryTrackerFeaturizer, BinarySingleStateFeaturizer, SingleStateFeaturizer
fallback = FallbackPolicy(fallback_action_name="utter_unclear",core_threshold=0.2, nlu_threshold=0.8)
agent = Agent(Domain.load(domain_file) , policies=[MemoizationPolicy(max_history=30),KerasPolicy(MaxHistoryTrackerFeaturizer(BinarySingleStateFeaturizer(),
                                            max_history=10)),fallback])
data = agent.load_data(training_data_file,augmentation_factor=0)
	agent.train(
				data,
				epochs = 700,
				batch_size = 50,
				validation_split = 0.0)
3 Likes
10

@ashukrishna100 : Did you try it with just setting augmentation to 0 and max_history=2?

Essentially, I believe the max_history needs to be set to # of (bot) actions in your longest story at most. In your example above max_history=2 should be sufficient.

But it also seems that the example above

Is not the full example, right?

11

The real story is a bit long with many inform steps in between, I would say 4 intents for ‘inform’. However, I checked the list of events in current tracker state list and its length was 10. And it started from some intermediate ‘inform’ intent. Tht’s why the prediction was not going accurate as the tracker was getting confused in predicting next action/utterance.

however, if there was some different step in both the stories in somewhere middle,the max_history=2 would have worked. The prediction of next action depends on the tracker state and stories completely.

1 Like
12

@souvikg10 @ashukrishna100 @smn-snkl @Juste can you clear my doubt here.see below code of stories with max_history=3(Memoization) and max_history=5(Keras) ,I am using augmentation =20 as I have added max stories 20. so my input hi->intent1->intent2 is working fine and following story 3(memoization) but when input is hi->intent1->intent3 it still doing action 2(following memoization) as past history is matching to story3. So my doubt is Why not on based on “entity”:“value” it is predicting correct action i.e. action 3?

#story1
* greet - utter greet

#story 2                                                                                             
*intent1{"entity":"value"}
 - action 1

#story 3                                                                                               
* greet
 - utter greet
*intent1{"entity":"value"}
 - action 1    
*intent2{"entity":"value"}
- action 2

#story4                                                                                              
*intent3{"entity":"value"}
- action 3

if I change story4 like this

#story 4

  • greet
  • utter greet
  • intent1{“entity”:“value”}
  • action 1
  • intent3{“entity”:“value”}
  • action 3

then hi->intent1 is not working correctly,as it is predicitng based on Keras and prediciting wrong action, What should I do so that hi->intent1->intent2 and hi->intent1->intent3 both would work fine?

and when I use --augmentaion 0 then hi->intent1 is not following any story(no memoization) and predicting entirly based on keras.I am really not following these stories’s story :confused: .

13

as per my understanding augmentation factor and augmentation is different …Augmentation factor was being used in earlier version . With Setting augmentation = 0 in current setup it strictly follows the story …Also it gives priority error between memoization and augmentation policy for Priority 3

14

hey @ashukrishna100 can you show me your config.yml in which u code ur policy??? It’ll really helpful to me

2 Likes
15

Hi Souvik Ghosh, could you please look into this. my bot is failing in between the conversation stories.yml

  • story: flow testing steps:
    • intent: dress_recommendation
    • action: utter_ask_user_gender
    • intent: user_gender
    • entities:
      • gender: male
      • gender: female
    • slot_was_set:
      • gender: “male”
    • slot_was_set:
      • gender: “female”
    • action: utter_ask_user_bodytype
    • intent: user_bodytype
    • entities:
      • bodytype: skinny
      • bodytype: slim
      • bodytype: medium
      • bodytype: heavy
      • bodytype: obese
    • slot_was_set:
      • bodytype: “skinny”
    • slot_was_set:
      • bodytype: “slim”
    • slot_was_set:
      • bodytype: “medium”
    • slot_was_set:
      • bodytype: “heavy”
    • slot_was_set:
      • bodytype: “obese”
    • action: utter_ask_user_location
    • intent: user_location
    • entities:
      • location: “delhi”
    • slot_was_set:
      • location: “delhi”
    • action: action_get_temperature
    • action: utter_user_temp

when try to run this, the code is working till 2nd step, it is giving options to choose gender using payload buttons, once choose anyone option, it is not going anywhere from there.(it is not predicting the next action). any help would be much appreciated I’m using rasa 2.0 open source

16

The story format is incorrect, You can’t have all slot was set event placed if they are categorical values and drive the conversation.

Slot was set predicts the next action and hence you need to set them according to how the story is supposed to be

17

Thank you responding Souvik ghosh, can you let me know how to achieve this? For instance, user first asks what should I wear today? then the bot should ask the user’s gender, and next ask user’s bodytype, then ask user’s location so that I hit the weather api and get the temperature. and I want to pass this gender, bodytype and weather details to some machine learning model and get the result from it to suggest user. it will be great help you tell me how to achieve this and how to pass this slots to machine learning model and post the result back to the user on the chatbot

18

Can you share your config.yml file?

19

Stories get mix with each other how to use it different different

  • story: login balance steps:
    • intent: login.balance
    • action: utter_mobile_number
    • intent: mobile_no entities:
      • mobile_number: 1234567890
    • action: action_mobile_no
    • intent: otp_number entities:
      • otp: 123056
    • action: action_otp
    • action: action_Account_Balnce

other story —

  • story: check ministatement steps:
    • intent: check.ministatement
    • action: utter_mobile_number
    • intent: mobile_no entities:
      • mobile_number: 1234567890
    • action: action_mobile_no
    • intent: otp_number entities:
      • otp: 123056
    • action: action_otp
    • action: utter_account_number
    • intent: account_no entities:
      • account_number: 1234
    • action: action_account_no
    • action: action_mini_statement

After action_otp in mini statement is goes to action_accountbalance