Probability of next action in rasa core for sequence prediction always coming 1

Rasa Open Source
1

Rasa Core version:
rasa-core (0.10.4) rasa-nlu (not used)

Python version: 3.6.0

Operating system (windows, osx, …): Linux

Issue: I am using memoization and keras policy. Next action is predicted with memoization if the current action is there in training file and with keras policy if the current action is not found with exact match. My stories have no checkpoint and max history is set to 1, keras policy is using MaxHistoryTrackerFeaturizer.

I am always getting probability =1 for the prediction with small number(3) stories, with augmentation factor = 50 and epoch=100. When So I tried with epoch=50 and no change in augmentation factor which worked fine. But now with more(6-8) stories and augmentation factor = 50 and epoch= 50/100, I am again getting probability 1 in prediction. I tried with all combination of augmentation factor =[0, 20, 50] and epoch=[10, 20, 50, 100].

With requests with little change from story file gives probability=1 with keras policy.

Content of domain file (if used & relevant):

In domain file, all the slots are of type 'bool’
Stories format

## story_1
* intent_A
- action_A
- slot{key1 :val1, key2: val2}
* inform({key1 :val1, key2: val2})
- action_B
- slot{key3 :val3, key4: val4}
* inform({key3 :val3, key4: val4})
- action_C
- slot{key4 :val4, key5: val5}
* inform({key4 :val4, key5: val5})
- action_D
- slot{key6 :val6, key7: val7}
* inform({key6 :val6, key7: val7})
- action_E.contextreset

2

sorry what is the question? is this prediction wrong?

3

Yes. The prediction is wrong but the confidence probability = 1. This is happening when the request is different from the story in training data.

4

This problem is still not solved?

  1. Please remove slots from the “stories” file, no need to add them explicitly.
  2. Try changing slot type from bool to text
5
  1. Removing slots doesn’t solve the problem.
  2. I am giving feature presence by bool value. So will try not to change the format.
    And what’s the logic behind changing slot type from bool to text?
6

If you just want to check the presence of a slot, text slot will suffice.

The logic is that, if you have bool then the “presence” of that slot and “value” of that slot both are used for prediction.

To check just the “presence”, use text

7

No no. Both the “presence” of that slot and “value” of that slot both are used for prediction in my case, so can’t be modified to text.

8

if your max history is 1 it learns 1to1 correspondence between next action and previous event. There is no way it can distinguish between different story paths.

9

Okay, can you please share your config, domain and stories files (sufficient enough to reproduce the error)? Also, are you emitting any events from actions.py that would modify the tracker?

10

Yes yes I get it. Can you explain me how the augmentation works, in a little bit detail?

What happens if my bool action slots are like {A: true, B: true} with non-zero augmentation factor?

Does the training data now have actions with all combination of slot values for A and B[(true,true), (true,false), (false,true), (false, false)] bcs of non-zero augmentation factor? I don’t have any checkpoint set in the story.

11

please check the docs on augmentation: Policies

12

Yeah I have seen this. Every documentation on augmentation covers just simple stories with single action with no slots. Documentation doesn’t cover the type of stories I am talking about.

From the code also, we see stories are augmented at checkpoints, but it’s not very clear what happens when a story doesn’t have any checkpoint and what happens to slot values? That’s what I asked in the last question.

1 Like
13

Try stories with bool slot value. For example

## story_1
* intent_A
  - action_A
* inform({key1 :val1, key2: val2})
  - action_B
* inform({key3 :val3, key4: val4})
  - action_C
* inform({key4 :val4, key5: val5})
  - action_D
* inform({key6 :val6, key7: val7})
  - action_E

Now while predicting next action, give slot values something different. For example to get action_B, you need to give {key1 :val1, key2: val2}, but you give {key1 :val1, key2: val8(anything different than val2)}. Domain, story, config etc are in normal format and augmentation factor should be non-zero.

14

Augmentation randomly concatenate some stories from training dataset, that’s it. No mixing, e.g.

## story_1_1
* intent_A
  - action_A
* inform({key1 :val1, key2: val2})
  - action_B
* inform({key3 :val3, key4: val4})
  - action_C
* inform({key4 :val4, key5: val5})
  - action_D
* inform({key6 :val6, key7: val7})
  - action_E
* intent_A
  - action_A
* inform({key1 :val1, key2: val2})
  - action_B
* inform({key3 :val3, key4: val4})
  - action_C
* inform({key4 :val4, key5: val5})
  - action_D
* inform({key6 :val6, key7: val7})
  - action_E
1 Like
15

I found the reason and the stated the solution in the rasa git.