Can I ask some questions about 'In-data fraction' in the "rasa test" command

Hello I would like to ask why my (In-data fraction: 0.167) is so low when I execute the ‘rasa test’? What is his influence? What does he mean?

But when I execute rasa test core --stories…the scores are all 1. I don’t understand the principle. Can you explain it to me?

In-data fraction measures the percentage of the predicted actions that was made by the Rule Policy, Memoization Policy or Augmented Memoization Policy. So, 0.167 means about 17 percent of the actions in your test data was predicted by these memorizing stories.

rasa test is equivalent to rasa test core --stories ./tests . If you run rasa test core --stories . then it will recursively look for stories in all folders starting from the root folder so it will pick up stories in the data folder too.

Thank you for your response! How can I increase the In-data fraction value? What changes do I need to make? If rasa test is equivalent to rasa test core --stories ./tests Why are the values in the above two pictures different?

To increase your in-data fraction value, the test stories flow must be identical to the training stories, so that every prediction is done by any of the “memorizing” policy. So, I’m not sure if this is a metric you really want to be optimizing for.

To clarify the equivalence of rasa test:

  • rasa test is equivalent to rasa test core --stories ./tests for the core policy testing part
  • rasa test core will use the stories in your data folder to test the core policy
  • rasa test core --stories . will use all the stories in can find starting from your project’s root folder (assuming this is where you run the command).

Thank you so much

1 Like