Indeterministic behaviour during "rasa test core"

Rasa Open Source
1

Hi, everyone. I have a problem regarding tests in RASA. Right now I have exactly 211 tests written and sometimes when I use rasa test core --stories tests it doesn’t work correctly. At most one test is failing and it happens 1 in 10 times (after new usage of “rasa train --force”). I’m not sure why this behaviour is so indeterministic.

I have some thoughts about it and I want to share it with You and maybe we come up with some solution.

  1. rasa train uses –augmentation, which takes a subsample of all possible stories created by merging original stories. By default, it takes subsample of 500. In my case, I have 74 original stories, which results in 574 stories used for training. Maybe in some runs augmentation doesn’t take into consideration some specific story?

  2. From a few examples of failed test stories, I have seen that these failed ones have an entity set in them. Maybe I’m writing test stories wrongly? Here is an example story:

## introduction confirm name
* intro
  - utter_greeting
  - utter_what_can_do
  - utter_if_name_question
* confirm
  - utter_name_question
* inform_name
  - greet_action
  - utter_age_question
* inform_age
  - utter_mood_question

In the intent inform_name I’m expecting an entity called name, but I don’t want to specify it in my story since the possibilities of user name are endless. And here is a test for this story:

## introduction confirm name entity found
* intro
  - utter_greeting
  - utter_what_can_do
  - utter_if_name_question
* confirm
  - utter_name_question
* inform_name{"name": "Christian"}
  - greet_action
  - utter_age_question
* inform_age{"age": "11"}
  - utter_mood_question

Should I specify these entities in my test story or not? Or maybe I should use tip written here. Cause I don’t care what will be the value of the entity and sometimes it can happen that for some reason entity isn’t recognized.

I would like to see Your ideas about the problem and how could I tackle that.

Thanks in advance for response!