Failed Story in test stories

[jalowiec]

Hello, I’ve a problem with rasa test. I had some issues there with my project so I decided to make it simple - one intent, one story and one test story (based on this intent and the story). After running rasa test I get my test story in failed_test_stories.yml. After running rasa test --fail-on-prediction-errors I get WrongPredictionException, altough when a type the same sentence in rasa shell the prediction is correct. Is there any reasonably explanation of it? In story_report.json my intent travel_office_general_info has precision = 1.0. Does it have to do with action_listen which has also precision = 1.0 in story_report.json. I will be grateful for any help, any tips, ideas. Regards

[AndreD (andre danner)]

Hi jalowiec,

I am hanging around with storytesting as well as you can see here:

End to end story testing; how to read results of failed_test_stories.yml

Hi all, I am hanging around for a while now without success regarding end to end testing. It all starts already how to interprete the results of the test run being written into failed_test_stories.yml. Currently the overall results show 3 of 30 test stories are correct, which results in having 27 in the failed_test_stories.yml. What I understood so far from the docs then is that one can expect the #tag to indicate where the test story deviates from the prediction. But here my problem starts - c…

I have currently two major topic with that:

1. Nobody seems to know how to read the failed test stories file. For my understanding the hash-tag (#) should indicate wrong predictions, which gives you a hint where the story fails.

Problem here: I have quite often more rejected stories in the file than hash-tags. → Why is that ?

2. I found differences between my CI/CD chain and my local test runs I am still in the middle of my investigation here, but honestly I have no clue what this might be, as all data- and config-files are the same …