Failed test stories have no comments

I am using rasa test to test the test stories in my tests/test_stories.yml file. However, I am having a hard time interpreting the results/failed_test_stories.yml file. Some of my test stories are included in the results file, but without any comments or corrections. I think that this is a bug, and that those test stories actually didn’t fail and shouldn’t have been included in the results file. But I’m not sure, and I wanted to see if anyone else has experience this problem.

In this screenshot, you can see that the first story has helpful comments, which indicate where the predicted action deviated from the expected action. However, in the second story, no such comments are made.

2 Likes

Hi Alex ,

Did you found a solution to the failed_test_stories.yml i have the same issues, the stories work with the input correctly , but they appear in the file failed_test_stories.yml without any comments.

Did you manage to fix it ?

Hi @Aleksandar.papazov, I did not find a solution.

Same issue here!

Looks like you can get some hint as to what rasa thinks is the error when using

rasa test --fail-on-prediction-errors

I have exactly the same issues.

I also put it here:

It seems that no one seems to have an answer.

As I lately also realized differences in between my CI/CD chain results and local testing I lost all my trust in that capability, which I find is crucial to have.