Test scores for failed intents

Rasa Open Source
1

When I test my stories in test_stories.yml, is there a way we can see the probabilities returned for each test case (especially the failed tests)?

In failed_test_stories.yml I just see something like

- intent: greet # predicted nlu_fallback: hello there!

Can I also see the probability that was returned so I know in what direction to work the model? Does this exist already?

2

you should be able to see them in intent_errors.json file.

3

@jupyterjazz .Thanks. What I see in intent_errors.json is something like

{
    "text": "cu",
    "intent": "sendoff",
    "intent_prediction": {
      "name": "affirm",
      "confidence": 0.998573899269104
    }

However, the texts are not the exact text in my user text in test_stories.yml. so I am wondering where the text in the above is coming from. Is something wrong? This happens for many other cases as shown in intent_errors.json.

4

You’re right it won’t show failed intents from the test stories in intent_errors.json, my bad.

I’m not sure if it’s possible to see the probabilities for the failed stories. However, one thing you could do is run rasa shell nlu which would give you an opportunity to test how your model predicts intents. Since NLU is stateless you can ignore the context and only check what it predicts on “hello there”.

5

@jupyterjazz . Yea I know of rasa shell nlu and I use it. It’s only if you have 100 test stories, one cannot try for all of them.

Thanks.