Rasa test: how to get intent ranking on failed intents?

When running rasa test, I get a lot of usefull information. F.e. failed_stories.md

But I don’t see intent ranking.

How do I report the intent ranking on these failed stories during end-to-end testing??

A little bit like interactive training gives me the intent ranking, f.e. in Rasa X:

When running your bot in the shell have you tried using rasa shell --debug ? This gives you a lot more information than just running it normally.

Another suggestion is that each bot response has 'intent_ranking" which similar to your screenshot contains all the probabilities of each intent for a user’s message. Perhaps you could create a new custom action to test a bot’s reponse, so you could just do:

   query = tracker.latest_message
   print(query)

This will just print the bot response to the terminal in which you’re running your actions and you will be able to see the intent rankings within that.

P.s. tracker.latest_message[‘intent_ranking’] should return just the probability scores if you want that

Hope this helps :slight_smile:

Alex, thanks for your reply!

I tried the --debug option: gives a lot of information, but I still can’t find he (failed) intent ranking.

So, I puzzled a while on your second suggestion. Seems I lack some experience with the tracker code

So for now, it would take me to much time to figure it out.

I’ll just stick to Rasa X interactive learning for getting the ranking.

Thanks anyway!

:blush: Had forgotton about the split train test, which is producing an intent histogram with confidence. Or the alternative cross validation test.

I was little bit confused, because the rasa test on end-to-end stories reported an intent histogram and intent config with no errors. Reason is, end-to-end-test doesn’t split intents, I guess