Rasa NLU predicts unseen data with confidence of 1

Rasa Open Source
1

Hi, I tried training a Rasa NLU model. I have 10 intents and about 1500 examples (total). I used the default pipeline, and when evaluating the model with unseen data, it seems to be very confident in its answers. Almost all responses have a confidence of 0.999+, and even in a lot of times when the model was wrong, the confidence was exactly 1.

What can I do to investigate this issue? I need the confidence score

2

You should change your own pipeline according to your needs. I suggest using Tensorboard with Rasa.

It would also help to create an out_of_scope intent.