I am getting confidence scores below 0. My understanding is that the confidence score should be between 0 and 1. Can anyone help me understand what the lower and upper bounds on the confidence score are? What circumstances I would cause a negative value?
Hi @nik202 - it varies, but in one case, 10 examples, some have more.
I’ve since found that the issue occurs when I use Rasa 2.3.3, but when I re-train the model using 2.8.12 I don’t see negative confidence scores.
I was using the older version as I needed to recreate my customer’s setup, which is where the issue was raised (our software was not expecting a value below 0 so I am trying to understand if we need to update our software, or whether it was a bug in Rasa that’s now been fixed)
Do you know if this was a bug that was fixed? I just need to know if we should expect values below 0 in the future.
My understanding is the model confidence comes from a calculation of similarity between an utterance and the intents in the model. Looking at the 3 ways similarity can be calculated in Rasa (“auto”, “inner” and “cosine”) it appears that all 3 would need to be in positive space and therefore bounded by [0,1]. Is that right?
Appreciate your help!
@lostinlogic Alright, in that case, can be a known bug, but in the changelog, there is nothing mentioned about such issue, verified from this link: Rasa Open Source Change Log
Yes, you are right 
it much depends on cosine similarity (I guess), can you please share your Rasa 2.3.3 config.yml file for reference?
@nik202 thank you for the quick response!
Here is my config file, although it’s petty much the default (no changes).
config.yml (1.4 KB).
I can also share the trained model file if you like (looks like there is a file size limit here, it’s ~24MB)
We will continue to treat negative values as erroneous for now then.
@lostinlogic it’s still uncommented or commented?
@lostinlogic are you getting these negative value inside form or for common conversations too?
I kept it commented out.
I am only seeing for the NLU confidence scores. (from rasa shell nlu)
@lostinlogic please uncomment and train the full model first and check the same intent confidence score.