Intent ranking, different measurement

Hi, I’m looking for the way I can get an intent ranking with no normalized confidence. That is, if I train with three intents that contain similar sentences, I would like the confidence ranking with, for example, 0.9, 0.8, 0.78 values. I want to use the intent ranking confidences as orchestator of several assistant and that is the reason of intents with similar sentences. With the config files I have seen in rasa docs, it brings me out intent rankings that sums 1, and then, in this cases, the confidences are aproximately 0.33. In my application (with a 0.85 confidence threshold), I wouldn’t say that these confidences are high enough althougth in fact, the sentence could be similar to the three intents. Other times you get a saturated result close to 1 (with other intents close to 0) which seems too high to me

What configuration/file/algorithm should I have to change?

Thanks

I have seen here Components that it is possible.

There’s also some additional parameters related to model confidence in 2.3.0 now: rasa/CHANGELOG.mdx at main · RasaHQ/rasa · GitHub

Hi, you mean model_confidence and constrain_similarities parameters? I have tried with cosine and True respectively but I get similar performances. One of the intent with aproximately 0.9 confidence while the rest obtain a negative value.

could you share the histogram output of a crossvalidation run?

constrain_similarities should be True btw

Here is the histogram. The pipeline is:

  - name: SpacyNLP
    model: "es_core_news_md"
  - name: SpacyTokenizer
  - name: SpacyFeaturizer
  - name: DIETClassifier
    epochs: 20
    entity_recognition: False
    model_confidence: cosine
    constrain_similarities: True

Can you try inner as model_confidence? The output confidence values will be the raw similarities that DIET computes to predict a label with no normalization / activation function on top of it. One thing to note is that the confidences will not lie in a bounded range, which means you’ll see values greater than 1 and possibly less than 0 too. Lmk if that helps.