Intent ranking, different measurement

Rasa Open Source
1

Hi, I’m looking for the way I can get an intent ranking with no normalized confidence. That is, if I train with three intents that contain similar sentences, I would like the confidence ranking with, for example, 0.9, 0.8, 0.78 values. I want to use the intent ranking confidences as orchestator of several assistant and that is the reason of intents with similar sentences. With the config files I have seen in rasa docs, it brings me out intent rankings that sums 1, and then, in this cases, the confidences are aproximately 0.33. In my application (with a 0.85 confidence threshold), I wouldn’t say that these confidences are high enough althougth in fact, the sentence could be similar to the three intents. Other times you get a saturated result close to 1 (with other intents close to 0) which seems too high to me

What configuration/file/algorithm should I have to change?

Thanks

2

I have seen here Components that it is possible.

3

There’s also some additional parameters related to model confidence in 2.3.0 now: rasa/CHANGELOG.mdx at main · RasaHQ/rasa · GitHub

4

Hi, you mean model_confidence and constrain_similarities parameters? I have tried with cosine and True respectively but I get similar performances. One of the intent with aproximately 0.9 confidence while the rest obtain a negative value.

5

could you share the histogram output of a crossvalidation run?

6

constrain_similarities should be True btw

7

Here is the histogram. The pipeline is:

  - name: SpacyNLP
    model: "es_core_news_md"
  - name: SpacyTokenizer
  - name: SpacyFeaturizer
  - name: DIETClassifier
    epochs: 20
    entity_recognition: False
    model_confidence: cosine
    constrain_similarities: True

intent_histogram
intent_histogram857×933 31.8 KB

8

Can you try inner as model_confidence? The output confidence values will be the raw similarities that DIET computes to predict a label with no normalization / activation function on top of it. One thing to note is that the confidences will not lie in a bounded range, which means you’ll see values greater than 1 and possibly less than 0 too. Lmk if that helps.

9

Hi, I have similar problem as Adrian but no clue helped me.

I want to ranking intent i.e. 4 intentions no normalized vector of confidence. I will give 2 examples explaining what is expected:

  1. For the first phrase, I would like to get an intent ranking (no normalise) e.g.: (0.9,0.9,0,0) instead of (0.5,0.5,0.,0)- with normalization, in this case my phrase it is not recognized with a confidence threshold equal to 0.85,

  2. For the second phrase, I would liket to get an intent ranking (no normalise) e.g.: (0.5,0,0,0) instead of (1,0,0,0)- with normalization, in this case, my phrase begins to be misidentified (with confidence=0.85).

To sum up, I would like get an intent ranking with normalized values for each intention, but without normalization for vector of values, which representing intent ranking.

Can I get such a non-normalized vector of normalized values ​​and how?

Thanks