The difference in NLU confidences for the top two intents is lower than the ambiguity threshold 0.10


I’ve been working on a custom component to replace an entity value and intent depending on the situation. It’s working, but I noticed that in certain scenarios I’m getting this message:

The difference in NLU confidences for the top two intents (0.034934624697795735) is lower than the ambiguity threshold 0.10.

I’m currently setting the confidence separately so I think that the “top two intents” it’s referring to here is from the original message. I’ve changed the names of my intents, but the intent ranking looks like this for me:

'intent_ranking': [{'name': 'mood_affirm', 'confidence': 0.12587440211501025}, {'name': 'goodbye', 'confidence': 0.09093977741721451}, {'name': 'intent_1', 'confidence': 0.08721471406713568}, {'name': 'greet', 'confidence': 0.07483678229421939}, {'name': 'intent_2', 'confidence': 0.07327140708101469}, {'name': 'intent_3', 'confidence': 0.05063039035481162}, {'name': 'intent_4', 'confidence': 0.04711009108408087}, {'name': 'intent_5', 'confidence': 0.04000977855265766}, {'name': 'intent_6', 'confidence': 0.03499751474986334}

In this case I wanted to set the intent to intent_1, but I think because of this ambiguity threshold it’s triggering my fallback action instead.

2019-10-02 14:10:15 DEBUG    rasa.core.policies.memoization  - There is no memorised next action
2019-10-02 14:10:15 DEBUG    rasa.core.policies.form_policy  - There is no active form
2019-10-02 14:10:15 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_1_FallbackPolicy
2019-10-02 14:10:15 DEBUG    rasa.core.processor  - Predicted next action 'action_fallback' with confidence 1.00.

Is there a way to override the original intent ranking or is this a bug with setting the intent from a custom component?

Yes, you can overwrite it in the config.yml, for example:

  - name: "FallbackPolicy"
    nlu_threshold: 0.4
    core_threshold: 0.3
    ambiguity_threshold: 0.1
    fallback_action_name: "action_default_fallback"

You should also check those scenarios and your training data, because the confidence seems to low.

1 Like

Thanks! I’ll try that. Yeah the fact that the confidence is so low make sense in this case because there was a certain entity that was hard to train for so I decided to write a separate component to identify it.

So I tried changing the threshold, but I think part of the issue I’m seeing is that, as you mentioned, the confidence is so low even though I’m manually setting the intent and confidence, so it’s still calling my fallback based on the original intent ranking. Is it possible overwrite the original intent ranking as well? Also what exactly is the ambiguity_threshold for?

I don’t know. You seem to have more experience with this than me.

To avoid ambiguity when the confidence is too close. You could give it a try setting it at 0.01 or lower to see of the story flows as you want. Also lower the nlu_threshold.

Thanks for getting back to me. I’ll try playing around with the threshold and see how it performs then. This is my first time writing a custom component, so a lot of what I’ve been doing is experimental. I appreciate the help!

In case anyone wants to know, you can change the intent_ranking by using message.set()

For example,

message.set("intent_ranking", new_intent_ranking, add_to_output=True)

new_intent_ranking just needs to be formatted the same as the original intent ranking.

This will allow you change the intent rankings based on what you need and you can resolve the ambiguity threshold issue.

1 Like