Attempting to handle confidence within a range with conditional rule and customized FallbackClassifier

Goal: I want a specific action triggered on the first user message if the confidence is within a range (.75 to .85) for only a specific intent.

Current plan: Within the existing FallbackClassifier class, I would like to add an additional condition to check the confidence and intent and send back a custom intent to trigger a rule

eg: The model is 76% confident of intent ‘hungry’ (the target for this case). Within FallbackClassifier the new intent will be ‘maybe_hungry’ and get pushed into the top confidence index (but have the same confidence level). Then a rule will check for ‘conversation_start: true’ and respond with a custom action.

Problem: If it is not the start of the conversation, changing the top intent name will obviously cause problems with story structures.

I’ve considered making another rule that will check for ‘conversation_start: false’ and then ‘maybe_hungry’ to trigger a different action that would revert ‘maybe_hungry’ to ‘hungry’, but I don’t know if that will still cause problems for stories. I saw the “undo fallback prediction” method could potentially be used for this, but I can’t find any usage examples. I assume it would be in a custom action somehow?

Alternatively, would checking within the FallbackClassifier itself for “first message” status work? After digging through the code I found that a generator can be used to make a list of prior trackers/actions, which is checked to see if it’s empty to verify ‘conversation_start’, but I can’t figure out how to leverage this within the FallbackClassifier.

Hi, this seems like an interesting problem!

I’m not quite sure I’ve understood what you’re trying to do, could you confirm or correct my understanding?

If an utterance is predicted to have intent hungry with a confidence between 0.75 and 0.85 and it’s the first utterance you’d like to assign this utterance the intent maybe_hungry instead?

Or would you always like this to happen regardless of when the utterance is made, but you’re thinking it can’t be done for anything but the first message?

@fkoerner Hi, Felicia!

Your first interpretation is correct, I want to assign maybe_hungry only at the start of the conversation. However, my goal is not the intent assignment of maybe_hungry but to trigger maybe_hungry_action, and my creation of maybe_hungry was only so I could trigger a rule (in tandem with conversation_start: true) that would execute maybe_hungry_action

The way I currently assign maybe_hungry is by checking within the FallbackClassifier for intent hungry and confidence >0.75 & <0.85. Since the Classifier does not have access to the conversation tracker (does it…?) it can’t tell if it’s the start of the conversation, so it ends up also re-labeling hungry in the middle of the conversation which breaks storylines.

So the priority here is to simply trigger maybe_hungry_action when hungry is within a confidence range and it is the start of the conversation. I do not necessarily have any reason to use a maybe_hungry intent, other than I currently do not know how else to trigger a rule for the desired action.

Oh, interesting! What if you made a rule to always execute the maybe_hungry_action whenever the user says they’re hungry at the beginning of the conversation:

 - rule: hungry rule
  conversation_start: True
  steps:
  - intent: hungry
  - action: maybe_hungry_action

Then inside the maybe_hungry_action first check what the confidence is, that information should be inside: tracker.latest_message

which will look something like this:

{'intent': {'id': 1234, 'name': 'hungry', 'confidence': 0.7512345},...<more stuff>...}

If the confidence isn’t in the right range, you can exit out of the action early and return what you would like the behaviour to be for example: return [FollowUpAction("action_listen")]

If the confidence is in the right range, continue and do whatever it is you want to do. What do you think?

It does seem like it could work! However, I have the same concern as with my other approach; would executing a rule break the stories those intents are used in?

I see action_two_stage_fallback has a _message_clarification that returns the event list [Listen(), last_message.intent.confidence = 1.0]; is this meant to simulate the progression of the story uninterrupted? If so, it seems I should add that to the maybe_hungry_action instead of just returning early.

**Edit: I just realized - would returning the event list to reset the conversation get the bot stuck in a loop, or since the event list is now longer than 0 would “conversation_start: true” not trigger?

Well, it might require some tweaking if you want the prediction to change based on whether the action was executed or not. One thing to think about is: what do you want to happen if the intent doesn’t fit the requirements (i.e. it’s not the start of the story, or the confidence isn’t in the desired range)? If you can share this with me, I can try and give you more specific advice.

There are several ways to encourage the behaviour you’d like to see. If you know that you would like intent: hungry to be followed by a particular action you can return this as a FollowUpAction. You can send a different FollowUpAction for each of the two cases.

Or, you can use a slot to disambiguate whether maybe_hungry occurred or not.

For example:

slots:
maybe_hungry:
    type: bool
    influence_conversation: true
 - story: maybe hungry conditions met
  steps:
  - intent: hungry
  - action: maybe_hungry_action
  - slot_was_set:
    - maybe_hungry: true
  < behaviour you want to see if maybe_hungry is true>
 - story: maybe hungry conditions not met
  steps:
  - intent: hungry
  - action: maybe_hungry_action
 - slot_was_set:
    - maybe_hungry: false
  < behaviour you want to see if maybe_hungry is false>

(The stories could also be rules. It all depends on your use case)