I have a question with regards to using Rasa Core with the HTTP API when predicted actions are uncertain.
Say we have the case that the user utterance is ambiguous, so that we have a probability of 0.55 for action A and 0.45 for action B. In that case, we would like to ask the user “did you mean A or B or something else?” and use the user feedback to determine further action.
I could see three possibilities to achieve this:
- Modify the Rasa Core response to include not only the most likely action but the N most likely actions and their probabilities. Then we have some logic in our channel component that checks the probabilities and instead of returning the answer from action A (the most confident one), it passes an utterance to the user as shown above. When the user responds with B, action B is executed.
This approach has several drawbacks. Rasa Core will assume that action A was executed, since it has a higher probability. Therefore, action A is already in the tracker and it is not trivial to remove the last action from the tracker. Also, if action A triggered a side-effect (like an API call), it could be too late to instead execute action B.
- We have a special action, say
ActionUncertain. If the probability for any one action is too low, we predict this action. It then returns the utterance above and if the user says B, we somehow hard-match this to action B (how?).
If I understand the inner workings of Rasa Core correctly, with this approach, the
ActionUncertain would be tracked in the
Tracker and thus be part of the features created for the machine learning model to predict the next step. Consequently, we would need stories where this
ActionUncertain appears – otherwise, the model cannot make sense of this feature. But introducing this action into existing stories seems to be cumbersome. Furthermore, since we are labelling real customer interactions with our service agents, this kind of action never occurs.
- Use the fallback action. However, AFAIK, this only covers the case when we didn’t understand the user at all, not for asking whether they meant A or B.
I hope that there is a simple way for us to implement this use case. Any help would be welcome.
PS: Are there more Rasa Action SDKs on the roadmap, say for JS (node)?