So here is the general scenario I am looking at and I don’t know how to do it so I am looking for guidance. We have a list of about 35 intents currently and a bunch of training utterances for each. I have trained a model. Now what I am wondering is what if I have a user enter an utterance which is completely different than the material which I have trained on. So I don’t want to force one of my 35 intents on this utterance, I want something like “No Intent” to come back to indicate that none of the intents is a good match. I also don’t want to create a new intent for “off-topic” utterances. When I run from evaluate.py and am trying to score the performance how could I indicate that no intents match that utterance and then get credit for successfully saying that none of the intents is relevant to the utterance.

Haha yeah! You got it! I’m not an expert at any of this. I think you might be able to customize the NLU pipeline to do what you want but I’m not much of a programmer so I wouldn’t exactly know how to help you with that!

There’s another trick you can try. This is to have an “out_of_scope” intent that you would fill with random chitchat that’s unlike any of your 35 specified examples. Then, especially if you have a lot, of examples for your “real” intents, anything that is “out of scope” will match the “out_of_scope” intent.

Good one. Just a note: You’ll have to think carefully about adding this. Here’s why: If you ever want to add an intent you’ll have to check the out_of_scope examples again. If there’s an example that matches the ‘new’ intent, you’ll have to move it to avoid NLU confusion. You’ll have to make absolutely 100% sure that the out_of_scope intent contains examples that match the other intents for a really small percentage, to avoid confusion.

I would only use an “out_of_scope” intent to rule out the obvious stuff like “I want pizza” (if you’re not doing a pizza delivery app, of course) or “I hate you”, that kind of stuff. It also makes your bot better if you answer differently when the user says this than when the bot fails to understand the user on relevant stuff. If you are not happy with your NLU always returning an intent for your application, I’m assuming your application is written by you; if you are writing a chatbot, the configuration for Rasa already enables you to put a threshold on the required confidence that the intent classification has to have to be considered as a valid classification. Otherwise, if you are using NLU for other purposes, the NLU returns a JSON object, so you can always write up a quick function that extracts the intent classification confidence of that JSON object and triggers whatever action you want to use NLU for given that the confidence goes above the threshold, and performs a fallback action otherwise.

[image] PatrickDS: “I want pizza” [image] PatrickDS: “I hate you” These are better to incorporate as small talk intents that have custom answers. It’s always better to make the bot respond accurately than saying “I dunno” all the time.

@Remy I am aware of this as a best practice in theory, but in practice I am still struggling with creating a bot that has 20+ intents with decent NLU predictions. Using a data generation tool and 13 intents, I can get something reasonable, but already with 14 intents I am starting to have problems. I tried the tensorflow pipeline with less good results than with the spacy NLU. So currently I am not recommending someone who wants to do a bot for a specific purpose to start putting functionality into the out of scope part because it’s just heavy load on the NLU. Of course I am not an expert either when it comes to that since I haven’t achieved a chatbot with more than 20 intents that actually looks good and is easy to manage, so it’s just my personal suggestion, not a best practice. I already asked this question in another topic where I am discussing my ideas, so I won’t elaborate here… :stuck_out_tongue:

Heya Patrick, I catch your drift, our bots usually have no less than 80 intents (including the small talk one) so it might be less of a problem for us. You might just need more examples for the intents your are actually using.

80 intents?? That’s insane. How many training examples do you have for each? I don’t even understand how to get a thousand examples of an intent called “/affirm” and still keep a good bot… the problem doesn’t lie in generating the examples themselves, sometimes they just don’t exist.

We don’t have 1000 examples per intent :wink: We’ve generated about 50 examples per intent, and our users will generate the rest. It works, kind of. Just implement the TwoStageFallback to have users classify intents for your bot.

@Remy Remy, how long does it take to train your NLU and core models?

NLU - When testing, does every utterance have to match an intent

Rasa Open Source

PatrickDS (Patrick Da Silva) March 26, 2019, 5:12pm 21

My gut feeling tells me that you will need some structure in your NLU and much more examples to get something viable. You may wanna give Chatito a try to start!

Topic		Replies	Views
Rasa NLU predicting random intent Rasa Open Source	14	1731	November 15, 2019
Yes/no intents do not go through NLU threshold Rasa Open Source	1	920	December 14, 2018
How Bot based on Rasa respond to utterances other than data provided in data.json file Rasa Open Source	3	459	December 19, 2018
Random input - intent classified with high confidence Rasa Open Source	5	687	December 22, 2020
NLU detects random input with wrong intent and high confidence Rasa Open Source	39	5495	July 27, 2022

NLU - When testing, does every utterance have to match an intent

Related topics