Hello, I have trained my model using pipeline: name: “DucklingHTTPExtractor” url: “http://localhost:8000” dimensions: [“duration”] name: WhitespaceTokenizer name: RegexFeaturizer name: LexicalSyntacticFeaturizer name: CountVectorsFeaturizer name: CountVectorsFeaturizer analyzer: char_wb min_ngram: 1 max_ngram: 4 name: DIETClassifier epochs: 100 name: EntitySynonymMapper name: ResponseSelector epochs: 100 name: FallbackClassifier threshold: 0.3 ambiguity_threshold: 0.1 However random words like ‘asdf’, ‘qwerty’ and even single letters are classified as intents (‘greet’, ‘neutral’ respectively). I could increase FallbackClassifier threshold, but some examples have confidence over 0.9. I’m using Rasa 2.8. How to deal with it?

@Jakub Hi! Can you please share some example or file or screenshot? @Jakub Why you are using ambiguity_threshold: 0.1 inside FallbackClassifier or share the link from where you get this idea? @Jakub Please share complete config.yml or update the above one.

[image] nik202: share some example or file or screenshot? Hi! Same on my site. No Matter what kind of input. There is always a intent recognition. Eg “123456”, “dfjjaspfj” or something like “i want pizza”. I’m looking forward for some help. Thanks! Using diffrent versions with docker: rasa/rasa:main-spacy-de rasa/rasa:2.8.1-spacy-de pipeline: language: de pipeline: # # No configuration for the NLU pipeline was provided. The following default pipeline was used to train your model. # # If you'd like to customize it, uncomment and adjust the pipeline. # # See https://rasa.com/docs/rasa/tuning-your-model for more information. - name: SpacyNLP model: de_core_news_sm - name: SpacyTokenizer - name: SpacyFeaturizer - name: RegexFeaturizer - name: LexicalSyntacticFeaturizer - name: CountVectorsFeaturizer - name: CountVectorsFeaturizer analyzer: char_wb min_ngram: 1 max_ngram: 4 - name: DIETClassifier epochs: 100 constrain_similarities: true - name: EntitySynonymMapper - name: FallbackClassifier threshold: 0.3 ambiguity_threshold: 0.1 nlu.yml: version: “2.0” nlu: - intent: greet examples: | - hey - hallo - hi - hallo du - guten morgen - guten abend - morgen - guten tag - intent: goodbye examples: | - tschüss - wiederhören - ciao - bis dann - bye result: { "text": "12345", "intent": { "id": 8761605359927853060, "name": "goodbye", "confidence": 0.9780691862106323 }, "entities": [], "intent_ranking": [ { "id": 8761605359927853060, "name": "goodbye", "confidence": 0.9780691862106323 }, { "id": -3994034142759590561, "name": "greet", "confidence": 0.02193082869052887 } ] }

[image] thorty: 12345 @thorty Means whatever input i.e “I want pizza” , “I need drinks” do such examples are in your nlu.yml file?; it’s giving you goodbye output?

[image] nik202: @thorty Means whatever input i.e “I want pizza” , “I need drinks” do such examples are in your nlu.yml file?; it’s giving you goodbye output? No they are not into my nlu.yml as examples. In this case I’m expecting a result like: intent: none. But what I’ m getting here is something like intent: goodbye with a conf of over 90. my nlu.yml is exactly like in the post above. the pipeline also. and the result is an example with this config. thank you @nik202 !

[image] thorty: ambiguity_threshold: 0.1 @thorty Why this? any significance? @thorty Link please, I guess its used for two-stage fallback not for FallbackClassifier

@thorty All your inputs will be given an intent. If you have very high confidence and a lot of incorrect classifications it’s often the case that this is due to your data not being a good representation of what your assistant is seeing in production. You should create an out of scope intent if you want to be able to capture things your assistant can’t do. This page goes over how to do it: Fallback and Human Handoff

@Nik202: Thanks, but this does not change enything. @rctatman: Thanks for your explanation and the Link to the right Step in Documentation. This helps a lot in understanding rasa. In my case I do not really know the utterances from users. So I will start with a small set of examples and train the NLU after going live with real-world examples. To handle the “non” trained utterances the right way it would be better to get no intent rather then the wrong intent. Especially when utterances are non-sense like in my example. Is there any chance to achieve this?

@Jakub Any progress about your error?

Rasa classifies random input as intents with high probability

Rasa Open Source

thorty (Thorsten Weiskopf) July 30, 2021, 3:12pm 7

@Nik202 No. It’s from the doc or from the example of the init pipeline. Should I use another threshold? I will try this out later.

Topic		Replies	Views
Random input - intent classified with high confidence Rasa Open Source	5	701	December 22, 2020
NLU detects random input with wrong intent and high confidence Rasa Open Source	39	5606	July 27, 2022
Error in intent classification - Without slots or entities Rasa Open Source	3	523	May 12, 2020
NLU Fallback is taking precedence to other intents irrespective of threshold value and their confidence score Feedback on Rasa Open Source	9	1492	August 16, 2022
Intent incorrectly recognised with high confidence Rasa Open Source	3	345	June 20, 2023

Rasa classifies random input as intents with high probability

Related topics