# As number of intents increases, confidence level decreases

**URL:** <https://forum.rasa.com/t/as-number-of-intents-increases-confidence-level-decreases/619>\
**Category:** Rasa Open Source\
**Created:** [August 23, 2018, 7:16am UTC](https://forum.rasa.com/t/as-number-of-intents-increases-confidence-level-decreases/619 "2018-08-23T07:16:58Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![harshitazilen](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/harshitazilen/32/321_2.png) [@harshitazilen](https://forum.rasa.com/u/harshitazilen)\
**Post date:** [August 23, 2018, 7:16am UTC](https://forum.rasa.com/t/as-number-of-intents-increases-confidence-level-decreases/619/1 "2018-08-23T07:16:58Z")

</div>

I am working with a chatbot project which involves Rasa NLU mainly.

It involves chatbot for FAQ support. As some of the FAQs are quite similar in terms of language, once number of intents (type of questions) are more, NLU gives lower confidence level.

e.g. Initially I tested with 2 intents and the confidence level was near 70 to 90%. Now the project has 80 intents, the confidence level is near to 8 to 10%.

I want to know in this kind of scenario, where around 80 intents are there, some of the intents are quite similar in nature, what should be the threshold for the NLU?

---

<div class="post-metadata">

**Author:** ![NikhilBansal21](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/nikhilbansal21/32/418_2.png) [@NikhilBansal21](https://forum.rasa.com/u/NikhilBansal21)\
**Post date:** [August 23, 2018, 8:34am UTC](https://forum.rasa.com/t/as-number-of-intents-increases-confidence-level-decreases/619/2 "2018-08-23T08:34:27Z")

</div>

what are you using in pipeline spacy or tensorflow?

---

<div class="post-metadata">

**Author:** ![harshitazilen](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/harshitazilen/32/321_2.png) [@harshitazilen](https://forum.rasa.com/u/harshitazilen)\
**Post date:** [August 23, 2018, 8:37am UTC](https://forum.rasa.com/t/as-number-of-intents-increases-confidence-level-decreases/619/3 "2018-08-23T08:37:53Z")

</div>

I am using spacy pipeline.

---

<div class="post-metadata">

**Author:** ![NikhilBansal21](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/nikhilbansal21/32/418_2.png) [@NikhilBansal21](https://forum.rasa.com/u/NikhilBansal21)\
**Post date:** [August 23, 2018, 8:42am UTC](https://forum.rasa.com/t/as-number-of-intents-increases-confidence-level-decreases/619/4 "2018-08-23T08:42:08Z")

</div>

use tensorflow for high intent confidence spacy’s confidence varies

---

<div class="post-metadata">

**Author:** ![harshitazilen](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/harshitazilen/32/321_2.png) [@harshitazilen](https://forum.rasa.com/u/harshitazilen)\
**Post date:** [August 23, 2018, 8:42am UTC](https://forum.rasa.com/t/as-number-of-intents-increases-confidence-level-decreases/619/5 "2018-08-23T08:42:58Z")

</div>

Ok. I recently trained using Tensorflow, Will let you know the results.

---

<div class="post-metadata">

**Author:** ![amn41](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/amn41/32/20_2.png) [@amn41](https://forum.rasa.com/u/amn41)\
**Post date:** [August 23, 2018, 7:25pm UTC](https://forum.rasa.com/t/as-number-of-intents-increases-confidence-level-decreases/619/6 "2018-08-23T19:25:41Z")

</div>

@NikhilBansal21 @harshitazilen do please read the ‘note about confidence scores’ here [http://rasa.com/docs/nlu/fallback/](http://rasa.com/docs/nlu/fallback/)

---

<div class="post-metadata">

**Author:** ![harshitazilen](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/harshitazilen/32/321_2.png) [@harshitazilen](https://forum.rasa.com/u/harshitazilen)\
**Post date:** [August 24, 2018, 5:39am UTC](https://forum.rasa.com/t/as-number-of-intents-increases-confidence-level-decreases/619/7 "2018-08-24T05:39:21Z")

</div>

Yes I read it. I used same training data with spacy and tensorflow\_embeding pipelines. Below is the report

**Spacy Pipeline:**

- Total Intents: 80
- Average confidence while detecting a training statement : 0.08 - 0.1

With spacy I am struggling to decide the threshold level as confidence is super low.

**Tensorflow\_embeding Pipeline:**

- Total Intents: 80
- Average confidence while detecting a training statement : 0.8 - 0.9

With tensorflow\_embeding, if this is overfitting, I will need to remove training data and epochs.

Kindly guide me.

---

<div class="post-metadata">

**Author:** ![souvikg10](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/souvikg10/32/93_2.png) [@souvikg10](https://forum.rasa.com/u/souvikg10)\
**Post date:** [August 24, 2018, 8:56am UTC](https://forum.rasa.com/t/as-number-of-intents-increases-confidence-level-decreases/619/8 "2018-08-24T08:56:22Z")

</div>

Do you have a test set? if not then use -3-fold cross-validation

> **[Testing Your Assistant](https://rasa.com/docs/rasa/testing-your-assistant/)**

For overfitting and underfitting, you should look at the F1 Score of test vs train

if test is low and train is high you have overfitting or underfitting the other way around.

I think F1 score is a good baseline that can indicate how good is your model.

Also is your dataset balanced? we have over 140 intents and most of them are not well balanced and ideally now we are trying to merge some classes and figure out programmatically how to deal with the difference. Since you are doing FAQ, one good reminder as I have seen from tests with real users is people don’t interact with FAQ pages the same way they do with a chatbot. So If you have reused the same classes for your chatbot, It won’t really make sense in the end.
