# Different DIET Intent confidence scores after each train

**URL:** <https://forum.rasa.com/t/different-diet-intent-confidence-scores-after-each-train/49014>\
**Category:** Rasa Open Source\
**Created:** [November 5, 2021, 1:07pm UTC](https://forum.rasa.com/t/different-diet-intent-confidence-scores-after-each-train/49014 "2021-11-05T13:07:15Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![pomegran](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/pomegran/32/16087_2.png) [@pomegran](https://forum.rasa.com/u/pomegran)\
**Post date:** [November 5, 2021, 1:07pm UTC](https://forum.rasa.com/t/different-diet-intent-confidence-scores-after-each-train/49014/1 "2021-11-05T13:07:15Z")

</div>

Hi,

I am using the DIET Classifier without about 15 intents. Nothing heavy at all. I have an 8GB RAM Virtual Machine with 4 cores and it takes about 4 minutes to train.

However, every time I retrain I get a different confidence score for my intents. Why would that be if my nlu.yml remains unchanged? Can somebody explain why this might be happening?

Thanks Mark

---

<div class="post-metadata">

**Author:** ![nik202](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/nik202/32/16098_2.png) [@nik202](https://forum.rasa.com/u/nik202)\
**Post date:** [November 5, 2021, 1:27pm UTC](https://forum.rasa.com/t/different-diet-intent-confidence-scores-after-each-train/49014/2 "2021-11-05T13:27:25Z")

</div>

@pomegran I am not sure but please try this approach:

```auto

- name: DIETClassifier
  epochs: 100
  model_confidence: linear_norm
  constrain_similarities: true
  random_seed: 1

```

set random\_seed to any number

@pomegran make sure to delete older trained models.

---

<div class="post-metadata">

**Author:** ![pomegran](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/pomegran/32/16087_2.png) [@pomegran](https://forum.rasa.com/u/pomegran)\
**Post date:** [November 5, 2021, 1:43pm UTC](https://forum.rasa.com/t/different-diet-intent-confidence-scores-after-each-train/49014/3 "2021-11-05T13:43:45Z")

</div>

Will do and will report back.

Thanks @nik202 !

---

<div class="post-metadata">

**Author:** ![ChrisRahme](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/chrisrahme/32/15572_2.png) [@ChrisRahme](https://forum.rasa.com/u/ChrisRahme)\
**Post date:** [November 5, 2021, 1:52pm UTC](https://forum.rasa.com/t/different-diet-intent-confidence-scores-after-each-train/49014/4 "2021-11-05T13:52:53Z")

</div>

Nik gave you the answer, just use `random_seed`.

It can also be set in the `ResponseSelector` and `TEDPolicy` (basically any component that needs training).

* * *

> [@pomegran](#):
>
> Can somebody explain why this might be happening?

But my message is gonna focus on this, because I’ve been seeing it a lot on the forum:

It’s because Machine Learning involves randomness, mainly while splitting the data into training and testing/validation sets. Each time, a different chunk of your data is selected for training and testing.

To “make that randomness not random”, you use a [random seed](https://en.wikipedia.org/wiki/Random_seed). I used quotes because setting a seed doesn’t remove the randomness, it just controls it.

Example:

- Generating numbers between 1 and 10 without seed:

```plaintext
Try 1: 1 4 2 5 6
Try 2: 8 7 2 4 1

```

- Generating numbers between 1 and 10 with seed:

```plaintext
Try 1: 3 1 6 9 4
Try 2: 3 1 6 9 4

```

The sequence is still random, but it’s the same randomness on each run.

Fun fact: When we talk about a Minecraft World Seed, this is exactly what it’s about. Using the same seed will create the same randomly generated world.

---

<div class="post-metadata">

**Author:** ![pomegran](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/pomegran/32/16087_2.png) [@pomegran](https://forum.rasa.com/u/pomegran)\
**Post date:** [November 5, 2021, 4:45pm UTC](https://forum.rasa.com/t/different-diet-intent-confidence-scores-after-each-train/49014/5 "2021-11-05T16:45:08Z")

</div>

Thank you both, this is very interesting.

I understand the concept of seeds, but I was unaware there was randomness in classifier training. I’m surprised by default they’re not fixed too.

If the seed is fixed, does the order of the intent training data impact scores? Apologies for my lack of understanding here, but what is the benefit of a different seed each time for training?

This does explain my weird results though!

---

<div class="post-metadata">

**Author:** ![ChrisRahme](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/chrisrahme/32/15572_2.png) [@ChrisRahme](https://forum.rasa.com/u/ChrisRahme)\
**Post date:** [November 5, 2021, 7:52pm UTC](https://forum.rasa.com/t/different-diet-intent-confidence-scores-after-each-train/49014/6 "2021-11-05T19:52:29Z")

</div>

> [@pomegran](#):
>
> does the order of the intent training data impact scores?

That’s a nice question… 🤔

> [@pomegran](#):
>
> Apologies for my lack of understanding here

No need to apologize! Those are nice genuine questions!

> [@pomegran](#):
>
> what is the benefit of a different seed each time for training?

To possibly avoid a very low confidence that was just a coincidence due to an unlucky seed.

---

<div class="post-metadata">

**Author:** ![nik202](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/nik202/32/16098_2.png) [@nik202](https://forum.rasa.com/u/nik202)\
**Post date:** [November 6, 2021, 1:50pm UTC](https://forum.rasa.com/t/different-diet-intent-confidence-scores-after-each-train/49014/7 "2021-11-06T13:50:26Z")

</div>

> [@pomegran](#):
>
> does the order of the intent training data impact scores?

No, I guess this will not be an issue. But on the other hand, If we want to understand the training data impact on confidence scores please see this explanation.

There are two main pitfalls when creating model accuracy and confidence scores, both due to training data. The first is when there isn’t very much training data available. The second is when there is an abundance of training data available, but it is skewed or not representative of the data to be predicted.

If this is the case, there is a significant risk that the model will overfit and produce high confidence scores for inaccurate predictions. This is because the scoring population is inconsistent with the training population. It’s like creating a model to identify oranges and using it to predict apples.

> [@pomegran](#):
>
> what is the benefit of a different seed each time for training?

We use random seed value **while creating training and test data set**. The goal is to make sure we get the same training and validation data set while we use different hyperparameters or machine learning algorithms in order to assess the performance of different models.

I hope this will give you clear idea and don’t worry about good or bad question, its good you asked the question! Good Luck!

---

<div class="post-metadata">

**Author:** ![pomegran](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/pomegran/32/16087_2.png) [@pomegran](https://forum.rasa.com/u/pomegran)\
**Post date:** [November 11, 2021, 2:55pm UTC](https://forum.rasa.com/t/different-diet-intent-confidence-scores-after-each-train/49014/8 "2021-11-11T14:55:07Z")

</div>

Thanks both. Very interesting input!

---

<div class="post-metadata">

**Author:** ![nik202](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/nik202/32/16098_2.png) [@nik202](https://forum.rasa.com/u/nik202)\
**Post date:** [November 11, 2021, 8:11pm UTC](https://forum.rasa.com/t/different-diet-intent-confidence-scores-after-each-train/49014/9 "2021-11-11T20:11:22Z")

</div>

@pomegran No worries Mark, please close this active thread for your reference and for other.