Hi,
I am using the DIET Classifier without about 15 intents. Nothing heavy at all. I have an 8GB RAM Virtual Machine with 4 cores and it takes about 4 minutes to train.
However, every time I retrain I get a different confidence score for my intents. Why would that be if my nlu.yml remains unchanged? Can somebody explain why this might be happening?
Thanks Mark
@pomegran I am not sure but please try this approach:
- name: DIETClassifier
epochs: 100
model_confidence: linear_norm
constrain_similarities: true
random_seed: 1
set random_seed to any number
@pomegran make sure to delete older trained models.
Nik gave you the answer, just use random_seed.
It can also be set in the ResponseSelector and TEDPolicy (basically any component that needs training).
But my message is gonna focus on this, because I’ve been seeing it a lot on the forum:
It’s because Machine Learning involves randomness, mainly while splitting the data into training and testing/validation sets. Each time, a different chunk of your data is selected for training and testing.
To “make that randomness not random”, you use a random seed. I used quotes because setting a seed doesn’t remove the randomness, it just controls it.
Example:
Try 1: 1 4 2 5 6
Try 2: 8 7 2 4 1
Try 1: 3 1 6 9 4
Try 2: 3 1 6 9 4
The sequence is still random, but it’s the same randomness on each run.
Fun fact: When we talk about a Minecraft World Seed, this is exactly what it’s about. Using the same seed will create the same randomly generated world.
Thank you both, this is very interesting.
I understand the concept of seeds, but I was unaware there was randomness in classifier training. I’m surprised by default they’re not fixed too.
If the seed is fixed, does the order of the intent training data impact scores? Apologies for my lack of understanding here, but what is the benefit of a different seed each time for training?
This does explain my weird results though!
That’s a nice question… 
No need to apologize! Those are nice genuine questions!
To possibly avoid a very low confidence that was just a coincidence due to an unlucky seed.
No, I guess this will not be an issue. But on the other hand, If we want to understand the training data impact on confidence scores please see this explanation.
There are two main pitfalls when creating model accuracy and confidence scores, both due to training data. The first is when there isn’t very much training data available. The second is when there is an abundance of training data available, but it is skewed or not representative of the data to be predicted.
If this is the case, there is a significant risk that the model will overfit and produce high confidence scores for inaccurate predictions. This is because the scoring population is inconsistent with the training population. It’s like creating a model to identify oranges and using it to predict apples.
We use random seed value while creating training and test data set . The goal is to make sure we get the same training and validation data set while we use different hyperparameters or machine learning algorithms in order to assess the performance of different models.
I hope this will give you clear idea and don’t worry about good or bad question, its good you asked the question! Good Luck!
Thanks both. Very interesting input!