During training, as described from the topic, i got the following figures Epochs: 41% 122/300 [1:40:08<1:51:25, 37.56s/it, t_loss=102, m_acc=0.992, i_acc=0.999] How should i interpret the result with such a high t_loss but also high m_acc and i_acc? Thanks

I don’t have the answer to that, to be honest, but I think it should be evenly split. But then, why does this field have to be a number and not a percentage? Anyway, I would suggest adding more examples to the intents that do not have many of them. I also suggest setting random_seed to any value so that you get comparable results between trainings.

@mayuanyang1 can you please share the config.yml file for us?

version: “2.0” language: ch pipeline: - name: HFTransformersNLP model_name: bert model_weights: bert-base-chinese - name: LanguageModelTokenizer - name: LanguageModelFeaturizer - name: RegexFeaturizer use_word_boundaries: False case_sensitive: False - name: CountVectorsFeaturizer - name: CountVectorsFeaturizer analyzer: char_wb min_ngram: 1 max_ngram: 6 - name: DIETClassifier epochs: 300 embedding_dimension: 150 number_of_transformer_layers: 3 transformer_size: 320 constrain_similarities: true entity_recognition: false use_masked_language_model: true policies: - name: MemoizationPolicy - name: TEDPolicy max_history: 5 epochs: 100 - name: RulePolicy

@mayuanyang1 Alright, means you have shared the DIETClassifier training for 300 Epochs. Give me some time and I’ll get back to you. @mayuanyang1 I hope your chat is working fine as per the use case?

thanks for your help. I have a model with about 900 intents and each one of them has about 30-50 examples, the intent classification seems not doing very well, any suggestion?

@mayuanyang1 always mention me for fast response :slight_smile: OK? [image] mayuanyang1: I have a model with about 900 intents and each one of them has about 30-50 examples, the intent classification seems not doing very well, Ok, please try to minimize some of the intents you have a lot of intents, try to merge the intents, I assume your model is getting overfit or underfit ( confused). Even, try to delete all previous trained model and train again and check it provides you with better results or not? Even, I’d recommend just mentioning at least 10-20 examples that are more than enough for training the model, but more you provide it’s also good but it will make training time increase a lot. @mayuanyang1 what is your model training time in total when you train?

@nik202 i use colab, here are the figures 2022-01-31 05:37:26 INFO rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer - 30301 vocabulary items were created for text attribute. 2022-01-31 05:37:38 INFO rasa.nlu.model - Finished training component. 2022-01-31 05:37:38 INFO rasa.nlu.model - Starting to train component CountVectorsFeaturizer 2022-01-31 05:37:41 INFO rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer - 465804 vocabulary items were created for text attribute. 2022-01-31 05:37:55 INFO rasa.nlu.model - Finished training component. 2022-01-31 05:37:55 INFO rasa.nlu.model - Starting to train component DIETClassifier /usr/local/lib/python3.7/dist-packages/rasa/utils/tensorflow/model_data_utils.py:395: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray np.array([v[0] for v in values]), number_of_dimensions=3 Epochs: 50% 150/300 [1:56:49<1:27:02, 34.82s/it, t_loss=102, m_acc=0.98, i_acc=0.998] /usr/local/lib/python3.7/dist-packages/rasa/utils/tensorflow/model_data.py:750: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray np.concatenate(np.array(f)), Epochs: 100% 300/300 [2:55:35<00:00, 35.12s/it, t_loss=44.4, m_acc=0.995, i_acc=0.999] I pretty much delete the old model and train from scratch every-time, i have tried split the intents into 3 models, about 300 intents/model, the yield confidence is significantly higher, however this for me seems like a work around, would be good to have just 1 model, any suggestion would be appreciated

Welcome to the forum :slight_smile: [image] mayuanyang1: How should i interpret the result This post on Stack Exchange explains the situation rather well. I strongly suggest using Tensorboard with Rasa to visualize training and validation. You can see an example of the config here . Once trained, open Tensorboard. You will see two curves for each component you enabled Tensorboard for (if you’re having problems with intents, you should at least enable it for DIET). One curve is for training, the other is for validation: [image] Usually, the training curve mainly goes up, while the validation curve starts going down at some point. DIET will have two pairs of curves, one for intents and one for entities. Note down the first epoch where the validation accuracy for both intents and entities reach a high enough level and set the number of epochs for DIET to that number. This will avoid overfitting, aka having a high accuracy on your training set but low accuracy on new data - similar to your case. In your config, you should set evaluate_on_number_of_examples to about 20% of your training data. (E.g. for diet, if you have 1000 example, set it to 200). I wrote this small script to calculate it for you.

[image] ChrisRahme: evaluate_on_number_of_examples Will definitely try them out, for evaluate_on_number_of_examples, as i have a not very well balanced of examples e.g. some intents might have 30 and others might just have 10, would DIET workout the split automatically?

Have tried your suggestion with a much better result :+1: Epochs: 100% 300/300 [1:45:35<00:00, 21.12s/it, t_loss=9.07, i_acc=1, val_t_loss=9.03, val_i_acc=0.999] Is the loss still relatively high?

Glad you go it better :slight_smile: Mind sharing your config.yml and Tensorboard results to see if I can suggest anything? I find the loss is a bit high, but as long as your bot works as expected it doesn’t matter. In the end, when your bot starts talking to real users, you will have to check those conversations and use them as training data (misclassified intents and unsuccessful stories).

Very high t_loss, but also with high m_acc and i_acc

Rasa Open Source

nik202 (NiK202) February 3, 2022, 8:53pm 15

@mayuanyang1 can you please close this thread, if you got the solution on this thread?

Topic		Replies	Views
DIETClassifier not working properly for french bot Rasa Open Source	16	800	October 5, 2021
Rasa train taking lot of time Rasa Open Source	22	5152	July 6, 2021
Output the training statistics Rasa Open Source	0	249	November 7, 2022
What changed in the DIETClassifier implementation or defaults? Significant drop in performance for rare intents Rasa Open Source	6	725	June 30, 2021
Feedback: Upgrading to Tensorflow 2.6 Rasa Open Source	30	9691	September 19, 2022

Very high t_loss, but also with high m_acc and i_acc

Related topics