@Akhil Taking ~10% of the total examples as validation data is a good approach. So ~300 for evaluate_on_number_of_examples sound like a good number in your case. We randomly pick those examples from the training examples. We guarantee that at least one example per intent will end up in the validation data. We don’t have any guarantee for entities. But normally you will also find a couple of entities in the validation data, especially if your training examples contain quite a lot entities. If you want to also validate the ResponseSelector during training, you need to set the option evaluate_on_number_of_examples for the ResponseSelector in the config.yml file. It is separate from the DIETClassifier. Hope that clarifies your questions.
1 Like