Welcome to the forum 
This post on Stack Exchange explains the situation rather well.
I strongly suggest using Tensorboard with Rasa to visualize training and validation. You can see an example of the config here.
Once trained, open Tensorboard. You will see two curves for each component you enabled Tensorboard for (if you’re having problems with intents, you should at least enable it for DIET). One curve is for training, the other is for validation:
Usually, the training curve mainly goes up, while the validation curve starts going down at some point. DIET will have two pairs of curves, one for intents and one for entities.
Note down the first epoch where the validation accuracy for both intents and entities reach a high enough level and set the number of epochs for DIET to that number. This will avoid overfitting, aka having a high accuracy on your training set but low accuracy on new data - similar to your case.
In your config, you should set evaluate_on_number_of_examples to about 20% of your training data. (E.g. for diet, if you have 1000 example, set it to 200). I wrote this small script to calculate it for you.