Hello! I would like to save metrics from validation data in each epoch during NLU training. How can I do that? I saw Rasa NLU in Depth: Part 3 tutorial but I want as a first step to see when the error stops getting better.

Hi magda, keras has a tool for this known as early_stopping. This is unfortunately not implemented in Rasa yet. You might use tensorboard for now to get an impression of when the turning point is in the meantime though.

Hi magda and @koaning ! Any updates on the early stopping functionality? Since Rasa is an incredibly well-written library, I think it would be actually quite easy to insert Early Stopping into the RasaModel training loop. Here is a naive but working implementation using the total loss of the DIETClassifier as criteria: training_steps = 0 PATIENCE = 9 MIN_DELTA = 0.001 patience_counter = 0 prev_loss = float('inf') for epoch in progress_bar: epoch_batch_size = self.linearly_increasing_batch_size( epoch, batch_size, epochs ) training_steps = self._batch_loop( train_dataset_function, tf_train_on_batch_function, epoch_batch_size, True, training_steps, self.train_summary_writer, ) ## Early Stopping current_loss = self.metrics[0].result().numpy() delta = prev_loss - current_loss if delta > MIN_DELTA: patience_counter = 0 else: patience_counter += 1 if patience_counter == PATIENCE: logger.info("Apply early stopping") break prev_loss = current_loss Probably a good idea to only activate Early Stopping after a certain amount of warm-up to avoid unwanted weirdness with the linearly increasing batch size. Also this ignores when working with a validation set or evaluate_on_num_examples > 0

Determine number of epochs in DIETClassifier

Rasa Open Source

magda (Magda) July 6, 2020, 12:50pm 3

thanks I’ll try it!

Topic		Replies	Views
How about to set early stopping function in rasa training? Rasa Open Source	4	1800	August 5, 2021
NLU validation data and early stopping Rasa Open Source	0	290	November 10, 2022
Model Training Rasa Open Source	2	299	October 11, 2021
Training the nlu Rasa Open Source	1	463	May 9, 2022
Output the training statistics Rasa Open Source	0	249	November 7, 2022

Determine number of epochs in DIETClassifier

Related topics