Determine number of epochs in DIETClassifier

Rasa Open Source
1

Hello!

I would like to save metrics from validation data in each epoch during NLU training. How can I do that? I saw Rasa NLU in Depth: Part 3 tutorial but I want as a first step to see when the error stops getting better.

2

Hi magda,

keras has a tool for this known as early_stopping. This is unfortunately not implemented in Rasa yet. You might use tensorboard for now to get an impression of when the turning point is in the meantime though.

3

thanks I’ll try it!

5

Hi magda and @koaning !

Any updates on the early stopping functionality? Since Rasa is an incredibly well-written library, I think it would be actually quite easy to insert Early Stopping into the RasaModel training loop. Here is a naive but working implementation using the total loss of the DIETClassifier as criteria:

    training_steps = 0
    PATIENCE = 9
    MIN_DELTA = 0.001
    patience_counter = 0
    prev_loss = float('inf')
    
    for epoch in progress_bar:
        epoch_batch_size = self.linearly_increasing_batch_size(
            epoch, batch_size, epochs
        )

        training_steps = self._batch_loop(
            train_dataset_function,
            tf_train_on_batch_function,
            epoch_batch_size,
            True,
            training_steps,
            self.train_summary_writer,
        )

        ## Early Stopping
        current_loss = self.metrics[0].result().numpy()
        delta = prev_loss - current_loss

        if delta > MIN_DELTA:
            patience_counter = 0
        else:
            patience_counter += 1

        if patience_counter == PATIENCE:
            logger.info("Apply early stopping")
            break
        
        prev_loss = current_loss
  • Probably a good idea to only activate Early Stopping after a certain amount of warm-up to avoid unwanted weirdness with the linearly increasing batch size.
  • Also this ignores when working with a validation set or evaluate_on_num_examples > 0