Hello!
I would like to save metrics from validation data in each epoch during NLU training. How can I do that? I saw Rasa NLU in Depth: Part 3 tutorial but I want as a first step to see when the error stops getting better.
Hello!
I would like to save metrics from validation data in each epoch during NLU training. How can I do that? I saw Rasa NLU in Depth: Part 3 tutorial but I want as a first step to see when the error stops getting better.
Hi magda,
keras has a tool for this known as early_stopping
. This is unfortunately not implemented in Rasa yet. You might use tensorboard for now to get an impression of when the turning point is in the meantime though.
thanks I’ll try it!
Hi magda and @koaning !
Any updates on the early stopping functionality?
Since Rasa is an incredibly well-written library, I think it would be actually quite easy to insert Early Stopping into the RasaModel
training loop. Here is a naive but working implementation using the total loss of the DIETClassifier as criteria:
training_steps = 0
PATIENCE = 9
MIN_DELTA = 0.001
patience_counter = 0
prev_loss = float('inf')
for epoch in progress_bar:
epoch_batch_size = self.linearly_increasing_batch_size(
epoch, batch_size, epochs
)
training_steps = self._batch_loop(
train_dataset_function,
tf_train_on_batch_function,
epoch_batch_size,
True,
training_steps,
self.train_summary_writer,
)
## Early Stopping
current_loss = self.metrics[0].result().numpy()
delta = prev_loss - current_loss
if delta > MIN_DELTA:
patience_counter = 0
else:
patience_counter += 1
if patience_counter == PATIENCE:
logger.info("Apply early stopping")
break
prev_loss = current_loss
Early Stopping
after a certain amount of warm-up to avoid unwanted weirdness with the linearly increasing batch size.evaluate_on_num_examples > 0