Terminating model training

Is there a way to terminate model training? This is not early stopping. Say you start model training and you realise you still need to make changes to training/labelled data and want to terminate the training. Is there a provision in the Trainer class for that? If NOT, any other suggestions for that?


Why not do Ctrl+C?

@nathanrbrandt Hi, If you want to terminate the training model, you can terminate at any time whilst the model is in training, as suggested by ChrisRahme you can kill the process or terminate the process using Ctrl+ C and even sometimes Ctrl + Z also.

1 Like

Thanks guys for the replies. Ctrl+C is fine when you are running it from the command line.

What if we have created a REST wrapper with endpoints like /build, /predict? We ideally want to provide an endpoint called /terminate which should terminate model building process. We have played around with

  1. Start the model building in a thread, try to kill the thread on /terminate call. but killing CPU bound threads can be tricky
  2. Start the model building in a process and kill the process on /terminate call

Just wanted to check if people have any other ideas.

1 Like