Custom LM with HFTransformers

I’m trying to use my own fine-tuned HFTransformers model weights in a pipeline. I’m guessing that it’s possible to do the following, but I’d love input or thoughts on if there’s an easier way:

  • build an initial fake pipeline using a specific set of HF Transformer model+model_weights combo (say distilbert + distilbert-base-uncased). Store the filename for the cached .h5 weights
  • use HF Transformer’s run_language_modeling.py script to create a new set of model weights.
  • copy the new weights to the old weight’s same {hash}.h5 cache dir

Before I write that up, does anyone see anything wrong with that or have ideas on a better approach?

Hi @jamesmf, I am not sure about the exact fine-tuning process. I am sure you’ll be able to get some help here - Examples — pytorch-transformers 1.0.0 documentation . Once you have the model fine-tuned, you can pass the directory containing your fine-tuned model to model_weights parameter of HFTransformersNLP component. Make sure you include all files that are generally present in the packaged pre-trained model tars(model weights, vocabulary, config, etc.). Keep model_name parameter to the base model architecture you are using(for e.g. - distilbert).

1 Like