With Rasa Open Source 1.8, we added support for leveraging language models like BERT, GPT-2, etc. These models can now be used as featurizers inside your NLU pipeline for intent classification, entity recognition and response selection models. The following snippet shows how to configure your pipeline to leverage
BERT model as an example -
pipeline: - name: HFTransformersNLP model_name: "bert" - name: LanguageModelTokenizer - name: LanguageModelFeaturizer - name: DIETClassifier
HFTransformersNLP is a utility component which relies on HuggingFace’s Transformers library for the core implementation of the selected language model. LanguageModelTokenizer and LanguageModelFeaturizer constructs the tokens and features respectively to be used inside the downstream NLU models.
You can load different variants of the same language model using the parameter
model_weights depending on the size of the model and language of your training corpus. For example, there are chinese (
bert-base-chinese) and japanese (
bert-base-japanese) variants of the BERT model which you can load if your training data is in chinese or japanese respectively. A full list of different variants of these language models is available in the official documentation of the Transformers library.
Please note, the current implementation uses these language models strictly as a featurizer which means that its weights are not fine-tuned along with the training of downstream NLU components like
As always, you can still use multiple featurizers in your pipeline, for example -
pipeline: - name: HFTransformersNLP model_name: "bert" - name: LanguageModelTokenizer - name: LanguageModelFeaturizer - name: CountVectorsFeaturizer - name: DIETClassifier
We would love to hear everyone’s feedback on it in terms of how it performs on your internal datasets, specially when used in combination with the newly introduced