Support for Language Models inside Rasa

Announcements Release Announcements
,
1

With Rasa Open Source 1.8, we added support for leveraging language models like BERT, GPT-2, etc. These models can now be used as featurizers inside your NLU pipeline for intent classification, entity recognition and response selection models. The following snippet shows how to configure your pipeline to leverage BERT model as an example -

pipeline:
   - name: HFTransformersNLP
     model_name: "bert"
   - name: LanguageModelTokenizer
   - name: LanguageModelFeaturizer
   - name: DIETClassifier

HFTransformersNLP is a utility component which relies on HuggingFace’s Transformers library for the core implementation of the selected language model. LanguageModelTokenizer and LanguageModelFeaturizer constructs the tokens and features respectively to be used inside the downstream NLU models.

You can load different variants of the same language model using the parameter model_weights depending on the size of the model and language of your training corpus. For example, there are chinese (bert-base-chinese) and japanese (bert-base-japanese) variants of the BERT model which you can load if your training data is in chinese or japanese respectively. A full list of different variants of these language models is available in the official documentation of the Transformers library.

Please note, the current implementation uses these language models strictly as a featurizer which means that its weights are not fine-tuned along with the training of downstream NLU components like DIETClassifier, etc.

As always, you can still use multiple featurizers in your pipeline, for example -

pipeline:
   - name: HFTransformersNLP
     model_name: "bert"
   - name: LanguageModelTokenizer
   - name: LanguageModelFeaturizer
   - name: CountVectorsFeaturizer
   - name: DIETClassifier

We would love to hear everyone’s feedback on it in terms of how it performs on your internal datasets, specially when used in combination with the newly introduced DIETClassifier.

13 Likes
2

I would also love to hear if the video on benchmarking helped navigate the settings. I would also love to hear if it didn’t!

8 Likes
3

Awesome :slight_smile:

4

Any plans on adding the XLM-RoBERTa (XLM-RoBERTa — transformers 2.5.1 documentation) to available language models?

2 Likes
5

Is there a way to load HF-transformers compatible model saved in pytorch format? Unfortunately there is no RuBERT model in TF2.0 format.

When I try to load pytorch model there is an error:

OSError: Error no file named ['pytorch_model.bin', 'tf_model.h5'] found in directory /opt/rubert/conversational_cased_L-12_H-768_A-12_pt/ or `from_pt` set to False

There exists pytorch_model.bin so I think the case is from_pt set to False.

6

@ezhvsalate If we set from_pt to True that would require pytorch in the backend to load that model. We don’t support that yet.

7

Maybe it will be possible to add an optional parameter defining if loaded model was saved as PyTorch checkpoint? And write in the docs that setting param to True will require installation of PyTorch. I tried it locally and it works - RuBert model was loaded. If this is ok - I’ll create a pull request.

8

@ezhvsalate You can also convert the pytorch checkpoint into a compatible tensorflow checkpoint using this script and then load the model - transformers/convert_pytorch_checkpoint_to_tf2.py at master ¡ huggingface/transformers ¡ GitHub

3 Likes
9

how can i load different variants of the same language model using the parameter model_weights

can you specify the process of doing so in pipeline.

@dakshvar22

10

For example, if the variant you want to use is bert-base-uncased, then your pipeline would look something like -

pipeline:
   - name: HFTransformersNLP
     model_name: "bert"
     model_weights: "bert-base-uncased"
   - name: LanguageModelTokenizer
   - name: LanguageModelFeaturizer
   - name: DIETClassifier

If you want to load the model weights from huggingface compatible model checkpoint stored locally, you can pass its path as well as the value of the model_weights parameter

1 Like
11

@dakshvar22 for loading the local model what will be the parameter i shall use?

12

Path to the directory containing the model checkpoint.

pipeline:
   - name: HFTransformersNLP
     model_name: "bert"
     model_weights: "path/to/your/model"
   - name: LanguageModelTokenizer
   - name: LanguageModelFeaturizer
   - name: DIETClassifier
2 Likes
13

Can someone take a look at this one? Training with BERT is constantly failing.

http://forum.rasa.com/t/uising-bert-with-rasa/28113

15 
# https://rasa.com/docs/rasa/nlu/components/
language: zh
pipeline:
  - name: HFTransformersNLP
    # Name of the language model to use
    model_name: "roberta"
    # Pre-Trained weights to be loaded
    model_weights: "data/roberta_chinese_base"
    # An optional path to a specific directory to download and cache the pre-trained model weights.
    # The `default` cache_dir is the same as https://huggingface.co/transformers/serialization.html#cache-directory .
    #cache_dir: null
  - name: LanguageModelTokenizer
    # Flag to check whether to split intents
    intent_tokenization_flag: False
    # Symbol on which intent should be split
    intent_split_symbol: "_"
  - name: LanguageModelFeaturizer
  #- name: CountVectorsFeaturizer
  - name: DIETClassifier
    epochs: 100
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 100

# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
  - name: MemoizationPolicy
  - name: TEDPolicy
    max_history: 5
    epochs: 100
  - name: MappingPolicy

2020-05-30 16:02:01 INFO transformers.tokenization_utils - Model name ‘data/roberta_chinese_base’ not found in model shortcut name list (roberta-base, roberta-large, roberta-large-mnli, distilroberta-base, roberta-base-openai-detector, roberta-large-openai-det ector). Assuming ‘data/roberta_chinese_base’ is a path, a model identifier, or url to a directory containing tokenizer files. 2020-05-30 16:02:01 INFO transformers.tokenization_utils - Didn’t find file data/roberta_chinese_base\vocab.json. We won’t load it. 2020-05-30 16:02:01 INFO transformers.tokenization_utils - Didn’t find file data/roberta_chinese_base\merges.txt. We won’t load it. 2020-05-30 16:02:01 INFO transformers.tokenization_utils - Didn’t find file data/roberta_chinese_base\added_tokens.json. We won’t load it. 2020-05-30 16:02:01 INFO transformers.tokenization_utils - Didn’t find file data/roberta_chinese_base\special_tokens_map.json. We won’t load it. 2020-05-30 16:02:01 INFO transformers.tokenization_utils - Didn’t find file data/roberta_chinese_base\tokenizer_config.json. We won’t load it.

OSError: Model name ‘data/roberta_chinese_base’ was not found in tokenizers model name list (roberta-base, roberta-large, roberta-large-mnli, distilroberta-base, roberta-base-openai-detector, roberta-large-openai-detector). We assumed ‘data/roberta_chinese_base’ wa s a path, a model identifier, or url to a directory containing vocabulary files named [‘vocab.json’, ‘merges.txt’] but couldn’t find such vocabulary files at this path or url.

16

OSError: Model name ‘bert-base-uncased’ was not found in tokenizers model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, b ert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbm dz-uncased, bert-base-finnish-cased-v1, bert-base-finnish-uncased-v1, bert-base-dutch-cased). We assumed ‘bert-base-uncased’ was a path, a model identifier, or url to a directory containing vocabulary files named [‘vocab.txt’] but couldn’t find such vocabulary file s at this path or url.

17

I’m trying to run this below config in rasa nlu

language: en pipeline:

  • name: HFTransformersNLP model_weights: “bert-base-uncased” model_name: “bert”
  • name: LanguageModelTokenizer
  • name: LanguageModelFeaturizer
  • name: DIETClassifier
  • epochs: 200

however, not able to run it , getting below errors

Traceback (most recent call last): File “/Users/malarvizhisaravanan/opt/anaconda3/bin/rasa”, line 10, in sys.exit(main()) File “/Users/malarvizhisaravanan/opt/anaconda3/lib/python3.7/site-packages/rasa/main.py”, line 91, in main cmdline_arguments.func(cmdline_arguments) File “/Users/malarvizhisaravanan/opt/anaconda3/lib/python3.7/site-packages/rasa/cli/train.py”, line 140, in train_nlu persist_nlu_training_data=args.persist_nlu_data, File “/Users/malarvizhisaravanan/opt/anaconda3/lib/python3.7/site-packages/rasa/train.py”, line 414, in train_nlu persist_nlu_training_data, File “uvloop/loop.pyx”, line 1456, in uvloop.loop.Loop.run_until_complete File “/Users/malarvizhisaravanan/opt/anaconda3/lib/python3.7/site-packages/rasa/train.py”, line 453, in _train_nlu_async persist_nlu_training_data=persist_nlu_training_data, File “/Users/malarvizhisaravanan/opt/anaconda3/lib/python3.7/site-packages/rasa/train.py”, line 482, in _train_nlu_with_validated_data persist_nlu_training_data=persist_nlu_training_data, File “/Users/malarvizhisaravanan/opt/anaconda3/lib/python3.7/site-packages/rasa/nlu/train.py”, line 75, in train trainer = Trainer(nlu_config, component_builder) File “/Users/malarvizhisaravanan/opt/anaconda3/lib/python3.7/site-packages/rasa/nlu/model.py”, line 142, in init components.validate_requirements(cfg.component_names) File “/Users/malarvizhisaravanan/opt/anaconda3/lib/python3.7/site-packages/rasa/nlu/components.py”, line 51, in validate_requirements component_class = registry.get_component_class(component_name) File “/Users/malarvizhisaravanan/opt/anaconda3/lib/python3.7/site-packages/rasa/nlu/registry.py”, line 173, in get_component_class return class_from_module_path(component_name) File “/Users/malarvizhisaravanan/opt/anaconda3/lib/python3.7/site-packages/rasa/utils/common.py”, line 196, in class_from_module_path if “.” in module_path: TypeError: argument of type ‘NoneType’ is not iterable

@dakshvar22 - any suggestions on this ?

18

@malarsarav Which version of Rasa are you using? Can you update to the latest version in a new virtual env and open a separate forum issue if the problem persists. Thanks

19

Did anyone solved the problem of bert-base-uncased no found error?

20

I am a newbie to RASA. Could someone please help me understand if I can use the above pipeline to create a Japanese language chatbot? Also, is it possible to use it using the Spacy pipeline provided in RASA docs and a language customization? Which of the two is recommended?

21

Hey @koaning , I’ve just got started with Rasa and have gone through basics. I have to mention that your videos and rasa NLU examples were quite helpful. As I understood, currently rasa doesn’t support pytorch based language models. I want to know if it’s possible to create a custom pipeline component to perform the language model related stuff in a python script by adding a pytorch based language model to the python script and then add it on the top of the pipeline (without using HFTransformersNLP on the top of the pipeline) just like we can add custom components for sentiment analysis for example. Sorry if the question is not that clear.

I’m using Rasa 2.2.8 btw. Even for newer versions, is it possible? I am currently doing a research project on fine-tuning a XLM-R based language model for Sinhalese using pytorch. It’ll be nice if I can add it as a custom component since Rasa doesn’t support that out of the box.