Using BERT with RASA

Rasa Open Source
1

BERT provides an option to include pre-trained language models from Hugging Face in pipline. As per the doc:

  • name: HFTransformersNLP

    Name of the language model to use

    model_name: “bert”

    Pre-Trained weights to be loaded

    model_weights: “bert-base-uncased”

    An optional path to a specific directory to download and cache the pre-trained model weights.

    The default cache_dir is the same as Export to ONNX .

    cache_dir: null

Following this I configured my pipeline as:

  • name: “HFTransformersNLP” # Name of the language model to use model_name: “bert” # Pre-Trained weights to be loaded model_weights: “bert-base-uncased” cache_dir: “C:/Project ABC/cache/”

But the problem is that on running the training steps, the model keeps failing with:

OSError: Model name ‘bert-base-uncased’ was not found in tokenizers model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased, bert-base-finnish-cased-v1, bert-base-finnish-uncased-v1, bert-base-dutch-cased). We assumed ‘bert-base-uncased’ was a path, a model identifier, or url to a directory containing vocabulary files named [‘vocab.txt’] but couldn’t find such vocabulary files at this path or url.

I did some research and it looks like that there might be issue in downloading the files from internet, so I manually downloaded the files config.json, pytorch_model.bin and placed it in C:/Project ABC/cache/ still I am getting the same error message. Any idea how to resolve this, not giving cache directory is failing too with the same error.

Posted the same in SO, but still did not get any response.

2

I am facing the exact same issue. Can someone help?

3

You’ll also need to download vocab.txt to your cache folder

4

Thanks for responding. Can u be more specific as to what cache u are referring here? I am using Pycharm IDE on a Windows 10 machine

5

Hi, thanks for the reply.

I downloaded the vocab.txt file from https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt and placed it in the cache folder as bert-base-uncased-vocab.txt. Still getting the below error.

OSError: Model name ‘bert-base-uncased’ was not found in tokenizers model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased, bert-base-finnish-cased-v1, bert-base-finnish-uncased-v1, bert-base-dutch-cased). We assumed ‘bert-base-uncased’ was a path, a model identifier, or url to a directory containing vocabulary files named [‘vocab.txt’] but couldn’t find such vocabulary files at this path or url.

I even tried renaming the file to vocab.txt still same issue

6

Can anyone please take a look at this… stuck for quite sometime on this issue.

1 Like
7

Trying to train with HF vectors in Rasa X with the docker-compose deployment, get a similar error:

rasa-x_1           | ERROR:rasax.community.api.blueprints.models:500, Internal Server Error, body='b'{"version":"1.9.5","status":"failure","message":"An unexpected error occurred during training. Error: Couldn\'t reach server at \'{}\' to download vocabulary files.","reason":"TrainingError","details":{},"help":null,"code":500}''
8

@dearc I am not sure how to reproduce the problem since I don’t have a windows machine. It can be a known bug in the Transformers library. Have you looked at their repo yet for known issues?

9

Hi, I have this issue when I use BERT in my pipeline. I’m using rasa/rasa:1.9.7-full docker image

docker run -v $(pwd):/app rasa/rasa:1.9.7-full test nlu --config pipelines/config-diet-bert.yml --cross-validation --runs 1 --folds 2 --out results/config-bert
2020-06-06 10:29:57 INFO     rasa.cli.test  - Test model using cross validation.
2020-06-06 10:29:58 INFO     transformers.file_utils  - TensorFlow version 2.1.0 available.
2020-06-06 10:29:59 INFO     filelock  - Lock 140131899698144 acquired on /tmp/96435fa287fbf7e469185f1062386e05a075cadbf6838b74da22bf64b080bc32.99bcd55fc66f4f3360bc49ba472b940b8dcf223ea6a345deb969d607ca900729.lock
Downloading: 100%|██████████| 996k/996k [00:00<00:00, 1.01MB/s] 
2020-06-06 10:30:00 INFO     filelock  - Lock 140131899698144 released on /tmp/96435fa287fbf7e469185f1062386e05a075cadbf6838b74da22bf64b080bc32.99bcd55fc66f4f3360bc49ba472b940b8dcf223ea6a345deb969d607ca900729.lock
2020-06-06 10:30:00 INFO     transformers.tokenization_utils  - loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-vocab.txt from cache at /tmp/96435fa287fbf7e469185f1062386e05a075cadbf6838b74da22bf64b080bc32.99bcd55fc66f4f3360bc49ba472b940b8dcf223ea6a345deb969d607ca900729
2020-06-06 10:30:01 INFO     filelock  - Lock 140131885427624 acquired on /tmp/45629519f3117b89d89fd9c740073d8e4c1f0a70f9842476185100a8afe715d1.65df3cef028a0c91a7b059e4c404a975ebe6843c71267b67019c0e9cfa8a88f0.lock
Downloading: 100%|██████████| 625/625 [00:00<00:00, 229kB/s]
2020-06-06 10:30:02 INFO     filelock  - Lock 140131885427624 released on /tmp/45629519f3117b89d89fd9c740073d8e4c1f0a70f9842476185100a8afe715d1.65df3cef028a0c91a7b059e4c404a975ebe6843c71267b67019c0e9cfa8a88f0.lock
2020-06-06 10:30:02 INFO     transformers.configuration_utils  - loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-config.json from cache at /tmp/45629519f3117b89d89fd9c740073d8e4c1f0a70f9842476185100a8afe715d1.65df3cef028a0c91a7b059e4c404a975ebe6843c71267b67019c0e9cfa8a88f0
2020-06-06 10:30:02 INFO     transformers.configuration_utils  - Model config BertConfig {
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "directionality": "bidi",
  "do_sample": false,
  "eos_token_ids": 0,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "layer_norm_eps": 1e-12,
  "length_penalty": 1.0,
  "max_length": 20,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_beams": 1,
  "num_hidden_layers": 12,
  "num_labels": 2,
  "num_return_sequences": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pad_token_id": 0,
  "pooler_fc_size": 768,
  "pooler_num_attention_heads": 12,
  "pooler_num_fc_layers": 3,
  "pooler_size_per_head": 128,
  "pooler_type": "first_token_transform",
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 119547
}

2020-06-06 10:30:03 INFO     filelock  - Lock 140132034860368 acquired on /tmp/65bfc60b1a6953086f6a6abb80a8a255a8db9ba53caef6c03392ecb155e1d447.1b4841f14bf42137fc7ecee17a46c1b2f22b417f636347e4b810bd06dd9c45ea.h5.lock
Downloading: 100%|██████████| 1.08G/1.08G [03:28<00:00, 5.18MB/s] 
2020-06-06 10:33:32 INFO     filelock  - Lock 140132034860368 released on /tmp/65bfc60b1a6953086f6a6abb80a8a255a8db9ba53caef6c03392ecb155e1d447.1b4841f14bf42137fc7ecee17a46c1b2f22b417f636347e4b810bd06dd9c45ea.h5.lock
2020-06-06 10:33:32 INFO     transformers.modeling_tf_utils  - loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-tf_model.h5 from cache at /tmp/65bfc60b1a6953086f6a6abb80a8a255a8db9ba53caef6c03392ecb155e1d447.1b4841f14bf42137fc7ecee17a46c1b2f22b417f636347e4b810bd06dd9c45ea.h5
2020-06-06 10:33:33.132481: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)

The process terminate after the last line and the results/config-bert folder is created but it remains empty.

10

Hi Team, I am trying to use BERT model in the pipeline in RASA version 1.10.1. image

After executing rasa train command getting below error;

(rasa_new) [dnachatbot@bbsr02cloud05 rasa_init]$ rasa train 2020-06-17 13:37:59 INFO rasa.model - Data (nlu-config) for NLU model section changed. Core stories/configuration did not change. No need to retrain Core model. Training NLU model… 2020-06-17 13:38:00 INFO transformers.file_utils - PyTorch version 1.5.0 available. 2020-06-17 13:38:00 INFO transformers.file_utils - TensorFlow version 2.1.0 available. 2020-06-17 13:38:00 INFO transformers.tokenization_utils - Model name ‘/dnachatbot/rahul/rasa_init/bert_config.json’ not found in model shortcut name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased, TurkuNLP/bert-base-finnish-cased-v1, TurkuNLP/bert-base-finnish-uncased-v1, wietsedv/bert-base-dutch-cased). Assuming ‘/dnachatbot/rahul/rasa_init/bert_config.json’ is a path, a model identifier, or url to a directory containing tokenizer files. 2020-06-17 13:38:00 WARNING transformers.tokenization_utils - Calling BertTokenizer.from_pretrained() with the path to a single file or url is deprecated 2020-06-17 13:38:00 INFO transformers.tokenization_utils - loading file /dnachatbot/rahul/rasa_init/bert_config.json 2020-06-17 13:38:00 INFO transformers.configuration_utils - loading configuration file /dnachatbot/rahul/rasa_init/bert_config.json 2020-06-17 13:38:00 INFO transformers.configuration_utils - Model config BertConfig { “attention_probs_dropout_prob”: 0.1, “hidden_act”: “gelu”, “hidden_dropout_prob”: 0.1, “hidden_size”: 768, “initializer_range”: 0.02, “intermediate_size”: 3072, “layer_norm_eps”: 1e-12, “max_position_embeddings”: 512, “model_type”: “bert”, “num_attention_heads”: 12, “num_hidden_layers”: 12, “pad_token_id”: 0, “type_vocab_size”: 2, “vocab_size”: 30522 }

2020-06-17 13:38:00 INFO transformers.modeling_tf_utils - loading weights file /dnachatbot/rahul/rasa_init/bert_config.json 2020-06-17 13:38:00.497039: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303) Traceback (most recent call last): File “/dnachatbot/murali/rasa_new/lib/python3.6/site-packages/transformers/modeling_tf_utils.py”, line 401, in from_pretrained model.load_weights(resolved_archive_file, by_name=True) File “/dnachatbot/murali/rasa_new/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py”, line 234, in load_weights return super(Model, self).load_weights(filepath, by_name, skip_mismatch) File “/dnachatbot/murali/rasa_new/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/network.py”, line 1215, in load_weights with h5py.File(filepath, ‘r’) as f: File “/dnachatbot/murali/rasa_new/lib/python3.6/site-packages/h5py/_hl/files.py”, line 408, in init swmr=swmr) File “/dnachatbot/murali/rasa_new/lib/python3.6/site-packages/h5py/_hl/files.py”, line 173, in make_fid fid = h5f.open(name, flags, fapl=fapl) File “h5py/_objects.pyx”, line 54, in h5py._objects.with_phil.wrapper File “h5py/_objects.pyx”, line 55, in h5py._objects.with_phil.wrapper File “h5py/h5f.pyx”, line 88, in h5py.h5f.open OSError: Unable to open file (file signature not found)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File “/dnachatbot/murali/rasa_new/bin/rasa”, line 8, in sys.exit(main()) File “/dnachatbot/murali/rasa_new/lib/python3.6/site-packages/rasa/main.py”, line 91, in main cmdline_arguments.func(cmdline_arguments) File “/dnachatbot/murali/rasa_new/lib/python3.6/site-packages/rasa/cli/train.py”, line 76, in train additional_arguments=extract_additional_arguments(args), File “/dnachatbot/murali/rasa_new/lib/python3.6/site-packages/rasa/train.py”, line 50, in train additional_arguments=additional_arguments, File “uvloop/loop.pyx”, line 1451, in uvloop.loop.Loop.run_until_complete File “/dnachatbot/murali/rasa_new/lib/python3.6/site-packages/rasa/train.py”, line 101, in train_async additional_arguments, File “/dnachatbot/murali/rasa_new/lib/python3.6/site-packages/rasa/train.py”, line 188, in _train_async_internal additional_arguments=additional_arguments, File “/dnachatbot/murali/rasa_new/lib/python3.6/site-packages/rasa/train.py”, line 245, in _do_training persist_nlu_training_data=persist_nlu_training_data, File “/dnachatbot/murali/rasa_new/lib/python3.6/site-packages/rasa/train.py”, line 482, in _train_nlu_with_validated_data persist_nlu_training_data=persist_nlu_training_data, File “/dnachatbot/murali/rasa_new/lib/python3.6/site-packages/rasa/nlu/train.py”, line 75, in train trainer = Trainer(nlu_config, component_builder) File “/dnachatbot/murali/rasa_new/lib/python3.6/site-packages/rasa/nlu/model.py”, line 145, in init self.pipeline = self._build_pipeline(cfg, component_builder) File “/dnachatbot/murali/rasa_new/lib/python3.6/site-packages/rasa/nlu/model.py”, line 157, in _build_pipeline component = component_builder.create_component(component_cfg, cfg) File “/dnachatbot/murali/rasa_new/lib/python3.6/site-packages/rasa/nlu/components.py”, line 769, in create_component component = registry.create_component_by_config(component_config, cfg) File “/dnachatbot/murali/rasa_new/lib/python3.6/site-packages/rasa/nlu/registry.py”, line 246, in create_component_by_config return component_class.create(component_config, config) File “/dnachatbot/murali/rasa_new/lib/python3.6/site-packages/rasa/nlu/components.py”, line 483, in create return cls(component_config) File “/dnachatbot/murali/rasa_new/lib/python3.6/site-packages/rasa/nlu/utils/hugging_face/hf_transformers.py”, line 47, in init self._load_model() File “/dnachatbot/murali/rasa_new/lib/python3.6/site-packages/rasa/nlu/utils/hugging_face/hf_transformers.py”, line 84, in _load_model self.model_weights, cache_dir=self.cache_dir File “/dnachatbot/murali/rasa_new/lib/python3.6/site-packages/transformers/modeling_tf_utils.py”, line 404, in from_pretrained "Unable to load weights from h5 file. " OSError: Unable to load weights from h5 file. If you tried to load a TF 2.0 model from a PyTorch checkpoint, please set from_pt=True.

Can you please help

11

Hi,

I got the same kind of problem using the french model of BERT, CamemBERT. I did follow advice from this link: RASA and camemBERT - #8 by MaxGdr

This helped me to configure my Rasa chatbot and now it’s working.

Hope it will help. :slight_smile: