Rasa version: 1.10.7 (also tried with 1.8.3)
Python version: 3.7
Operating system (windows, osx, …): Linux and Windows
Issue: I have a problem with integrating BERT into the configuration of my model. Namely, at the very beginning of training it gives the error message shown below. There is also a config.yml file I used.
It is interesting that I am not getting this error when using a very small train dataset, so the model is able to finish with training successfully.
Error (including full traceback):
2020-07-16 13:05:16 INFO rasa.nlu.model - Starting to train component HFTransformersNLP
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/bin/rasa", line 8, in <module>
sys.exit(main())
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/__main__.py", line 92, in main
cmdline_arguments.func(cmdline_arguments)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/cli/train.py", line 140, in train_nlu
persist_nlu_training_data=args.persist_nlu_data,
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/train.py", line 414, in train_nlu
persist_nlu_training_data,
File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/train.py", line 453, in _train_nlu_async
persist_nlu_training_data=persist_nlu_training_data,
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/train.py", line 482, in _train_nlu_with_validated_data
persist_nlu_training_data=persist_nlu_training_data,
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/nlu/train.py", line 90, in train
interpreter = trainer.train(training_data, **kwargs)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/nlu/model.py", line 191, in train
updates = component.train(working_data, self.config, **context)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/nlu/utils/hugging_face/hf_transformers.py", line 472, in train
batch_docs = self._get_docs_for_batch(batch_messages, attribute)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/nlu/utils/hugging_face/hf_transformers.py", line 423, in _get_docs_for_batch
) = self._get_model_features_for_batch(batch_token_ids)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/nlu/utils/hugging_face/hf_transformers.py", line 387, in _get_model_features_for_batch
batch_attention_mask, padded_token_ids
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/rasa/nlu/utils/hugging_face/hf_transformers.py", line 347, in _compute_batch_sequence_features
np.array(padded_token_ids), attention_mask=np.array(batch_attention_mask)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 822, in __call__
outputs = self.call(cast_inputs, *args, **kwargs)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/transformers/modeling_tf_bert.py", line 708, in call
outputs = self.bert(inputs, **kwargs)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 822, in __call__
outputs = self.call(cast_inputs, *args, **kwargs)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/transformers/modeling_tf_bert.py", line 570, in call
embedding_output = self.embeddings([input_ids, position_ids, token_type_ids, inputs_embeds], training=training)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 822, in __call__
outputs = self.call(cast_inputs, *args, **kwargs)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/transformers/modeling_tf_bert.py", line 150, in call
return self._embedding(inputs, training=training)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/transformers/modeling_tf_bert.py", line 173, in _embedding
position_embeddings = self.position_embeddings(position_ids)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 822, in __call__
outputs = self.call(cast_inputs, *args, **kwargs)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/layers/embeddings.py", line 184, in call
out = embedding_ops.embedding_lookup(self.embeddings, inputs)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/embedding_ops.py", line 323, in embedding_lookup
transform_fn=None)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/embedding_ops.py", line 137, in _embedding_lookup_and_transform
array_ops.gather(params[0], ids, name=name), ids, max_norm)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/util/dispatch.py", line 180, in wrapper
return target(*args, **kwargs)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py", line 4106, in gather
return params.sparse_read(indices, name=name)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/resource_variable_ops.py", line 645, in sparse_read
self._handle, indices, dtype=self._dtype, name=name)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_resource_variable_ops.py", line 556, in resource_gather
_ops.raise_from_not_ok_status(e, name)
File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 6606, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[0,512] = 512 is not in [0, 512) [Op:ResourceGather] name: tf_bert_model/bert/embeddings/position_embeddings/embedding_lookup/
Command or request that led to error:
rasa train nlu
Content of configuration file (config.yml) (if relevant):
language: en
num_threads: 8
pipeline:
- name: HFTransformersNLP
model_weights: "bert-base-uncased"
model_name: "bert"
- name: LanguageModelTokenizer
- name: LanguageModelFeaturizer
- name: "CRFEntityExtractor"
- name: "EmbeddingIntentClassifier"