Rasa X Training Error

So I have set up Rasa-X on an AWS EC2 instance and all was going well until I started using ‘HFTransformersNLP’. Now I get errors saying: File “/opt/venv/lib/python3.6/site-packages/transformers/tokenization_utils.py”, line 400, in _from_pretrained raise EnvironmentError(msg) OSError: Couldn’t reach server at ‘{}’ to download vocabulary files.

and

File "/usr/local/lib/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/.cache'

from the rasa-worker pod. Any ideas on how to fix this?

1 Like

Hi Shay,

Can you paste your full config.yml? You should also review this post with an example pipeline.

Greg

I am also getting this after training a bert model and importing it to Rasa X.

My config.yml is the following:

# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/
language: "en"

pipeline:
  - name: HFTransformersNLP
    model_weights: "bert-base-uncased"
    model_name: "bert"
  - name: LanguageModelTokenizer
    intent_tokenization_flag: True
    intent_split_symbol: "+"
  - name: LanguageModelFeaturizer    
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: "char_wb"
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 30
    num_transformer_layers: 4
    transformer_size: 256
    use_masked_language_model: True
    drop_rate: 0.25
    weight_sparsity: 0.7
    batch_size: [32, 128]
    embedding_dimension: 30
    hidden_layer_sized:
      text: [512, 128]
  - name: "DucklingHTTPExtractor"
    url: "http://localhost:8000"
    dimensions: ["time", "number"]
    locale: "en_US"
    timezone: "US/Pacific"
    timeout : 3
  - name: EntitySynonymMapper

# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
  - name: MemoizationPolicy
  - name: TEDPolicy
    featurizer:
    - name: MaxHistoryTrackerFeaturizer
      max_history: 5
      state_featurizer:
        - name: LabelTokenizerSingleStateFeaturizer
  - name: MappingPolicy
  - name: FormPolicy
  - name: FallbackPolicy
    ambiguity_threshold: 0.1
    nlu_threshold: 0.4
    core_threshold: 0.3
    fallback_action_name: "action_default_fallback"

Here is the error in my docker-compose environment for Rasa X:

rasa-worker_1      | 2020-04-30 20:09:05 ERROR    rasa.core.agent  - An exception was raised while fetching a model. Continuing anyways...
rasa-worker_1      | Traceback (most recent call last):
rasa-worker_1      |   File "/opt/venv/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 383, in _from_pretrained
rasa-worker_1      |     resume_download=resume_download,
rasa-worker_1      |   File "/opt/venv/lib/python3.6/site-packages/transformers/file_utils.py", line 238, in cached_path
rasa-worker_1      |     user_agent=user_agent,
rasa-worker_1      |   File "/opt/venv/lib/python3.6/site-packages/transformers/file_utils.py", line 349, in get_from_cache
rasa-worker_1      |     os.makedirs(cache_dir, exist_ok=True)
rasa-worker_1      |   File "/usr/local/lib/python3.6/os.py", line 210, in makedirs
rasa-worker_1      |     makedirs(head, mode, exist_ok)
rasa-worker_1      |   File "/usr/local/lib/python3.6/os.py", line 210, in makedirs
rasa-worker_1      |     makedirs(head, mode, exist_ok)
rasa-worker_1      |   File "/usr/local/lib/python3.6/os.py", line 220, in makedirs
rasa-worker_1      |     mkdir(name, mode)
rasa-worker_1      | PermissionError: [Errno 13] Permission denied: '/.cache'
rasa-worker_1      | 
rasa-worker_1      | During handling of the above exception, another exception occurred:
rasa-worker_1      | 
rasa-worker_1      | Traceback (most recent call last):
rasa-worker_1      |   File "/opt/venv/lib/python3.6/site-packages/rasa/core/agent.py", line 203, in _run_model_pulling_worker
rasa-worker_1      |     await _update_model_from_server(model_server, agent)
rasa-worker_1      |   File "/opt/venv/lib/python3.6/site-packages/rasa/core/agent.py", line 125, in _update_model_from_server
rasa-worker_1      |     _load_and_set_updated_model(agent, model_directory, new_model_fingerprint)
rasa-worker_1      |   File "/opt/venv/lib/python3.6/site-packages/rasa/core/agent.py", line 86, in _load_and_set_updated_model
rasa-worker_1      |     interpreter = RasaNLUInterpreter(model_directory=nlu_path)
rasa-worker_1      |   File "/opt/venv/lib/python3.6/site-packages/rasa/core/interpreter.py", line 274, in __init__
rasa-worker_1      |     self._load_interpreter()
rasa-worker_1      |   File "/opt/venv/lib/python3.6/site-packages/rasa/core/interpreter.py", line 297, in _load_interpreter
rasa-worker_1      |     self.interpreter = Interpreter.load(self.model_directory)
rasa-worker_1      |   File "/opt/venv/lib/python3.6/site-packages/rasa/nlu/model.py", line 301, in load
rasa-worker_1      |     return Interpreter.create(model_metadata, component_builder, skip_validation)
rasa-worker_1      |   File "/opt/venv/lib/python3.6/site-packages/rasa/nlu/model.py", line 328, in create
rasa-worker_1      |     component_meta, model_metadata.model_dir, model_metadata, **context
rasa-worker_1      |   File "/opt/venv/lib/python3.6/site-packages/rasa/nlu/components.py", line 718, in load_component
rasa-worker_1      |     component_meta, model_dir, model_metadata, cached_component, **context
rasa-worker_1      |   File "/opt/venv/lib/python3.6/site-packages/rasa/nlu/registry.py", line 231, in load_component_by_meta
rasa-worker_1      |     component_meta, model_dir, metadata, cached_component, **kwargs
rasa-worker_1      |   File "/opt/venv/lib/python3.6/site-packages/rasa/nlu/components.py", line 445, in load
rasa-worker_1      |     return cls(meta)
rasa-worker_1      |   File "/opt/venv/lib/python3.6/site-packages/rasa/nlu/utils/hugging_face/hf_transformers.py", line 47, in __init__
rasa-worker_1      |     self._load_model()
rasa-worker_1      |   File "/opt/venv/lib/python3.6/site-packages/rasa/nlu/utils/hugging_face/hf_transformers.py", line 81, in _load_model
rasa-worker_1      |     self.model_weights, cache_dir=self.cache_dir
rasa-worker_1      |   File "/opt/venv/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 309, in from_pretrained
rasa-worker_1      |     return cls._from_pretrained(*inputs, **kwargs)
rasa-worker_1      |   File "/opt/venv/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 400, in _from_pretrained
rasa-worker_1      |     raise EnvironmentError(msg)
rasa-worker_1      | OSError: Couldn't reach server at '{}' to download vocabulary files.

this is file permission error when running in docker

u can use cache_dir for HFTransformersNLP

mkdir cache
chmod -R 777 cache

modify docker-compose.yml on module x-rasa-services: &default-rasa-service

add this:

volumes:
    - ./cache:/app/cache_dir