InvalidModelError: Model 'da_core_news_md' is not a linked spaCy model

Rasa Open Source
1

I want to create an assisant using a docker file, and when i try to train the bot the following error occurs:

InvalidModelError: Model ‘da_core_news_md’ is not a linked spaCy model. Please download and/or link a spaCy model, e.g. by running: python -m spacy download en_core_web_md python -m spacy link en_core_web_md en

But in the docker file i have made that exact command. I also tried to change the language in the config file.

What am I doing wrong?

Thanks in advance.

System information: Docker version: 20.10.5 OS: Windows 10

2

I’ve narrowed it down to that the error is caused by another error:

Failed to write global config. Error: [Errno 13] Permission denied: '/.config'. Skipping

Also it says that (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory

Are these two errors the cause of the SpaCy problem, and are they correlated in any way?

And the big question: How do i fix them

Thanks in advance

3

With Rasa 2.5 this should all be a thing of the past. We’ve upgraded to spaCy 3.0 which deprecates the spacy link feature. After downloading the model you only need to make sure that you’re referring to the correct model. Like so:

pipeline:
- name: SpacyNLP
  model: da_core_news_md

Let me know if upgrading doesn’t help.

4

But in the docker file i have made that exact command

The exacy command above is for the english medium model, do you have that command in your dockerfile? In yours, it should look like

python -m spacy download da_core_news_md
python -m spacy link da_core_news_md da

FYI, in rasa 2.5, the spacy dependency is updated to spacy 3.0, where spacy link has been deprecated. More info about migrating can be found here: Version Migration Guide

5

My configuration is like this:

pipeline:

  • name: SpacyNLP

  • model: da_core_news_md

I tried running the command again, and this it makes a different error.

Actually i cant recreate the old error for some weird reason.

The new error looks like this: Traceback (most recent call last): File "/opt/venv/bin/rasa", line 8, in <module> sys.exit(main()) File "/opt/venv/lib/python3.8/site-packages/rasa/__main__.py", line 116, in main cmdline_arguments.func(cmdline_arguments) File "/opt/venv/lib/python3.8/site-packages/rasa/cli/train.py", line 58, in <lambda> train_parser.set_defaults(func=lambda args: train(args, can_exit=True)) File "/opt/venv/lib/python3.8/site-packages/rasa/cli/train.py", line 90, in train training_result = rasa.train( File "/opt/venv/lib/python3.8/site-packages/rasa/train.py", line 94, in train return rasa.utils.common.run_in_loop( File "/opt/venv/lib/python3.8/site-packages/rasa/utils/common.py", line 307, in run_in_loop result = loop.run_until_complete(f) File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete File "/opt/venv/lib/python3.8/site-packages/rasa/train.py", line 155, in train_async domain = await file_importer.get_domain() File "/opt/venv/lib/python3.8/site-packages/rasa/shared/utils/common.py", line 141, in decorated return await cache.cached_result() File "/opt/venv/lib/python3.8/site-packages/rasa/shared/importers/importer.py", line 413, in get_domain original, e2e_domain = await asyncio.gather( File "/opt/venv/lib/python3.8/site-packages/rasa/shared/importers/importer.py", line 420, in _get_domain_with_e2e_actions stories = await self.get_stories() File "/opt/venv/lib/python3.8/site-packages/rasa/shared/importers/importer.py", line 455, in get_stories return await self.importer.get_stories( File "/opt/venv/lib/python3.8/site-packages/rasa/shared/importers/importer.py", line 369, in get_stories return await self._importer.get_stories( File "/opt/venv/lib/python3.8/site-packages/rasa/shared/utils/common.py", line 141, in decorated return await cache.cached_result() File "/opt/venv/lib/python3.8/site-packages/rasa/shared/importers/importer.py", line 254, in get_stories stories = await asyncio.gather(*stories) File "/opt/venv/lib/python3.8/site-packages/rasa/shared/importers/rasa.py", line 51, in get_stories await self.get_domain(), File "/opt/venv/lib/python3.8/site-packages/rasa/shared/importers/rasa.py", line 67, in get_domain domain = Domain.load(self._domain_path) File "/opt/venv/lib/python3.8/site-packages/rasa/shared/core/domain.py", line 123, in load other = cls.from_path(path) File "/opt/venv/lib/python3.8/site-packages/rasa/shared/core/domain.py", line 133, in from_path domain = cls.from_file(path) File "/opt/venv/lib/python3.8/site-packages/rasa/shared/core/domain.py", line 146, in from_file return cls.from_yaml(rasa.shared.utils.io.read_file(path), path) File "/opt/venv/lib/python3.8/site-packages/rasa/shared/core/domain.py", line 161, in from_yaml return cls.from_dict(data) File "/opt/venv/lib/python3.8/site-packages/rasa/shared/core/domain.py", line 185, in from_dict return cls( File "/opt/venv/lib/python3.8/site-packages/rasa/shared/core/domain.py", line 545, in __init__ action_names += overridden_form_actions TypeError: unsupported operand type(s) for +=: 'NoneType' and 'list'

Thanks in advance

6

Can you provide your domain file? Looks like you don’t have actions correctly listed in it

7

I found the error so I can recreate the old error :sweat_smile:

Now my domain file looks like this:

version: '2.0'
config:
  store_entities_as_slots: true
session_config:
  session_expiration_time: 60
  carry_over_slots_to_new_session: true
intents:
- mood_unhappy:
    use_entities: true
- Datasikkerhed:
    use_entities: true
- Hilsen:
    use_entities: true
- greet:
    use_entities: true
- goodbye:
    use_entities: true
- affirm:
    use_entities: true
- deny:
    use_entities: true
- mood_great:
    use_entities: true
- bot_challenge:
    use_entities: true
- nlu_fallback:
    use_entities: true
- Betafunktioner:
    use_entities: true
- DreamplanVsBank:
    use_entities: true
- KontaktBank:
    use_entities: true
- AndreTingAfValue:
    use_entities: true
- SkalJegGoreDet:
    use_entities: true
- HvordanGorJeg:
    use_entities: true
entities: []
slots: {}
responses:
  utter_greet:
  - text: Hey! How are you?
  utter_cheer_up:
  - image: https://i.imgur.com/nGF1K8f.jpg
    text: 'Here is something to cheer you up:'
  utter_did_that_help:
  - text: Did that help you?
  utter_happy:
  - text: Great, carry on!
  utter_goodbye:
  - text: Bye
  utter_iamabot:
  - text: I am a bot, powered by Rasa.
  utter_data:
  - text: Vi opbevarer din data fortroligt i forhold til de nuværende GDPR regler.
  utter_fallback:
  - text: Det forstod jeg desværre ikke, kan du prøve at omformulere det
  utter_beta:
  - text: Det er desværre ikke muligt lige pt, men vi arbejder på at få det implementeret.
  utter_DreamplanVsBank:
  - text: Vi har ikke de samme værktøjer som din bank har, så vi kan ikke lave om i din økonomi. Men vi kan præsentere nogle løsninger du kan lave om i ganske simpelt. Samtidig kan løsningen ændres baseret på dine drømme.
  utter_KontaktBank:
  - text: Ideen med vores service, at du bliver bedre forberedt til samtalen med din bank. Det betyder ikke at du skal snakke med banken.
  utter_AndreTingAfValue:
  - text: Vores beregninger er lavet ud fra nogle antagelser og baseret på bankers samt finanstilsynets anbefalinger, så vi kan desværre ikke medregne alle former for indtægt endnu.
  utter_HvordanGorJeg:
  - text: Det ville vi anbefale at du snakkede med din bankrådgiver om, da vi ikke kender din livssituation.
actions: []
forms: {}
e2e_actions: []
9

Do you mean that you fixed the long traceback and are now back to the spacy thing?

10

Yes I got back to getting the spacy error :sweat_smile:

11

Ok, then I go back to my initial response :grinning_face_with_smiling_eyes: Do you download the da_core_news_md model in your dockerfile?

12

I would think so,

I run this command: RUN python -m spacy download da_core_news_md

13

Ok, and then do you link it with

RUN python -m spacy link da_core_news_md da

(it might be easier if you just provide your dockerfile and config file)

14

Thanks for the quick reply

my dockerfile looks like this:

FROM python:3.7.6-slim-buster

RUN pip install -U pip setuptools wheel

RUN pip install -U spacy==3.0.0

RUN apt-get update

RUN apt-get upgrade -y

RUN apt-get install -y git

RUN python -m spacy download da_core_news_md

RUN pip install Ujson

RUN pip install tensorflow

COPY script.py .

CMD python script.py

And my config file: pipeline:

  • name: SpacyNLP

    model: da_core_news_md

  • name: SpacyTokenizer

  • name: SpacyEntityExtractor

  • name: SpacyFeaturizer

    pooling: mean

  • name: CountVectorsFeaturizer

    analyzer: char_wb

    min_ngram: 1

    max_ngram: 2

  • name: DIETClassifier

  • name: FallbackClassifier

    threshold: 0.8

    ambiguity_threshold: 0.1

epochs: 1 `

15

This is a bit odd – ideally you would be building your image on top of the rasa spacy image instead of building from python. This will make sure that you’re running versions of spacy, tensorflow, etc. that are compatible with the rasa version you’re running (of which I’m still not sure which one you’re trying to run).

For example, you could take this rasa spacy image

And then make sure to add RUN python -m spacy download da_core_news_md to your dockerfile extending it. You’d already have spacy and tensrflow installed in that case, and the compatible versions to rasa 2.5.0

16

The reason for that is because im not smart :sweat_smile: Of course Rasa has a docker image available.

I will try changing to this and report back to you.

Again thanks for the quick reply

1 Like
17

Haha, no worries, it’s a big hard to spot in the docs :grinning_face_with_smiling_eyes:

18

So to clarify:

My dockerfile should look like this:

FROM rasa/rasa:2.5.0-spacy-de

RUN python -m spacy download da_core_news_md

COPY script.py .

CMD python script.py

Or have i misunderstood something

19

That looks like it should work, yes! Because you’ll have both spacy and your model installed, and you reference the model directly by name in your config file. Are you still running into the issue?

20

sorry for the late reply.

I tried putting it in, and now I cant run the usual cmd of docker build -t <name> .

Now i get the following error:

[2/3] RUN python -m spacy download da_core_news_md:

5 0.683 2021-05-02 19:15:32.128996: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library ‘libcudart.so.10.1’; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory

5 0.683 2021-05-02 19:15:32.129087: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.

5 2.669 WARNING: The directory ‘/.cache/pip/http’ or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo’s -H flag.

5 2.670 WARNING: The directory ‘/.cache/pip’ or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo’s -H flag.

5 2.703 Collecting da-core-news-md==3.0.0

5 2.902 Downloading https://github.com/explosion/spacy-models/releases/download/da_core_news_md-3.0.0/da_core_news_md-3.0.0-py3-none-any.whl (49.7MB)

5 13.72 Requirement already satisfied: spacy<3.1.0,>=3.0.0 in /opt/venv/lib/python3.8/site-packages (from da-core-news-md==3.0.0) (3.0.5)

5 13.77 Requirement already satisfied: pydantic<1.8.0,>=1.7.1 in /opt/venv/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (1.7.3)

5 13.78 Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.0 in /opt/venv/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (3.0.2)

5 13.78 Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /opt/venv/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (1.0.5)

5 13.78 Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /opt/venv/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (2.0.5)

5 13.78 Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /opt/venv/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (3.0.5)

5 13.79 Requirement already satisfied: jinja2 in /opt/venv/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (2.11.3)

5 13.79 Requirement already satisfied: pathy>=0.3.5 in /opt/venv/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (0.4.0)

5 13.80 Requirement already satisfied: numpy>=1.15.0 in /opt/venv/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (1.18.5)

5 13.80 Requirement already satisfied: catalogue<2.1.0,>=2.0.1 in /opt/venv/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (2.0.1)

5 13.81 Requirement already satisfied: srsly<3.0.0,>=2.4.0 in /opt/venv/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (2.4.1)

5 13.82 Requirement already satisfied: requests<3.0.0,>=2.13.0 in /opt/venv/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (2.25.1)

5 13.83 Requirement already satisfied: packaging>=20.0 in /opt/venv/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (20.9)

5 13.84 Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in /opt/venv/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (0.8.2)

5 13.84 Requirement already satisfied: typer<0.4.0,>=0.3.0 in /opt/venv/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (0.3.2)

5 13.86 Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /opt/venv/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (4.59.0)

5 13.88 Requirement already satisfied: setuptools in /opt/venv/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (44.0.0)

5 13.88 Requirement already satisfied: thinc<8.1.0,>=8.0.2 in /opt/venv/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (8.0.2)

5 13.91 Requirement already satisfied: blis<0.8.0,>=0.4.0 in /opt/venv/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (0.7.4)

5 13.92 Requirement already satisfied: MarkupSafe>=0.23 in /opt/venv/lib/python3.8/site-packages (from jinja2->spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (1.1.1)

5 13.92 Requirement already satisfied: smart-open<4.0.0,>=2.2.0 in /opt/venv/lib/python3.8/site-packages (from pathy>=0.3.5->spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (3.0.0)

5 13.93 Requirement already satisfied: certifi>=2017.4.17 in /opt/venv/lib/python3.8/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (2020.12.5)

5 13.93 Requirement already satisfied: idna<3,>=2.5 in /opt/venv/lib/python3.8/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (2.10)

5 13.95 Requirement already satisfied: chardet<5,>=3.0.2 in /opt/venv/lib/python3.8/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (3.0.4)

5 13.95 Requirement already satisfied: pyparsing>=2.0.2 in /opt/venv/lib/python3.8/site-packages (from packaging>=20.0->spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (2.4.7)

5 13.95 Requirement already satisfied: click<7.2.0,>=7.1.1 in /opt/venv/lib/python3.8/site-packages (from typer<0.4.0,>=0.3.0->spacy<3.1.0,>=3.0.0->da-core-news-md==3.0.0) (7.1.2)

5 14.55 Installing collected packages: da-core-news-md

5 14.55 ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: ‘/opt/venv/lib/python3.8/site-packages/da_core_news_md-3.0.0.dist-info’

5 14.55 Consider using the --user option or check the permissions.

5 14.55

5 14.80 You should consider upgrading via the ‘pip install --upgrade pip’ command

executor failed running [/bin/bash -o pipefail -c python -m spacy download da_core_news_md]: exit code: 1

21

I’m unfamiliar with the permission details in our docker container, but this is a bit strange. @erohmensing are users not allowed to install python packages to go along with Rasa here?

Technically, spaCy does do a pip install here. After downloading a spaCy model locally you are able to see it installed in pip freeze. Here’s an example from a local repo on my end.

whatlies git:(main) > pip freeze | grep en-core
en-core-web-md==3.0.0
en-core-web-sm==3.0.0

I’m also wondering if it makes sense to install the Danish model in a container that has the German model. Won’t the container be needlessly big? Looking at the hub file it seems like we’re switching to a 1001 user and I think we’re also assuming that the user won’t install anything with pip after.