Rasa X training fails when using SpaCy model

[Deprecated] Rasa X Community Edition
1

Hello,

I have installed Rasa X on a server using quick-install. I then add the Action Server using a DockerHub image.

I have language: en_core_web_lg in my config.yml. When I test the bot in local mode on my personal computer, everything works fine. But on the server, when I train the bot using Rasa X, it fails.

I check on the Kubernetes logs by using kubectl logs rasa-rasa-worker-xxx and I find this:

2021-02-24 15:16:10 ERROR    rasa.server  - Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/rasa/nlu/utils/spacy_utils.py", line 49, in load_model
    return spacy.load(spacy_model_name, disable=["parser"])
  File "/opt/venv/lib/python3.8/site-packages/spacy/__init__.py", line 30, in load
    return util.load_model(name, **overrides)
  File "/opt/venv/lib/python3.8/site-packages/spacy/util.py", line 169, in load_model
    raise IOError(Errors.E050.format(name=name))
OSError: [E050] Can't find model 'en_core_web_lg'. It doesn't seem to be a shortcut link, a Python package or a valid path to a data directory.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/rasa/server.py", line 1020, in train
    training_result = await train_async(**training_payload)
  File "/opt/venv/lib/python3.8/site-packages/rasa/train.py", line 163, in train_async
    return await _train_async_internal(
  File "/opt/venv/lib/python3.8/site-packages/rasa/train.py", line 342, in _train_async_internal
    await _do_training(
  File "/opt/venv/lib/python3.8/site-packages/rasa/train.py", line 388, in _do_training
    model_path = await _train_nlu_with_validated_data(
  File "/opt/venv/lib/python3.8/site-packages/rasa/train.py", line 812, in _train_nlu_with_validated_data
    await rasa.nlu.train(
  File "/opt/venv/lib/python3.8/site-packages/rasa/nlu/train.py", line 97, in train
    trainer = Trainer(
  File "/opt/venv/lib/python3.8/site-packages/rasa/nlu/model.py", line 163, in __init__
    self.pipeline = self._build_pipeline(cfg, component_builder)
  File "/opt/venv/lib/python3.8/site-packages/rasa/nlu/model.py", line 174, in _build_pipeline
    component = component_builder.create_component(component_cfg, cfg)
  File "/opt/venv/lib/python3.8/site-packages/rasa/nlu/components.py", line 855, in create_component
    component = registry.create_component_by_config(component_config, cfg)
  File "/opt/venv/lib/python3.8/site-packages/rasa/nlu/registry.py", line 193, in create_component_by_config
    return component_class.create(component_config, config)
  File "/opt/venv/lib/python3.8/site-packages/rasa/nlu/utils/spacy_utils.py", line 81, in create
    nlp = cls.load_model(spacy_model_name)
  File "/opt/venv/lib/python3.8/site-packages/rasa/nlu/utils/spacy_utils.py", line 51, in load_model
    raise InvalidModelError(
rasa.nlu.model.InvalidModelError: Model 'en_core_web_lg' is not a linked spaCy model.  Please download and/or link a spaCy model, e.g. by running:
python -m spacy download en_core_web_md
python -m spacy link en_core_web_md en

2021-02-24 15:16:10 ERROR    rasa.server  - An unexpected error occurred during training. Error: Model 'en_core_web_lg' is not a linked spaCy model.  Please download and/or link a spaCy model, e.g. by running:
python -m spacy download en_core_web_md
python -m spacy link en_core_web_md en

I am using SpaCy 3, so no need to use the link command. I instead write language: en_core_web_lg in config.yml.

SpaCy 3 is installed on the server, and I downloaded en_core_web_lg 3.0.3.

I tried adding RUN python -m spacy download en_core_web_lg in the Dockerfile, still doesn’t fix it.

However, if I write language: en in config.yml, it works.

2

Hi @ChrisRahme the default Rasa Open Source image I believe only contains the medium sized english model. Did you try building a custom container, and if yes, could you share the Dockerfile you used?

1 Like
3

Thank you for your reply! Here is my Dockerfile:

FROM rasa/rasa-sdk:2.2.0

WORKDIR /app

COPY actions/requirements-actions.txt ./

USER root

RUN pip install -r requirements-actions.txt

COPY ./actions /app/actions
COPY data/lookups/* data/lookups/

USER 1001

I just tested tried training with language: en_core_web_md in config.yml, and it works, thank you very much.

Though I would prefer it if there was a way to use lg or French models for example.

EDIT: For some reason, when I use en_core_web_md instead of en, Rasa X stops answering. More like, it doesn’t even try to answer, it stays stuck on the initial action_listen.

I checked on the rasa-x pod’s logs and I found this:

ERROR:rasax.community.api.blueprints.stack:Failed to send message to Rasa Chat webhook. Error: [Errno 104] Connection reset by peer

ERROR:rasax.community.api.blueprints.stack:Failed to send message to Rasa Chat webhook. Error: [Errno 104] Connection reset by peer
[2021-02-25 19:41:01 +0000] - (sanic.access)[INFO][10.42.0.37:58268]: POST http://194.126.17.114/api/conversations/062ad93b8b4e42b88698d855f7c465b7/messages?environment=production  404 176
[2021-02-25 19:41:01 +0000] - (sanic.access)[INFO][10.42.0.37:59120]: POST http://194.126.17.114/api/conversations  422 204
[2021-02-25 19:41:01 +0000] [33] [ERROR] Exception occurred while handling uri: 'http://194.126.17.114/api/projects/default/models?limit=1&offset=0'
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/sanic/app.py", line 973, in handle_request
    response = await response
  File "/usr/local/lib/python3.8/dist-packages/rasax/community/api/decorators.py", line 216, in decorated_function
    return await await_and_return_response(args, kwargs, request)
  File "/usr/local/lib/python3.8/dist-packages/rasax/community/api/decorators.py", line 146, in await_and_return_response
    response = await response
  File "/usr/local/lib/python3.8/dist-packages/rasax/community/api/blueprints/models.py", line 45, in get_models
    models, total_models = await _model_service(request).get_models(
  File "/usr/local/lib/python3.8/dist-packages/rasax/community/services/model_service.py", line 748, in get_models
    minimum_compatible_version = await self.minimum_compatible_version()
  File "/usr/local/lib/python3.8/dist-packages/rasax/community/services/model_service.py", line 161, in minimum_compatible_version
    info = await stack_service.version()
  File "/usr/local/lib/python3.8/dist-packages/rasax/community/services/stack_service.py", line 95, in version
    response = await session.get(
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/client.py", line 504, in _request
    await resp.start(conn)
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/client_reqrep.py", line 847, in start
    message, payload = await self._protocol.read()  # type: ignore  # noqa
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/streams.py", line 591, in read
    await self._waiter
aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer
[2021-02-25 19:41:01 +0000] - (sanic.access)[INFO][10.42.0.37:58988]: GET http://194.126.17.114/api/projects/default/models?limit=1&offset=0  500 214
[2021-02-25 19:41:01 +0000] [29] [ERROR] Exception occurred while handling uri: 'http://194.126.17.114/api/projects/default/models?limit=1&offset=0&tag=production'
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/connector.py", line 936, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "uvloop/loop.pyx", line 1974, in create_connection
  File "uvloop/loop.pyx", line 1951, in uvloop.loop.Loop.create_connection
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/sanic/app.py", line 973, in handle_request
    response = await response
  File "/usr/local/lib/python3.8/dist-packages/rasax/community/api/decorators.py", line 216, in decorated_function
    return await await_and_return_response(args, kwargs, request)
  File "/usr/local/lib/python3.8/dist-packages/rasax/community/api/decorators.py", line 146, in await_and_return_response
    response = await response
  File "/usr/local/lib/python3.8/dist-packages/rasax/community/api/blueprints/models.py", line 45, in get_models
    models, total_models = await _model_service(request).get_models(
  File "/usr/local/lib/python3.8/dist-packages/rasax/community/services/model_service.py", line 748, in get_models
    minimum_compatible_version = await self.minimum_compatible_version()
  File "/usr/local/lib/python3.8/dist-packages/rasax/community/services/model_service.py", line 161, in minimum_compatible_version
    info = await stack_service.version()
  File "/usr/local/lib/python3.8/dist-packages/rasax/community/services/stack_service.py", line 95, in version
    response = await session.get(
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/client.py", line 480, in _request
    conn = await self._connector.connect(
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/connector.py", line 523, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/connector.py", line 858, in _create_connection
    _, proto = await self._create_direct_connection(
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/connector.py", line 1004, in _create_direct_connection
    raise last_exc
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/connector.py", line 980, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/connector.py", line 943, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host rasa-rasa-x-rasa-production.rasa.svc:5005 ssl:default [Connection refused]
4

if you want to use the large model, then you’d have to build a custom Rasa Open Source image, not a custom rasa SDK image.

Something like:

FROM rasa/rasa:2.2.3-full

RUN python -m spacy download en_core_web_lg

If you just use en in your config file with the default image, that should use the en_core_web_md by default, so you can switch that back in the config file.

Could you share the rasa-production logs from that error though? Looks like that container has become unreachable for some reason.

1 Like
5

Thank you!

Unfortunately, I don’t have access to the logs anymore.

6

@akelad in this case, how would you run CMD [“start”, “–actions”, “actions”] in the same Dockerfile to run the action server?

I have the same issue as above and my Dockerfile is as follows:

FROM rasa/rasa-sdk:latest
USER root
WORKDIR /app
COPY requirements.txt /app
COPY actions.py /app
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
CMD ["start", "--actions", "actions"]
USER 1001