Rasa X: Worker cannot train the model after modifying the docker file for third language

I have successfully installed the rasa-x in the Docker-compose mode, trained the demo model from rasa-worker, and launched the chatbot with rasa-production and rasa-x.

Then, I modified the docker image to support Chinese language using spacy. Specifically, it installed a Chinese dictionary, zh_core_web_md. After updating the docker-compose.yml, the rasa-x is launched and the chatbot works with both English and Chinese model. I can talk to and share the bot via rasa-x, which means the rasa-production works well too. However, neither the English model nor the Chinese model can be trained via rasa-x.

After going through the docker logs, seem the rasa-worker failed to extract the model which leads to an error. However, when we initialized the rasa-x, the rasa-worker was successfully initialized and the model was extracted without error. Why it failed to extract the model for training?

The Dockerfile we used to edit docker image:

FROM rasa/rasa:2.1.2-full
# Change back to root user to install dependencies
USER root
COPY ./zh_core_web_md-2.3.1.tar.gz /app/
RUN pip install zh_core_web_md-2.3.1.tar.gz -i
# By best practices, don't run the code with root user
USER 1001

The log of rasa-worker which has been successfully initialized.

2021-03-05 06:51:12 DEBUG    rasa.telemetry  - Could not read telemetry settings from configuration file: Configuration 'metrics' key not found.,
2021-03-05 06:51:13 DEBUG    sanic_jwt.configuration  - validating provided secret,
2021-03-05 06:51:13 DEBUG    sanic_jwt.configuration  - validating keys (if needed),
2021-03-05 06:51:13 DEBUG    sanic_jwt.configuration  - loading secret and/or keys (if needed),
2021-03-05 06:51:13 DEBUG    rasa.core.utils  - Available web server routes: ,
/conversations/<conversation_id:path>/messages     POST                           add_message,
/conversations/<conversation_id:path>/tracker/events POST                           append_events,
/auth                                              POST                           auth_bp.AuthenticateEndpoint,
/auth/me                                           GET                            auth_bp.RetrieveUserEndpoint,
/auth/verify                                       GET                            auth_bp.VerifyEndpoint,
/webhooks/rasa                                     GET                            custom_webhook_RasaChatInput.health,
/webhooks/rasa/webhook                             POST                           custom_webhook_RasaChatInput.receive,
/webhooks/rest                                     GET                            custom_webhook_RestInput.health,
/webhooks/rest/webhook                             POST                           custom_webhook_RestInput.receive,
/model/test/intents                                POST                           evaluate_intents,
/model/test/stories                                POST                           evaluate_stories,
/conversations/<conversation_id:path>/execute      POST                           execute_action,
/domain                                            GET                            get_domain,
/                                                  GET                            hello,
/model                                             PUT                            load_model,
/model/parse                                       POST                           parse,
/conversations/<conversation_id:path>/predict      POST                           predict,
/conversations/<conversation_id:path>/tracker/events PUT                            replace_events,
/conversations/<conversation_id:path>/story        GET                            retrieve_story,
/conversations/<conversation_id:path>/tracker      GET                            retrieve_tracker,
/status                                            GET                            status,
/model/predict                                     POST                           tracker_predict,
/model/train                                       POST                           train,
/conversations/<conversation_id:path>/trigger_intent POST                           trigger_intent,
/model                                             DELETE                         unload_model,
/version                                           GET                            version,
2021-03-05 06:51:13 INFO     root  - Starting Rasa server on http://localhost:5005,
2021-03-05 06:51:13 DEBUG    rasa.core.utils  - Using the default number of Sanic workers (1).,
2021-03-05 06:51:14 INFO     root  - Enabling coroutine debugging. Loop id 94682458493984.,
2021-03-05 06:51:14 DEBUG    root  - Could not load interpreter from 'None'.,
2021-03-05 06:51:14 INFO     rasa.core.brokers.pika  - Connecting to RabbitMQ ...,
2021-03-05 06:51:14 DEBUG    rasa.core.brokers.pika  - Connecting to 'rabbit' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.,
2021-03-05 06:51:19 DEBUG    rasa.core.brokers.pika  - Connecting to 'rabbit' failed with error '[Errno 111] Connection refused'. Trying again.,
2021-03-05 06:51:24 DEBUG    rasa.core.brokers.pika  - Connecting to 'rabbit' failed with error '[Errno 111] Connection refused'. Trying again.,
2021-03-05 06:51:29 INFO     rasa.core.brokers.pika  - RabbitMQ connection to 'rabbit' was established.,
2021-03-05 06:51:29 DEBUG    rasa.core.brokers.pika  - RabbitMQ channel was opened. Declaring fanout exchange.,
2021-03-05 06:51:29 DEBUG    rasa.core.brokers.broker  - Instantiated event broker to 'PikaEventBroker'.,
2021-03-05 06:51:29 DEBUG    rasa.core.tracker_store  - Attempting to connect to database via 'postgresql://admin:***@db:5432/rasa'.,
2021-03-05 06:51:29 DEBUG    rasa.core.tracker_store  - Connection to SQL database 'worker_tracker' successful.,
2021-03-05 06:51:29 DEBUG    rasa.core.tracker_store  - Connected to SQLTrackerStore.,
2021-03-05 06:51:29 DEBUG    rasa.core.lock_store  - Connected to lock store 'RedisLockStore'.,
2021-03-05 06:51:29 DEBUG    rasa.core.nlg.generator  - Instantiated NLG to 'TemplatedNaturalLanguageGenerator'.,
2021-03-05 06:51:29 DEBUG    rasa.core.agent  - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:51:29 DEBUG    rasa.core.agent  - Model server could not find a model at the requested endpoint 'http://rasa-x:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.,
2021-03-05 06:51:29 DEBUG    rasa.core.agent  - No new model found at URL http://rasa-x:5002/api/projects/default/models/tags/production,
2021-03-05 06:51:29 INFO     root  - Rasa server is up and running.,
2021-03-05 06:51:39 DEBUG    rasa.core.agent  - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:51:40 DEBUG    rasa.core.agent  - Unzipped model to '/tmp/tmpqtxyh7l_',
2021-03-05 06:51:40 DEBUG    rasa.core.agent  - Found new model with fingerprint 51973938fc5e9fc2bffd93e5efe71652. Loading...,
/opt/venv/lib/python3.7/site-packages/thinc/neural/_custom_kernels.py:36: ResourceWarning: unclosed file <_io.TextIOWrapper name='/opt/venv/lib/python3.7/site-packages/thinc/neural/_custom_kernels.cu' mode='r' encoding='utf8'>,
  SRC = (PWD / "_custom_kernels.cu").open("r", encoding="utf8").read(),
ResourceWarning: Enable tracemalloc to get the object allocation traceback,
/opt/venv/lib/python3.7/site-packages/thinc/neural/_custom_kernels.py:39: ResourceWarning: unclosed file <_io.TextIOWrapper name='/opt/venv/lib/python3.7/site-packages/thinc/neural/_murmur3.cu' mode='r' encoding='utf8'>,
  MMH_SRC = (PWD / "_murmur3.cu").open("r", encoding="utf8").read(),
ResourceWarning: Enable tracemalloc to get the object allocation traceback,
Building prefix dict from the default dictionary ...,
2021-03-05 06:51:41 DEBUG    jieba  - Building prefix dict from the default dictionary ...,
Dumping model to file cache /tmp/jieba.cache,
2021-03-05 06:51:42 DEBUG    jieba  - Dumping model to file cache /tmp/jieba.cache,
Loading model cost 0.854 seconds.,
2021-03-05 06:51:42 DEBUG    jieba  - Loading model cost 0.854 seconds.,
Prefix dict has been built successfully.,
2021-03-05 06:51:42 DEBUG    jieba  - Prefix dict has been built successfully.,
2021-03-05 06:51:44 INFO     rasa.nlu.components  - Added 'SpacyNLP' to component cache. Key 'SpacyNLP-zh_core_web_md'.,
2021-03-05 06:51:44 DEBUG    rasa.utils.tensorflow.models  - Loading the model ...,
2021-03-05 06:51:44 DEBUG    rasa.nlu.classifiers.diet_classifier  - Following metrics will be logged during training: ,
2021-03-05 06:51:44 DEBUG    rasa.nlu.classifiers.diet_classifier  -   t_loss (total loss),
2021-03-05 06:51:44 DEBUG    rasa.nlu.classifiers.diet_classifier  -   i_acc (intent acc),
2021-03-05 06:51:44 DEBUG    rasa.nlu.classifiers.diet_classifier  -   i_loss (intent loss),
2021-03-05 06:51:44 DEBUG    rasa.nlu.classifiers.diet_classifier  -   e_f1 (entity f1),
2021-03-05 06:51:44 DEBUG    rasa.nlu.classifiers.diet_classifier  -   e_loss (entity loss),
2021-03-05 06:51:45 DEBUG    rasa.utils.tensorflow.models  - Finished loading the model.,
2021-03-05 06:51:45 DEBUG    rasa.utils.tensorflow.models  - Building tensorflow prediction graph...,
2021-03-05 06:51:50 DEBUG    rasa.utils.tensorflow.models  - Finished building tensorflow prediction graph.,
2021-03-05 06:51:50 DEBUG    rasa.nlu.classifiers.diet_classifier  - Failed to load model for 'ResponseSelector'. Maybe you did not provide enough training data and no model was trained or the path '/tmp/tmpqtxyh7l_/nlu' doesn't exist?,
2021-03-05 06:51:50 DEBUG    rasa.utils.tensorflow.models  - Loading the model ...,
2021-03-05 06:51:51 DEBUG    rasa.utils.tensorflow.models  - Finished loading the model.,
2021-03-05 06:51:51 DEBUG    rasa.utils.tensorflow.models  - Building tensorflow prediction graph...,
2021-03-05 06:51:52 DEBUG    rasa.utils.tensorflow.models  - Finished building tensorflow prediction graph.,
2021-03-05 06:51:52 DEBUG    rasa.core.agent  - Finished updating agent to new model.,
2021-03-05 06:51:59 DEBUG    rasa.core.agent  - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:51:59 DEBUG    rasa.core.agent  - Model server returned 204 status code, indicating that no new model is available. Current fingerprint: 51973938fc5e9fc2bffd93e5efe71652,
2021-03-05 06:51:59 DEBUG    rasa.core.agent  - No new model found at URL http://rasa-x:5002/api/projects/default/models/tags/production,
2021-03-05 06:52:09 DEBUG    rasa.core.agent  - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:52:09 DEBUG    rasa.core.agent  - Model server returned 204 status code, indicating that no new model is available. Current fingerprint: 51973938fc5e9fc2bffd93e5efe71652,
2021-03-05 06:52:09 DEBUG    rasa.core.agent  - No new model found at URL http://rasa-x:5002/api/projects/default/models/tags/production,
2021-03-05 06:52:19 DEBUG    rasa.core.agent  - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:52:19 DEBUG    rasa.core.agent  - Model server returned 204 status code, indicating that no new model is available. Current fingerprint: 51973938fc5e9fc2bffd93e5efe71652,
2021-03-05 06:52:19 DEBUG    rasa.core.agent  - No new model found at URL http://rasa-x:5002/api/projects/default/models/tags/production,
2021-03-05 06:52:29 DEBUG    rasa.core.agent  - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:52:29 DEBUG    rasa.core.agent  - Model server returned 204 status code, indicating that no new model is available. Current fingerprint: 51973938fc5e9fc2bffd93e5efe71652,
2021-03-05 06:52:29 DEBUG    rasa.core.agent  - No new model found at URL http://rasa-x:5002/api/projects/default/models/tags/production,

The error logs of rasa-worker:

2021-03-05 06:55:09 DEBUG    rasa.core.agent  - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:55:09 DEBUG    rasa.core.agent  - Model server returned 204 status code, indicating that no new model is available. Current fingerprint: 51973938fc5e9fc2bffd93e5efe71652,
2021-03-05 06:55:09 DEBUG    rasa.core.agent  - No new model found at URL http://rasa-x:5002/api/projects/default/models/tags/production,
2021-03-05 06:55:19 DEBUG    rasa.core.agent  - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:55:19 DEBUG    rasa.core.agent  - Model server returned 204 status code, indicating that no new model is available. Current fingerprint: 51973938fc5e9fc2bffd93e5efe71652,
2021-03-05 06:55:19 DEBUG    rasa.core.agent  - No new model found at URL http://rasa-x:5002/api/projects/default/models/tags/production,
2021-03-05 06:55:29 DEBUG    rasa.core.agent  - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:55:29 DEBUG    rasa.core.agent  - Model server returned 204 status code, indicating that no new model is available. Current fingerprint: 51973938fc5e9fc2bffd93e5efe71652,
2021-03-05 06:55:29 DEBUG    rasa.core.agent  - No new model found at URL http://rasa-x:5002/api/projects/default/models/tags/production,
2021-03-05 06:55:38 DEBUG    rasa.server  - Extracting YAML training data from request body.,
2021-03-05 06:55:38 DEBUG    rasa.shared.nlu.training_data.loading  - Training data format of '/tmp/tmpxmegthuk/data.yml' is 'rasa_yml'.,
2021-03-05 06:55:38 DEBUG    rasa.shared.importers.autoconfig  - The provided configuration does not contain the key(s) policies and pipeline. Values will be provided from the default configuration.,
2021-03-05 06:55:38 DEBUG    rasa.shared.nlu.training_data.loading  - Training data format of '/tmp/tmpxmegthuk/data.yml' is 'rasa_yml'.,
2021-03-05 06:55:39 DEBUG    rasa.shared.importers.importer  - Added 14 training data examples from the story training data.,
2021-03-05 06:55:39 DEBUG    rasa.shared.nlu.training_data.loading  - Training data format of '/tmp/tmpxmegthuk/data.yml' is 'rasa_yml'.,
2021-03-05 06:55:39 DEBUG    rasa.core.agent  - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:55:39 DEBUG    rasa.core.agent  - Model server returned 204 status code, indicating that no new model is available. Current fingerprint: 51973938fc5e9fc2bffd93e5efe71652,
2021-03-05 06:55:39 DEBUG    rasa.core.agent  - No new model found at URL http://rasa-x:5002/api/projects/default/models/tags/production,
2021-03-05 06:55:40 DEBUG    rasa.model  - Extracted model to '/tmp/tmpz2gamxfm'.,
2021-03-05 06:55:40 ERROR    rasa.server  - Traceback (most recent call last):,
  File "/opt/venv/lib/python3.7/site-packages/rasa/server.py", line 860, in train,
    None, functools.partial(train_model, **training_payload),
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run,
    result = self.fn(*self.args, **self.kwargs),
  File "/opt/venv/lib/python3.7/site-packages/rasa/train.py", line 55, in train,,
,,
  File "/opt/venv/lib/python3.7/site-packages/rasa/utils/common.py", line 308, in run_in_loop,
    result = loop.run_until_complete(f),
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete,
  File "/opt/venv/lib/python3.7/site-packages/rasa/train.py", line 110, in train_async,
    nlu_additional_arguments=nlu_additional_arguments,,
  File "/opt/venv/lib/python3.7/site-packages/rasa/train.py", line 191, in _train_async_internal,
    new_fingerprint, old_model, train_path,
  File "/opt/venv/lib/python3.7/site-packages/rasa/model.py", line 440, in should_retrain,
    old_core, old_nlu = get_model_subdirectories(unpacked),
  File "/opt/venv/lib/python3.7/site-packages/rasa/model.py", line 239, in get_model_subdirectories,
    unpacked_model_path,
rasa.exceptions.ModelNotFound: No NLU or Core data for unpacked model at: '/tmp/tmpz2gamxfm'.,
,
2021-03-05 06:55:40 ERROR    rasa.server  - An unexpected error occurred during training. Error: No NLU or Core data for unpacked model at: '/tmp/tmpz2gamxfm'.,
2021-03-05 06:55:49 DEBUG    rasa.core.agent  - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:55:49 DEBUG    rasa.core.agent  - Model server returned 204 status code, indicating that no new model is available. Current fingerprint: 51973938fc5e9fc2bffd93e5efe71652,
2021-03-05 06:55:49 DEBUG    rasa.core.agent  - No new model found at URL http://rasa-x:5002/api/projects/default/models/tags/production,