I have successfully installed the rasa-x in the Docker-compose mode, trained the demo model from rasa-worker, and launched the chatbot with rasa-production and rasa-x.
Then, I modified the docker image to support Chinese language using spacy. Specifically, it installed a Chinese dictionary, zh_core_web_md. After updating the docker-compose.yml, the rasa-x is launched and the chatbot works with both English and Chinese model. I can talk to and share the bot via rasa-x, which means the rasa-production works well too. However, neither the English model nor the Chinese model can be trained via rasa-x.
After going through the docker logs, seem the rasa-worker failed to extract the model which leads to an error. However, when we initialized the rasa-x, the rasa-worker was successfully initialized and the model was extracted without error. Why it failed to extract the model for training?
The Dockerfile we used to edit docker image:
FROM rasa/rasa:2.1.2-full
# Change back to root user to install dependencies
USER root
COPY ./zh_core_web_md-2.3.1.tar.gz /app/
RUN pip install zh_core_web_md-2.3.1.tar.gz -i
# By best practices, don't run the code with root user
USER 1001
The log of rasa-worker which has been successfully initialized.
2021-03-05 06:51:12 DEBUG rasa.telemetry - Could not read telemetry settings from configuration file: Configuration 'metrics' key not found.,
2021-03-05 06:51:13 DEBUG sanic_jwt.configuration - validating provided secret,
2021-03-05 06:51:13 DEBUG sanic_jwt.configuration - validating keys (if needed),
2021-03-05 06:51:13 DEBUG sanic_jwt.configuration - loading secret and/or keys (if needed),
2021-03-05 06:51:13 DEBUG rasa.core.utils - Available web server routes: ,
/conversations/<conversation_id:path>/messages POST add_message,
/conversations/<conversation_id:path>/tracker/events POST append_events,
/auth POST auth_bp.AuthenticateEndpoint,
/auth/me GET auth_bp.RetrieveUserEndpoint,
/auth/verify GET auth_bp.VerifyEndpoint,
/webhooks/rasa GET custom_webhook_RasaChatInput.health,
/webhooks/rasa/webhook POST custom_webhook_RasaChatInput.receive,
/webhooks/rest GET custom_webhook_RestInput.health,
/webhooks/rest/webhook POST custom_webhook_RestInput.receive,
/model/test/intents POST evaluate_intents,
/model/test/stories POST evaluate_stories,
/conversations/<conversation_id:path>/execute POST execute_action,
/domain GET get_domain,
/ GET hello,
/model PUT load_model,
/model/parse POST parse,
/conversations/<conversation_id:path>/predict POST predict,
/conversations/<conversation_id:path>/tracker/events PUT replace_events,
/conversations/<conversation_id:path>/story GET retrieve_story,
/conversations/<conversation_id:path>/tracker GET retrieve_tracker,
/status GET status,
/model/predict POST tracker_predict,
/model/train POST train,
/conversations/<conversation_id:path>/trigger_intent POST trigger_intent,
/model DELETE unload_model,
/version GET version,
2021-03-05 06:51:13 INFO root - Starting Rasa server on http://localhost:5005,
2021-03-05 06:51:13 DEBUG rasa.core.utils - Using the default number of Sanic workers (1).,
2021-03-05 06:51:14 INFO root - Enabling coroutine debugging. Loop id 94682458493984.,
2021-03-05 06:51:14 DEBUG root - Could not load interpreter from 'None'.,
2021-03-05 06:51:14 INFO rasa.core.brokers.pika - Connecting to RabbitMQ ...,
2021-03-05 06:51:14 DEBUG rasa.core.brokers.pika - Connecting to 'rabbit' failed with error 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'. Trying again.,
2021-03-05 06:51:19 DEBUG rasa.core.brokers.pika - Connecting to 'rabbit' failed with error '[Errno 111] Connection refused'. Trying again.,
2021-03-05 06:51:24 DEBUG rasa.core.brokers.pika - Connecting to 'rabbit' failed with error '[Errno 111] Connection refused'. Trying again.,
2021-03-05 06:51:29 INFO rasa.core.brokers.pika - RabbitMQ connection to 'rabbit' was established.,
2021-03-05 06:51:29 DEBUG rasa.core.brokers.pika - RabbitMQ channel was opened. Declaring fanout exchange.,
2021-03-05 06:51:29 DEBUG rasa.core.brokers.broker - Instantiated event broker to 'PikaEventBroker'.,
2021-03-05 06:51:29 DEBUG rasa.core.tracker_store - Attempting to connect to database via 'postgresql://admin:***@db:5432/rasa'.,
2021-03-05 06:51:29 DEBUG rasa.core.tracker_store - Connection to SQL database 'worker_tracker' successful.,
2021-03-05 06:51:29 DEBUG rasa.core.tracker_store - Connected to SQLTrackerStore.,
2021-03-05 06:51:29 DEBUG rasa.core.lock_store - Connected to lock store 'RedisLockStore'.,
2021-03-05 06:51:29 DEBUG rasa.core.nlg.generator - Instantiated NLG to 'TemplatedNaturalLanguageGenerator'.,
2021-03-05 06:51:29 DEBUG rasa.core.agent - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:51:29 DEBUG rasa.core.agent - Model server could not find a model at the requested endpoint 'http://rasa-x:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.,
2021-03-05 06:51:29 DEBUG rasa.core.agent - No new model found at URL http://rasa-x:5002/api/projects/default/models/tags/production,
2021-03-05 06:51:29 INFO root - Rasa server is up and running.,
2021-03-05 06:51:39 DEBUG rasa.core.agent - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:51:40 DEBUG rasa.core.agent - Unzipped model to '/tmp/tmpqtxyh7l_',
2021-03-05 06:51:40 DEBUG rasa.core.agent - Found new model with fingerprint 51973938fc5e9fc2bffd93e5efe71652. Loading...,
/opt/venv/lib/python3.7/site-packages/thinc/neural/_custom_kernels.py:36: ResourceWarning: unclosed file <_io.TextIOWrapper name='/opt/venv/lib/python3.7/site-packages/thinc/neural/_custom_kernels.cu' mode='r' encoding='utf8'>,
SRC = (PWD / "_custom_kernels.cu").open("r", encoding="utf8").read(),
ResourceWarning: Enable tracemalloc to get the object allocation traceback,
/opt/venv/lib/python3.7/site-packages/thinc/neural/_custom_kernels.py:39: ResourceWarning: unclosed file <_io.TextIOWrapper name='/opt/venv/lib/python3.7/site-packages/thinc/neural/_murmur3.cu' mode='r' encoding='utf8'>,
MMH_SRC = (PWD / "_murmur3.cu").open("r", encoding="utf8").read(),
ResourceWarning: Enable tracemalloc to get the object allocation traceback,
Building prefix dict from the default dictionary ...,
2021-03-05 06:51:41 DEBUG jieba - Building prefix dict from the default dictionary ...,
Dumping model to file cache /tmp/jieba.cache,
2021-03-05 06:51:42 DEBUG jieba - Dumping model to file cache /tmp/jieba.cache,
Loading model cost 0.854 seconds.,
2021-03-05 06:51:42 DEBUG jieba - Loading model cost 0.854 seconds.,
Prefix dict has been built successfully.,
2021-03-05 06:51:42 DEBUG jieba - Prefix dict has been built successfully.,
2021-03-05 06:51:44 INFO rasa.nlu.components - Added 'SpacyNLP' to component cache. Key 'SpacyNLP-zh_core_web_md'.,
2021-03-05 06:51:44 DEBUG rasa.utils.tensorflow.models - Loading the model ...,
2021-03-05 06:51:44 DEBUG rasa.nlu.classifiers.diet_classifier - Following metrics will be logged during training: ,
2021-03-05 06:51:44 DEBUG rasa.nlu.classifiers.diet_classifier - t_loss (total loss),
2021-03-05 06:51:44 DEBUG rasa.nlu.classifiers.diet_classifier - i_acc (intent acc),
2021-03-05 06:51:44 DEBUG rasa.nlu.classifiers.diet_classifier - i_loss (intent loss),
2021-03-05 06:51:44 DEBUG rasa.nlu.classifiers.diet_classifier - e_f1 (entity f1),
2021-03-05 06:51:44 DEBUG rasa.nlu.classifiers.diet_classifier - e_loss (entity loss),
2021-03-05 06:51:45 DEBUG rasa.utils.tensorflow.models - Finished loading the model.,
2021-03-05 06:51:45 DEBUG rasa.utils.tensorflow.models - Building tensorflow prediction graph...,
2021-03-05 06:51:50 DEBUG rasa.utils.tensorflow.models - Finished building tensorflow prediction graph.,
2021-03-05 06:51:50 DEBUG rasa.nlu.classifiers.diet_classifier - Failed to load model for 'ResponseSelector'. Maybe you did not provide enough training data and no model was trained or the path '/tmp/tmpqtxyh7l_/nlu' doesn't exist?,
2021-03-05 06:51:50 DEBUG rasa.utils.tensorflow.models - Loading the model ...,
2021-03-05 06:51:51 DEBUG rasa.utils.tensorflow.models - Finished loading the model.,
2021-03-05 06:51:51 DEBUG rasa.utils.tensorflow.models - Building tensorflow prediction graph...,
2021-03-05 06:51:52 DEBUG rasa.utils.tensorflow.models - Finished building tensorflow prediction graph.,
2021-03-05 06:51:52 DEBUG rasa.core.agent - Finished updating agent to new model.,
2021-03-05 06:51:59 DEBUG rasa.core.agent - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:51:59 DEBUG rasa.core.agent - Model server returned 204 status code, indicating that no new model is available. Current fingerprint: 51973938fc5e9fc2bffd93e5efe71652,
2021-03-05 06:51:59 DEBUG rasa.core.agent - No new model found at URL http://rasa-x:5002/api/projects/default/models/tags/production,
2021-03-05 06:52:09 DEBUG rasa.core.agent - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:52:09 DEBUG rasa.core.agent - Model server returned 204 status code, indicating that no new model is available. Current fingerprint: 51973938fc5e9fc2bffd93e5efe71652,
2021-03-05 06:52:09 DEBUG rasa.core.agent - No new model found at URL http://rasa-x:5002/api/projects/default/models/tags/production,
2021-03-05 06:52:19 DEBUG rasa.core.agent - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:52:19 DEBUG rasa.core.agent - Model server returned 204 status code, indicating that no new model is available. Current fingerprint: 51973938fc5e9fc2bffd93e5efe71652,
2021-03-05 06:52:19 DEBUG rasa.core.agent - No new model found at URL http://rasa-x:5002/api/projects/default/models/tags/production,
2021-03-05 06:52:29 DEBUG rasa.core.agent - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:52:29 DEBUG rasa.core.agent - Model server returned 204 status code, indicating that no new model is available. Current fingerprint: 51973938fc5e9fc2bffd93e5efe71652,
2021-03-05 06:52:29 DEBUG rasa.core.agent - No new model found at URL http://rasa-x:5002/api/projects/default/models/tags/production,
The error logs of rasa-worker:
2021-03-05 06:55:09 DEBUG rasa.core.agent - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:55:09 DEBUG rasa.core.agent - Model server returned 204 status code, indicating that no new model is available. Current fingerprint: 51973938fc5e9fc2bffd93e5efe71652,
2021-03-05 06:55:09 DEBUG rasa.core.agent - No new model found at URL http://rasa-x:5002/api/projects/default/models/tags/production,
2021-03-05 06:55:19 DEBUG rasa.core.agent - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:55:19 DEBUG rasa.core.agent - Model server returned 204 status code, indicating that no new model is available. Current fingerprint: 51973938fc5e9fc2bffd93e5efe71652,
2021-03-05 06:55:19 DEBUG rasa.core.agent - No new model found at URL http://rasa-x:5002/api/projects/default/models/tags/production,
2021-03-05 06:55:29 DEBUG rasa.core.agent - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:55:29 DEBUG rasa.core.agent - Model server returned 204 status code, indicating that no new model is available. Current fingerprint: 51973938fc5e9fc2bffd93e5efe71652,
2021-03-05 06:55:29 DEBUG rasa.core.agent - No new model found at URL http://rasa-x:5002/api/projects/default/models/tags/production,
2021-03-05 06:55:38 DEBUG rasa.server - Extracting YAML training data from request body.,
2021-03-05 06:55:38 DEBUG rasa.shared.nlu.training_data.loading - Training data format of '/tmp/tmpxmegthuk/data.yml' is 'rasa_yml'.,
2021-03-05 06:55:38 DEBUG rasa.shared.importers.autoconfig - The provided configuration does not contain the key(s) policies and pipeline. Values will be provided from the default configuration.,
2021-03-05 06:55:38 DEBUG rasa.shared.nlu.training_data.loading - Training data format of '/tmp/tmpxmegthuk/data.yml' is 'rasa_yml'.,
2021-03-05 06:55:39 DEBUG rasa.shared.importers.importer - Added 14 training data examples from the story training data.,
2021-03-05 06:55:39 DEBUG rasa.shared.nlu.training_data.loading - Training data format of '/tmp/tmpxmegthuk/data.yml' is 'rasa_yml'.,
2021-03-05 06:55:39 DEBUG rasa.core.agent - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:55:39 DEBUG rasa.core.agent - Model server returned 204 status code, indicating that no new model is available. Current fingerprint: 51973938fc5e9fc2bffd93e5efe71652,
2021-03-05 06:55:39 DEBUG rasa.core.agent - No new model found at URL http://rasa-x:5002/api/projects/default/models/tags/production,
2021-03-05 06:55:40 DEBUG rasa.model - Extracted model to '/tmp/tmpz2gamxfm'.,
2021-03-05 06:55:40 ERROR rasa.server - Traceback (most recent call last):,
File "/opt/venv/lib/python3.7/site-packages/rasa/server.py", line 860, in train,
None, functools.partial(train_model, **training_payload),
File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run,
result = self.fn(*self.args, **self.kwargs),
File "/opt/venv/lib/python3.7/site-packages/rasa/train.py", line 55, in train,,
,,
File "/opt/venv/lib/python3.7/site-packages/rasa/utils/common.py", line 308, in run_in_loop,
result = loop.run_until_complete(f),
File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete,
File "/opt/venv/lib/python3.7/site-packages/rasa/train.py", line 110, in train_async,
nlu_additional_arguments=nlu_additional_arguments,,
File "/opt/venv/lib/python3.7/site-packages/rasa/train.py", line 191, in _train_async_internal,
new_fingerprint, old_model, train_path,
File "/opt/venv/lib/python3.7/site-packages/rasa/model.py", line 440, in should_retrain,
old_core, old_nlu = get_model_subdirectories(unpacked),
File "/opt/venv/lib/python3.7/site-packages/rasa/model.py", line 239, in get_model_subdirectories,
unpacked_model_path,
rasa.exceptions.ModelNotFound: No NLU or Core data for unpacked model at: '/tmp/tmpz2gamxfm'.,
,
2021-03-05 06:55:40 ERROR rasa.server - An unexpected error occurred during training. Error: No NLU or Core data for unpacked model at: '/tmp/tmpz2gamxfm'.,
2021-03-05 06:55:49 DEBUG rasa.core.agent - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...,
2021-03-05 06:55:49 DEBUG rasa.core.agent - Model server returned 204 status code, indicating that no new model is available. Current fingerprint: 51973938fc5e9fc2bffd93e5efe71652,
2021-03-05 06:55:49 DEBUG rasa.core.agent - No new model found at URL http://rasa-x:5002/api/projects/default/models/tags/production,