Can't train model with spaCy

[Deprecated] Rasa X Community Edition
1

I cannot train my model when using spaCyNLP. My config.yml

pipeline:
  - name: SpacyNLP
  - name: SpacyTokenizer
  - name: SpacyFeaturizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: char_wb
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 150
    random_seed: 1
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 100
language: en
policies:
  - name: TEDPolicy
    max_history: 10
    epochs: 50
    batch_size:
      - 32
      - 64
      - max_history: 5
  - name: FallbackPolicy
    nlu_threshold: 0.6
    core_threshold: 0.6
    fallback_action_name: utter_fallback

rasa-x logs:

rasa-x_1           | INFO:alembic.runtime.migration:Context impl PostgresqlImpl.
rasa-x_1           | INFO:alembic.runtime.migration:Will assume transactional DDL.
rasa-x_1           | INFO:rasax.community.services.event_service:Starting event service (standalone: False).
rasa-x_1           | Starting Rasa X server... 🚀
rasa-x_1           | /usr/local/lib/python3.7/site-packages/rasax/community/services/test_service.py:43: UserWarning: Unable to get tests from tests/conversation_tests.md:
rasa-x_1           | File 'tests/conversation_tests.md' does not exist. Please, make sure you have end-to-end tests added to your assistant. See https://rasa.com/docs/rasa-x/user-guide/test-assistant for more information.
rasa-x_1           |   UserWarning,
rasa-x_1           | INFO:rasax.community.services.event_consumers.pika_consumer:Start consuming queue 'rasa_production_events' on pika host 'rabbit'.
rasa-x_1           | ERROR:pika.adapters.blocking_connection:Unexpected connection close detected: ConnectionClosedByBroker: (320) "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
rasa-x_1           | ERROR:rasax.community.services.event_service:Caught an exception while consuming events. Will retry in 5 s.
rasa-x_1           | Traceback (most recent call last):
rasa-x_1           |   File "/usr/local/lib/python3.7/site-packages/rasax/community/services/event_service.py", line 1667, in continuously_consume
rasa-x_1           |     consumer.consume()
rasa-x_1           |   File "/usr/local/lib/python3.7/site-packages/rasax/community/services/event_consumers/pika_consumer.py", line 180, in consume
rasa-x_1           |     self.channel.start_consuming()
rasa-x_1           |   File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 1866, in start_consuming
rasa-x_1           |     self._process_data_events(time_limit=None)
rasa-x_1           |   File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 2027, in _process_data_events
rasa-x_1           |     self.connection.process_data_events(time_limit=time_limit)
rasa-x_1           |   File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 825, in process_data_events
rasa-x_1           |     self._flush_output(common_terminator)
rasa-x_1           |   File "/usr/local/lib/python3.7/site-packages/pika/adapters/blocking_connection.py", line 522, in _flush_output
rasa-x_1           |     raise self._closed_result.value.error
rasa-x_1           | pika.exceptions.ConnectionClosedByBroker: (320, "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'")
rasa-x_1           | INFO:rasax.community.services.event_consumers.pika_consumer:Start consuming queue 'rasa_production_events' on pika host 'rabbit'.
Processed Story Blocks: 100%|██████████| 1/1 [00:00<00:00, 1354.31it/s, # trackers=1]
rasa-x_1           | /usr/local/lib/python3.7/site-packages/rasa/core/training/dsl.py:431: UserWarning: Found unknown intent 'chat.bot_correct' on line 3. Please, make sure that all intents are listed in your domain yaml.
rasa-x_1           |   docs=DOCS_URL_DOMAINS,
Processed Story Blocks: 100%|██████████| 1/1 [00:00<00:00, 1363.56it/s, # trackers=1]
Processed Story Blocks: 100%|██████████| 1/1 [00:00<00:00, 1790.14it/s, # trackers=1]
Processed Story Blocks: 100%|██████████| 1/1 [00:00<00:00, 1818.08it/s, # trackers=1]
Processed Story Blocks: 0it [00:00, ?it/s]
 Processed Story Blocks: 100%|██████████| 1/1 [00:00<00:00, 1930.19it/s, # trackers=1]
Processed Story Blocks: 100%|██████████| 1/1 [00:00<00:00, 1642.25it/s, # trackers=1]
rasa-x_1           | /usr/local/lib/python3.7/site-packages/rasa/utils/common.py:363: UserWarning: There is no starting story block in the training data. All your story blocks start with some checkpoint. There should be at least one story block that starts without any checkpoint.
rasa-x_1           |   More info at https://rasa.com/docs/rasa/core/stories/#stories
rasa-x_1           | ERROR:rasax.community.api.blueprints.models:500, message='Internal Server Error', url=URL('http://rasa-worker:5005/model/train?token=KpLtuEKG3wuisXm')
rasa-x_1           | Encountered an exception while training. Please check the logs of the rasa worker container for more information.

rasa-worker logs:

rasa-worker_1      | /opt/venv/lib/python3.7/site-packages/rasa/core/brokers/pika.py:294: FutureWarning: Your Pika event broker config contains the deprecated `queue` key. Please use the `queues` key instead.
rasa-worker_1      |   self.queues = self._get_queues_from_args(queues, kwargs)
rasa-worker_1      | 2020-12-07 10:42:18 ERROR    pika.adapters.utils.io_services_utils  - Socket failed to connect: <socket.socket fd=22, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.21.0.8', 38874)>; error=111 (Connection refused)
rasa-worker_1      | 2020-12-07 10:42:18 ERROR    pika.adapters.utils.connection_workflow  - TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.21.0.3', 5672))
rasa-worker_1      | 2020-12-07 10:42:18 ERROR    pika.adapters.utils.connection_workflow  - AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rasa-worker_1      | 2020-12-07 10:42:23 ERROR    pika.adapters.utils.io_services_utils  - Socket failed to connect: <socket.socket fd=25, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.21.0.8', 38894)>; error=111 (Connection refused)
rasa-worker_1      | 2020-12-07 10:42:23 ERROR    pika.adapters.utils.connection_workflow  - TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.21.0.3', 5672))
rasa-worker_1      | 2020-12-07 10:42:23 ERROR    pika.adapters.utils.connection_workflow  - AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rasa-worker_1      | 2020-12-07 10:42:51.960288: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)

Another weird thing, after training failed, I switch back to the old working config but it is also failed. The old config.yml:

pipeline:
  - name: WhitespaceTokenizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: char_wb
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 100
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 100
language: en
policies:
  - name: TEDPolicy
    max_history: 10
    epochs: 50
    batch_size:
      - 32
      - 64
      - max_history: 5
  - name: FallbackPolicy
    nlu_threshold: 0.6
    core_threshold: 0.6
    fallback_action_name: utter_fallback

This is frustrating. Anyone has any idea?

2

I solved the problem. One of my story has no corresponding intent in nlu.md so training could not be succeeded. It took a few sudo docker-compose down and up for the problem to be appeared in the logs, so basically the is no definite indicator for the problem.