NLU model training finishes immediately, but model returns 'null' for any input [RASA 3.6.20]

Rasa Open Source
1

Hi, thanks for checking this post.

I am trying to upgrade from RASA 2 to RASA 3, and am having a problem I don’t understand. I’m trying to train an NLU model, and when I run the ‘rasa train’ command I get terminal output indicating that my model has trained successfully–but the epoch bar never displays, and training seems to complete immediately. When I open the model using ‘rasa shell’, it returns null with 0 confidence for any input.

I’m running RASA version 3.6.20 with spaCy version 3.7.6 in a python 3.9 virtual environment. Here’s my config file:

version: “3.1” recipe: “default.v1” language: en pipeline:

  • name: SpacyNLP model: en_core_web_sm
  • name: SpacyTokenizer
  • name: SpacyFeaturizer
  • name: SpacyEntityExtractor
  • name: RegexFeaturizer analyzer: char_wb min_ngram: 1 max_ngram: 4
  • name: DIETClassifier epochs: 100 constrain_similarities: true
  • name: EntitySynonymMapper policies:
  • name: MemoizationPolicy assistant_id: 20240813-193423-felt-modal

and here is the last part of the terminal output from training:

2024-08-29 17:45:41 INFO rasa.nlu.utils.spacy_utils - Trying to load SpaCy model with name ‘en_core_web_sm’. 2024-08-29 17:45:42 INFO rasa.nlu.utils.spacy_utils - Trying to load SpaCy model with name ‘en_core_web_sm’. 2024-08-29 17:45:43 INFO rasa.engine.training.hooks - Starting to train component ‘RegexFeaturizer’. 2024-08-29 17:45:43 INFO rasa.engine.training.hooks - Finished training component ‘RegexFeaturizer’. 2024-08-29 17:45:43 INFO rasa.engine.training.hooks - Starting to train component ‘DIETClassifier’. 2024-08-29 17:45:43 INFO rasa.engine.training.hooks - Finished training component ‘DIETClassifier’. 2024-08-29 17:45:43 INFO rasa.engine.training.hooks - Starting to train component ‘EntitySynonymMapper’. 2024-08-29 17:45:43 INFO rasa.engine.training.hooks - Finished training component ‘EntitySynonymMapper’. Your Rasa model is trained and saved at ‘models/nlu-20240829-174541-heartless-bayou.tar.gz’. Rasa model training completed successfully.

Here’s the terminal output from running ‘rasa shell’ with a sample input:

2024-08-29 18:05:26 INFO rasa.core.processor - Loading model models/nlu-20240829-174541-heartless-bayou.tar.gz… 2024-08-29 18:05:26 INFO rasa.nlu.utils.spacy_utils - Trying to load SpaCy model with name ‘en_core_web_sm’. 2024-08-29 18:05:28 INFO rasa.nlu.utils.spacy_utils - Trying to load SpaCy model with name ‘en_core_web_sm’. NLU model loaded. Type a message and press enter to parse it. Next message: Hello, this is a test message. { “text”: “Hello, this is a test message.”, “intent”: { “name”: null, “confidence”: 0.0 }, “entities”: , “text_tokens”: [ [ 0, 5 ], [ 5, 6 ], [ 7, 11 ], [ 12, 14 ], [ 15, 16 ], [ 17, 21 ], [ 22, 29 ], [ 29, 30 ] ], “intent_ranking”: } Next message:

Any ideas what could be causing this issue? Thanks again.

2

Adding some more information about the size of the models created. Some of the RASA 3 models initially would train and display the epoch bar, but these models would take 3+ hours to train for 100 epochs and then only return very low (<.08) confidences for any input except for an “affirmative” recognizing intent that would score very confidently, .95+, but these models would display meaningful confidences in intents that were part of their domain file. More recently trained models “complete training successfully” as noted above–no epoch bar, <30 seconds to train model, “null” only intent response in rasa shell with 0 confidence. These models are also much smaller than the ones that took 3 hours to train in RASA 3.

Just for comparison, I dusted off my RASA 2.1.1 pipeline, which still works, trained quickly (~30 minutes for 100 epochs), showed the epoch bar, and produces the kind of results you’d expect in rasa shell, high confidences and so on. These RASA 2 models are also about twice the size of the suspicious-but-completed-training RASA 3 models, and many times larger than the RASA 3 models that don’t really seem to train but say they do. Data below, first number is filesize.

RASA 3.6

-models that “trained successfully” in <30s without epoch bar display and writes only null confidence in rasa shell:

  • 2456 Aug 30 14:20 nlu-20240830-142056-poky-turret.tar.gz

  • 2453 Aug 29 17:45 nlu-20240829-174541-heartless-bayou.tar.gz

  • 2466 Aug 29 17:45 nlu-20240829-174510-scalding-easement.tar.gz

  • 2469 Aug 29 17:42 nlu-20240829-174158-product-vinegar.tar.gz

  • 2461 Aug 29 17:41 nlu-20240829-174120-modern-combustion.tar.gz

  • 2480 Aug 29 17:38 nlu-20240829-173834-humble-bingo.tar.gz

  • 2481 Aug 29 17:37 nlu-20240829-173756-forward-loan.tar.gz

  • 2472 Aug 28 20:57 nlu-20240828-205711-numerous-theme.tar.gz

  • 2455 Aug 28 20:53 nlu-20240828-205355-stale-bell.tar.gz

  • 2452 Aug 28 14:56 nlu-20240828-145611-inflammable-panini.tar.gz

  • 2459 Aug 28 14:13 nlu-20240828-141305-flat-groove.tar.gz

  • 1729 Aug 28 14:04 nlu-20240828-140442-basic-force.tar.gz

  • 1723 Aug 28 14:00 nlu-20240828-140057-dry-satchel.tar.gz

  • 1729 Aug 27 21:50 nlu-20240827-215016-staff-deadline.tar.gz

  • 1710 Aug 27 20:25 nlu-20240827-202558-absolute-burger.tar.gz

  • 1722 Aug 23 19:16 nlu-20240823-191617-sluggish-canoe.tar.gz

  • 1718 Aug 20 19:10 nlu-20240820-191053-perfect-head.tar.gz

  • 1714 Aug 19 18:55 nlu-20240819-185552-global-spring.tar.gz

  • 1708 Aug 19 18:43 nlu-20240819-184320-ebony-subspace.tar.gz

  • 1707 Aug 19 18:42 nlu-20240819-184209-timid-loon.tar.gz

  • 1706 Aug 19 18:28 nlu-20240819-182807-vain-corona.tar.gz

  • 1711 Aug 14 14:31 nlu-20240814-143102-chromatic-club.tar.gz

  • 1708 Aug 14 14:29 nlu-20240814-142909-flat-prefix.tar.gz

  • 1716 Aug 13 20:07 nlu-20240813-200742-tall-sprite.tar.gz

  • 1710 Aug 13 20:03 nlu-20240813-200355-concave-tree.tar.gz

  • 1723 Aug 13 19:57 nlu-20240813-195703-brilliant-mountain.tar.gz

  • 8674 Aug  6 21:32 nlu-20240806-213223-kind-audience.tar.gz

  • 8696 Aug  6 21:28 nlu-20240806-212831-crunchy-reveal.tar.gz

-models that trained (3+ hours) with epoch bar display and write meaningful (but low, <.07) confidences for intents in domain in rasa shell:

  • 19678231 Aug 1 00:10 nlu-20240731-210526-booming-reflection.tar.gz
  • 19842547 Jul 31 21:01 nlu-20240731-202405-resolving-provolone.tar.gz
  • 19923109 Aug 13 16:08 nlu-20240730-180020-hot-event.tar.gz

RASA 2.1.1

-models trained 100 epochs ~25 mins, write meaningful (and high) confidences for intents in domain:

  • 37432244 Jun 27 21:33 nlu-20240627-client-v2-beta.tar.gz
  • 37456829 Sep 3 18:24 nlu-20240903-182449.tar.gz

Would appreciate any suggestions about what the issue could be. Thanks

3

Since the vm I am working on does not have a GPU allocated to it, I thought that downloading the CPU-only version of tensorflow might solve my problem. But after uninstalling the default version, installing tensorflow-cpu, and linking the two so that rasa will recognize the latter as the former, the behavior when training is the same: it still trains immediately and produces a junk model.

4

I have downgraded to RASA 3.2.13 in a fresh python 3.9.x virtual environment hoping that might address the issue, but I am still seeing the same issue with the model training immediately and not displaying the epoch bar.