Rasa Train Error Function call stack: train_on_batch

trinhminhhieu · October 7, 2021, 3:51am

Hi everyone, I have a problem running “rasa train” , I am using rasa 2.3.1 and I am getting the following error:

Here is my config.yml file

# https://rasa.com/docs/rasa/nlu/components/

language: en 

pipeline:

  - name: WhitespaceTokenizer

  - name: RegexFeaturizer

  - name: LexicalSyntacticFeaturizer

  - name: CountVectorsFeaturizer

  - name: CountVectorsFeaturizer

    analyzer: char_wb

    min_ngram: 1

    max_ngram: 4

  - name: DIETClassifier

    epochs: 100

    constrain_similarities: true

    model_confidence: softmax

  - name: EntitySynonymMapper

  - name: ResponseSelector

    epochs: 100

    retrieval_intent: faq

  - name: ResponseSelector

    epochs: 100

    retrieval_intent: chitchat

  - name: FallbackClassifier

    threshold: 0.3

    ambiguity_threshold: 0.1

  - name: "DucklingEntityExtractor"

    # url of the running duckling server

    url: "http://localhost:8000"

    # dimensions to extract

    dimensions: ["number", "email"]

policies:

  - name: MemoizationPolicy

  - name: TEDPolicy

    max_history: 5

    epochs: 100

    constrain_similarities: true

    model_confidence: softmax

  - name: RulePolicy

    core_fallback_threshold: 0.4

    core_fallback_action_name: "action_default_fallback"

    enable_fallback_prediction: True

is there any solution to this problem? please help! Thanks!

nik202 · October 7, 2021, 6:30am

@trinhminhhieu please share the rasa --version screenshot while activating conda environment ?

trinhminhhieu · October 7, 2021, 6:50am

Hi, Below are the results

nik202 · October 7, 2021, 6:53am

@trinhminhhieu can you install pip install rasa==2.8.1 or 2.8.2 and share rasa --version please?

ChrisRahme · October 7, 2021, 6:55am

You have Rasa 2.3 but Rasa SDK 2.8.

Make sure they’re both the same, either by upgrading rasa as Nik said, or downgrading rasa-sdk by the same method if you need Rasa 2.3.

trinhminhhieu · October 7, 2021, 6:59am

Thanks for the feedback, i just installed rasa==2.8.1 and below is the result

nik202 · October 7, 2021, 7:02am

@trinhminhhieu Nice, now try train the model please and share the output? do run rasa init if you don’t have rasa project else I guess you are good to go.

trinhminhhieu · October 7, 2021, 7:21am

Hi, it still doesn’t solve i still get the error

I try to train model from this repository

nik202 · October 7, 2021, 7:27am

@trinhminhhieu share the complete screenshot please and what command you have typed.

trinhminhhieu · October 7, 2021, 8:00am

(chat) C:\Users\DELL\OneDrive\Desktop\chat-ecom>rasa train
C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\rasa\shared\utils\io.py:97: UserWarning: Action 'utter_chitchat' is listed as a response action in the domain file, but there is no matching response defined. Please check your domain.
  More info at https://rasa.com/docs/rasa/responses
C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\rasa\shared\utils\io.py:97: UserWarning: Action 'utter_faq' is listed as a response action in the domain file, but there is no matching response defined. Please check your domain.
  More info at https://rasa.com/docs/rasa/responses
2021-10-07 14:09:50 INFO     rasa.model  - Data (core-config) for Core model section changed.
2021-10-07 14:09:50 INFO     rasa.model  - Data (nlu-config) for NLU model section changed.
2021-10-07 14:09:50 INFO     rasa.model  - Data (nlg) for NLG responses section changed.
Training NLU model...
C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\rasa\utils\train_utils.py:646: UserWarning: constrain_similarities is set to `False`. It is recommended to set it to `True` when using cross-entropy loss. It will be set to `True` by default, Rasa Open Source 3.0.0 onwards.
  category=UserWarning,
2021-10-07 14:09:55 INFO     rasa.shared.nlu.training_data.training_data  - Training data stats:
2021-10-07 14:09:55 INFO     rasa.shared.nlu.training_data.training_data  - Number of intent examples: 203 (14 distinct intents)

2021-10-07 14:09:55 INFO     rasa.shared.nlu.training_data.training_data  -   Found intents: 'faq', 'order_shoes', 'greet', 'product_updates', 'whats_your_name', 'order_status', 'deny', 'return', 'chitchat', 'product_stock', 'affirm', 'order_cancel', 'out_of_scope', 'inform'
2021-10-07 14:09:55 INFO     rasa.shared.nlu.training_data.training_data  - Number of response examples: 66 (11 distinct responses)
2021-10-07 14:09:55 INFO     rasa.shared.nlu.training_data.training_data  - Number of entity examples: 15 (2 distinct entities)
2021-10-07 14:09:55 INFO     rasa.shared.nlu.training_data.training_data  -   Found entity types: 'color', 'negation'
2021-10-07 14:09:55 INFO     rasa.nlu.model  - Starting to train component WhitespaceTokenizer
2021-10-07 14:09:55 INFO     rasa.nlu.model  - Finished training component.
2021-10-07 14:09:55 INFO     rasa.nlu.model  - Starting to train component RegexFeaturizer
2021-10-07 14:09:55 INFO     rasa.nlu.model  - Finished training component.
2021-10-07 14:09:55 INFO     rasa.nlu.model  - Starting to train component LexicalSyntacticFeaturizer
2021-10-07 14:09:55 INFO     rasa.nlu.model  - Finished training component.
2021-10-07 14:09:55 INFO     rasa.nlu.model  - Starting to train component CountVectorsFeaturizer
2021-10-07 14:09:55 INFO     rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer  - 325 vocabulary items were created for text attribute.
2021-10-07 14:09:55 INFO     rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer  - 95 vocabulary items were created for response attribute.
2021-10-07 14:09:56 INFO     rasa.nlu.model  - Finished training component.
2021-10-07 14:09:56 INFO     rasa.nlu.model  - Starting to train component CountVectorsFeaturizer
2021-10-07 14:09:56 INFO     rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer  - 2328 vocabulary items were created for text attribute.
2021-10-07 14:09:56 INFO     rasa.nlu.featurizers.sparse_featurizer.count_vectors_featurizer  - 792 vocabulary items were created for response attribute.
2021-10-07 14:09:56 INFO     rasa.nlu.model  - Finished training component.
2021-10-07 14:09:56 INFO     rasa.nlu.model  - Starting to train component DIETClassifier
Epochs: 100%|████████████████| 100/100 [02:06<00:00,  1.27s/it, t_loss=1.59, i_acc=1, e_f1=0.941]
2021-10-07 14:12:03 INFO     rasa.nlu.model  - Finished training component.
2021-10-07 14:12:03 INFO     rasa.nlu.model  - Starting to train component EntitySynonymMapper
2021-10-07 14:12:03 INFO     rasa.nlu.model  - Finished training component.
2021-10-07 14:12:03 INFO     rasa.nlu.model  - Starting to train component ResponseSelector
Epochs: 100%|████████████████████████████| 100/100 [00:13<00:00,  7.56it/s, t_loss=80.7, r_acc=1]
2021-10-07 14:12:16 INFO     rasa.nlu.model  - Finished training component.
2021-10-07 14:12:16 INFO     rasa.nlu.model  - Starting to train component ResponseSelector
Epochs: 100%|████████████████████████████| 100/100 [00:15<00:00,  6.38it/s, t_loss=5.64, r_acc=1]
2021-10-07 14:12:32 INFO     rasa.nlu.model  - Finished training component.
2021-10-07 14:12:32 INFO     rasa.nlu.model  - Starting to train component FallbackClassifier
2021-10-07 14:12:32 INFO     rasa.nlu.model  - Finished training component.
2021-10-07 14:12:32 INFO     rasa.nlu.model  - Starting to train component DucklingEntityExtractor
2021-10-07 14:12:32 INFO     rasa.nlu.model  - Finished training component.
2021-10-07 14:12:35 INFO     rasa.nlu.model  - Successfully saved model into 'C:\Users\DELL\AppData\Local\Temp\tmpjrmch2pi\nlu'
NLU model training completed.
Training Core model...
Processed story blocks: 100%|██████████████████████| 8/8 [00:00<00:00, 1601.49it/s, # trackers=1]
Processed story blocks: 100%|███████████████████████| 8/8 [00:00<00:00, 333.31it/s, # trackers=8]
Processed story blocks: 100%|███████████████████████| 8/8 [00:00<00:00, 38.10it/s, # trackers=50]
Processed story blocks: 100%|███████████████████████| 8/8 [00:00<00:00, 31.13it/s, # trackers=50]
Processed rules: 100%|███████████████████████████| 20/20 [00:00<00:00, 1669.01it/s, # trackers=1]
Processed trackers: 100%|████████████████████████████| 8/8 [00:00<00:00, 800.00it/s, # action=32]
Processed actions: 32it [00:00, 80.80it/s, # examples=32]
Processed trackers: 100%|██████████████████████| 508/508 [00:00<00:00, 1154.55it/s, # action=584]
Epochs:   0%|                                                            | 0/100 [00:00<?, ?it/s]Traceback (most recent call last):
  File "C:\Users\DELL\anaconda3\envs\chat\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\DELL\anaconda3\envs\chat\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\DELL\anaconda3\envs\chat\Scripts\rasa.exe\__main__.py", line 7, in <module>
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\rasa\__main__.py", line 117, in main
    cmdline_arguments.func(cmdline_arguments)
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\rasa\cli\train.py", line 59, in <lambda>
    train_parser.set_defaults(func=lambda args: run_training(args, can_exit=True))
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\rasa\cli\train.py", line 103, in run_training
    finetuning_epoch_fraction=args.epoch_fraction,
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\rasa\api.py", line 124, in train
    loop,
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\rasa\utils\common.py", line 296, in run_in_loop
    result = loop.run_until_complete(f)
  File "C:\Users\DELL\anaconda3\envs\chat\lib\asyncio\base_events.py", line 583, in run_until_complete
    return future.result()
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\rasa\model_training.py", line 119, in train_async
    finetuning_epoch_fraction=finetuning_epoch_fraction,
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\rasa\model_training.py", line 299, in _train_async_internal
    finetuning_epoch_fraction=finetuning_epoch_fraction,
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\rasa\model_training.py", line 361, in _do_training
    finetuning_epoch_fraction=finetuning_epoch_fraction,
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\rasa\model_training.py", line 556, in _train_core_with_validated_data
    model_to_finetune=model_to_finetune,
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\rasa\core\train.py", line 70, in train
    agent.train(training_data, **additional_arguments)
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\rasa\core\agent.py", line 754, in train
    training_trackers, self.domain, interpreter=self.interpreter, **kwargs
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\rasa\core\policies\ensemble.py", line 207, in train
    trackers_to_train, domain, interpreter=interpreter, **kwargs
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\rasa\core\policies\ted_policy.py", line 690, in train
    self.run_training(model_data, label_ids)
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\rasa\core\policies\ted_policy.py", line 647, in run_training
    shuffle=False,  # we use custom shuffle inside data generator
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\tensorflow\python\keras\engine\training.py", line 108, in _method_wrapper
    return method(self, *args, **kwargs)
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\rasa\utils\tensorflow\temp_keras_modules.py", line 191, in fit
    tmp_logs = train_function(iterator)
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\tensorflow\python\eager\def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\tensorflow\python\eager\def_function.py", line 840, in _call
    return self._stateless_fn(*args, **kwds)
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\tensorflow\python\eager\function.py", line 2829, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\tensorflow\python\eager\function.py", line 1848, in _filtered_call
    cancellation_manager=cancellation_manager)
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\tensorflow\python\eager\function.py", line 1924, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\tensorflow\python\eager\function.py", line 550, in call
    ctx=ctx)
  File "C:\Users\DELL\anaconda3\envs\chat\lib\site-packages\tensorflow\python\eager\execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError:  reshape cannot infer the missing input size for an empty tensor unless all specified input sizes are non-zero
         [[{{node cond_1/then/_82/cond_1/concatenate_sparse_dense_features_active_loop_sentence/sparse_to_dense.active_loop_sentence/SparseReshape}}]] [Op:__inference_train_function_65875]

Function call stack:
train_function

nik202 · October 7, 2021, 8:03am

@trinhminhhieu solve the errors as shown in debug and even delete the older trained models and re-train, sometime that trick also help, if you are sure with the code.

trinhminhhieu · October 7, 2021, 8:17am

I deleted the model file and dug again but I still get the error

Function call stack:
train_function

nik202 · October 7, 2021, 9:30am

@trinhminhhieu Hello again! I have seen and dry run your code, but the issue is not your coding, I guess your pipeline in config.yml as you are developing the vi (Vietnamese bot), so you need to decide the pipeline first; I am sharing some link for your reference how you can customise the pipeline, I hope this will help you. Even, I can understand the pressure about your project and university for the same.

Reference Paper (check page 4): https://www.researchgate.net/publication/348390901_Enhancing_Rasa_NLU_model_for_Vietnamese_chatbot

Reference Vietnamese Community: Rasa Community in Vietnam

Reference Github (Its 2 year old- just for idea don’t try to run): https://github.com/heraclex12/vietnamese-chat-with-rasa

I’m really sorry, I tried my level best to help you!

Urgent Note: I can be totally wrong in this too, If any other person seeing this post please try suggest the solution for the same.

Currently the bot is working as you can see in below post, I am not expert in pipeline on any other language except English

trinhminhhieu · October 7, 2021, 9:57am

Thanks for your suggestion, I also tried training the model in English but it still doesn’t work. Can you test this model? It still gives me the error

Function call stack:
train_function

nik202 · October 7, 2021, 10:00am

@trinhminhhieu I can but your domain file have vi words and this is not parsing

@trinhminhhieu wait let me check your nlu file again, you did a blunder there.

Please stay!!

trinhminhhieu · October 7, 2021, 10:07am

Thanks, I’m still waiting for you

nik202 · October 7, 2021, 10:07am

@trinhminhhieu model trained wait

Yeah!! I don’t know Vietnamese.

Errors: need to change:

You mentioned the forms and responses in nlu.yml
Even, I have updated something in config.yml
In domain,yml there is button payload which is not in proper syntax, so I commented them
I copied the forms and responses from nlu to domain.yml

These are your issue.

domain.yml (8.3 KB) config.yml (1.0 KB) endpoints.yml (1.4 KB) credentials.yml (980 Bytes) nlu.yml (8.1 KB)

Update by nik202

trinhminhhieu · October 7, 2021, 10:13am

oh, it works! How did you do?

nik202 · October 7, 2021, 10:22am

@trinhminhhieu Now, its working can you close this thread as a solution for others and good luck!

trinhminhhieu · October 7, 2021, 11:15am

@nik202 Thank you very much!

Topic		Replies	Views
Traceback most recent call Rasa Open Source	14	1456	October 29, 2021
New rasa core dialogue training error Rasa Open Source	2	1389	November 15, 2018
[Rasa] 'rasa train core" returns GraphicsSchemaValidationException error Getting Started with Rasa	0	149	July 6, 2022
Error when training RASA model, please help! Rasa Open Source	1	396	October 27, 2021
NLU training failing Rasa Open Source	5	1832	November 20, 2019

Rasa Train Error Function call stack: train_on_batch

Related Topics