Segmentation fault rasa train on enabling Intentless Policy

Rasa Pro CALM
1

When I turn on IntentlessPolicy as below

  • name: rasa_plus.ml.IntentlessPolicy nlu_abstention_threshold: 0.2 I get 2024-02-27 18:58:29 INFO rasa.engine.training.hooks - Starting to train component ‘IntentlessPolicy’. 2024-02-27 18:58:31 INFO faiss.loader - Loading faiss with AVX2 support. 2024-02-27 18:58:31 INFO faiss.loader - Successfully loaded faiss with AVX2 support. zsh: segmentation fault rasa train

Kindly please help me to debug and solve this issue

Thanks, Geeta

2

Hi @geeta.m.desai ,

Can you provide more detail that would allow us to reproduce your issue?

Perhaps you have your model in a Github repo so we can have a look?

3

Hi Arjaan,

Thank you for respondng back. Please find my project here https://github.com/geeta-m-desai/rasa-calm/blob/main/config.yml

Earlier I had lot of code, I removed everything and simplified to see if it fixes problem but it did not help

I have tried creating new environment and other such techniques but did not help

Thanks,

Geeta

4

Hi Geeta, I cloned your repo and was able to train successfully with just rasa train. This is with python 3.9.16 and rasa-plus 3.7.8. What versions are you using?

5

Links:

6

Could you paste the output of the command rasa --version? This is a good start to debugging.

7

(venv) geetadesai@MacBook-Pro rasa-calm % rasa --version Rasa Version : 3.7.7 Minimum Compatible Version: 3.7.0 Rasa SDK Version : 3.7.1 Python Version : 3.10.10 Operating System : macOS-14.3.1-x86_64-i386-64bit Python Path : /Users/geetadesai/rasa-calm/venv/bin/python3

8

Here is my rasa --version:

geeta-rasa-calm % rasa --version
Rasa Version      :         3.7.8
Minimum Compatible Version: 3.7.0
Rasa SDK Version  :         3.7.1
Python Version    :         3.10.13
Operating System  :         macOS-14.3.1-x86_64-i386-64bit
Python Path       :         /Users/john/src/github.com/trammell/bots/geeta-rasa-calm/.direnv/python-3.10/bin/python
	Rasa Plus Version  :         3.7.8
geeta-rasa-calm %

And my rasa train:

geeta-rasa-calm % rasa train
2024-03-03 13:40:55 INFO     rasa_plus.tracing.config  - No endpoint for tracing type available in endpoints.yml,tracing will not be configured.
2024-03-03 13:40:55 INFO     rasa_plus.telemetry  - Initialised global config file with Rasa Pro telemetry tracking set to True.
2024-03-03 13:40:55 INFO     rasa.cli.train  - Started validating domain and training data...
2024-03-03 13:40:56 INFO     rasa.validator  - Validating intents...
/Users/john/src/github.com/trammell/bots/geeta-rasa-calm/.direnv/python-3.10/lib/python3.10/site-packages/rasa/shared/utils/io.py:99: UserWarning: The intent 'affirm' is not used in any story, rule or flow.
/Users/john/src/github.com/trammell/bots/geeta-rasa-calm/.direnv/python-3.10/lib/python3.10/site-packages/rasa/shared/utils/io.py:99: UserWarning: The intent 'ask_question' is not used in any story, rule or flow.
/Users/john/src/github.com/trammell/bots/geeta-rasa-calm/.direnv/python-3.10/lib/python3.10/site-packages/rasa/shared/utils/io.py:99: UserWarning: The intent 'deny' is not used in any story, rule or flow.
/Users/john/src/github.com/trammell/bots/geeta-rasa-calm/.direnv/python-3.10/lib/python3.10/site-packages/rasa/shared/utils/io.py:99: UserWarning: The intent 'goodbye' is not used in any story, rule or flow.
2024-03-03 13:40:56 INFO     rasa.validator  - Validating uniqueness of intents and stories...
2024-03-03 13:40:56 INFO     rasa.validator  - Validating utterances...
2024-03-03 13:40:56 INFO     rasa.validator  - Story structure validation...
Processed story blocks: 100%|█████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 464.07it/s, # trackers=1]
2024-03-03 13:40:56 INFO     rasa.core.training.story_conflict  - Considering all preceding turns for conflict analysis.
2024-03-03 13:40:56 INFO     rasa.validator  - No story structure conflicts found.
2024-03-03 13:40:56 INFO     rasa.validator  - [info     ] validation.flows.started
2024-03-03 13:40:56 INFO     rasa.validator  - [info     ] validation.flows.ended
Processed story blocks: 100%|████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1876.65it/s, # trackers=1]
Processed story blocks: 100%|████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1615.68it/s, # trackers=1]
Processed story blocks: 100%|████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1364.00it/s, # trackers=2]
Processed story blocks: 100%|████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1180.50it/s, # trackers=3]
2024-03-03 13:40:58 INFO     rasa.engine.training.hooks  - Starting to train component 'FlowPolicy'.
2024-03-03 13:40:58 INFO     rasa.engine.training.hooks  - Finished training component 'FlowPolicy'.
2024-03-03 13:40:58 INFO     rasa.engine.training.hooks  - Starting to train component 'IntentlessPolicy'.
2024-03-03 13:40:59 INFO     faiss.loader  - Loading faiss with AVX2 support.
2024-03-03 13:40:59 INFO     faiss.loader  - Successfully loaded faiss with AVX2 support.
2024-03-03 13:40:59 INFO     rasa_plus.ml.intentless_policy  - [info     ] intentless_policy.training.completed
2024-03-03 13:40:59 INFO     rasa.engine.training.hooks  - Finished training component 'IntentlessPolicy'.
2024-03-03 13:40:59 INFO     rasa.engine.training.hooks  - Starting to train component 'LLMCommandGenerator'.
2024-03-03 13:40:59 INFO     rasa.engine.training.hooks  - Finished training component 'LLMCommandGenerator'.
Your Rasa model is trained and saved at 'models/20240303-134057-golden-beagle.tar.gz'.
geeta-rasa-calm %

I notice a couple of things:

  • I have a more recent version of rasa-plus than you do, can you try updating and try again?
  • Some cache files got into your git repo, if the updates don’t work can you try deleting the contents of .rasa/ and try again?
9

It seems like you’re encountering a segmentation fault error when training your Rasa model with the IntentlessPolicy enabled. Here are a few steps you can take to debug and resolve this issue:

  1. Check Dependencies: Ensure that all required dependencies for Rasa, including Faiss, are properly installed and up to date. The segmentation fault could be caused by a compatibility issue or missing dependency.

  2. Update Libraries: Update Rasa and Faiss to the latest versions to see if the issue persists. Sometimes, bugs or compatibility issues are fixed in newer releases.

  3. Review Configuration: Double-check your configuration settings for the IntentlessPolicy. Verify that the values are correct and compatible with your Rasa version.

  4. Memory Usage: Segmentation faults can sometimes occur due to memory issues. Monitor your system’s memory usage during training to see if it’s reaching its limits. Consider optimizing your system’s memory allocation if necessary.

  5. Isolate the Issue: Try training your Rasa model without the IntentlessPolicy enabled to see if the segmentation fault persists. If it doesn’t occur without the policy, it could indicate a problem with the policy implementation.

  6. Error Logs: Check for any additional error logs or messages that may provide more insight into the cause of the segmentation fault. Look for any specific patterns or clues that could help diagnose the issue.

  7. Community Support: Reach out to the Rasa community forums or GitHub repository for assistance. Other users or developers may have encountered similar issues and could provide helpful advice or solutions.

By following these steps and carefully investigating the possible causes of the segmentation fault, you should be able to debug and resolve the issue with training your Rasa model.

10

Hi John,

I ahve upgraded to 3.7.8

It is failing b/c I am tring to create my own version of KnowledgeAnswerCommand

Thanks,
Geeta
11

I checked out branch new-branch2, I see a couple of things that need to be changed:

12

But I am using CALM and I want to route all the unidentified queries to this new CustomKnowledgeAnswerCommand

Is it possble to specify a specific action as a fallback when using CALM ?

I had tried below in flows.yml

fallback:
  description: fallback action
  name: fallback
  steps:
      - action: action_custom_knowledge_answer
it had not worked , hence was trying to triggerCustomKnowledgeAnswerCommand through policies.
13

There are instructions on how to build custom actions with CALM here.

14

Thank you. I have moved past of these issues for building my voice bot using Rasa-CALM.

I have a functionaity for reminders. I ahve configured as per documentation.But I see warning as (venv) geetadesai@MacBook-Pro rasa-calm % rasa data validate 2024-03-10 04:02:04 INFO rasa_plus.telemetry - Initialised global config file with Rasa Pro telemetry tracking set to True. 2024-03-10 04:02:06 INFO rasa.validator - Validating intents… /Users/geetadesai/rasa-calm/venv/lib/python3.10/site-packages/rasa/shared/utils/io.py:99: UserWarning: The intent ‘EXTERNAL_reminder_callback’ is not used in any story, rule or flow. Though my domain.yml has that entry responses: utter_time_for_medication: - text: “It’s time for your medication.” rules:

  • rule: Schedule medication reminder steps:

    • intent: schedule_medication
    • action: action_schedule_medication_reminder

  • rule: reminder callback steps: - intent: EXTERNAL_reminder_callback - action: action_notify_medication Git URL GitHub - geeta-m-desai/rasa-calm at new-branch2 Could you pelase provide some hints on this, I am stuck on this for 2 days.

(venv) geetadesai@MacBook-Pro rasa-calm % rasa --version Rasa Version : 3.7.8 Minimum Compatible Version: 3.7.0 Rasa SDK Version : 3.7.1 Python Version : 3.10.10 Operating System : macOS-14.3.1-x86_64-i386-64bit Python Path : /Users/geetadesai/rasa-calm/venv/bin/python3 Rasa Plus Version : 3.7.8

It always triggers intentlessPOlicy Your input → reminders
2024-03-10 04:48:11 INFO rasa.utils.log_utils - [info ] llm_command_generator.predict_commands.finished commands=[StartFlowCommand(flow=‘schedule_reminders’)] I will remind you in 5 seconds. What more can I assist you with? Your input → 2024-03-10 04:49:12 INFO rasa_plus.ml.intentless_policy - [info ] intentless_policy.prediction.completed predicted_action_name=utter_boolean_slot_rejection score=0.6538142578480671

Action server logs reminder ===> {‘event’: ‘reminder’, ‘timestamp’: None, ‘intent’: ‘EXTERNAL_reminder_callback’, ‘entities’: , ‘date_time’: ‘2024-03-10T04:49:11.686317’, ‘name’: ‘my_reminder’, ‘kill_on_user_msg’: False} 2024-03-10 04:48:11 DEBUG rasa_sdk.executor - Finished running ‘action_schedule_medication_reminder’

Thanks, Geeta

15

Hi Geeta, it looks like that link is broken, so I can’t see your code.

Can you share your config.yml and domain.yml files? I think I see that you’re trying to use intentless policy, is that correct?

16

Hi John,

Sorry I made it private now.

I am using CALM and intentless policy. Attached is the code and reminders EXTERNAL_reminder_callback is aways triggering intenless policy

Thanks,

Geeta

rasa-calm-new-branch2.zip (770 KB)

17

I’m able to train and run the assistant you provided. You include a news flow that looks like this:

  read_news_flow:
    description: news
    name: news
    steps:
        - action: action_fetch_news

It is important to use the description field to describe the flow, because that is what is used by the LLM to choose the flow that is started. A better flow would be:

  read_news_flow:
    description: Inform the user about news and current events.
    name: news
    steps:
        - action: action_fetch_news

When I prompt the assistant with “news” it predicts the news flow and tries to run the custom action:

2024-03-13 18:41:21 DEBUG    rasa.engine.graph  - [debug    ] graph.node.running_component   clazz=DefaultPolicyPredictionEnsemble fn=combine_predictions_from_kwargs node_name=select_prediction
2024-03-13 18:41:21 DEBUG    rasa.core.policies.ensemble  - Made prediction using user intent.
2024-03-13 18:41:21 DEBUG    rasa.core.policies.ensemble  - Added `DefinePrevUserUtteredFeaturization(False)` event.
2024-03-13 18:41:21 DEBUG    rasa.core.policies.ensemble  - Predicted next action using FlowPolicy.
2024-03-13 18:41:21 DEBUG    rasa.core.processor  - Predicted next action 'action_fetch_news' with confidence 1.00.
18

Hi John,

Issue is not about News flows it is about reminders

(venv) geetadesai@MacBook-Pro rasa-calm % rasa --version Rasa Version : 3.7.8 Minimum Compatible Version: 3.7.0 Rasa SDK Version : 3.7.1 Python Version : 3.10.10 Operating System : macOS-14.3.1-x86_64-i386-64bit Python Path : /Users/geetadesai/rasa-calm/venv/bin/python3 Rasa Plus Version : 3.7.8

It always triggers intentlessPOlicy Your input → reminders 2024-03-10 04:48:11 INFO rasa.utils.log_utils - [info ] llm_command_generator.predict_commands.finished commands=[StartFlowCommand(flow=‘schedule_reminders’)] I will remind you in 5 seconds. What more can I assist you with? Your input → 2024-03-10 04:49:12 INFO rasa_plus.ml.intentless_policy - [info ] intentless_policy.prediction.completed predicted_action_name=utter_boolean_slot_rejection score=0.6538142578480671

Action server logs reminder ===> {‘event’: ‘reminder’, ‘timestamp’: None, ‘intent’: ‘EXTERNAL_reminder_callback’, ‘entities’: , ‘date_time’: ‘2024-03-10T04:49:11.686317’, ‘name’: ‘my_reminder’, ‘kill_on_user_msg’: False} 2024-03-10 04:48:11 DEBUG rasa_sdk.executor - Finished running ‘action_schedule_medication_reminder’

I have made the git repo public again

https://github.com/geeta-m-desai/rasa-calm/tree/new-branch3

Thanks,

Geeta

blocked.gif

19

I see. The reminders functionality is coupled with the NLU architecture and I would not expect it to work with CALM, at least not until we have released the coexistence functionality.

20

So if we are using Rasa pro 3.7 ,how we should build reminder functionality