Unable to properly use Rule Policy

Rasa Open Source
1

Hi, I am using rasa 2.x and using Rule Policy to handle queries which my model is not trained on.

Here is the rue policy:

> - name: RulePolicy
>     core_fallback_threshold: 0.7
>     core_fallback_action_name: "action_default_fallback"
>     enable_fallback_prediction: True

I have given the text:

can you help me?

My model is not trained for this query and the prediction score is 0.50 (as seen below). So, as per Rule policy, It should predict ‘utter_default’ but it has perdicted ‘greet’ Here is the output: rasa.core.processor - Received user message 'can you help me?' with intent '{'id': -3112698601225084444, 'name': 'greet', 'confidence': 0.5097352862358093}' and entities '[]'

Please help.

2

@Sajjadmanal can you try this configuration and do confirm the output, please also delete the older trained model and try run it again :

Note: This is for demo purpose only, please change the values as per your use case.


policies:
  - name: MemoizationPolicy
  - name: TEDPolicy
    max_history: 5
    epochs: 100
  - name: RulePolicy
    nlu_threshold: 0.4
    core_threshold: 0.4
    fallback_action_name: "action_default_fallback"

I hope you had mentioned the utter_default: in the domain.yml

1 Like
3

@nik202 Yes, I have utter_default in domain.yml why the threshold is 0.4 and not 0.7? greet was being predicted with 0.50 accuracy. (with my configuration)

4

@Sajjadmanal you can set any threshold value, for my use case it’s giving me the default message on these values, so I just shared with you, please do try with different values and check the response.

@Sajjadmanal do even check this input text “can you help me?” have some example mentioned in the greet intent?

5

@nik202 Thanks. It worked. Can you let me know taking threshold as 0.4 means what? The accuracy of prediction should be at least 0.4?

6

@Sajjadmanal its basically the confidence score values, we tried to mention the values between .4-.6 for the best case scenarios, but it can vary as per the use case. In one of my project I set the value to 5 or 6 even.

7

@nik202 Then what does this mean? When i take threshold= 0.7 2021-12-13 19:43:30 DEBUG rasa.core.processor - Received user message 'can you help me?' with intent '{'id': -1385349519514546219, 'name': 'out_of_scope', 'confidence': 0.47837963700294495}' and entities '[]'

8

@Sajjadmanal ok, as you see first screenshot you have shared the intent name is greet and now its out of scope. It means your model is confused and for that you need to provide a good amount of training examples. How many training examples you are mentioned in per intents for your project?

9

@nik202 I have used from 6 to 12. some are 6, some are 8 some are 10-11.