Custom Policy Not Working

I’m trying to create a custom policy that predicts one specific action all the time when a slot is set, but it is not working for some reason. Below is my policy’s code:

from rasa.core.policies import Policy
from rasa.core.policies.policy import confidence_scores_for
import numpy as np

class MyPolicy(Policy):
    def predict_action_probabilities(self, tracker, domain):
        if tracker.get_slot("my_slot") == True:
            return confidence_scores_for("action_my_custom_action", 1.0, domain) 
        else:
            return np.zeros(domain.num_actions)

the policy file is in a directory called custom_policies/ in a file called testpolicy.py. I also have a file called __init__.py in that directory. Below is the relevant part of my config.yml:

...
policies:
  - name: MemoizationPolicy
    max_history: 1
  - name: MappingPolicy
  - name: FormPolicy
  - name: TwoStageFallbackPolicy
  - name: "custom_policies.testpolicy.MyPolicy"
    priority: 15

Hi @basil-chatha, when you say it doesn’t work what do you mean? Is there an error? Or does the policy not get used? If it’s the second case, could you post the debug logs of the conversation?

Also, are you sure using a custom policy is the right course here? It could also be achieved with a story that predicts a handoff and then pauses the conversation for example.

@akelad so the error I was getting before was because my model was not training using the rasax ui so I had to fix that. Here are the error logs I get for rasa-production now:

rasa-production_1  | Traceback (most recent call last):
rasa-production_1  |   File "/opt/venv/lib/python3.7/site-packages/rasa/core/agent.py", line 100, in _load_and_set_updated_model
rasa-production_1  |     policy_ensemble = PolicyEnsemble.load(core_path)
rasa-production_1  |   File "/opt/venv/lib/python3.7/site-packages/rasa/core/policies/ensemble.py", line 253, in load
rasa-production_1  |     policy_cls = registry.policy_from_module_path(policy_name)
rasa-production_1  |   File "/opt/venv/lib/python3.7/site-packages/rasa/core/registry.py", line 24, in policy_from_module_path
rasa-production_1  |     raise ImportError(f"Cannot retrieve policy from path '{module_path}'")
rasa-production_1  | ImportError: Cannot retrieve policy from path 'custom_policies.testpolicy.MyPolicy'
rasa-production_1  | 2020-07-16 02:25:57 ERROR    rasa.core.agent  - Failed to load policy and update agent. The previous model will stay loaded instead.
rasa-production_1  | Traceback (most recent call last):
rasa-production_1  |   File "/opt/venv/lib/python3.7/site-packages/rasa/core/registry.py", line 21, in policy_from_module_path
rasa-production_1  |     module_path, lookup_path="rasa.core.policies.registry"
rasa-production_1  |   File "/opt/venv/lib/python3.7/site-packages/rasa/utils/common.py", line 198, in class_from_module_path
rasa-production_1  |     m = importlib.import_module(module_name)
rasa-production_1  |   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
rasa-production_1  |     return _bootstrap._gcd_import(name[level:], package, level)
rasa-production_1  |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
rasa-production_1  |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
rasa-production_1  |   File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
rasa-production_1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
rasa-production_1  |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
rasa-production_1  |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
rasa-production_1  |   File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
rasa-production_1  | ModuleNotFoundError: No module named 'custom_policies'

Have you mounted this file to the rasa-production container in your docker-compose.override.yml?

@akelad Ok thanks yes now I’m doing that and am no longer getting that error. For some reason this method is not triggering the action I want it to run for some reason. Here’s my code:

is_triggered = tracker.get_slot('my_slot')

if is_triggered:
    logger.info('if statement ' + str(is_triggered) + ' ' + str(type(is_triggered)))
    return confidence_scores_for("action_my_action", 1.0, domain)
else:
    logger.info('in else statement ' + str(is_triggered) + ' ' + str(type(is_triggered)))
    return self._default_predictions(domain)

I’m not getting any errors in my logs and I know it is triggering this policy now because I was getting errors from within the policy earlier and am no longer getting them. Any suggestions?

@basil-chatha can you post the debug logs of the conversation where the action isn’t being run? I want to make sure that the policy is actually the one predicting the action

@akelad Actually I retried it and it seems to be working now. However, I’m getting a circuit breaker tripped error. Is that likely because my model is not trained on stories with this new conversation flow?

Yes, that’s usually the issue with the circuit braker tripped error