Circuit breaker tripped. Stopped predicting more actions for sender 'XXX'

Rasa Open Source
1

Hi all,

I have an issue with Rasa deployed on slack. When an error is found on the action server, the rasa server seems to lock all future actions for the sender that provoked this error. This was not the case in local mode, where the rasa server was simply ignoring the error message from the action server.

I was wondering if there is a way to deactivate this ‘circuit breaker’ option and keep it working for the user that caused the error, like in local mode.

Logs form action server:

2022-05-20 10:11:32	
2022-05-20T10:11:32.325066976Z stderr F Exception occurred while handling uri: 'xxx'
2022-05-20 10:11:32	
2022-05-20T10:11:32.325299731Z stderr F Traceback (most recent call last):
2022-05-20 10:11:32	
2022-05-20T10:11:32.32534708Z stderr F   File "handle_request", line 83, in handle_request
2022-05-20 10:11:32	
2022-05-20T10:11:32.325380654Z stderr F     )
2022-05-20 10:11:32	
2022-05-20T10:11:32.325410827Z stderr F   File "/opt/venv/lib/python3.8/site-packages/rasa_sdk/endpoint.py", line 104, in webhook
2022-05-20 10:11:32	
2022-05-20T10:11:32.325440146Z stderr F     result = await executor.run(action_call)
2022-05-20 10:11:32	
2022-05-20T10:11:32.325470595Z stderr F   File "/opt/venv/lib/python3.8/site-packages/rasa_sdk/executor.py", line 398, in run
2022-05-20 10:11:32	
2022-05-20T10:11:32.325500154Z stderr F     action(dispatcher, tracker, domain)
2022-05-20 10:11:32	
2022-05-20T10:11:32.325529215Z stderr F   File "/app/actions/actions.py", line 61, in run
2022-05-20 10:11:32	
2022-05-20T10:11:32.325561142Z stderr F     buttons.append({"title": f'👍 : {intent.replace("_", " ")}' , "payload": '/affirm'})
2022-05-20 10:11:32	
2022-05-20T10:11:32.325610422Z stderr F AttributeError: 'NoneType' object has no attribute 'replace'type or paste code here


Logs from rasa server:


2022-05-20 10:11:32	
2022-05-20T10:11:32.314086964Z stderr F 2022-05-20 10:11:32 DEBUG    rasa.core.processor  - Predicted next action 'pi_outside' with confidence 0.53.
2022-05-20 10:11:32	
2022-05-20T10:11:32.314790889Z stderr F 2022-05-20 10:11:32 DEBUG    rasa.core.actions.action  - Calling action endpoint to run action 'pi_outside'.
2022-05-20 10:11:32	
2022-05-20T10:11:32.327614004Z stderr F 2022-05-20 10:11:32 ERROR    rasa.core.processor  - Encountered an exception while running action 'pi_outside'.Bot will continue, but the actions events are lost. Please check the logs of your action server for more information.
2022-05-20 10:11:32	
2022-05-20T10:11:32.32780414Z stderr F Traceback (most recent call last):
2022-05-20 10:11:32	
2022-05-20T10:11:32.327844478Z stderr F   File "/opt/venv/lib/python3.8/site-packages/rasa/core/actions/action.py", line 742, in run
2022-05-20 10:11:32	
2022-05-20T10:11:32.327853161Z stderr F     response: Any = await self.action_endpoint.request(
2022-05-20 10:11:32	
2022-05-20T10:11:32.327859462Z stderr F   File "/opt/venv/lib/python3.8/site-packages/rasa/utils/endpoints.py", line 173, in request
2022-05-20 10:11:32	
2022-05-20T10:11:32.327886837Z stderr F     raise ClientResponseError(
2022-05-20 10:11:32	
2022-05-20T10:11:32.327893664Z stderr F rasa.utils.endpoints.ClientResponseError: 500, Internal Server Error, body='b'{"description":"Internal Server Error","status":500,"message":"The server encountered an internal error and cannot complete your request."}''
2022-05-20 10:11:32	
2022-05-20T10:11:32.327955422Z stderr F 
2022-05-20 10:11:32	
2022-05-20T10:11:32.327965006Z stderr F The above exception was the direct cause of the following exception:
2022-05-20 10:11:32	
2022-05-20T10:11:32.327970555Z stderr F 
2022-05-20 10:11:32	
2022-05-20T10:11:32.327976526Z stderr F Traceback (most recent call last):
2022-05-20 10:11:32	
2022-05-20T10:11:32.32798281Z stderr F   File "/opt/venv/lib/python3.8/site-packages/rasa/core/processor.py", line 869, in _run_action
2022-05-20 10:11:32	
2022-05-20T10:11:32.327988871Z stderr F     events = await action.run(
2022-05-20 10:11:32	
2022-05-20T10:11:32.327994768Z stderr F   File "/opt/venv/lib/python3.8/site-packages/rasa/core/actions/action.py", line 766, in run
2022-05-20 10:11:32	
2022-05-20T10:11:32.328005762Z stderr F     raise RasaException("Failed to execute custom action.") from e
2022-05-20 10:11:32	
2022-05-20T10:11:32.328036077Z stderr F rasa.shared.exceptions.RasaException: Failed to execute custom action.
2022-05-20 10:11:32	
2022-05-20T10:11:32.328057571Z stderr F 2022-05-20 10:11:32 DEBUG    rasa.core.processor  - Policy prediction ended with events '[]'.
2022-05-20 10:11:32	
2022-05-20T10:11:32.328068288Z stderr F 2022-05-20 10:11:32 DEBUG    rasa.core.processor  - Action 'pi_outside' ended with events '[]'.
2022-05-20 10:11:32	
2022-05-20T10:11:32.328074527Z stderr F 2022-05-20 10:11:32 DEBUG    rasa.core.processor  - Current slot values: 
2022-05-20 10:11:32	
2022-05-20T10:11:32.328081599Z stderr F 	kb_filter: [False, False, False, False, False, True, True, True, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False]
2022-05-20 10:11:32	
2022-05-20T10:11:32.328087675Z stderr F 	change_of_heart: False
2022-05-20 10:11:32	
2022-05-20T10:11:32.328156864Z stderr F 	current_intent: None
2022-05-20 10:11:32	
2022-05-20T10:11:32.328164498Z stderr F 	prev_intent: None
2022-05-20 10:11:32	
2022-05-20T10:11:32.328170204Z stderr F 	change_of_heart_ent: False
2022-05-20 10:11:32	
2022-05-20T10:11:32.328175535Z stderr F 	entity_out_of_filt_value: False
2022-05-20 10:11:32	
2022-05-20T10:11:32.328181251Z stderr F 	entity_out_of_filt_name: False
2022-05-20 10:11:32	
2022-05-20T10:11:32.328186944Z stderr F 	solution: None
2022-05-20 10:11:32	
2022-05-20T10:11:32.32819235Z stderr F 	produits: Epargne salariale
2022-05-20 10:11:32	
2022-05-20T10:11:32.328197973Z stderr F 	categories: Participation
2022-05-20 10:11:32	
2022-05-20T10:11:32.328229377Z stderr F 	situation_parents: Not applicable
2022-05-20 10:11:32	
2022-05-20T10:11:32.328237173Z stderr F 	situation_enfants: Not applicable
2022-05-20 10:11:32	
2022-05-20T10:11:32.328243048Z stderr F 	sous_categories: Affectation
2022-05-20 10:11:32	
2022-05-20T10:11:32.328249209Z stderr F 	mode_payement: None
2022-05-20 10:11:32	
2022-05-20T10:11:32.32825477Z stderr F 	nom_canaux: None
2022-05-20 10:11:32	
2022-05-20T10:11:32.328260342Z stderr F 	type_canaux: None
2022-05-20 10:11:32	
2022-05-20T10:11:32.328266407Z stderr F 	save_produits: Epargne salariale
2022-05-20 10:11:32	
2022-05-20T10:11:32.328272029Z stderr F 	save_categories: Participation
2022-05-20 10:11:32	
2022-05-20T10:11:32.328277524Z stderr F 	save_situation_parents: None
2022-05-20 10:11:32	
2022-05-20T10:11:32.328282919Z stderr F 	save_situation_enfants: None
2022-05-20 10:11:32	
2022-05-20T10:11:32.328288212Z stderr F 	save_sous_categories: Affectation
2022-05-20 10:11:32	
2022-05-20T10:11:32.328305846Z stderr F 	save_mode_payement: None
2022-05-20 10:11:32	
2022-05-20T10:11:32.328311815Z stderr F 	save_nom_canaux: None
2022-05-20 10:11:32	
2022-05-20T10:11:32.328347167Z stderr F 	save_type_canaux: None
2022-05-20 10:11:32	
2022-05-20T10:11:32.328355587Z stderr F 	requested_slot: None
2022-05-20 10:11:32	
2022-05-20T10:11:32.328392491Z stderr F 	session_started_metadata: None
2022-05-20 10:11:32	
2022-05-20T10:11:32.32841341Z stderr F 2022-05-20 10:11:32 WARNING  rasa.core.processor  - Circuit breaker tripped. Stopped predicting more actions for sender 'U02EHGKKWMQ'.
2022-05-20 10:11:32	
2022-05-20T10:11:32.340818276Z stderr F 2022-05-20 10:11:32 DEBUG    rasa.core.tracker_store  - Tracker with sender_id 'U02EHGKKWMQ' stored to database
2022-05-20 10:11:32	
2022-05-20T10:11:32.342845195Z stderr F 2022-05-20 10:11:32 DEBUG    rasa.core.lock_store  - Deleted lock for conversation 'U02EHGKKWMQ'.
2022-05-20 10:11:32	
2022-05-20T10:11:32.577524006Z stderr F 2022-05-20 10:11:32 DEBUG    rasa.core.channels.slack  - Received message from Slack which doesn't look like a user message. Skipping message.
2022-05-20 10:11:32	
2022-05-20T10:11:32.781231574Z stderr F 2022-05-20 10:11:32 DEBUG    rasa.core.channels.slack  - Received message from Slack which doesn't look like a user message. Skipping message.
2022-05-20 10:12:18	
2022-05-20T10:12:18.153712918Z stderr F 2022-05-20 10:12:18 DEBUG    rasa.core.lock_store  - Issuing ticket for conversation 'U02EHGKKWMQ'.
2022-05-20 10:12:18	
2022-05-20T10:12:18.157956828Z stderr F 2022-05-20 10:12:18 DEBUG    rasa.core.lock_store  - Acquiring lock for conversation 'U02EHGKKWMQ'.
2022-05-20 10:12:18	
2022-05-20T10:12:18.158709788Z stderr F 2022-05-20 10:12:18 DEBUG    rasa.core.lock_store  - Acquired lock for conversation 'U02EHGKKWMQ'.
2022-05-20 10:12:18	
2022-05-20T10:12:18.166105541Z stderr F 2022-05-20 10:12:18 DEBUG    rasa.core.tracker_store  - Recreating tracker from sender id 'U02EHGKKWMQ'
2

Hello, try set this variable (default 10)

export MAX_NUMBER_OF_PREDICTIONS=30

See processor.py for reference

1 Like
3

The error is clear:

Please send your custom action

1 Like
4

Hi @artemsnegirev ,

thanks for the suggestion. Where should I add: export MAX_NUMBER_OF_PREDICTIONS=30?

5

Hi @ChrisRahme,

thanks for your reply. I have noticed as well that the error comes from the actions server, as explained in the question.

The error is triggered because the custom action ‘pi_outside’ (find code below) was called by the rasa server when it shouldn’t have been, before the slot ‘current_intent’ was set. I tried fixing the stories to avoid this behaviour.

The problem is that the bot doesn’t answer afterwards to the user ‘XXX’, and I get the warning ‘circuit breaker has tripped’ for all future messages, until I clear the tracker store for this user.

class ActionIntOutOfForm(Action):
    def name(self):
        return 'pi_outside'

    def run(self, dispatcher, tracker, domain):
        # Init filter
        intent = tracker.get_slot('current_intent')
        prev_intent = tracker.get_slot('prev_intent')
        buttons =[]
        try:
            buttons.append({"title": f'👍 : {intent.replace("_", " ")}' , "payload": '/affirm'})
            buttons.append({"title": f'👎 : {prev_intent.replace("_", " ")}' , "payload": '/deny'})
        except AttributeError:
            return [FollowupAction('action_ask_produits')]
        dispatcher.utter_message(text= f"Nous venons de detecter un changement d'intention de votre part. Pouvez-vous nous confirmer qu'il ne s'agit pas d'une erreur, et que vous voulez bien {intent.replace('_', ' ')}, et non pas {prev_intent.replace('_', ' ')} \n Voulez-vous changer de demande ?", buttons=buttons)
1 Like
6

First, you should always handle potential errors, just in case stuff like that happens, and it will :slight_smile:

Good to see you used a try block now! I also suggest adding return [] at the end instead of nothing or even return.

I’m sorry, I don’t understand this

1 Like
7

First, enter in your bash session

export MAX_NUMBER_OF_PREDICTIONS=30

And after in same bash session run/restart your rasa server as usual

rasa run --enable-api
1 Like
8

Hi Chris, thanks again for this reply.

I mean that what I would like to avoid the state of ‘circuit breaker’, where the rasa core ‘Stopped predicting more actions for sender ‘U02EHGKKWMQ’.’

Yes, sorry I forgot to mention I added the try block since the bug :smile: . Could you explain why you are suggesting to finish the functions by return [] or return rather then leave blank?

9

Great, thanks artsemsneigirev, will try this.

1 Like