What is the recommended approach to send External Events to a load balanced Core Server stack

Hello everyone,

RASA version: 1.10.1
RASA CDK: 1.10.1

Similar to the Raspberry Pi example in the RASA documentation external events, I have an external process that needs to modify the conversation based on customer behavior.
My development p.o.c utilized the trigger_intent API endpoint. In a one CORE server configuration the expected behavior was successful!!

To mirror the production environment, I decided to test the functionality with a multiple CORE server implementation locally. I suspected that if a conversation started on core_server_1 but the load balancer routed the external trigger_intent POST request to core_server_2, the user would not receive the utterance that accompanies the intent even if the intent was correctly injected. My tests results appear to confirm this behavior.

run multiple core instances locally:

docker-compose --scale core_server=3
docker-compose ps
a_core_server_1     rasa run --cors * --enable ...   Up      0.0.0.0:32794->5005/tcp
a_core_server_2     rasa run --cors * --enable ...   Up      0.0.0.0:32793->5005/tcp
a_core_server_3     rasa run --cors * --enable ...   Up      0.0.0.0:32792->5005/tcp
a_lock_store_1      docker-entrypoint.sh redis ...   Up      0.0.0.0:6379->6379/tcp 
a_tracker_store_1   docker-entrypoint.sh postgres    Up      0.0.0.0:5432->5432/tcp 

In my client I forced the connection to Core Server on localhost:32794 and upgraded to wssocket protocol. With Postman I tested the external trigger intent:
POST: http://localhost:32794/conversations/12345/trigger_intent?output_channel=socketio
BODY:

{
 "name": "greet",
 "entities": {

 }
}

Event in RESPONSE:

            {
                "event": "user",
                "timestamp": 1591477068.711168,
                "metadata": {
                    "is_external": true
                },
                "text": "EXTERNAL: greet",
                "parse_data": {
                    "intent": {
                        "name": "greet"
                    },
                    "entities": [],
                    "text": "EXTERNAL: greet",
                    "message_id": null,
                    "metadata": {
                        "is_external": true
                    }
                },
                "input_channel": null,
                "message_id": null
            },
            {
                "event": "action",
                "timestamp": 1591477068.7393892,
                "name": "utter_how_can_i_help",
                "policy": "policy_1_AugmentedMemoizationPolicy",
                "confidence": 1.0
            },
            {
                "event": "bot",
                "timestamp": 1591477068.739402,
                "text": "Hi! How can I help you??",
                "data": {
                    "elements": null,
                    "quick_replies": null,
                    "buttons": null,
                    "attachment": null,
                    "image": null,
                    "custom": null
                },
                "metadata": {}
            },

The correct intent response would be utter to the client: “Hi how can I help you”

Changing the port to send the POST to server_2 will not display the utterance back to the user despite processing the external event correctly.
http://localhost:32793/conversations/12345/trigger_intent?output_channel=socketio

core_server_2    | 2020-06-06 21:04:00 DEBUG    rasa.core.processor  - Action 'utter_how_can_i_help' ended with events '[BotUttered('Hi! How can I help you??'

This to me makes sense since the socket connection was never set with core_server_2 so I does not know where to reply back to!

I guess my question is: How should I set this up if I am running multiple core servers behind a load balancer and the POST requests to the API could be routed to a different C.S than the one the client established a connection from? Is this possible to achieve?

Regards

Setup
docker-compose:

  core_server:
    image: rasa/rasa:1.10.1-full
    ports:
      - "5005"
    command: >
      run 
      --cors '*'
      --enable-api
      --debug  
      --endpoints endpoints.yml
      --credentials credentials.yml

endpoints:

action_endpoint:
...
tracker_store:
   dialect: "postgresql"
lock_store:
   type: "redis"

@akelad, @rctatman, @Juste

Hi @danieled, your setup looks sensible, and Rasa is designed to broker messages across multiple backends. We should first confirm that your Rasa instances actually connect to the postgresql tracker store, and to the redis lock store. Would you mind sharing logs for the Rasa instances right after they start up? The command to do this should be sudo docker-compose logs a_core_server_1/2/3.

Of course @ricwo

core_server_1

2020-06-26 16:18:14 DEBUG    rasa.core.utils  - Available web server routes: 
/conversations/<conversation_id>/messages          POST                           add_message
/conversations/<conversation_id>/tracker/events    POST                           append_events
/webhooks/rest                                     GET                            custom_webhook_RestInput.health
/webhooks/rest/webhook                             POST                           custom_webhook_RestInput.receive
/model/test/intents                                POST                           evaluate_intents
/model/test/stories                                POST                           evaluate_stories
/conversations/<conversation_id>/execute           POST                           execute_action
/domain                                            GET                            get_domain
/socket.io                                         POST                           handle_request
/                                                  GET                            hello
/model                                             PUT                            load_model
/model/parse                                       POST                           parse
/conversations/<conversation_id>/predict           POST                           predict
/conversations/<conversation_id>/tracker/events    PUT                            replace_events
/conversations/<conversation_id>/story             GET                            retrieve_story
/conversations/<conversation_id>/tracker           GET                            retrieve_tracker
/webhooks/socketio                                 GET                            socketio_webhook.health
/status                                            GET                            status
/model/predict                                     POST                           tracker_predict
/model/train                                       POST                           train
/conversations/<conversation_id>/trigger_intent    POST                           trigger_intent
/model                                             DELETE                         unload_model
/version                                           GET                            version
2020-06-26 16:18:14 INFO     root  - Starting Rasa server on http://localhost:5005
2020-06-26 16:18:14 DEBUG    rasa.core.utils  - Using the default number of Sanic workers (1).
2020-06-26 16:18:14 INFO     root  - Enabling coroutine debugging. Loop id 94090200019360.
2020-06-26 16:18:15 DEBUG    rasa.model  - Extracted model to '/tmp/tmp1qpmy9rd'.
2020-06-26 16:18:43 INFO     rasa.nlu.components  - Added 'SpacyNLP' to component cache. Key 'SpacyNLP-en_core_web_md'.
/opt/venv/lib/python3.7/site-packages/sklearn/externals/joblib/__init__.py:15: FutureWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=FutureWarning)
2020-06-26 16:18:43 DEBUG    rasa.utils.tensorflow.models  - Loading the model ...
2020-06-26 16:18:43.575096: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
2020-06-26 16:18:44 DEBUG    rasa.utils.tensorflow.models  - Finished loading the model.
2020-06-26 16:18:44 DEBUG    rasa.utils.tensorflow.models  - Building tensorflow prediction graph...
2020-06-26 16:18:47 DEBUG    rasa.utils.tensorflow.models  - Finished building tensorflow prediction graph.
2020-06-26 16:18:47 DEBUG    rasa.utils.tensorflow.models  - Loading the model ...
2020-06-26 16:18:47 DEBUG    rasa.utils.tensorflow.models  - Finished loading the model.
2020-06-26 16:18:47 DEBUG    rasa.utils.tensorflow.models  - Building tensorflow prediction graph...
2020-06-26 16:18:47 DEBUG    rasa.utils.tensorflow.models  - Finished building tensorflow prediction graph.
2020-06-26 16:18:47 DEBUG    rasa.core.tracker_store  - Attempting to connect to database via 'postgresql://rasa:rasapw!@tracker_store:5432/rasa'.
2020-06-26 16:18:48 DEBUG    rasa.core.tracker_store  - Connection to SQL database 'rasa' successful.
2020-06-26 16:18:48 DEBUG    rasa.core.tracker_store  - Connected to SQLTrackerStore.
2020-06-26 16:18:48 DEBUG    rasa.core.lock_store  - Connected to lock store 'RedisLockStore'.
2020-06-26 16:18:48 DEBUG    rasa.model  - Extracted model to '/tmp/tmpe4jz4eb2'.
2020-06-26 16:18:48 DEBUG    pykwalify.compat  - Using yaml library: /opt/venv/lib/python3.7/site-packages/ruamel/yaml/__init__.py
2020-06-26 16:18:49 DEBUG    rasa.utils.tensorflow.models  - Loading the model ...
2020-06-26 16:18:50 DEBUG    rasa.utils.tensorflow.models  - Finished loading the model.
2020-06-26 16:18:50 DEBUG    rasa.utils.tensorflow.models  - Building tensorflow prediction graph...
2020-06-26 16:18:53 DEBUG    rasa.utils.tensorflow.models  - Finished building tensorflow prediction graph.
2020-06-26 16:18:53 DEBUG    rasa.core.nlg.generator  - Instantiated NLG to 'TemplatedNaturalLanguageGenerator'.

core_server_2

2020-06-26 16:18:14 DEBUG    rasa.core.utils  - Available web server routes: 
/conversations/<conversation_id>/messages          POST                           add_message
/conversations/<conversation_id>/tracker/events    POST                           append_events
/webhooks/rest                                     GET                            custom_webhook_RestInput.health
/webhooks/rest/webhook                             POST                           custom_webhook_RestInput.receive
/model/test/intents                                POST                           evaluate_intents
/model/test/stories                                POST                           evaluate_stories
/conversations/<conversation_id>/execute           POST                           execute_action
/domain                                            GET                            get_domain
/socket.io                                         POST                           handle_request
/                                                  GET                            hello
/model                                             PUT                            load_model
/model/parse                                       POST                           parse
/conversations/<conversation_id>/predict           POST                           predict
/conversations/<conversation_id>/tracker/events    PUT                            replace_events
/conversations/<conversation_id>/story             GET                            retrieve_story
/conversations/<conversation_id>/tracker           GET                            retrieve_tracker
/webhooks/socketio                                 GET                            socketio_webhook.health
/status                                            GET                            status
/model/predict                                     POST                           tracker_predict
/model/train                                       POST                           train
/conversations/<conversation_id>/trigger_intent    POST                           trigger_intent
/model                                             DELETE                         unload_model
/version                                           GET                            version
2020-06-26 16:18:14 INFO     root  - Starting Rasa server on http://localhost:5005
2020-06-26 16:18:14 DEBUG    rasa.core.utils  - Using the default number of Sanic workers (1).
2020-06-26 16:18:14 INFO     root  - Enabling coroutine debugging. Loop id 94549718162304.
2020-06-26 16:18:15 DEBUG    rasa.model  - Extracted model to '/tmp/tmpwttpq__v'.
2020-06-26 16:18:42 INFO     rasa.nlu.components  - Added 'SpacyNLP' to component cache. Key 'SpacyNLP-en_core_web_md'.
/opt/venv/lib/python3.7/site-packages/sklearn/externals/joblib/__init__.py:15: FutureWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=FutureWarning)
2020-06-26 16:18:42 DEBUG    rasa.utils.tensorflow.models  - Loading the model ...
2020-06-26 16:18:42.886643: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
2020-06-26 16:18:43 DEBUG    rasa.utils.tensorflow.models  - Finished loading the model.
2020-06-26 16:18:43 DEBUG    rasa.utils.tensorflow.models  - Building tensorflow prediction graph...
2020-06-26 16:18:46 DEBUG    rasa.utils.tensorflow.models  - Finished building tensorflow prediction graph.
2020-06-26 16:18:46 DEBUG    rasa.utils.tensorflow.models  - Loading the model ...
2020-06-26 16:18:47 DEBUG    rasa.utils.tensorflow.models  - Finished loading the model.
2020-06-26 16:18:47 DEBUG    rasa.utils.tensorflow.models  - Building tensorflow prediction graph...
2020-06-26 16:18:47 DEBUG    rasa.utils.tensorflow.models  - Finished building tensorflow prediction graph.
2020-06-26 16:18:47 DEBUG    rasa.core.tracker_store  - Attempting to connect to database via 'postgresql://rasa:rasapw!@tracker_store:5432/rasa'.
2020-06-26 16:18:47 DEBUG    rasa.core.tracker_store  - Connection to SQL database 'rasa' successful.
2020-06-26 16:18:47 DEBUG    rasa.core.tracker_store  - Connected to SQLTrackerStore.
2020-06-26 16:18:47 DEBUG    rasa.core.lock_store  - Connected to lock store 'RedisLockStore'.
2020-06-26 16:18:48 DEBUG    rasa.model  - Extracted model to '/tmp/tmp3vjspaoy'.
2020-06-26 16:18:48 DEBUG    pykwalify.compat  - Using yaml library: /opt/venv/lib/python3.7/site-packages/ruamel/yaml/__init__.py
2020-06-26 16:18:49 DEBUG    rasa.utils.tensorflow.models  - Loading the model ...
2020-06-26 16:18:49 DEBUG    rasa.utils.tensorflow.models  - Finished loading the model.
2020-06-26 16:18:49 DEBUG    rasa.utils.tensorflow.models  - Building tensorflow prediction graph...
2020-06-26 16:18:52 DEBUG    rasa.utils.tensorflow.models  - Finished building tensorflow prediction graph.
2020-06-26 16:18:52 DEBUG    rasa.core.nlg.generator  - Instantiated NLG to 'TemplatedNaturalLanguageGenerator'.

core_server_3

2020-06-26 16:18:14 DEBUG    rasa.core.utils  - Available web server routes: 
/conversations/<conversation_id>/messages          POST                           add_message
/conversations/<conversation_id>/tracker/events    POST                           append_events
/webhooks/rest                                     GET                            custom_webhook_RestInput.health
/webhooks/rest/webhook                             POST                           custom_webhook_RestInput.receive
/model/test/intents                                POST                           evaluate_intents
/model/test/stories                                POST                           evaluate_stories
/conversations/<conversation_id>/execute           POST                           execute_action
/domain                                            GET                            get_domain
/socket.io                                         GET                            handle_request
/                                                  GET                            hello
/model                                             PUT                            load_model
/model/parse                                       POST                           parse
/conversations/<conversation_id>/predict           POST                           predict
/conversations/<conversation_id>/tracker/events    PUT                            replace_events
/conversations/<conversation_id>/story             GET                            retrieve_story
/conversations/<conversation_id>/tracker           GET                            retrieve_tracker
/webhooks/socketio                                 GET                            socketio_webhook.health
/status                                            GET                            status
/model/predict                                     POST                           tracker_predict
/model/train                                       POST                           train
/conversations/<conversation_id>/trigger_intent    POST                           trigger_intent
/model                                             DELETE                         unload_model
/version                                           GET                            version
2020-06-26 16:18:14 INFO     root  - Starting Rasa server on http://localhost:5005
2020-06-26 16:18:14 DEBUG    rasa.core.utils  - Using the default number of Sanic workers (1).
2020-06-26 16:18:14 INFO     root  - Enabling coroutine debugging. Loop id 94012376344224.
2020-06-26 16:18:15 DEBUG    rasa.model  - Extracted model to '/tmp/tmp5t5ihj8u'.
2020-06-26 16:18:42 INFO     rasa.nlu.components  - Added 'SpacyNLP' to component cache. Key 'SpacyNLP-en_core_web_md'.
/opt/venv/lib/python3.7/site-packages/sklearn/externals/joblib/__init__.py:15: FutureWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=FutureWarning)
2020-06-26 16:18:42 DEBUG    rasa.utils.tensorflow.models  - Loading the model ...
2020-06-26 16:18:42.920947: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
2020-06-26 16:18:43 DEBUG    rasa.utils.tensorflow.models  - Finished loading the model.
2020-06-26 16:18:43 DEBUG    rasa.utils.tensorflow.models  - Building tensorflow prediction graph...
2020-06-26 16:18:46 DEBUG    rasa.utils.tensorflow.models  - Finished building tensorflow prediction graph.
2020-06-26 16:18:46 DEBUG    rasa.utils.tensorflow.models  - Loading the model ...
2020-06-26 16:18:47 DEBUG    rasa.utils.tensorflow.models  - Finished loading the model.
2020-06-26 16:18:47 DEBUG    rasa.utils.tensorflow.models  - Building tensorflow prediction graph...
2020-06-26 16:18:47 DEBUG    rasa.utils.tensorflow.models  - Finished building tensorflow prediction graph.
2020-06-26 16:18:47 DEBUG    rasa.core.tracker_store  - Attempting to connect to database via 'postgresql://rasa:rasapw!@tracker_store:5432/rasa'.
2020-06-26 16:18:47 DEBUG    rasa.core.tracker_store  - Connection to SQL database 'rasa' successful.
2020-06-26 16:18:47 DEBUG    rasa.core.tracker_store  - Connected to SQLTrackerStore.
2020-06-26 16:18:47 DEBUG    rasa.core.lock_store  - Connected to lock store 'RedisLockStore'.
2020-06-26 16:18:48 DEBUG    rasa.model  - Extracted model to '/tmp/tmp0z1h1340'.
2020-06-26 16:18:48 DEBUG    pykwalify.compat  - Using yaml library: /opt/venv/lib/python3.7/site-packages/ruamel/yaml/__init__.py
2020-06-26 16:18:49 DEBUG    rasa.utils.tensorflow.models  - Loading the model ...
2020-06-26 16:18:49 DEBUG    rasa.utils.tensorflow.models  - Finished loading the model.
2020-06-26 16:18:49 DEBUG    rasa.utils.tensorflow.models  - Building tensorflow prediction graph...
2020-06-26 16:18:53 DEBUG    rasa.utils.tensorflow.models  - Finished building tensorflow prediction graph.
2020-06-26 16:18:53 DEBUG    rasa.core.nlg.generator  - Instantiated NLG to 'TemplatedNaturalLanguageGenerator'.

Thanks for sharing those logs. There is currently no way to broker a single socket connection across multiple backends.

One solution in your case might be to configure sticky sessions. What type of load balancer are you using? This links describes how to configure sticky sessions using traefik: Sticky Session With Docker Swarm (CE) on CentOS 7 - Vultr.com

@ricwo,

Thank you, that confirms the P.O.C.

The Core Servers sit behind a layer 7 (application) load balancer with sticky sessions enabled. I will work on a solution that forwards the cookies across the requests to the external client and hope they are routed correctly through the load balancer.

Do you know if something like this has been accomplished before?

Regards, Daniel

Not that I know of. I’d be curious to hear how it goes!