Rasa X and Rasa same server deployment with Twilio

[Deprecated] Rasa X Community Edition
1

  1. I have a Rasa Open Source docker-compose deployment on the same server as a Rasa X docker-compose deployment. Both are on a Google Cloud vm.

  2. I would like to also connect Twilio with Rasa X / Rasa Open Source. I am assuming giving the Rasa Open Source bot the Twilio action endpoint and then connecting it to a Production worker container in Rasa X would be able to connect them all.

  3. I have followed the instructions here in order to try to connect them by RabbitMQ on the same machine: Connect a Rasa Deployment

However, logs from my Rasa Open Source bot gets the following errors after adding the Rabbit information in the endpoints.yml for the docker-compose Rasa Open Source bot:

rasa_1           | 2020-03-18 23:35:34 ERROR    pika.adapters.utils.selector_ioloop_adapter  - Address resolution failed: gaierror(-2, 'Name or service not known')
rasa_1           | 2020-03-18 23:35:34 ERROR    pika.adapters.utils.connection_workflow  - getaddrinfo failed: gaierror(-2, 'Name or service not known').
rasa_1           | 2020-03-18 23:35:34 ERROR    pika.adapters.utils.connection_workflow  - AMQP connection workflow failed: AMQPConnectionWorkflowFailed: 20 exceptions in all; last exception - gaierror(-2, 'Name or service not known'); first exception - gaierror(-2, 'Name or service not known').
rasa_1           | 2020-03-18 23:35:34 ERROR    pika.adapters.utils.connection_workflow  - AMQPConnectionWorkflow - reporting failure: AMQPConnectionWorkflowFailed: 20 exceptions in all; last exception - gaierror(-2, 'Name or service not known'); first exception - gaierror(-2, 'Name or service not known')
rasa_1           | 2020-03-18 23:35:34 ERROR    pika.adapters.base_connection  - Full-stack connection workflow failed: AMQPConnectionWorkflowFailed: 20 exceptions in all; last exception - gaierror(-2, 'Name or service not known'); first exception - gaierror(-2, 'Name or service not known')
rasa_1           | 2020-03-18 23:35:34 ERROR    pika.adapters.base_connection  - Self-initiated stack bring-up failed: AMQPConnectionWorkflowFailed: 20 exceptions in all; last exception - gaierror(-2, 'Name or service not known'); first exception - gaierror(-2, 'Name or service not known')
rasa_1           | 2020-03-18 23:35:34 WARNING  rasa.core.brokers.pika  - Connecting to 'http://localhost:5002/' failed with error ''. Trying again.

My Rasa Core bot’s endpoints.yml is the following:

action_endpoint:
  url: "http://localhost:5100/webhooks/twilio/webhook"

event_broker:
  type: "pika"
  url: "http://localhost:5002/"
  username: "user"
  password: "XXXXXXXXREDACTEDZZZZZZZZ"
  queue: "rasa_production_events"

I receive these logs on the Rasa X side:

nginx_1            | 34.204.12.72 - - [18/Mar/2020:23:56:02 +0000] "POST /:5100/webhooks/twilio/webhook HTTP/1.1" 200 3173 "-" "TwilioProxy/1.1"
rasa-production_1  | 2020-03-18 23:56:02 DEBUG    rasa.core.agent  - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...
rasa-worker_1      | 2020-03-18 23:56:02 DEBUG    rasa.core.agent  - Requesting model from server http://rasa-x:5002/api/projects/default/models/tags/production...
rasa-production_1  | 2020-03-18 23:56:02 DEBUG    rasa.core.agent  - Model server returned 204 status code, indicating that no new model is available. Current fingerprint: f458c849c770d6986a53c8f5d9d60b4a

I opened the port in Rasa X docker-compose.override.yml:

services:
  rabbit:
    ports:
      - "5672:5672"
  app:
    restart: always
    image: "rasa/rasa-sdk:latest"
    volumes:
      - 'XXXXXXXX'
    expose:
      - "5055"
      - "5100"
    depends_on:
      - rasa-production
  1. Here is my Rasa Open Source docker file - do I need to add any options in order to allow it to recognize incoming twilio calls?
version: '3.4'

services:
  rasa:
    image: rasa/rasa:latest-full
    networks: ['rasa-network']
    ports:
    - "5005:5005"
    volumes:
    - ./:/app
    command:
    - run

  action_server:
    image: rasa/rasa-sdk:latest
    networks: ['rasa-network']
    ports:
    - "5055:5055"
    volumes:
    - ./actions:/app/actions

  duckling:
    image: rasa/duckling:latest
    networks: ['rasa-network']
    ports:
    - "8000:8000"

Can anyone assist me in getting my Rasa Open Source Bot working with my Rasa X server and Twilio?

2

Rasa X includes Rasa Open Source so this means you have duplicate Rasa OSS installs. I assume you have installed them with separate $RASA_HOME directories. If not, you should remove everything and re-start, only doing the Rasa X install.

If you have installed them with separate RASA_HOME directories. Shutdown and remove the Rasa OSS install and start only the Rasa X docker-compose setup.

With this configuration, step #3 is not necessary.

Greg

3

Hi @stephens , thanks for your response. I believe I have Rasa X correctly installed. It is working correctly and I can interact with it via my browser. When I installed Rasa X, I followed all instructions in the docs Docker-Compose Quick Install

However, what is the best way to connect a Twilio bot to Rasa X?Thanks again for your help.

4

In my credentials.yml, I have correctly added my twilio credentials. How would the endpoint look? I assume I need to change the action_endpoint. My current Rasa X endpoints.yml is:

models:
  url: ${RASA_MODEL_SERVER}
  token: ${RASA_X_TOKEN}
  wait_time_between_pulls: ${RASA_MODEL_PULL_INTERVAL}
tracker_store:
  type: sql
  dialect: "postgresql"
  url: ${DB_HOST}
  port: ${DB_PORT}
  username: ${DB_USER}
  password: ${DB_PASSWORD}
  db: ${DB_DATABASE}
  login_db: ${DB_LOGIN_DB}
lock_store:
  type: "redis"
  url: ${REDIS_HOST}
  port: ${REDIS_PORT}
  password: ${REDIS_PASSWORD}
  db: ${REDIS_DB}
event_broker:
  type: "pika"
  url: ${RABBITMQ_HOST}
  username: ${RABBITMQ_USERNAME}
  password: ${RABBITMQ_PASSWORD}
  queue: ${RABBITMQ_QUEUE}
action_endpoint:
  url: ${RASA_USER_APP}/webhook
  token:  ""
#action_endpoint:
#  url: "http://localhost:5100/webhooks/twilio/webhook"
5

The endpoints.yml is not involved in the channel configuration. The credentials.yml should be setup as shown on the Twilio connector page with the values from your Twilio account.

You also need to setup the Twilio configuration to point to the Rasa Twilio channel webhook URL. That endpoint looks like this (but you need to supply your host and port):

http://<host>:<port>/webhooks/twilio/webhook

Greg

6

Oh, for Rasa Open Source, you used to have to change the action_endpoints url to something like:

action_endpoint:`
  url: "http://localhost:5100/webhooks/twilio/webhook"

I didn’t know if things were different for Rasa X. Per your suggestion, I have left my endpoints.yml as it is upon Rasa X install. For Rasa X, I added my credentials and have the twilio hook in. I consistently receive a response in my nginx container:

nginx_1            | 52342342349 - - [20/Mar/2020:03:10:29 +0000] "POST /:5100/webhooks/twilio/webhook HTTP/1.1" 200 3173 "-" "TwilioProxy/1.1"

However, my bot does not respond to the message and Twilio is saying the response is malformed. From Twilio:

58%20PM
58%20PM818Ă—864 62.2 KB
06%20PM
06%20PM1544Ă—522 24.8 KB
This is the body:

<!doctype html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><link rel="shortcut icon" href="/icons/favicon.ico"><link rel="apple-touch-icon" href="/icons/apple-touch-icon-57x57.png"/><link rel="apple-touch-icon" sizes="72x72" href="/icons/apple-touch-icon-72x72.png"/><link rel="apple-touch-icon" sizes="114x114" href="/icons/apple-touch-icon-114x114.png"/><link rel="apple-touch-icon" sizes="144x144" href="/icons/apple-touch-icon-144x144.png"/><script>try{window.SERVER_DATA=__SERVER_DATA__}catch(_){}</script><title>Rasa X</title><link href="/static/css/2.20a0f44c.chunk.css" rel="stylesheet"></head><body><div id="root"></div><script>!function(c){function e(e){for(var r,t,n=e[0],o=e[1],a=e[2],u=0,i=[];u<n.length;u++)t=n[u],Object.prototype.hasOwnProperty.call(l,t)&&l[t]&&i.push(l[t][0]),l[t]=0;for(r in o)Object.prototype.hasOwnProperty.call(o,r)&&(c[r]=o[r]);for(d&&d(e);i.length;)i.shift()();return s.push.apply(s,a||[]),f()}function f(){for(var e,r=0;r<s.length;r++){for(var t=s[r],n=!0,o=1;o<t.length;o++){var a=t[o];0!==l[a]&&(n=!1)}n&&(s.splice(r--,1),e=p(p.s=t[0]))}return e}var t={},l={1:0},s=[];function p(e){if(t[e])return t[e].exports;var r=t[e]={i:e,l:!1,exports:{}};return c[e].call(r.exports,r,r.exports,p),r.l=!0,r.exports}p.e=function(o){var e=[],t=l[o];if(0!==t)if(t)e.push(t[2]);else{var r=new Promise(function(e,r){t=l[o]=[e,r]});e.push(t[2]=r);var n,a=document.createElement("script");a.charset="utf-8",a.timeout=120,p.nc&&a.setAttribute("nonce",p.nc),a.src=p.p+"static/js/"+({}[o]||o)+"."+{3:"1e3e9bd1",4:"44c98c82",5:"e2dd33fa",6:"37c0f0e2"}[o]+".chunk.js";var u=new Error;n=function(e){a.onerror=a.onload=null,clearTimeout(i);var r=l[o];if(0!==r){if(r){var t=e&&("load"===e.type?"missing":e.type),n=e&&e.target&&e.target.src;u.message="Loading chunk "+o+" failed.\n("+t+": "+n+")",u.name="ChunkLoadError",u.type=t,u.request=n,r[1](u)}l[o]=void 0}};var i=setTimeout(function(){n({type:"timeout",target:a})},12e4);a.onerror=a.onload=n,document.head.appendChild(a)}return Promise.all(e)},p.m=c,p.c=t,p.d=function(e,r,t){p.o(e,r)||Object.defineProperty(e,r,{enumerable:!0,get:t})},p.r=function(e){"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},p.t=function(r,e){if(1&e&&(r=p(r)),8&e)return r;if(4&e&&"object"==typeof r&&r&&r.__esModule)return r;var t=Object.create(null);if(p.r(t),Object.defineProperty(t,"default",{enumerable:!0,value:r}),2&e&&"string"!=typeof r)for(var n in r)p.d(t,n,function(e){return r[e]}.bind(null,n));return t},p.n=function(e){var r=e&&e.__esModule?function(){return e.default}:function(){return e};return p.d(r,"a",r),r},p.o=function(e,r){return Object.prototype.hasOwnProperty.call(e,r)},p.p="/",p.oe=function(e){throw console.error(e),e};var r=this["webpackJsonprasa-interface"]=this["webpackJsonprasa-interface"]||[],n=r.push.bind(r);r.push=e,r=r.slice();for(var o=0;o<r.length;o++)e(r[o]);var d=n;f()}([])</script><script src="/static/js/2.2703e766.chunk.js"></script><script src="/static/js/main.8f2e2e65.chunk.js"></script></body></html>

I don’t know what I need to do to get this to hit my action server correctly be stored as a conversation in rasa x.

Is there an option in my docker-compose.override.yml or something that I need to add to get this to work, or is this behavior not expected?

7

@argideritzalpea no, you don’t need to change your action_endpoint for Rasa Open Source or for Rasa X when using twilio.

Also, the correct url would be /core/webhooks/twilio/webhook - can you try querying that?

8

Hi @akelad

I changed Twilio’s connection url to: http://[ip_address]/:5100/core/webhooks/twilio/webhook

I still don’t get any action from the incoming Twilio message.

Here are my logs:

db_1               | Welcome to the Bitnami postgresql container
db_1               | Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql
db_1               | Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql/issues
db_1               | Send us your feedback at containers@bitnami.com
db_1               | 
db_1               | INFO  ==> ** Starting PostgreSQL setup **
db_1               | INFO  ==> Validating settings in POSTGRESQL_* env vars..
db_1               | INFO  ==> Initializing PostgreSQL database...
db_1               | INFO  ==> postgresql.conf file not detected. Generating it...
db_1               | INFO  ==> pg_hba.conf file not detected. Generating it...
db_1               | INFO  ==> Deploying PostgreSQL with persisted data...
db_1               | INFO  ==> Configuring replication parameters
db_1               | INFO  ==> Configuring fsync
db_1               | INFO  ==> Loading custom scripts...
db_1               | INFO  ==> Enabling remote connections
db_1               | INFO  ==> Stopping PostgreSQL...
db_1               | INFO  ==> ** PostgreSQL setup finished! **
db_1               | 
db_1               | INFO  ==> ** Starting PostgreSQL **
db_1               | 2020-03-20 15:49:55.795 GMT [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
duckling_1         | no port specified, defaulting to port 8000
duckling_1         | Listening on http://0.0.0.0:8000
db_1               | 2020-03-20 15:49:55.795 GMT [1] LOG:  listening on IPv6 address "::", port 5432
db_1               | 2020-03-20 15:49:55.800 GMT [1] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
db_1               | 2020-03-20 15:49:55.827 GMT [150] LOG:  database system was shut down at 2020-03-20 15:49:48 GMT
db_1               | 2020-03-20 15:49:55.833 GMT [1] LOG:  database system is ready to accept connections
rabbit_1           |  15:49:56.93 
rabbit_1           |  15:49:56.96 Welcome to the Bitnami rabbitmq container
rabbit_1           |  15:49:56.97 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-rabbitmq
rabbit_1           |  15:49:56.98 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-rabbitmq/issues
rabbit_1           |  15:49:56.99 Send us your feedback at containers@bitnami.com
rabbit_1           |  15:49:57.01 
rabbit_1           |  15:49:57.03 INFO  ==> ** Starting RabbitMQ setup **
rabbit_1           |  15:49:57.11 INFO  ==> Validating settings in RABBITMQ_* env vars..
rabbit_1           |  15:49:57.19 INFO  ==> Initializing RabbitMQ...
rabbit_1           |  15:49:57.23 INFO  ==> Generating random cookie
rabbit_1           |  15:49:57.36 INFO  ==> Starting RabbitMQ in background...
redis_1            |  15:49:56.79 
redis_1            |  15:49:56.80 Welcome to the Bitnami redis container
redis_1            |  15:49:56.80 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-redis
redis_1            |  15:49:56.80 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-redis/issues
redis_1            |  15:49:56.81 Send us your feedback at containers@bitnami.com
redis_1            |  15:49:56.82 
redis_1            |  15:49:56.82 INFO  ==> ** Starting Redis setup **
redis_1            |  15:49:56.88 INFO  ==> Initializing Redis...
redis_1            |  15:49:57.31 INFO  ==> ** Redis setup finished! **
redis_1            |  15:49:57.38 INFO  ==> ** Starting Redis **
redis_1            | 
redis_1            | 1:C 20 Mar 2020 15:49:57.402 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1            | 1:C 20 Mar 2020 15:49:57.423 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1            | 1:C 20 Mar 2020 15:49:57.423 # Configuration loaded
redis_1            |                 _._                                                  
redis_1            |            _.-``__ ''-._                                             
redis_1            |       _.-``    `.  `_.  ''-._           Redis 5.0.5 (00000000/0) 64 bit
redis_1            |   .-`` .-```.  ```\/    _.,_ ''-._                                   
redis_1            |  (    '      ,       .-`  | `,    )     Running in standalone mode
redis_1            |  |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
redis_1            |  |    `-._   `._    /     _.-'    |     PID: 1
redis_1            |   `-._    `-._  `-./  _.-'    _.-'                                   
redis_1            |  |`-._`-._    `-.__.-'    _.-'_.-'|                                  
redis_1            |  |    `-._`-._        _.-'_.-'    |           http://redis.io        
redis_1            |   `-._    `-._`-.__.-'_.-'    _.-'                                   
redis_1            |  |`-._`-._    `-.__.-'    _.-'_.-'|                                  
redis_1            |  |    `-._`-._        _.-'_.-'    |                                  
redis_1            |   `-._    `-._`-.__.-'_.-'    _.-'                                   
redis_1            |       `-._    `-.__.-'    _.-'                                       
redis_1            |           `-._        _.-'                                           
redis_1            |               `-.__.-'                                               
redis_1            | 
redis_1            | 1:M 20 Mar 2020 15:49:57.425 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1            | 1:M 20 Mar 2020 15:49:57.425 # Server initialized
redis_1            | 1:M 20 Mar 2020 15:49:57.425 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1            | 1:M 20 Mar 2020 15:49:57.425 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1            | 1:M 20 Mar 2020 15:49:57.425 * Ready to accept connections
nginx_1            | SSL encryption is not used since no certificates were provided.
nginx_1            |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
nginx_1            |                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to app port 80: Connection refused
nginx_1            | 7
nginx_1            | >> exec docker CMD
nginx_1            | nginx
nginx_1            | 2020/03/20 15:50:02 [notice] 1#0: using the "epoll" event method
nginx_1            | 2020/03/20 15:50:02 [notice] 1#0: nginx/1.14.2
nginx_1            | 2020/03/20 15:50:02 [notice] 1#0: built by gcc 6.3.0 20170516 (Debian 6.3.0-18+deb9u1) 
nginx_1            | 2020/03/20 15:50:02 [notice] 1#0: OS: Linux 4.9.0-12-amd64
nginx_1            | 2020/03/20 15:50:02 [notice] 1#0: getrlimit(RLIMIT_NOFILE): 1048576:1048576
nginx_1            | 2020/03/20 15:50:02 [notice] 1#0: start worker processes
nginx_1            | 2020/03/20 15:50:02 [notice] 1#0: start worker process 12
nginx_1            | 2020/03/20 15:50:04 [error] 12#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 66.170.190.22, server: , request: "GET /api/conversations/ba4d354bc9194935ba4b888211664029?since=0&environment=production HTTP/1.1", upstream: "http://172.23.0.6:5002/api/conversations/ba4d354bc9194935ba4b888211664029?since=0&environment=production", host: "XXXXXXX", referrer: "http://XXXXXXXX/interactive"
nginx_1            | 66.170.190.22 - - [20/Mar/2020:15:50:04 +0000] "GET /api/conversations/ba4d354bc9194935ba4b888211664029?since=0&environment=production HTTP/1.1" 502 575 "http://XXXXXX/interactive" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
app_1              | 2020-03-20 15:50:06 INFO     rasa_sdk.endpoint  - Starting action endpoint server...
app_1              | 2020-03-20 15:50:06 INFO     rasa_sdk.executor  - Registered function for 'action_inform_peopleCount'.
app_1              | 2020-03-20 15:50:06 INFO     rasa_sdk.executor  - Registered function for 'action_inform_vehicleCount'.
app_1              | 2020-03-20 15:50:06 INFO     rasa_sdk.executor  - Registered function for 'action_inform_partyVehics'.
app_1              | 2020-03-20 15:50:06 INFO     rasa_sdk.executor  - Registered function for 'utter_goodbye'.
app_1              | 2020-03-20 15:50:06 INFO     rasa_sdk.executor  - Registered function for 'action_inform_countTime'.
app_1              | 2020-03-20 15:50:06 INFO     rasa_sdk.executor  - Registered function for 'action_inform_source'.
app_1              | 2020-03-20 15:50:06 INFO     rasa_sdk.executor  - Registered function for 'action_inform_returnStatus'.
app_1              | 2020-03-20 15:50:06 INFO     rasa_sdk.executor  - Registered function for 'action_inform_trailVisits'.
app_1              | 2020-03-20 15:50:06 INFO     rasa_sdk.executor  - Registered function for 'action_inform_yearBorn'.
app_1              | 2020-03-20 15:50:06 INFO     rasa_sdk.executor  - Registered function for 'action_inform_howLong'.
app_1              | 2020-03-20 15:50:06 INFO     rasa_sdk.executor  - Registered function for 'action_inform_zipCode'.
app_1              | 2020-03-20 15:50:06 INFO     rasa_sdk.executor  - Registered function for 'action_inform_seenBefore'.
app_1              | 2020-03-20 15:50:06 INFO     rasa_sdk.executor  - Registered function for 'action_inform_seenPastDay'.
app_1              | 2020-03-20 15:50:06 INFO     rasa_sdk.executor  - Registered function for 'action_inform_ambigNumber'.
app_1              | 2020-03-20 15:50:06 INFO     rasa_sdk.executor  - Registered function for 'action_inform_ambigVehic'.
app_1              | 2020-03-20 15:50:06 INFO     rasa_sdk.executor  - Registered function for 'action_setDefaultSlotValues'.
app_1              | 2020-03-20 15:50:06 INFO     rasa_sdk.executor  - Registered function for 'action_default_fallback'.
app_1              | 2020-03-20 15:50:06 INFO     rasa_sdk.executor  - Registered function for 'feedback_form'.
app_1              | 2020-03-20 15:50:06 INFO     rasa_sdk.executor  - Registered function for 'count_episode'.
app_1              | 2020-03-20 15:50:06 INFO     rasa_sdk.executor  - Registered function for 'personal_form'.
rasa-x_1           | INFO:rasax.community.services.event_service:Waiting until database migrations have been executed...
nginx_1            | 2020/03/20 15:50:08 [error] 12#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 66.170.190.22, server: , request: "GET /api/conversations/ba4d354bc9194935ba4b888211664029?since=0&environment=production HTTP/1.1", upstream: "http://172.23.0.6:5002/api/conversations/ba4d354bc9194935ba4b888211664029?since=0&environment=production", host: "XXXXXXXX", referrer: "http://XXXXXXXX/interactive"
nginx_1            | 66.170.190.22 - - [20/Mar/2020:15:50:08 +0000] "GET /api/conversations/ba4d354bc9194935ba4b888211664029?since=0&environment=production HTTP/1.1" 502 575 "http://XXXXXXXX/interactive" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
rasa-x_1           | INFO:alembic.runtime.migration:Context impl PostgresqlImpl.
rasa-x_1           | INFO:alembic.runtime.migration:Will assume transactional DDL.
rasa-x_1           | INFO:rasax.community.services.event_service:Check for database migrations completed.
rasa-x_1           | ERROR:pika.adapters.utils.io_services_utils:Socket failed to connect: <socket.socket fd=18, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.23.0.6', 51424)>; error=111 (Connection refused)
rasa-x_1           | ERROR:pika.adapters.utils.connection_workflow:TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.23.0.2', 5672))
rasa-x_1           | ERROR:pika.adapters.utils.connection_workflow:AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rasa-x_1           | INFO:alembic.runtime.migration:Context impl PostgresqlImpl.
rasa-x_1           | INFO:alembic.runtime.migration:Will assume transactional DDL.
rasa-x_1           | INFO:rasax.community.services.integrated_version_control.git_service:Cloning git repository from URL 'git@bitbucket.org:outdoorrd/crowdy.git'.
nginx_1            | 2020/03/20 15:50:12 [error] 12#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 66.170.190.22, server: , request: "GET /api/conversations/ba4d354bc9194935ba4b888211664029?since=0&environment=production HTTP/1.1", upstream: "http://172.23.0.6:5002/api/conversations/ba4d354bc9194935ba4b888211664029?since=0&environment=production", host: "XXXXXXXX", referrer: "http://XXXXXXXX/interactive"
nginx_1            | 66.170.190.22 - - [20/Mar/2020:15:50:12 +0000] "GET /api/conversations/ba4d354bc9194935ba4b888211664029?since=0&environment=production HTTP/1.1" 502 575 "http://XXXXXXXXX/interactive" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
rasa-x_1           | You are using an outdated version of Rasa X (0.26.0).
rasa-x_1           | A new version 0.26.1) is available.
rasa-x_1           | Please follow these instructions to update:
rasa-x_1           | https://rasa.com/docs/rasa-x/installation-and-setup/updating
rasa-x_1           | 
rasa-x_1           | Starting Rasa X server... 🚀
nginx_1            | 2020/03/20 15:50:13 [error] 12#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 66.170.190.22, server: , request: "GET /api/projects/default/git_repositories/1/status HTTP/1.1", upstream: "http://172.23.0.6:5002/api/projects/default/git_repositories/1/status", host: "XXXXXXX", referrer: "http://XXXXXXXXX/interactive"
nginx_1            | 66.170.190.22 - - [20/Mar/2020:15:50:13 +0000] "GET /api/projects/default/git_repositories/1/status HTTP/1.1" 502 575 "http://XXXXXXXXX/interactive" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
rasa-x_1           | ERROR:pika.adapters.utils.io_services_utils:Socket failed to connect: <socket.socket fd=18, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.23.0.6', 51450)>; error=111 (Connection refused)
rasa-x_1           | ERROR:pika.adapters.utils.connection_workflow:TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.23.0.2', 5672))
rasa-x_1           | ERROR:pika.adapters.utils.connection_workflow:AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rasa-production_1  | 2020-03-20 15:50:15.596463: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
rasa-worker_1      | 2020-03-20 15:50:15.601240: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
rasa-production_1  | 2020-03-20 15:50:15.606189: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
rasa-production_1  | 2020-03-20 15:50:15.608389: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
rasa-worker_1      | 2020-03-20 15:50:15.613701: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
rasa-worker_1      | 2020-03-20 15:50:15.614378: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
nginx_1            | 66.170.190.22 - - [20/Mar/2020:15:50:16 +0000] "GET /api/conversations/ba4d354bc9194935ba4b888211664029?since=0&environment=production HTTP/1.1" 401 57 "http://XXXXXXXX/interactive" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
rabbit_1           | Waiting for erlang distribution on node 'rabbit@localhost' while OS process '43' is running
rasa-x_1           | ERROR:pika.adapters.utils.io_services_utils:Socket failed to connect: <socket.socket fd=18, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.23.0.6', 51462)>; error=111 (Connection refused)
rasa-x_1           | ERROR:pika.adapters.utils.connection_workflow:TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.23.0.2', 5672))
rasa-x_1           | ERROR:pika.adapters.utils.connection_workflow:AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rasa-x_1           | Job "GitService.run_background_synchronization (trigger: cron[minute='*'], next run at: 2020-03-20 15:51:00 UTC)" raised an exception
rasa-x_1           | Traceback (most recent call last):
rasa-x_1           |   File "/usr/local/lib/python3.6/site-packages/apscheduler/executors/base.py", line 125, in run_job
rasa-x_1           |     retval = job.func(*job.args, **job.kwargs)
rasa-x_1           |   File "/usr/local/lib/python3.6/site-packages/rasax/community/services/integrated_version_control/git_service.py", line 826, in run_background_synchronization
rasa-x_1           |     git_service.synchronize_project(force_data_injection)
rasa-x_1           |   File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
rasa-x_1           |   File "/usr/local/lib/python3.6/site-packages/rasax/community/services/integrated_version_control/git_service.py", line 675, in synchronize_project
rasa-x_1           |     await self._inject_data()
rasa-x_1           |   File "/usr/local/lib/python3.6/site-packages/rasax/community/services/integrated_version_control/git_service.py", line 710, in _inject_data
rasa-x_1           |     str(self.repository_path()), str(data_path), self.session, SYSTEM_USER
rasa-x_1           |   File "/usr/local/lib/python3.6/site-packages/rasax/community/initialise.py", line 293, in inject_files_from_disk
rasa-x_1           |     username,
rasa-x_1           |   File "/usr/local/lib/python3.6/site-packages/rasax/community/initialise.py", line 129, in inject_domain
rasa-x_1           |     rasa_cli_utils.error_and_exit(
rasa-x_1           | AttributeError: module 'rasa.cli.utils' has no attribute 'error_and_exit'
rabbit_1           | Error: operation wait on node rabbit@localhost timed out. Timeout value used: 5000
rasa-production_1  | 2020-03-20 15:50:23 ERROR    pika.adapters.utils.io_services_utils  - Socket failed to connect: <socket.socket fd=21, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.23.0.8', 49322)>; error=111 (Connection refused)
rasa-production_1  | 2020-03-20 15:50:23 ERROR    pika.adapters.utils.connection_workflow  - TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.23.0.2', 5672))
rasa-worker_1      | 2020-03-20 15:50:23 ERROR    pika.adapters.utils.io_services_utils  - Socket failed to connect: <socket.socket fd=21, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.23.0.7', 52742)>; error=111 (Connection refused)
rasa-production_1  | 2020-03-20 15:50:23 ERROR    pika.adapters.utils.connection_workflow  - AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rasa-worker_1      | 2020-03-20 15:50:23 ERROR    pika.adapters.utils.connection_workflow  - TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.23.0.2', 5672))
rasa-worker_1      | 2020-03-20 15:50:23 ERROR    pika.adapters.utils.connection_workflow  - AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rasa-x_1           | ERROR:pika.adapters.utils.io_services_utils:Socket failed to connect: <socket.socket fd=18, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.23.0.6', 51490)>; error=111 (Connection refused)
rasa-x_1           | ERROR:pika.adapters.utils.connection_workflow:TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.23.0.2', 5672))
rasa-x_1           | ERROR:pika.adapters.utils.connection_workflow:AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
db_1               | 2020-03-20 15:50:23.898 GMT [165] WARNING:  there is no transaction in progress
db_1               | 2020-03-20 15:50:23.918 GMT [166] WARNING:  there is no transaction in progress
rasa-production_1  | 2020-03-20 15:50:28 ERROR    pika.adapters.utils.io_services_utils  - Socket failed to connect: <socket.socket fd=27, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.23.0.8', 49344)>; error=111 (Connection refused)
rasa-production_1  | 2020-03-20 15:50:28 ERROR    pika.adapters.utils.connection_workflow  - TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.23.0.2', 5672))
rasa-production_1  | 2020-03-20 15:50:28 ERROR    pika.adapters.utils.connection_workflow  - AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rasa-worker_1      | 2020-03-20 15:50:28 ERROR    pika.adapters.utils.io_services_utils  - Socket failed to connect: <socket.socket fd=25, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.23.0.7', 52764)>; error=111 (Connection refused)
rasa-worker_1      | 2020-03-20 15:50:28 ERROR    pika.adapters.utils.connection_workflow  - TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.23.0.2', 5672))
rasa-worker_1      | 2020-03-20 15:50:28 ERROR    pika.adapters.utils.connection_workflow  - AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rasa-x_1           | ERROR:pika.adapters.utils.io_services_utils:Socket failed to connect: <socket.socket fd=18, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.23.0.6', 51512)>; error=111 (Connection refused)
rasa-x_1           | ERROR:pika.adapters.utils.connection_workflow:TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.23.0.2', 5672))
rasa-x_1           | ERROR:pika.adapters.utils.connection_workflow:AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rabbit_1           | Waiting for erlang distribution on node 'rabbit@localhost' while OS process '43' is running
rabbit_1           | Waiting for applications 'rabbit_and_plugins' to start on node 'rabbit@localhost'
rasa-production_1  | 2020-03-20 15:50:33 ERROR    pika.adapters.utils.io_services_utils  - Socket failed to connect: <socket.socket fd=25, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.23.0.8', 49350)>; error=111 (Connection refused)
rasa-production_1  | 2020-03-20 15:50:33 ERROR    pika.adapters.utils.connection_workflow  - TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.23.0.2', 5672))
rasa-production_1  | 2020-03-20 15:50:33 ERROR    pika.adapters.utils.connection_workflow  - AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rasa-worker_1      | 2020-03-20 15:50:33 ERROR    pika.adapters.utils.io_services_utils  - Socket failed to connect: <socket.socket fd=25, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.23.0.7', 52770)>; error=111 (Connection refused)
rasa-worker_1      | 2020-03-20 15:50:33 ERROR    pika.adapters.utils.connection_workflow  - TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.23.0.2', 5672))
rasa-worker_1      | 2020-03-20 15:50:33 ERROR    pika.adapters.utils.connection_workflow  - AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rasa-x_1           | ERROR:pika.adapters.utils.io_services_utils:Socket failed to connect: <socket.socket fd=18, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.23.0.6', 51518)>; error=111 (Connection refused)
rasa-x_1           | ERROR:pika.adapters.utils.connection_workflow:TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.23.0.2', 5672))
rasa-x_1           | ERROR:pika.adapters.utils.connection_workflow:AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rasa-production_1  | 2020-03-20 15:50:38 ERROR    pika.adapters.utils.io_services_utils  - Socket failed to connect: <socket.socket fd=26, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.23.0.8', 49364)>; error=111 (Connection refused)
rasa-production_1  | 2020-03-20 15:50:38 ERROR    pika.adapters.utils.connection_workflow  - TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.23.0.2', 5672))
rasa-worker_1      | 2020-03-20 15:50:38 ERROR    pika.adapters.utils.io_services_utils  - Socket failed to connect: <socket.socket fd=26, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.23.0.7', 52784)>; error=111 (Connection refused)
rasa-worker_1      | 2020-03-20 15:50:38 ERROR    pika.adapters.utils.connection_workflow  - TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.23.0.2', 5672))
rasa-production_1  | 2020-03-20 15:50:38 ERROR    pika.adapters.utils.connection_workflow  - AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rasa-worker_1      | 2020-03-20 15:50:38 ERROR    pika.adapters.utils.connection_workflow  - AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rasa-x_1           | ERROR:pika.adapters.utils.io_services_utils:Socket failed to connect: <socket.socket fd=18, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.23.0.6', 51532)>; error=111 (Connection refused)
rasa-x_1           | ERROR:pika.adapters.utils.connection_workflow:TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.23.0.2', 5672))
rasa-x_1           | ERROR:pika.adapters.utils.connection_workflow:AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rabbit_1           | Applications 'rabbit_and_plugins' are running on node 'rabbit@localhost'
nginx_1            | 54.234.185.216 - - [20/Mar/2020:15:50:42 +0000] "POST /:5100/core/webhooks/twilio/webhook HTTP/1.1" 200 3173 "-" "TwilioProxy/1.1"
rasa-production_1  | 2020-03-20 15:50:43 ERROR    pika.connection  - Connection closed while authenticating indicating a probable authentication error
rasa-production_1  | 2020-03-20 15:50:43 WARNING  rasa.core.brokers.pika  - Connecting to 'rabbit' failed with error 'ConnectionClosedByBroker: (403) 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.''. Trying again.
rasa-production_1  | 2020-03-20 15:50:43 ERROR    pika.adapters.utils.connection_workflow  - AMQPConnector - reporting failure: AMQPConnectorAMQPHandshakeError: ProbableAuthenticationError: Client was disconnected at a connection stage indicating a probable authentication error: ("ConnectionClosedByBroker: (403) 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'",)
rasa-worker_1      | 2020-03-20 15:50:43 ERROR    pika.connection  - Connection closed while authenticating indicating a probable authentication error
rasa-production_1  | 2020-03-20 15:50:43 ERROR    pika.adapters.utils.connection_workflow  - AMQPConnectionWorkflow - reporting failure: AMQPConnectionWorkflowFailed: 5 exceptions in all; last exception - AMQPConnectorAMQPHandshakeError: ProbableAuthenticationError: Client was disconnected at a connection stage indicating a probable authentication error: ("ConnectionClosedByBroker: (403) 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'",); first exception - AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rasa-worker_1      | 2020-03-20 15:50:43 WARNING  rasa.core.brokers.pika  - Connecting to 'rabbit' failed with error 'ConnectionClosedByBroker: (403) 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.''. Trying again.
rasa-worker_1      | 2020-03-20 15:50:43 ERROR    pika.adapters.utils.connection_workflow  - AMQPConnector - reporting failure: AMQPConnectorAMQPHandshakeError: ProbableAuthenticationError: Client was disconnected at a connection stage indicating a probable authentication error: ("ConnectionClosedByBroker: (403) 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'",)
rasa-x_1           | ERROR:pika.connection:Connection closed while authenticating indicating a probable authentication error
rasa-x_1           | ERROR:pika.adapters.utils.connection_workflow:AMQPConnector - reporting failure: AMQPConnectorAMQPHandshakeError: ProbableAuthenticationError: Client was disconnected at a connection stage indicating a probable authentication error: ("ConnectionClosedByBroker: (403) 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'",)
rasa-worker_1      | 2020-03-20 15:50:43 ERROR    pika.adapters.utils.connection_workflow  - AMQPConnectionWorkflow - reporting failure: AMQPConnectionWorkflowFailed: 5 exceptions in all; last exception - AMQPConnectorAMQPHandshakeError: ProbableAuthenticationError: Client was disconnected at a connection stage indicating a probable authentication error: ("ConnectionClosedByBroker: (403) 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'",); first exception - AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rasa-worker_1      | 2020-03-20 15:50:43 ERROR    pika.adapters.base_connection  - Full-stack connection workflow failed: AMQPConnectionWorkflowFailed: 5 exceptions in all; last exception - AMQPConnectorAMQPHandshakeError: ProbableAuthenticationError: Client was disconnected at a connection stage indicating a probable authentication error: ("ConnectionClosedByBroker: (403) 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'",); first exception - AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rasa-production_1  | 2020-03-20 15:50:43 ERROR    pika.adapters.base_connection  - Full-stack connection workflow failed: AMQPConnectionWorkflowFailed: 5 exceptions in all; last exception - AMQPConnectorAMQPHandshakeError: ProbableAuthenticationError: Client was disconnected at a connection stage indicating a probable authentication error: ("ConnectionClosedByBroker: (403) 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'",); first exception - AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rasa-worker_1      | 2020-03-20 15:50:43 ERROR    pika.adapters.base_connection  - Self-initiated stack bring-up failed: AMQPConnectionWorkflowFailed: 5 exceptions in all; last exception - AMQPConnectorAMQPHandshakeError: ProbableAuthenticationError: Client was disconnected at a connection stage indicating a probable authentication error: ("ConnectionClosedByBroker: (403) 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'",); first exception - AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rasa-production_1  | 2020-03-20 15:50:43 ERROR    pika.adapters.base_connection  - Self-initiated stack bring-up failed: AMQPConnectionWorkflowFailed: 5 exceptions in all; last exception - AMQPConnectorAMQPHandshakeError: ProbableAuthenticationError: Client was disconnected at a connection stage indicating a probable authentication error: ("ConnectionClosedByBroker: (403) 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.'",); first exception - AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rasa-worker_1      | 2020-03-20 15:50:46.781888: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
rasa-production_1  | 2020-03-20 15:50:46.792396: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
rasa-x_1           | INFO:rasax.community.services.event_consumers.pika_consumer:Start consuming queue 'rasa_production_events' on pika host 'rabbit'.
rabbit_1           |  15:50:58.14 INFO  ==> Stopping RabbitMQ...
nginx_1            | 54.235.58.137 - - [20/Mar/2020:15:50:58 +0000] "POST /:5100/core/webhooks/twilio/webhook HTTP/1.1" 200 3173 "-" "TwilioProxy/1.1"
rasa-x_1           | ERROR:pika.adapters.blocking_connection:Unexpected connection close detected: ConnectionClosedByBroker: (320) "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
rasa-x_1           | ERROR:rasax.community.services.event_service:Caught an exception while consuming events. Will retry in 5 s.
rasa-x_1           | Traceback (most recent call last):
rasa-x_1           |   File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_service.py", line 1400, in continuously_consume
rasa-x_1           |     consumer.consume()
rasa-x_1           |   File "/usr/local/lib/python3.6/site-packages/rasax/community/services/event_consumers/pika_consumer.py", line 123, in consume
rasa-x_1           |     self.channel.start_consuming()
rasa-x_1           |   File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 1866, in start_consuming
rasa-x_1           |     self._process_data_events(time_limit=None)
rasa-x_1           |   File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 2027, in _process_data_events
rasa-x_1           |     self.connection.process_data_events(time_limit=time_limit)
rasa-x_1           |   File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 825, in process_data_events
rasa-x_1           |     self._flush_output(common_terminator)
rasa-x_1           |   File "/usr/local/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 522, in _flush_output
rasa-x_1           |     raise self._closed_result.value.error
rasa-x_1           | pika.exceptions.ConnectionClosedByBroker: (320, "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'")
rasa-x_1           | ERROR:pika.adapters.utils.io_services_utils:Socket failed to connect: <socket.socket fd=18, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.23.0.6', 51562)>; error=111 (Connection refused)
rasa-x_1           | ERROR:pika.adapters.utils.connection_workflow:TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.23.0.2', 5672))
rasa-x_1           | ERROR:pika.adapters.utils.connection_workflow:AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rabbit_1           |  15:51:07.07 INFO  ==> ** RabbitMQ setup finished! **
rabbit_1           | 
rabbit_1           |  15:51:07.08 INFO  ==> ** Starting RabbitMQ **
rasa-x_1           | ERROR:pika.adapters.utils.io_services_utils:Socket failed to connect: <socket.socket fd=18, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.23.0.6', 51564)>; error=111 (Connection refused)
rasa-x_1           | ERROR:pika.adapters.utils.connection_workflow:TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.23.0.2', 5672))
rasa-x_1           | ERROR:pika.adapters.utils.connection_workflow:AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rasa-x_1           | ERROR:pika.adapters.utils.io_services_utils:Socket failed to connect: <socket.socket fd=18, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.23.0.6', 51570)>; error=111 (Connection refused)
rasa-x_1           | ERROR:pika.adapters.utils.connection_workflow:TCP Connection attempt failed: ConnectionRefusedError(111, 'Connection refused'); dest=(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.23.0.2', 5672))
rasa-x_1           | ERROR:pika.adapters.utils.connection_workflow:AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: ConnectionRefusedError(111, 'Connection refused')
rabbit_1           | 2020-03-20 15:51:17.807 [info] <0.8.0> Feature flags: list of feature flags found:
rabbit_1           | 2020-03-20 15:51:17.808 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
rabbit_1           | 2020-03-20 15:51:17.960 [info] <0.270.0> 
rabbit_1           |  Starting RabbitMQ 3.7.17 on Erlang 22.0
rabbit_1           |  Copyright (C) 2007-2019 Pivotal Software, Inc.
rabbit_1           |  Licensed under the MPL.  See https://www.rabbitmq.com/
rabbit_1           | 
rabbit_1           |   ##  ##
rabbit_1           |   ##  ##      RabbitMQ 3.7.17. Copyright (C) 2007-2019 Pivotal Software, Inc.
rabbit_1           |   ##########  Licensed under the MPL.  See https://www.rabbitmq.com/
rabbit_1           |   ######  ##
rabbit_1           |   ##########  Logs: <stdout>
rabbit_1           | 
rabbit_1           |               Starting broker...
rabbit_1           | 2020-03-20 15:51:17.968 [info] <0.270.0> 
rabbit_1           |  node           : rabbit@localhost
rabbit_1           |  home dir       : /opt/bitnami/rabbitmq/.rabbitmq
rabbit_1           |  config file(s) : /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq.config
rabbit_1           |  cookie hash    : FZXzV5U5RznbHldN2mReDw==
rabbit_1           |  log(s)         : <stdout>
rabbit_1           |  database dir   : /bitnami/rabbitmq/mnesia/rabbit@localhost
rabbit_1           | 2020-03-20 15:51:17.985 [info] <0.270.0> Running boot step pre_boot defined by app rabbit
rabbit_1           | 2020-03-20 15:51:17.986 [info] <0.270.0> Running boot step rabbit_core_metrics defined by app rabbit
rabbit_1           | 2020-03-20 15:51:17.989 [info] <0.270.0> Running boot step rabbit_alarm defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.024 [info] <0.284.0> Memory high watermark set to 4001 MiB (4196004659 bytes) of 10004 MiB (10490011648 bytes) total
rabbit_1           | 2020-03-20 15:51:18.049 [info] <0.306.0> Enabling free disk space monitoring
rabbit_1           | 2020-03-20 15:51:18.053 [info] <0.306.0> Disk free limit set to 1049MB
rabbit_1           | 2020-03-20 15:51:18.073 [info] <0.270.0> Running boot step code_server_cache defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.074 [info] <0.270.0> Running boot step file_handle_cache defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.075 [info] <0.345.0> Limiting to approx 1048476 file handles (943626 sockets)
rabbit_1           | 2020-03-20 15:51:18.075 [info] <0.346.0> FHC read buffering:  OFF
rabbit_1           | 2020-03-20 15:51:18.076 [info] <0.346.0> FHC write buffering: ON
rabbit_1           | 2020-03-20 15:51:18.080 [info] <0.270.0> Running boot step worker_pool defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.080 [info] <0.273.0> Will use 1 processes for default worker pool
rabbit_1           | 2020-03-20 15:51:18.081 [info] <0.273.0> Starting worker pool 'worker_pool' with 1 processes in it
rabbit_1           | 2020-03-20 15:51:18.083 [info] <0.270.0> Running boot step database defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.088 [info] <0.270.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbit_1           | 2020-03-20 15:51:18.123 [info] <0.270.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbit_1           | 2020-03-20 15:51:18.124 [info] <0.270.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
rabbit_1           | 2020-03-20 15:51:18.125 [info] <0.270.0> Running boot step database_sync defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.126 [info] <0.270.0> Running boot step feature_flags defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.127 [info] <0.270.0> Running boot step codec_correctness_check defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.127 [info] <0.270.0> Running boot step external_infrastructure defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.127 [info] <0.270.0> Running boot step rabbit_registry defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.128 [info] <0.270.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.129 [info] <0.270.0> Running boot step rabbit_queue_location_random defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.129 [info] <0.270.0> Running boot step rabbit_event defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.130 [info] <0.270.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.130 [info] <0.270.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.131 [info] <0.270.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.131 [info] <0.270.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.132 [info] <0.270.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.132 [info] <0.270.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.133 [info] <0.270.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.133 [info] <0.270.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.133 [info] <0.270.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.134 [info] <0.270.0> Running boot step rabbit_priority_queue defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.134 [info] <0.270.0> Priority queues enabled, real BQ is rabbit_variable_queue
rabbit_1           | 2020-03-20 15:51:18.135 [info] <0.270.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.135 [info] <0.270.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.136 [info] <0.270.0> Running boot step kernel_ready defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.136 [info] <0.270.0> Running boot step rabbit_sysmon_minder defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.137 [info] <0.270.0> Running boot step rabbit_epmd_monitor defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.140 [info] <0.270.0> Running boot step guid_generator defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.145 [info] <0.270.0> Running boot step rabbit_node_monitor defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.147 [info] <0.370.0> Starting rabbit_node_monitor
rabbit_1           | 2020-03-20 15:51:18.148 [info] <0.270.0> Running boot step delegate_sup defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.149 [info] <0.270.0> Running boot step rabbit_memory_monitor defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.149 [info] <0.270.0> Running boot step core_initialized defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.149 [info] <0.270.0> Running boot step upgrade_queues defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.173 [info] <0.270.0> Running boot step rabbit_connection_tracking defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.174 [info] <0.270.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.174 [info] <0.270.0> Running boot step rabbit_exchange_parameters defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.175 [info] <0.270.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.176 [info] <0.270.0> Running boot step rabbit_policies defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.177 [info] <0.270.0> Running boot step rabbit_policy defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.177 [info] <0.270.0> Running boot step rabbit_queue_location_validator defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.178 [info] <0.270.0> Running boot step rabbit_vhost_limit defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.178 [info] <0.270.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
rabbit_1           | 2020-03-20 15:51:18.178 [info] <0.270.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
rabbit_1           | 2020-03-20 15:51:18.179 [info] <0.270.0> Management plugin: using rates mode 'basic'
rabbit_1           | 2020-03-20 15:51:18.180 [info] <0.270.0> Running boot step recovery defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.182 [info] <0.404.0> Making sure data directory '/bitnami/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
rabbit_1           | 2020-03-20 15:51:18.191 [info] <0.404.0> Starting message stores for vhost '/'
rabbit_1           | 2020-03-20 15:51:18.193 [info] <0.408.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
rabbit_1           | 2020-03-20 15:51:18.197 [info] <0.404.0> Started message store of type transient for vhost '/'
rabbit_1           | 2020-03-20 15:51:18.198 [info] <0.411.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
rabbit_1           | 2020-03-20 15:51:18.203 [info] <0.404.0> Started message store of type persistent for vhost '/'
rabbit_1           | 2020-03-20 15:51:18.212 [info] <0.270.0> Running boot step load_definitions defined by app rabbitmq_management
rabbit_1           | 2020-03-20 15:51:18.212 [info] <0.270.0> Running boot step empty_db_check defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.213 [info] <0.270.0> Running boot step rabbit_looking_glass defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.214 [info] <0.270.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.215 [info] <0.270.0> Running boot step background_gc defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.216 [info] <0.270.0> Running boot step connection_tracking defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.216 [info] <0.270.0> Setting up a table for connection tracking on this node: tracked_connection_on_node_rabbit@localhost
rabbit_1           | 2020-03-20 15:51:18.217 [info] <0.270.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost_on_node_rabbit@localhost
rabbit_1           | 2020-03-20 15:51:18.218 [info] <0.270.0> Running boot step routing_ready defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.218 [info] <0.270.0> Running boot step pre_flight defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.218 [info] <0.270.0> Running boot step notify_cluster defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.219 [info] <0.270.0> Running boot step networking defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.223 [warning] <0.438.0> Setting Ranch options together with socket options is deprecated. Please use the new map syntax that allows specifying socket options separately from other options.
rabbit_1           | 2020-03-20 15:51:18.224 [info] <0.452.0> started TCP listener on [::]:5672
rabbit_1           | 2020-03-20 15:51:18.225 [info] <0.270.0> Running boot step direct_client defined by app rabbit
rabbit_1           | 2020-03-20 15:51:18.254 [info] <0.502.0> Management plugin: HTTP (non-TLS) listener started on port 15672
rabbit_1           | 2020-03-20 15:51:18.255 [info] <0.608.0> Statistics database started.
rabbit_1           | 2020-03-20 15:51:18.255 [info] <0.607.0> Starting worker pool 'management_worker_pool' with 3 processes in it
rabbit_1           |  completed with 3 plugins.
rabbit_1           | 2020-03-20 15:51:18.703 [info] <0.8.0> Server startup complete; 3 plugins started.
rabbit_1           |  * rabbitmq_management
rabbit_1           |  * rabbitmq_management_agent
rabbit_1           |  * rabbitmq_web_dispatch
rabbit_1           | 2020-03-20 15:51:21.002 [info] <0.618.0> accepting AMQP connection <0.618.0> (172.23.0.6:51572 -> 172.23.0.2:5672)
rabbit_1           | 2020-03-20 15:51:21.009 [info] <0.618.0> connection <0.618.0> (172.23.0.6:51572 -> 172.23.0.2:5672): user 'user' authenticated and granted access to vhost '/'
rasa-x_1           | INFO:rasax.community.services.event_consumers.pika_consumer:Start consuming queue 'rasa_production_events' on pika host 'rabbit'.
nginx_1            | 54.175.237.160 - - [20/Mar/2020:15:51:25 +0000] "POST /:5100/core/webhooks/twilio/webhook HTTP/1.1" 200 3173 "-" "TwilioProxy/1.1"
nginx_1            | 3.95.185.118 - - [20/Mar/2020:15:51:40 +0000] "POST /:5100/core/webhooks/twilio/webhook HTTP/1.1" 200 3173 "-" "TwilioProxy/1.1"
9

I changed my webhook url in Twilio to http://<host>:5100/core/webhooks/twilio/webhook and still Rasa X returns a malformed response. The logs are in a pending spam-flagged post that an automated message should be reviewed by a team member.

I don’t see documentation on the webhook address including the “/core/” directory in the docs and the webhook URL was likewise referred to me as http://<host>:<port>/webhooks/twilio/webhook in a post earlier in the thread:

I have kept endpoints.yml the exact same. Thanks again for your help.

10

@argideritzalpea where does this port number come from? it shouldn’t be needed. It should just be http://host/core/webhooks/twilio/webhook. The /core part is described here.

Can you show the output of docker-compose logs rasa-production when you’ve tried out the URL i suggested?

1 Like
11

The port I was using was the old default port that was required for the connections webhooks. Also, previous answers before yours @akelad did not suggest that a port needed to be deleted. Thank you for providing the correct answer here!

1 Like
12

The /core part is not described there anymore because

from the changelog:

#2379: When connecting to external channels on /webhooks/{channel}/webhook , it is no longer required to add the /core/ prefix to the endpoint. Nginx will route http://<HOST>/webhooks/ requests to the rasa-production container.