Hi there!
I am relatively new to python and RASA. I have developed a chatbot and hosted in the cloud which works pretty well. Now i want to integrate the ELK stack with my chatbot for user analytics. I have read the post by gstephens on his website but it is not making any sense to me as it is way over my head. If anyone can point me in the right direction, it will be great.
Thanks.
@jusce17 Can you please take a look?
Hello @sealedhermit and that it is a great use-case, can you share with me some basic information:
Thanks.
I have hosted my bot online in okteto and have logstash, kibana and elastic installed and running in the same instance. I have read gstephens blog, but as i said, i am basically stuck.
@sealedhermit so in summary you just want to get the analysis of user queries and display them in the dashboard? Right?
Yes. Right now i am stuck at understanding the configuration of ELK which gstephens has detailed in his blog. Since i am a newbie trying out with RASA, i pretty much is blank about this.
@sealedhermit If you allow me can I give my honest feedback? As I already implemented the complete pipeline starting from Rasa to Kibana.
Sure. I am welcome to suggestions.
@stephens This is what i am not understanding. I am pretty new with Rasa and ELK stack, so that might be the reason.
Do i need to edit the event_broker in rasa open source? If so, is it the same format as mentioned in this blog? Or do i only need to add rabbitMQ in docker for rasa open source?
event_broker:
type: "pika"
url: ${RABBITMQ_HOST}
username: ${RABBITMQ_USERNAME}
password: ${RABBITMQ_PASSWORD}
queue: ${RABBITMQ_QUEUE}
In the second part, where logstash is setup, what is the host address i need to enter? Is it the instance address of the RabbitMQ server inside the docker?
input {
rabbitmq {
queue => "elk_log"
host => "<host ip addr>"
port => 15672
user => "user"
password => "<rabbit user password>"
durable => true
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
user => "elastic"
password => "<elastic password>"
}
}
filter {
if [@metadata][rabbitmq_properties][timestamp] {
date {
match => ["[@metadata][rabbitmq_properties][timestamp]", "UNIX"]
}
}
}
@sealedhermit Well, honestly (I will not recommend using the Elastic Search, then Logstach, and then used Kibana for creating dashboard) as, the intermediate steps are really complex and you need to run the commands every time, you need to push the data to Kibana. As, I told you I already tried this, as I am an expert and working on different projects on Kibana but for me, it’s making our lives more complex and utilizing a lot of resources. So, the alternative solution, which I followed was, rasa bot connect with database i.e Postgres or MySQL or etc using Trackerstore and I used Grafana Dashboards | Grafana Labs ( Note: I’m not promoting this website or I never worked with Grafana ) for creating different dashboards.
Summary, whilst using Kibana there are not many details you can display honestly. Except for user queries or bot responses that’s it, that you can even archive whilst using Rasa X.
I’m not saying you should not use ELK but for me, it has not done the purpose and even my manager also felt the same.
I hope my suggestion will help you and guide you in the right direction. Good Luck!
Is there a tutorial or any public repo where i can see this rasa-grafana in action?
I needed to implement something open-source for my setup. That’s why i was rooting for ELK. As i am already invested too much in this, i will try ELK and Grafana and switch over if possible. The first problem in hand is the ELK integration though.
@sealedhermit give me some time, let me find the resource for you, else Stephen is the best person to deal with ELK.
Sure. Thanks for your help.
No worries.
Do i need to edit the event_broker in rasa open source?
Yes, you need the event_broker config in the endpoints.yml in the format shown in the post.
Or do i only need to add rabbitMQ in docker for rasa open source?
I think you’re confused about this point. The endpoints.yml is a Rasa OSS configuration file (not docker or ELK).
In the second part, where logstash is setup…
This configuration is telling logstash how to connect to the rabbitmq service and queue so this is the host IP of your rabbit server. With docker, you can use the name of the rabbit service in your docker-compose.yml
Hi @stephens !
Sorry, i may sound dumb asking this. I have added the entries in the event_broker inside my endpoints.yml as per your blog.
event_broker:
type: "pika"
url: <url>
username: guest
password: guest
queues:
- rasa_production_events
- elk_log
the url here, is this url of the rabbit server i have installed inside my docker container on the cloud?
And my docker-compose.yml for the rabbit looks like this:
rabbit:
restart: always
image: "bitnami/rabbitmq:3.7.17"
environment:
RABBITMQ_HOST: "rabbit"
RABBITMQ_USERNAME: "guest"
RABBITMQ_PASSWORD: guest
RABBITMQ_DISK_FREE_LIMIT: "{mem_relative, 0.1}"
expose:
- "5672"
ports:
- "5672:5672"
- "15672:15672"
Here the rabbitMQ username and the password is the same as that of the one mentioned in the endpoints.yml?
And finally in the ports.conf for logstash,
input {
rabbitmq {
queue => "elk_log"
host => "<url>"
port => 15672
user => "guest"
password => "guest"
durable => true
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
index => "test-elk"
}
}
filter {
if [@metadata][rabbitmq_properties][timestamp] {
date {
match => ["[@metadata][rabbitmq_properties][timestamp]", "UNIX"]
}
}
}
The host here is the same as that of the one mentioned in the event_broker configuratiion? i.e the rabbit server endpoint that i have hosted in the cloud?
My logstash returns logstash-5c4d756bbd-9lmr8logstash[2022-03-05T03:53:44,505][ERROR][logstash.inputs.rabbitmq ][main][93243bf60bab34d0677494d07601102a4c63a522d6bd14f35d119f465ab90056] RabbitMQ connection error, will retry. {:error_message=>"Connection was refused. Target host list: rabbit-15672-testbedelk-sealedhermit.cloud.okteto.net:15672, target virtual host: /, username: guest", :exception=>"MarchHare::ConnectionRefused"} error with the following setup.
And my rasa server hosted on okteto cloud returns this error:
2022-03-05 04:05:10.26 UTCrasa-66f858d6cb-vd4gxrasa2022-03-05 04:05:10 DEBUG rasa.core.brokers.pika - Connecting to 'https://rabbit-15672-testbedelk-sealedhermit.cloud.okteto.net/' failed with error 'Port could not be cast to integer value as '//rabbit-15672-testbedelk-sealedhermit.cloud.okteto.net/:5672''. Trying again.
the url here, is this url of the rabbit server i have installed inside my docker container on the cloud?
docker-compose networking is discussed here. In your config, the hostname is rabbit.
Before proceeding with the logstash config, make sure you can read the rasa events from the rabbit event queue. My blog post provides example curl commands for this.
Hi @stephens !
Now i am able to successfully read the rasa events from rabbit event queue. What changes should i make to the logstash config to complete the integration?
Great, so now the logstash config is going to read from that rabbit queue, so the parameter values in the log stash config will be the same that you used for the curl command.
@stephens I have successfully connected the RabbitMQ with rasa and then the logstash configuration. RabbitMQ now get’s successfully connected with logstash as well.
[logstash.inputs.rabbitmq ][main][fe26ba5c444dccf07a2fda433a36e9ab13aae75972bab003b2cd97665c45369f] Connected to RabbitMQ at
[2022-03-07T08:56:28,400][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
But when i go to stack monitoring and click logstash overview, i am not seeing any events received or emitted here.
Elasticsearch says no log found and asks me to setup filebeat. Is there any step that i have missed in between?