# Benchmark Rasa Performance

**URL:** <https://forum.rasa.com/t/benchmark-rasa-performance/12759>\
**Category:** Rasa Open Source\
**Created:** [July 10, 2019, 4:43pm UTC](https://forum.rasa.com/t/benchmark-rasa-performance/12759 "2019-07-10T16:43:48Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ggbond](https://avatars.discourse-cdn.com/v4/letter/g/8e8cbc/32.png) [@ggbond](https://forum.rasa.com/u/ggbond)\
**Post date:** [July 10, 2019, 4:43pm UTC](https://forum.rasa.com/t/benchmark-rasa-performance/12759/1 "2019-07-10T16:43:48Z")

</div>

Has anyone done any stress-testing against the Rasa? Do you guys mind to share the result of your stress-testing? I am wondering how many concurrent “active” users are able to talk to the chatbot through the Rasa.

---

<div class="post-metadata">

**Author:** ![lahsuk](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/lahsuk/32/3127_2.png) [@lahsuk](https://forum.rasa.com/u/lahsuk)\
**Post date:** [July 16, 2019, 4:49am UTC](https://forum.rasa.com/t/benchmark-rasa-performance/12759/2 "2019-07-16T04:49:06Z")

</div>

Hi ChiaJun,

I’ve done a small test to see how the server copes with stress and I don’t have good results to share. But regardless, here it is:

![image](https://europe1.discourse-cdn.com/flex013/uploads/rasa/original/2X/e/ecd6f10d723359ba123a31eb61963e404099f721.png)

You can see here that the message _response time per message increases with the number of messages that the user has sent_. This is probably because of the tracker store and looks like it scales linearly with the size of conversation history.

For small conversation history of 10 messages, response time was as low as 60ms but it increases to 300ms for history of 700+ messages.

This test was done in JupyterLab so the tracker store used was InMemory. You can expect worse results when databases are used. (In fact, initially, I tested with a script and the session timed out for 800 messages for 2 users for a connection timeout time of 1 hour.) Also, almost all of the actions run on the action server which may have contributed to a little bit of time here.

So, to answer your question, it can probably support between `3-15 active users` based on their conversation length.

Here is the sample code that was run:

```auto
import time
from rasa.core.agent import Agent

data = [
    # messages go here
    # had 848 messages here
]
data_counts = [i*10 for i in range(1, 80, 3)]

for message_count in data_counts:
    # create new agent to reset tracker store
    agent = Agent.load("models/model.tar.gz")
    
    start_time = time.time()
    for message in data[:message_count]:
        try:
            response = await agent.handle_message(message);
        except:
            pass
    end_time = time.time()

    print("message count:", message_count)
    print("total time:", end_time - start_time)
    print("time per message:", (end_time - start_time) / message_count)
    
    del agent

```

Hope that helps.

**Edit:** Looks like the performance deteriorates \> 10x when using mongo tracker store in comparison to InMemory tracker store. My custom actions run ~3ms with InMemory and ~55ms for Mongo tracker store.

---

<div class="post-metadata">

**Author:** ![twittmin](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/twittmin/32/2091_2.png) [@twittmin](https://forum.rasa.com/u/twittmin)\
**Post date:** [July 17, 2019, 5:41am UTC](https://forum.rasa.com/t/benchmark-rasa-performance/12759/3 "2019-07-17T05:41:29Z")

</div>

Hi, Lahsuk:

Thanks for sharing. This is my policy settings: policies: - name: KerasPolicy epochs: 300 max\_history: 3 - name: AugmentedMemoizationPolicy max\_history: 3 - name: FallbackPolicy nlu\_threshold: 0.6 core\_threshold: 0.3 - name: FormPolicy - name: MappingPolicy

I only keep a conversation history of 3. Does the tracker store keeps all history even if max\_history=3? If it stores all history data and won’t use all history data for next action prediction, which doesn’t make sense, why does the performance decreases quickly?

In my case, I simply take a file of 10,000 sentences as input, and process sentence one by one without concurrency , after it processes about 500 sentences, the speed decreases quickly. Is this similar to multiple active users to talk to the bot simultaneously?

---

<div class="post-metadata">

**Author:** ![lahsuk](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/lahsuk/32/3127_2.png) [@lahsuk](https://forum.rasa.com/u/lahsuk)\
**Post date:** [July 17, 2019, 6:04am UTC](https://forum.rasa.com/t/benchmark-rasa-performance/12759/4 "2019-07-17T06:04:38Z")

</div>

Hi @twittmin,

I don’t think the history size effects performance (it only effects prediction).

The numbers of active users it can support depends on their conversation length. So, for very short conversations it can support 20+ users (depends on other factors too including custom actions) and for long conversations, it will struggle to support even single user.

And no, bombarding it with 10,000 sentences is not similar to multiple users talking to the bot simultaneously. It is just a single user conversing with the bot.

And, looks like other people have faced this issue too:

> [@Rasa performance for increasingly large trackers](http://forum.rasa.com/t/rasa-performance-for-increasingly-large-trackers/11126):
>
> We are currently building a bot using rasa\_core==0.14.4 . During testing we notice considerable slowdown, to the point that it is no longer usable, when having longer conversation. Typically, this happens after around 20 rounds, in our bot this corresponds to 80 events. After profiling our bots we came to the conclusion that the constant (de)serializing and recreating of the tracker objects is the current bottleneck. Looking at the code, it seems that the processing becomes slower as more event…

---

<div class="post-metadata">

**Author:** ![twittmin](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/twittmin/32/2091_2.png) [@twittmin](https://forum.rasa.com/u/twittmin)\
**Post date:** [July 17, 2019, 3:28pm UTC](https://forum.rasa.com/t/benchmark-rasa-performance/12759/5 "2019-07-17T15:28:41Z")

</div>

Hi, @lahsuk, Since it is not used for prediction, why does it slow down so much? Storage wise, it should only increase memory usage, right? My observation is that memory stay almost constant, compared with its size.

Since it seems a lot of people are using rasa to build bots, there should be way to overcome the problem.

---

<div class="post-metadata">

**Author:** ![twittmin](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.rasa.com/twittmin/32/2091_2.png) [@twittmin](https://forum.rasa.com/u/twittmin)\
**Post date:** [July 17, 2019, 3:47pm UTC](https://forum.rasa.com/t/benchmark-rasa-performance/12759/6 "2019-07-17T15:47:03Z")

</div>

This event broker may be helpful:

> **[Event Brokers](https://rasa.com/docs/rasa/event-brokers/)**
>
> Find out how open source chatbot framework Rasa allows you to stream events to a message broker.
