Performance issue with action end point

@shyamzacharia Did You find a solution to this issue?? I am facing the same with Rasa 1.4.3 version. @Tobias_Wochinger I dont think there is any issue with any of those things you suggested.

My normal pre defined actions like “utter_greet” etc are getting replied quickly. But its taking approx 2-3 seconds when a custom action is called.

It has nothing to do with the content of the Actions.py .For me even a small custom action like this-

class ActionCheckContract(Action):
def name(self):
    return "contract_action"

def run(self, dispatcher, tracker, domain):
    intent = tracker.latest_message["intent"].get("name")
    if intent == "inform_contract_owner":
        dispatcher.utter_template("utter_change_mind", tracker)
        return []

**Takes the equal amount of time to response in compared to this Action.py (5% of Actions,py)

class ActionCheckContract(Action):
def name(self):
    return "contract_action"

def run(self, dispatcher, tracker, domain):
    client = pymongo.MongoClient("localhost", 27017)
    db = client.in_dev
    collection = db.Contract
    entity = tracker.get_latest_entity_values("contract_name")
    entity_DB = next((j for j in collection.find({"_id": ObjectId(entity)})), None)
    print(entity_DB)
    intent = tracker.latest_message["intent"].get("name")
    intents_check_dict = {"inform_contract_owner": 'contractOwner', "inform_contract_number": "contractNumber",
                          "inform_contract_risk_rating": "contractRisk",
                          "inform_contract_annual_spend": "annualSpend",
                          "inform_business_risk_rating": "riskRating", "inform_contract_value": "value",
                          "inform_contract_renewal_period": "renewalPeriodId",
                          "inform_contract_liquidation_damages": "liquidationDamage",
                          "inform_contract_minimum_spend": "minimumSpend",
                          "inform_contract_addressable_spend": "addressableSpend",
                          "inform_contract_policy_number": "policyNumber"}

    def intent_lookup(intent):
        return intents_check_dict.get(intent)

    c = intent_lookup(intent)
    if c is not None:
        print(intents_check_dict)
        dispatcher.utter_message(str(entity_DB[c]))
        return []

It takes the same time ,even if I alter the interface file to change a function and get my entity through text search API locally. So could u please suggest a solution to this? @akelad @Juste @alexweidauer @souvikg10 @erohmensing Please help me out on this.

@shyamzacharia Did you find a solution to this?

@pranay_raj what does your setup look? Where are you running the action server and the rasa server? Are they on the same machine?

@akelad Thank you so much for replying back. My deployement date is approaching :stuck_out_tongue: . I could really use some help of yours. My setup as in?what exactly you mean by setup?

My config.yml-

pipeline:         
name: "WhitespaceTokenizer"*
name: "CountVectorsFeaturizer"*
name: "EmbeddingIntentClassifier"*

policies:
      - name: MemoizationPolicy
      - name: KerasPolicy
      - name: MappingPolicy

I am not using any entity extractor/classifier because Rasa’s Entity extraction for named entity recognition was not efficient enough for my use case, so i went for text search approach.

And both the servers are run separately locally as of now from cmd. The predefined functions gives reply within a second.But custom actions take time from 2-3 seconds.

I have tried executing my project with different Rasa versions.- 1.0.0.a1 , 1.3.1, 1.4 , 1.5 , 1.6.1(latest) Out of which, in version 1.4 and 1.5 I was getting replies from custom action even after 4-5 seconds. In the latest version also it takes approx 2 seconds or more to show result.

So this is all just running on the command line? That’s strange that it takes so long. Can you show me the logs of both rasa and the action server? Specifically i’m interested in the time stamps of the action being predicted and the action server executing the action

This is Rasa shell debug log-

@akelad

Yeah! So, it takes approximately 2 seconds between- (27-29 seconds)

20-01-20 19:23:27 DEBUG rasa.core.actions.action - Calling action endpoint to run action ‘supplier_action’.

2020-01-20 19:23:29 DEBUG rasa.core.processor steps.

This is Action Server log- The request to Action server is also received at : 29 seconds time stamp.(2 seconds late)

@akelad Ma’am, can you please direct me as to what I should do to reduce the request time.? And why exactly is it happening?

@pranay_raj we’re looking into it on our end, but we’ve never seen this before so it’s entirely possible it’s something to do with your machine. Do you have the option of trying this on a different machine?

@akelad OK, I shall try on different machine and let you know within an hour. TYSM for replying.

@akelad Hey. So I checked it on another system as well, it has the same lagging problem of 2 seconds. What may be the reason behind this? The same Issue persist. From the other system as well, after the call is made from core.actions to predicted action, the request is received after two seconds in action server log

@akelad Can you please tell me the files or classes or functions which might be causing this delay. As i can see the request received itself is delayed. So what may be the exact cause or which call could be lagging?? please reply.

Hm, that’s odd. Few more questions:

  • have you modified the rasa or rasa-sdk code in any way?
  • which versions are you using?
  • what commands are you using to start rasa and the rasa-sdk?

@akelad . Thank You for replying.

  1. I did modify rasa-sdk in my system in the interface file, but i guess thats irrelevant since I didnt change anything on another machine.

Currently I am using 1.6.1 of Rasa 3. rasa shell --debug and rasa run actions are the command i am using And also rasa run -m models --enable-api --cors “*” --debug

are you running rasa run -m models... and rasa shell simultaneously?

@akelad No only when I have to use chatbot on UI i run rasa run… else for normal testing i use rasa shell

As far as I can trace the call to action endpoint is getting delayed. May I know what happens after

What calls are made ?? And Just what may be probably the reason???

Does the delay occur in both rasa shell and rasa run?

Yes, in both! On the UI side it feels more than 2 sec. But the timestamp of rasa run shows the equal lag of 2 seconds, as in rasa shell.

@akelad Did you find any solution to this or any reason perhaps, as to why the call is getting delayed?? And btw the call is getting delayed only when action server is called from rasa, I tested calling action server from Postman and there wasnt any noticable lag. Please reply! :pray:

Run the following command to monitor the rest call to the action server. I’d like to know if the 2 second delay is before the rasa POST or after.

sudo tcpdump -A -t -s 0 'tcp port 5055 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'