Sending Custom JSON Payload from actions(workaround for utter_custom_message)

Hi all, I wanted to share some interesting workaround for utter_custom_message, I saw many people were facing difficulties while using this method(I too had faced the same issue) so I had came up with some workaround:

so below is the json data that I want to sent but when I used utter_custom_message(), I used to get an empty array in the response

snip1

so what I did instead of using utter_custom_message I used utter_message and sent the above data in JSON string format using json.dumps

Since I am using custom channel I mean had created my own UI for Rasa Core so I can easily get the JSON data in my custom UI(i.e handled through Javascript) and then I can further process that JSON data as required.

image

I hope this helps :wink: :slight_smile:

3 Likes

hey @JiteshGaikwad your bot ui awesome.Can you tell me how to create that? or more help like can you share me that code ya github repo or link?Thanks in advance.

hey @Nishet I had created the UI using CSS and JS, right now I don’t have any GitHub link to share with you as this UI was developed by me😃

Ohk @JiteshGaikwad Thanks for reply. Can you share that css and js ?Thanks.

hey @Nishet the UI is still under development

Great way to deal with this… I tried using “custom tag” shown in Domains doc but it just doesn’t send like we want it to be, it sends with the key “metadata” having value as Custom JSON provided in domains.yml. If anyone could help me with this would be great!

Hey folks. Any one found how to make sure the custom element is not null? We are running Rasa 1.1.4 on Ubuntu 16.04 facing a similiar issue when using the SocketIO.py channel for custom message utterance.

In the domain file we have:

utter_csmb_hardware_grid:
  - custom:
      title: "click here to see the page with the info table, and here is the video: https://myinernalurl.com/index.html"
      video: "https://my.url.com/video"

When running rasa shell from CLI, the custom message presents itself when this utterance is hit:

? What type of segment for would you like to see: 2: Consumer (/view_grid{“grid_hw_segment”: “consumer_smb”})

Custom json: { “title”: “click here to see the page with the info table, and here is the video: https://myinernalurl.com/index.html”, “video”: “https://my.url.com/video” }

Can I help you with something else? (Yes/No)

Your input ->

But when run it through the webchat via socket.io we get nothing back and following in logs.

2019-07-02 22:10:45 INFO engineio.server - 0c2df059678d48fbaa4244c99faeb099: Sending packet MESSAGE data 2["bot_uttered",null]

But in the core I can see it has an utterance with the custom attribute populated:

2019-07-02 22:10:45 DEBUG    rasa.core.processor  - Action 'utter_csmb_hardware_grid' ended with events '['BotUttered(text: None, data: {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": {"title": "click here to see the page with the info table, and here is the video: https://myinernalurl.com/index.html", "video": "https://my.url.com/video"}}, metadata: {})']

@akelad Any thoughts on this one? It’s clear that the socketio.py send_custom_json method is called because in the log i get the “bot_uttered” part:

async def send_custom_json( self, recipient_id: Text, json_message: Dict[Text, Any], **kwargs: Any ) -> None: “”“Sends custom json to the output”""

    json_message.setdefault("room", self.sid)

    await self.sio.emit(self.bot_message_evt, **json_message) 

We will try the workaround, but the custom message should work also?

1 Like

@erohmensing do you have any ideas about this one?

@Serge in order for the element to make it to the front-end, it has to match what the front-end is expecting. The custom is specifically for custom json payloads, so it has to be a piece of json you could feasibly send to the front-end completely separate of rasa, just turned into yml format. Looking at the socket wrapper we use, it looks like your video format isn’t the socket video format. Are you using the rasa-webchat? If so, this is how video messages are parsed to send to the front-end: rasa-webchat/msgProcessor.js at master · mrbot-ai/rasa-webchat · GitHub

Hi @erohmensing.

Thank you kindly for replying.

Yes we use mrbot rasa webchat.

Key is when we provide the custom utterance in the domain file the socket adapter of rasa sents “NULL” - see bot_uttered, null, so it simply never makes to the UI layer.

But in general, yes, we are actively trying to establish the contract between rasa and ui.

This is what we have in the domain file for this utterance:

utter_csmb_hardware_grid:
  - custom:
      title: "click here to see the page with the info table, and here is the video: https://myinernalurl.com/index.html"
      video: "https://my.url.com/video"

And yes, we can adapt the rasa web chat to what is needed, but key for that we need rasa to return something to the UI as part of customer versus null.

Let me know if there is something that can be done. This is not a problem in the rasa shell CLI as the implementation of the command line channel is simpler - it just checks for (message.contains), but it doesn’t work in the socket adapter.

Thank you

Hm okay I see what you mean.

Not sure if you are on an ngrok server or not – if you can see the http request and whether it’s completely empty that would be helpful too – my instinct is that the information is ctually being sent, you’re just not seeing it because that “sending message data” only prints the “text” part of the data. But my guess is that the information is actually hitting the server in any case, it’s just not correctly formatted – so the http req Would be super helpful!

However the connector should also definitely print any data it gets too, if it doesn’t, I need to fix that. If you send a message buttons (another example of data), what does that log message look like?

Hi I’m getting the same problem. How do I get the http req data for you?

@ghost are you using ngrok?

@Serge @ghost Heya, I was having the same issue, so I looked a bit into the source code (Rasa 1.1.3).

Well, I’m actually directly calling the dispatcher.utter_custom_json method from custom actions, but I could confirm it was the same issue by adding logs on the method eventually called on the socketio channel, send_custom_json(). That method is itself calling sio.emit(self.bot_message_evt, **json_message).

As you can see, that method is unpacking the json message before emitting through socketio (**json_message), so in your case it would be something like calling sio.emit(self.bot_message_evt, title="click here to see[...]", video="https://[...]", room=self.sid) (the room arg is added in the json message by the send_custom_json method).

Now, from what I understood, if you look at that emit method, you’ll notice that what’s actually sent to the client is in the data parameter. Your title and video args though, are included in the extra kwargs and sent to the asyncio_manager.AsyncManager() manager, which does… nothing with them, since it’s not expecting any title or video arg. Meanwhile, as the data arg is actually empty, what’s sent is a null message to the client, resulting in an error on rasa-webchat (since it’s trying to convert a null to an object to check its length, according to what I see from their source code).

Fortunately, it’s pretty easy to solve that issue. From what I see, you’d have two options:

  1. Make a custom SocketIo channel to override the send_custom_json method so that it doesn’t unpack. For it to work, you could look at how other methods are implemented, but you could do something as simple as this:
async def send_custom_json(
        self, recipient_id: Text, json_message: Dict[Text, Any], **kwargs: Any
    ) -> None:
        """Sends custom json to the output"""
        await self.sio.emit(self.bot_message_evt, json_message, room=self.sid)
        // Or use await self._send_message(self.sid, json_message) instead, same thing
  1. Pack your custom message in a data block. When calling send_custom_json, it will make it so it ends up calling sio.emit(self.bot_message_evt, data={title="click here to see[...]", video="https://[...]"}, room=self.sid), which will then send that json to the front-end.
  • For those using custom actions. you would do something like this:
message = {}
message["data"] = {"my_custom_element": "var1", "my_other_custom_element": "Hello world"}
dispatcher.utter_custom_json(message)
  • For those using the domain file, like what I can see you’re doing, it would be something like this:
utter_csmb_hardware_grid
  - custom:
      data:
        title: "click here to see the page with the info table, and here is the video: https://myinernalurl.com/index.html"
        video: "https://my.url.com/video"

Both should be working from what I tried. I could check that the message was actually sent:

2019-07-29 12:20:18 INFO     engineio.server  - c1b8c2bf0b7d4396b23f68778762abf4: Sending packet MESSAGE data 2["bot_uttered",{"title":"click here to see the page with the info table, and here is the video: https://myinernalurl.com/index.html","video":"https://my.url.com/video"}]

On the Firefox Console, I also checked that it was received by adding some logging:

But then again, as @erohmensing, for it to do anything, it would need to be data formatted in a format expected by the front-end, which doesn’t look like to be the case for rasa-webchat (for videos, another format is expected). So while it has been received, nothing is displayed.

Hope it helps.

7 Likes

Thanks for the thorough debugging and explanation @Mgo2! I think that your option number 2 is the better option and is very well thought out. Thanks for the help :smiley:

You’re welcome °w°

Yeah, I think it’s the better option too, especially if you have no other use in making a custom socketio channel. In my case, I did implement a custom channel to add some stuff, but even then I preferred using that option since it felt “cleaner”. I wrote the first option in case someone had for some reason tons of custom messages to send and didn’t want to keep adding that data block.

Sorry but, please share one full example. I am unable to understand it. @Mgo2 @erohmensing

@Mgo2 @erohmensing I just figured out, after some inspecting and comparison with quick_replies

Answer is check below… If any have issue like this problems…

Facebook’s example demo for generic template :

curl -X POST -H "Content-Type: application/json" -d '{
  "recipient":{
    "id":"<PSID>"
  },
  "message":{
    "attachment":{
      "type":"template",
      "payload":{
        "template_type":"generic",
        "elements":[
           {
            "title":"Welcome!",
            "image_url":"https://petersfancybrownhats.com/company_image.png",
            "subtitle":"We have the right hat for everyone.",
            "default_action": {
              "type": "web_url",
              "url": "https://petersfancybrownhats.com/view?item=103",
              "webview_height_ratio": "tall",
            },
            "buttons":[
              {
                "type":"web_url",
                "url":"https://petersfancybrownhats.com",
                "title":"View Website"
              },{
                "type":"postback",
                "title":"Start Chatting",
                "payload":"DEVELOPER_DEFINED_PAYLOAD"
              }              
            ]      
          }
        ]
      }
    }
  }
}' "https://graph.facebook.com/v2.6/me/messages?access_token=<PAGE_ACCESS_TOKEN>"



And this way, You can use above example in our custom action file :

       gt = {
            "attachment": {
                "type": "template",
                "payload": {
                    "template_type": "generic",
                    "elements": [
                        {
                            "title": "Welcome! 1",
                            "image_url": "https://picsum.photos/200",
                            "subtitle": "We have the right hat for everyone.",
                            "default_action": {
                                "type": "web_url",
                                "url": "https://tithal.life",
                                "webview_height_ratio": "tall",
                            },
                            "buttons": [
                                {
                                    "type": "web_url",
                                    "url": "https://tithal.life",
                                    "title": "View Website"
                                },
                                {
                                    "type": "postback",
                                    "title": "Start Chatting",
                                    "payload": "DEVELOPER_DEFINED_PAYLOAD"
                                }
                            ]
                        },
                        {
                            "title": "Welcome! 2",
                            "image_url": "https://picsum.photos/200",
                            "subtitle": "We have the right hat for everyone.",
                            "default_action": {
                                "type": "web_url",
                                "url": "https://tithal.life",
                                "webview_height_ratio": "tall",
                            },
                            "buttons": [
                                {
                                    "type": "web_url",
                                    "url": "https://tithal.life",
                                    "title": "View Website"
                                },
                                {
                                    "type": "postback",
                                    "title": "Start Chatting",
                                    "payload": "DEVELOPER_DEFINED_PAYLOAD"
                                }
                            ]
                        }
                    ]
                }
            }
        }
        dispatcher.utter_custom_json(gt)

Compare both code and anyone can figure it out. Thanks :+1: :tada::tada:

1 Like

Hey @Mgo2 After some debugging a while ago, we figured that what is missing in Rasa documentation is the mandatory data: layer in between customer and actual values :slight_smile:

So ya, that’s the route that’'s cleanest, regarding if we go with the template definition or create the utterance via action code.

@erohmensing Would be good for Rasa to update the Custom action documentation, especially as an example for websocket based chat ui. thank you

What do you mean by this?