Question about Rasa and Mirador

Hi guys, I need to solve a problem for my thesis. I built a bot about some architectural structures of my city, and I implemented this bot in my local webpage where I instantiated Mirador with direct opening on an image. This is an example:

Now I’d like to create some interactions , like for example if I ask “what is this?” without adding the thing itself like “what is cryptoporticus?” , the bot answers in the right way according to the photo that the user clicked on. Can you help me?

you can inject context to your bot using the Rasa HTTP API by appending SlotSet event to the tracker.

Assuming here you have some kind of web front end(react, vue etc), you can take the image context, and create a slot in Rasa domain, create stories with the context of this slot value and add value in real time to the slot using the SlotSet Event, this way, the bot already has the context of what the image is about, thus any question triggered on this image page, would be about the image already.

Thank you so much for the reply! Unluckily, I still don’t understand well. I share some codes I did:

  1. this is a code I inserted in my doctype html:

  2. the function in the code is :

  3. at last I created the file mir_action.php in this way just for testing:


[ {
	"action" : 
	{
		"type" : "CHANGE_CANVAS", 
		"parameters" : 
		{
			"window_id": "main_window", 
			"argument" : "I put here the link of the first img that the user views in mirador viewer"
			}
		}
			}
		  ]
		  

Now, if I click on a photo that isn’t the first one and if I ask to bot anything, like “hallo” or any quote of my intents, instantly mirador changes photo and goes on the first one. So something works. Now I’d like to understand how to do what I asked in the topic, for example, how can I put in a slot the value of entity linked to the photo that I clicked , like u suggested in the reply ? Can you give me an example? Thank you so much

Hi , I tried to create the interaction in an other way: every time I change page on website, I trigger a function “transmit” that sends to rasa bot an intent with the link of the photo. Every time this intent comes to the bot , the intent triggers a custom action that sets the slot I need with the link of the page. But there is a problem: if I do "“rasa run actions” and “rasa run -m models --enable-api --cors “*” --debug -p 5005” the intent looks like it doesn’t come (or the custom action isn’t triggered, I don’t know). But if I close the cmd with “rasa run actions”, it works like in the photo:

Can you help me?

@luisshinken - i don’t understand the issue you mentioned.

You have two separate terminals running with each of the command, and when you turn off rasa run models it is working?

nono when I turn off the server for the custom action it works. This is the intent sent by the agent to the bot every time a user change photo in the webpage:

- intent: EXTERNAL_page_change
  examples: |
    - page change [https://parsec2.unicampania.it/cleopatra/cleodemo/summerschool/index.json/canvas/0](page_url)```

this is the action :

utter_EXTERNAL_page_change:

  • text: “You have changed page : {page_url}.”

this is the CUSTOM action:

from rasa_sdk import Action, Tracker

from rasa_sdk.executor import CollectingDispatcher

from rasa_sdk.events import SlotSet

class setTopic(Action):

def name(self):

    return "action_set_Topic"

def run(self, dispatcher, tracker, domain):

    link_page = tracker.get_slot("page_url")    

    return[SlotSet("topic", link_page)]

At last, this is the rule:

  • rule: Say that page has changed

    steps:

    • intent: EXTERNAL_page_change

    • action: action_set_Topic

    • action: utter_EXTERNAL_page_change


If I turn on the server for rasa, and the server for the custom actions, when I change photo the utter_EXTERNAL_page_change isn't triggered (it should appears in this way : "You have changed page : none." because the server for rasa action is off, but nothing, this msg doesn't appear).

If I turn on the server for rasa and turn off the server for rasa custom actions, the interaction works like in the previous photo, the problem is that the custom action is important for me to set an important slot.

Is it clear now? Thank you anyway for you reply :clap:

probably something you have done but still asking, did you retrain your bot after adding the rule?

Dont’t worry, every suggestion is welcome. Anyway yes, I did it everytime I change something in nlu , domain or any file linked to the bot

does the action server works for other actions? I am going to go through a list of diagnostics

UPDATE: I think the problem is solved:

I added the 3 lines of my rules:

- rule: Say that page has changed
  steps:
    - intent: EXTERNAL_page_change
    - action: utter_EXTERNAL_page_change
    - action: action_set_Topic

I added them to the first part of my stories:

# what is stories

- story: what_is_arch
  steps:
  - intent: EXTERNAL_page_change
  - action: utter_EXTERNAL_page_change
  - action: action_set_Topic
  - intent: what_is  
    entities:
    - topic: arch  
  - slot_was_set:
    - topic: arch
  - action: utter_what_is
  

- story: what_is_triumphal_arch
  steps:
  - intent: EXTERNAL_page_change
  - action: utter_EXTERNAL_page_change
  - action: action_set_Topic
  - intent: what_is    
    entities:
    - topic: triumphal_arch  
  - slot_was_set:
    - topic: triumphal_arch
  - action: utter_what_is



- story: what_is_conocchia
  steps:
  - intent: EXTERNAL_page_change
  - action: utter_EXTERNAL_page_change
  - action: action_set_Topic
  - intent: what_is    
    entities:
    - topic: conocchia  
  - slot_was_set:
    - topic: conocchia
  - action: utter_what_is


- story: what_is_cryptoporticus
  steps:
  - intent: EXTERNAL_page_change
  - action: utter_EXTERNAL_page_change
  - action: action_set_Topic
  - intent: what_is    
    entities:
    - topic: cryptoporticus  
  - slot_was_set:
    - topic: cryptoporticus
  - action: utter_what_is

- story: what_is_theater
  steps:
  - intent: EXTERNAL_page_change
  - action: utter_EXTERNAL_page_change
  - action: action_set_Topic
  - intent: what_is 
    entities:
    - topic: theater  
  - slot_was_set:
    - topic: theater
  - action: utter_what_is

- story: what_is_domus
  steps:
  - intent: EXTERNAL_page_change
  - action: utter_EXTERNAL_page_change
  - action: action_set_Topic
  - intent: what_is 
    entities:
    - topic: domus  
  - slot_was_set:
    - topic: domus
  - action: utter_what_is

- story: what_is_domus_confuleius
  steps:
  - intent: EXTERNAL_page_change
  - action: utter_EXTERNAL_page_change
  - action: action_set_Topic
  - intent: what_is 
    entities:
    - topic: domus_confuleius  
  - slot_was_set:
    - topic: domus_confuleius
  - action: utter_what_is

- story: what_is_domus_orti
  steps:
  - intent: EXTERNAL_page_change
  - action: utter_EXTERNAL_page_change
  - action: action_set_Topic
  - intent: what_is 
    entities:
    - topic: domus_orti  
  - slot_was_set:
    - topic: domus_orti
  - action: utter_what_is   

- story: what_is not set
  steps:
    - entities:
      - topic: other
      intent: what_is
    - slot_was_set:
        - topic: __other__
    - action: utter_ask_topic

Now everytime I change page , the bot notify me of that and if I ask “What is this?” he answers me in the right way:

Thank you so much for your support @souvikg10

Now I have a new question : I would like to create another interaction from BOT --TO–> AGENT , that is : if I am on the photo of X , but I ask the bot “What is Y?” , I’d like to change photo of Y automatically in the website. Could you help me for this interaction?

@souvikg10 Thinking about the problem, the solution I want to implement is to create a custom action in rasa that send to the webpage the link of the photo, and when the site receive the link he changes photo thanks to the link that I send with the custom action. Do you know how to do this?

@luisshinken - i suppose you answered your own question here. you can create a custom action to send your webpage the link of the photo to show.

Your manipulation of the chat is mostly related to the front end. Rasa here, plays little role, it feeds your front end widget with what to do next given an intent. so you can customize what should happen next and in this case forwarding an image url to the widget.

I solved the problem. If it can help anyone:

I created a custom action that sends a JSON to my webpage in this way :

class get_and_send_Topic(Action):

    def name(self):
        return "send_Topic"

    def run(self, dispatcher, tracker, domain):
        get_topic = tracker.get_slot("topic")
   j = [ {"action" : {"type" : "CHANGE_CANVAS", "parameters" : {"window_id": "main_window", "argument" : "URL OF THE PHOTO"}}}]

    dispatcher.utter_message(attachment = J)
return []

and in my page I acquire di url with JSON.parse

it works! Thanks anyway!