I am trying to implement Object detection, navigation, and other Deep learning within the custom actions in the rasa Open Source version==2.8.0. I have set a rule and added an intent, also wrote the function completely. when I type in a user message, it is perfectly identifying the intent and even starts the execution of the custom action. But as soon as it enters the object_detection() function and the camera turns on, the rasa server immediately says timeout and stops! but on the other hand, the Actions server is executing the object_detection() function and is giving the correct desired output and working as expected. This same thing is happening with all the functions like navigation, face recognition, etc. maybe it’s because of the opening of the camera or something related to flow control.
I have attached the screenshot of the error that I am getting on the server and you can also see that the actions server is working fine in the background
You can’t block Rasa for a long running action. You’ll need to take a different approach. External events may be useful but the the capture/import of the picture would likely need to be done in the front end.
You also should try sending multimedia files through the Rasa tracker. There’s a post here on that.
Thankyou very much for your response @stephens . Actually I am a newbie and exploring things. I am working on a project where I have to create a chatbot for a blind person which can narrate the outside world so I want to implement theses deep learning models and also add microphone input and speaker output. Can you please help me getting along with that. I have all the functions ready, just a matter of implementing it using a channel like RASA. So can you please help me with how should I approach it?
Also, can you please share any resource where I can learn more about External Events, That would be very helpful.
Interesting application. I think you’ll need a layer of software in front of the channel to handle this.
The helpdesk example bot does something similar to what you need with it’s handoff feature. In that example, when the user tells rasa it they want to switch to another bot, the Rasa utterance that is sent, tells the React widget to change it’s configuration.
I assume you have some software on the computer/phone that the user interacts with anyway. This software would know to activiate the camera for instance. When the user requires object detection and says something to that effect, your rasa response would include a message to the software on the device which would handle taking the picture, etc.
Blind person use screen readers to navigate. I’m academic research, most volutes tells me prefer mensagem text format and use the screen reader installed in his device (NVDA, VoiceOver and TallBack)
For microphone input, the volutes preferred to use native mobile function for that
Stephens is right. You need develop image capture in front-end and send it to your deep learning services to identify it but the problem, is take a some time and Rasa Custom Action return timeout
I think you can try something using (Custom Connector)[Custom Connectors]
Look my example, the sound button and other accessibility screen reader software integration
I builded all this thinks in front-end using Web Content Accessibility Guidelines (WCAG)
Thankyou very much @stephens and @itsjhonny . actually I have found a way to get along with it. I bypassed the rasa shell and implemented it using a simple rest API. this way I have gotten more control over the code and it has become super easy. And the main point is the timeout() problem is solved and it is perfectly running. you can see the following image for the same.
All that is left is to host it on My Website using a GUI or something. and here I am totally doomed. being an ML person I do have very few knowledge of backend and it is making me super frustrated. Can you guys please help me in connecting it to a frontend. All the videos on YouTube and all the articles that I read. None of them was helpful and wasted my entire day. I would really appreciate it if you could guide me through that.
I’m doing master using rasa + accessibility
