Best approach to invoke classifier from custom action

Rasa Open Source
1

Hello!

I’m building a helpdesk chatbot to log service requests and incidents for our users. Basically, after collecting user data, I want to ask for which IT service the event should be logged, and given an IT service, which category. For example: a service request may be for Notebook (an IT service) replacement (category applicable for this IT service). Another example could be a configuration (category) for Firefox (IT service). An incident could be: HR system unavailable.

I’d like some ideas on how to best approach the identification of IT service and category. There are more than 200 IT services. One problem we have with our current ITSM solution is that it does not support synonyms, and it seems Rasa would be of great help for that. For instance: some users ask for help with Firefox refering to it by Mozilla.

Currently, I think that the best approach would be to implement a custom extract_it_service method in my form validation class. However, by using this method, I could not find an easy way to get from Rasa API a rank of most likely it_service candidates.

Any ideas?

2

Take a look at Lookup Tables for the list of IT services. You can use synonyms with lookup tables.

You can’t lookup F1 scores with entities, only intents.

3

Hello @stephens .

So basically I must trust on the lookup and their synonyms to correctly identify the IT service for intent inform_it_service?

Should I expect any difference if my slot is categorical?

4

The slot types are only for featurization so categorical doesn’t help with this issue.

I would prompt the user to confirm the IT service before using it to create/update a ticket and review your user history for conversations where the user said it was incorrect - improve iteratively.

5

Thank you for the clarifications, @stephens

I’m already prompting the user to confirm the IT service. I was struggling to find a way to ensure that it service are correctly identified, and I tought that running tests to check their f1-score would give me some clues.

Another approach I had in mind: based on the fact that 20 of my ~200 it services are responsible for 80% of the tickets, could I improve my classification testability - by knowing their f1-score - if I created intents dedicated for those top 20 it services?

Something along the lines of:

intent: request_for_top_it_service examples: | xxxxxxx xxxx

intent: request_for_2nd_it_service: examples: | xxxxxxx xxxx

and so on, up to intent for the 20th most popular it service.

6

Yes, I think that’s a good approach, try to identify the IT service based on the intent for the most common.

1 Like