Check Accuracy of our Chatbot

Rasa Open Source
1

I am doing a project, And I have been asked to measure the accuracy of our chatbot using Bleu or some other techniques available to do the same. If you guys can help it would be really appreciated, My Project is due in next 3 weeks

2

Hey @nikhillmishra

I’m not sure I understand the question. What is “accuracy of chatbot”? AFAIK Bleu is used to measure quality of the machine translated text.

Could you please elaborate on that?

3

We need a quantitative analysis of how precise is the output of the Chatbot. We came across the concept of BLEU. Is it possible to use BLEU in RASA?

4

Hey @nikhillmishra

I’m still not sure what the expectation is, but you can export your conversations from the chatbot database and apply any kind of analysis on top of that. We don’t a tool that would use BLEU out of the box.

5

@nikhillmishra,

I’m not sure I fully understand what it is you’re trying to measure, either. However, Rasa does have built in evaluation functions that help measure how well your Core and NLU models perform. There is also end-to-end evaluation available, that determines the accuracy of the NLU and Core predictions together.

Information on evaluating models can be found here: