Hi, I’m testing out my rasa model and I see that my model performs well in one channel and in another it doesn’t. Does next action prediction depend on channel?
@ChrisRahme can you throw some light here? Thanks in advance
1 Like
That’s weird, logically I don’t think it should depend on the channel.
Try the same exact conversation at least 5 times in each channel to make sure it was not by chance. Check the confidences on Rasa X as well.
sure @ChrisRahme. About confidence, can you point me to an article (if any) to understand the difference between END-TO-END level and ACTION-level evaluation. Basically I want to understand how to improve if I see less confidence in either of them.
@rctatman can you help me with identifying the difference between END-to-END accuracy and ACTION-level accuracy and what should I do to improve them
Lower confidence isn’t necessarily something you should fix. https://twitter.com/rctatman/status/1441431900698988546?s=20&t=BC4-3B3Pw7shnOaN2mo-8A
But in general end-to-end is the whole conversation & action-level is a specific turn.
1 Like
Alright, so if I have a model that is 95% accurate on action level and 68% accurate on end-to-end level, what would you say?
mmm, you might need more representative stories