Does next step prediction depend on channel?

Hi, I’m testing out my rasa model and I see that my model performs well in one channel and in another it doesn’t. Does next action prediction depend on channel?

@ChrisRahme can you throw some light here? Thanks in advance

1 Like

That’s weird, logically I don’t think it should depend on the channel.

Try the same exact conversation at least 5 times in each channel to make sure it was not by chance. Check the confidences on Rasa X as well.

sure @ChrisRahme. About confidence, can you point me to an article (if any) to understand the difference between END-TO-END level and ACTION-level evaluation. Basically I want to understand how to improve if I see less confidence in either of them.

@rctatman can you help me with identifying the difference between END-to-END accuracy and ACTION-level accuracy and what should I do to improve them

Lower confidence isn’t necessarily something you should fix.

But in general end-to-end is the whole conversation & action-level is a specific turn.

1 Like

Alright, so if I have a model that is 95% accurate on action level and 68% accurate on end-to-end level, what would you say?

mmm, you might need more representative stories