Hey @BlackSwan. Well, you have some great points about the evaluation already. You should for sure first test the model using the evaluation scripts, also you can vizualise the training stories to see if the layout of the conversations covered make sense.
Once that’s covered and works well, then it’s the matter of evaluating the usefulness of your bot. Checking the conversations and evaluating if the users achieved the goal could be one metric you could use. Feedback from the users is very helpful as well (mid-conversation or at the end of the conversation).
As of feedback look: what we have implemented already is an interactive learning which is specifically built to improve the bot through the feedback provided while talking to a bot.