I was wondering why Rasa expects stories to be consistent?
Shouldn’t an AI Conversational Agent create a model even if there are some conflicting stories and or rules? Of course, assuming we’re having enough training examples.
Hypothetical, I could feed the CA with live-chats. We can not expect all chats from all live-agents to be consistent.
Good question.
Stories should be consistent because they represent the stories you want the bot to go through, not all possible user actions. Simply, you want the bot, under a certain context, to respond as you desgined it to respond.
The AI/ML part of it is adapting in case the user deviates from the story.
Exaggeration: It’s like labeling the sentence “I want a pizza” under the intent “get_weather” because the user might say that sarcastically while they in fact want the weather.
Adding to Chris suggestion; as from your this statement, if you thinking to feed all the live-conversation of bot/user or even live-agent; then it can even some time confused the model to pick the right response, or can be trained on wrong annotated data. So, the stories, dialogues and conversation should be design with more precisely for AI driven chatbot.
Chris, Nik, first thanks for joining this discussion!
Still I’m not convinced yet.
To be sure to avoid ambiguity: I should have used the term rasa used when reporting the error: contradicting rules or stories
In my opinion, law of the large numbers should deal with contradicting stories or rules: as long as most of the data is non-contradicting.
More over big chance that these unhappy paths are contradicting
In case of help-desk human-agents, most of them should be experts as you don’t want them to give customers wrong answers.
But you can’t avoid some of them make mistakes, f.e. because they are not expert yet.
So my imaginary help-desk AI CA is fed with a lot of non-contradicting stories and a few contradicting.
Can you show us what these contradicting stories are?
Contradicting stories means you’re telling the bot to do two different things in the exact same situation! If there is a case in which you want action X to be followed by action Y (happy path) or action Z (unhappy path), you need to put the deciding factor in your stories as well.
I like to see bots as humans. If I tell you to eat after receiving your meal, then I tell you not to, you will be confused. I should tell you to eat after receiving your meal if your meal smells good, then I tell you not to if it smells bad.
Chris, although I could show you some stories, it doesn’t add much to this discussion.
On the other hand, your and Nics arguments do indeed help!
So, Suppose I’am on a turning point in my life. F.e. I want to make a trip either to Iceland or to Greece. I ask 100 people what they advise me to do All those people ask me questions about what I want to see, what I like to eat and so on.
Then, 90 people advice me Greece and 10 Iceland.
Although this is a simple example and not yet machine-learning, the principle is the same.
@HermanH In terms of NLP just helping the discussion 
:
What is a contradiction?
One standard is to adopt a strict logical definition of contradiction: sentences A and B are contradictory if there is no possible world in which A and B are both true. However, for contradiction detection to be useful, a closer definition that more closely matches human intuitions is necessary; contradiction occurs when two sentences are extremely unlikely to be true simultaneously. Pairs such as Sally sold a boat to John and John sold a boat to Sally are tagged as contradictory even though it could be that each sold a boat to the other. This definition captures intuitions of incompatibility, and perfectly fits applications that seek to highlight discrepancies in descriptions of the same event.
For texts to be contradictory, they must involve the same event. Two phenomena must be considered in this determination: implied coreference and embedded texts. Given limited context, whether two entities are coreferent may be probable rather than certain. To match human intuitions, compatible noun phrases between sentences are assumed to be coreferent in the absence of clear countervailing evidence.
Hope this will help so ways!
The essence of stories is linking an input to an output. It’s just that.
If you ask someone what country you should go to, you expect them to answer by giving you a country name. Now the country name itself could have a 90% chance of being Greece and 10% chance of being Ireland - but this is a detail stories do not care about (if this is what you want to achieve, I will explain how in the second section).
If you’re the bot, you know you should answer with a country name. Intent ask_country links to action utter_country via a story/rule.
On the other hand, if you’re the user who should provide a country after action utter_ask_country, but don’t and change topic, the bot should forget about countries and adapt to respond to your message (if you want to keep asking for the country, I will explain how in the third section). The first part, utter_ask_country followed by a random other_intent should no appear in your stories, obviously; but the second part other_intent followed by a specific utter_other_reply should be a story/rule.
Now you might notice bigger stories, in which user intents do follow an action. These bigger intents are either:
Used to complete an action depending on a sequence of two (or more) intents:
- story:
- intent: hello
- action: utter_greet
- story:
- intent: goodbye
- action: utter_goodbye
- story:
- intent: hello
- action: utter_greet
- intent: goodbye
- action: utter_but_you_just_arrived
Or because they are taken/imported from real conversations with users, in which the users took different paths after a certain action (because humans can be unpredictable, or because the topics are unrelated anyway).
In this case, stories are made to reinforce the bot’s behavior and in a way tell it that it did a great job adapting to these unpredictable events. They’re in no way necessary to training and are just used to reinforce correct behavior.
You can see some examples here. None of these stories were used initially for training, it was all simple rules that link intent to action and that’s it.
About how to insert randomness in a response.
Again, with the country examples. You want the bot to reply with “Greece” 2 times out of 3 and “Ireland” 1 time out of 3. These should not be classified as different actions.
For this particular example (or more generally simply uttering something with a certain probability), you can do that in the domain by specifying the same utterance multiple times according to its probability.
When defining the text key of a response multiple times, Rasa will randomly and equally pick one of the entries (reference). So, mentioning the same entry twice will give it a twice higher chance of appearing.
utter_country:
- text: "Greece"
- text: "Ireland"
- text: "Ireland"
I don’t really find thing solution clean, especially with more choices and less practical probabilities like 61%, 37%, and 2%. I would use a Custom Action for that.
Inside the custom action, you can write whatever Python code you want. Inside its run() method, you define two lists of equal length: One for the outputs, one for their respective probabilities.weights. Then, an element gets picked using the random.choices() function:
countries = ['Greece', 'Ireland']
weights = [2, 1]
output = random.choices(population = countries, weights = weights)
dispatcher.utter_message(text = output)
return []
Stories (and rules) do not care about any of this. In their point of view, they should reply with utter_country or action_country, whatever random output they may give.
After the action’s execution, the user may say whatever they want to say, regardless of whether or not stories mention it is possible, and the bot will continue as it should.
You may want to execute completely different actions after the same intent.
In this case, use the event called FollowupAction inside a custom action. That “main” custom action is the one that should be defined in stories/rules as the action that executes as a response to a given intent.
For example, the intent random_action that the user triggers when saying “Do something random” will be linked to action_random as such:
- story:
- intent: random_action
- action: action_random
That action_random will in turn call a random action. As before, this can be used with random.choices():
actions = ['action_joke', 'action_order_pizza', 'action_weather']
weights = [3, 1, 2]
next_action = random.choices(population = actions, weights = weights)
return [FollowupAction(next_action)]
About keeping asking for a slot (information).
This is done via a Form. The bot will keep requesting a slot until it has been filled (or a stop condition that you defined has been fulfilled).
If this time, it is the bot asking the user for a country, whether the user replied with “Greece” or “Ireland”, the important part is that they come from the intent give_country, which will fill the slot (using either entity or text mapping).
If the user decided to instead go on an unrelated topic and ask about the weather, and you provided a rule which links ask_weather to action_weather, it will execute that unexpected action and go back to asking for the slot right after.
No need to mention the possibility of that happening in the stories at all. It’s practically impossible to cover all possibilities. The bot will adapt.
Stories and rules are made for linking an intent (or context) to an action.
If you want the action to be random, introduce randomness inside the action as a custom action, under the same action name.
Nik202, although your thoughts on contradictions partly addresses the discussion, at least as far as I concern, I like your definition and examples indeed!
@HermanH Glad to hear that atleast I helped you in some way 
. I guess this gave you better idea now about contradiction of stories. Can I request you please close this thread for your reference and for other readers. Good Luck and Keep safe!
Chris, thanks for your effort, I really appriciate it!
Now, some thoughts on each of your sections
Section 1, bots reaction This one addresses the bots reaction on previous turns, the reaction doesn’t only depend on the last intent. Including deviating paths, the user takes.
Section 2, Randomness I am aware of this Rasa capability.
Although it’s a nice feature to give the CA some variations in it’s utterances, I don’t think it’s what I’m aiming on.
Section 3 Forms is a great Rasa feature, I already use it.
Wrapping it all up, assuming that I have understood both Chris’and nik202 contributions correctly, I still believe, a CA should be capable of being trained on data with a small amount off contradictions in the stories.
And as far as Rasa concerns, the #end-to-end training comes close to it. I have not tried it yet, partly because Rasa X doesn’t yet support it (at least version I’m using) and partly because I haven’t the data yet. But I’m working on that!
Due to some organisational circumstances, this experience with real data, possibly containing some conflicting stories, could take a while.
So that’s why I started this discusion: getting to know thoughts on conflicting training data, from Rasa’s community members
And so, I’m realy greatfull to both of you!