Incorrect Predictions: My model is predicting incorrectly and I'm not sure what to do

Rasa Open Source
1

I have a lot of slots, templates, intents and actions/forms in my domain.yml file.

I used to use rasa_core v0.12.2 and then switched to rasa_core v0.14.1. After switching, the model started to train much quicker but the very same domain and stories files used to train the model started giving incorrect predictions.

Problem Points: Suppose I have a form which asks two slots and as a way to stop the form execution, I’ve added the following story:

## Create Appointment
* create_appointment
   - create_appointment_form
* deny
   - utter_ask_continue
* stop
   - action_deactivate_form
   - reset_slots
   - utter_default

I have a couple other forms where I’ve added this way to stop the form.

The problem is that on a random story which involves two/three forms one after another, the prediction goes haywire.

E.g.

## Personal Details
* personal_details
    - personal_details_form
* create_ticket
    - create_ticket_form   <!-- predicted: utter_ask_continue -->

Why is Rasa not giving an emphasis to the intent predicted when deciding the next action? Can I add the extra bias to intents during featurization? If so, how do I go about doing that?

What I’ve tried/know:

  • I know that the Memoization Form was modified and that the Augmented Trackers no longer are used for the memoization lookup. This results in both a smaller model size and weaker predictions.

  • I didn’t notice any change in the Keras Policy Code or the featurizer.py between the two versions (0.12.2 and 0.14.1)

  • Changing the Max History in the training config.yml file. Increasing it to be around 10 when my max actions in a story is 8 seems to improve the model a bit but not much.

  • Commented this line of code to improve Memoization Policy (by allowing augmented stories to add to the lookup):

     training_trackers = [
     t
     for t in training_trackers
     if not hasattr(t, 'is_augmented') or not t.is_augmented
 ]

What I would like to know:

  • What should my approach be? Should I change my training data so that every story is concatenated with every other in a round robin fashion to improve predicitions?
  • Should I modify any code and if so which code?
  • Should I write my own policy and if so then how?
  • Should I just upgrade to Rasa 1.0.x? Is this changed or is it the same since v0.14.1?
  • Any other I’m not thinking of?