Core prediction on stories not consistent with stories and slot status

Rasa Open Source
21

Thank you! I was able to get it working with Embedding Policy, where it’s much more consistent with correct predictions if i use FullDialogueTrackerFeaturizer, and less when i use MaxHistoryTrackerFeaturizer (with max_history set to 4 or 5).

However, the caveat is, I had to turn off FallbackPolicy and Memoization in order to see Embedding Policy result (if i don’t, the former always takes over). This is because, in the cases above, all the predictions are low confidence usually in the range of 10-20%.

Ideally, i’d like to use this along with Memoization and FallbackPolicy in place so the bot can be certain about what’s already in training data AND is producing medium-high confidence predictions (so Fallback can be used in low confidence predictions)

Is there any way i can increase the confidence on action predictions (other than adding more training data with new stories)?

Thanks again for your help! I really appreciate it

22

Memoization should not affect prediction, because it will only predict the action if the story is exactly like one of the training stories, which I assume what you want. You can modify core_threshold in FallbackPolicy

How many actions do you have?

23

I have almost around 500 actions

Let me explain with an example and subset of stories+domain, I have 4 intents and 1 entity of Car Model

intents:
  - cost
  - specification
  - downPayment
  - expectedDelivery

entity:
  - carModel

slots:
 carModel:
    type: categorical
    values:
        - modelX
        - cybertruck

actions:
  - cost
  - specification
  - downPayment
  - expectedDelivery
  - cost_modelX
  - specification_modelX
  - downPayment_modelX
  - expected_delivery_modelX
  - cost_cybertruck
  - specification_cybertruck
  - downPayment_cybertruck
  - expectedDelivery_cybertruck

obvious way of making this work is when both intent and entity is specified in the the utterance, for example, ‘what are the specs of cybertruck?’ or ‘when can i expect the delivery of Model X?’

But for a much better experience you cant force/expect the user to mention a car model everytime such as this conversation flow

> What are the specs of Cybertruck?
> Whats the cost of Cybertruck?
> What about the down payment of Cybertruck?
> when can i expect delivery of Cybertruck?

instead you’d like the flow of conversation from user to be

> What are the specs of Cybertruck?
> How about the cost?
> What about the down payment?
> when can i expect its delivery?

So i want to use the slot set in the first utterance to influence the next predictions (to predict cost_cybertruck without user mentioning it ), and when that same slot is not set (implying that user never mentioned an entity in previous turns) you provide generic response (thus, only predicts cost action).

The Stories exist as following:

Generic cost response story

* cost
   - cost

Generic specification response story

* specification
   - specification

Generic downPayment response story

 * downPayment
    - downPayment

Generic expectedDelivery response story

 * expectedDelivery
    - expectedDelivery

Intent + Entity story (similar stories exist for all intent+entity combinations)

 * downPayment{"carModel": "modelX"}
    - slot{"carModel": "modelX"}
    - downPayment_modelX

Story to remember slot and influence next turn predictions

 * specification{"carModel": "cybertruck"}
    - slot{"carModel": "cybertruck"}
    - specification_cybertruck
 * cost
    - cost_cybertruck
 * downPayment
    - downPayment_cybertruck
 * expectedDelivery
    - expectedDelivery_cybertruck

does my expectation of slot influence in such a scenario align with how it’s supposed to work? Thank you!

24

softmax saturates over 500 actions, that’s why confidences are so low. We have a PR to fix it: https://github.com/RasaHQ/rasa/pull/4902

regarding slots, once you set the slot, it is set for all the next steps in the dialogue until you reset it or set to different value, so the behavior would be exactly the same in both cases:

> What are the specs of Cybertruck?
> Whats the cost of Cybertruck?
> What about the down payment of Cybertruck?
> when can i expect delivery of Cybertruck?

> What are the specs of Cybertruck?
> How about the cost?
> What about the down payment?
> when can i expect its delivery?