Looking for some advice with incorrect intent classification

Rasa Open Source
1

Hi folks,

I’m trying to address an issue I’m having with getting accurate intent detection and I’m hoping someone can offer some advice.

Essentially, I have more than one intent with similar training data. (e.g. “Jennifer rolled a 1.”) In order to ensure that it picks up the right intent at the right time, I’m using slot_was_set entries in my stories with a set of conditions that should ensure that the system will pick the right one at the right time. But it doesn’t.

Here’s my story:

- story: battle
  steps:
  - intent: Start Battle
  - action: my_custom_action
  - intent: Start Round
  - slot_was_set:
      - battle_id
  - action: my_custom_action
  - intent: Identify Play Order
  - slot_was_set:
      - is_play_order: true
      - battle_round_id
      - battle_id
  - action: my_custom_action
  - intent: Identify Battle Turn
  - slot_was_set:
      - battle_id
      - battle_round_id
  - action: my_custom_action
  - intent: Start Attack
  - slot_was_set:
      - battle_id
      - battle_round_id
  - action: my_custom_action
  - intent: Attack Roll
  - slot_was_set:
      - battle_round_attack_id
      - battle_id
      - battle_round_id

The two intents with the similar training data that get confused are “Identify Play Order” and “Attack Role”.

When the battle starts (it’s a game), all the “id” slots are null, and is_play_order is false. My custom action for “Start Round” sets the boolean slot “is_play_order” to true as well as a battle_round_id and battle_id (integer values). For my “problem utterance” of “Megan got a 7.” since is_play_order is true and battle_round_attack_id is null, it should pick Identify Play Order and not Attack Roll. But it doesn’t. It picks Attack Roll (with double the confidence).

Here’s what the slots look like in the tracker for the “Megan got a 7.” utterance BTW (so I know my custom action SlotSets are working):

"slots": {"battle_id": 19, "battle_round_id": 20, "battle_round_attack_id": null, "is_play_order": true, "session_started_metadata": null}}}

Here’s something else that’s important to note (I think). In my test scenario, there are multiple “Identify Play Order” utterances happening in sequence but it only seems to mess one of them up, and it’s one immediately after a fallback. Here’s how the conversation flow goes:

  1. Jerry rolled a 1. (correctly identified as “Identify Play Order”)
  2. What did the rest of you get? (correctly identified as an nlu_fallback)
  3. Megan got a 7. (incorrectly identified as Attack Roll)
  4. Jill got a 5. (correctly identified as “Identify Play Order”)
  5. Phil got a 2. (correctly identified as “Identify Play Order”)

Given how all of those are so similar in format and most were picked up ok, it seems like maybe the nlu_fallback caused the subsequent one to get misclassified.

It’s also worth noting that I have a nearly identical blurb in the training data of the Identify Play Order (e.g. “Mary got a 7.”) but none exactly like that in Attack Roll. (Which makes it choosing that one even more confusing).

Here’s what my slot configurations in domain.yml look like:

slots:
  battle_id:
    type: text
    initial_value: null
    auto_fill: false
    influence_conversation: true
  battle_round_id:
    type: text
    initial_value: null
    auto_fill: false
    influence_conversation: true
  is_play_order:
    type: bool
    initial_value: false
    auto_fill: false
    influence_conversation: true

Any thoughts on why that one instance is getting misclassified and what I might do to prevent it?

Thanks!

2

Hi folks,

After “sleeping on it” and reading the docs a bit more, I appear to have solved my issue here. Here’s how:

I added an intent inside the “battle” story to serve as a “battle specific” out of scope intent rather than letting the nlu_fallback handle the unrecognized utterance.

Also, but I’m not sure if this actually contributed to the solution, I put all of the intents that occur when there’s a battle_id into their own story with a checkpoint triggered by the intent that starts the battle. The rationale was that I could restrict the checkpoint to only trigger when a battle_id was present via slot_was_set rather than each individual intent within it. That part’s not working right (and I’ll be posting separately about that) but in the end, the intent classification issue I originally posted about is resolved so that’s great.

For reference, my story now looks like this:

- story: battle
  steps:
  - checkpoint: battle_started
  - intent: Battle Fallback
  - slot_was_set:
      - battle_id
  - action: my_custom_action
  - intent: Start Round
  - slot_was_set:
      - battle_id
  - action: my_custom_action
  - intent: Identify Play Order
  - slot_was_set:
      - is_play_order: true
      - battle_round_id
      - battle_id
  - action: my_custom_action
  - intent: Identify Battle Turn
  - slot_was_set:
      - battle_id
      - battle_round_id
  - action: my_custom_action
  - intent: Start Attack
  - slot_was_set:
      - battle_id
      - battle_round_id
  - action: my_custom_action
  - intent: Attack Roll
  - slot_was_set:
      - battle_round_attack_id
      - battle_id
      - battle_round_id

- story: startbattle
  steps:
  - intent: Start Battle
  - action: my_custom_action
  - checkpoint: battle_started

Cheers!

3

Suppose we implemented the Renew Subscription intent, which answers with “To renew your subscription, do this […]” to questions like “How can I renew my subscription?”. We can distinguish the intents in:

  • System Intents: common intents that are always present in every chatbot, like the Welcome and Fallback intents.
  • Response Intents: intents whose response should solve the user’s request. Later, we discover that there are two distinct renewal procedures: one in case the subscription has not expired yet, and one when the subscription has already expired. The naive solution would be to create a second intent and try to distinguish the training phrases of the two intents as much as possible. We create the Renew Subscription Expired intent, which answers with “To renew your subscription, do this […]” to questions like “How do I renew my expired subscription” . Then, we update the Renew Subscription Not Expired intent to answer with “To renew your subscription, do that […]” to questions like “How do I renew my not yet expired subscription” .
[krnt.run](https://krnl.run) [indigocard.ltd](https://indigocard.ltd)