RASA NLU Intent and Entity Extraction Issue

Rasa Open Source
1

I have trained a currency convert intent with “from_currency” and “to_currency” as entites and I trained with utterances like below

- Can you convert [250](convert_amount) [euros](from_currency:EUR)
- convert [10](convert_amount) [euro](from_currency:EUR) to [usd](to_currency:USD)
- Worth of [100](convert_amount) [canadian dollar](from_currency:CAD) in [rupee](to_currency:INR)
- What is [794](convert_amount) [euros](from_currency:EUR) to [dollar](to_currency:USD)
- Worth of [10](convert_amount) [US dollars](from_currency:USD) to [yen](to_currency:YEN)
- What is [794](convert_amount) [rupee](from_currency:INR) in [US dollars](to_currency:USD)
- Worth of [10](convert_amount) [$](from_currency:USD) to [Indian rupees](to_currency:INR)
- What is [1000](convert_amount) [YEN](from_currency) in [euro](to_currency:EUR)
- Convert [1000](convert_amount) [yen](from_currency) into [euros](to_currency:EUR)
- Worth of [10](convert_amount) [euro](from_currency:EUR) in [euro](to_currency:EUR)
- Worth of [794](convert_amount) [Indian rupees](from_currency:INR) in [YEN](to_currency)
- What is [794](convert_amount) [YEN](from_currency) in [Rupee](to_currency:INR)
- Convert [1000](convert_amount)[$](from_currency:USD) into [canadian dollar](to_currency:CAD)
- What is [100](convert_amount) [US dollars](from_currency:USD) in [canadian dollars](to_currency:CAD)
- Convert [1000](convert_amount) [Indian rupees](from_currency:INR) in [dollars](to_currency:USD)
- Worth of [424](convert_amount) [rupee](from_currency:INR) in [yen](to_currency:YEN)
- Convert [10](convert_amount) [rupees](from_currency:INR) in [euros](to_currency:EUR)
- Worth of [424](convert_amount) [euros](from_currency:EUR) in [euro](to_currency:EUR)
- What is [1000](convert_amount) [Indian rupees](from_currency:INR) in [euros](to_currency:EUR)
- Worth of [100](convert_amount) [dollars](from_currency:USD) in [rupees](to_currency:INR)
- Worth of [10](convert_amount) [canadian dollars](from_currency:CAD) in [dollar](to_currency:USD)
- Worth of [794](convert_amount) [Indian rupees](from_currency:INR) in [Indian rupees](to_currency:INR)
- Convert [1000](convert_amount) [Indian rupees](from_currency:INR) into [rupee](to_currency:INR)
- Worth of [1000](convert_amount) [US dollars](from_currency:USD) in [euro](to_currency:EUR)

Now during testing when I give a statement like

“Convert 10 dollars to buses”

buses value is being taken as to_currency value, So how can we make sure wrong values like these do not get filled automatically and I tried another statement similarly

“convert donkey to monkey”

{
  "intent": {

    "name": "convert_currency",
    "confidence": 0.9697902159344353

  },

  "entities": [

    {

      "start": 18,
      "end": 24,
      "value": "monkey",
      "entity": "to_currency",
      "confidence": 0.6973585377838782,
      "extractor": "CRFEntityExtractor"

    }

  ],

  "intent_ranking": [

    {

      "name": "convert_currency",
      "confidence": 0.9697902159344353

    },

    {

      "name": "navigate_customer_fundtransfer_page",
      "confidence": 0.015523231714319201

    },

    {

      "name": "enter_data",
      "confidence": 0.008887848262814135

    },

    {

      "name": "customer_onboarding_document_checklist",
      "confidence": 0.005798704088431311

    }

  ],

  "text": "convert donkey to moneky"

}

So here even the intent classification went wrong, Please let me know how can I make sure certain statement almost similar to trained ones do not get classified wrongly like this with high confidence score.

2

Looking at your data intent classification is right. How would the algorithm know that donkey is not a currency?

One way would be to add additional intents that would contain examples with donkeys and buses and not marked entities

3

Thanks for the reply @Ghostvv. Yes its true algorithm may not know that donkey is not a currency but the problem is when I give this chatbot for testing to my friends they tried these kinds of scenarios so it started making me think, How can we make sure that certain things which are very similar to trained statements do not extract entities in the above manner. I can add more intents with this kind of examples to make them not classify wrongly, but how many such statements can we add…I think the list would be exhaustive. I am planning to use the confidence score which comes along with the entity extraction to validate the extracted entity value. what do you think…?

4

since currencies are a limited, did you try to use lookup table?

5

I have added lookup also in the nlu data

Capture

But that also did’nt work

6

you can also add a custom action to check whether extracted entity is in database

7

ok, thanks for the help.