Upgrading Rasa returns incorrect Intent and recognised with high confidence value

Rasa Open Source
1

Hi all,

We are using Rasa 1.1 with the ubuntu 18.04 but recently we have upgraded our servers to ubuntu 24.04 and we have to upgrade RASA also to the Rasa version 3.1. So we are using rasa to understand the patients responses wheather they want to book new appointment or are they replying for their existing appointment confirmation or cancelation. With rasa 1.1 we are getting acturate result in most of the case but by using rasa 3.1 it is very unpredictible results. I will share one example:

For the message : “Hi, my daughter Paisley Estrada has an appointment at 5:10 today but the bug bite is getting better so we would like to cancel”

So in above example patient want to cancel appointment. and following are the results from rasa 1.1.:

{"intent":{"name":"cancel","confidence":0.5234062075614929},"entities":[],"intent_ranking":[{"name":"cancel","confidence":0.5234062075614929},{"name":"confirm","confidence":0.39480534195899963},{"name":"moved","confidence":0.040613360702991486},{"name":"reschedule","confidence":0.03156549111008644},{"name":"new","confidence":0.0032740195747464895},{"name":"unmatched","confidence":0.0031129431445151567},{"name":"list","confidence":0.0014362988295033574},{"name":"sched","confidence":0.0012881943257525563},{"name":"stop","confidence":0.0004980870289728045}],"text":"Hi, my daughter Paisley Estrada has an appointment at 5:10 today but the bug bite is getting better so we would like to cancel"}

Rasa 3.1 result:

{
  "text": "Hi, my daughter Paisley Estrada has an appointment at 5:10 today but the bug bite is getting better so we would like to cancel",
  "intent": {
    "name": "moved",
    "confidence": 0.9755521416664124
  },
  "entities": [],
  "text_tokens": [
    [
      0,
      2
    ],
    [
      4,
      6
    ],
    [
      7,
      15
    ],
    [
      16,
      23
    ],
    [
      24,
      31
    ],
    [
      32,
      35
    ],
    [
      36,
      38
    ],
    [
      39,
      50
    ],
    [
      51,
      53
    ],
    [
      54,
      58
    ],
    [
      59,
      64
    ],
    [
      65,
      68
    ],
    [
      69,
      72
    ],
    [
      73,
      76
    ],
    [
      77,
      81
    ],
    [
      82,
      84
    ],
    [
      85,
      92
    ],
    [
      93,
      99
    ],
    [
      100,
      102
    ],
    [
      103,
      105
    ],
    [
      106,
      111
    ],
    [
      112,
      116
    ],
    [
      117,
      119
    ],
    [
      120,
      126
    ]
  ],
  "intent_ranking": [
    {
      "name": "moved",
      "confidence": 0.9755521416664124
    },
    {
      "name": "cancel",
      "confidence": 0.015571106225252151
    },
    {
      "name": "sched",
      "confidence": 0.0032032011076807976
    },
    {
      "name": "stop",
      "confidence": 0.0025991001166403294
    },
    {
      "name": "list",
      "confidence": 0.0011804726673290133
    },
    {
      "name": "unmatched",
      "confidence": 0.0008496760856360197
    },
    {
      "name": "new",
      "confidence": 0.0006487199570983648
    },
    {
      "name": "confirm",
      "confidence": 0.00022960604110267013
    },
    {
      "name": "reschedule",
      "confidence": 0.00016609011800028384
    }
  ],
  "response_selector": {
    "all_retrieval_intents": [],
    "default": {
      "response": {
        "responses": null,
        "confidence": 0.0,
        "intent_response_key": null,
        "utter_action": "utter_None"
      },
      "ranking": []
    }
  }
}

As you can see the different rasa 1.1 has detected as cancel with 0.5234062076 but in 3.1 it come as completely different intent “moved” and with higer confidence value 0.975552141666412.

So I would like to know if my training was not correct or pipelines are not correct. We use default pipelines only on both the versions.

Please let me know if you need more details.

Thank you.