Rasa worker can't find model tag

I am using rasa 2.8.1, rasa-x 0.42.0 deployed using the official helm chart. The following issue started happening randomly and now I can’t get it to work. Nothing has changed from when it worked to now.

When training and activating a model in Rasa-X the model seems to activate but when analyzing the conversations they all go like this:

It is using every word or phrase as the intent and it never answers.
The model is 100% functional as I can train it and use it on my computer via rasa shell:

(venv) $ rasa shell models/20210817-182209.tar.gz 
2021-08-17 18:23:29 INFO     rasa.model  - Loading model models/20210817-182209.tar.gz...
2021-08-17 18:23:31 INFO     root  - Starting Rasa server on http://localhost:5005
2021-08-17 18:23:31 INFO     rasa.model  - Loading model models/20210817-182209.tar.gz...
/home/jsolis/Desktop/venv/lib/python3.8/site-packages/rasa/utils/train_utils.py:641: UserWarning: constrain_similarities is set to `False`. It is recommended to set it to `True` when using cross-entropy loss. It will be set to `True` by default, Rasa Open Source 3.0.0 onwards.
  rasa.shared.utils.io.raise_warning(
2021-08-17 18:23:39 WARNING  rasa.core.policies.ted_policy  - Failed to load type model. Path '/tmp/tmpa5v68dzp/core/policy_1_TEDPolicy' doesn't exist.
/home/jsolis/Desktop/venv/lib/python3.8/site-packages/rasa/shared/utils/io.py:97: UserWarning: the value of 'evaluate_every_number_of_epochs' is greater than the value of 'epochs'. No evaluation will occur.
2021-08-17 18:23:39 INFO     root  - Rasa server is up and running.
Bot loaded. Type a message and press enter (use '/stop' to exit): 
Your input ->  hi                                                                                                                                                 
Hey there.

After enabling debug mode I get these logs every 10 seconds:

2021-08-18 00:30:12 DEBUG    rasa.core.agent  - Model server could not find a model at the requested endpoint 'http://k8s-rasa-rasa-x-rasa-x.rasa.svc:5002/api/projects/default/models/tags/production'. It's possible that no model has been trained, or that the requested tag hasn't been assigned.

2021-08-18 00:30:12 DEBUG    rasa.core.agent  - No new model found at URL http://k8s-rasa-rasa-x-rasa-x.rasa.svc:5002/api/projects/default/models/tags/production

When hitting the endpoint api/projects/default/models I get this which clearly shows the model trained and tagged:

[{"hash":"cdda3350bde594061531191f10f5590a",
"model":"20210817-233608",
"path":"\/app\/models\/20210817-233608.tar.gz",
"project":"default",
"trained_at":1629243368.0236799717,
"version":"2.8.1",
"tags":["production"],
"is_compatible":true}]

but when hitting this endpoint /api/projects/default/models/tags/production I get:

{"version":"0.42.0",
"status":"failure",
"message":"Tag 'production' not found for project 'default'.",
"reason":"TagNotFound",
"details":{},
"help":null,"code":404}

As I mentioned before no changes were made before it broke. I saw it working with the same version for both rasa and rasa-x today.