I have a rasa project deployed to k8s cluster. In one commit I added few training phrases + intents, changed the config.yaml and updated some custom actions. I run the bot locally and it worked as expected. But when deployed to the cluster the bot fails to predict intents. ▶ The logs from the remote instance I suppose the prediction fails due to Failed to load ABCMeta from model storage. Resource 'train_DIETClassifier5' doesn't exist. Or Failed to load ABCMeta from model storage. Resource 'train_TEDPolicy2' doesn't exist. When I run the bot locally there are no these messages and there’s a log ‘Finished loading the model.’. Local versions: Rasa Version : 3.6.10 Minimum Compatible Version: 3.5.0 Rasa SDK Version : 3.6.2 Python Version : 3.10.7 Operating System : macOS-12.2.1-arm64-arm-64bit Remote versions: Rasa Version : 3.6.10 Minimum Compatible Version: 3.5.0 Rasa SDK Version : 3.6.2 Python Version : 3.10.6 Operating System : Linux-5.4.0-148-generic-x86_64-with-glibc2.35 My latest config - name: LexicalSyntacticFeaturizer - name: CountVectorsFeaturizer alias: "cvf-word" - name: DIETClassifier epochs: 100 constrain_similarities: true model_confidence: softmax - name: EntitySynonymMapper - name: FallbackClassifier threshold: 0.3 ambiguity_threshold: 0.02 policies: - name: RulePolicy priority: 2 enable_fallback_prediction: False - name: AugmentedMemoizationPolicy priority: 3 max_history: 10 - name: TEDPolicy priority: 1 epochs: 100 constrain_similarities: true nlu_fallback_threshold: 0.3 core_fallback_threshold: 0.3 core_fallback_action_name: action_default_fallback enable_fallback_prediction: true My previous config - name: LexicalSyntacticFeaturizer - name: CountVectorsFeaturizer alias: "cvf-word" - name: CountVectorsFeaturizer analyzer: char_wb min_ngram: 1 max_ngram: 4 - name: DIETClassifier epochs: 100 constrain_similarities: true model_confidence: softmax - name: EntitySynonymMapper - name: ResponseSelector featurizers: ["cvf-word"] epochs: 100 constrain_similarities: true model_confidence: softmax - name: FallbackClassifier threshold: 0.2 ambiguity_threshold: 0.02 policies: - name: RulePolicy enable_fallback_prediction: False - name: AugmentedMemoizationPolicy max_history: 0 - name: TEDPolicy epochs: 100 constrain_similarities: true nlu_fallback_threshold: 0.2 core_fallback_threshold: 0.2 core_fallback_action_name: action_default_fallback enable_fallback_prediction: true And it’s not just that the bot works locally and doesn’t remotely. Another thing is that I tried to retrain model with the previous config but with the rest of the latest changes and it doesn’t work exactly the same as it doesn’t work with the latest config. Although when I deploy the previous version of the code all together - it works fine. That’s confusing, cause the action code changes shouldn’t af…

Failed to load ABCMeta from model storage. Resource 'train_DIETClassifier5' doesn't exist

stephens (Greg Stephens) September 30, 2023, 2:59am 8

There’s a recently discovered issue training on arm under 2.6 and then using the model on amd64. This seems to have been introduced in 3.6.x and does not happen with 3.5.x

1 Like

Topic		Replies	Views
Rasa model deployment issue Rasa Open Source	2	276	October 5, 2023
Unable to load trained nlu model on different server/instance Rasa Open Source	2	213	August 5, 2023
Error loading model Rasa Open Source	0	177	April 5, 2024
Model Loading Error and Cache Directory Problems - Issues Upgrading Rasa from 3.4.0 to 3.6.2 Rasa Open Source	0	50	November 18, 2024
Rasa-server \| 2023-01-19 07:30:56 ERROR rasa.core.agent - Could not load model due to Error initializing graph component for node run_LanguageModelFeaturizer1 Rasa Open Source	2	480	March 7, 2023

Failed to load ABCMeta from model storage. Resource 'train_DIETClassifier5' doesn't exist

Related topics