Hello and thanks @akelad for looking into this issue.
I’m certainly sure that the tensorflow-gpu is configured correctly on my system. If I just run rasa train nlu --fixed-model-name <my-model-name> it generates the NLU model correctly without any hiccups using the GPU.
Similarly, if I run training for only the core model, it works too (TypeError: Object of type MaxHistoryTrackerFeaturizer is not JSON serializable is worked with a temporary solution provided here.).
But when I run rasa train --fixed-model-name <my-model-name>, the training starts correctly (core training works fine), nlu training works fine up to the process of training the EmbeddingIntentClassifier.
After EmbeddingIntentClassifier, it throws this error:
Attempting to fetch value instead of handling error Internal: failed to get device attribute 13 for device 0: CUDA_ERROR_UNKNOWN: unknown error
I checked regarding this error here and here but no fixes yet. Not sure if any other members are facing this issues as I haven’t seen anyone reporting about this neither in forums nor in github issues.