Error in implementing sanic workers concept with rasa version 3.6.10

I am getting difficultly in implementing sanic worker concept for rasa server as well as action server. Actually for ‘rasa run’ command to run rasa server does not have config to increase the sanic workers within the commands, so I tried directly changing in rasa base code i.e for rasa server I changed in rasa/constant.py file and for action server I changed in rasa_sdk/constant.py file and then tried to run rasa server. Here is the error that I got

/usr/lib/python3.10/random.py:370: DeprecationWarning: non-integer
arguments to randrange() have been deprecated since Python 3.10 and will
be removed in a subsequent version
   return self.randrange(a, b+1)
/usr/lib/python3.10/random.py:370: DeprecationWarning: non-integer
arguments to randrange() have been deprecated since Python 3.10 and will
be removed in a subsequent version
   return self.randrange(a, b+1)
2023-11-09 07:08:33.790088: E
tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:219] failed to
create cublas handle: the library was not initialized
2023-11-09 07:08:33.790138: E
tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:222] Failure
to initialize cublas may be due to OOM (cublas needs some free memory
when you initialize it, and your deep-learning framework may have
preallocated more than its fair share), or may be because this binary
was not built with support for the GPU in your machine.
2023-11-09 07:08:33.797893: E
tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:219] failed to
create cublas handle: the library was not initialized
2023-11-09 07:08:33.797924: E
tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:222] Failure
to initialize cublas may be due to OOM (cublas needs some free memory
when you initialize it, and your deep-learning framework may have
preallocated more than its fair share), or may be because this binary
was not built with support for the GPU in your machine.
2023-11-09 07:08:33.809678: E
tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:219] failed to
create cublas handle: the library was not initialized
2023-11-09 07:08:33.809714: E
tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:222] Failure
to initialize cublas may be due to OOM (cublas needs some free memory
when you initialize it, and your deep-learning framework may have
preallocated more than its fair share), or may be because this binary
was not built with support for the GPU in your machine.
2023-11-09 07:08:33.811228: E
tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:219] failed to
create cublas handle: the library was not initialized
2023-11-09 07:08:33.811262: E
tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:222] Failure
to initialize cublas may be due to OOM (cublas needs some free memory
when you initialize it, and your deep-learning framework may have
preallocated more than its fair share), or may be because this binary
was not built with support for the GPU in your machine.
2023-11-09 07:08:33.819931: E
tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:219] failed to
create cublas handle: the library was not initialized
2023-11-09 07:08:33.819967: E
tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:222] Failure
to initialize cublas may be due to OOM (cublas needs some free memory
when you initialize it, and your deep-learning framework may have
preallocated more than its fair share), or may be because this binary
was not built with support for the GPU in your machine.
2023-11-09 07:08:34 ERROR    rasa.core.agent  - Could not load model due
to Error initializing graph component for node run_DIETClassifier5..
Traceback (most recent call last):
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/graph.py",
line 403, in _load_component
     self._component: GraphComponent = constructor(  # type:
ignore[no-redef]
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1120, in load
     return cls._load(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1151, in _load
     model = cls._load_model(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1244, in _load_model
     model = cls._load_model_class(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1275, in _load_model_class
     return cls.model_class().load(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/utils/tensorflow/models.py",
line 445, in load
     model.fit(data_generator, verbose=False)
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 70, in error_handler
     raise e.with_traceback(filtered_tb) from None
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/tensorflow/python/eager/execute.py",
line 52, in quick_execute
     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InternalError: Graph execution
error:

Detected at node 'embed_label/embed_layer_label/MatMul' defined at (most
recent call last):
     File "/home/intern/chatbot/venv/bin/rasa", line 8, in <module>
       sys.exit(main())
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/__main__.py",
line 133, in main
       cmdline_arguments.func(cmdline_arguments)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/cli/run.py",
line 93, in run
       rasa.run(**vars(args))
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/api.py",
line 56, in run
       rasa.core.run.serve_application(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/run.py",
line 231, in serve_application
       app.run(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/app.py",
line 1206, in run
       serve_multiple(server_settings, workers)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/server/runners.py",
line 264, in serve_multiple
       process.start()
     File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in
start
       self._popen = self._Popen(self)
     File "/usr/lib/python3.10/multiprocessing/context.py", line 281, in
_Popen
       return Popen(process_obj)
     File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19,
in __init__
       self._launch(process_obj)
     File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 71,
in _launch
       code = process_obj._bootstrap(parent_sentinel=child_r)
     File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in
_bootstrap
       self.run()
     File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
       self._target(*self._args, **self._kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/server/runners.py",
line 130, in serve
       loop.run_until_complete(app._server_event("init", "before"))
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/app.py",
line 2000, in _server_event
       await self.dispatch(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/signals.py",
line 191, in dispatch
       return await dispatch
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/signals.py",
line 161, in _dispatch
       retval = await maybe_coroutine
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/app.py",
line 1524, in _listener
       await maybe_coro
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/run.py",
line 253, in load_agent_on_start
       app.ctx.agent = await agent.load_agent(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/agent.py",
line 254, in load_agent
       agent.load_model(model_path)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/agent.py",
line 352, in load_model
       self.processor = MessageProcessor(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/processor.py",
line 101, in __init__
       self.model_filename, self.model_metadata, self.graph_runner =
self._load_model(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/processor.py",
line 138, in _load_model
       metadata, runner = loader.load_predict_graph_runner(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/loader.py",
line 29, in load_predict_graph_runner
       runner = graph_runner_class.create(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 51, in create
       return cls(graph_schema, model_storage, execution_context, hooks)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 37, in __init__
       self._instantiated_nodes: Dict[Text, GraphNode] =
self._instantiate_nodes(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 60, in _instantiate_nodes
       return {
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 61, in <dictcomp>
       node_name: GraphNode.from_schema_node(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/graph.py",
line 566, in from_schema_node
       return cls(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/graph.py",
line 392, in __init__
       self._load_component()
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/graph.py",
line 403, in _load_component
       self._component: GraphComponent = constructor(  # type:
ignore[no-redef]
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1120, in load
       return cls._load(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1151, in _load
       model = cls._load_model(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1244, in _load_model
       model = cls._load_model_class(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1275, in _load_model_class
       return cls.model_class().load(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/utils/tensorflow/models.py",
line 445, in load
       model.fit(data_generator, verbose=False)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/training.py",
line 1685, in fit
       tmp_logs = self.train_function(iterator)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/training.py",
line 1284, in train_function
       return step_function(self, iterator)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/training.py",
line 1268, in step_function
       outputs = model.distribute_strategy.run(run_step, args=(data,))
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/training.py",
line 1249, in run_step
       outputs = model.train_step(data)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/utils/tensorflow/models.py",
line 161, in train_step
       prediction_loss = self.batch_loss(batch_in)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1644, in batch_loss
       loss = self._batch_loss_intent(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1683, in _batch_loss_intent
       loss, acc = self._calculate_label_loss(sentence_vector, label,
label_ids)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1581, in _calculate_label_loss
       all_label_ids, all_labels_embed = self._create_all_labels()
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1532, in _create_all_labels
       all_labels_embed = self._tf_layers[f"embed.{LABEL}"](x)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/base_layer.py",
line 1145, in __call__
       outputs = call_fn(inputs, *args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 96, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/utils/tensorflow/layers.py",
line 463, in call
       x = self._dense(x)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/base_layer.py",
line 1145, in __call__
       outputs = call_fn(inputs, *args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 96, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/layers/core/dense.py",
line 241, in call
       outputs = tf.matmul(a=inputs, b=self.kernel)
Node: 'embed_label/embed_layer_label/MatMul'
Detected at node 'embed_label/embed_layer_label/MatMul' defined at (most
recent call last):
     File "/home/intern/chatbot/venv/bin/rasa", line 8, in <module>
       sys.exit(main())
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/__main__.py",
line 133, in main
       cmdline_arguments.func(cmdline_arguments)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/cli/run.py",
line 93, in run
       rasa.run(**vars(args))
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/api.py",
line 56, in run
       rasa.core.run.serve_application(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/run.py",
line 231, in serve_application
       app.run(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/app.py",
line 1206, in run
       serve_multiple(server_settings, workers)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/server/runners.py",
line 264, in serve_multiple
       process.start()
     File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in
start
       self._popen = self._Popen(self)
     File "/usr/lib/python3.10/multiprocessing/context.py", line 281, in
_Popen
       return Popen(process_obj)
     File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19,
in __init__
       self._launch(process_obj)
     File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 71,
in _launch
       code = process_obj._bootstrap(parent_sentinel=child_r)
     File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in
_bootstrap
       self.run()
     File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
       self._target(*self._args, **self._kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/server/runners.py",
line 130, in serve
       loop.run_until_complete(app._server_event("init", "before"))
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/app.py",
line 2000, in _server_event
       await self.dispatch(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/signals.py",
line 191, in dispatch
       return await dispatch
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/signals.py",
line 161, in _dispatch
       retval = await maybe_coroutine
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/app.py",
line 1524, in _listener
       await maybe_coro
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/run.py",
line 253, in load_agent_on_start
       app.ctx.agent = await agent.load_agent(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/agent.py",
line 254, in load_agent
       agent.load_model(model_path)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/agent.py",
line 352, in load_model
       self.processor = MessageProcessor(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/processor.py",
line 101, in __init__
       self.model_filename, self.model_metadata, self.graph_runner =
self._load_model(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/processor.py",
line 138, in _load_model
       metadata, runner = loader.load_predict_graph_runner(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/loader.py",
line 29, in load_predict_graph_runner
       runner = graph_runner_class.create(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 51, in create
       return cls(graph_schema, model_storage, execution_context, hooks)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 37, in __init__
       self._instantiated_nodes: Dict[Text, GraphNode] =
self._instantiate_nodes(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 60, in _instantiate_nodes
       return {
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 61, in <dictcomp>
       node_name: GraphNode.from_schema_node(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/graph.py",
line 566, in from_schema_node
       return cls(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/graph.py",
line 392, in __init__
       self._load_component()
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/graph.py",
line 403, in _load_component
       self._component: GraphComponent = constructor(  # type:
ignore[no-redef]
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1120, in load
       return cls._load(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1151, in _load
       model = cls._load_model(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1244, in _load_model
       model = cls._load_model_class(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1275, in _load_model_class
       return cls.model_class().load(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/utils/tensorflow/models.py",
line 445, in load
       model.fit(data_generator, verbose=False)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/training.py",
line 1685, in fit
       tmp_logs = self.train_function(iterator)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/training.py",
line 1284, in train_function
       return step_function(self, iterator)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/training.py",
line 1268, in step_function
       outputs = model.distribute_strategy.run(run_step, args=(data,))
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/training.py",
line 1249, in run_step
       outputs = model.train_step(data)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/utils/tensorflow/models.py",
line 161, in train_step
       prediction_loss = self.batch_loss(batch_in)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1644, in batch_loss
       loss = self._batch_loss_intent(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1683, in _batch_loss_intent
       loss, acc = self._calculate_label_loss(sentence_vector, label,
label_ids)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1581, in _calculate_label_loss
       all_label_ids, all_labels_embed = self._create_all_labels()
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1532, in _create_all_labels
       all_labels_embed = self._tf_layers[f"embed.{LABEL}"](x)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/base_layer.py",
line 1145, in __call__
       outputs = call_fn(inputs, *args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 96, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/utils/tensorflow/layers.py",
line 463, in call
       x = self._dense(x)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/base_layer.py",
line 1145, in __call__
       outputs = call_fn(inputs, *args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 96, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/layers/core/dense.py",
line 241, in call
       outputs = tf.matmul(a=inputs, b=self.kernel)
Node: 'embed_label/embed_layer_label/MatMul'
2 root error(s) found.
   (0) INTERNAL:  Attempting to perform BLAS operation using
StreamExecutor without BLAS support
          [[{{node embed_label/embed_layer_label/MatMul}}]]
[[crf/cond/StatefulPartitionedCall/crf/cond/else/_236/crf/cond/Maximum/_312]]
   (1) INTERNAL:  Attempting to perform BLAS operation using
StreamExecutor without BLAS support
          [[{{node embed_label/embed_layer_label/MatMul}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_50041]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/agent.py",
line 254, in load_agent
     agent.load_model(model_path)
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/agent.py",
line 352, in load_model
     self.processor = MessageProcessor(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/processor.py",
line 101, in __init__
     self.model_filename, self.model_metadata, self.graph_runner =
self._load_model(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/processor.py",
line 138, in _load_model
     metadata, runner = loader.load_predict_graph_runner(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/loader.py",
line 29, in load_predict_graph_runner
     runner = graph_runner_class.create(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 51, in create
     return cls(graph_schema, model_storage, execution_context, hooks)
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 37, in __init__
     self._instantiated_nodes: Dict[Text, GraphNode] =
self._instantiate_nodes(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 60, in _instantiate_nodes
     return {
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 61, in <dictcomp>
     node_name: GraphNode.from_schema_node(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/graph.py",
line 566, in from_schema_node
     return cls(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/graph.py",
line 392, in __init__
     self._load_component()
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/graph.py",
line 416, in _load_component
     raise GraphComponentException(
rasa.engine.exceptions.GraphComponentException: Error initializing graph
component for node run_DIETClassifier5.
2023-11-09 07:08:34 ERROR    rasa.core.agent  - Could not load model due
to Error initializing graph component for node run_DIETClassifier5..
Traceback (most recent call last):
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/graph.py",
line 403, in _load_component
     self._component: GraphComponent = constructor(  # type:
ignore[no-redef]
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1120, in load
     return cls._load(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1151, in _load
     model = cls._load_model(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1244, in _load_model
     model = cls._load_model_class(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1275, in _load_model_class
     return cls.model_class().load(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/utils/tensorflow/models.py",
line 445, in load
     model.fit(data_generator, verbose=False)
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 70, in error_handler
     raise e.with_traceback(filtered_tb) from None
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/tensorflow/python/eager/execute.py",
line 52, in quick_execute
     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InternalError: Graph execution
error:

Detected at node
'rasa_sequence_layer_text/text_encoder/randomly_connected_dense/Tensordot/MatMul'
defined at (most recent call last):
     File "/home/intern/chatbot/venv/bin/rasa", line 8, in <module>
       sys.exit(main())
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/__main__.py",
line 133, in main
       cmdline_arguments.func(cmdline_arguments)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/cli/run.py",
line 93, in run
       rasa.run(**vars(args))
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/api.py",
line 56, in run
       rasa.core.run.serve_application(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/run.py",
line 231, in serve_application
       app.run(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/app.py",
line 1206, in run
       serve_multiple(server_settings, workers)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/server/runners.py",
line 264, in serve_multiple
       process.start()
     File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in
start
       self._popen = self._Popen(self)
     File "/usr/lib/python3.10/multiprocessing/context.py", line 281, in
_Popen
       return Popen(process_obj)
     File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19,
in __init__
       self._launch(process_obj)
     File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 71,
in _launch
       code = process_obj._bootstrap(parent_sentinel=child_r)
     File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in
_bootstrap
       self.run()
     File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
       self._target(*self._args, **self._kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/server/runners.py",
line 130, in serve
       loop.run_until_complete(app._server_event("init", "before"))
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/app.py",
line 2000, in _server_event
       await self.dispatch(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/signals.py",
line 191, in dispatch
       return await dispatch
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/signals.py",
line 161, in _dispatch
       retval = await maybe_coroutine
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/app.py",
line 1524, in _listener
       await maybe_coro
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/run.py",
line 253, in load_agent_on_start
       app.ctx.agent = await agent.load_agent(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/agent.py",
line 254, in load_agent
       agent.load_model(model_path)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/agent.py",
line 352, in load_model
       self.processor = MessageProcessor(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/processor.py",
line 101, in __init__
       self.model_filename, self.model_metadata, self.graph_runner =
self._load_model(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/processor.py",
line 138, in _load_model
       metadata, runner = loader.load_predict_graph_runner(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/loader.py",
line 29, in load_predict_graph_runner
       runner = graph_runner_class.create(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 51, in create
       return cls(graph_schema, model_storage, execution_context, hooks)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 37, in __init__
       self._instantiated_nodes: Dict[Text, GraphNode] =
self._instantiate_nodes(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 60, in _instantiate_nodes
       return {
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 61, in <dictcomp>
       node_name: GraphNode.from_schema_node(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/graph.py",
line 566, in from_schema_node
       return cls(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/graph.py",
line 392, in __init__
       self._load_component()
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/graph.py",
line 403, in _load_component
       self._component: GraphComponent = constructor(  # type:
ignore[no-redef]
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1120, in load
       return cls._load(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1151, in _load
       model = cls._load_model(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1244, in _load_model
       model = cls._load_model_class(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1275, in _load_model_class
       return cls.model_class().load(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/utils/tensorflow/models.py",
line 445, in load
       model.fit(data_generator, verbose=False)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/training.py",
line 1685, in fit
       tmp_logs = self.train_function(iterator)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/training.py",
line 1284, in train_function
       return step_function(self, iterator)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/training.py",
line 1268, in step_function
       outputs = model.distribute_strategy.run(run_step, args=(data,))
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/training.py",
line 1249, in run_step
       outputs = model.train_step(data)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/utils/tensorflow/models.py",
line 161, in train_step
       prediction_loss = self.batch_loss(batch_in)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1614, in batch_loss
       ) = self._tf_layers[f"sequence_layer.{self.text_name}"](
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/base_layer.py",
line 1145, in __call__
       outputs = call_fn(inputs, *args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 96, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/utils/tensorflow/rasa_layers.py",
line 1030, in call
       if self._has_transformer:
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/utils/tensorflow/rasa_layers.py",
line 1032, in call
       outputs, attention_weights = self._tf_layers[self.TRANSFORMER](
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/base_layer.py",
line 1145, in __call__
       outputs = call_fn(inputs, *args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 96, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/utils/tensorflow/transformer.py",
line 611, in call
       x = self._embedding(x)  # (batch_size, length, units)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/base_layer.py",
line 1145, in __call__
       outputs = call_fn(inputs, *args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 96, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/utils/tensorflow/layers.py",
line 368, in call
       return super().call(inputs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/layers/core/dense.py",
line 244, in call
       outputs = tf.tensordot(inputs, self.kernel, [[rank - 1], [0]])
Node:
'rasa_sequence_layer_text/text_encoder/randomly_connected_dense/Tensordot/MatMul'
Detected at node
'rasa_sequence_layer_text/text_encoder/randomly_connected_dense/Tensordot/MatMul'
defined at (most recent call last):
     File "/home/intern/chatbot/venv/bin/rasa", line 8, in <module>
       sys.exit(main())
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/__main__.py",
line 133, in main
       cmdline_arguments.func(cmdline_arguments)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/cli/run.py",
line 93, in run
       rasa.run(**vars(args))
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/api.py",
line 56, in run
       rasa.core.run.serve_application(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/run.py",
line 231, in serve_application
       app.run(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/app.py",
line 1206, in run
       serve_multiple(server_settings, workers)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/server/runners.py",
line 264, in serve_multiple
       process.start()
     File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in
start
       self._popen = self._Popen(self)
     File "/usr/lib/python3.10/multiprocessing/context.py", line 281, in
_Popen
       return Popen(process_obj)
     File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19,
in __init__
       self._launch(process_obj)
     File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 71,
in _launch
       code = process_obj._bootstrap(parent_sentinel=child_r)
     File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in
_bootstrap
       self.run()
     File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
       self._target(*self._args, **self._kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/server/runners.py",
line 130, in serve
       loop.run_until_complete(app._server_event("init", "before"))
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/app.py",
line 2000, in _server_event
       await self.dispatch(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/signals.py",
line 191, in dispatch
       return await dispatch
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/signals.py",
line 161, in _dispatch
       retval = await maybe_coroutine
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/sanic/app.py",
line 1524, in _listener
       await maybe_coro
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/run.py",
line 253, in load_agent_on_start
       app.ctx.agent = await agent.load_agent(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/agent.py",
line 254, in load_agent
       agent.load_model(model_path)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/agent.py",
line 352, in load_model
       self.processor = MessageProcessor(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/processor.py",
line 101, in __init__
       self.model_filename, self.model_metadata, self.graph_runner =
self._load_model(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/processor.py",
line 138, in _load_model
       metadata, runner = loader.load_predict_graph_runner(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/loader.py",
line 29, in load_predict_graph_runner
       runner = graph_runner_class.create(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 51, in create
       return cls(graph_schema, model_storage, execution_context, hooks)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 37, in __init__
       self._instantiated_nodes: Dict[Text, GraphNode] =
self._instantiate_nodes(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 60, in _instantiate_nodes
       return {
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 61, in <dictcomp>
       node_name: GraphNode.from_schema_node(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/graph.py",
line 566, in from_schema_node
       return cls(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/graph.py",
line 392, in __init__
       self._load_component()
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/graph.py",
line 403, in _load_component
       self._component: GraphComponent = constructor(  # type:
ignore[no-redef]
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1120, in load
       return cls._load(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1151, in _load
       model = cls._load_model(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1244, in _load_model
       model = cls._load_model_class(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1275, in _load_model_class
       return cls.model_class().load(
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/utils/tensorflow/models.py",
line 445, in load
       model.fit(data_generator, verbose=False)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/training.py",
line 1685, in fit
       tmp_logs = self.train_function(iterator)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/training.py",
line 1284, in train_function
       return step_function(self, iterator)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/training.py",
line 1268, in step_function
       outputs = model.distribute_strategy.run(run_step, args=(data,))
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/training.py",
line 1249, in run_step
       outputs = model.train_step(data)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/utils/tensorflow/models.py",
line 161, in train_step
       prediction_loss = self.batch_loss(batch_in)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/nlu/classifiers/diet_classifier.py",
line 1614, in batch_loss
       ) = self._tf_layers[f"sequence_layer.{self.text_name}"](
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/base_layer.py",
line 1145, in __call__
       outputs = call_fn(inputs, *args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 96, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/utils/tensorflow/rasa_layers.py",
line 1030, in call
       if self._has_transformer:
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/utils/tensorflow/rasa_layers.py",
line 1032, in call
       outputs, attention_weights = self._tf_layers[self.TRANSFORMER](
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/base_layer.py",
line 1145, in __call__
       outputs = call_fn(inputs, *args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 96, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/utils/tensorflow/transformer.py",
line 611, in call
       x = self._embedding(x)  # (batch_size, length, units)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 65, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/engine/base_layer.py",
line 1145, in __call__
       outputs = call_fn(inputs, *args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/utils/traceback_utils.py",
line 96, in error_handler
       return fn(*args, **kwargs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/utils/tensorflow/layers.py",
line 368, in call
       return super().call(inputs)
     File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/keras/layers/core/dense.py",
line 244, in call
       outputs = tf.tensordot(inputs, self.kernel, [[rank - 1], [0]])
Node:
'rasa_sequence_layer_text/text_encoder/randomly_connected_dense/Tensordot/MatMul'
2 root error(s) found.
   (0) INTERNAL:  Attempting to perform BLAS operation using
StreamExecutor without BLAS support
          [[{{node
rasa_sequence_layer_text/text_encoder/randomly_connected_dense/Tensordot/MatMul}}]]
[[crf/cond/StatefulPartitionedCall/crf/cond/else/_236/crf/cond/Cast/_310]]
   (1) INTERNAL:  Attempting to perform BLAS operation using
StreamExecutor without BLAS support
          [[{{node
rasa_sequence_layer_text/text_encoder/randomly_connected_dense/Tensordot/MatMul}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_50041]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/agent.py",
line 254, in load_agent
     agent.load_model(model_path)
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/agent.py",
line 352, in load_model
     self.processor = MessageProcessor(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/processor.py",
line 101, in __init__
     self.model_filename, self.model_metadata, self.graph_runner =
self._load_model(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/core/processor.py",
line 138, in _load_model
     metadata, runner = loader.load_predict_graph_runner(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/loader.py",
line 29, in load_predict_graph_runner
     runner = graph_runner_class.create(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 51, in create
     return cls(graph_schema, model_storage, execution_context, hooks)
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 37, in __init__
     self._instantiated_nodes: Dict[Text, GraphNode] =
self._instantiate_nodes(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 60, in _instantiate_nodes
     return {
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/runner/dask.py",
line 61, in <dictcomp>
     node_name: GraphNode.from_schema_node(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/graph.py",
line 566, in from_schema_node
     return cls(
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/graph.py",
line 392, in __init__
     self._load_component()
   File
"/home/intern/chatbot/venv/lib/python3.10/site-packages/rasa/engine/graph.py",
line 416, in _load_component
     raise GraphComponentException(
rasa.engine.exceptions.GraphComponentException: Error initializing graph
component for node run_DIETClassifier5.

can someone help me in resolving this or is there any other way to handle concurrent users in rasa

@choudharynamit01 - you are running out of GPU memory. How many workers are you trying to spin up. Each worker duplicates the memory of the model you are trying to load, if you have a LanguageModelFeaturizer with a fairly big BERT model, adding more sanic workers would duplicate the memory n times and you will run out of memory.

Also not sure you need a GPU to run DIET in production.

You can add additional sanic workers with the environment variable SANIC_WORKERS in the rasa server.