LanguageModelFeaturizer doesn't work with multiple sanic worker


I’m using LanguageModelFeaturizer in the training pipeline

- name: LanguageModelFeaturizer
  model_name: "bert"
  model_weights: "rasa/LaBSE"

The trained model can be load correctly (with Agent.load()), but while serving the model in sanic app, I got following following message in the log:

2022-07-06T20:50:30.601228416Z 2022-07-06 20:50:30	[DEBUG]	Attempting to release lock 140077744492064 on /opt/nlu/.cache/huggingface/transformers/b60f119db02852a2a5fb7bee76b15b5937e623ef9b243df1c6de8c7cb976dd94.f0e007221889d8582aa9b987b273083365e918107c3546a4f7db5969110c7f27.h5.lock ( filelock)
2022-07-06T20:50:30.601240916Z 2022-07-06 20:50:30	[DEBUG]	Lock 140077744492064 released on /opt/nlu/.cache/huggingface/transformers/b60f119db02852a2a5fb7bee76b15b5937e623ef9b243df1c6de8c7cb976dd94.f0e007221889d8582aa9b987b273083365e918107c3546a4f7db5969110c7f27.h5.lock ( filelock)

And when I tried to parse the utterance by using:

async def parse(request: Request):
    text = request.json.get("text")
    result = await model.parse_message(text) # model is a global variable initialized in main function
    return response.json(result)

The sanic app can not process that message with loaded model, and in the log, I can see it stuck at LanguageModelFeaturizer.process step

2022-07-06T20:48:44.922773400Z 2022-07-06 20:48:44	[DEBUG]	Node 'run_SpacyTokenizer1' running 'SpacyTokenizer.process'. ( rasa.engine.graph)
2022-07-06T20:48:44.923236406Z 2022-07-06 20:48:44	[DEBUG]	Node 'run_RegexFeaturizer2' running 'RegexFeaturizer.process'. ( rasa.engine.graph)
2022-07-06T20:48:44.924587124Z 2022-07-06 20:48:44	[DEBUG]	Node 'run_LanguageModelFeaturizer3' running 'LanguageModelFeaturizer.process'. ( rasa.engine.graph)

I tried to add model into sanic app by doing

app.ctx.model = Agent.load("model_path")

And it doesn’t work either.

This error only occurs if sanic worker >1, by setting sanic work to 1, the text can be processed correctly

my environment:

rasa 3.1.0

python 3.8

tf 2.7.3