Issue running Rasa Open Source on Google Cloud Run: Could not load model due to [Errno 39] Directory not empty

Rasa Open Source
1

Hi,

I’m trying to deploy Rasa Open Source on Cloud Run and use it’s webhook REST endpoints

I did a rasa init with the following command:

docker run -v $(pwd):/app rasa/rasa:3.0.6 init --no-prompt

Then, created a Dockerfile with following contents:

FROM rasa/rasa:3.0.6
USER root
COPY . /app
WORKDIR /app
USER 1001
RUN rasa train
ENTRYPOINT ["sh", "-c", "rasa run -p $PORT"]
CMD ["sh"]

On my local Ubuntu 18.04 when I run this docker I see no error.

Running on local:

docker run -e PORT=8080 -p 8080:8080 babblebots/rasa-simple:latest

I can call the webhook endpoint as follows:

curl --location --request POST 'http://localhost:8080/webhooks/rest/webhook' \
> --header 'Content-Type: application/json' \
> --data-raw '{
>   "sender": "test_user",
>   "message": "Hi"
> }'
[{"recipient_id":"test_user","text":"Hey! How are you?"}]

After deploying to cloud run the behaviour is different.

I see following error in logs:

ERROR rasa.core.agent - Could not load model due to [Errno 39] Directory not empty: '/tmp/tmpfnrvrdqf'.
INFO root - Rasa server is up and running.

On doing the same REST call on the cloud run app I get empty response with following message in logs:

INFO rasa.core.agent - Ignoring message as there is no agent to handle it.

Any insight here would be valuable. Thanks in advance.

2

For Demo:

FROM rasa/rasa:3.0.6
WORKDIR  '/app'
COPY . /app
USER root

RUN  rasa train 

VOLUME /app/models


CMD [ "run","-m","/app/models","--enable-api","--cors","*","--debug" ,"--endpoints", "endpoints.yml", "--log-file", "out.log", "--debug"]

EXPOSE 5005

update you port, I mentioned the generic port.

3

I changed the Dockerfile like above. Running cloud run instance on port 5005 now. This did not change the outcome though

2022-02-07T05:16:37.363167Z2022-02-07 05:16:37 ERROR rasa.core.agent - Could not load model due to [Errno 39] Directory not empty: '/tmp/tmpxjw4yz_2'.
2022-02-07T05:16:37.363180Z2022-02-07 05:16:37 INFO root - Rasa server is up and running.
...
2022-02-07T05:19:04.914524Z2022-02-07 05:19:04 INFO rasa.core.agent - Ignoring message as there is no agent to handle it.

Still getting empty response on calling webhook API

4

@jsingh check on the docker that you were able to mount the volume or not and able to see all the data?

5

It’s difficult (maybe even not possible), to exec into the docker container running on Google cloud run. Not sure how that can be done.

As I said earlier, when the same docker runs on my Ubuntu 18.04 everything works fine.

The following error:

ERROR rasa.core.agent - Could not load model due to [Errno 39] Directory not empty: '/tmp/tmpxjw4yz_2'.

Tells me there is a requirement for /tmp/ folder to be empty but somehow in cloud run it’s not. I have also tried to empty the /tmp/ folder just before doing ‘rasa run’ but still got the same error.

6

@jsingh Try following these steps on VM: Dockerizing my rasa chatbot application that has botfront - #11 by nik202 and let me know is it working or not.

7

Thanks @nik202 . Appreciate your detailed response.

I am able to run my Docker instance on Google Compute Engine VM so taking that route for now. Interestingly cloud run also requires more memory than normal GCE VM.

I suppose this problem is specific to cloud run environment only. So if anyone had tried deploying Rasa on cloud run maybe able to point out the issue here.

8

@jsingh Thanks for the. feedback, can I request you to please close this thread as solution.

9

Ok. In summary, cloud run does not seem to be compatible with Rasa for now, unless somebody can solve this one.

10

I’ve stumbled upon the same issue. Anyone found a way to run it on Cloud Run?

It’s such a bummer to require a whole Compute Instance only for these tmp folder issues.

11

I deployed Rasa on Cloud Run, this is my Dockerfile:

FROM rasa/rasa:3.5.2-full as step1

USER root
 
COPY ./app/ /app/

RUN pip3 install --no-cache-dir spacy
RUN python -m spacy download pt_core_news_sm
RUN python -m spacy link pt_core_news_sm pt

FROM step1 as runner

COPY --from=step1 /opt/venv /opt/venv

# make sure we use the virtualenv
ENV PATH="/opt/venv/bin:$PATH"

# update permissions & change user to not run as root
WORKDIR /app
RUN chgrp -R 0 /app && chmod -R g=u /app
USER 1001

# the entry point
# EXPOSE 5005
ENTRYPOINT ["rasa", "run"]

My Cloud Run setup has port 5005 and a minimum of 2GiB of memory.