Issues with S3 Remote Storage

Rasa Open Source
1

Having issues with using S3 remote storage:

Environmental variables have been set:

export AWS_SECRET_ACCESS_KEY=<MY_SECRET>
export AWS_ACCESS_KEY_ID=<MY_ACCESS>
export AWS_DEFAULT_REGION=<MY_REGION>
export BUCKET_NAME=<MY_BUCKET>
export AWS_ENDPOINT_URL=https://<MY_BUCKET>.s3-<MY_REGION>.amazonaws.com

Now when I try to start the rasa server, a file <MY_BUCKET> is created in the bucket, ie, of the following address:

https://<MY_BUCKET>.s3-<MY_REGION>.amazonaws.com/<MY_BUCKET>

Which is strange to me that the bucket name is the same as the filename.

Anyway, the command I am running is as follows:

rasa run -m models --remote-storage aws --debug --endpoints ./endpoints.yml --credentials ./credentials.yml --enable-api

I have also tried it without the -m models directory being specified:

rasa run --remote-storage aws --debug --endpoints ./endpoints.yml --credentials ./credentials.yml --enable-api

I have also tried it with/without the --enable-api argument, and, with/without various *tar.gz model files already pre-uploaded into the bucket.

In all instances, the program crashes with a whole bunch of error code, including the following:

2019-08-19 07:29:58 DEBUG    s3transfer.tasks  - Exception raised.
Traceback (most recent call last):
  File "/Users/nick/Projects/Python/rasa-x/env/lib/python3.7/site-packages/s3transfer/tasks.py", line 255, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/Users/nick/Projects/Python/rasa-x/env/lib/python3.7/site-packages/s3transfer/download.py", line 345, in _submit
    **transfer_future.meta.call_args.extra_args
  File "/Users/nick/Projects/Python/rasa-x/env/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/nick/Projects/Python/rasa-x/env/lib/python3.7/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found
2019-08-19 07:29:58 DEBUG    s3transfer.utils  - Releasing acquire 0/None
2019-08-19 07:29:58 ERROR    rasa.core.agent  - Could not load model due to An error occurred (404) when calling the HeadObject operation: Not Found.

Running the above without the --remote-storage argument, everything works as expected, with the obvious exception of using locally stored models, rather than the desired S3 stored model.

So all these kinds of bugs leads me to ask the question, has anyone actually scaled a rasa model?

2

I’m curious if you ever got further on this, I seem to be hitting similar issues when trying to specify the model file that is in my s3 for remote storage as well. I’m getting list index out of range on mine.

3

Not sure it helps (it’s been some time since this issue was posted in the forum :sweat_smile:) but I’ve run into the exact same error after trying those alternatives / running those commands too.

Somehow after running with this command, it works fine:

rasa run -m 20191105-174239.tar.gz --remote-storage aws --enable-api --credentials /rasa/credentials.yml --endpoints /rasa/endpoints.yml --log-file out.log

The model had to be specified with the -m argument, and apparently this also helped:

  • creating in the bucket a folder with the same name as <MY_BUCKET>
  • uploading my current model there

The model was successfully fetched from storage and it finally stopped creating that <MY_BUCKET> file

1 Like
4

Joining the conversation a little late but I’m struggling with this issue as well. Tried a lot of different alternatives in this forum and in github but I can’t seem to get it to work.

Just wondering what version of Rasa you are using or if you have any more advice to solve this issue.

5

Haha I got it! If anyone is curious, I was using the wrong endpoint (follow the format above and you should be fine) which results in a 403 error. After fixing that, a file gets created named <MY_REGION> (I’m using rasa 1.10.3) and then I get a 404 error. So I placed the model in a folder called <MY_REGION> in my s3 bucket and it worked :slight_smile:

1 Like
6

Hi all,

I am also encountering the same issue. My endpoint is export AWS_ENDPOINT_URL=https://<MY_BUCKET>.s3-<MY_REGION>.amazonaws.com

A new file name <MY_BUCKET> is created instead and I followed some of the advice here (creating a folder named <MY_BUCKET> and put my model in there) but it kept creating the file instead. I have also tried creating the folder but to no success.

Would appreciate if anyone can provide me with some advice on this.

Additionally, I would like to check if the model will be updated without restarting the server if a new model is uploaded?

Thanks!

[Updates]

I have successfully loaded the model from S3! It turns out that I have been using the wrong model name. I realized that I could not make RASA pull the model every interval (similar to what the model server is doing). Is there anyway to automate the pulling of model instead of restarting the bot everytime?

1 Like
7

Glad you figured it out. We use the remote storage on S3 as well and deploy to AWS with ECS and ECR and use Fargate. I was this close || from chiming in.

I know there is a “load from server” function, but I’ve not tried to do this with an S3 url

I would love if this auto-loading worked, because I have to change the model filename in our deployment system each time I redeploy

But like I said I haven’t tried it with the remote-storage setup