AWS upload to S3 after training?

mbukovy · November 19, 2020, 2:36pm

Hello,

according to this Model Storage it is possible to use AWS S3 as a model storage. Loading from S3 bucket works fine but we wonder, is it also possible to upload the model to S3 after the training? It doesn’t seem to work out-of-the box.

We have set out credentials right, model is correctly loaded, access rights for S3 write are properly set but Rasa’s still saving the models into models/ directory on disk.

We want to use it within POST /model/train API.

Please suggest, how to achieve this.

Edit: We’ve noticed this code

github.com

RasaHQ/rasa/blob/e5db1cd3838db695551842c19c1533fde48349ed/rasa/nlu/persistor.py#L172-L176


def _persist_tar(self, file_key: Text, tar_path: Text) -> None:
    """Uploads a model persisted in the `target_dir` to s3."""

    with open(tar_path, "rb") as f:
        self.s3.Object(self.bucket_name, file_key).put(Body=f)

so we guess it should be working out-of-the box, right? Then what could be wrong, when loading a model from S3 works fine for us?

Tobias_Wochinger · November 20, 2020, 12:50pm

Hi @mbukovy, unfortunately this is currently not possible. Would you mind creating a feature request for this? I think that would be great feature to have.

mbukovy · November 20, 2020, 3:07pm

Thanks @Tobias_Wochinger

github.com/RasaHQ/rasa

Feature Request - upload model to S3 after training is done

opened 03:06PM - 20 Nov 20 UTC

mbukovy

type:enhancement

type:discussion

area:rasa-oss

**Description of Problem**: According to this [https://rasa.com/docs/rasa/model-…storage](https://rasa.com/docs/rasa/model-storage) it is possible to use AWS S3 as a model storage. Loading from S3 bucket works fine, but we're missing upload right after training. **Overview of the Solution**: There should be a possibility to upload the model to S3 (or other clouds eventually) instead of saving it into models/ folder on disc when training is done. I would expect this to be automatic when AWS credentials are set (if I want to load the model from S3, I want to save into it as well), but there could also be some separated switch. Expected behavior - when training is finished (by `rasa train` command or `POST /model/train` endpoint), model is being sent into S3 automatically instead of saved to the disc. My forum thread [https://forum.rasa.com/t/aws-upload-to-s3-after-training/37156](https://forum.rasa.com/t/aws-upload-to-s3-after-training/37156) **Definition of Done**: TBD

Tobias_Wochinger · November 23, 2020, 8:58am

Nice, thanks a lot @mbukovy!

pal7mentor · March 12, 2021, 3:22pm

read this hope this will help you to upload the model to S3 in aws

Topic		Replies	Views
AWS S3 complication Rasa Open Source	4	523	July 26, 2020
Does Rasa automatically saves trained model to S3 bucket when configured? Rasa Open Source	6	1059	November 24, 2021
Automatic storage of model after training on aws Rasa Open Source	6	679	January 23, 2020
Storing rasa models into minio server Rasa Open Source	5	1164	July 6, 2020
Model not storing in to remote storage Rasa Open Source	7	859	July 23, 2020

AWS upload to S3 after training?

Related topics