Now Available: Integrated Version Control for Rasa X

Today we’re announcing experimental access for a new feature in Rasa X: Integrated Version Control. This feature allows you to securely connect any Git-based code hosting server with Rasa X, to manage version history for training data. Currently, Integrated Version Control is available in Rasa X version 0.23.0, under an experimental feature flag.

What does it do?

Integrated Version Control creates a two-way sync with a remote Git repository that detects diffs between the training data in Rasa X and the remote repo. As you annotate conversations, add stories, and label training examples in Rasa X, you can commit and push those changes to the remote repo. You can also pull down changes from the remote to Rasa X, if changes exist on the Git server but not in Rasa X. Integrated Version Control allows developers to preserve a complete record of each incremental change made to training data.

What are the benefits?

The ability to connect with a remote Git server from within Rasa X significantly streamlines the process of using Rasa X with code hosting platforms and Continuous Integration/Continuous Delivery tooling. Developers can take advantage of a full range of workflows and automations built into and around Git, including end-to-end testing, code reviews, and branch-based development.

Our viewpoint is that industry standard best practices for building software help teams build better AI assistants. We have found the most successful product teams working in conversational AI iterate quickly and apply what they learn from user interactions back into their development process. With the workflows enabled by Integrated Version Control, developers can accelerate the process of continually improving their assistants using Rasa X.

Get Started

Integrated Version Control is compatible with any Git-based code hosting platform, including GitHub, Bitbucket, and GitLab. Get started by downloading Rasa X 0.23 and enabling the Integrated Version Control experiment in your Experiments tab. Check out the documentation for step-by-step instructions.

Share your feedback

Feedback from the community is incredibly important. Tell us what works and what you think could be improved, which tools you use for version control and CI/CD, and the use cases you want supported. We can’t wait for you to try it out.

Happy building!

15 Likes

I’m having problems creating the repo. (Integrated Version Control) I get 401 unauthorized when I try to POST

{“reasons”:[“Authorization header is invalid.”],“exception”:“Unauthorized”}

I have tried decoding the token and it is valid. I get my JSON Web Token from /auth.

My request looks like this

curl --request POST \
     --url https://<URL goes here>/api/projects/default/git_repositories \
     --header 'Authorization: <JWT goes here>' \
     --header 'content-type: application/json' \
     --data-binary @repository.json

I also tried with the API toke, then I get 500 Internal Server Error.

curl --request POST \
     --url https://<URL>/api/projects/default/git_repositories?api_token=<api token goes here> \
     --header 'content-type: application/json' \
     --data-binary @repository.json

Hi @Krogsager,

thanks for trying this out!

I get 401 unauthorized when I try to POST

Sorry, I put the wrong command in the docs. You need the Bearer prefix in the curl:

     --header 'Authorization: Bearer <JWT goes here>' \

I also tried with the API toke, then I get 500 Internal Server Error.

Do you have a server log for me? That’d be great!

JWT version first. I added ‘Bearer’ and get this response:

500 internal server error

The nginx log does not give much

127.0.0.1 - - [29/Nov/2019:15:56:23 +0100] “POST /api/projects/default/git_repositories HTTP/1.1” 500 169 “-” “curl/7.58.0”

I’m running rasa x in a docker container, and nginx on the host machine. I don’t think that nginx can give me much info on an error 500, right? It’s just the messenger.

Can you please do docker-compose logs rasa-x to get the logs from the rasa-x container?

I’m not using the official docker image so my names are a bit different. I have no /app dir

Also, the uri in the log is not https. If that makes a difference.


rasa_vejle              | 2019-11-29 14:56:46 ERROR    sanic.root  - Exception occurred while handling uri: 'http://vejlechat.itkdigital.etek.dk/api/projects/default/git_repositories'                                              
rasa_vejle              | Traceback (most recent call last):                                                                                                                                                                         
rasa_vejle              |   File "/usr/local/lib/python3.7/site-packages/sanic/app.py", line 942, in handle_request                                                                                                                  
rasa_vejle              |     response = await response                                                                                                                                                                              
rasa_vejle              |   File "/usr/local/lib/python3.7/site-packages/rasax/community/api/decorators.py", line 177, in decorated_function                                                                                         
rasa_vejle              |     return await await_and_return_response(args, kwargs, request)                                                                                                                                          
rasa_vejle              |   File "/usr/local/lib/python3.7/site-packages/rasax/community/api/decorators.py", line 107, in await_and_return_response                                                                                  
rasa_vejle              |     response = await response                                                                                                                                                                              
rasa_vejle              |   File "/usr/local/lib/python3.7/site-packages/rasax/community/api/decorators.py", line 197, in decorated_function                                                                                         
rasa_vejle              |     return await f(request, *args, **kwargs)                                                                                                                                                               
rasa_vejle              |   File "/usr/local/lib/python3.7/site-packages/rasax/community/api/blueprints/git.py", line 48, in add_repository                                                                                          
rasa_vejle              |     git_service = _git(request, project_id)                                                                                                                                                                
rasa_vejle              |   File "/usr/local/lib/python3.7/site-packages/rasax/community/api/blueprints/git.py", line 22, in _git                                                                                                    
rasa_vejle              |     repository_id=repository_id,                                                                                                                                                                           
rasa_vejle              |   File "/usr/local/lib/python3.7/site-packages/rasax/community/services/git_service.py", line 52, in __init__                                                                                              
rasa_vejle              |     io_utils.create_directory(directory_for_git_clones)                                                                                                                                                    
rasa_vejle              |   File "/usr/local/lib/python3.7/site-packages/rasa/utils/io.py", line 380, in create_directory                                                                                                            
rasa_vejle              |     os.makedirs(directory_path)                                                                                                                                                                            
rasa_vejle              |   File "/usr/local/lib/python3.7/os.py", line 211, in makedirs                                                                                                                                             
rasa_vejle              |     makedirs(head, exist_ok=exist_ok)                                                                                                                                                                      
rasa_vejle              |   File "/usr/local/lib/python3.7/os.py", line 221, in makedirs                                                                                                                                             
rasa_vejle              |     mkdir(name, mode)                                                                                                                                                                                      
rasa_vejle              | PermissionError: [Errno 13] Permission denied: '/app'  

I’m not using the official docker image so my names are a bit different. I have no /app dir

Well, that’s gonna make everything a bit complicated :thinking: What prevents you from switching to the offical one? An easy hack could be to create an /app folder as part of your docker image. You should also set an environment variable LOCAL_MODE=false so that Rasa X knows it’s not running locally

What prevents you from switching to the offical one?

I only have one server available for my containerized services. Your official docker image requires a dedicated machine. The install modifies the host machine and messes with my setup.

I lold; LOCAL_MODE=True equals not running locally :stuck_out_tongue:

I’ve added the app dir and added env var. compose logs now return:


rasa_vejle              | 2019-12-02 08:15:14 ERROR    sanic.root  - Exception occurred while handling uri: 'http://vejlechat.itkdigital.etek.dk/api/projects/default/git_repositories'                                              
rasa_vejle              | Traceback (most recent call last):                                                                                                                                                                         
rasa_vejle              |   File "/usr/local/lib/python3.7/site-packages/sanic/app.py", line 942, in handle_request                                                                                                                  
rasa_vejle              |     response = await response                                                                                                                                                                              
rasa_vejle              |   File "/usr/local/lib/python3.7/site-packages/rasax/community/api/decorators.py", line 177, in decorated_function                                                                                         
rasa_vejle              |     return await await_and_return_response(args, kwargs, request)                                                                                                                                          
rasa_vejle              |   File "/usr/local/lib/python3.7/site-packages/rasax/community/api/decorators.py", line 107, in await_and_return_response                                                                                  
rasa_vejle              |     response = await response                                                                                                                                                                              
rasa_vejle              |   File "/usr/local/lib/python3.7/site-packages/rasax/community/api/decorators.py", line 197, in decorated_function                                                                                         
rasa_vejle              |     return await f(request, *args, **kwargs)                                                                                                                                                               
rasa_vejle              |   File "/usr/local/lib/python3.7/site-packages/rasax/community/api/blueprints/git.py", line 48, in add_repository                                                                                          
rasa_vejle              |     git_service = _git(request, project_id)                                                                                                                                                                
rasa_vejle              |   File "/usr/local/lib/python3.7/site-packages/rasax/community/api/blueprints/git.py", line 22, in _git                                                                                                    
rasa_vejle              |     repository_id=repository_id,                                                                                                                                                                           
rasa_vejle              |   File "/usr/local/lib/python3.7/site-packages/rasax/community/services/git_service.py", line 52, in __init__                                                                                              
rasa_vejle              |     io_utils.create_directory(directory_for_git_clones)                                                                                                                                                    
rasa_vejle              |   File "/usr/local/lib/python3.7/site-packages/rasa/utils/io.py", line 380, in create_directory                                                                                                            
rasa_vejle              |     os.makedirs(directory_path)                                                                                                                                                                            
rasa_vejle              |   File "/usr/local/lib/python3.7/os.py", line 221, in makedirs                                                                                                                                             
rasa_vejle              |     mkdir(name, mode)                                                                                                                                                                                      
rasa_vejle              | PermissionError: [Errno 13] Permission denied: '/app/git' 

User/group ownership of the dir matches the rest of the project-

Your official docker image requires a dedicated machine.

That’s not correct. With docker-compose you can run on a single machine.

I lold; LOCAL_MODE=True equals not running locally :stuck_out_tongue:

sorry, that was a type :laughing: It’s LOCAL_MODE=false :smiley:

I’ve added the app dir and added env var. compose logs now return:

Can you make sure user 1001 or group 0 has read / write permissions to this directory?

I nearly lost my mind today. Setting the env var LOCAL_MODE=false broke rasa.

Rasa x crashes on start - missing psycopg2

Regarding the official docker image, the instructions here on Rasa X downloads and runs this ansible playbook, which I’m wary of.

@Tobias_Wochinger Why might LOCAL_MODE=false break psycopg2?

Hi @Krogsager,

there is also documentation how to use the official docker image without using the ansible script :slight_smile: Please see the docs in this section: Deploy to a Server .

I nearly lost my mind today. Setting the env var LOCAL_MODE=false broke rasa.

Rasa x crashes on start - missing psycopg2

Oh dear :grimacing: Sorry for the hard time!

Rasa X in server mode uses postgresql as a database since sqlite is not suited for a production setup. In your docker image you have to install libpq-dev and you should install rasa with pip install rasa-x[sql] --extra-index-url https://pypi.rasa.com/simple (or simply do pip install psycopg2~=2.8).

I previously tried to pip install psycopg2 but that threw another error, so I’m doing the “right” thing instead:

I did RUN pip install rasa-x[sql]==0.23.2 --extra-index-url https://pypi.rasa.com/simple in a new container but I get a new error:

File "/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py", line 126, in connect                                                                                                                                               
  conn = _connect(dsn, connection_factory=connection_factory, **kwasync)                                                                                                                                                           
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "db" to address: Temporary failure in name resolution

Should I cherry-pick something from the manual configuration?

Hi,

I was looking forward to trying out the integrated version control. But I fail at the very beginning. I can’t find the way to "select Experimental -> Integrated Version Control ". I installed rasa-x 0.23.4 using pip and run it in the local model on my machine. However, I don’t see the menu option “Experimental”. What am I doing wrong?

Thanks in advance, Lily

Hi @liliya.avdiyenko,

Integrated Version Control is only available in the server edition of Rasa X which is deployed using the docker images. In local mode, you can just git add, commit, and push your changes yourself.

Hi Tobias, Thanks, I will try the server edition.

Just passing by to share a (perhaps) useful advice. When adding your repository to Rasa X, on Step 2, there is an important step that’s missing in the docs. That is, when you paste your private key, you need to replace new lines to '\n' character as that’s needed to successfully parse the key from JSON.

Before that I was getting this error (makes it easy for people to find this out afterwards):

{
    "version": "0.23.5.dev391+g95e6e50",
    "status": "failure",
    "message": "Insufficient permissions for remote repository.",
    "reason": "RepositoryCreationFailed",
    "details": {
        "args": [
            "Given repository credentials don't provide write permissions to the repository. Please make sure the ssh key is correct and the administrator of the remote repository gave you the required permissions."
        ]
    },
    "help": null,
    "code": 422
}

conversion can be accomplished by a few lines on an IPython session on the same directory as the private key:

replaces_nl = lambda line: line.replace('\n', '\\n')
concat_replace = lambda a, b: replaces_nl(f'{a}{b}')
with open('git-deploy-key') as f:
  one_liner = reduce(concat_replace, f.readlines())
with open('git-deploy-key-oneliner', 'w') as f:
  f.write(one_liner)

seizing any opportunity to use functools :laughing:

You can now paste git-deploy-key-oneliner as a one-liner string on repository.json and follow through to sending the request to link both services (Step 3).

And after going through all steps, I had to recreate rasa-x service in order to integrations apply de-facto:

docker-compose up -d --force-recreate rasa-x

:rocket:

1 Like

I’ve been trying to activate Integrated Version Control and now I believe I overcome the problems others had so far in this thread (ie. successfully activated server mode, checked the experimental box, successfully connected to a sql tracker store, successfully uploaded the git ssh key). But the next step according to the docs would be to see the version control status on the side bar, showing sync status with the git repository. But unfortunately the Rasa-X interface did not change at all. I even tried to modify something in the training data to see if it would do anything, to no avail.

Logs are quite clean. No error message related to version control. No message when the key is uploaded. Should there be one? I start rasa with -v. When the key is uploaded the response to the client is:

{“id":1,“name”:null,“repository_url”:"git@github.com:xxxxxxxx/xxxxxxxxxxxxxxx”,“git_service”:null,“target_branch”:“master”}

The following is the only leftover warning in my setup, could this be related?

/build/lib/python3.6/site-packages/rasax/community/services/event_service.py:1006: UserWarning: Could not find ‘event_broker’ section in endpoint config file at path ‘endpoints.yml’. f"Could not find ‘{event_broker_key}’ section in "

Any clue how to troubleshoot the sidebar not appearing at all after all steps being completed?

Thanks!

I had the same error and tried to solve it the way you wrote, but I’m still getting it.

This is how my repository.json file looks like:

image

And this is the error that I get:

{"version":"0.24.1","status":"failure","message":"Insufficient permissions for remote repository.","reason":"RepositoryCreationFailed","details":{"args":["Given repository credentials don't provide write permissions to the repository. Please make sure the ssh key is correct and the administrator of the remote repository gave you the required permissions."]},"help":null,"code":422}

I don’t know what else to try…

Hey @tiziano, please make sure all these items are covered:

  1. you created a brand new SSH key-pair for this purpose
  2. this public key is registered on smartpdt-chatbot repository under “Settings → Deploy Keys
  3. that you created git-deploy-oneliner from this private key and it is still a valid key (meaning you didn’t edit anything accidentaly)

As a matter of fact, it seems to me that you are using a PEM private key format. I would definitely (go back to step 1 and create one) try using a OpenSSH key format. I’m 100% not sure about this specification but I would give it a try since worked for me.