Rasa X (v.33.0) connects to GitLab but does not pull data

[Deprecated] Rasa X Community Edition
1

Version: Rasa X 0.33.0

I’ve been having trouble with Git integration with GitLab.

As part of my troubleshooting I’ve:

  • disconnected Git from Rasa X
  • made a change to my training data directly in Git
  • connected Rasa X to Git with a new deploy key

Expected results:

  • integration icon shows “green” status
  • training data is pulled from Git, replacing existing training data in Rasa X

Actual results:

  • integration icon shows “green” status
  • training data is old/out of date and does not match what is in GitLab

My install method is the Server Quick Install method (K8S cluster) with an additional environment variable only to enable the REST webhook.

I’m going to try this in an environment where I have more visibility to logs. Anyone else having this issue? How should I troubleshoot?

Edit:

  • This bot was previously running on Rasa OSS v1.x but has since been migrated to v2.x
  • In part of my troubleshooting this morning (after already experiencing my problem above) I converted the data from the markdown format to yaml format.
  • My git push included deleting the old .md files and adding the new yaml files
2

@Chris98106 could you share your logs from the rasa-x pod?

3

I found two exceptions in the logs:

[2020-11-12 10:41:57 +0000] - (sanic.access)[INFO][10.42.0.35:38674]: GET http://rasa-rasa-x-rasa-x:5002/api/projects/default/models/tags/production?token=REDACTEDTOKEN  204 0
[2020-11-12 10:41:57 +0000] - (sanic.access)[INFO][10.42.0.37:49532]: GET http://rasa-rasa-x-rasa-x:5002/api/projects/default/models/tags/production?token=REDACTEDTOKEN  204 0
Job "run_background_synchronization (trigger: cron[minute='*'], next run at: 2020-11-12 10:43:00 UTC)" raised an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/apscheduler/executors/base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
  File "/usr/local/lib/python3.7/site-packages/rasax/community/services/integrated_version_control/git_service.py", line 131, in run_background_synchronization
    loop.run_until_complete(git_service.synchronize_project(force_data_injection))
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.7/site-packages/rasax/community/services/integrated_version_control/git_service.py", line 1065, in synchronize_project
    if not self._fetch():
  File "/usr/local/lib/python3.7/site-packages/rasax/community/services/integrated_version_control/git_service.py", line 313, in _fetch
    self._repository.git.fetch(env=self._env())
  File "/usr/local/lib/python3.7/site-packages/git/cmd.py", line 542, in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/git/cmd.py", line 1005, in _call_process
    return self.execute(call, **exec_kwargs)
  File "/usr/local/lib/python3.7/site-packages/git/cmd.py", line 822, in execute
    raise GitCommandError(command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git fetch
  stderr: 'Could not create directory '/root/.ssh'.
Warning: Permanently added 'gitlab.com,172.65.251.78' (ECDSA) to the list of known hosts.
git@gitlab.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.'
[2020-11-12 10:42:01 +0000] - (sanic.access)[INFO][10.42.0.1:56844]: GET http://10.42.0.41:5002/  200 3217

Second:

[2020-11-12 08:49:56 +0000] - (sanic.access)[INFO][10.42.0.1:46290]: GET http://10.42.0.41:5002/  200 3217
[2020-11-12 08:49:57 +0000] - (sanic.access)[INFO][10.42.0.35:56398]: GET http://rasa-rasa-x-rasa-x:5002/api/projects/default/models/tags/production?token=REDACTEDTOKEN  204 0
[2020-11-12 08:49:57 +0000] - (sanic.access)[INFO][10.42.0.37:39024]: GET http://rasa-rasa-x-rasa-x:5002/api/projects/default/models/tags/production?token=REDACTEDTOKEN  204 0
Job "run_background_synchronization (trigger: cron[minute='*'], next run at: 2020-11-12 08:51:00 UTC)" raised an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/apscheduler/executors/base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
  File "/usr/local/lib/python3.7/site-packages/rasax/community/services/integrated_version_control/git_service.py", line 131, in run_background_synchronization
    loop.run_until_complete(git_service.synchronize_project(force_data_injection))
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.7/site-packages/rasax/community/services/integrated_version_control/git_service.py", line 1065, in synchronize_project
    if not self._fetch():
  File "/usr/local/lib/python3.7/site-packages/rasax/community/services/integrated_version_control/git_service.py", line 313, in _fetch
    self._repository.git.fetch(env=self._env())
  File "/usr/local/lib/python3.7/site-packages/git/cmd.py", line 542, in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/git/cmd.py", line 1005, in _call_process
    return self.execute(call, **exec_kwargs)
  File "/usr/local/lib/python3.7/site-packages/git/cmd.py", line 822, in execute
    raise GitCommandError(command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git fetch
  stderr: 'Could not create directory '/root/.ssh'.
Warning: Permanently added 'gitlab.com,172.65.251.78' (ECDSA) to the list of known hosts.
git@gitlab.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.'
[2020-11-12 08:50:01 +0000] - (sanic.access)[INFO][10.42.0.1:46338]: GET http://10.42.0.41:5002/  200 3217
[2020-11-12 08:50:06 +0000] - (sanic.access)[INFO][10.42.0.1:46380]: GET http://10.42.0.41:5002/  200 3217
4

Sorry - I missed your response - are you still having issues with it? It seems like you don’t have access to create a directory under /root/ - do you have some permission configuration on your kubernetes environment?