Failed to get a proper response from remote server

[Deprecated] Rasa X Community Edition
1

Hi, I’m having trouble getting Rasa X running on Google Cloud Platform Kubernetes Engine.

I did the install using Helm:

# Add the repository which contains the Rasa X Helm chart
helm repo add rasa-x https://rasahq.github.io/rasa-x-helm

# Deploy Rasa X
helm install \
    --generate-name \
    --namespace <name of your namespace> \
    --values values.yml \
    rasa-x/rasa-x

rasa==1.9.5
rasa-sdk==1.9.0

$ kubectl --namespace rasa get pods
NAME                                                 READY   STATUS             RESTARTS   AGE
rasa-x-1586477260-app-64ff85c5c-x5vtz                1/1     Running            0          28m
rasa-x-1586477260-duckling-7bb45fd959-q7kqh          1/1     Running            0          28m
rasa-x-1586477260-event-service-b9b96bc69-jtzq9      1/1     Running            1          28m
rasa-x-1586477260-nginx-5fd6dc5569-rvvf6             1/1     Running            0          28m
rasa-x-1586477260-postgresql-0                       1/1     Running            0          133m
rasa-x-1586477260-rabbit-0                           1/1     Running            0          28m
rasa-x-1586477260-rasa-production-795688666b-cfngp   0/1     CrashLoopBackOff   8          28m
rasa-x-1586477260-rasa-worker-858bf7764f-69b79       0/1     CrashLoopBackOff   8          28m
rasa-x-1586477260-rasa-x-8d5ddcb6c-pjdln             1/1     Running            0          27m
rasa-x-1586477260-redis-master-0                     1/1     Running            0          133m

$ kubectl --namespace rasa logs rasa-x-1586477260-rasa-production-795688666b-cfngp
2020-04-10 02:26:30 DEBUG    rasa.cli.x  - Failed to get a proper response from remote server. Status Code: 401. Response: '{"reasons":["Authorization header not present."],"exception":"Unauthorized"}'

$ kubectl --namespace rasa logs rasa-x-1586477260-rasa-worker-858bf7764f-69b79
2020-04-10 02:31:40 DEBUG    rasa.cli.x  - Failed to get a proper response from remote server. Status Code: 401. Response: '{"reasons":["Authorization header not present."],"exception":"Unauthorized"}'

Looks like the error comes from here:

I can log into the Rasa X interface, but nothing happens when I click the “Train” button, and when I try to upload a model, I get the error:

$ curl -k -F "model=@models/20200409-115717.tar.gz" "http://XX.XX.XX.XXXX:8000/api/projects/default/models?api_token=35fdab3f9f340aad110a633deecffb2d5999871a"
{"version":"0.27.4","status":"failure","message":"See the server logs for more information.","reason":"The server encountered an internal error and cannot complete the request.","details":{},"help":null,"code":500}

Any clues as to what I did wrong? Thank you!

2

I seemed to have fixed the problem but I’ve no idea how! Rather than following the docs:

To configure the credentials, copy the section below into the values.yml

I started with values.yaml from the helm chart.

The only thing I changed was the app: section so that my action server is pulled from docker hub.

Then I logged into the rasa-x pod and updated the admin user.

I linked my GitHub repo but by then Rasa X had crashed?

ERROR:rasax.community.services.test_service:Unable to get tests from tests/conversation_tests.md:
File 'tests/conversation_tests.md' does not exist.
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<run_operation_in_single_sanic_worker.<locals>.execute() done, defined at /usr/local/lib/python3.6/site-packages/rasax/community/utils.py:760> exception=SystemExit(1,)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/rasa/utils/io.py", line 137, in read_file
    with open(filename, encoding=encoding) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmptnsqms0v/core/domain.yml'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/rasax/community/database/utils.py", line 148, in session_scope
    yield session
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service.py", line 753, in discover_models
    await model_service.discover_models()
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service.py", line 110, in discover_models
    await _run_loop(max_retries, sleep_in_seconds)
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service.py", line 96, in _run_loop
    await self._discover_models()
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service.py", line 176, in _discover_models
    _ = await self.add_model(project_id, model_name, path)
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service.py", line 282, in add_model
    await self.inject_data_from_first_model(project, path)
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service.py", line 338, in inject_data_from_first_model
    await self._inject_domain_from_model(project, domain_path)
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service.py", line 311, in _inject_domain_from_model
    data = read_yaml_file(domain_path)
  File "/usr/local/lib/python3.6/site-packages/rasa/utils/io.py", line 210, in read_yaml_file
    return read_yaml(read_file(filename, DEFAULT_ENCODING))
  File "/usr/local/lib/python3.6/site-packages/rasa/utils/io.py", line 140, in read_file
    raise ValueError(f"File '{filename}' does not exist.")
ValueError: File '/tmp/tmptnsqms0v/core/domain.yml' does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "uvloop/loop.pyx", line 436, in uvloop.loop.Loop._on_idle
  File "uvloop/cbhandles.pyx", line 90, in uvloop.loop.Handle._run
  File "uvloop/cbhandles.pyx", line 70, in uvloop.loop.Handle._run
  File "/usr/local/lib/python3.6/site-packages/rasax/community/utils.py", line 768, in execute
    await f()
  File "/usr/local/lib/python3.6/site-packages/rasax/community/services/model_service.py", line 756, in discover_models
    await model_service.mark_latest_as_production()
  File "/usr/local/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/site-packages/rasax/community/database/utils.py", line 152, in session_scope
    f"Requested PostgreSQL schema '{e}' was not found in the database. To "
  File "/usr/local/lib/python3.6/site-packages/rasa/cli/utils.py", line 241, in print_error_and_exit
    sys.exit(exit_code)
SystemExit: 1
Requested PostgreSQL schema 'File '/tmp/tmptnsqms0v/core/domain.yml' does not exist.' was not found in the database. To continue, please create the schema by running 'CREATE DATABASE File '/tmp/tmptnsqms0v/core/domain.yml' does not exist.;' or unset the
'POSTGRESQL_SCHEMA' environment variable in order to use the default schema. Exiting application.

How do you restart the service? You delete the pod of course!

kubectl delete pod <pod name> -n <namespace>

Now everything seems to be working.

Apparently system admin is 10X harder than data science and coding these days :grimacing:

3

In your initial install, it looks like you followed the steps to install the Enterprise version of our product but you were not able to install that version without an enterprise key.

You solved it by grabbing the open source repos.

Good job!

Greg

1 Like
4

Ah yes, I see. In general, I’ve found the Rasa X docs quite hard to navigate. It seems like there are options but it’s not clear what bits apply to my configuration.

Separately, it took me ages to figure out the paths parameter, which on GKE should look like this:

@@ -381,7 +381,7 @@ ingress:
   hosts:
     - host: rasa-x.example.com
       paths:
-      - /
+      - /*

Without the additional asterix, requests for static assets, e.g. /css aren’t passed through to nginx.