Kubernetes fails to install from Helm Chart

[Deprecated] Rasa X Community Edition
1

Try as I might I can not get it to work. It appears unstable in that I get a new and different error each time I work. Currently, both the event and rasa x will not talk with each other. This appears to be some sort of password problem. But I am using the absolutely standard install process which worked 48 hours ago and now it does not. But then other things did not work. Currently, I can not get anything to install however stripped down an install I do.

image
image916×377 52.1 KB

The rasa-x pod has this error

jamestagg@macbook-pro-2 CenDigitalHuman % kubectl logs rasacen-rasa-x-89994fd57-jkj7w
DEBUG:__main__:Environment variable 'RUN_EVENT_CONSUMER_AS_SEPARATE_SERVICE' set to 'True', meaning Rasa X expects the event consumer to run as a separate service.
DEBUG:__main__:Starting API service.
DEBUG:rasax.community.utils:Generating JWT RSA key pair.
DEBUG:rasax.community.utils:Saved RSA private key to temporary file '/tmp/tmppd274oo9'.
DEBUG:sanic_jwt.configuration:validating provided secret
DEBUG:sanic_jwt.configuration:validating keys (if needed)
DEBUG:sanic_jwt.configuration:loading secret and/or keys (if needed)
DEBUG:rasax.community.sql_migrations:Start running schema migrations.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 303, in unique_connection
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 760, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
    self._dec_overflow()
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 129, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
    self.__connect(first_connect_check=True)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 639, in __connect
    connection = pool._invoke_creator(self)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 453, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL:  password authentication failed for user "postgres"

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/site-packages/rasax/community/server.py", line 108, in <module>
    main()
  File "/usr/local/lib/python3.6/site-packages/rasax/community/server.py", line 32, in main
    sql_migrations.run_migrations(session)
  File "/usr/local/lib/python3.6/site-packages/rasax/community/sql_migrations.py", line 25, in run_migrations
    _run_schema_migrations(session)
  File "/usr/local/lib/python3.6/site-packages/rasax/community/sql_migrations.py", line 41, in _run_schema_migrations
    _run_alembic_migration(alembic_config)
  File "/usr/local/lib/python3.6/site-packages/rasax/community/sql_migrations.py", line 65, in _run_alembic_migration
    command.upgrade(alembic_config, target_revision)
  File "/usr/local/lib/python3.6/site-packages/alembic/command.py", line 276, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.6/site-packages/alembic/script/base.py", line 475, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line 90, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.6/site-packages/alembic/util/compat.py", line 156, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.6/site-packages/rasax/community/database/schema_migrations/alembic/env.py", line 87, in <module>
    run_migrations_online()
  File "/usr/local/lib/python3.6/site-packages/rasax/community/database/schema_migrations/alembic/env.py", line 65, in run_migrations_online
    with connectable.connect() as connection:
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2193, in connect
    return self._connection_cls(self, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 103, in __init__
    else engine.raw_connection()
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2293, in raw_connection
    self.pool.unique_connection, _connection
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2266, in _wrap_pool_connect
    e, dialect, self
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1536, in _handle_dbapi_exception_noconnection
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 303, in unique_connection
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 760, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
    self._dec_overflow()
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 129, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
    self.__connect(first_connect_check=True)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 639, in __connect
    connection = pool._invoke_creator(self)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 453, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  password authentication failed for user "postgres"

(Background on this error at: Error Messages — SQLAlchemy 2.0 Documentation)

While the event service does this…

NAMESPACE   NAME                                HOSTS                ADDRESS         PORTS   AGE
rasaxkube   ingress.extensions/rasacen-rasa-x   rasa-x.example.com   35.244.229.89   80      7m50s
jamestagg@macbook-pro-2 CenDigitalHuman % kubectl logs rasacen-event-service-989c9c844-f5km9
INFO:__main__:Waiting until database migrations have been executed...
WARNING:__main__:Could not retrieve the database revision due to: (psycopg2.OperationalError) FATAL:  password authentication failed for user "postgres"

(Background on this error at: Error Messages — SQLAlchemy 2.0 Documentation).

WARNING:__main__:Database revision does not match migrations' latest, trying again in 2 seconds.
WARNING:__main__:Could not retrieve the database revision due to: (psycopg2.OperationalError) FATAL:  password authentication failed for user "postgres"
2

And here is my test values.yml file with silly passwords that I am using for debug. Slightly oddly if I use the ones with “” they used to work (3 days ago) but now they fail with parsing splitting the tokens into “<safe” "credentials>"and seeing it as two tokens which crashes the server! But i moved to silly passwords instead and these still don’t work but in a more sensible way.

rasax specific settings

rasax: # initialUser is the user which is created upon the initial start of Rasa X initialUser: # password for the Rasa X user password: “safe1credential” # passwordSalt Rasa X uses to salt the user passwords passwordSalt: “safeScredential” # token Rasa X accepts as authentication token from other Rasa services token: “safe2credential” # jwtSecret which is used to sign the jwtTokens of the users jwtSecret: “safe3credential” tag: “0.27.4”

rasa: Settings common for all Rasa containers

rasa: # token Rasa accepts as authentication token from other Rasa services token: “safe4credential” tag: “1.9.5-full”

RabbitMQ specific settings

rabbitmq: # rabbitmq settings of the subchart rabbitmq: # password which is used for the authentication password: “safe5credential”

global settings of the used subcharts

global: # postgresql: global settings of the postgresql subchart postgresql: # postgresqlPassword is the password which is used when the postgresqlUsername equals “postgres” postgresqlPassword: “safe6credential” # redis: global settings of the postgresql subchart redis: # password to use in case there no external secret was provided password: “safe7credential” debugMode: “true”

3

James, do you mind putting some markdown on your messages so they’re readble.