New Version does not allow special characters in DB passwords

The issue

When trying to upgrade to the latest version (2.022.24) from my previously installed version (2.021.45), I get the following error when entering my Database passwords during the setup process initiated by python3 run.py --setup:

Rules: Alphanumeric characters only, 8 characters minimum

My DB passwords for Postgres, MongoDB and Redis all contain special characters. Upon entering new passwords without special characters during the installation, the database fails to connect, as is expected. kobo-install output:

CommandError: Retries exceeded; failed to connect to default database

What I’ve tried

The obvious thing to do was to try and change the database passwords from inside the backend containers to something without special characters. After some trial and error, I was able to change the password for the postgres container using psql shell commands.
I wasn’t able to figure out how to change the database passwords for mongo and redis. I tried db.changeUserPassword(username, password) but ran into errors like:

E QUERY    [thread1] Error: Updating user failed: not authorized on root to execute command { updateUser: "kobouser", pwd: "xxx", writeConcern: { w: "majority", wtimeout: 600000.0 }, digestPassword: false } :

Anyway after changing the postgres password and executing python3 run.py --setup, the containers were up and running. However, the instance is empty: none of the previous users, forms, data or anything was visible, it was as if I had done a completely clean, fresh installation.

I brought over a backup of the .vols folder and tried again, same results.

Support requested

Did I “lose” the data because the mongo and/or redis container passwords were not changed? If so, how can I change those passwords?
Would appreciate your support on this @jnm @Josh. Also I am curious to understand why the password rules changed between these versions.

1 Like

Do you run your databases separately from kobo-install? Unless you do, you should not have needed to set any passwords manually (outside of run.py --setup), unless you were using PostgreSQL replication. @nolive is the expert here, though.

No, everything was setup from within run.py --setup.

In the previous versions of kobo-install, it was possible to enter special characters in passwords during the setup process. In the current version, it doesn’t allow it. Since my existing passwords have special characters, it breaks the installation process.

Hello @ks_1, as you have noticed, we don’t support special characters anymore. To respond to your question, it was a hard decision to take but we had lots of issues with special characters at some point that we decided to use only safe characters. The suggested passwords are now 20 characters for new installs (even if kobo-install needs only 8 characters minimum) to counterbalance the use of special characters.

As @jnm said, you can change your passwords when asked for while running ./run.py --setup, kobo-install should update your PostgreSQL, MongoDB and Redis passwords when the containers start. It seems you already tried to change your passwords with ./run.py --setup. I would recommend to try it another time (i.e. run ./run.py --setup and change your all your passwords). We did it for few instances without any issues.

1 Like

@nolive
Thanks for your response. I made a snapshot of my production VM this time (earlier I had just copied over the .vols folder to a new VM, which used to work).

I managed to get through the setup process, but now I’m getting these errors inside the kobocat container and it keeps restarting. It seems to be related to changes in the DB structure. Any tips on getting through this?

  Applying oauth2_provider.0006_alter_application_client_secret...KoBoCAT intializing...
Waiting for container `mongo`.
wait-for-it: waiting 40 seconds for mongo.<domain>.private:27017
wait-for-it: mongo.<domain>.private:27017 is available after 0 seconds
Container `mongo` up.
Waiting for container `postgres`.
wait-for-it: waiting 20 seconds for postgres.<domain>.private:5432
wait-for-it: postgres.<domain>.private:5432 is available after 0 seconds
Container `postgres` up.
Waiting for Postgres service.
postgres.<domain>.private:5432 - accepting connections
Postgres service running; ensuring <db> database exists and has PostGIS extensions...
ERROR:  database "<db>" already exists
You are now connected to database "<db>" as user "<user>".
NOTICE:  extension "postgis" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "postgis_topology" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "fuzzystrmatch" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "postgis_tiger_geocoder" already exists, skipping
CREATE EXTENSION
Postgres database <db> ready for use!
Running migrations...
Your environment is:"onadata.settings.prod"
Operations to perform:
  Apply all migrations: admin, api, auth, authtoken, contenttypes, django_celery_beat, django_digest, guardian, logger, main, oauth2_provider, restservice, reversion, sessions, sites, taggit, viewer
Running migrations:
Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedColumn: column oauth2_provider_application.created does not exist
LINE 1: ...auth2_provider_application"."skip_authorization", "oauth2_pr...
                                                             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/srv/src/kobocat/manage.py", line 17, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 232, in handle
    post_migrate_state = executor.migrate(
  File "/opt/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/opt/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/opt/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/opt/venv/lib/python3.10/site-packages/django/db/migrations/migration.py", line 124, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/opt/venv/lib/python3.10/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/opt/venv/lib/python3.10/site-packages/oauth2_provider/migrations/0006_alter_application_client_secret.py", line 14, in forwards_func
    for application in applications:
  File "/opt/venv/lib/python3.10/site-packages/django/db/models/query.py", line 274, in __iter__
    self._fetch_all()
  File "/opt/venv/lib/python3.10/site-packages/django/db/models/query.py", line 1242, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/opt/venv/lib/python3.10/site-packages/django/db/models/query.py", line 55, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/opt/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql
    cursor.execute(sql, params)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 80, in _execute
    with self.db.wrap_database_errors:
  File "/opt/venv/lib/python3.10/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: column oauth2_provider_application.created does not exist
LINE 1: ...auth2_provider_application"."skip_authorization", "oauth2_pr...
                                                             ^

Edit: I just noticed that another forum member is having the same issue: Kobo server is not starting

@ks_1, Can you try to follow these steps:

At the end, do not upgrade to 1.3.2 but to 1.7.1.

1 Like