Background
KoBoToolbox previously used a shared Postgres database for its two Django applications, KPI and KoBoCAT. Since this has some disadvantages—including the requirement that Django be updated in lockstep across both apps—we’ve decided to give KPI and KoBoCAT their own, separate Postgres databases.
KoBoCAT has responsibility for storing all form submissions and, as a result, generally has a larger amount of data than KPI. For this reason, our approach is to copy the KPI tables from the original shared database into a new database. We leave the KoBoCAT tables where they are: if your original, shared database is called kobotoolbox (the default), then, when the migration is finished, KoBoCAT will use kobotoolbox and KPI will use a new database, called koboform by default.
Warnings
Upgrading to two databases requires downtime, because all the KPI data must be copied from one Postgres database to another. If your installation is small, this will be negligible. If you run a large server, however, please plan carefully: expect at least one minute of downtime for every 1,500 KPI assets. Assets are surveys and library items: questions, blocks, and templates. Anticipate an additional 30 minutes for the systems administration work itself: running commands, waiting for Docker images to download and containers to start, etc.
Sessions will be reset, meaning that all users will need to log in again after the upgrade. This happens because we now use Redis to share sessions between KPI and KoBoCAT instead of relying on both applications accessing a shared Postgres table.
Make sure you have enough disk space before proceeding. You’ll be creating a backup of your entire KoBo installation (configuration as well as data), and you’ll be copying the entire KPI database. For safety, we don’t yet delete anything from the original database after the copy completes, but if you’re an experienced database administrator, you can remove all the KPI tables from the KoBoCAT database once you’re satisfied with the upgrade.
It’s difficult to roll back to a single-database setup once you’ve upgraded and started making changes in the KPI application, because you would need to merge, manually, the two databases back into a single database. You can easily restore from a backup, but that would lose any work you may have done in KPI after the database split. If you manage an instance for multiple users, make sure to test the application thoroughly after upgrading before inviting everyone else to continue their work with KoBoToolbox.
Performing the upgrade
This procedure assumes you are using kobo-install to run KoBoToolbox on a single server. If you have a different setup and need assistance, please post a message below.
- Please make sure you have Python 3 installed in the environment where you run
kobo-install/run.py. - Stop all KoBo containers:
$ python3 kobo-install/run.py --stop - Back up everything related to your KoBo installation using
rsync. For a standalone server, this is usually four directories:kobo-deployments,kobo-docker,kobo-install, andnginx-certbot. If your containers are not stopped, your backup will be corrupt.$ sudo rsync -avP \ kobo-deployments \ kobo-docker \ kobo-install \ nginx-certbot \ backup-20200430/ - Run the same
rsynccommand again; no files should be listed, indicating that everything was already copied. Example output:sending incremental file list sent 93,627 bytes received 428 bytes 188,110.00 bytes/sec total size is 492,598,759 speedup is 5,237.35 - Enter the
kobo-installdirectory:$ cd kobo-install - Fetch new tags from GitHub:
$ git fetch --tags - Check out the
2.020.24brelease:$ git checkout 2.020.24b - Update the installation:
$ python3 run.py --update - Choose
Yesto proceed when prompted thatit's strongly recommended to run `./run.py --setup` to regenerate environment files.
At each prompt, you may press enter to accept your previously configured value without re-typing it, but take care not to skip over the next step. - After you are asked
Do you want to (re)start containers?, to which you should answerYes, you will be asked aboutUpgrading from single-database setup to separate databases for KPI and KoBoCAT. Read the notice about downtime and chooseYesto proceed if acceptable. This is the last opportunity to abort the upgrade without restoring from a backup.
If you decide not to proceed at this point, you have two options:- Follow the on-screen instructions to downgrade to the
shared-database-obsoletebranch; - Or, roll back using the
rsyncbackup you made at the beginning of this procedure. See the section titled “Rolling back a failed upgrade” below.
- Follow the on-screen instructions to downgrade to the
- If you elected to proceed, you should see messages related database creation and copying tables, followed by a notice that
The database upgrade finished successfully! Thanks for using KoBoToolbox. The message WARNING: Found orphan containerscan safely be ignored. - Your containers should now start as usual, which may take several minutes.
It is not uncommon to see
`KoBoToolbox` has not started yet. This is can be normal with low CPU/RAM computers. On a VPS with 2 GB RAM, 512 MB swap, and a single core of an AMD EPYC 7601, starting up the first time takes over 10 minutes.
If you want to get a sense of what’s happening, open a separate terminal and runtop. You should seegrunt,node, ornpmconsuming lots of CPU time (as Enketo completes its initial build).- If the CPU is idle but the application still does not load, there’s likely a problem. Answer
Noat the next prompt toWait for another 600 seconds?and selectYesto tryRestarting the frontend containers. - If startup continues to fail, inspect:
- The output of
python3 run.py --logs; - The uWSGI logs:
kobo-docker/log/kpi/uwsgi.log;kobo-docker/log/kobocat/uwsgi.log;
- NGINX logs in
kobo-docker/log/nginx, particularly*.error.log.
- The output of
- If necessary, roll back using the
rsyncbackup you made at the beginning of this procedure. See the section titled “Rolling back a failed upgrade” below.
- If the CPU is idle but the application still does not load, there’s likely a problem. Answer
- Once your containers start successfully, update the KoBoCAT database so that REST Services continues to function correctly:
- Enter the KoBoCAT Docker container:
$ python3 run.py -cf exec kobocat bash - Run the
update_kpi_hooks_endpointmanagement command:# ./manage.py update_kpi_hooks_endpoint - Once you see
Done!, typeexitto leave the KoBoCAT Docker container.
- Enter the KoBoCAT Docker container:
- Test to make sure the application is functioning well. Check as much as your time budget allows, but at a minimum, make sure that you can:
- Access old projects and their submissions;
- Access previously-created Enketo survey URLs and submit data through them;
- Create, deploy, and submit data to new projects;
- Export data to XLS.
- If everything looks good, you’re likely done!
Some installations will require a few extra steps; continue reading the “Less common scenarios” section below to see if this applies to you.
Less common scenarios
- If your users have previously uploaded forms directly to KoBoCAT, a.k.a. “Projects (legacy)”, and now want to use KPI REST Services to push submissions from those forms to external servers, you’ll need to run a management command:
- Enter the KPI Docker container—from your
kobo-installdirectory, execute:$ python3 run.py -cf exec kpi bash - Run the
populate_kc_xform_kpi_asset_uidmanagement command:# ./manage.py populate_kc_xform_kpi_asset_uid Done!will appear on the screen when the process is finished. You can thenexitthe container and close your terminal.
- Enter the KPI Docker container—from your
- KPI allows you to configure CORS headers using the Django admin interface, i.e. at
https://[your KPI host]/admin/external_integrations/corsmodel/. The underlying library we use for CORS has changed their logic to require that the scheme (httporhttps) be specified in each origin. If you previously had, for instance,example.comas an allowed CORS origin, simply change it tohttps://example.com.
Rolling back a failed upgrade
- If you discover problems or were unable to start your containers, roll back using the
rsyncbackup you made earlier:- Ensure all containers are stopped:
$ python3 kobo-install/run.py --stop - Move all directories pertaining to the failed upgrade into a new directory, e.g.
$ sudo mv \ kobo-deployments \ kobo-docker \ kobo-install \ nginx-certbot \ failed-update-20200430/ - Restore the backup, e.g.
$ sudo rsync -a backup-20200430/* ./ - Start your instance as you would have before attempting this procedure:
$ python3 kobo-install/run.py --start - Report error messages or trouble symptoms here by posting a message below.
- Ensure all containers are stopped:
