Release Notes - version 2.022.24, 2.022.24a, 2.022.24b, 2.022.24c, 2.022.24d

Hello, everyone.

A new release with the changes described below was deployed to:

Components comprising this release:

  • KPI version 2.022.24
  • KoBoCAT version 2.022.24
  • Enketo Express version 3.0.5 (unchanged from the previous release)
  • Pyxform version 1.9.0, unchanged in KPI but upgraded in KoBoCAT from version 1.7.0. For a list of Pyxform changes, please refer to the Pyxform change log.

Removal of obsolete metadata

The simserial and subscriberid metadata fields have been removed from the form builder sidebar. Google disallowed apps distributed in the Play Store from reading this information back in November of 2020:

Web-based clients such as Enketo have never been able to read permanent unique identifiers like simserial or subscriberid. Neither have iOS clients for a long time. Android clients used to be able to access them but won’t be able to do so from Play Store installs starting Nov 2020.
Display warning when using simserial or subscriberid · Issue #471 · XLSForm/pyxform · GitHub

If your form contains these fields, don’t worry: it should continue to function properly. The simserial and subscriberid fields have likely been empty for all submissions since late 2020, unless you explicitly ran an old version of Android and Collect. Please note that if you open a form containing these fields in the form builder and then save it, these fields will be removed automatically. If you have an unusual situation where you do run old software in order to collect this information, avoid the graphical form builder and edit your form using XLSForm only.

Fixes and improvements

PR Description Related Issues
kpi#3849 Add export API documentation for lang field closes #3848
kpi#3752 Add new photo gallery to replace the one from the legacy interface
Add a new photo gallery that supports filtering by form question to replace the legacy KoBoCAT gallery, which was previously shown via an iframe.
Depends on Add elemMatch support to mongo helper by bufke · Pull Request #3749 · kobotoolbox/kpi · GitHub
Fixes #1349
Related to #3145
kpi#3937 Allow form builder to open forms using deprecated metadata, but then force those metadata to be disabled
The simserial and subscriberid metadata cannot be read from contemporary devices (#2842) and were removed entirely from the form builder in #3836, but that caused an error when trying to load old forms that had already enabled these metadata. This change allows those forms to be loaded without error, but the obsolete metadata are then disabled as soon as the form is saved.
Part of #2842
Related to #3836
kpi#3773 Change password reset message
Add more detail to the password reset message: indicate that a message is only sent if the email address provided is linked to an account.
closes #3746
kpi#3777 Changed public access warning to only show for surveys
Changed the public access warning so it would only be visible for surveys and not library items
closes #3721
kpi#3861 Correct spelling error


kpi#3852 Display select_one_from_file and select_multiple_from_file responses properly
Adds a bit better support for select_one_from_file and select_multiple_from_file in KPI. Now these are being displayed properly in Single Submission Modal.
Fixes #3845
kpi#3855 Fix issue with synchronizing forms from the legacy interface
Fix sync_kobocat_xforms issue failing with xlrd.biffh.XLRDError: Excel xlsx file; not supported, resulting in forms not being synced between kobocat and kpi through the legacy interface’s “SYNC FORMS” button.
closes #3854
kpi#3803 Fix random semicolon appearing in UI
A semicolon was appearing in the Form’s Downloads view on the bottom. It caused a second scrollbar to appear and breaking UI a bit. Removed it and it’s fixed.
kpi#3801 Fix table view fullscreen functionality
Fixes broken fullscreen styles for Table View. When used, the table was placed after the fold instead of taking the whole screen area.
kpi#3814 Move audit settings in the form builder sidebar
Audit settings were in the wrong spot in the form builder sidebar. This PR moves them directly under the check boxes and modernizes the styles.
Fixes #3808
kpi#3859 Remove duplicated labels
Remove duplicated labels in account settings selects.
kpi#3836 Remove simserial and subscriberid from metadata fields
Removed simserial and subscriberid from metadata because the information is no longer provided by Google
Closes #2842
kobocat#809 Upgrade pyxform to 1.9.0
Upgrade pyxform to 1.9.0 and use openpyxl for reading XLSX.
closes #808
kpi#3759 Upgrade to python 3.10
Use Python 3.10 in docker image.

Of interest to self-hosters

PR Description Related Issues
kpi#3783 Allow configurable session cookie name
kpi.domain.org sets a session cookie scoped to .domain.org so that kobocat.domain.org and enketo.domain.org can access it. However, this causes conflicts with instances running on subdomains of domain.com, e.g. kpi.sub.domain.org. This change allows the session cookie name (not scope) to be changed to avoid these conflicts. Note that the Enketo CSRF cookie name must also be changed for this use case; see #3798.
Related to Allow configurable session cookie name by bufke · Pull Request #814 · kobotoolbox/kobocat · GitHub, Allow configuration of Enketo CSRF cookie name by bufke · Pull Request #3798 · kobotoolbox/kpi · GitHub
kpi#3811 Add CSP exception for Google Analytics img src Internal issue: OCHA-CSP-4
kobocat#813 Add CSP headers via django-csp
Add support for optionally setting Content Security Policy (CSP) Headers. Defaults to disabled. Related to Add CSP headers via django-csp by bufke · Pull Request #3710 · kobotoolbox/kpi · GitHub
kpi#3872 Add map layer domains to default content security policy
When CSP is enabled (not default), always allow map related domains including open street map and arcgis.
Follow up of adding CSP headers Add CSP headers via django-csp by bufke · Pull Request #3710 · kobotoolbox/kpi · GitHub
kpi#3807 Add support for MongoDB connections over TLS/SSL
Add the possibility to use a secure connection between KPI and MongoDB.
Closes #3806
Related to kobotoolbox/kobocat#819
kobocat#819 Add support for MongoDB connections over TLS/SSL
Add the possibility to use a secure connection between KPI and MongoDB.
Closes #818
Related to kobotoolbox/kpi#3807
kpi#3838 Add support for replica set strings in MONGO_DB_URL
Add support for MongoDB replica sets configured in the MONGO_DB_URL environment variable, for example:
<br>MONGO_DB_URL=mongodb://username:password@server0.mongodb.cluster:27017,server1.mongodb.cluster:27017,server2.mongodb.cluster:27017<br>
When using a database name other than the default (formhub), then please set MONGO_DB_NAME as well. Do not append the database name to the end of MONGO_DB_URL.
Fixes #3821
Related to Add support for replica set strings in MONGO_DB_URL by bufke · Pull Request #830 · kobotoolbox/kobocat · GitHub
kobocat#830 Add support for replica set strings in MONGO_DB_URL
See description at Add support for replica set strings in MONGO_DB_URL by bufke · Pull Request #3838 · kobotoolbox/kpi · GitHub
kobocat#823 Allow AWS S3 region to be set through environment variables
AWS_S3_REGION_NAME environment variable can be used to set the region.
Closes #805
kpi#3816 Allow AWS S3 region to be set through environment variables
AWS_S3_REGION_NAME environment variable can be used to set the region.
Related to kobotoolbox/kobocat#823
kobocat#814 Allow configurable session cookie name
kpi#3798 Allow configuration of Enketo CSRF cookie name
Allow Enketo CSRF cookie name to be configurable. Defaults to standard __csrf, and thus is not a breaking change. See #3783 for the purpose of this setting.
- Requires #3783
- Docs Add cookie related settings · kobotoolbox/kobotoolbox.github.io@982cfbb · GitHub
kobocat#824 Allow passing extra parameters to MongoDB driver
Implement a new environment variable MONGO_DB_URL which gives more flexibility to pass extra parameters (e.g. tls=True).
Supersedes #819
Related to kobotoolbox/kpi#3817
kpi#3797 Azure private storage
Adds support for Azure in django-private-storage
Related to Upgrade to latest django-storages and add support for Azure Blob Storage by bufke · Pull Request #3794 · kobotoolbox/kpi · GitHub
kobocat#812 Change how the default database URL is read
Change the default database to read from KC_DATABASE_URL and fallback to DATABASE_URL. See kobotoolbox/kpi#3780
closes #804
kpi#3883 Fix KPI issue when the app is not responding anymore even under low load
Fix 502/504 errors that KPI starts to return after running few hours/days even if the load is very low.
Fixes #3566
kobocat#834 Fix KoBoCAT issue when the app is not responding anymore even under low load
Fix 502/504 errors that KoBoCAT starts to return after running few hours/days even if the load is very low.
Fixes kobotoolbox/kpi#3566
Related to kobotoolbox/kpi#3883
kpi#3809 Fix ImportError with Python 3.8
Bring back Python 3.8 support for users who do not use kobo-docker/docker containers.
kobotoolbox/kobocat#820
kobocat#820 Fix ImportError with Python 3.8
Bring back Python 3.8 support for users who do not use kobo-docker/docker containers.
kobotoolbox/kpi#3809
kpi#3862 Prevent certain configuration settings from unexpectedly reverting to defaults
Work around django-constance issue where values are undesirably reset to their default.

Add redis cache for constance database backend.

Introduces a new environment variable CACHE_URL to define the default Django cache. This is only used for constance right now, but could be used elsewhere.
kpi#3841 Reduce Docker image size
Reduce Docker image size via multi-step build, to improve docker pull speed and use less disk space. Removes 860MB of disk space as reported by docker images.
Related to #3840
kobocat#821 Reduce Sentry sample rate for performance traces
Allow editing Sentry traces rate with env var SENTRY_TRACES_SAMPLE_RATE which defaults to 0.05 (%)
Related to Sentry - configure traces rate by bufke · Pull Request #3810 · kobotoolbox/kpi · GitHub
kpi#3584 Two-Factor Authentication (front end)
Add support for 2FA with mobile app (such as Google authenticator, Authy etc…)
Closes #3563
Related to kobotoolbox/kobocat#780
kobocat#811 Upgrade Python version to 3.10
Use Python 3.10 in docker image
kobocat#816 Upgrade to latest django-storages and add support for Azure Blob Storage Upgrade to latest django-storages and add support for Azure Blob Storage by bufke · Pull Request #3794 · kobotoolbox/kpi · GitHub
kpi#3794 Upgrade to latest django-storages and add support for Azure Blob Storage
Upgrade django-storages package. Adds support for Azure Blob Storage, outside of django-private-storage
kpi#3680 [ MFA ] Improve MFA admin list view
Add a search box (to search for username) and display the MFA Method with the user’s username instead of their id.
Part of #3563
kpi#3674 [ MFA ] Make the help message editable
Super user can modify (and translate) the message to help users who lost their device and the backup codes.
Part of #3563

Of interest to developers

PR Description Related Issues
kobocat#832 Add pip compile script to work around pip tools python version limitation
kpi#3796 Add InlineMessage component
A simple component for displaying an inline message. The component will unify all existing messages under single design.
kpi#3847 Add data-cy attribute to TextBox component to facilitate testing
Enables easier testing of components with Cypress
kpi#3868 Add pip compile script to work around pip tools python version limitation
Work around pip tools limitation where packages that depend on a certain Python version are not carried over into the requirements txt file when the command is run from a system with certain Python versions. The desire is for the txt file to act as a universal lock file for dependencies. This change appends necessary packages that only should be installed on Python < 3.9.
Fixes #One command to generate requirements.txt files · Issue #3867 · kobotoolbox/kpi · GitHub
kpi#3817 Allow passing extra parameters to MongoDB driver
Implement a new environment variable MONGO_DB_URL which gives more flexibility to pass extra parameters (e.g. tls=True).
Supersedes #3807
Related to kobotoolbox/kobocat#824
kpi#3885 Avoid using cache for dynamic settings while running unit tests
Deactivate cache system for Constance config values in testing environment
kpi#3780 Change how default database URL is read
Set the default database to read from KPI_DATABASE_URL and fallback on DATABASE_URL. This allows for simpler configuration, i.e. KPI_DATABASE_URL and KC_DATABASE_URL can be set simultaneously in an environment.
closes #3748
part of kobotoolbox/kobocat#804
kpi#3871 Cleanup assetUtils functions
Moving some refactor/clanup code from other branches to avoid complex merge conflicts.
kpi#3825 Cope with change in django-constance newline handling
In django-constance 2.7+, newlines are converted from \r\n to \n using the undocumented Django function normalize_newlines(). This PR accommodates that change while continuing to support old values from prior versions of django-constance.
Explain django-constance newline handling by bufke · Pull Request #3831 · kobotoolbox/kpi · GitHub
kpi#3839 Do not install raven package at runtime. No longer necessary.
kpi#3831 Explain django-constance newline handling Related to Cope with change in django-constance newline handling by bufke · Pull Request #3825 · kobotoolbox/kpi · GitHub
kpi#3858 Fix build dependency error
Be more specific as what node version (16.15.0) github actions are using.
Followup to #3851
kobocat#825 Fix media zip export to work with newer version of django-storages
kpi#3851 Freeze npm version to 8.5.5
Fix broken build when using a newer version of npm.
kpi#3874 Make GitHub Actions pytest routine stop failing on a particular case
Change logic to create mock data to avoid unit tests failing randomly.
kpi#3828 Make HEAD requests to OpenRosa endpoints return empty responses
Hopefully fixes the Parse Error: Expected HTTP/ seen in Enketo when NGINX is not used, e.g. in local development environments
kobocat#831 Make HEAD requests to OpenRosa endpoints return empty responses
Hopefully fixes the Parse Error: Expected HTTP/ seen in Enketo when NGINX is not used, e.g. in local development environments. See also kobotoolbox/kpi#3828
kpi#3866 More linter fixes
Internal code cleanup. Reorders functions and fixes rest of linter issues in assetUtils.
kobocat#817 Pin pip version
kpi#3805 Pin pip version
kpi#3774 Preload JS chunks. Use more Suspense to avoid using Suspense.
Avoid brief flicker when loading pages for the first time by adjusting React Suspense. Set JS chunks to preload, for faster loading.
kpi#3865 Run linter on asset utils
Internal code cleanup.
kpi#3779 Send .xlsx file extension to kobocat for XLSForm (re)deployments
Pyxform 1.9.0 requires that we differentiate between xls and xlsx files explicitly. Ensure this is happening when kpi sends XLSForm contents to kobocat.
required change from Upgrade pyxform to 1.9.0 by joshuaberetta · Pull Request #809 · kobotoolbox/kobocat · GitHub
closes #3778
kpi#3810 Sentry - configure traces rate
Allow editing Sentry traces rate with env var SENTRY_TRACES_SAMPLE_RATE which defaults to 0.05 (%)
Related to Reduce Sentry sample rate for performance traces by bufke · Pull Request #821 · kobotoolbox/kobocat · GitHub
kpi#3685 TS linter configuration
Update linting configuration to include TypeScript.
Fixes #3678
kpi#3649 Types for myLibraryStore, tableStore, dataInterface, and mixins
Types for tableStore, with all necessary types for it: myLibraryStore, dataInterface, and mixins.
kpi#3832 Update GitHub actions
kpi#3884 Updating cypress branch, and adding testserver and three cypress tests.


kpi#3850 Upgrade css-loader dependency
Upgrade css-loader dependency to make sure it’s compatible with Webpack v5.
kpi#3827 Use KPI_DATABASE_URL in the settings for testing
kpi#3835 Use cache to display number of submissions per project
Get the number of all submissions from a pre-calculated field instead of calculating it each time for users who can view all submissions of a project.
kpi#3618 [NLP] Add lists of languages supported by ASR and MT engines
Created lists for languages supported by ASR and MT engines.
Created endpoints to access the lists at environment/transcription_lists and environment/translation_lists
closes #3610
closes #3511
3 Likes

Attention self-hosters:

When you are running kobo-install/run.py --setup, pay attention to the “AWS Region Name” question. This is newly added and defaults to us-east-1. If you use Amazon and host in a different region, you must change this value to avoid authentication problems, which manifest as X-Amz-Credential errors.

1 Like

Attention self-hosters:

If you encounter an error with KoBoCAT container like this one:

psycopg2.errors.UndefinedColumn: column oauth2_provider_application.created does not exist

Please follow these instructions to fix it with one minor change: Upgrade the library back to 1.7.1 (instead of 1.3.2).

1 Like

Hello, everyone.

A new release of KPI (2.022.24b) has been deployed to:

Fixes and improvements

PR Description Related Issues
kpi#3965 Fix error “Failed to load environment data” on login page
1 Like

Hello, everyone.

New releases of KPI (2.022.24c) and KoBoCAT (2.022.24a) have been deployed to:

Fixes and improvements

PR Description Related Issues
kpi#3970 Generate the external XML data of connected parent projects faster.
Fix 502/504 errors when fetching the XML data of connected parent projects.
kobocat#844 Fix “Code challenge required” error when using OAuth2 applications

Of interest to developers

PR Description Related Issues
kpi#3969 Optimize code to avoid loading not required fields.
Avoid slow queries when loading (not required) large data from DB.
2 Likes

Hello, everyone.

New release of KPI (2.022.24d) has been deployed to:

Fixes and improvements

PR Description Related Issues
kpi#3989 Move background tasks for REST Services to their own queue.
Use a different queue to send submissions to REST Services external servers to avoid blocking imports and exports running in the background.
Part of kpi#3988
2 Likes