Hello, everyone.
A new release with the changes described below was deployed to:
- kf.kobotoolbox.org at 12:32 UTC on 15 July 2022;
- kobo.humanitarianresponse.info at 15:22 UTC on 19 July 2022.
Components comprising this release:
- KPI version 2.022.24
- KoBoCAT version 2.022.24
- Enketo Express version 3.0.5 (unchanged from the previous release)
- Pyxform version 1.9.0, unchanged in KPI but upgraded in KoBoCAT from version 1.7.0. For a list of Pyxform changes, please refer to the Pyxform change log.
Removal of obsolete metadata
The simserial and subscriberid metadata fields have been removed from the form builder sidebar. Google disallowed apps distributed in the Play Store from reading this information back in November of 2020:
Web-based clients such as Enketo have never been able to read permanent unique identifiers like simserial or subscriberid. Neither have iOS clients for a long time. Android clients used to be able to access them but won’t be able to do so from Play Store installs starting Nov 2020.
Display warning when using simserial or subscriberid · Issue #471 · XLSForm/pyxform · GitHub
If your form contains these fields, don’t worry: it should continue to function properly. The simserial and subscriberid fields have likely been empty for all submissions since late 2020, unless you explicitly ran an old version of Android and Collect. Please note that if you open a form containing these fields in the form builder and then save it, these fields will be removed automatically. If you have an unusual situation where you do run old software in order to collect this information, avoid the graphical form builder and edit your form using XLSForm only.
Fixes and improvements
| PR | Description | Related Issues |
|---|---|---|
| kpi#3849 | Add export API documentation for lang field |
closes #3848 |
| kpi#3752 | Add new photo gallery to replace the one from the legacy interface Add a new photo gallery that supports filtering by form question to replace the legacy KoBoCAT gallery, which was previously shown via an iframe. |
Depends on Add elemMatch support to mongo helper by bufke · Pull Request #3749 · kobotoolbox/kpi · GitHub Fixes #1349 Related to #3145 |
| kpi#3937 | Allow form builder to open forms using deprecated metadata, but then force those metadata to be disabled The simserial and subscriberid metadata cannot be read from contemporary devices (#2842) and were removed entirely from the form builder in #3836, but that caused an error when trying to load old forms that had already enabled these metadata. This change allows those forms to be loaded without error, but the obsolete metadata are then disabled as soon as the form is saved. |
Part of #2842 Related to #3836 |
| kpi#3773 | Change password reset message Add more detail to the password reset message: indicate that a message is only sent if the email address provided is linked to an account. |
closes #3746 |
| kpi#3777 | Changed public access warning to only show for surveys Changed the public access warning so it would only be visible for surveys and not library items |
closes #3721 |
| kpi#3861 | Correct spelling error |
|
| kpi#3852 | Display select_one_from_file and select_multiple_from_file responses properlyAdds a bit better support for select_one_from_file and select_multiple_from_file in KPI. Now these are being displayed properly in Single Submission Modal. |
Fixes #3845 |
| kpi#3855 | Fix issue with synchronizing forms from the legacy interface Fix sync_kobocat_xforms issue failing with xlrd.biffh.XLRDError: Excel xlsx file; not supported, resulting in forms not being synced between kobocat and kpi through the legacy interface’s “SYNC FORMS” button. |
closes #3854 |
| kpi#3803 | Fix random semicolon appearing in UI A semicolon was appearing in the Form’s Downloads view on the bottom. It caused a second scrollbar to appear and breaking UI a bit. Removed it and it’s fixed. |
|
| kpi#3801 | Fix table view fullscreen functionality Fixes broken fullscreen styles for Table View. When used, the table was placed after the fold instead of taking the whole screen area. |
|
| kpi#3814 | Move audit settings in the form builder sidebar Audit settings were in the wrong spot in the form builder sidebar. This PR moves them directly under the check boxes and modernizes the styles. |
Fixes #3808 |
| kpi#3859 | Remove duplicated labels Remove duplicated labels in account settings selects. |
|
| kpi#3836 | Remove simserial and subscriberid from metadata fields Removed simserial and subscriberid from metadata because the information is no longer provided by Google |
Closes #2842 |
| kobocat#809 | Upgrade pyxform to 1.9.0 Upgrade pyxform to 1.9.0 and use openpyxl for reading XLSX. |
closes #808 |
| kpi#3759 | Upgrade to python 3.10 Use Python 3.10 in docker image. |
Of interest to self-hosters
| PR | Description | Related Issues |
|---|---|---|
| kpi#3783 | Allow configurable session cookie namekpi.domain.org sets a session cookie scoped to .domain.org so that kobocat.domain.org and enketo.domain.org can access it. However, this causes conflicts with instances running on subdomains of domain.com, e.g. kpi.sub.domain.org. This change allows the session cookie name (not scope) to be changed to avoid these conflicts. Note that the Enketo CSRF cookie name must also be changed for this use case; see #3798. |
Related to Allow configurable session cookie name by bufke · Pull Request #814 · kobotoolbox/kobocat · GitHub, Allow configuration of Enketo CSRF cookie name by bufke · Pull Request #3798 · kobotoolbox/kpi · GitHub |
| kpi#3811 | Add CSP exception for Google Analytics img src | Internal issue: OCHA-CSP-4 |
| kobocat#813 | Add CSP headers via django-csp Add support for optionally setting Content Security Policy (CSP) Headers. Defaults to disabled. Related to Add CSP headers via django-csp by bufke · Pull Request #3710 · kobotoolbox/kpi · GitHub |
|
| kpi#3872 | Add map layer domains to default content security policy When CSP is enabled (not default), always allow map related domains including open street map and arcgis. |
Follow up of adding CSP headers Add CSP headers via django-csp by bufke · Pull Request #3710 · kobotoolbox/kpi · GitHub |
| kpi#3807 | Add support for MongoDB connections over TLS/SSL Add the possibility to use a secure connection between KPI and MongoDB. |
Closes #3806 Related to kobotoolbox/kobocat#819 |
| kobocat#819 | Add support for MongoDB connections over TLS/SSL Add the possibility to use a secure connection between KPI and MongoDB. |
Closes #818 Related to kobotoolbox/kpi#3807 |
| kpi#3838 | Add support for replica set strings in MONGO_DB_URL Add support for MongoDB replica sets configured in the MONGO_DB_URL environment variable, for example:<br>MONGO_DB_URL=mongodb://username:password@server0.mongodb.cluster:27017,server1.mongodb.cluster:27017,server2.mongodb.cluster:27017<br>When using a database name other than the default ( formhub), then please set MONGO_DB_NAME as well. Do not append the database name to the end of MONGO_DB_URL. |
Fixes #3821 Related to Add support for replica set strings in MONGO_DB_URL by bufke · Pull Request #830 · kobotoolbox/kobocat · GitHub |
| kobocat#830 | Add support for replica set strings in MONGO_DB_URL See description at Add support for replica set strings in MONGO_DB_URL by bufke · Pull Request #3838 · kobotoolbox/kpi · GitHub |
|
| kobocat#823 | Allow AWS S3 region to be set through environment variablesAWS_S3_REGION_NAME environment variable can be used to set the region. |
Closes #805 |
| kpi#3816 | Allow AWS S3 region to be set through environment variablesAWS_S3_REGION_NAME environment variable can be used to set the region. |
Related to kobotoolbox/kobocat#823 |
| kobocat#814 | Allow configurable session cookie name | |
| kpi#3798 | Allow configuration of Enketo CSRF cookie name Allow Enketo CSRF cookie name to be configurable. Defaults to standard __csrf, and thus is not a breaking change. See #3783 for the purpose of this setting. |
- Requires #3783 - Docs Add cookie related settings · kobotoolbox/kobotoolbox.github.io@982cfbb · GitHub |
| kobocat#824 | Allow passing extra parameters to MongoDB driver Implement a new environment variable MONGO_DB_URL which gives more flexibility to pass extra parameters (e.g. tls=True). |
Supersedes #819 Related to kobotoolbox/kpi#3817 |
| kpi#3797 | Azure private storage Adds support for Azure in django-private-storage |
Related to Upgrade to latest django-storages and add support for Azure Blob Storage by bufke · Pull Request #3794 · kobotoolbox/kpi · GitHub |
| kobocat#812 | Change how the default database URL is read Change the default database to read from KC_DATABASE_URL and fallback to DATABASE_URL. See kobotoolbox/kpi#3780
|
closes #804 |
| kpi#3883 | Fix KPI issue when the app is not responding anymore even under low load Fix 502/504 errors that KPI starts to return after running few hours/days even if the load is very low. |
Fixes #3566 |
| kobocat#834 | Fix KoBoCAT issue when the app is not responding anymore even under low load Fix 502/504 errors that KoBoCAT starts to return after running few hours/days even if the load is very low. |
Fixes kobotoolbox/kpi#3566 Related to kobotoolbox/kpi#3883 |
| kpi#3809 | Fix ImportError with Python 3.8Bring back Python 3.8 support for users who do not use kobo-docker/docker containers. |
kobotoolbox/kobocat#820 |
| kobocat#820 | Fix ImportError with Python 3.8Bring back Python 3.8 support for users who do not use kobo-docker/docker containers. |
kobotoolbox/kpi#3809 |
| kpi#3862 | Prevent certain configuration settings from unexpectedly reverting to defaults Work around django-constance issue where values are undesirably reset to their default. Add redis cache for constance database backend. Introduces a new environment variable CACHE_URL to define the default Django cache. This is only used for constance right now, but could be used elsewhere. |
|
| kpi#3841 | Reduce Docker image size Reduce Docker image size via multi-step build, to improve docker pull speed and use less disk space. Removes 860MB of disk space as reported by docker images. |
Related to #3840 |
| kobocat#821 | Reduce Sentry sample rate for performance traces Allow editing Sentry traces rate with env var SENTRY_TRACES_SAMPLE_RATE which defaults to 0.05 (%) |
Related to Sentry - configure traces rate by bufke · Pull Request #3810 · kobotoolbox/kpi · GitHub |
| kpi#3584 | Two-Factor Authentication (front end) Add support for 2FA with mobile app (such as Google authenticator, Authy etc…) |
Closes #3563 Related to kobotoolbox/kobocat#780 |
| kobocat#811 | Upgrade Python version to 3.10 Use Python 3.10 in docker image |
|
| kobocat#816 | Upgrade to latest django-storages and add support for Azure Blob Storage | Upgrade to latest django-storages and add support for Azure Blob Storage by bufke · Pull Request #3794 · kobotoolbox/kpi · GitHub |
| kpi#3794 | Upgrade to latest django-storages and add support for Azure Blob Storage Upgrade django-storages package. Adds support for Azure Blob Storage, outside of django-private-storage |
|
| kpi#3680 | [ MFA ] Improve MFA admin list view Add a search box (to search for username) and display the MFA Method with the user’s username instead of their id. |
Part of #3563 |
| kpi#3674 | [ MFA ] Make the help message editable Super user can modify (and translate) the message to help users who lost their device and the backup codes. |
Part of #3563 |
Of interest to developers
| PR | Description | Related Issues |
|---|---|---|
| kobocat#832 | Add pip compile script to work around pip tools python version limitation | |
| kpi#3796 | Add InlineMessage component A simple component for displaying an inline message. The component will unify all existing messages under single design. |
|
| kpi#3847 | Add data-cy attribute to TextBox component to facilitate testing Enables easier testing of components with Cypress |
|
| kpi#3868 | Add pip compile script to work around pip tools python version limitation Work around pip tools limitation where packages that depend on a certain Python version are not carried over into the requirements txt file when the command is run from a system with certain Python versions. The desire is for the txt file to act as a universal lock file for dependencies. This change appends necessary packages that only should be installed on Python < 3.9. |
Fixes #One command to generate requirements.txt files · Issue #3867 · kobotoolbox/kpi · GitHub |
| kpi#3817 | Allow passing extra parameters to MongoDB driver Implement a new environment variable MONGO_DB_URL which gives more flexibility to pass extra parameters (e.g. tls=True). |
Supersedes #3807 Related to kobotoolbox/kobocat#824 |
| kpi#3885 | Avoid using cache for dynamic settings while running unit tests Deactivate cache system for Constance config values in testing environment |
|
| kpi#3780 | Change how default database URL is read Set the default database to read from KPI_DATABASE_URL and fallback on DATABASE_URL. This allows for simpler configuration, i.e. KPI_DATABASE_URL and KC_DATABASE_URL can be set simultaneously in an environment. |
closes #3748 part of kobotoolbox/kobocat#804 |
| kpi#3871 | Cleanup assetUtils functions Moving some refactor/clanup code from other branches to avoid complex merge conflicts. |
|
| kpi#3825 | Cope with change in django-constance newline handling In django-constance 2.7+, newlines are converted from \r\n to \n using the undocumented Django function normalize_newlines(). This PR accommodates that change while continuing to support old values from prior versions of django-constance. |
Explain django-constance newline handling by bufke · Pull Request #3831 · kobotoolbox/kpi · GitHub |
| kpi#3839 | Do not install raven package at runtime. No longer necessary. | |
| kpi#3831 | Explain django-constance newline handling | Related to Cope with change in django-constance newline handling by bufke · Pull Request #3825 · kobotoolbox/kpi · GitHub |
| kpi#3858 | Fix build dependency error Be more specific as what node version (16.15.0) github actions are using. |
Followup to #3851 |
| kobocat#825 | Fix media zip export to work with newer version of django-storages | |
| kpi#3851 | Freeze npm version to 8.5.5Fix broken build when using a newer version of npm. |
|
| kpi#3874 | Make GitHub Actions pytest routine stop failing on a particular case Change logic to create mock data to avoid unit tests failing randomly. |
|
| kpi#3828 | Make HEAD requests to OpenRosa endpoints return empty responses Hopefully fixes the Parse Error: Expected HTTP/ seen in Enketo when NGINX is not used, e.g. in local development environments |
|
| kobocat#831 | Make HEAD requests to OpenRosa endpoints return empty responses Hopefully fixes the Parse Error: Expected HTTP/ seen in Enketo when NGINX is not used, e.g. in local development environments. See also kobotoolbox/kpi#3828
|
|
| kpi#3866 | More linter fixes Internal code cleanup. Reorders functions and fixes rest of linter issues in assetUtils. |
|
| kobocat#817 | Pin pip version | |
| kpi#3805 | Pin pip version | |
| kpi#3774 | Preload JS chunks. Use more Suspense to avoid using Suspense. Avoid brief flicker when loading pages for the first time by adjusting React Suspense. Set JS chunks to preload, for faster loading. |
|
| kpi#3865 | Run linter on asset utils Internal code cleanup. |
|
| kpi#3779 | Send .xlsx file extension to kobocat for XLSForm (re)deploymentsPyxform 1.9.0 requires that we differentiate between xls and xlsx files explicitly. Ensure this is happening when kpi sends XLSForm contents to kobocat. |
required change from Upgrade pyxform to 1.9.0 by joshuaberetta · Pull Request #809 · kobotoolbox/kobocat · GitHub closes #3778 |
| kpi#3810 | Sentry - configure traces rate Allow editing Sentry traces rate with env var SENTRY_TRACES_SAMPLE_RATE which defaults to 0.05 (%) |
Related to Reduce Sentry sample rate for performance traces by bufke · Pull Request #821 · kobotoolbox/kobocat · GitHub |
| kpi#3685 | TS linter configuration Update linting configuration to include TypeScript. |
Fixes #3678 |
| kpi#3649 | Types for myLibraryStore, tableStore, dataInterface, and mixins Types for tableStore, with all necessary types for it: myLibraryStore, dataInterface, and mixins. |
|
| kpi#3832 | Update GitHub actions | |
| kpi#3884 | Updating cypress branch, and adding testserver and three cypress tests. |
|
| kpi#3850 | Upgrade css-loader dependency Upgrade css-loader dependency to make sure it’s compatible with Webpack v5. |
|
| kpi#3827 | Use KPI_DATABASE_URL in the settings for testing |
|
| kpi#3835 | Use cache to display number of submissions per project Get the number of all submissions from a pre-calculated field instead of calculating it each time for users who can view all submissions of a project. |
|
| kpi#3618 | [NLP] Add lists of languages supported by ASR and MT engines Created lists for languages supported by ASR and MT engines. Created endpoints to access the lists at environment/transcription_lists and environment/translation_lists
|
closes #3610 closes #3511 |
