Release Notes - version 2.022.16, 2.022.16a, b, c

Hello, everyone. A new release with the changes described below has been deployed to

Sorry for the previous typos in the dates listed in this announcement.

Components comprising this release:

  • KPI version 2.022.16
  • KoBoCAT version 2.022.16
  • Enketo Express version 3.1.0, upgraded from version 2.8.1. For a list of Enketo changes, please see the Enketo change log.
  • Pyxform version 1.9.0, upgraded from version 1.7.0. For a list of Pyxform changes, please refer to the Pyxform change log.

Fixes and improvements

PR Description Related Issues
kpi#3648 Fix parsing of submission XML for bulk editing
Use lxml.etree consistently for parsing and editing XML tree.
Closes #3647
formpack#271 Fix GeoJSON choice labels
Fix GeoJSON exports for select-type questions and include media urls in properties. Translated choice labels were improperly handled and caused exports to fail. This was especially the case for select_multiple questions that would fail regardless of the label if more than one option was selected. Rather rely on the existing label handling done by the field classes.
closes #266
kpi#3727 Fix 500 error when serving dynamic data.
Fix an error that happens in some circumstances when Enketo is trying to fetch data from another external form (i.e. referenced as xml-external).
Close #3726
kpi#3730 Fix autonaming for KoboRank and KoboScore
Fix autonaming for non-Latin characters for KoboRank and KoboScore questions.
closes #3729
kpi#3744 Fixes 404 error when attempting to edit submissions after switching accounts
Make Enketo display login form instead of 404 error.
Closes #3743
kpi#3745 Serve synchronous exports without redirecting for some ill-behaved user agents
Work around limitations in some clients (Power BI, Excel, LibreOffice) by avoiding HTTP 302 redirects when serving synchronous CSV exports
kpi#3753 Fix delete matrix row button not working properly
Fix case when clicking on delete matrix row button was not causing a row to be deleted.
Fixes #3697
kpi#3757 Allow duplicate choices in snapshots
Relaxes the pyxform requirement for uniqueness among choice values (also called choice names) for existing forms. This prevents failures when trying to edit submissions for these forms or retrieve these forms’ XForm XML.
Fixes #3751
kpi#3768 Fix validation badge colors
Change “Approved” badge background color to improve readability.
kpi#3771 Bulk permission assignment optimization
Improve the performance of sharing permission assignments, especially for projects that are shared with many accounts.
Fixes #2779
Related to #3433
kpi#3637 Replace instances of “KoBoToolbox” with "KoboToolbox"
Use consistent spelling of “KoboToolbox” and “Kobo” in the UI.
closes #3636

Of interest to self-hosters

PR Description Related Issues
kobocat#795 Enable optional HSTS headers via environment variables
Same as Enable optional settings for hsts by bufke · Pull Request #3708 · kobotoolbox/kpi · GitHub
kobocat#810 Remove select_for_update() locking from paired data timestamp update
Fix a performance bottleneck in the “Connect Projects” / “dynamic data attachments” feature
kpi#3704 Remove unused custom admin actions
They were used when kpi and kobocat shared a single database, but not any longer; see Delete users from KPI and KC if they do not have any forms or submiss… · kobotoolbox/kpi@0a1f3e8 · GitHub
kpi#3708 Enable optional settings for hsts
Enable optional (and off by default) HSTS headers. Via environment variable SECURE_HSTS_SECONDS.
kpi#3710 Add CSP headers via django-csp
Adds optional support for setting Content Security Policy (CSP) Headers. Defaults to disabled.
kpi#3761 Add task to flush Enketo preview after some delay
Attempt to solve a persistent Redis memory exhaustion problem caused by the accumulation of many gigabytes of Enketo previews (specifically, cached transformations of their XML). Enketo stores these for 30 days, and that lifetime is not configurable. This change uses the Enketo API to remove each preview from the cache 30 minutes after it was created.
enketo/enketo-express#357
kpi#3787 Allow an in-app message to appear on each reload…
Lets those who manage instances of KoboToolbox create persistent in-app messages that reappear whenever the application is loaded. Previously, the only way to display a message repeatedly to users who’d already read it was to create a new in-app message with the same content. This feature is available in the Django admin interface.

Of interest to developers

PR Description Related Issues
formpack#293 Pin pyxform to 1.7.0
formpack#296 Upgrade pyxform
formpack#297 Add support to Python 3.9 and 3.10
formpack#298 Remove xlwt requirement
kobocat#785 Build Docker images with GitLab CI and deploy to beta instance
- Build docker images on commit, tag as both short sha hash and ref (branch/tag). Push to gitlab docker registry.
- On beta branch (and this branch to test) push to a beta instance via helm chart
kobotoolbox/kpi#3600
kobocat#794 Dockerfile: move copying src to runtime image, instead of build step
kobocat#796 Add django-environ and refactor environment variable setting
kobocat#797 Fix django-redis-sessions url
kobocat#798 Use UTC timezone in Django settings
kobocat#802 Removed deprecated u from ugettext and ugettext_lazy
kpi#3600 Build Docker images with GitLab CI and deploy to beta instance
Build internal docker images on GitLab CI automatically. Deploy them automatically to kf.beta.kobotoolbox.org
kobotoolbox/kobocat#785
kpi#3606 KoboSelect component
New component for handling simple select dropdowns. Also changes KoboDropdown to be a styleless base for all our dropdowns.
Fixes #3568

Depends on #3540
kpi#3701 Add API support for fetching submissions by UUID
uuid and id can be used in data endpoint (i.e.: /api/v2/assets/<asset_uid>/data/<id_or_uuid>/)
kpi#3709 Add django-environ and refactor environment variable setting
Refactor code on how environment variables are set. Reduces code complexity.
kpi#3715 Update sentry
For error monitoring, update to latest Sentry client SDK.
kpi#3717 Pass REDIS connection as an URL to django-redis-session
kpi#3720 Quick magical hack fix for crashing tests
Fixes a crash Resource Failed to Load on /me and /environment endpoint calls while running frontend tests.
Part of #3719
kpi#3723 Fixes broken sessions
Fixed an unreleased problem related to invalid sessions that occurred only in beta branch.
kpi#3733 Update environment endpoint API test for new export-status-check timeouts
Goes with #3711
kpi#3737 Status filter for hook log API
kpi#3742 Ignore env in docker and git
For development environments, ignore env directory in git and docker. env is commonly used for python virtual environments.
kpi#3747 Debug toolbar enable
For development, add DEBUG_TOOLBAR environment variable to enable django-debug-toolbar.
kpi#3749 Add elemMatch support to mongo helper
Adds $elemMatch support to form data api filtering
kpi#3754 Use reports endpoints from v2 API
Updates reports endpoint url to use v2.
Uses endpoint added with #3009
kpi#3765 Upgrade pyxform to 1.9.0
Upgrade pyxform to 1.9.0 and use openpyxl and xlrd to handle XLSX and XLS excel file formats respectively.
closes #3686
blocked by kobotoolbox/formpack#296
3 Likes

We’ve decided to proceed with deploying Enketo 3.1.0 today. I’ve updated the release notes above. Thanks to everyone who helped to test this new version of Enketo.

2 Likes

As of now, 17:45 UTC on 14 June 2022, a patch has been deployed on both servers fixing a permission assignments issue

Fixes and improvements

PR Description Related Issues
kpi#3869 Permission assignments bug fix.
Save permission assignments correctly when usernames start with the same letter but contain non alphanumeric characters (such as _) and have a mix of upper and lower case.
Closes #3864

Of interest to self-hosters

PR Description Related Issues
kobo-docker#337 Fix NGINX DNS cache resolution issue. Closes #332
1 Like

As of now, 18:35 UTC on 20 June 2022, a patch has been deployed on both servers fixing a permission assignments issue

Fixes and improvements

PR Description Related Issues
kpi#3877 Fix submission edit authentication loop where attempts to submit edits to a submission get caught in an authentication loop. This loop can initiate if two or more submissions of the same project have been opened for editing. Only the most recently opened submission will submit successfully — all others will be stuck in an authentication loop and the edits will be lost. Closes #3876
3 Likes

We’ve decided to downgrade Enketo to version 3.0.5 today on both servers because of a memory leak affecting version 3.1.0 and above. There should be no noticeable difference between 3.0.5 and 3.1.0 given that the only change in the latter was:

#391 Add Enketo version (release tag, or git commit hash, or package.json version) to User-Agent header to server-sent requests

2 Likes