Release Notes - version 2.021.24

Hi everyone, we’re pleased to announce the latest release of KoBoToolbox, with significant new features and many behind-the-scenes updates. It has been deployed on:

  • kf.kobotoolbox.org as of 21:42 UTC on 15 June 2021;
  • kobo.humanitarianresponse.info as of 20:40 UTC on 16 June 2021.

As always, thank you for using KoBoToolbox, and please do not hesitate to use this forum to report any new issues that you may discover.

New features and enhancements

PR Description Related Issues
#3127
#3082
formpack#238
#3209
Library Locking
A template can now specify (in XLSForm only) that a form created from that template should disallow certain kinds of editing
Fixes #3032
Fixes #2994
Closes #3005
Closes kobotoolbox/formpack#236
Fixes #3205
Known translation management caveats: #3213
#3146 Delete entire group functionality. Now, whenever use clicks the delete group button, a modal will be displayed to choose whether they want to simply ungroup or delete everything. Fixes #3132
Needed for #3032
#3255 Enable anonymous limited access in KPI interface
Allows anonymous (not logged in) access to a limited part of KPI interface.
Fixes #1659
Fixes #3222
Requires #3215 #3256
#3262
#2925
#2695
#2537
Handle media files from KPI
Users can attach media files to their form and preview them directly from KPI. The legacy interface is not used anymore.
Files are synchronized (asynchronously) on deployment only.
Closes #2673
Related to #2675
Fixes #2673, #2647
#3276 Adds background audio recording settings to the side bar of the form builder Part of #3193
Fixes #3275
#2984 Add dropdown menu item that lets non-owners remove their own permissions from a project
Adds a dropdown menu item on the project list page that lets a non-owner remove forms that are shared with them
Front end of #2940
#3111 Automated pushing translation strings to Transifex
Automated the task of pushing an updated list of translation strings
This is to keep our translations as accurate as possible and all strings will be pushed to Transifex every time a PR to the Master branch is made.
Blocked by #3064
#3226 Include form media when syncing forms from the legacy interface (KoBoCAT) to the new (KPI) interface
New management command, sync_kobocat_form_media, created to allow for form-media from kobocat to be synced with KPI’s replacement model.
closes #2675
kobocat#729 Add guidance and redirect links to KPI for legacy form and data views
This will hopefully help ease the transition away from removed features of KoBoCAT’s legacy UI, especially for those arriving via anonymous-access links.

kobocat#634
kobocat#677
kobocat#678
kobocat#679
kobocat#680
kobocat#681
kobocat#682
kobocat#683
Upgrade KoBoCAT to Python 3 and Django 2.2 LTS

Bug fixes and other improvements

PR Description Related Issues
#3129 Fix not redirecting to form list after deleting project
Fixed bug where if user deleted a project they would not be redirected back to the form list
#3143 A better new icon for “Expand / collapse questions” button. Fixes #3140
#3152 Add non-owner self removal to form landing
Add the non-owner self removal option introduced in https://github.com/kobotoolbox/kpi/pull/2984 to the actions available in form landing (to complement where a user would expect the sharing options to be for an owned form):
Fixes #3151
#3155 Fix to allow non-owners with “manage” permission to remove other non-owners’ permissions
A fix to allow non-owners with manage_asset permissions to manage all permissions of a form including removing all permissions of other non-owners.
closes #3154
#3165 Don’t show last asset after it’s removed from asset list
Adds a hack fix that removes last asset from list view after it’s been deleted
Fixes #3163
#3184 Missing collection link handling
Whenever collection method link is missing from endpoint (e.g. due to Enketo error), show a warning information instead of broken button.
#3189 Create project button fix
Allow creating projects from any viewable templates (previously user needed edit permissions).
Fixes #3162
#3211 Better light button disabled styles
More (visually) distinct disabled state for light buttons.
#3229 Fix hiding columns
Fix hiding Submission links column. Never hide checkboxes column.
Fixes #3227
Fixes #3228
#3245 Fixed positioning of form view close button.
#3265 Fix report styles tabs
Fixes a small bug with missing styles for report styles tabs.
#3267 Make some of the settings dropdowns searchable
Selectors like “Country”, “Sector” and few more are searchable now.
#3272 Change Data side tabs order
"Table" tab is first now and is the default tab for “Data” section. “Reports” tab will now be second.
Fixes #3269
#3291 Fix unreleased visual bug that results in tiny icons on the form view sidebar.
kobocat#728 Fix form media permissions
Only allow users with permission to submit to a form to download that form’s associated media files

Of interest to self-hosters

PR Description Related Issues
#3161 Create User Statistics Report
Created an admin view to track users usage for number of submissions, number of forms created and number of forms deployed
part of #3069
part of kobotoolbox/kobocat#703
#3259 Make the admin user report pull data from KoBoCAT tables instead of KPI
KPI can read data directly from some KoBoCAT tables. The admin user report needs to access those tables, otherwise it fails to be generated.
Fixes #3253
#3287 Remove unnecessary filter from database queries
When deleting submissions through the legacy KoBoCAT UI, a special marker was applied to the “deleted” submissions instead of removing them outright. This meant that every query to retrieve submitted data needed to filter out submissions having this marker. We now always remove submissions outright when deleting and therefore no longer need this extra filter, which caused high database overhead.
Part of kobotoolbox/kobocat#696

Of interest to developers

PR Description Related Issues
#3064 cleaned up translation strings
Removed the _() wrapping used to identify which strings should be translated from any of the strings which do not appear to the front end users.
Added a shell script to automate creating PO files and pushing them to Transifex.
No related issues
#3076 Better routes checking
Alternative solution to react router’s isActive for checking wheter a route is your current route. Needed for non-components.
Started because of #3012
#3077 Add comment about _status field
Add comments for developers describing the _status field, which comes from KoBoCAT
#3112 Move surveyCompanion store to separate file
This is small code refactor change, nothing user oriented. I moved one of the stores to a separate file to avoid potential circular import warning.
Part of #3032
#3118 Link export with export-settings
Include export-settings name in the export data object if provided.
backend for #3078
blocked by kobotoolbox/formpack#240
#3120 Update icons, get rid of FontAwesome
All of our icons now are self designed, we stopped using FontAwesome. Some of the icons got updated. Also introduced consistent loading spinner with message.
Fixes #2869
Part of #3032
#3126 Add tests to asset deployments
This PR modifies the MockDeploymentBackend class to make it behave a little bit more like KobocatDeploymentBackend when deploying assets. It lets our unit tests cover more code that’s used for asset deployment.
Part of #2487
#3130 Alphabetization refactoring
Reorder methods of deployment back-end classes alphabetically for better readability.
Part of #2487
#3136 Coffeelint config
Added coffee code linting configuration file. Almost all the rules added are set to “warn” level. This is going to be helpful for #3018 to clean up the code before running scripts.
Part of #3018
#3141 Post coffeelint config fixes
Fixing a few coffeescript lines that were raisng compiler errors for linter.
#3185 Split reports code
Split Reports code into multiple files to make maintaining it easier.
#3217 Add prettier config
Add prettier configuration file for better development.
#3220 Linter arrow functions improvement
Improve linter configuration to allow more compact arrow functions.
#3223 Fixed GenericForeignKey errors with shadow models
Overloads GenericField class to check for KobocatContentType objects instead of ContentType.
Shadow models are reading data from KoBoCAT databases, so related objects should get content types from the same DB.
Content types can differ between databases
Fixes issue found here https://github.com/kobotoolbox/kpi/pull/3161#discussion_r628494780
#3224 Initial table cleanup
Splitting the Table View related code into separate directory with some cleanup.
Preparing work for #3199 and #3198
Part of #3225
#3246 New names for gray colors
Introduced new future friendly names for gray color variables. Also updated the general hue and saturation of grays (went a bit more blue).
#3251 Add the formbuilder message box from library locking
#3256 List permissions assigned to AnonymousUser when an asset is public closes #3215 (on master branch)
#3261 Enforce readonly state at the DB router level
When a DB model class has the attribute readonly equals to True, creating or changing entries will raise an exception.
Fixes #3258
#3264 Upgrade react-select dependency version
#3273 Make drop zone enlargement apply only to form media
#3278 Update sync_kobocat_xform command
Minor fix to syncing form media that was uploaded to legacy KoBoCAT interface
Part of #2675
#3281 Fix pre-release skip logic problem
Fixes missing skip logic value selector issue that was introduced on the beta branch but never released.
Fixes #3279
#3284 MongoDB query timeout calculation fix
MongoDB query timeout, which is in millisecond, is based on Celery timeout, which is in seconds.
The calculation on MongoDB was wrong. It was converted from minutes to milliseconds instead instead of
seconds to milliseconds.
Fixes #3283
Related to kobotoolbox/kobocat#731
kobocat#646 Removed old check for ziggy
kobocat#692 Remove old SMS model functions and run migrations
kobocat#693 Skip AlterUniqueTogether for logger_xform
kobocat#694 Add a flag from_kpi on media files
Adds a new field on the main_metadata table to distinguish between files uploaded directly to KoBoCAT and files synchronised from KPI.
The name and hash of files can now be set explicitly in POST requests that create metadata objects. This helps KPI to synchronise files correctly.
kobocat#695 Make GitHub actions run unit tests
Git rid of Travis CI and use GitHub actions instead
kobocat#702 Fix docs typo
kobocat#703 Created Submission Counter Model
kobocat#704 Fixed apt dependencies breaking
kobocat#708 Fix argparse config for sync_mongo
kobocat#711 Re-add secondary data api endpoint after it was removed from an unreleased version
kobocat#713 Python3 and Django2.2 upgrade punch list items
kobocat#719 Apply pyxform 1.5.1 upgrade to beta branch
Addresses any conflicts and Python 3 / Django 2 incompatibilities with the previous upgrade to pyxform 1.5.1 (kobocat#716).
Incorporates kobocat#716
Closes kobocat#604 (on beta branch)
Supersedes kobocat#699 (completely)
Begins work on kobocat#718
kobocat#720 Removed recaptcha client dependency and registration HTML templates
kobocat#722 Re-add background documents after they were removed from an unreleased version
kobocat#726 Bring metadata endpoint enhancements and changes from master to KoBoCAT 2.0 Bring kobocat#706, kobocat#714, kobocat#723 to beta
kobocat#731 MongoDB query timeout calculation fix
2 Likes

Thank you to everyone who reported the ZIP export issue for large exports (over about 50 megabytes). We have deployed a fix for this now (2.021.24a) to both servers.

2 Likes

We’ve deployed more fixes (as release 2.021.24b) to both servers. Thanks for your continued patience.

PR Description Related Issues
#3316 Synchronize media files when redeploying a project with no changes
Force the synchronization of the project media files if any have been added / deleted / updated when the button REDEPLOY is clicked, even if the project has not changed yet.
#3311 Update supported form media types closes #3310
#3306 Fix broken partial permissions exports
Fix broken exports for partial permissions.
closes #3113
2 Likes

If you run your own server and have noticed 500 errors when attempting to use OAuth authentication, there are some steps you need to take manually due to a change in the django-oauth-toolkit library that we use.

The steps are:

  1. Enter your KoBoCAT container (or, better yet, start a one-off container);
  2. Downgrade django-oauth-toolkit from version 1.3.2 to version 1.2.0;
  3. Run the Django manage.py migrate command;
  4. Upgrade django-oauth-toolkit to 1.3.2;
  5. Re-run the manage.py migrate command.

Specifically, if you’re using kobo-install:

  1. In your kobo-install directory, execute ./run.py -cf run --rm kobocat bash, which will start a one-off KoBoCAT container;
  2. Now, inside that one-off container, execute pip install django-oauth-toolkit==1.2.0;
  3. Now that django-oauth-toolkit has been downgraded, execute ./manage.py migrate;
  • It’s okay if you see Your models have changes that are not yet reflected in a migration, and so won't be applied. You do not need to take any action.
  1. Upgrade the library back to the 1.3.2 version with pip install django-oauth-toolkit==1.3.2;
  2. Run ./manage.py migrate.
  • It’s fine if no migrations are applied in this step, and, again, you can ignore the message Your models have changes that are not yet reflected in a migration….

If you have trouble with this procedure, please reply here. This is only needed if you use OAuth for integration with external applications. If you don’t know what that is, don’t worry about it!

For search purposes, some common exceptions raised due to this problem are:

psycopg2.errors.UndefinedColumn: column oauth2_provider_application.created does not exist

psycopg2.errors.UndefinedColumn: column oauth2_provider_accesstoken.source_refresh_token_id does not exist

psycopg2.ProgrammingError: column "source_refresh_token_id" of relation "oauth2_provider_accesstoken" does not exist
2 Likes