Release Notes - version 2.023.37

Hello everyone,

We’re pleased to announce the release of the following:

  • KPI version 2.023.37
  • KoboCAT version 2.023.37
  • Pyxform version 1.9.0, unchanged from the previous release
  • Enketo Express version 6.0.0 with additions, unchanged from the previous release.

These were deployed to

  • kf.kobotoolbox.org at approximately 19:00 UTC on Wednesday, 20 September 2023, and
  • eu.kobotoolbox.org (formerly kobo.humanitarianresponse.info) at approximately 20:20 UTC on Thursday, 21 September 2023.

New languages

KoboToolbox now has user interface support for 8 new languages:

  • Bengali
  • Burmese/Myanmar
  • Indonesian
  • Lingala
  • Nepali
  • Nyanja/Chewa
  • Thai
  • Vietnamese

Support was added in kpi#4610, kpi#4459, kpi#4486, kpi#4444, and kpi#4649. Additionally, kpi#4624 fixes an issue where the user interface language would incorrectly fall back to English.

Improved projects list

The list of projects has been totally revamped for better filtering and ordering, customization of displayed columns, and the ability to perform bulk actions that affect multiple projects at once: kpi#4240, kpi#4404, kpi#4219, kpi#4580, kpi#4529, kpi#4502, kpi#4408, kpi#4402, kpi#4409, kpi#4383, kpi#4382, kpi#4386, kpi#4310.

Notifications when exceeding limits

We’ve taken the guesswork out of monitoring your KoboToolbox usage and will now alert you when you’ve exceeded any of your account limits: kpi#4571, kobocat#889.

New features for self hosters

If you administer your own instance of KoboToolbox, it’s now possible for you to:

Removal of the legacy UI

On 14 June 2021, we announced:

  • that the legacy (KoboCAT) user interface would no longer be supported,
  • that the legacy user interface would be removed after “a few more months”, and
  • that form uploads were only supported through the regular interface from that point forward.

Now that more than two years have passed, this release removes the legacy user interface entirely and removes various obsolete endpoints related to it. These changes were implemented in kpi#4516, kobocat#892, and kobocat#885.

If you are a developer, please note that this release also disables uploading XLSForms directly to the legacy v1 API, as projects created this way would be inaccessible from any user interface. Please update your integrations to use the v2 API. This change only affects uploading XLSForms, not retrieving data, which we still support with the v1 API—although you should likewise future-proof your code by migrating to the v2 API as soon as possible.

Other fixes and improvements

PR Description
kpi#4522, kpi#4518 Improve redeployment logic
Always update “Date deployed” when “REDEPLOY” is clicked, even if the form version is the same (i.e. the content has not been changed). Other things about the project may have changed, e.g. form media files.
kpi#4559 Hide meta/rootUuid from table view
This field is not ready for general use yet, but curious people have discovered it.

When (if) it comes back, it should be positioned at the end of the table.
kpi#4472 Clarify the error message when REST Services attempts to send a submission that has been deleted.
kpi#4501 Add usage section to user settings menu
This allows users to view their account submission, storage, transcription minutes, and translation minutes usage.
kpi#4286 Project Summary Details improvement
Merges together two boxes of information on project from Project Summary route. Adds few missing pieces of information to it too.
formpack#315 Handle Excel limits: URLs, cell content, and worksheet boundaries
formpack#316 Speed up the import of XLSForms, especially when many blank columns are present
kpi#4647 Fix CSP problem with language picker on login/signup page

Of interest to self hosters

PR Description
kpi#4613 Fix inability to edit in-app messages in Django Admin
kpi#4582 Update CSP settings for Google Analytics 4
kpi#4549 Allow default MFA help text to be translated along with the rest of the application’s translatable strings
kpi#4460 Add Redis status checks to service health endpoint
Add cache, broker, and session Redis checks on service_health endpoint.
kpi#4508 Allow setting SSO config fully in env vars and support multiple public OIDC providers
Adds front-end support for multiple OIDC SSO providers.
kpi#4435 Add filtering by date joined on the user detail report for superusers
kpi#4441 Fix spacing for customizable email template
Fix the registration email template spacing
kpi#4637 Allow gif and webp extensions to be uploaded in markdown editor
Validation check is based on the content type. Allowed content types are:

- image/jpeg
- image/png
- image/svg+xml
- image/gif
- image/webp
kobocat#895 Retain counts for deleted forms in DailyXFormSubmissionCounter

Of interest to developers

PR Description
kpi#4628 Align language selector dropdown with form beneath it
kpi#4606 Use fuzzy integer for tests which assert of number of queries
Make the number of queries assertions a little bit less strict when running unit tests
kpi#4609 Make language selector available in all templates
kpi#4579 Update TextBox component
Update looks of TextBox component and cleanup its usage.
kpi#4601 Update copy/styling for usage limit banners and modals
Changed the text and styling of usage limit notifications for public servers.
kpi#4617 Add notes to PR template
kpi#4603 Add support for old KoBoCAT domain name
kpi#4602 Update formpack requirement
kpi#4578 Usage limits setting
kpi#4593 Fix incorrect labels in static project metadata list
kpi#4590 Code refactoring to get translated utilities at the same place
kpi#4483 Use only one markdown dependency for markdown fields in Django Admin
Replace django-markitup with markdownx everywhere.
kpi#4575 Support alias t() for gettext and gettext_lazy
Extract strings used with t() (and nt() for plural) alias(es) instead of expected Django alias: _()
kpi#4568 Remove dead KOBOFORM_INTERNAL_URL setting
kpi#4570 Mass user email script using Amazon SES
Added a script to send broadcast emails to users using Amazon SES.
kpi#4541 Use python3 for npm copy-fonts command
kpi#4535 Typescriptize MainHeader component
Internal code improvements.
kpi#4548 Resolve conflicts and api changes from react-router-6-13 branch
kpi#4561 Usage limits 2fa fixes
Small changes to how MFA configuration is sent to the browser.
kpi#4560 Support Django Constance settings with translatable strings
Add a wrapper class to be used python objects such as dict or list with translatable strings
kpi#4550 Clean up the environment (API endpoint)
kpi#4552 Remove unused variable report__base
kpi#4505 Upgrade React Router to 6.14.2 and use upstream usePrompt
kpi#4547 Bump pip-tools version to 7.x.x
Fixes broken GitHub CI Python unit tests because of conflicting version of pip and pip-tools.
kpi#4538 Improve fetch api handler to reject non-ok responses
Make Update Password Form use more detailed error messages.
kpi#4506 Updating usage endpoints for organizations and subscriptions
Updated api/v2/service_usage to account for organizations and subscriptions.
kpi#4531 Rename the default plan
kpi#4517 Remove build frontend code in pytest CI step
kpi#4512 Split out PasswordUpdateComponent
Code cleanup.
kpi#4450 split top tabs from formViewTabs file and remove x button
With introduction of Project Views, the “x” button on a single project route stopped making sense. It was removed to avoid confusion.
kpi#4526 Redo package-lock
Recreate frontend package-lock file from an earlier version to fix build problems.
kpi#4528 Add private_data field to ExtraUserDetail
Added ‘private_data’ field to the extra user detail model.
kpi#4523 Avoid CharField for dates and use ReadOnlyField instead
kpi#4521 Clarify where (re)deployment writes to database
kpi#4503 Upgrade to Storybook 7
kpi#4520 Get Storybook compatible with TypeScript 5.x
Fixed a compatibility issue between Storybook and TypeScript 5.x.
kpi#4509 Organizations API
Allow creation/updating/deleting of organizations via API.
kpi#4387 Toolchain: Warn about undefined CSS Module classes in TypeScript
Show a warning in your code editor if you use an undefined CSS module class in a TypeScript file.
kpi#4366 Print a warning when used with Node versions other than v16.15.0
Warn in development when running the JavaScript frontend with unsupported Node or NPM version.
kpi#4474 Cleanup password related components
Internal code improvements.
kpi#4457 Drop underscore
Inner cleanup. Removes unnecessary underscore dependency in favor of granular lodash packages.
kpi#4326 Run prettier on projects code
kpi#4634 Fixes annual price showing as a large monthly amount
kobocat#894 Prevent SKIP_HEAVY_MIGRATIONS warning unless running the relevant migrations
Refactors main.0011_drop_old_kpi_tables to not print a heavy migration warning every time Django loads the migration file (e.g. when running ./manage.py migrate for an unrelated app).
kobocat#893 Update populate_xform_submission migration dependencies
Applying logger.0028 before main.0012 (from #885) raises an error (column main_userprofile.validated_password does not exist). This PR updates the dependency list for logger.0028 accordingly.
kobocat#891 Purge non-dict UserProfile.metadata when calculating submission counters
kobocat#886 Code refactoring: Use default_storage instead of get_storage_class()()
Django provides a default_storage variable which exactly what get_storage_class()() is doing.
See https://github.com/django/django/blob/3.2/django/core/files/storage.py#L362-L367
kobocat#890 Revise line endings and comments in #889
kobocat#888 Bump pip-tools version to 7.x.x
Fixes broken GitHub CI Python unit tests because of conflicting version of pip and pip-tools.
kobocat#869 Improve XML parsing with defusedxml
Switch to defusedxml instead of the standard xml library for parsing
formpack#310 Clean up unused and badly formatted code
kpi#4635 Account for null XForm instances in service_usage endpoint
Fix /api/v2/service_usage/ calculating NLP and submission counts incorrectly.
kpi#4636 Use NLPUsageCounter date field to fix User Statistics report
Fix the superusers User statistics report
kpi#4644 Replace curl to bash with Nodesource Debian package
Removes a 60-second delay caused by Nodesource’s deprecation of their curl to bash installation method.
kpi#4640 Stripe checkout now collects billing address info.
3 Likes

The latest release is now 2.023.37d and includes the following additional changes:

PR Description
kpi#4665 Improve /api/v2/service_usage endpoint response
kpi#4661 Make storage calculation more accurate with deleted projects
Do not wait for the background task to delete projects before update the user’s storage
kpi#4655 Usage limits release fixes
Fixes a bug where users on the Professional plan were showing usage limits from the Legacy plan.
kpi#4639 Refactor Usage frontend data fetching
Prevent some extra API calls that were slowing down the Project list/Usage page.
kpi#4654 Tone down over-limit messaging for now
kpi#4487 Improve labeling/ordering of transcripts/translations for grouped audio questions
Place transcripts and translations immediately after their related audio questions, and use field labels instead of XML values when ‘Labels’ is selected in the Table display option.

These fixes for grouped audio apply to the table view column headers and the single submission model rows.
kpi#4640 Stripe endpoint improvements
Stripe checkout now collects billing address info.
1 Like

We have now released 2.023.37e, which fixes a problem where newly-created accounts could not view their API tokens in the account settings user interface (kpi#4672).

Other changes include:

  • kpi#4666, Cache service usage endpoint
    Caches the api/v2/service_usage endpoint to prevent slowdown on sites with a large number of users/projects. Adds a ‘last updated’ widget to the Usage page. Allows promotion codes to be used when checking out for a Plan.
  • kpi#4653, Add check that SOCIALACCOUNT_PROVIDERS don’t use kobo as ID, which would cause a conflict

Thanks for using KoboToolbox!

1 Like

We have released few patch versions:

2.023.37f , which fixes a problem where Usage page was not showing correct date for single users with a Plan (kpi#4673).

  • KPI version 2.023.37f
  • KoboCAT version 2.023.37a

Other changes include:

  • kpi#4674, Fixes reloading page on select all and empty import causing page crash in dynamic data attachments.
  • kobocat#897, Regroup storage counter calculation in the same DB transaction
  • kobocat#896, Improve update_attachment_storage_bytes management command and return a more accurate global storage metric when user has no projects

2.023.37g, which fixes a problem where no default disclaimers were set (kpi#4677).

  • KPI version 2.023.37g
  • KoboCAT version 2.023.37b

Other changes include:

  • kobocat#899, Restore missing monthly submission counters data when management command populate_submission_counters was run prior to this version

2.023.37h, which removes replaced/deleted attachments on submission edit in API responses (kpi#4681, kobocat#792).

  • KPI version 2.023.37h
  • KoboCAT version 2.023.37c

Other changes include:

  • kpi#4686, Fixes 500 error when loading projects list with PostgreSQL 13 and older
  • kpi#4682, Makes name field required by default (It brings back to previous behaviour, i.e. Full name is required all the time when users create a new account)
  • kpi#4660, Shows storage add-ons in usage limits. Previously purchase storage add-ons are now used in usage limit calculations. Corrects some issues that were causing incorrect limits to be displayed.
  • kpi#4679, Fixes 500 error when previewing multilanguage forms with disclaimer message
  • kobocat#901, Fixes failing ZIP exports
  • kobocat#900, Makes user not nullable in DailyXFormSubmissionCounter

Self hosters

You need to run some management command:

  • in KPI: ./manage.py populate_asset_date_deployed -v 2. This one is not mandatory but strongly recommended if the results of this query below returns something greater than 0
Asset.objects.filter(_deployment_status='deployed', date_deployed__isnull=True).count()
  • in Kobocat: ./manage.py soft_delete_orphan_attachments -v 2
1 Like

For reference:

1 Like

A new patch release is available and has been deployed to our production servers at 19:00 UTC on Monday, 20 October 2023:

2.023.37i , which fixes the project search (kpi#4694).

Other changes include:

  • kpi#4658, Support animated pictures on drap&drop upload with Admin markdown editor (Backport PR from beta branch).

  • kpi#4688, Clarify that OrderCustom supports CharField only.

  • kobocat#850, Expunge UserProfile.created_by.

  • kobocat#902, Improve soft_delete_orphan_attachments management command speed. Only process edited submissions.

  • kobocat#903, Fix 500 error when fetching data with a date range.
    :warning: /api/v1/data/<pk>.<format> is deprecated. /api/v2/assets/<uid>/data.<format> should be used instead.

1 Like

A new patch release is available and has been deployed to our production servers at 16:00 UTC on Tuesday, 14 November 2023:

2.023.37j , which fixes the missing background audio recordings and audit logs (kobocat#905).

Self hosters

If your instance is already on version 2.023.37c and later, you need to run this management command in Kobocat ./manage.py soft_delete_orphan_attachments_2_023_37c_fix -v 3 to bring back missing background audio recordings and audit logs in existing data.

2 Likes

A new patch release is available and has been deployed to our production servers at 22:00 UTC on Thursday, 16 November 2023:

2.023.37k , which fixes the missing encrypted files (kobocat#906).

Self hosters

If your instance is already on version 2.023.37c and later, you need to run this management command in Kobocat ./manage.py soft_delete_orphan_attachments_2_023_37c_fix -v 3 to bring back missing encrypted files (even if you have already run it once with 2.023.37j).

1 Like

A new patch release is available. It has not been deployed yet to our production servers.

2.023.37l , which fixes the export settings being grayed out in certain circumstances (Backport PR from beta branch).

1 Like

The latest release is now 2.023.37m, which removes project submission counts from the view of users who have row-level “partial” permissions. The sometimes-complex filters for these row-level permissions led to 504 errors for large projects when the database could not the number of matching submissions fast enough. For technical details, see Display submission counts only for users who have unrestricted access to view submissions by jnm · Pull Request #4739 · kobotoolbox/kpi · GitHub.

This release has been deployed to both Global and EU instances as of 18:30 on 28 November 2023.

1 Like

A new patch release is available and has been deployed to our production servers at 17:00 UTC on Thursday, 7 December 2023:

2.023.37n , which fixes an error when using connected projects (kpi#4741, kobocat#909).

Other changes include:

  • kpi#4749: Require “edit submissions”, not “add submissions”, for working with transcripts and translations
    Doing any kind of work with transcripts or translations—whether adding a new one or editing an existing one—now always requires the ““edit submissions”” permission."
1 Like