Release Notes - version 2.022.44

Hello KoboToolbox community,

We’re pleased to announce that a new release of KoboToolbox has been deployed to kf.kobotoolbox.org as of 21:00 UTC on 17 November 2022 and to kobo.humanitarianresponse.info as of 21:30 UTC on 28 November 2022.

The components comprising this release are:

  • KPI version 2.022.44
  • KoBoCAT version 2.022.44
  • Enketo Express version 4.1.3, upgraded from 4.1.2, which was previously deployed without announcement. For a list of Enketo changes, please see the Enketo change log.
  • Pyxform version 1.9.0 (unchanged)

:warning: Attention self-hosters: This release contains a new version of Django, which requires you to upgrade your databases. Please review the database upgrade procedure carefully before installing this release.

New features

Translation and transcription of audio questions (including natural language processing, for beta testers only)

With this release, it is now possible to transcribe audio collected during surveys into text, and to translate these transcriptions. Speech to text (automatic speech recognition) and machine translation are also supported for several dozen languages, using Google Cloud Compute. Transcripts and transcript translations can be exported along with other data.

Automatic transcription and translation is currently available only for partners and beta testers. To sign up to become a beta tester, please fill in this form with your details to apply.

Improvements

PR Description
kpi#4040 Duplicate attachments with duplicated submissions
Ensure that attachments are duplicated along with duplicated submissions. This was not included in the original work done in #2897.
kpi#4026 Show more text in Data Table column headers
Display up to 3 lines of text for column headers, trimmed if longer. Users can still see the whole text when hovering.
kpi#3363 Make it possible to add whole groups to the question library from within the form builder
kpi#3999 Better filtering for select_multiple columns in Data Table
Changes the text input filtering into a dropdown with all available choices
kpi#3973 Enable filtering by calculate and barcode in Table View

Fixes

PR Description
kpi#3980 Add time to gallery date filters
Add time to the dates generated by the date pickers in the gallery
kpi#4029 Fix crash in Single Submission Modal for choices without labels
kpi#3972 Revised notification system, patched some vulnerabilities
Finished migrating old alertify toast notifications to react-hot-toast notifications.

Escaped user generated HTML before it is rendered by alertify dialogs.
kpi#3956 Resize all image elements to fit in their container by default
Prevent large images from stretching across table cell boundaries in the single submission modal.
kpi#3950 Treat unmarked timezone as UTC, convert _submission_time from UTC to local
Convert _submission_time from UTC to local time in the table view.
kpi#3667 Fix upload and import modal properties not being applied
Fix a bug introduced by configurable metadata feature - in new project modal, when uploading or importing a file, the imported name and settings were not applied.
kpi#3901 Update archive button
Update archive button looks to distinguish it from delete button.
kobocat#849 Fix timeout when deleting many submissions at a time.
Improve (drastically) the time to delete multiple submissions at a time.
kobocat#848 Forbid DELETE requests on data endpoint.
DELETE requests are not allowed anymore on KoBoCAT API. API v2 (i.e. KPI) should be used instead.
Only service account is allowed to call the API and make DELETE requests to it.
kpi#3786 Upgrade to Django 3.2
Upgrade code base to use Django 3.2 LTS
kobocat#815 Upgrade to Django 3.2
Upgrade Django code to latest Django LTS
kobocat#840 Reject submissions with error if they cannot be written to MongoDB
formpack#304 Handle multiple spaces in or_other
Handle multiple spaces in select type questions between list_name and or_other option.
formpack#300 Remove simserial and subscriberid from metadata
removed simserial and subscriberid from the meta data as Google no longer provides this information
formpack#274 Handle inf and nan values for XLS exports
Fix handling of inf and nan values when attempting to export types to XLS format.
formpack#276 Fix combining fields for repeat groups across versions
Update FormPack.get_fields_for_versions() to look for the field and section names rather than just the field names when collecting all the versions’ fields.
formpack#300 Remove simserial and subscriberid from metadata
removed simserial and subscriberid from the meta data as Google no longer provides this information
formpack#304 Handle multiple spaces in or_other
Handle multiple spaces in select type questions between list_name and or_other option.
kpi#4025 Fix “different root names” Enketo error when editing a submission
Always regenerate (and update the root of the <instance> node) of the XML of the form when editing a submission to match the submission root name.
kpi#4119 Fix lost Project Settings name
Update the name in Project Settings after it was being changed in the app header.

Of interest to self-hosters

PR Description
kpi#4098 Fix missing media files after overwriting the same files in Azure environment
Make Django Storage behave the same with AWS S3 and Azure Blob Storage.
(included in patch release 2.022.43a)
kobocat#853 Do not override files when using Azure Storage
Make Django Storage behave the same with AWS S3 and Azure Blob Storage.
(included in patch release 2.022.43a)
kpi#3770 Superuser Reports
Created reports for superusers:
- Continued usage report
- Domains report
- Forms count by submission range
- Media storage report
- User count by organization

Also added a page to /superuser_stats/ to list all reports
kpi#4031 Allow registration from listed domains only
Adds a feature which will only allow users with specified email domains to register for the the app.
kpi#4038 Remove newlines in user details export
Remove newline characters in user details CSV export that cause issues in Excel, Data Studio, etc.
kpi#4027 Log submissions delete actions
When submissions are deleted, the action is stored in DB and expose to superusers (only) through a new JSON API endpoint (i.e.: /api/v2/audit-logs/)
kpi#3976 Superuser user list endpoint and export
Create a user-list endpoint with only superuser access and ability to export list of users to CSV.
kpi#3741 Service usage API endpoint
Created a service usage endpoint so users can monitor their use of services
kpi#3997 404 error fix when using the “Login as” button in Admin interface
When a super user tries to log in another user account, they do not receive a Page not found anymore.
kpi#3863 Manage celery autocale via environment variables instead of constance
When managing Celery autoscaling, previously this would be set by a Django Constance setting in the Database. This was set during application startup using a Celery task to apply the setting. This logic would always run at app startup, slowing it down. It would also run multiple times in multiple server deployments.

This change makes it possible to set Celery autoscaling min/max using the new environment variables CELERY_AUTOSCALE_MIN and CELERY_AUTOSCALE_MAX which both default to 2. Most users can ignore this entirely. However large deployments that set this, need to migrate to the new settings.
kpi#3767 Extended Django Admin User View
Added a date joined field to the Django Admin user list and added deployed forms count and submissions count to the changelist view
kpi#3986 Use new counters to display users’ submission statistics in Admin interface
Fixes the error saying django.db.utils.ProgrammingError: relation "logger_submissioncounter" does not exist.
kpi#3844 added attachment_storage_bytes to KobocatUserProfile and KobocatXForm…
Added an attachment_storage_bytes field to KobocatXForm and KobocatUserProfile shadow models
kpi#3978 Avoid crash when kobocat database does not exist
In a fresh-install situation, such as when kobo-docker/.vols has been removed, the kobocat database may not yet exist when kpi configuration checks run. That led to a django.db.utils.OperationalError: FATAL: database "kobocat" does not exist error caused by check_for_kpi_data_in_kobocat_database(). This change allows the configuration checks to pass without a kobocat database. Both kobocat and kpi use the wait_for_postgres.bash script to create their databases, but each application only creates its own database (for example, kpi does not create a database for kobocat).
kobocat#841 Add a management command to populate users’ daily and monthly submission counters
The management command calculates all submissions per user, per form for specific time number of weeks (default is 8).
kobocat#828 Attachment storage bytes management command
Added a management command to recursively add storage amounts to XForm and UserProfile attachment_storage_bytes fields
kobocat#846 Fix wrong schedule for submission counter background task
Change schedule of daily submission count deletion from every 0 seconds to once everyday at midnight UTC.
kobocat#836 Create daily and monthly xform submission counter models
Created DailyXFormSubmissionCounter and MonthlyXFormSubmissionCounter models to improve the request speed for form statistics
kobocat#827 Add attachment_storage_bytes fields to XForm and UserProfile models
Add attachment_storage_bytes fields to XForm and UserProfile models to for reporting purposes.

For a given submission, the storage counter will always increase after editing and changing (or removing) attachments because Attachment objects are never modified nor deleted when editing a submission. To reclaim storage space, the submission must be deleted entirely.
kobocat#835 Raise superuser_stats time limit to one hour
kpi#4112 Fix non dismissable in-app messages
It is again possible to dismiss in-app messages created with always_display_as_new option.

Of interest to developers

PR Description
kobocat#852 Merge deleted xforms’ monthly submission counters
kobocat#847 Update KOBOCAT_BROKER_URL default for consistency with kobo-install
kpi#4092 Refactor modal
Introducing new modal component with demo available at /design-system.
kpi#4086 Do NOT enable verbose = true for Black
kpi#4072 Refactor ApiTokenDisplay component
kpi#4030 Make envStore use Mobx
Part of gradual move away from Reflux.
kpi#3979 Add Stripe data sync to track public kf.kobotoolbox.org usage limit exemptions
Add Stripe model syncing for tracking kf.kobotoolbox.org usage exemptions.
kpi#4046 Remove ‘metadata’ field from the Account Settings UI
Remove ‘metadata’ field from the Account Settings UI.
kpi#4044 Merge Cypress tests
Merge some work done on the Cypress branch. Adds Cypress tests and accompanying documentation.
kpi#4042 Shorten toast notification duration to 5 seconds
Show toast notifications for 5 seconds, rather than 10 seconds.
kpi#3735 Wrap KoBoCAT DB writes into transactions where needed
Rollback inserts/updates in KoBoCAT database if one of queries sent to the DB fail inside the same transaction.
kpi#3983 Use a service account to authenticate KPI requests to KoBoCAT
All requests to KoBoCAT are authenticated with the service account user (provided by kobotoolbox/kobo-service-account) on behalf of the current authenticated user.
kpi#3998 Refactoring: Make the spelling of submission count variables consistent
Make the spelling of submission_count variables consistent
kpi#3793 Update django-trench to 0.3.1
Update MFA third-party library (django-trench) to add better support for Python 3.10
kpi#3966 Add Stripe data sync to track public kf.kobotoolbox.org usage limit exemptions
kpi#3962 Add SAST to Gitlab pipeline
Add SAST to Gitlab pipeline for automated security testing
kpi#3961 Initial PR for switching alertify with react-hot-toast notification handlers
Moving from alertify to react-hot-toast due to reasons outlined in #3837. This PR just adds the library and replaces the utility method notify which handles a lot of global notifications.

Work that needs to be done to complete the switch:
- Find page specific alertify overrides and replace them with notify
- Add override cases to notify or a similar utility function
- We use alertify to show dialogs as well, so we need to:
- Create an extendable component for dialogs
- Move the CSS over ← might be a good time to update the styles and standardize them
kpi#3918 Consistent usage of Mfa acronym in the code base
Refactor code base to write every class name containing the acronym MFA with a capital M and fa in lower case.
kpi#3953 Add mfa_code_length to environment API test
kpi#3899 Refactor HelpBubble code using MobX
Simplified and refactored code for HelpBubble UI using MobX as a state management solution.
kpi#3931 Use mfa_code_length in UI
Display the same MFA code length in UI as set on backend.
kpi#3691 Fix crash when getRowLockingProfile is called without asset
Fixing error that is showing up in Sentry (GlitchTip), but I wasn’t able to reproduce it - some edge case perhaps?
kpi#3882 MiniAudioPlayer component
Add new MiniAudioPlayer component. It has a play/stop button and a MM:SS time text.
kpi#3927 Improve TODO comments
Improve TODO comments in the code by providing each with a GH issue url.
kpi#3842 Typescriptize help bubbles code
Move help bubble code to typescript and trim half of it that we didn’t use anymore.
kpi#3860 Add data-cy attribute to all common components
Adds data-cy attribute to all our common components (buttons, dropdowns, etc.), so they can be easily targeted while writing Cypress tests.
kpi#3889 Lint Toggle Switch component code
Internal code improvements.
kpi#3884 Updating cypress branch, and adding testserver and three cypress tests.
kobocat#845 Use a service account to authenticate KPI requests to KoBoCAT
Grant access to API for requests with special service account authentication headers
kobocat#839 Add SAST to Gitlab pipeline
kobocat#833 django-digest redis cache nonce storage backend
kobocat#826 Remove unused savReaderWriter dependency
Avoids an approximately 90 MB download
formpack#278 MVP: Handle extra fields to export
Allow for translations, transcriptions and coding questions to be integrated into the export.
formpack#308 Update tests to match qpath in supplementalData
kpi#3428 Single processing view
New route with skeleton UI for all the processing features.
kpi#3488 Single processing view improvements
Some improvements for Single Processing View after first round of testing.
kpi#3513 Language selector
A language selector component for NLP feature. Filter a list, choose an existing language or use typed value as custom - end up with getting a language code/custom value.
kpi#3532 Button component (broken branch)
Accidently I merged to much stuff here. I created a clean branch with PR at #3540. Will merge this into feature/nlp as it’s already there.
kpi#3546 [NLP] The single processing view feature
This is a feature branch with zero changes. It was created to gather all PRs from NLP project, so when each of them are ready, they will not go directly to beta until whole feature is ready.

This will probably include all work from the NLP MVP milestone.
kpi#3578 Add analysis fields to be toggled in export
Allow for inclusion of analysis fields to be toggled in export.
kpi#3581 NLP transcript and translations editors
This adds a transcript and translations editors to the NLP interface.
kpi#3819 Add nlp_features_enabled boolean to restrict ASR/MT usage for specified user list
Allow for list of users to be added to the Constance config setting NLP_USER_WHITELIST through the Django admin interface. Add nlp_features_enabled boolean to /environment endpoint with value of True if the requesting user is in the specified whitelist, False otherwise.
kpi#3822 Read Google ASR/MT API credentials from Constance
kpi#3826 Automatic processing (simplified UI)
Enables automatic transcription and translation (via just one service: Google) using a simplified UI (a button with a spinner).
kpi#3955 New API endpoint to expose all languages
Add a new API endpoint (at /api/v2/languages/) to list all languages (and their regions) imported in the admin interface.
kpi#3968 [NLP] Extend language selector to almost 7000 languages available
Introduces a bigger list of available languages (around 7000) for the NLP feature. Due to the size of the list, users can expect some spinners in the language selector component.
kpi#3990 [NLP] Small UI tweaks for Processing View
kpi#3991 [NLP] Automatic transcription and translation for Processing View
kpi#3992 [NLP] Fixing how we handle questions in groups for Processing View
This fixes mostly the automatic transcription and translation for the questions in groups by introducing a better way to identify them.
kpi#3993 [NLP] Introduce regions to Language Selector for Processing View
Bring the big languages list into the automated part of Processing View, and introduce regions to the language selector.
kpi#4002 Async NLP transcript and refactor gcloud and speech interaction
kpi#4028 nlp - transcribe in progress status
kpi#4087 Separate languageCode and regionCode from automatic transcripts
kpi#4123 remove fields starting with $ from XLSForm exports
kpi#4117 Fix project settings name not being able to be edited (unreleased bug)
Project name in the settings was not editable due to bad merge conflict resolving.
kpi#4113 Move LanguageSelector separator line below featured languages
Places the separator below featured languages - now they will be grouped together with suggested languages.
kpi#4140 Remove extra ‘Primary Sector’ label in Library Asset form
kpi eda7758 Fix <asset uid>/xform debugging route
#4133 Improve audit log API documentation
4 Likes

Hello KoboToolbox community,

As of 2022-11-22 17:00 UTC, a new patch release of KPI and KoBoCAT 2.022.44a has been deployed to kf.kobotoolbox.org.

Improvements

PR Description
kobocat#855 Update the logos and the favicon

Fixes

PR Description
kpi#4144 Fix 500 error when fetching data from service usage endpoint
kpi#4149 Fix service usage endpoint all time calculation
kpi#4150 Fix timeout when superusers try to open MFA user’s details Admin UI
1 Like

Hello KoboToolbox community,

As of 2022-11-28 21:30 UTC, a new patch release of KPI 2.022.44b has been deployed to kf.kobotoolbox.org and kobo.humanitarianresponse.info.

Improvements

PR Description
kpi#4159 Use months range for user statistics report

Fixes

PR Description
kpi#4162 Fix forms count by submissions report
kpi#4160 Fix 404 when date range parameters are omitted
1 Like