Hello KoboToolbox community,
We’re pleased to announce that a new release of KoboToolbox has been deployed to kf.kobotoolbox.org as of 21:00 UTC on 17 November 2022 and to kobo.humanitarianresponse.info as of 21:30 UTC on 28 November 2022.
The components comprising this release are:
- KPI version 2.022.44
- KoBoCAT version 2.022.44
- Enketo Express version 4.1.3, upgraded from 4.1.2, which was previously deployed without announcement. For a list of Enketo changes, please see the Enketo change log.
- Pyxform version 1.9.0 (unchanged)
Attention self-hosters: This release contains a new version of Django, which requires you to upgrade your databases. Please review the database upgrade procedure carefully before installing this release.
New features
Translation and transcription of audio questions (including natural language processing, for beta testers only)
With this release, it is now possible to transcribe audio collected during surveys into text, and to translate these transcriptions. Speech to text (automatic speech recognition) and machine translation are also supported for several dozen languages, using Google Cloud Compute. Transcripts and transcript translations can be exported along with other data.
Automatic transcription and translation is currently available only for partners and beta testers. To sign up to become a beta tester, please fill in this form with your details to apply.
Improvements
| PR | Description |
|---|---|
| kpi#4040 | Duplicate attachments with duplicated submissions Ensure that attachments are duplicated along with duplicated submissions. This was not included in the original work done in #2897. |
| kpi#4026 | Show more text in Data Table column headers Display up to 3 lines of text for column headers, trimmed if longer. Users can still see the whole text when hovering. |
| kpi#3363 | Make it possible to add whole groups to the question library from within the form builder |
| kpi#3999 | Better filtering for select_multiple columns in Data Table Changes the text input filtering into a dropdown with all available choices |
| kpi#3973 | Enable filtering by calculate and barcode in Table View |
Fixes
| PR | Description |
|---|---|
| kpi#3980 | Add time to gallery date filters Add time to the dates generated by the date pickers in the gallery |
| kpi#4029 | Fix crash in Single Submission Modal for choices without labels |
| kpi#3972 | Revised notification system, patched some vulnerabilities Finished migrating old alertify toast notifications to react-hot-toast notifications.Escaped user generated HTML before it is rendered by alertify dialogs. |
| kpi#3956 | Resize all image elements to fit in their container by default Prevent large images from stretching across table cell boundaries in the single submission modal. |
| kpi#3950 | Treat unmarked timezone as UTC, convert _submission_time from UTC to local Convert _submission_time from UTC to local time in the table view. |
| kpi#3667 | Fix upload and import modal properties not being applied Fix a bug introduced by configurable metadata feature - in new project modal, when uploading or importing a file, the imported name and settings were not applied. |
| kpi#3901 | Update archive button Update archive button looks to distinguish it from delete button. |
| kobocat#849 | Fix timeout when deleting many submissions at a time. Improve (drastically) the time to delete multiple submissions at a time. |
| kobocat#848 | Forbid DELETE requests on data endpoint.DELETE requests are not allowed anymore on KoBoCAT API. API v2 (i.e. KPI) should be used instead.Only service account is allowed to call the API and make DELETE requests to it. |
| kpi#3786 | Upgrade to Django 3.2 Upgrade code base to use Django 3.2 LTS |
| kobocat#815 | Upgrade to Django 3.2 Upgrade Django code to latest Django LTS |
| kobocat#840 | Reject submissions with error if they cannot be written to MongoDB |
| formpack#304 | Handle multiple spaces in or_other Handle multiple spaces in select type questions between list_name and or_other option. |
| formpack#300 | Remove simserial and subscriberid from metadata removed simserial and subscriberid from the meta data as Google no longer provides this information |
| formpack#274 | Handle inf and nan values for XLS exportsFix handling of inf and nan values when attempting to export types to XLS format. |
| formpack#276 | Fix combining fields for repeat groups across versions Update FormPack.get_fields_for_versions() to look for the field and section names rather than just the field names when collecting all the versions’ fields. |
| formpack#300 | Remove simserial and subscriberid from metadata removed simserial and subscriberid from the meta data as Google no longer provides this information |
| formpack#304 | Handle multiple spaces in or_other Handle multiple spaces in select type questions between list_name and or_other option. |
| kpi#4025 | Fix “different root names” Enketo error when editing a submission Always regenerate (and update the root of the <instance> node) of the XML of the form when editing a submission to match the submission root name. |
| kpi#4119 | Fix lost Project Settings name Update the name in Project Settings after it was being changed in the app header. |
Of interest to self-hosters
| PR | Description |
|---|---|
| kpi#4098 | Fix missing media files after overwriting the same files in Azure environment Make Django Storage behave the same with AWS S3 and Azure Blob Storage. (included in patch release 2.022.43a) |
| kobocat#853 | Do not override files when using Azure Storage Make Django Storage behave the same with AWS S3 and Azure Blob Storage. (included in patch release 2.022.43a) |
| kpi#3770 | Superuser Reports Created reports for superusers: - Continued usage report - Domains report - Forms count by submission range - Media storage report - User count by organization Also added a page to /superuser_stats/ to list all reports |
| kpi#4031 | Allow registration from listed domains only Adds a feature which will only allow users with specified email domains to register for the the app. |
| kpi#4038 | Remove newlines in user details export Remove newline characters in user details CSV export that cause issues in Excel, Data Studio, etc. |
| kpi#4027 | Log submissions delete actions When submissions are deleted, the action is stored in DB and expose to superusers (only) through a new JSON API endpoint (i.e.: /api/v2/audit-logs/) |
| kpi#3976 | Superuser user list endpoint and export Create a user-list endpoint with only superuser access and ability to export list of users to CSV. |
| kpi#3741 | Service usage API endpoint Created a service usage endpoint so users can monitor their use of services |
| kpi#3997 | 404 error fix when using the “Login as” button in Admin interface When a super user tries to log in another user account, they do not receive a Page not found anymore. |
| kpi#3863 | Manage celery autocale via environment variables instead of constance When managing Celery autoscaling, previously this would be set by a Django Constance setting in the Database. This was set during application startup using a Celery task to apply the setting. This logic would always run at app startup, slowing it down. It would also run multiple times in multiple server deployments. This change makes it possible to set Celery autoscaling min/max using the new environment variables CELERY_AUTOSCALE_MIN and CELERY_AUTOSCALE_MAX which both default to 2. Most users can ignore this entirely. However large deployments that set this, need to migrate to the new settings. |
| kpi#3767 | Extended Django Admin User View Added a date joined field to the Django Admin user list and added deployed forms count and submissions count to the changelist view |
| kpi#3986 | Use new counters to display users’ submission statistics in Admin interface Fixes the error saying django.db.utils.ProgrammingError: relation "logger_submissioncounter" does not exist. |
| kpi#3844 | added attachment_storage_bytes to KobocatUserProfile and KobocatXForm… Added an attachment_storage_bytes field to KobocatXForm and KobocatUserProfile shadow models |
| kpi#3978 | Avoid crash when kobocat database does not exist In a fresh-install situation, such as when kobo-docker/.vols has been removed, the kobocat database may not yet exist when kpi configuration checks run. That led to a django.db.utils.OperationalError: FATAL: database "kobocat" does not exist error caused by check_for_kpi_data_in_kobocat_database(). This change allows the configuration checks to pass without a kobocat database. Both kobocat and kpi use the wait_for_postgres.bash script to create their databases, but each application only creates its own database (for example, kpi does not create a database for kobocat). |
| kobocat#841 | Add a management command to populate users’ daily and monthly submission counters The management command calculates all submissions per user, per form for specific time number of weeks (default is 8). |
| kobocat#828 | Attachment storage bytes management command Added a management command to recursively add storage amounts to XForm and UserProfile attachment_storage_bytes fields |
| kobocat#846 | Fix wrong schedule for submission counter background task Change schedule of daily submission count deletion from every 0 seconds to once everyday at midnight UTC. |
| kobocat#836 | Create daily and monthly xform submission counter models Created DailyXFormSubmissionCounter and MonthlyXFormSubmissionCounter models to improve the request speed for form statistics |
| kobocat#827 | Add attachment_storage_bytes fields to XForm and UserProfile models Add attachment_storage_bytes fields to XForm and UserProfile models to for reporting purposes.For a given submission, the storage counter will always increase after editing and changing (or removing) attachments because Attachment objects are never modified nor deleted when editing a submission. To reclaim storage space, the submission must be deleted entirely. |
| kobocat#835 | Raise superuser_stats time limit to one hour |
| kpi#4112 | Fix non dismissable in-app messages It is again possible to dismiss in-app messages created with always_display_as_new option. |
Of interest to developers
| PR | Description |
|---|---|
| kobocat#852 | Merge deleted xforms’ monthly submission counters |
| kobocat#847 | Update KOBOCAT_BROKER_URL default for consistency with kobo-install |
| kpi#4092 | Refactor modal Introducing new modal component with demo available at /design-system. |
| kpi#4086 | Do NOT enable verbose = true for Black |
| kpi#4072 | Refactor ApiTokenDisplay component |
| kpi#4030 | Make envStore use MobxPart of gradual move away from Reflux. |
| kpi#3979 | Add Stripe data sync to track public kf.kobotoolbox.org usage limit exemptions Add Stripe model syncing for tracking kf.kobotoolbox.org usage exemptions. |
| kpi#4046 | Remove ‘metadata’ field from the Account Settings UI Remove ‘metadata’ field from the Account Settings UI. |
| kpi#4044 | Merge Cypress tests Merge some work done on the Cypress branch. Adds Cypress tests and accompanying documentation. |
| kpi#4042 | Shorten toast notification duration to 5 seconds Show toast notifications for 5 seconds, rather than 10 seconds. |
| kpi#3735 | Wrap KoBoCAT DB writes into transactions where needed Rollback inserts/updates in KoBoCAT database if one of queries sent to the DB fail inside the same transaction. |
| kpi#3983 | Use a service account to authenticate KPI requests to KoBoCAT All requests to KoBoCAT are authenticated with the service account user (provided by kobotoolbox/kobo-service-account) on behalf of the current authenticated user. |
| kpi#3998 | Refactoring: Make the spelling of submission count variables consistent Make the spelling of submission_count variables consistent |
| kpi#3793 | Update django-trench to 0.3.1Update MFA third-party library ( django-trench) to add better support for Python 3.10 |
| kpi#3966 | Add Stripe data sync to track public kf.kobotoolbox.org usage limit exemptions |
| kpi#3962 | Add SAST to Gitlab pipeline Add SAST to Gitlab pipeline for automated security testing |
| kpi#3961 | Initial PR for switching alertify with react-hot-toast notification handlersMoving from alertify to react-hot-toast due to reasons outlined in #3837. This PR just adds the library and replaces the utility method notify which handles a lot of global notifications.Work that needs to be done to complete the switch: - Find page specific alertify overrides and replace them with notify- Add override cases to notify or a similar utility function- We use alertify to show dialogs as well, so we need to:- Create an extendable component for dialogs - Move the CSS over ← might be a good time to update the styles and standardize them |
| kpi#3918 | Consistent usage of Mfa acronym in the code base Refactor code base to write every class name containing the acronym MFA with a capital M and fa in lower case. |
| kpi#3953 | Add mfa_code_length to environment API test |
| kpi#3899 | Refactor HelpBubble code using MobX Simplified and refactored code for HelpBubble UI using MobX as a state management solution. |
| kpi#3931 | Use mfa_code_length in UI Display the same MFA code length in UI as set on backend. |
| kpi#3691 | Fix crash when getRowLockingProfile is called without asset Fixing error that is showing up in Sentry (GlitchTip), but I wasn’t able to reproduce it - some edge case perhaps? |
| kpi#3882 | MiniAudioPlayer component Add new MiniAudioPlayer component. It has a play/stop button and a MM:SS time text. |
| kpi#3927 | Improve TODO comments Improve TODO comments in the code by providing each with a GH issue url. |
| kpi#3842 | Typescriptize help bubbles code Move help bubble code to typescript and trim half of it that we didn’t use anymore. |
| kpi#3860 | Add data-cy attribute to all common components Adds data-cy attribute to all our common components (buttons, dropdowns, etc.), so they can be easily targeted while writing Cypress tests. |
| kpi#3889 | Lint Toggle Switch component code Internal code improvements. |
| kpi#3884 | Updating cypress branch, and adding testserver and three cypress tests. |
| kobocat#845 | Use a service account to authenticate KPI requests to KoBoCAT Grant access to API for requests with special service account authentication headers |
| kobocat#839 | Add SAST to Gitlab pipeline |
| kobocat#833 | django-digest redis cache nonce storage backend |
| kobocat#826 | Remove unused savReaderWriter dependency Avoids an approximately 90 MB download |
| formpack#278 | MVP: Handle extra fields to export Allow for translations, transcriptions and coding questions to be integrated into the export. |
| formpack#308 | Update tests to match qpath in supplementalData |
| kpi#3428 | Single processing view New route with skeleton UI for all the processing features. |
| kpi#3488 | Single processing view improvements Some improvements for Single Processing View after first round of testing. |
| kpi#3513 | Language selector A language selector component for NLP feature. Filter a list, choose an existing language or use typed value as custom - end up with getting a language code/custom value. |
| kpi#3532 | Button component (broken branch) Accidently I merged to much stuff here. I created a clean branch with PR at #3540. Will merge this into feature/nlp as it’s already there. |
| kpi#3546 | [NLP] The single processing view feature This is a feature branch with zero changes. It was created to gather all PRs from NLP project, so when each of them are ready, they will not go directly to beta until whole feature is ready.This will probably include all work from the NLP MVP milestone. |
| kpi#3578 | Add analysis fields to be toggled in export Allow for inclusion of analysis fields to be toggled in export. |
| kpi#3581 | NLP transcript and translations editors This adds a transcript and translations editors to the NLP interface. |
| kpi#3819 | Add nlp_features_enabled boolean to restrict ASR/MT usage for specified user listAllow for list of users to be added to the Constance config setting NLP_USER_WHITELIST through the Django admin interface. Add nlp_features_enabled boolean to /environment endpoint with value of True if the requesting user is in the specified whitelist, False otherwise. |
| kpi#3822 | Read Google ASR/MT API credentials from Constance |
| kpi#3826 | Automatic processing (simplified UI) Enables automatic transcription and translation (via just one service: Google) using a simplified UI (a button with a spinner). |
| kpi#3955 | New API endpoint to expose all languages Add a new API endpoint (at /api/v2/languages/) to list all languages (and their regions) imported in the admin interface. |
| kpi#3968 | [NLP] Extend language selector to almost 7000 languages available Introduces a bigger list of available languages (around 7000) for the NLP feature. Due to the size of the list, users can expect some spinners in the language selector component. |
| kpi#3990 | [NLP] Small UI tweaks for Processing View |
| kpi#3991 | [NLP] Automatic transcription and translation for Processing View |
| kpi#3992 | [NLP] Fixing how we handle questions in groups for Processing View This fixes mostly the automatic transcription and translation for the questions in groups by introducing a better way to identify them. |
| kpi#3993 | [NLP] Introduce regions to Language Selector for Processing View Bring the big languages list into the automated part of Processing View, and introduce regions to the language selector. |
| kpi#4002 | Async NLP transcript and refactor gcloud and speech interaction |
| kpi#4028 | nlp - transcribe in progress status |
| kpi#4087 | Separate languageCode and regionCode from automatic transcripts |
| kpi#4123 | remove fields starting with $ from XLSForm exports |
| kpi#4117 | Fix project settings name not being able to be edited (unreleased bug) Project name in the settings was not editable due to bad merge conflict resolving. |
| kpi#4113 | Move LanguageSelector separator line below featured languages Places the separator below featured languages - now they will be grouped together with suggested languages. |
| kpi#4140 | Remove extra ‘Primary Sector’ label in Library Asset form |
kpi eda7758 |
Fix <asset uid>/xform debugging route |
| #4133 | Improve audit log API documentation |
