Update
We have recovered what we believe is all data affected by this error, i.e. projects that were lost after 07:00 UTC on 22 January. More specifically, our logs show the first effects of this problem on HHI at 07:28 and on OCHA at 07:32.
Original Message
Dear community,
Thank for your making us aware of a serious issue today with the KoBoToolbox platform. We have fixed the issue and no further projects should be impacted. Recovering all missing data is now our top priority, and our technical team is dedicating their full attention to this right now.
- The 2.020.52b release at the end of 2020 included a small change intended to fix an issue where attempting to create a new project from an invalid XLSForm upload resulted in an empty “Untitled” form being added to the list of drafts. We were not aware of this change having any negative side-effects.
- The 2.021.03 release that was deployed this morning around 07:20 UTC suffered from a server problem that caused many more imports to fail than normally would, including many imports with valid XLSForm. The technical reason for this is: imports are handled by a pool of separate worker processes, and some of them failed to update to the latest version of our code. The previous version of the code expected a database column to exist that had been removed, so the workers running that old version could not complete any imports.
- This increased failure rate made it obvious that the change in 2.020.52b definitely did have a serious side-effect: existing projects could be deleted if an attempt to replace them with new XLSForm failed.
- Thanks to reports from this community, we identified the problem and deployed a fix (2.021.03a) to stop these deletions from happening at 13:31 UTC today. Some imports were still failing—without any deletion resulting from that—up through 14:41.
- We are now in the process of recovering all deleted data from backups. It’s likely this process will take several hours to complete. We will post updates here as we make our way through the work.
Thank you, as always, for your patience and support as we endeavor to provide a useful data-collection tool for those who need it most. As the lead developer of KoBoToolbox, I personally apologize for the stress and disruption caused by this failure.