This work still in progress. We’re staying up through the night (Eastern Time) to get it done.
The 250+ point-in-time recoveries needed for HHI are complete, and the 400+ needed for OCHA are in progress. While OCHA finishes, we will proceed with restoring the recovered data to the HHI production server.
We believe that all deleted projects on the HHI server (kobotoolbox.org) have been recovered to KoBoCAT only. They will accept new submissions and appear in the “Projects (legacy)” interface. However, viewing or exporting old data may not work yet as we resynchronize the MongoDB read-only replica used for these tasks.
These projects, as well as undeployed draft forms, will be restored to the main HHI interface shortly.
The OCHA server is still processing point-in-time recoveries.
This process is now complete. Hopefully, all missing projects on the HHI (kobotoolbox.org) instance are now recovered. (@pcranley, please check again now)
While the OCHA server continues to process recovery points, we will perform some additional checks to verify that the HHI restoration is complete.
OCHA point-in-time recoveries are complete. Projects will be restored to KoBoCAT first, then synchronized to MongoDB, and then restored to the main interface (KPI, kobo.humanitarianresponse.info).
The restoration of all data on OCHA / kobo.humanitarianresponse.info should now be complete. We will continue to run tests verifying the recovery on both OCHA and HHI servers, but you may use KoBoToolbox normally.
If you find that your project is still missing, please respond here. Alternatively, you may email 202103.recovery@kobotoolbox.org, but we’re able to respond more quickly to replies on this forum than to emails. Thank you!
Thanks for the hard work of fixing these problems. Must have been a tough few days for y’all.
And more importantly: thanks for keeping kobo.humanitarianresponse.info running smoothly most of the time. Not having to run our own servers saves us a huge amount of time and money. I think we don’t express this enough, so I wanted to take this opportunity to say thanks.
Thank you @Sjlver we appreciate your compliments and most importantly your patience. We will strive to continue providing support to all of you who do critical work within the development sector and the wider research areas.
This kind of thing makes me very nervous, because I’m running Kobo on my own server and I have limited technical skills… I am running an older version of Kobo, and would like to update it to sort out some bugs. And of course avoiding data loss is the topmost priority.
After going through the github docs I didn’t find any update process to ensure that things go smoothly. Is there any documentation anywhere?
Hi there, at this point we’re considering the recovery to be finished. We haven’t found evidence of other missing data and haven’t heard from anyone about problems related to this event in over two weeks. Please do contact us ASAP if you believe any of your projects were lost and we failed to recover them.
I am missing one of my forms - it was last reported appearing a month ago. I don’t believe it has been deleted from our end. This is the error message received when trying to follow the form link -
Error occured during the loading of this form. It is recommended not to use this form for data entry until this is resolved.
Please contact undefined with the link to this page and the error message below:
Oops, this form doesn’t exist (anymore). Most likely the owner of the form has deleted it, archived it, or disabled it. Please contact the form’s owner to confirm. Detailed error: Form with ID a2aNEz6EhxmAHaifEkcLWd not found in /formList.