This work still in progress. We’re staying up through the night (Eastern Time) to get it done.
The 250+ point-in-time recoveries needed for HHI are complete, and the 400+ needed for OCHA are in progress. While OCHA finishes, we will proceed with restoring the recovered data to the HHI production server.
We believe that all deleted projects on the HHI server (kobotoolbox.org) have been recovered to KoBoCAT only. They will accept new submissions and appear in the “Projects (legacy)” interface. However, viewing or exporting old data may not work yet as we resynchronize the MongoDB read-only replica used for these tasks.
These projects, as well as undeployed draft forms, will be restored to the main HHI interface shortly.
The OCHA server is still processing point-in-time recoveries.
This process is now complete.
Thanks for all your work trying to recover the data. My project still isn’t back. Should I be worried?
This process is now complete. Hopefully, all missing projects on the HHI (kobotoolbox.org) instance are now recovered. (@pcranley, please check again now)
While the OCHA server continues to process recovery points, we will perform some additional checks to verify that the HHI restoration is complete.
OCHA point-in-time recoveries are complete. Projects will be restored to KoBoCAT first, then synchronized to MongoDB, and then restored to the main interface (KPI, kobo.humanitarianresponse.info).
OCHA KoBoCAT restoration complete; MongoDB sync underway.
Your projects will not yet appear in the main interface and submissions may not appear when viewing or exporting from the legacy interface.
The MongoDB resynchronization for OCHA is complete. Now restoring to KPI (the main interface of kobo.humanitarianresponse.info).
The restoration of all data on OCHA / kobo.humanitarianresponse.info should now be complete. We will continue to run tests verifying the recovery on both OCHA and HHI servers, but you may use KoBoToolbox normally.
If you find that your project is still missing, please respond here. Alternatively, you may email 202103.recovery@kobotoolbox.org, but we’re able to respond more quickly to replies on this forum than to emails. Thank you!
Thank you for your efforts and hard work. My 2 projects are back.
Thanks for the hard work of fixing these problems. Must have been a tough few days for y’all.
And more importantly: thanks for keeping kobo.humanitarianresponse.info running smoothly most of the time. Not having to run our own servers saves us a huge amount of time and money. I think we don’t express this enough, so I wanted to take this opportunity to say thanks.
Thank you @Sjlver we appreciate your compliments and most importantly your patience. We will strive to continue providing support to all of you who do critical work within the development sector and the wider research areas.
Stephane
Appreciate the great work!
This kind of thing makes me very nervous, because I’m running Kobo on my own server and I have limited technical skills… I am running an older version of Kobo, and would like to update it to sort out some bugs. And of course avoiding data loss is the topmost priority.
After going through the github docs I didn’t find any update process to ensure that things go smoothly. Is there any documentation anywhere?
@ks_1, you won’t encounter this issue on your own server, but should still be careful of data loss regardless when maintaining your own instance.
Which version of kobo-install are you running? Note that there is a word of caution when upgrading from releases before 2.020.18.
@Josh
How can I check the version running on the server? We had installed it back in July '20 and I believe it was the latest version at the time.
Hi there, at this point we’re considering the recovery to be finished. We haven’t found evidence of other missing data and haven’t heard from anyone about problems related to this event in over two weeks. Please do contact us ASAP if you believe any of your projects were lost and we failed to recover them.
Thanks, as always, for your patience!
Hi,
I am missing one of my forms - it was last reported appearing a month ago. I don’t believe it has been deleted from our end. This is the error message received when trying to follow the form link -
Error occured during the loading of this form. It is recommended not to use this form for data entry until this is resolved.
Please contact undefined with the link to this page and the error message below:
@fmi, could you share with me the following information through a private message so that we could have a closer look at your issue:
