Access a deployment of 60,000+ instances

I’ve just started working with a program which deployed a registration form which required a photo of every person in the household. They went forth and collected registration data from over 60,000 households, and now (given poor internet) are unable to access their data. General best practice tips aside (such as regularly pulling and reviewing instances) - are there any solutions which would all a pull of all this data without requiring a continuous connection to the internet?

I’m on a solid connection (in the US) but only able to pull about 15-25 forms a minute (using briefcase or the Kobo dashboard).

Using briefcase I believe I have downloaded all instances, but I am still facing a ‘FAILED’ message – (see attached screenshot). Based on the Kobo dashboard I know the deployment only has 62,920 instances. Briefcase’s status window (before the ‘FAILED’ notice), cites it has fetched 62,920 instances. Even though it has a ‘Failed’ notice, I have tried to export, but get a second error message resulting from an expected .jpg that isn’t found – allowing me to export less than 200 cases.

On Kobo’s dashboard I’m unable to download the data directly, I get a perpetual ‘pending…click to refresh’ message when attempting to download any format of the data.

Any insights, tips, thoughts, to obtain this data would be very welcome.

Hi Lloyd,

If you’re trying to download a very large number of photos you can use an alternative route for getting a list of all the photos as URLs, and then use a download manager to download them to your local computer. We have instructions for this procedure here: http://support.kobotoolbox.org/customer/en/portal/articles/2278188-downloading-photos-and-other-media

Best regards,

Tino

···

On Mon, Jul 11, 2016 at 9:03 AM Lloyd Owen Banwart lloyd....@gmail.com wrote:

I’ve just started working with a program which deployed a registration form which required a photo of every person in the household. They went forth and collected registration data from over 60,000 households, and now (given poor internet) are unable to access their data. General best practice tips aside (such as regularly pulling and reviewing instances) - are there any solutions which would all a pull of all this data without requiring a continuous connection to the internet?

I’m on a solid connection (in the US) but only able to pull about 15-25 forms a minute (using briefcase or the Kobo dashboard).

Using briefcase I believe I have downloaded all instances, but I am still facing a ‘FAILED’ message – (see attached screenshot). Based on the Kobo dashboard I know the deployment only has 62,920 instances. Briefcase’s status window (before the ‘FAILED’ notice), cites it has fetched 62,920 instances. Even though it has a ‘Failed’ notice, I have tried to export, but get a second error message resulting from an expected .jpg that isn’t found – allowing me to export less than 200 cases.

On Kobo’s dashboard I’m unable to download the data directly, I get a perpetual ‘pending…click to refresh’ message when attempting to download any format of the data.

Any insights, tips, thoughts, to obtain this data would be very welcome.

You received this message because you are subscribed to the Google Groups “Kobo Users” group.

To unsubscribe from this group and stop receiving emails from it, send an email to kobo-users+...@googlegroups.com.

To post to this group, send email to kobo-...@googlegroups.com.

Visit this group at https://groups.google.com/group/kobo-users.

For more options, visit https://groups.google.com/d/optout.

Hi Lloyd,

I checked the project in question (one of your colleagues has already been in touch with us separately) and I can confirm that using Briefcase the speed you described is normal due to all the attachments in this case. I checked and it downloads 25 cases per minute for this project at a speed of about 23 MB per minute - many instances have several photos attached. For a project that doesn’t have media attachments the speed is much higher - about 400-500 instances per minute. The reason is probably that the files are served by Amazon and each request takes a little bit of time to resolve each file, download it, then start the next one. Briefcase is not a fast download manager but will get there eventually.

To download your attachments as quickly as possible I recommend the route described earlier. This way you download your files more quickly by saving multiple files concurrently to your computer instead of downloading one at a time.

Best,

Tino

···

On Mon, Jul 11, 2016 at 12:36 PM, Tino Kreutzer tino.k...@kobotoolbox.org wrote:

Hi Lloyd,

If you’re trying to download a very large number of photos you can use an alternative route for getting a list of all the photos as URLs, and then use a download manager to download them to your local computer. We have instructions for this procedure here: http://support.kobotoolbox.org/customer/en/portal/articles/2278188-downloading-photos-and-other-media

Best regards,

Tino

On Mon, Jul 11, 2016 at 9:03 AM Lloyd Owen Banwart lloyd....@gmail.com wrote:

I’ve just started working with a program which deployed a registration form which required a photo of every person in the household. They went forth and collected registration data from over 60,000 households, and now (given poor internet) are unable to access their data. General best practice tips aside (such as regularly pulling and reviewing instances) - are there any solutions which would all a pull of all this data without requiring a continuous connection to the internet?

I’m on a solid connection (in the US) but only able to pull about 15-25 forms a minute (using briefcase or the Kobo dashboard).

Using briefcase I believe I have downloaded all instances, but I am still facing a ‘FAILED’ message – (see attached screenshot). Based on the Kobo dashboard I know the deployment only has 62,920 instances. Briefcase’s status window (before the ‘FAILED’ notice), cites it has fetched 62,920 instances. Even though it has a ‘Failed’ notice, I have tried to export, but get a second error message resulting from an expected .jpg that isn’t found – allowing me to export less than 200 cases.

On Kobo’s dashboard I’m unable to download the data directly, I get a perpetual ‘pending…click to refresh’ message when attempting to download any format of the data.

Any insights, tips, thoughts, to obtain this data would be very welcome.

You received this message because you are subscribed to the Google Groups “Kobo Users” group.

To unsubscribe from this group and stop receiving emails from it, send an email to kobo-users+...@googlegroups.com.

To post to this group, send email to kobo-...@googlegroups.com.

Visit this group at https://groups.google.com/group/kobo-users.

For more options, visit https://groups.google.com/d/optout.