Doanload Large Dataset in XLS form

User Support Data Management
, ,
1

Hi
I need help with downloading large Kobo exports reliably (especially geotagging / GPS + photos type forms) and saving them into SharePoint for our automation pipeline.

:white_check_mark: What we need:
Just a reliable way to download/export the full data (preferably Excel, or XLS ) and then save it in SharePoint.
This works fine for small forms, but fails for large/high-intensity exports.

What we’ve tried (still failing):

  1. Manual export from Kobo UI

    • Often stays on “Exporting/Processing…” and then fails

    • Sometimes works late at night but fails again the next time

  2. Power Automate download

    • Sometimes fails completely

    • Sometimes downloads partial/half data (not full records)

  3. Python via Kobo API + Azure Automation/Function

    • Works occasionally

    • Most of the time fails due to long runtime / unstable export completion for large datasets

  4. Paging approach (API pagination)

    • We tried paging as well, but still not able to reliably fetch the full dataset for these heavy forms

:pushpin: My questions:

  • Is there a known limitation for exporting/downloading very large datasets (geotags + images metadata)?

  • What is the best practice to download large data reliably? (bulk export vs API vs attachments separate?)

  • Is there any recommended approach to schedule exports/downloads daily?

  • Do we need a paid subscription / Kobo Enterprise for stable large exports, background exports, or higher limits?

Any guidance, tips, or examples would be really appreciated :folded_hands:
Thanks in advance!

2

Welcome to the community, @Parnoti! Could you also share with us the total number of submissions you have in your project that is failing when downloading?

3

Hello, Thanks for your reply.
so basically we have 1969 submissions, and if we count values for each( like if we download the data as a excel we get 2 sheets, 1 is for form submission and 2nd sheet is in depth unit values which we required everytime) its around 400K+.

when i tried power automate and python, both time i used download the file when “Status = “complete” , then it wait for 3..4 hours and fails.

also our technical team tried to download manually it takes 6..7 hours to them. for me if i tried to download it in the night it sometimes works sometime dont work ( fails).

as in future we want it to automate the process , need your help to understand how we can deal with large data? or if we take paid version of kobo will it be work?

Thanks

4

@Parnoti, if you are comfortable working with data in JSON format, this API endpoint should expose your project data:

`https://Kobo_URL/api/v2/assets/asset_id/data.json`