Need urgent help to download data from one of my projects (Having download error)

[Kal_Lam]

Does it have a repeat group question?

[icydtransfer]

Yes it does

[Kal_Lam]

Did you make any changes to the repeat group and then redeploy?

[icydtransfer]

aGbBAuoGnzFPS678RqntjR.xlsx (22.8 KB)
Please find attached the XLS Form

[Kal_Lam]

Did you edit the repeat group and then redeploy your survey project?

[icydtransfer]

No. I haven’t edited the repeat group

[Kal_Lam]

Well have you tried downloading your data as legacy format as outlined in the image shared below:

[icydtransfer]

Yes I have tried but it also showed the same error

[Kal_Lam]

Seems like you are having issues as you have a lot of submissions.

[icydtransfer]

True there are quiet a number of submissions. How can i access the data for now? Is it possible for your to download it for me from the back end?

[Kal_Lam]

OK maybe you could download your data as outlined in our previous post:

What's the best way to retrieve records through API when survey is over 30,000 records?

Hi all, I’m trying to retrieve csv data from the API but I’m failing to do that when the survey has more than 30,000 submissions. I found that, in theory, it should be possible to use MongoDB filters to sort and paginate the queries. However I’m having trouble getting it to work. Has anyone faced a similar challenge? Is there an alternative that can be used to get the same result? For example when calling: https://kc.humanitarianresponse.info/api/v1/data/XXXXX?format=csv&start=30000&sort=%7B%22…

[icydtransfer]

This isn’t yet working

[Kal_Lam]

@icydtransfer, could you share the following though a private message:

• Username
• Project name
• Server

Maybe we could have a closer look at it?

[icydtransfer]

Username: xxxxxxx
Project name: home visit tool
Server: Humanitarian server

Thanks

[Josh]

Hi @icydtransfer, I apologize for this inconvenience — it’s due to the export timing out on the server with the large number of submissions. I’m looking at it now to see what I can do

[icydtransfer]

Thanks @Josh. Looking forward to the solution

[Josh]

Hi @icydtransfer, the legacy XLS export was successful so please use that if it’s urgent. We are going to be doing maintenance on the server soon so I’ve been advised not to start a large export until after that has been completed

[Josh]

Hi @icydtransfer I was able successfully export your data. Unfortunately due to the size of your project it will continue to fail if you try exporting from the UI or through the API — so I will have to due it manually from the server.

We will be introducing a new date range filtering feature for exports soon, so hopefully this won’t be an issue for much longer as you can just export the data submitted since your previous export.

[tomisin]

Thank you so much the KoboCollect team. This has also been a concern for me too.

Your timely support is the only thing that gives me confidence around any misappropriation, well done.

While the date filtering option for download will be of a great help - I can’t wait to see it on the download option - for those who wants to download to a local file, how would we be able to go around it if we are using the API and it is connected to a BI tool for visualization ?

I currently have a survey with over 30000 submissions and more than 120 questions that is connected to Power BI for real time visualization. However, I do sometimes get error messages that the data extraction time out, what can I do to solve this?

Can the time for the data extraction be increased ?

[Josh]

Hi @tomisin, we appreciate your feedback

You can track the export filtering progress here and hopefully it can slip into one of the next releases. Unfortunately it will not initially address the integration issue yet, although this is something that will be incorporated in the future.

Unfortunately server load can affect projects of your size and result in occasional time-outs. We are considering how to better handle this at the moment, so hopefully it will be resolved in the coming months.