Dear Community,
Please help with next issue:
I have 138000 submissions and i cannot export them to power bi, it is always this error 504 (Gateway Timeout). I read that it was a submission limit of 30000. Could it be the case?
Please suggest the method to connect it all to power bi.
Thank you!
Power BI and Kobo don’t have a limit of 30000. There is a connection mistake somewhere.
I will share a tutorial with you, if you still have problem, let me know. https://youtu.be/53Se9rWc9bc
Hi @osmanburcu
Thank you for quick response and video.
Tried ypur method on two forms 1) 138000 submissions 2) 7.,200.
For 1) it is not working - still have a 504 problem.
for 2) works perfectly.
Do you know any other methods?
I don’t think it is about the row number, because i just imported 720,086 row data to power bi. HTTP 504 error is a time-out error, it is probably because your data takes too much time to process by the Kobo for exporting as excel, and that causes time-out error. Just try to download the data as excel and try to import it to Power BI, if it works that way, the problem is the processing time of the Kobo
Dear @osmanburcu
Thank you for support!
Im just trying to connect power bi directly to kobo to avoid excel and other programs. As Im having 600+ new submissions per day. Also, I want to make it reltime data.
Are you connecting this big dataset directly to power bi?
I understand what you are trying to achieve, what i wanted to show you was the total number of the row was not the problem, you can import data directly to Power BI till 1GB. So what i understood from your side is, how long does it take to create your data by Kobo. Each time you would like to update your data Kobo needs to prepare as you are going to download it as long as you are not importing it as JSON. So first you need to able to tell us how long does it take to prepare your data, the easiest way is to click download and see.
Dear @osmanburcu kobo creating the dataset for 5 minutes, all submissions. I`ve tried JSON, but it is limited to 30000 submissions.
Thank you for your support!
Did you have any chance to solve your problem? I might find your problem while reviewing the API document. As you can see below, the 30000 limit comes from the Kobo side not from the Power BI side, I believe what you need to do is you need to split you data set into 30000 batches, and import each one and merge them in the Power BI. can you check and let me know?
https://eu.kobotoolbox.org/api/v2/assets/{formuid}/data.json/?start=0&limit=30000
https://eu.kobotoolbox.org/api/v2/assets/{formuid}/data.json/?start=30000&limit=60000
https://eu.kobotoolbox.org/api/v2/assets/{formuid}/data.json/?start=60000&limit=90000
https://eu.kobotoolbox.org/api/v2/assets/{formuid}/data.json/?start=90000&limit=120000
https://eu.kobotoolbox.org/api/v2/assets/{formuid}/data.json/?start=120000&limit=1500000
Every time I tried to export the data, I encountered a recurring error message indicating a 504 Gateway Timeout. After researching potential causes, I discovered that there is a submission limit of 100,000 records for exporting data from KoboToolbox. This limitation posed a challenge for me as I needed to analyze the entire dataset in Power BI for a comprehensive evaluation.
Hi @MacroDavis,
You can split them into a few groups import them to Power BI and Append them in Power BI. If it doesn’t work for you, check if you are able to download your data as XLS, if you can, just use API v2 to import it kobo as Excel the whole file, I might work
