Slow data export / import

User Support Data Management
1

Hello KoBo Community.

I am importing KoBo data from one account in kobotoolbox[dot]org to another kobo account in a self-hosted kobo installation. I am using the kobo-transfer tool and automating it with a python script using the KoBo API as explained here in the forum and in the docs.

Now, I have run into a problem and would appreciate any kind of help from any member who has experienced this or from the core team : the API response is quite slow. I am moving huge amounts of data (millions of submissions, hundreds of forms, several accounts) and at the current speed of import, it may take months to complete this task which is not practical.

  • Is there an alternative way to export and re-import data to another account - sort of like download + upload in one fell swoop ?
  • Can the API at kobotoolbox [dot] org be tweaked to respond faster (a few orders of magnitude faster) ?

NB : My hardware is 4 CPUs and 16GB of RAM. The load average rarely exceeds 0.4 . So I doubt that compute power is the bottleneck here.

–
Kind Regards,
Ndungi

2

Welcome to the community, @ndungi! Have you tried trasfering the projects from one KoboToolbox public server to the other KoboToolbox public server to see the speed? I assume the speed of your self-hosted server is making it take longer. Besides, please also be informed that if your projects are too large, the transfer could take some time.

3

Thank you Lam for your reply. Given that our self-hosted server’s location is unlikely to change, would you happen to have another option to dump/download data as one huge tarball which can then be re-imported into the destination server ? This would be much faster.