Move media to S3 command modification

Kobo On Your Own Server
, , ,
1

As noted in another thread, I had to move ~100,000 files from our server to S3. When running the kobocat move_media_to_s3 command, our server ran out of memory (with 16GB). We looked into the code a bit and made a small modification.

from
Line 43: for i in cls.objects.all():

to
for i in cls.objects.all().iterator()

the iterator() method resolved our server running out of memory issue. see image showing first two attempts before adding iterator().

image
image1645×911 47.2 KB

hopefully this might help someone else in the case they have to move a lot of files!

Cheers

2 Likes
2

Thank you for sharing this with the entire community, @tolexy! This should be helpful for the community who wish to perform the same. :clap: :heart: :partying_face:

3

Where can I find information about this command? I searched github repo and couldn’t find this command

4

it is under Kobocat/onadata/apps/logger/management/commands. There is however no information on the command itself aside from reading the command and understanding what is actually happening line by line

2 Likes