Move media to S3 command modification

As noted in another thread, I had to move ~100,000 files from our server to S3. When running the kobocat move_media_to_s3 command, our server ran out of memory (with 16GB). We looked into the code a bit and made a small modification.

Line 43: for i in cls.objects.all():

for i in cls.objects.all().iterator()

the iterator() method resolved our server running out of memory issue. see image showing first two attempts before adding iterator().

hopefully this might help someone else in the case they have to move a lot of files!



Thank you for sharing this with the entire community, @tolexy! This should be helpful for the community who wish to perform the same. :clap: :heart: :partying_face:

Where can I find information about this command? I searched github repo and couldn’t find this command

it is under Kobocat/onadata/apps/logger/management/commands. There is however no information on the command itself aside from reading the command and understanding what is actually happening line by line