Kobocat move_media_to_s3 wrong url for files

tolexy · October 12, 2022, 2:01pm

Hello,

I am experiencing strange behavior from the uploaded media in Kobo after moving our files to S3.
Self-hosted, currently on:
KC 2.022.08b
KPI 2.022.08a
Enketo 2.8.1

Background: We were running out of disk space on our server and decided to move the media files added to form submissions to S3. We found the management command to move the files to an S3 bucket we set up. All told we had to move about ~100,000 files and the operation appeared to complete successfully (see other thread for note on minor modification to the command to avoid running out of memory during transfer of so many files).

Over the next several days, some users began to notice some of their media files from the submissions were not accessible. I checked the server filesystem and found that for at least one project, ~150 folders did not copy over for some reason (curiosity #1). I was able to compare and move those manually to S3.

Then we began to notice that some media files were not downloadable from the UI at the “View” submission screen under Data. Some files were available only if viewing the submission via the “Edit” view for that submission. Some media files were not accessible in either view despite verifying that they all existed in S3. Then we exported the XLS and including media URLs, and those URLs worked for some files but not for others (also confirmed existed in S3). The URLs in the XLS appear as “…com/media/original?media_file=…” and the subsequent download is named “original” whereas others appear as expected (…com/api/v2/assets/<form_id>/data/222000/attachments/222000/)

I suspect there is some inconsistency with the URLs that are stored in the KC/KPI databases?

#1: Is there a known reason why some folders may not have been transferred to S3 via this command? I will likely run the command again, but it takes some time given the shear number of files.

#2: Is there another management command that updates the stored URLs for the media uploaded with submissions? I was not able to see anything obvious in the codebase. Have you come across issues like this before or have any recommendations on how to get the media files URLs and download links back in sync?

Regards

manu_j · October 29, 2022, 7:15am

Were you able to solve this issue?

tolexy · November 3, 2022, 2:38am

I have not found a good solution. and you are the only response so far

ks_1 · November 10, 2022, 5:43pm

Hi,
I had faced a similar problem; you can see the Github issues I opened here:

github.com/kobotoolbox/kpi

Trying to make KPI work with S3 Compatible Cloud Storage

opened 09:45AM - 03 Apr 22 UTC

amks1

## Preface I know it's probably not a priority currently to get Kobotoolbox r…unning with [other S3-like providers](https://help.servmask.com/knowledgebase/list-of-s3-compatible-storage-providers/) like DigitalOcean, but I do feel that there could be many interested parties. This is because of ***much*** lower costs compared to AWS S3 including no charges per request. ## Description Over the past 2 days, I've been trying to make Kobotoolbox work with DigitalOcean Spaces. Here's what I've done so far: ### Trial 1: Trying with Kpi and Kobocat installation `v2.021.45` I added the following lines to settings of both Kpi and Kobocat: ``` AWS_S3_SIGNATURE_VERSION = 's3v4' AWS_S3_REGION_NAME = 'ams3' AWS_S3_ENDPOINT_URL = f"https://{AWS_S3_REGION_NAME}.digitaloceanspaces.com" ``` Image submissions, image fetching and asset xls exports were working well, but data exports returned a weird error: ![image](https://user-images.githubusercontent.com/68153785/161420005-996b32f6-dd4d-43c5-b64b-c7e5dfab2c9c.png) However, legacy exports were working well so I figured it's an issue with Kpi. After a long time (long time because of my inexperience with Django) I figured that Kpi data exports use `PrivateStorageDetailView`, and adding the following line to settings solved the issue: ``` AWS_PRIVATE_S3_ENDPOINT_URL = AWS_S3_ENDPOINT_URL ``` Now `v2.021.45` was working well with DigitalOcean Spaces. ### Trial 2: Trying with Kpi and Kobocat installation `v2.022.08` On upgrading the earlier installation to the new version, I find that data exports and media URLs therein work, but Kpi frontend view does not. The URLs used to access media from KPI uses the `v2/assets/.../data/../attachment/...` endpoint in this version compared to the `kc.url...` in the previous version. It now returns a 404 for all media files (they get uploaded no issues). The endpoint uses Django's `models.FileField` to retrieve the media file, but I've not been able to get this working with my S3 custom URL. And it looks like I've exhausted my Django capabilities. Would anyone be able to guide me forward?

github.com/kobotoolbox/kobocat

S3 Authentication issue: AWS demands updated SDK

opened 04:19PM - 01 Apr 22 UTC

amks1

I've installed Kobocat version `2.021.45`. When trying to use Amazon S3 storage,… the collected media gets stored in the bucket and I am able to browse it there. However when trying to view the submission in the Kobotoolbox KPI interface, the image doesn't load and the following error is returned on clicking the broken image: ``` The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256. ``` Doing a quick search, I came across this post: https://stackoverflow.com/questions/26533245/the-authorization-mechanism-you-have-provided-is-not-supported-please-use-aws4, which essentially says that the SDK needs to be updated for S3 instances created in regions that were added after January 2014.