XML Submission via Bulk Upload Works, But Audio File Is Not Attached (500 Error)

I am having an issue similar to this one:

  1. I created a brand new form, with 1 question and background audio.
  2. I publish the form
  3. I fill the form without internet connection, to get a zip file with the xml of the submission.
  4. I go to the bulk-submission-form referenced here: Manually Uploading Submissions — KoboToolbox documentation
  5. I modify the xml by adding a reference to an AMR audio file I had, and add the file to the zip file.
  6. I add this zip file to the submission.
  7. After I send and wait for the submission to go through, I get: “Server error (500) Something went wrong when trying to process your request.”

This is the key information for debugging purposes:

  • form_id: a2EbErQYjXcPQdp5HHVrBK

BTW the xml form is submitted, the file is not attached.

And I tried attachments of multiple sizes, from 9.7MB (close to the 10MB limit), to 3MB, it doesn’t seem to be the issue.

1 Like

Here is the mandatory screenshot that shows the 500 error happening, in case is needed.

BTW I also tried to use ODK Briefcase as mentioned in this thread, but had no luck.

I configured it correctly, as I was able to PULL form and submissions from KoboToolbox.
But was not able to PUSH, no forms appeared when I populated the PUSH screen with the same information I used in the PULL screen.

1 Like

Just to complete the picture, the need for using the bulk submission came out of a debugging session with some KoboCollect devices that had unsent submissions. They were receiving a 500 error submitting their forms using KoboCollect, so I requested them to send me their files, their Android KoboCollect directory to me, so I could dig what was happening.

The submissions that were having issues were those that had heavy files (>10MB), so I attempted to send the submissions shrinking those files to a file size that was reasonable for Kobo toolbox API.

But no matter the size of the file, the failed. So I decided to create a new form and attempt to do submissions on this new form with just one field and the background audio. Still no luck, so that’s why I’m trying to debug this situation.

I’ve been playing around with this and it seems there may be an issue with the bulk-upload specifically around background-audio files (!?). Even without futzing around replacing audio files and editing XML, I have been unable to use the bulk-upload interface to re-submit a simple form with a short background-audio, after pulling it off the mobile device using adb. Whereas the identical submission comes up fine when I submit it directly from KoboCollect.

I’ll keep investigating but, suffice to say, “it’s not just you” :slightly_smiling_face:

If you just want to get the audio file uploaded into Kobo, somehow, somewhere, you might also try a simple form with a single audio or file upload widget, and attach your audio new file to that instead.

bulk-upload doesn’t appear to be behaving for even plain-vanilla audio files either :slightly_frowning_face: Still investigating…

Thanks for researching into it @Xiphware .
On a sidenote, if there is a programmatic way of doing this, I am happy to explore it.
I posted a complementary issue asking about it: Guidance on Using Python to Interact with KoboToolbox – Compatibility with PyODK or Recommended Alternatives

I presume from your description that you’ve already figured out how to pull off the too-large background audio files - off the devices with the failed submissions - and been able to shrink the files down to something more manageable?

If all you basically need are a way to get these new audio files up into your Kobo account, have you considered just creating a simple form/project with a single audio question, and firing up Enketo web interface to submit them all?

Or do you still need them to be re-associated intimately with the original form submission data? If that’s the case, you might also try replacing the too-large audio file in the submission folder on the physical device with your shrunk one - keeping the identical file name - and retrying the still pending submit from KoboCollect. Messy, but it might be an option.

Yes

It is not that.
The form has 72 questions, and the background audio is for auditting purposes.
I guess I could edit the form, add an audio field that is optional and at the end of the form, and upload it manually…
This would imply a usability issue, as audios could be find in one of two places, and our users will have to jump over that hoop to get to the audios.
I have not tried this myself yet.

In any case, it would be great to have this working properly still…

I did this in the past and it works, but I don’t have access to the devices so.I can’t do that maual process.

Thanks for the suggestions though!

Understood. I wanted to make sure you at least have the ability to preserve the critical data, somewhere. Obviously, the goal is to capture the original submission in its entire former glory… The bulk upload should have worked but, as we can see, its misbehaving. Still investigating.

1 Like

I dag into the kpi sourcecode a little bit…
I am not sure that this is related, but in the import functionality there are mentions about the images but not about audios or any other attachments like videos:

Could this be related to this?
This function is called for every submission that is sent in the zip file over the bulk submissions url. The trace is this one:

  1. kpi/kobo/apps/openrosa/apps/main/urls.py at d6464023bdaf1365e07d5e273877351c6ba7f2eb · kobotoolbox/kpi · GitHub
  2. kpi/kobo/apps/openrosa/apps/main/urls.py at d6464023bdaf1365e07d5e273877351c6ba7f2eb · kobotoolbox/kpi · GitHub
  3. kpi/kobo/apps/openrosa/apps/logger/import_tools.py at main · kobotoolbox/kpi · GitHub
  4. kpi/kobo/apps/openrosa/apps/logger/import_tools.py at main · kobotoolbox/kpi · GitHub
  5. kpi/kobo/apps/openrosa/apps/logger/import_tools.py at main · kobotoolbox/kpi · GitHub
  6. kpi/kobo/apps/openrosa/apps/logger/import_tools.py at d6464023bdaf1365e07d5e273877351c6ba7f2eb · kobotoolbox/kpi · GitHub
  7. kpi/kobo/apps/openrosa/libs/utils/logger_tools.py at d6464023bdaf1365e07d5e273877351c6ba7f2eb · kobotoolbox/kpi · GitHub

In the call to create_instance, only images are being sent, where a lot of other attachments types could be sent.

My suggestion would be to change this line:

I hope this is useful @Xiphware .

1 Like

Did a bug report documenting this issue here:

1 Like

Thanks for the report; sorry about the trouble. I responded to your other topic, but I’ll cross-post here for convenience the part that links to an example of Python code that uploads submissions and attachments:

1 Like

We’ve written a guide for our users that you can find in here:

This is the current strategy that we will use. Thanks for this suggestion.

@Xiphware @jnm Is there any addiotional information I can share to help troubleshoot the issue or support in testing out code?

Thanks,

This is great, @nicopace, you are to be commended! :smiling_face_with_three_hearts:

Couple minor suggestions:

Project_hash: This is your user folder inside KoboToolbox. If you have multiple projects set up, you will have several folders. There’s no way to identify them from the folder names alone, so you’ll need to browse inside each folder to determine which one contains your project.

Correct. Each project folder hash name is rather indecipherable… But, if you look inside each project folder - that is, if the user is connecting to multiple different projects in their KoboCollect - then there should be a single zero-length file that has the name of your project (!). So you can use this to reliably differentiate and locate submissions for different projects; eg

From Using Android Debug Bridge with Collect - ODK Docs

= 2021.2: /sdcard/Android/data/org.odk.collect.android/files/projects. Only accessible by Collect. The Project directories will contain a blank file with the same name as the Project itself.

Also, and as you might guess from above, you can also do all this from macOS too - you just need to install adb to do it [I note your instructions are specifically for Microsoft Windows: “On your computer, open the file manager, access the phone’s storage,…”]. There are some instructions for the Mac-only approach here:

If you have any issues using adb to pull stuff off an Android device onto a Mac please post back :slightly_smiling_face:

1 Like

FYI adb also support the find command, so you could also use this to immediately locate your desired project folder:

Screenshot 2025-07-25 at 3.11.16 PM

1 Like

Is it worth it?

What do you mean?