XML Submission via Bulk Upload Works, But Audio File Is Not Attached (500 Error)

nicopace · July 22, 2025, 4:50pm

I am having an issue similar to this one:

I created a brand new form, with 1 question and background audio.
I publish the form
I fill the form without internet connection, to get a zip file with the xml of the submission.
I go to the bulk-submission-form referenced here: Manually Uploading Submissions — KoboToolbox documentation
I modify the xml by adding a reference to an AMR audio file I had, and add the file to the zip file.
I add this zip file to the submission.
After I send and wait for the submission to go through, I get: “Server error (500) Something went wrong when trying to process your request.”

This is the key information for debugging purposes:

form_id: a2EbErQYjXcPQdp5HHVrBK

BTW the xml form is submitted, the file is not attached.

And I tried attachments of multiple sizes, from 9.7MB (close to the 10MB limit), to 3MB, it doesn’t seem to be the issue.

nicopace · July 22, 2025, 5:04pm

Here is the mandatory screenshot that shows the 500 error happening, in case is needed.

nicopace · July 22, 2025, 5:25pm

BTW I also tried to use ODK Briefcase as mentioned in this thread, but had no luck.

I configured it correctly, as I was able to PULL form and submissions from KoboToolbox.
But was not able to PUSH, no forms appeared when I populated the PUSH screen with the same information I used in the PULL screen.

nicopace · July 22, 2025, 6:23pm

Just to complete the picture, the need for using the bulk submission came out of a debugging session with some KoboCollect devices that had unsent submissions. They were receiving a 500 error submitting their forms using KoboCollect, so I requested them to send me their files, their Android KoboCollect directory to me, so I could dig what was happening.

The submissions that were having issues were those that had heavy files (>10MB), so I attempted to send the submissions shrinking those files to a file size that was reasonable for Kobo toolbox API.

But no matter the size of the file, the failed. So I decided to create a new form and attempt to do submissions on this new form with just one field and the background audio. Still no luck, so that’s why I’m trying to debug this situation.

Xiphware · July 23, 2025, 1:18am

I’ve been playing around with this and it seems there may be an issue with the bulk-upload specifically around background-audio files (!?). Even without futzing around replacing audio files and editing XML, I have been unable to use the bulk-upload interface to re-submit a simple form with a short background-audio, after pulling it off the mobile device using adb. Whereas the identical submission comes up fine when I submit it directly from KoboCollect.

I’ll keep investigating but, suffice to say, “it’s not just you”

If you just want to get the audio file uploaded into Kobo, somehow, somewhere, you might also try a simple form with a single ~~audio or~~ file upload widget, and attach your audio new file to that instead.

Xiphware · July 23, 2025, 3:27am

bulk-upload doesn’t appear to be behaving for even plain-vanilla audio files either Still investigating…

nicopace · July 23, 2025, 6:35am

Thanks for researching into it @Xiphware .
On a sidenote, if there is a programmatic way of doing this, I am happy to explore it.
I posted a complementary issue asking about it: Guidance on Using Python to Interact with KoboToolbox – Compatibility with PyODK or Recommended Alternatives

Xiphware · July 23, 2025, 8:01am

I presume from your description that you’ve already figured out how to pull off the too-large background audio files - off the devices with the failed submissions - and been able to shrink the files down to something more manageable?

If all you basically need are a way to get these new audio files up into your Kobo account, have you considered just creating a simple form/project with a single audio question, and firing up Enketo web interface to submit them all?

Or do you still need them to be re-associated intimately with the original form submission data? If that’s the case, you might also try replacing the too-large audio file in the submission folder on the physical device with your shrunk one - keeping the identical file name - and retrying the still pending submit from KoboCollect. Messy, but it might be an option.

nicopace · July 23, 2025, 8:21am

Yes

It is not that.
The form has 72 questions, and the background audio is for auditting purposes.
I guess I could edit the form, add an audio field that is optional and at the end of the form, and upload it manually…
This would imply a usability issue, as audios could be find in one of two places, and our users will have to jump over that hoop to get to the audios.
I have not tried this myself yet.

In any case, it would be great to have this working properly still…

I did this in the past and it works, but I don’t have access to the devices so.I can’t do that maual process.

Thanks for the suggestions though!

Xiphware · July 23, 2025, 8:26am

Understood. I wanted to make sure you at least have the ability to preserve the critical data, somewhere. Obviously, the goal is to capture the original submission in its entire former glory… The bulk upload should have worked but, as we can see, its misbehaving. Still investigating.

nicopace · July 23, 2025, 11:56am

I dag into the kpi sourcecode a little bit…
I am not sure that this is related, but in the import functionality there are mentions about the images but not about audios or any other attachments like videos:

github.com/kobotoolbox/kpi

kobo/apps/openrosa/apps/logger/import_tools.py

d6464023b


      
                  zf.extractall(temp_directory)
              except zipfile.BadZipfile as e:
                  errors = ["%s" % e]
                  return 0, 0, errors
              else:
                  return import_instances_from_path(temp_directory, user, status)
              finally:
                  shutil.rmtree(temp_directory)
          
          
          def import_instances_from_path(path, user, status="zip"):
              def callback(xform_fs):
                  """
                  This callback is passed an instance of a XFormInstanceFS.
                  See xform_fs.py for more info.
                  """
                  with django_file(xform_fs.path, field_name="xml_file",
                                   content_type="text/xml") as xml_file:
                      images = [django_file(jpg, field_name="image",
                                content_type="image/jpeg") for jpg in xform_fs.photos]
                      # TODO: if an instance has been submitted make sure all the

Could this be related to this?
This function is called for every submission that is sent in the zip file over the bulk submissions url. The trace is this one:

In the call to create_instance, only images are being sent, where a lot of other attachments types could be sent.

My suggestion would be to change this line:

github.com/kobotoolbox/kpi

kobo/apps/openrosa/apps/logger/import_tools.py

d6464023b


      
          
          
          def import_instances_from_path(path, user, status="zip"):
              def callback(xform_fs):
                  """
                  This callback is passed an instance of a XFormInstanceFS.
                  See xform_fs.py for more info.
                  """
                  with django_file(xform_fs.path, field_name="xml_file",
                                   content_type="text/xml") as xml_file:
                      images = [django_file(jpg, field_name="image",
                                content_type="image/jpeg") for jpg in xform_fs.photos]
                      # TODO: if an instance has been submitted make sure all the
                      # files are in the database.
                      # there shouldn't be any instances with a submitted status in the
                      # import.
                      instance = create_instance(user.username, xml_file, images, status)
          
                      for i in images:
                          i.close()
                          i.close()

I hope this is useful @Xiphware .

nicopace · July 23, 2025, 1:47pm

Did a bug report documenting this issue here:

github.com/kobotoolbox/kpi

bulk-submission-form doesn't accept non-image attachments

opened 01:46PM - 23 Jul 25 UTC

nicopace

When using the bulk submission form, it's possible to upload ZIP files containin…g XML submissions and their corresponding attachments. However, the current implementation only accepts JPEG files as attachments, excluding all others. The relevant code is: [https://github.com/kobotoolbox/kpi/blob/d6464023bdaf1365e07d5e273877351c6ba7f2eb/kobo/apps/openrosa/apps/logger/import\_tools.py#L86-L89](https://github.com/kobotoolbox/kpi/blob/d6464023bdaf1365e07d5e273877351c6ba7f2eb/kobo/apps/openrosa/apps/logger/import_tools.py#L86-L89) This behavior is incorrect. The expected logic should be: * Exclude the XML file itself, * Include **any** attachment referenced in the XML submission, regardless of its file type or extension. This limitation prevents the use of valid non-JPEG attachments (e.g., PDFs, audio files), which are supported in other parts of the platform. Thanks!

jnm · July 23, 2025, 9:43pm

Thanks for the report; sorry about the trouble. I responded to your other topic, but I’ll cross-post here for convenience the part that links to an example of Python code that uploads submissions and attachments:

nicopace · July 25, 2025, 2:05am

We’ve written a guide for our users that you can find in here:

This is the current strategy that we will use. Thanks for this suggestion.

@Xiphware @jnm Is there any addiotional information I can share to help troubleshoot the issue or support in testing out code?

Thanks,

Xiphware · July 25, 2025, 3:06am

This is great, @nicopace, you are to be commended!

Couple minor suggestions:

Project_hash: This is your user folder inside KoboToolbox. If you have multiple projects set up, you will have several folders. There’s no way to identify them from the folder names alone, so you’ll need to browse inside each folder to determine which one contains your project.

Correct. Each project folder hash name is rather indecipherable… But, if you look inside each project folder - that is, if the user is connecting to multiple different projects in their KoboCollect - then there should be a single zero-length file that has the name of your project (!). So you can use this to reliably differentiate and locate submissions for different projects; eg

From Using Android Debug Bridge with Collect - ODK Docs

= 2021.2: /sdcard/Android/data/org.odk.collect.android/files/projects. Only accessible by Collect. The Project directories will contain a blank file with the same name as the Project itself.

Also, and as you might guess from above, you can also do all this from macOS too - you just need to install adb to do it [I note your instructions are specifically for Microsoft Windows: “On your computer, open the file manager, access the phone’s storage,…”]. There are some instructions for the Mac-only approach here:

If you have any issues using adb to pull stuff off an Android device onto a Mac please post back

Xiphware · July 25, 2025, 3:13am

FYI adb also support the find command, so you could also use this to immediately locate your desired project folder: