Hi there, we are doing assessment surveys requiring a large number of media files (photographs) to be collected in each submission.
We already request enumerators to set camera settings as low as possible, and are aware of the 5MB limit per attachment/photograph. Currently there are up to 15 photos per submission however ideally (in order to suit the ideal data structure/logic) there would be more photos (up to ~80) per submission. This would allow us to capture all buildings within a complex in the same survey form, rather than requiring our enumerators to re-enter common data for each building).
However, we are finding that even with Head Office internet (let alone with the slow internet in the field) the KoboCollect uploads are often unsuccessful even with the current number of photos. We are regularly getting “Write error. IO Error during system call. Broken pipe.” and other errors when trying to upload data from the tablets. Each individual media item was around 1.5MB so well under the 5MB max.
I have since been told that perhaps the 5MB media file size limit applies to each submission rather than to each file
Can someone please advise what the Maxima are of the KoboCollect tool connecting to the Kobo Toolbox site when uploading submissions involving media files?
I realise we can manually upload if necessary, but that requires technical capability that some of our enumerators don’t yet have - so ideally we’d design the survey so it fits within any technical constraints in order to allow auto-upload.
There is no 5MB limit on images or files. I tested this to make sure.
Created a form with 1 question and tested with one image that is 8.53MB. Upload was successful.
Since that test was succesful, I proceeded to test with 15 images.
Created a new form with 15 questions, each question required an image as a response.
With KoboCollect v1.14.0a: reported a failed submission, but 6 out of 15 images uploaded succesfully. A notification dialogue box appeared and said, “Write error. IO Error during system call. Broken pipe - Error: Generic Exception: Read error: ssl=0x245f63d580: Failure in SSL library, usually a protocol error”. See screenshot.
With ODK Collect v1.18.2: all 15 images uploaded, but a notification dialogue box appeared and said, “Write error. IO Error during system call. Broken pipe” but also said, “Succesful Submission”. See screenshot.
Thanks @CaS for doing the testing! I hadn’t thought of uploading using ODK Collect - we only had Kobo Collect installed on the tablets we were using, will try both next time.
Am I correct in guessing that the form you were using was named “TESTING for: Write error. IO Error during system call. Broken pipe”? The name of the form generally appears before any message (success or otherwise), which would indicate that your ODK Collect submission was completely successful whereas your KoboCollect form had a SSL Library error - which was different wording to the to the “I/O… Broken pipe” error I was repeatedly getting from multiple devices. After a number of retries eventually everything got through in my case.
I will keep an eye out on what happens next time and perhaps log a ticket if the same error keeps occurring. Thanks again.
I’ve tried testing this too, albeit not as thoroughly as I’d like. It’d be really helpful if someone could get the Android logcat output while these upload problems occur. Check out ODK’s guide for installing adb and obtaining logs.
A preliminary guess is that this is related to connection speed / reliability, as I could reproduce weirdness more frequently while using my phone’s mobile connection than while using WiFi.
KoBoCAT sends an X-OpenRosa-Accept-Content-Length header with value 10000000 to clients, requesting that they split submissions larger than ~10 MB into separate HTTPPOST requests. The specification reads:
If a full POST of the form’s XML submission and its additional parts (for example, captured image, audio or video clips) would exceed the size specified in the X-OpenRosa-Accept-Content-Length header (the maxSize ), it is recommended that the client split the POST into multiple individual POST requests, each containing the form’s XML submission and one or more additional parts such that each partial POST request is no greater than maxSize ; if a single additional part is greater than maxSize , the POST should contain the form’s XML submission and that single additional part. Regardless of whether the client observes and honors the X-OpenRosa-Accept-Content-Length header, a compliant server with give its best effort to accept submissions of any length.
There is not a limit on the number of POSTs that a client can send for a single submission, so theoretically submissions—not attachments—of unlimited size can be uploaded. Of course, with a large enough submission, something will eventually break
Individual POSTs, however, cannot exceed 100 MB on our servers (kc.kobotoolbox.org and kc.humanitarianresponse.info), which effectively sets the ceiling for attachment size. The actual maximum allowable size of an attachment will be somewhat less than 100 MB due to overhead.
It’s worth noting that ODK Collect and Enketo differ as to how they interpret X-OpenRosa-Accept-Content-Length:
Enketo interprets the value of this header as an upper bound on the size of any individual attachment and refuses to accept anything larger.
ODK Collect currently ignores the header and “always splits files into 10Mb posts. If a file exceeds 10Mb, it will be sent in its own post with the submission XML.” I know this to be true from having uploaded a single image in excess of 30 MB and even larger videos.
