Migrating Submissions from ONA to Kobo Humanitarian Response

Hi there,

We’re looking to migrate from ONA to Kobo. I’m trialling this process and want to migrate a single project. The project has ~110 submissions, each with photo and video media attachments. I planned to do this migration with the following steps, as highlighted in this previous support post here:

  1. Download form XLS from ONA.

  2. Upload form XLS to Kobo

  3. Use ODK Briefcase to pull data from ONA

  4. Use ODK Briefcase to push data to Kobo

Step 1, 2, and 3 are successful but when I try step 4, I have no luck. The ODK Briefcase log gives the following:
Start pushing form and submissions
Form doesn’t exist in Aggregate
Sending form
Error sending form: Internal Server Error
Sending submission 1 of 106 (1/3)
Sending submission 2 of 106 (1/5)
Sending submission 3 of 106 (1/5)

The credentials seem correct and the forms are identical (When checking the XLS versions against one another).

What is going wrong here?

@sjed1, it could be a compatibility issue. Maybe try another approach outlined here …

Hi Kal,

Thanks for the link.

That solution is for single submissions and only and doesn’t deal with media attachements.

I’ve been tinkering with writing a python script to run essentially that same solution, but iterated to upload multiple submissions. However it’s hitting a snag with the media attachements.

I was hoping someone in the community had already solved this, but I’ll keep at it and post a solution if I crack it.

1 Like

OK - So I have managed to write a python script to upload the collection of submissions downloaded from ONA into Kobo.

However I cannot work out how to upload the associated media attachments. Is this even possible? I couldn’t find any documentation on this. Is it possible to upload media attachments to Kobo?

1 Like

I checked the same thing and tried few methods, but no luck. It looks your submission number is low, can’t you upload it manuallt incase?

1 Like

@sjed1, could you share the scripts with the community? Maybe the community would also benefit from your script, or if there is anyone who has already worked on this could also share with you the missing part of your script.

1 Like

The first step is to pull your form submissions using ODK briefcase. If you do this correctly, you will have a folder structure in a path that resembles something like “C:\…\…\Documents\ODK Briefcase\ODK Briefcase Storage\forms\FORM NAME\instances”

Under instances, you will have a separate folder for each submission containing an .xml file.

The first script will make a .json copy of all these xml files under the instances folder.
The python script is:

import os
import json
import xml.etree.ElementTree as ET
from pathlib import Path

def xml_to_dict(element):
    result = {}
    for child in element:
        child_tag = child.tag.split('}')[-1]  # Remove the namespace
        if child_tag not in result:
            result[child_tag] = []
        child_dict = xml_to_dict(child)
        if child.attrib:
            child_dict.update(child.attrib)
        result[child_tag].append(child_dict)

    if len(element) == 0:
        return element.text
    return result


def convert_xml_to_json(xml_file_path, json_file_path):
    tree = ET.parse(xml_file_path)
    root = tree.getroot()
    data_dict = xml_to_dict(root)
    with open(json_file_path, 'w') as f:
        json.dump(data_dict, f, indent=2)

def process_directory(directory):
    for root, _, files in os.walk(directory):
        for file in files:
            if file.endswith(".xml"):
                xml_file_path = os.path.join(root, file)
                json_file_path = os.path.splitext(xml_file_path)[0] + ".json"
                convert_xml_to_json(xml_file_path, json_file_path)
                print(f"Converted {xml_file_path} to {json_file_path}")

if __name__ == "__main__":
    # Set the directory path here // Should be something like: C:\\...\\...\\Documents\\ODK Briefcase\\ODK Briefcase Storage\\forms\\FORM NAME\\instances
    directory = "{DIRECTORY_PATH_HERE}}"

    if os.path.exists(directory):
        process_directory(directory)
    else:
        print("Directory not found. Please set a valid directory path.")

Once you run this script, you can run a second python script that will run through each folder and push the json to the new location (Note, the form structure needs to be identical between the pull and push locations).

import json
import requests
import os

# Replace these with your own Kobo Toolbox credentials and the form ID
form_id = '{FORM_ID}'
api_token = '{API_TOKEN}'
instances_folder = '{SAME_PATH_AS_FIRST_SCRIPT}'

submit_url = f'https://kc.humanitarianresponse.info/api/v1/submissions.json'
headers = {
    'Authorization': f'Token {api_token}',
    'Content-Type': 'application/json'
}

for folder in os.listdir(instances_folder):
    folder_path = os.path.join(instances_folder, folder)
    if os.path.isdir(folder_path):
        json_file_path = os.path.join(folder_path, 'submission.json')
        if os.path.isfile(json_file_path):
            # Load the JSON data
            with open(json_file_path, 'r') as json_file:
                json_data = json.load(json_file)

            # Prepare the payload
            payload = {
                'id': form_id,
                'submission': json_data
            }

            # Submit the JSON data
            try:
                submit_response = requests.post(submit_url, headers=headers, json=payload)
                submit_response.raise_for_status()
                print(f"Form submitted successfully for instance {folder}!")
            except requests.exceptions.HTTPError as e:
                print(f"Error submitting form for instance {folder}: {e}")
                print(f"Server Response: {submit_response.text}")

print("Finished processing all instances.")

If anyone could work out how to include the media files in the submission for the second script, I’d be eternally grateful.

@osmanburcu - this is a pilot migration. The eventual migration will cover multiple forms and thousands of submissions so while manual upload of the media files is possible, it’s not feasible for our case.