Error 500 on Data Import with Python

I am using the script attached to perform this task but unfortunately, I get the Error 500 every time I do so.

I have tried copying and pasting the JSON data as well onto the kc.kobotoolbox.org/api/v1/submissions page and I still get the same error 500.

The Python script producing this error is below.

import requests
import json

def post_data_to_kobo(form_id, submission_data, username, password):
    url = f"https://kc.kobotoolbox.org/api/v1/submissions"
    auth = (username, password)

    data = {
        "id": form_id,
        "submission": submission_data
    }

    headers = {
        "Content-Type": "application/json"
    }

    try:
        response = requests.post(url, json=data, auth=auth, headers=headers)

        if response.status_code == 201:
            print("Data submission successful!")
        else:
            print(f"Data submission failed with status code: {response.status_code}")
            print(response.text)

    except requests.exceptions.RequestException as e:
        print("An error occurred during the data submission:")
        print(e)

if __name__ == "__main__":
    # Replace the following variables with your actual data
    form_id = "<my_form_id>"
    submission_data_file = "census.json"
    username = "<my_username>"
    password = "<my_password>"

    with open(submission_data_file) as file:
        submission_data = json.load(file)

    post_data_to_kobo(form_id, submission_data, username, password)

Unfortunately, it seems I am not allowed to paste attachments as a new user but the JSON looks like this.

[
  {
    "172938293": 172938293,
    "start": "end"
  },
  {
    "172938293": "3/8/2023",
    "start": "Date"
  },
  {
    "172938293": "",
    "start": "location_note"
  },
  {
    "172938293": "ZW",
    "start": "country"
  },
  {
    "172938293": "ZW13",
    "start": "adm_one"
  },
  {
    "172938293": "ZW1303",
    "start": "adm_two"
  },
  {
    "172938293": "ZW130309",
    "start": "adm_three"
  },
  {
    "172938293": "",
    "start": "adm_four"
  },
  {
    "172938293": "Example Village",
    "start": "Village_Name"
  },
  {
    "172938293": "-19.947651666666665 30.6872 1183.2 21.125",
    "start": "Location_Coordinates"
  },
  {
    "172938293": -19.94765167,
    "start": "_Location_Coordinates_latitude"
  },
  {
    "172938293": 30.6872,
    "start": "_Location_Coordinates_longitude"
  },
  {
    "172938293": 1183.2,
    "start": "_Location_Coordinates_altitude"
  },
  {
    "172938293": 21.125,
    "start": "_Location_Coordinates_precision"
  },
  {
    "172938293": "rtsl",
    "start": "Pillar"
  },
  {
    "172938293": "rra",
    "start": "Project"
  },
  {
    "172938293": "farmers",
    "start": "Types_of_Participants"
  },
  {
    "172938293": "",
    "start": "Other_Participant_Type"
  },
  {
    "172938293": "",
    "start": "Group_Name"
  },
  {
    "172938293": 13105,
    "start": "random_number"
  },
  {
    "172938293": "ZW1303_223700626",
    "start": "Participant_ID"
  },
  {
    "172938293": "Person Name",
    "start": "Full_name"
  },
  {
    "172938293": "Male",
    "start": "Gender"
  },
  {
    "172938293": "6/2/1982",
    "start": "Date_of_Birth"
  },
  {
    "172938293": "",
    "start": "Age"
  },
  {
    "172938293": 41,
    "start": "participant_age"
  },
  {
    "172938293": "ordinary",
    "start": "Position"
  },
  {
    "172938293": "",
    "start": "Organisation"
  },
  {
    "172938293": 7392,
    "start": "National_ID_Number"
  },
  {
    "172938293": 7712345687,
    "start": "Mobile_Phone_Number"
  },
  {
    "172938293": "ZW",
    "start": "Country"
  },
  {
    "172938293": "",
    "start": "Specify"
  },
  {
    "172938293": 223700626,
    "start": "_id"
  },
  {
    "172938293": "32780e14-9c5f-49e8-b568-ec37e0153849",
    "start": "_uuid"
  },
  {
    "172938293": "########",
    "start": "_submission_time"
  },
  {
    "172938293": "",
    "start": "_validation_status"
  },
  {
    "172938293": "",
    "start": "_notes"
  },
  {
    "172938293": "submitted_via_web",
    "start": "_status"
  },
  {
    "172938293": "",
    "start": "_submitted_by"
  },
  {
    "172938293": "vApGFsoZmEWyyUVW7igvCj",
    "start": "__version__"
  },
  {
    "172938293": "",
    "start": "_tags"
  },
  {
    "172938293": 7778,
    "start": "_index"
  }
]

Any assistance would be appreciated.

I also tried this solution available here - Data import though Python but server did not accept the data when edit - User Support / API - KoboToolbox Community Forum but I get an error that says:

InvalidJSONError: Out of range float values are not JSON compliant

Hi Surveyorjr,

can you edit the url as “https://kc.kobotoolbox.org/api/v1/submissions.json”? because the link you are using is for the reference sheet, this might be causing the issue?

1 Like

Thanks @osmanburcu. I have added the additional part to the URL. Unfortunately, I am now getting a new error which is the same error with the solution that I had found on the previous post.

The error:

InvalidJSONError: Out of range float values are not JSON compliant

I checked your code, the first part working fine, but the JSON style is not correct i believe because i replaced your JSON file with my test JSON format, and i was able to import data.

I remember, might be wrong as well, in documentation there should be spesific JSON structure requirment that’s why you may not be able to upload your data

{
“id”:“FORM ID”,
“submission” :{
“formhub”: {
“uuid”: “YOUR FORM UUID”
},
“q1”: “1”,
“q2”: “2”,
“q3”: “33”,
“meta”: {
“instanceID”: “SUBMISSION UUID”
}
}
}

2 Likes

Thanks very much @osmanburcu , I restructured the JSON file and it worked very well.

1 Like

@surveyorjr, would appreciate if you could share the final snippet with the community. It would be very helpful for those having similar issue.

So the main issue was to do with the JSON format that my data was in which is not compatible with this process.

After the modifications as per the guidance and response from @osmanburcu I reformated my JSON file and it ended up looking like so;

{
  "id": "<my_form_id>",
  "submission": {
    "formhub": {
      "uuid": "form_uuid"
    },
    "start": "172938293",
    "end": "172938293",
    "Date": "3/8/2023",
    "location_note": "",
    "country": "ZW",
    "adm_one": "ZW13",
    "adm_two": "ZW1303",
    "adm_three": "ZW130309",
    "adm_four": "",
    "Village_Name": "Example Village",
    "Location_Coordinates": "-19.947651666666665 30.6872 1183.2 21.125",
    "_Location_Coordinates_latitude": "-19.94765167",
    "_Location_Coordinates_longitude": "30.6872",
    "_Location_Coordinates_altitude": "1183.2",
    "_Location_Coordinates_precision": "21.125",
    "Pillar": "rtsl",
    "Project": "rra",
    "Types_of_Participants": "farmers",
    "Other_Participant_Type": "",
    "Group_Name": "",
    "random_number": "13105",
    "Participant_ID": "ZW1303_223700626",
    "Full_name": "Person Name",
    "Gender": "Male",
    "Date_of_Birth": "6/2/1982",
    "Age": "",
    "participant_age": "41",
    "Position": "ordinary",
    "Organisation": "",
    "National_ID_Number": "7392",
    "Mobile_Phone_Number": "7712345687",
    "Country": "ZW",
    "Specify": "",
    "_id": "223700626",
    "_uuid": "32780e14-9c5f-49e8-b568-ec37e0153849",
    "_submission_time": "########",
    "_validation_status": "",
    "_notes": "",
    "_status": "submitted_via_web",
    "_submitted_by": "",
    "__version__": "vApGFsoZmEWyyUVW7igvCj",
    "_tags": "",
    "_index": "7778"
  }
}
1 Like

Thank you for sharing this with the entire community, @surveyorjr! :clap: :heart: :partying_face:

I would like to make a small addition to @surveyorjr’s code, If you would like to import a set of questions in group, you can find an example below.

{
  "id": "<my_form_id>",
  "submission": {
    "formhub": {
      "uuid": "form_uuid"
    },
    "start": "172938293",
    "end": "172938293",
    "Date": "3/8/2023",
    "location_note": "",
    "country": "ZW",
    "adm_one": "ZW13",
    "adm_two": "ZW1303",
    "adm_three": "ZW130309",
    "adm_four": "",
    "Village_Name": "Example Village",
    "Location_Coordinates": "-19.947651666666665 30.6872 1183.2 21.125",
    "_Location_Coordinates_latitude": "-19.94765167",
    "_Location_Coordinates_longitude": "30.6872",
    "_Location_Coordinates_altitude": "1183.2",
    "_Location_Coordinates_precision": "21.125",
    "Pillar": "rtsl",
    "Project": "rra",
    "Types_of_Participants": "farmers",
    "Other_Participant_Type": "",
    "Group_Name": "",
    "random_number": "13105",
    "Total_number_of_partificants" :"2",
    "Group_name": {
                     "Participant_ID": "ZW1303_223700626",
                    "Full_name": "Person Name",
                    "Gender": "Male",
                    "Date_of_Birth": "6/2/1982",
                    "Age": "",
                    "participant_age": "41",
                    "Position": "ordinary",
                    "Organisation": "",
                    "National_ID_Number": "7392",
                    "Mobile_Phone_Number": "7712345687",
                    "Country": "ZW",
                    "Specify": ""},
    "_id": "223700626",
    "_uuid": "32780e14-9c5f-49e8-b568-ec37e0153849",
    "_submission_time": "########",
    "_validation_status": "",
    "_notes": "",
    "_status": "submitted_via_web",
    "_submitted_by": "",
    "__version__": "vApGFsoZmEWyyUVW7igvCj",
    "_tags": "",
    "_index": "7778"
  }
}
2 Likes