Thanks @osmanburcu, I was posting but i have finally figured it out.
I was missing the right URL’s.
so, it looks like there are three URLs:
https://kc.kobotoolbox.org/ - used for API V1.
https://kf.kobotoolbox.org/ - used for API V2 & the form editor
https://ee.kobotoolbox.org/ - Form Submission
I have from there created a simple python script to upload the entries synchronously. It reads a FLAT CSV file, with the headers matching the data fields. where there are groups, they should be entered as group/field, and the script will separate them
data.csv header example
remove/uuidGeneration, meta/instanceID, field1, remove/calculationData ,group1/field1, group1/field2, group2,field1
- any headers that have “remove” in them will not be uploaded. These can be used to calculate other fields in the spreadsheet, before saving as a CSV.
- meta/instanceID is required, and must have the shape of
uuid:<UUID>
- Every new item to upload MUST have a UUID, this prevents duplicate data from being uploaded.
- UUIDs can be calculated in libre office calc with the following formula:
=LOWER(CONCATENATE(DEC2HEX(RANDBETWEEN(0,4294967295),8),"-",DEC2HEX(RANDBETWEEN(0,65535),4),"-",DEC2HEX(RANDBETWEEN(0,65535),4),"-",DEC2HEX(RANDBETWEEN(0,65535),4),"-",DEC2HEX(RANDBETWEEN(0,4294967295),8),DEC2HEX(RANDBETWEEN(0,65535),4)))
you need to update in the script below:
- the exact
title of the form you want to upload to, which can be found using this API
- username and password
There are a number of things that it does not do, and if i fix up the script or add functionality, i will update it here.
- data validation
- keeping track of where it was last at
- validate the form data is in the correct shape
- graceful error handling
- only handles one level of grouping/nesting. if you have more than one
/ in your headers, it will not work!
koboFormSubmission.py - it will take the form name, get the required metadata used for upload, then collect the submission data.csv, parse it into the correct JSON shape and upload each item one by one, with a 1 second pause between POSTs
# Reference Documents:
# https://kc.kobotoolbox.org/api/v1/?format=json
# Youtube Video: https://www.youtube.com/watch?v=9q3kVr4m7LY
# Forum Post: https://community.kobotoolbox.org/t/error-500-on-data-import-with-python/43200
## Useful URLS:
# https://kc.kobotoolbox.org/ - used for API V1.
# https://kf.kobotoolbox.org/ - used for API V2 & the form editor
# https://ee.kobotoolbox.org/ - Form Submission
## Generate UUID in LibreOffice with the following Calc:
## =LOWER(CONCATENATE(DEC2HEX(RANDBETWEEN(0,4294967295),8),"-",DEC2HEX(RANDBETWEEN(0,65535),4),"-",DEC2HEX(RANDBETWEEN(0,65535),4),"-",DEC2HEX(RANDBETWEEN(0,65535),4),"-",DEC2HEX(RANDBETWEEN(0,4294967295),8),DEC2HEX(RANDBETWEEN(0,65535),4)))
import requests
import json
import uuid
import csv
from pprint import pprint
import time
def create_uuid():
return str(uuid.uuid4())
def post_data_to_kobo(formId, formUUID, submissionData, submissionURL, username, password):
print("Uploading item number " + str(index))
url = submissionURL
auth = (username, password)
submissionData["id"] = formId
submissionData["submission"]["formhub"] = {"uuid": formUUID}
headers = {
"Content-Type": "application/json"
}
try:
response = requests.post(url, json=submissionData, auth=auth, headers=headers)
if response.status_code == 201:
print("Data submission successful!")
else:
print(f"Data submission failed with status code: {response.status_code}")
print(response.text)
except requests.exceptions.RequestException as e:
print("An error occurred during the data submission:")
print(e)
def csvToDict(csvFile):
data = []
# Open a csv reader called DictReader
with open(csvFile, encoding='utf-8') as csvf:
csvReader = csv.DictReader(csvf)
data = list(csvReader)
return data
def formatKeys(data):
# pprint(data)
newDict = {"submission": {}}
for key in data:
if "remove" not in key:
if "/" in key:
newKeys = key.split("/")
# print(newKeys)
if newKeys[0] == "meta":
newDict["submission"][newKeys[0]] = {}
newDict["submission"][newKeys[0]][newKeys[1]] = data[key]
else:
if newKeys[0] not in newDict["submission"]:
newDict["submission"][newKeys[0]] = {}
newDict["submission"][newKeys[0]][newKeys[1]] = data[key]
else:
newDict["submission"][key] = data[key]
return newDict
def getFormMetaData(title, username, password):
url = "https://kc.kobotoolbox.org/api/v1/forms?format=json"
auth = (username, password)
response = requests.get(url, auth=auth)
if response.status_code == 200:
print("form MetaData request successful!")
data = response.json()
else:
print(f"form MetaData request failed with status code: {response.status_code}")
print(response.text)
for form in data:
if form["title"] == title:
metaData = {"title": title,
"formId": form["id_string"],
"formUUID": form["uuid"]}
pprint(metaData)
return(metaData)
if __name__ == "__main__":
submissionDataFile = "data.csv"
submissionURL = "https://kc.kobotoolbox.org/api/v1/submissions"
username = "{{Username}}"
password = "{{Password}}"
formTitle = "{{Title of the Form}}"
print("Retrieving details about form " + formTitle)
formMetaData = getFormMetaData(formTitle, username, password)
submissionData = csvToDict(submissionDataFile)
cleansedData = []
for item in submissionData:
cleansedData.append(formatKeys(item))
input()
index = 0
for data in cleansedData:
post_data_to_kobo(formMetaData["formId"], formMetaData["formUUID"], data, submissionURL, username, password)
time.sleep(1)
index+=1
Expecting similar participation in the future too.
