How to use API V2 to submit new data

Hi, i am very new to Kobo, but fairly familiar with using API’s to get and create data. There seems to be a lot of good info on the forums about how to create new submissions with V1 of the API, but no equivalent for V2.

I have explored the API generated docs: KoboToolbox Form Building API

From what i can work out, the API is now structured in a RESTful manner, that allows you to nicely drill down into Assets (forms) and their data.

However, there does not seem to be a data POST method that would allow me to submit a new piece of data. I have a large backlog of data that i would like to programmatically upload.

How do i POST new data to my table?

Thanks :smiley:

Leigh

Welcome to the community, @leigh_vandermerwe! Could you kindly let the community know what you found and what you expect that is still not there in the community discussion? Maybe the community could then help you out.

Hi Kal_Lam,

Maybe it is just a misunderstanding on how the system and API fits together.

I am using https://ee.kobotoolbox.org as the form URL, and https://kf.kobotoolbox.org/ for the Form Editor.

maybe for submissions i should only be using API V1 on [ee.kobo](http://ee.kobotoolbox.org?

and, V2 is only for managing forms and retrieving submitted data?

I am following the Youtube Video, which is referenced often in the forum as a way to upload bulk data.: https://www.youtube.com/watch?v=9q3kVr4m7LY

and at the first step i have an issue, i get the same result using chrome, firefox, postman and curl

Video:
image

Mine:
GET https://ee.kobotoolbox.org/api/v1/submissions.json

{
    "code": 400,
    "message": "Bad Request. Server URL missing."
}

And once i follow it all the way through, it fails to work with the same error message.

@leigh_vandermerwe Hi, Happy to see that my video is contributing. I am happy to help to solve your problem. It looks like you are sending GET request to a POST request link. Can you make sure you are sending the POST request to server?

Regarding to your Using the API V2 to import data, unfortunately documentation is really weak about this topic as well as there is no POST request. API V2 is only accepting GET requests.

I also just realized the link you are trying to use (https://ee.kobotoolbox.org/api/v1/submissions.json) is wrong, https://kc.kobotoolbox.org/api/v1/submissions.json you can try to use this link instead. ee. links are for enketo and it will not work. If you are using eu server it should be https://kc-eu.kobotoolbox.org/api/v1/submissions.json

2 Likes

Thanks @osmanburcu, I was posting but i have finally figured it out.

I was missing the right URL’s.

so, it looks like there are three URLs:
https://kc.kobotoolbox.org/ - used for API V1.
https://kf.kobotoolbox.org/ - used for API V2 & the form editor
https://ee.kobotoolbox.org/ - Form Submission

I have from there created a simple python script to upload the entries synchronously. It reads a FLAT CSV file, with the headers matching the data fields. where there are groups, they should be entered as group/field, and the script will separate them

data.csv header example

remove/uuidGeneration, meta/instanceID, field1, remove/calculationData ,group1/field1, group1/field2, group2,field1
  • any headers that have “remove” in them will not be uploaded. These can be used to calculate other fields in the spreadsheet, before saving as a CSV.
  • meta/instanceID is required, and must have the shape of uuid:<UUID>
  • Every new item to upload MUST have a UUID, this prevents duplicate data from being uploaded.
  • UUIDs can be calculated in libre office calc with the following formula:
 =LOWER(CONCATENATE(DEC2HEX(RANDBETWEEN(0,4294967295),8),"-",DEC2HEX(RANDBETWEEN(0,65535),4),"-",DEC2HEX(RANDBETWEEN(0,65535),4),"-",DEC2HEX(RANDBETWEEN(0,65535),4),"-",DEC2HEX(RANDBETWEEN(0,4294967295),8),DEC2HEX(RANDBETWEEN(0,65535),4)))

you need to update in the script below:

  • the exact title of the form you want to upload to, which can be found using this API
  • username and password

There are a number of things that it does not do, and if i fix up the script or add functionality, i will update it here.

  • data validation
  • keeping track of where it was last at
  • validate the form data is in the correct shape
  • graceful error handling
  • only handles one level of grouping/nesting. if you have more than one / in your headers, it will not work!

koboFormSubmission.py - it will take the form name, get the required metadata used for upload, then collect the submission data.csv, parse it into the correct JSON shape and upload each item one by one, with a 1 second pause between POSTs

# Reference Documents: 
#  https://kc.kobotoolbox.org/api/v1/?format=json
#  Youtube Video: https://www.youtube.com/watch?v=9q3kVr4m7LY
#  Forum Post: https://community.kobotoolbox.org/t/error-500-on-data-import-with-python/43200

## Useful URLS:
# https://kc.kobotoolbox.org/ - used for API V1.  
# https://kf.kobotoolbox.org/ - used for API V2 & the form editor
# https://ee.kobotoolbox.org/ - Form Submission

## Generate UUID in LibreOffice with the following Calc:
## =LOWER(CONCATENATE(DEC2HEX(RANDBETWEEN(0,4294967295),8),"-",DEC2HEX(RANDBETWEEN(0,65535),4),"-",DEC2HEX(RANDBETWEEN(0,65535),4),"-",DEC2HEX(RANDBETWEEN(0,65535),4),"-",DEC2HEX(RANDBETWEEN(0,4294967295),8),DEC2HEX(RANDBETWEEN(0,65535),4)))

import requests
import json
import uuid
import csv
from pprint import pprint
import time


def create_uuid():
    return str(uuid.uuid4())

def post_data_to_kobo(formId, formUUID, submissionData, submissionURL, username, password):
    print("Uploading item number " + str(index))
    url = submissionURL
    auth = (username, password)

    submissionData["id"] = formId
    submissionData["submission"]["formhub"] = {"uuid": formUUID}

    headers = {
        "Content-Type": "application/json"
    }

    try:
        response = requests.post(url, json=submissionData, auth=auth, headers=headers)

        if response.status_code == 201:
            print("Data submission successful!")
        else:
            print(f"Data submission failed with status code: {response.status_code}")
            print(response.text)

    except requests.exceptions.RequestException as e:
        print("An error occurred during the data submission:")
        print(e)

def csvToDict(csvFile):
    data = []

    # Open a csv reader called DictReader
    with open(csvFile, encoding='utf-8') as csvf:
        csvReader = csv.DictReader(csvf)
        data = list(csvReader)
    
    return data

def formatKeys(data):
    # pprint(data)
    newDict = {"submission": {}}
    for key in data:
        if "remove" not in key:
            if "/" in key:
                newKeys = key.split("/")
                # print(newKeys)
                
                if newKeys[0] == "meta":
                    newDict["submission"][newKeys[0]] = {}
                    newDict["submission"][newKeys[0]][newKeys[1]] = data[key]
                else:
                    if newKeys[0] not in newDict["submission"]:
                        newDict["submission"][newKeys[0]] = {}

                    newDict["submission"][newKeys[0]][newKeys[1]] = data[key]
            
            else:
                newDict["submission"][key] = data[key]
        
    return newDict
            

def getFormMetaData(title, username, password):
    url = "https://kc.kobotoolbox.org/api/v1/forms?format=json"
    auth = (username, password)

    response = requests.get(url, auth=auth)

    if response.status_code == 200:
        print("form MetaData request successful!")
        data = response.json()
    else:
        print(f"form MetaData request failed with status code: {response.status_code}")
        print(response.text)

    for form in data:
        if form["title"] == title:
            metaData = {"title": title,
                        "formId": form["id_string"],
                        "formUUID": form["uuid"]}
    pprint(metaData)
    return(metaData)
    
    

if __name__ == "__main__":
    submissionDataFile = "data.csv"
    submissionURL =  "https://kc.kobotoolbox.org/api/v1/submissions"

    username = "{{Username}}"
    password = "{{Password}}"


    formTitle = "{{Title of the Form}}"
    
    print("Retrieving details about form " + formTitle)
    formMetaData = getFormMetaData(formTitle, username, password)
    
    submissionData = csvToDict(submissionDataFile)

    cleansedData = []
    for item in submissionData:
      cleansedData.append(formatKeys(item))
    

    input()
    index = 0
    for data in cleansedData:
      
      post_data_to_kobo(formMetaData["formId"], formMetaData["formUUID"], data, submissionURL, username, password)
      time.sleep(1)
      index+=1


1 Like

Thank you @leigh_vandermerwe, for documenting and making the community a rich forum of resources. :bowing_man: Expecting similar participation in the future too.

all good, i did not want to solve it and disappear into the night.

Ultimately it came down to confusion about the 3 different subdomains, and the functions of each.

Even now, i do not understand what or why they are there, just that i need to use different ones. An official document that outlines them would be fantastic. I don’t actually know how i stumbled across all three, as i really only use two in the front end (kf & ee).

Otherwise, the API’s seem to have been written quite nicely, in a proper restful manner, which presents all the resources that you have available, and allows it to be self-discoverable (as long as you are in the correct API!)

cheers :smiley:

Hello,
I have been trying for some time to export my data to my Kobo server, but I am encountering errors in my code. I would like to have scripts that can allow me to do this.

this is a python script "

Convertir les colonnes de type ‘Timestamp’ en chaînes de caractères

for col in data.select_dtypes(include=[‘datetime64[ns]’]).columns:
data[col] = data[col].astype(str)

Convertir les données en JSON

data_json = data.to_dict(orient=‘records’)

Définir l’URL de l’API KoBoToolbox pour soumettre des données

url = ‘https://kf.kobotoolbox.org/api/v2/assets/aRyD46H2VpGvb24tf9vYbZ/submissions/

Token d’API

api_token = ‘d954872a1e4fef8bc152859667ca05c43f4cf673’

En-têtes de la requête

headers = {
‘Authorization’: f’Token {api_token}',
‘Content-Type’: ‘application/json’
}

Faire la requête POST pour chaque enregistrement de données

for record in data_json:
response = requests.post(url, headers=headers, data=json.dumps(record))

# Vérifier la réponse
if response.status_code == 201:
    print('Données uploadées avec succès!')
else:
    print(f"Erreur lors de l'upload des données: {response.status_code}")
    print(response.text)

and this the error
Erreur lors de l’upload des données: 404

Not Found

Not Found

The requested resource was not found on this server.

can you please put your code in a code block?

image

or use three back ticks (same key as ~)

ok, looking at your code:

  1. there is no API V2 submissions URL. look at the submissionURL value in my code above for the correct one. This is why you are getting 404 - Not Found. It literally does not exist.
  2. you need to have the data in the correct shape to POST. have a look at the API reference link in my post to see the data shape. I cannot see the shape of your data.
  3. you need to have the formID and the formUUID within the JSON of the POST.

tl;dr: you are trying to hit an API that does not exist. you need to use the V1 API submission end point.

@leigh_vandermerwe,

Thank you so much for taking the time to sort that out, and sharing it with the community!
Had been struggling with the same URL issue, but had given up pretty fast!
Your solution worked perfectly for me.

As a side note for non-experts: I happened to have a " in my pw, which caused the script authentication part to fail: even if I had the python string enclose in single-quotes, seems like the server was still getting confused!

1 Like


# Convertir les données en JSON
data_json = data.to_json(orient='records')

# Définir l'URL de l'API KoBoToolbox
url = 'https://kf.kobotoolbox.org/api/v2/assets/aRyD46H2VpGvb24tf9vYbZ/data/'

# Remplacer {form_id} par l'ID de votre formulaire
form_id = 'aRyD46H2VpGvb24tf9vYbZ'
url = url.format(form_id=form_id)

# Token d'API
api_token = 'd954872a1e4fef8bc152856875ca05c43f4cf673'

# En-têtes de la requête
headers = {
    'Authorization': f'Token {api_token}',
    'Content-Type': 'application/json'
}

# Faire la requête POST
response = requests.post(url, headers=headers, data=data_json)

# Vérifier la réponse
if response.status_code == 201:
    print('Données uploadées avec succès!')
else:
    print(f"Erreur lors de l'upload des données: {response.status_code}")
    print(response.text)

edit your previous posts and remove the token and any other ID’s in there.

Hi
there is all the information to submit the data in this code, please help me to adapt it.

yeah, i dont think you even looked at my code to work out why yours doesnt work.

API V2 does not allow for submissions.

If you look at my code it DOES NOT use API V2.

If you read my post above it has details on how to use the API for submissions.