Using API to upload media files?

No, hoping for a reply from the developers…

Hi @ivanradisson, @ks_1 sorry for not seeing this — please ping one of the team otherwise it’s easy to miss messages.

You can view the API v2 documentation for the changes here:

https://<kf_url>/api/v2/assets/<asset_uid>/files/

2 Likes

@Josh, the c# code that I am using (v1 API) to upload/update media files to kobotoolbox is not working. I get a 403 code or Forbidden. How do I do this using V2 api? Thanks

Hi @raffy_m, please see my previous message above pointing to the API v2 docs for media files :+1:

1 Like

@josh

I got a bad request response (status code:400) in submitting the payload which looks like the formatted example below: (The actual payload string does not have the carriage returns that are in the example)

{
"user":"https://kf.kobotoolbox.org/api/v2/users/a_user_id/", "asset":"https://kf.kobotoolbox.org/api/v2/asset/asset_guid/", "description":"default", "base64Encoded":"aLongStringThatIBelieveIsBase64EncodedStringDerivedFromCSVData", "metadata":{"filename":"a_csv_file.csv"}
}

This is the C# code i use to Post to the V2 api:

var base64string = Convert.ToBase64String(Encoding.Unicode.GetBytes(File.ReadAllText(f.FullName)));

string payload =
"{" +
$"\"user\":\"https://kf.kobotoolbox.org/api/v2/users/{_user}/\"," +
$"\"asset\":\"https://kf.kobotoolbox.org/api/v2/assets/{assetID}/\"," +
"\"description\":\"default\"," +
$"\"base64Encoded\":\"{base64string}\"," +
$"\"metadata\":{{\"filename\":\"{f.Name}\"}}" +
"}";

using (request = new HttpRequestMessage(new HttpMethod("POST"), $"https://kf.kobotoolbox.org/api/v2/assets/{assetID}/files/"))
{
base64authorization = Convert.ToBase64String(Encoding.ASCII.GetBytes($"{_user}:{_password}"));
request.Headers.TryAddWithoutValidation("Authorization", $"Basic {base64authorization}");
request.Content = new StringContent(payload);
request.Content.Headers.ContentType = MediaTypeHeaderValue.Parse("application/json");

response = await httpClient.SendAsync(request);

The response returned contains a status code:400 and additionally it says bad request

Thanks

Hi @raffy_m, I’m not familiar with C#, but here’s how you can do it with curl and Python:

curl

curl -X POST https://kf.kobotoolbox.org/api/v2/assets/$ASSET_UID/files.json \
-H "Authorization: Token $TOKEN" \
-F content=@something.png \
-F file_type=form_media \
-F description=something \
-F metadata='{"filename": "something.png"}'

Python

import requests
import json

TOKEN = 'your secret token'
ASSET_UID = 'your asset uid'
URL = f'https://<kf_url>/api/v2/assets/{ASSET_UID}/files.json'
HEADERS = {'Authorization': f'Token {TOKEN}'}

payload = {'filename': 'something.png'}
data = {'description': 'something', 'metadata': json.dumps(payload), 'file_type': 'form_media'}
files = {'content': open('something.png', 'rb')}
res = requests.post(url=URL, headers=HEADERS, data=data, files=files)
1 Like

@Josh , Can I ask for an example specific to a posting a csv file? I still get a bad request error and I think its because the parameter names are different when sending a csv file compared to a graphic.

Thanks

@Josh, the curl example that you sent worked. I converted it to c# using this online converter:
https://curl.olsh.me/

So thanks for the help.

This is the code i C# that i used based on the curl example posted earlier

string baseURL = $"https://kf.kobotoolbox.org/api/v2/assets/{assetID}";
using (var httpClient = new HttpClient())
{
  using (request = new HttpRequestMessage(new HttpMethod("POST"), $"{baseURL}/files.json"))
  {
     request.Headers.TryAddWithoutValidation("Authorization", $"Token {_token}");
        var multipartContent = new MultipartFormDataContent();
        multipartContent.Add(new ByteArrayContent(File.ReadAllBytes(f.FullName)), "content", $"{f.Name}");
        multipartContent.Add(new StringContent("form_media"), "file_type");
        multipartContent.Add(new StringContent("default"), "description");
        multipartContent.Add(new StringContent($"{{\"filename\": \"{f.Name}\"}}"), "metadata");

        request.Content = multipartContent;
        var response = await httpClient.SendAsync(request);
  }
}
2 Likes

Hello,

I have tried to use this python code as is but getting error 400. what could be the problem. In addition, is it right to have the KF and KC URLs defined as below:-

KC_URL = ‘https://kc.kobotoolbox.org/api/v1/
KF_URL = ‘https://kf.kobotoolbox.org/api/v2/

regards,

Hi @franklwambo21, are you referring to this Python code?

1 Like

Yes that is the code I am referring to. I am not sure why I get the bad request error.

Interesting… I just tried the code again and successfully uploaded a file. If you are referring to KF_URL and KC_URL I’m guessing you are referring to the snippet here and not the updated snippet here using API v2. Can you please confirm this @franklwambo21?

1 Like

hello @Josh I am actually using the python code snippet here . Just to follow up, what should be the convention of specifying the KC_URL and KF_URL for forms hosted on the free kobo toolbox server. I have set the form ID as the xforms parameter. And by the way what would change in the code if my form is hosted in the non-humanitarian platform.

Finally, I got this working when I removed the quotes around the XForm value

2 Likes

Hi @ks_1 @Josh,

Could you please check my code below. There is no error but as I checked on the kobo media files there is no uploaded csv files on the kobo form medial files.

import os
import requests
import json

KC_URL = ‘KoBoCAT REST API
KF_URL = ‘https://kf.kobotoolbox.org/api/v2/assets/aSXwstUL******************/submissions/

TOKEN = ‘5b359613c8f***************************’ # Replace with your KoBoToolbox API token
XFORM = ‘aSXwstUL2U3W**********’ # Replace with the relevant KoBoToolbox form ID
FILE_FOLDER = ‘/Users/andisulasikin/PycharmProjects/pythonProject/uploadtokobo’ # Replace with the local folder path where your CSV files are stored
MIME = ‘text/csv’
headers = {‘Authorization’: f’Token {TOKEN}'}

#Get a list of all files in the folder
file_list = [f for f in os.listdir(FILE_FOLDER) if os.path.isfile(os.path.join(FILE_FOLDER, f))]

for filename in file_list:
files = {‘data_file’: (filename, open(os.path.join(FILE_FOLDER, filename), ‘rb’).read(), MIME)}
data = {
‘data_value’: filename,
‘xform’: XFORM,
‘data_type’: ‘media’,
‘data_file_type’: MIME,
}

# Download metadata.json
response = requests.get(fr"{KC_URL}/metadata.json", headers=headers)
try:
    dict_response = json.loads(response.text)
except json.JSONDecodeError:
    dict_response = []
# Delete appropriate entry in the metadata.json (delete old file)
found = False
for each in dict_response:
    if isinstance(each, dict) and each.get('xform') == XFORM and each.get('data_value') == filename:
        found = True
        del_id = each['id']
        response = requests.delete(fr"{KC_URL}/metadata/{del_id}", headers=headers)
        break

if not found:
    print(f"File {filename} not found in metadata. Proceeding with initial upload.")

# Upload the changed file
response = requests.post(fr"{KC_URL}/metadata.json", data=data, files=files, headers=headers) print("All files uploaded successfully.")

Add a print(response) in the last line and see what is returned by the server.

1 Like

Oh sorry I missed to add print(“All files uploaded successfully.”)

here the response

But on the koboform media files, there is no uploaded csv files.

The files uploaded through this method are not visible in the new kpi interface (as mentioned in this thread somewhere iirc), but they are available to the forms.

You can do a test to confirm.

1 Like

Unfortunately, I’ve checked the preview form and they are still not available.

What do you think @ks_1 @Josh @Kal_Lam ? Do you have any solution?

Thanks for those inputs! I manage to do most of it in R, including deleting the csv file that I use to pulldata. However, I cannot upload a new csv file to substitute the deleted one using POST() (library httr).
Would you be able to write the R statement(s) that would upload a csv to the Kobotoolbox server?
Thanks for considering.