Using API to upload media files?

User Support API
, , , ,
21

Hi @raffy_m,

My guess is that your issue lies here:

multipartContent.Add(new ByteArrayContent(File.ReadAllBytes(file.FullName)), "data_file", file.Name);

Note the structure of this from the Python code — the content of data_file is within a tuple:

{'data_file': (<filename>, <bites>, <mime>)}

i.e.

{'data_file': (FILENAME, open(fr'{FILE_FOLDER}\{FILENAME}', 'rb').read(), MIME)}
1 Like
22

I was able to solve it.

@Josh posted this curl script earlier:

curl https://<kc_url>/api/v1/metadata.json -H “Authorization: Token ”
-F “xform=<xform_id>”
-F “data_value=”
-F “data_type=media”
-F “data_file=@<path/to/file>”

I used this online C# to curl converter and this is the code i got:

using (var httpClient = new HttpClient())
{
using (var request = new HttpRequestMessage(new HttpMethod("POST"), "http://https//<kc_url>/api/v1/metadata.json"))
{
request.Headers.TryAddWithoutValidation("Authorization", "Token <token>");

    var multipartContent = new MultipartFormDataContent();
    multipartContent.Add(new StringContent(File.ReadAllText("xform_id>")), "xform");
    multipartContent.Add(new StringContent(File.ReadAllText("filename>")), "data_value");
    multipartContent.Add(new StringContent("media"), "data_type");
    multipartContent.Add(new ByteArrayContent(File.ReadAllBytes("<path/to/file>")), "data_file", Path.GetFileName("<path/to/file>"));
    request.Content = multipartContent; 

    var response = await httpClient.SendAsync(request);
}

}

Some lines did not translate well:

multipartContent.Add(new StringContent(File.ReadAllText("xform_id>")), "xform");
multipartContent.Add(new StringContent(File.ReadAllText("filename>")), "data_value");

These should have been

multipartContent.Add(new StringContent("xform_id>"), "xform");
multipartContent.Add(new StringContent("filename>"), "data_value");

This is the working code that I use now:

    private async Task<HttpResponseMessage> UploadMediaToServer(FileInfo file)

    {
        HttpResponseMessage result = null;
        string baseURL = "https://kc.kobotoolbox.org/api/v1/metadata.json";
        using (var httpClient = new HttpClient())
        {
            using (var request = new HttpRequestMessage(new HttpMethod("POST"), baseURL))
            {
                var base64authorization = Convert.ToBase64String(Encoding.ASCII.GetBytes($"{_user}:{_password}"));
                request.Headers.TryAddWithoutValidation("Authorization", $"Basic {base64authorization}");

                var multipartContent = new MultipartFormDataContent();
                multipartContent.Add(new StringContent(_formID), "xform");
                multipartContent.Add(new StringContent(file.Name), "data_value");
                multipartContent.Add(new StringContent("media"), "data_type");
                multipartContent.Add(new StringContent("text/csv"), "data_file_type");
                multipartContent.Add(new ByteArrayContent(File.ReadAllBytes(file.FullName)), "data_file", file.Name);
                request.Content = multipartContent;

                result = await httpClient.SendAsync(request);
            }
        }
        return result;
    }

The status code returned is 201, meaning it works perfectly.

Requerying the form metadata, the `text/csv’ attribute of the updated CSV file is not recognized as shown by the JSON text below:

{"url":"https://kc.kobotoolbox.org/api/v1/metadata/1331767?format=json",
"id":1331767,
"xform":<xformID>,
"data_value":"enumerator_select.csv",
"data_type":"media",
"data_file":"https://kobocat-s3.s3.amazonaws.com/<form_username>/form-media/<long random string>",
"data_file_type":"", CSV ATTRIBUTE IS NOT RECOGNIZED
"file_hash":"md5:be30dc27789387659818e428b63d867a"},

This is the json text of a csv file that was not updated. The text/csv value is still there.

{"url":"https://kc.kobotoolbox.org/api/v1/metadata/1299805?format=json", "id":1299805,
"xform":<xformID>,
"data_value":"gear.csv",
"data_type":"media",
"data_file":"https://kobocat-s3.s3.amazonaws.com/<form_username>/form-media/<long random string>",
"data_file_type":"text/csv",
"file_hash":"md5:e89a4c6e61e958a5e7b3a624fc7fa8f1"},

Is the missing attribute something that should be of concern?

Thanks

1 Like
23

Hi @raffy_m,

Great to hear that you got it working! I’m not certain if it will cause issues, but perhaps try leaving out this line where you explicitly set it and see what happens?

multipartContent.Add(new StringContent("text/csv"), "data_file_type");

Or instead see if you can add it to the tuple assigned to “data_file” as is the case in the Python code.

1 Like
24

So far, I see that the missing text/csv attribute in the metadata after a media file is updated using API has no effect on the files that are downloaded by the mobile device. The e-form using ODK-Collect works as usual.

Additionally, there is no need to re-deploy the form. After updating all the CSV files via API, I downloaded the updated form in ODK-Collect . All the new items in the updated CSVs were visible in the e-form

1 Like
25

In the new versions where the media files are uploaded using the KPI interface, adding them programmatically using this approach seems to work, but the files are not seen in the frontend.
Is this a bug? Or is there a KPI API to now upload the files?

26

@ks_1 any news on this?

27

No, hoping for a reply from the developers…

28

Hi @ivanradisson, @ks_1 sorry for not seeing this — please ping one of the team otherwise it’s easy to miss messages.

You can view the API v2 documentation for the changes here:

https://<kf_url>/api/v2/assets/<asset_uid>/files/

2 Likes
29

@Josh, the c# code that I am using (v1 API) to upload/update media files to kobotoolbox is not working. I get a 403 code or Forbidden. How do I do this using V2 api? Thanks

30

Hi @raffy_m, please see my previous message above pointing to the API v2 docs for media files :+1:

1 Like
31

@josh

I got a bad request response (status code:400) in submitting the payload which looks like the formatted example below: (The actual payload string does not have the carriage returns that are in the example)

{
"user":"https://kf.kobotoolbox.org/api/v2/users/a_user_id/", "asset":"https://kf.kobotoolbox.org/api/v2/asset/asset_guid/", "description":"default", "base64Encoded":"aLongStringThatIBelieveIsBase64EncodedStringDerivedFromCSVData", "metadata":{"filename":"a_csv_file.csv"}
}

This is the C# code i use to Post to the V2 api:

var base64string = Convert.ToBase64String(Encoding.Unicode.GetBytes(File.ReadAllText(f.FullName)));

string payload =
"{" +
 $"\"user\":\"https://kf.kobotoolbox.org/api/v2/users/{_user}/\"," +
 $"\"asset\":\"https://kf.kobotoolbox.org/api/v2/assets/{assetID}/\"," +
 "\"description\":\"default\"," +
 $"\"base64Encoded\":\"{base64string}\"," +
 $"\"metadata\":{{\"filename\":\"{f.Name}\"}}" +
"}";

using (request = new HttpRequestMessage(new HttpMethod("POST"), $"https://kf.kobotoolbox.org/api/v2/assets/{assetID}/files/"))
{
 base64authorization = Convert.ToBase64String(Encoding.ASCII.GetBytes($"{_user}:{_password}"));
 request.Headers.TryAddWithoutValidation("Authorization", $"Basic {base64authorization}");
 request.Content = new StringContent(payload);
 request.Content.Headers.ContentType = MediaTypeHeaderValue.Parse("application/json");

response = await httpClient.SendAsync(request);

The response returned contains a status code:400 and additionally it says bad request

Thanks

32

Hi @raffy_m, I’m not familiar with C#, but here’s how you can do it with curl and Python:

curl

curl -X POST https://kf.kobotoolbox.org/api/v2/assets/$ASSET_UID/files.json \
-H "Authorization: Token $TOKEN" \
-F content=@something.png \
-F file_type=form_media \
-F description=something \
-F metadata='{"filename": "something.png"}'

Python

import requests
import json

TOKEN = 'your secret token'
ASSET_UID = 'your asset uid'
URL = f'https://<kf_url>/api/v2/assets/{ASSET_UID}/files.json'
HEADERS = {'Authorization': f'Token {TOKEN}'}

payload = {'filename': 'something.png'}
data = {'description': 'something', 'metadata': json.dumps(payload), 'file_type': 'form_media'}
files = {'content': open('something.png', 'rb')}
res = requests.post(url=URL, headers=HEADERS, data=data, files=files)
1 Like
33

@Josh , Can I ask for an example specific to a posting a csv file? I still get a bad request error and I think its because the parameter names are different when sending a csv file compared to a graphic.

Thanks

34

@Josh, the curl example that you sent worked. I converted it to c# using this online converter:
https://curl.olsh.me/

So thanks for the help.

This is the code i C# that i used based on the curl example posted earlier

string baseURL = $"https://kf.kobotoolbox.org/api/v2/assets/{assetID}";
using (var httpClient = new HttpClient())
{
  using (request = new HttpRequestMessage(new HttpMethod("POST"), $"{baseURL}/files.json"))
  {
     request.Headers.TryAddWithoutValidation("Authorization", $"Token {_token}");
        var multipartContent = new MultipartFormDataContent();
        multipartContent.Add(new ByteArrayContent(File.ReadAllBytes(f.FullName)), "content", $"{f.Name}");
        multipartContent.Add(new StringContent("form_media"), "file_type");
        multipartContent.Add(new StringContent("default"), "description");
        multipartContent.Add(new StringContent($"{{\"filename\": \"{f.Name}\"}}"), "metadata");

        request.Content = multipartContent;
        var response = await httpClient.SendAsync(request);
  }
}
2 Likes
35

Hello,

I have tried to use this python code as is but getting error 400. what could be the problem. In addition, is it right to have the KF and KC URLs defined as below:-

KC_URL = ‘https://kc.kobotoolbox.org/api/v1/’
KF_URL = ‘https://kf.kobotoolbox.org/api/v2/’

regards,

36

Hi @franklwambo21, are you referring to this Python code?

1 Like
37

Yes that is the code I am referring to. I am not sure why I get the bad request error.

38

Interesting… I just tried the code again and successfully uploaded a file. If you are referring to KF_URL and KC_URL I’m guessing you are referring to the snippet here and not the updated snippet here using API v2. Can you please confirm this @franklwambo21?

1 Like
39

hello @Josh I am actually using the python code snippet here . Just to follow up, what should be the convention of specifying the KC_URL and KF_URL for forms hosted on the free kobo toolbox server. I have set the form ID as the xforms parameter. And by the way what would change in the code if my form is hosted in the non-humanitarian platform.

40

Finally, I got this working when I removed the quotes around the XForm value

2 Likes