Using API to upload media files?

User Support API
, , , ,
1

I want to upload a new CSV file for use with pulldata() every day. Since this file is programmatically created from our back end, I want to update the form on a daily basis using the API. Is there a way to do this?

2

I have not tried this out but it should be possible if your program is able to automatically update the csv file that is present in the KoBoToolbox server replacing the earlier csv file. Hope our community would add to this.

3

What I wanted was to add form-media for a particular form, using an API endpoint. My program can then call this API every day and attach the CSV file. I tried linking the CSV file in this area after uploading it to a public folder:

image
image1167×318 9.14 KB

But it doesn’t work. I guess it only works for images.

4

Maybe you could make a post to /api/v1/metadata/ with the following payload:

{
    xform: <xform ID>,  # can be retrieve from `/api/v1/forms.json`
   data_value: "<URL to your CSV file>",
   data_type: "media",
}
1 Like
5

I’m not able to access the endpoint /api/v1/metadata

6

Hi @ks_1,

Try /api/v1/metadata.json instead.

2 Likes
7

Thanks @Kal_Lam and @nolive, I was able to add the URL using the API. But this just created a media URL link in the media section, as I tried manually earlier. And this approach doesn’t help to pull the data from the remote CSV file.

Can you help me to change this payload to attach a file and upload it from the local system?

{
    xform: <xform ID>,  # can be retrieve from `/api/v1/forms.json`
   data_value: "<URL to your CSV file>",
   data_type: "media",
}
8

Hi @ks_1, here are two ways you can achieve this for a file upload:

  1. Using curl:
curl https://<kc_url>/api/v1/metadata.json -H "Authorization: Token <token>" \
-F "xform=<xform_id>" \
-F "data_value=<filename>" \
-F "data_type=media" \
-F "data_file=@<path/to/file>"
  1. Using a Python script:
import requests

URL = 'https://<kc_url>/api/v1/metadata.json'
TOKEN = '<token>'
FILE_PATH = '<path/to/file>'
FILENAME = '<filename>'
MIME = 'text/csv'
XFORM = '<xform_id>'

headers = {'Authorization': f'Token {TOKEN}'}
files = {'data_file': (FILENAME, open(f'{FILE_PATH}{FILENAME}', 'rb').read(), MIME)}
data = {
    'data_value': FILENAME,
    'xform': XFORM,
    'data_type': 'media',
    'data_file_type': MIME,
}
response = requests.post(URL, data=data, files=files, headers=headers)
2 Likes
9

@Josh Thank you so much!! The python script worked great! I have a couple more questions:

  1. How do I use the API to delete a file that I’ve uploaded? When I tried to upload the changed file again, I get the error:
    {'non_field_errors': ['The fields xform, data_type, data_value must make a unique set.']}
    So I would need to delete the previous uploaded file before attempting to reupload.

  2. I am guessing that ODK collect will not download the changed CSV unless I redeploy the form. So how can I redeploy this form with the API?

Once again, thanks a lot.

10

@ks_1, great!

  1. Yes, you have to have unique files. If you look at the return data in the response from the upload, you will see an id field. You can do a DELETE request to /api/v1/metadata/<id> to remove the file. You can get all our form’s data in the response from a GET to /api/v1/forms.json
  2. I’m not certain about how ODK will treat the new CSV, but you can redeploy through the API to check

To redeploy your form:

curl -X PATCH https://<kf url>/api/v2/assets/<form uid>/deployment/?format=json \
-d 'active=true' \
-d 'version_id=<form version id>' \
-H 'Authorization: Token <token>'
2 Likes
11

You can get your form’s version_id by doing the following:

curl 'https:/<kf url>/api/v2/assets/<form uid>.json' \                                                      
-H 'Authorization: Token <token>'

If you have jq installed, you can find it easily by piping to jq:

curl -s 'https:/<kf url>/api/v2/assets/<form uid>.json' \                                                      
-H 'Authorization: Token <token>' | jq '.version_id'
2 Likes
12

@Josh It works beautifully! Many thanks.
I did some experimenting and I found that ODK will download the new CSV file if you sync manually, even if you don’t redeploy the form. Here’s the final python script for anyone who wants to programmatically upload the CSV file for pulldata purposes:

    import requests
    import json

    KC_URL = 'https://<kc_url>/api/v1/'
    KF_URL = 'https://<kf_url>/api/v2/'

    TOKEN = '<token>'      
    XFORM = <xform_id>

    FILE_FOLDER = r"<file_folder>"
    FILENAME = '<file_name>'
    MIME = 'text/csv'


    headers = {'Authorization': f'Token {TOKEN}'}
    files = {'data_file': (FILENAME, open(fr'{FILE_FOLDER}\{FILENAME}', 'rb').read(), MIME)}
    data = {
        'data_value': FILENAME,
        'xform': XFORM,
        'data_type': 'media',
        'data_file_type': MIME,
    }

    # Download metadata.json
    response = requests.get(fr"{KC_URL}/metadata.json", headers=headers)
    dict_response = json.loads(response.text)

    # Delete appropriate entry in the metadata.json (delete old file)
    for each in dict_response:
        if each['xform'] == XFORM and each['data_value'] == FILENAME:
            del_id = each['id']
            response = requests.delete(fr"{KC_URL}/metadata/{del_id}", headers=headers)
            break

    # Upload the changed file
    response = requests.post(fr"{KC_URL}/metadata.json", data=data, files=files, headers=headers)
4 Likes
13

Awesome! Thanks for sharing :muscle:

1 Like
14

My pleasure. Is there any detailed documentation about all the available API endpoints?

15

Hi @ks_1,

We don’t have a great central location with all the API endpoints and their methods (something that we’ll hopefully get sorted soon), but you can have a look at some of these locations to get started:

2 Likes
16

One of the posts in the community shows how to upload media files (csv).

Using API to upload media files?

I would like to ask if I upload a csv file as an update to a file already in the server, would it overwrite that file? If not then how do I delete media files in the server using API so that I can upload an update?

Thanks

17

@raffy_m
You need to delete the old file. I have commented the section in the code where it does this.

1 Like
18

I have been trying to post a csv file in c#. Here is my code:

    private async Task<HttpResponseMessage> UploadMediaToServer(FileInfo file)

    {
        HttpResponseMessage result = null;
        string baseURL = "https://kc.kobotoolbox.org/api/v1/metadata.json";


        using (var httpClient = new HttpClient())
        {
            using (var request = new HttpRequestMessage(new HttpMethod("POST"), baseURL))
            {
                request.Headers.TryAddWithoutValidation("Authorization", "Token <token>");
                var multipartContent = new MultipartFormDataContent();
                multipartContent.Headers.Add("xform", _formID);
                multipartContent.Headers.Add("data_value", file.Name);
                multipartContent.Headers.Add("data_type", "media");
                multipartContent.Headers.ContentType = MediaTypeHeaderValue.Parse("text/csv");
                multipartContent.Add(new ByteArrayContent(File.ReadAllBytes(file.FullName)), "data_file", file.Name);
                request.Content = multipartContent;

                result = await httpClient.SendAsync(request);
            }
        }
        return result;
    }

I keep on getting a result with status code 415 - unsupported media type.

Any help is greatly appreciated. Thanks

19

@raffy_m Try adding

‘data_file_type’: ‘text/csv’

to the header

1 Like
20

When i add
'data_file_type’: ‘text/csv'

I get a 400 status code or Bad Request.

Do you think it has something to do with how the payload is encoded using ByteArrayContent?

I also tried this:

byte[] bytes = File.ReadAllBytes(file.FullName);
HttpContent fileContent = new ByteArrayContent(bytes);
multipartContent.Add(fileContent);
request.Content = multipartContent;

Thanks