Thanks @Kal_Lam and @nolive, I was able to add the URL using the API. But this just created a media URL link in the media section, as I tried manually earlier. And this approach doesn’t help to pull the data from the remote CSV file.
Can you help me to change this payload to attach a file and upload it from the local system?
{
xform: <xform ID>, # can be retrieve from `/api/v1/forms.json`
data_value: "<URL to your CSV file>",
data_type: "media",
}Hi @ks_1, here are two ways you can achieve this for a file upload:
curl:curl https://<kc_url>/api/v1/metadata.json -H "Authorization: Token <token>" \
-F "xform=<xform_id>" \
-F "data_value=<filename>" \
-F "data_type=media" \
-F "data_file=@<path/to/file>"
import requests
URL = 'https://<kc_url>/api/v1/metadata.json'
TOKEN = '<token>'
FILE_PATH = '<path/to/file>'
FILENAME = '<filename>'
MIME = 'text/csv'
XFORM = '<xform_id>'
headers = {'Authorization': f'Token {TOKEN}'}
files = {'data_file': (FILENAME, open(f'{FILE_PATH}{FILENAME}', 'rb').read(), MIME)}
data = {
'data_value': FILENAME,
'xform': XFORM,
'data_type': 'media',
'data_file_type': MIME,
}
response = requests.post(URL, data=data, files=files, headers=headers)
@Josh Thank you so much!! The python script worked great! I have a couple more questions:
How do I use the API to delete a file that I’ve uploaded? When I tried to upload the changed file again, I get the error:
{'non_field_errors': ['The fields xform, data_type, data_value must make a unique set.']}
So I would need to delete the previous uploaded file before attempting to reupload.
I am guessing that ODK collect will not download the changed CSV unless I redeploy the form. So how can I redeploy this form with the API?
Once again, thanks a lot.
@ks_1, great!
id field. You can do a DELETE request to /api/v1/metadata/<id> to remove the file. You can get all our form’s data in the response from a GET to /api/v1/forms.json
To redeploy your form:
curl -X PATCH https://<kf url>/api/v2/assets/<form uid>/deployment/?format=json \
-d 'active=true' \
-d 'version_id=<form version id>' \
-H 'Authorization: Token <token>'
You can get your form’s version_id by doing the following:
curl 'https:/<kf url>/api/v2/assets/<form uid>.json' \
-H 'Authorization: Token <token>'
If you have jq installed, you can find it easily by piping to jq:
curl -s 'https:/<kf url>/api/v2/assets/<form uid>.json' \
-H 'Authorization: Token <token>' | jq '.version_id'
@Josh It works beautifully! Many thanks.
I did some experimenting and I found that ODK will download the new CSV file if you sync manually, even if you don’t redeploy the form. Here’s the final python script for anyone who wants to programmatically upload the CSV file for pulldata purposes:
import requests
import json
KC_URL = 'https://<kc_url>/api/v1/'
KF_URL = 'https://<kf_url>/api/v2/'
TOKEN = '<token>'
XFORM = <xform_id>
FILE_FOLDER = r"<file_folder>"
FILENAME = '<file_name>'
MIME = 'text/csv'
headers = {'Authorization': f'Token {TOKEN}'}
files = {'data_file': (FILENAME, open(fr'{FILE_FOLDER}\{FILENAME}', 'rb').read(), MIME)}
data = {
'data_value': FILENAME,
'xform': XFORM,
'data_type': 'media',
'data_file_type': MIME,
}
# Download metadata.json
response = requests.get(fr"{KC_URL}/metadata.json", headers=headers)
dict_response = json.loads(response.text)
# Delete appropriate entry in the metadata.json (delete old file)
for each in dict_response:
if each['xform'] == XFORM and each['data_value'] == FILENAME:
del_id = each['id']
response = requests.delete(fr"{KC_URL}/metadata/{del_id}", headers=headers)
break
# Upload the changed file
response = requests.post(fr"{KC_URL}/metadata.json", data=data, files=files, headers=headers)
Awesome! Thanks for sharing 
My pleasure. Is there any detailed documentation about all the available API endpoints?
Hi @ks_1,
We don’t have a great central location with all the API endpoints and their methods (something that we’ll hopefully get sorted soon), but you can have a look at some of these locations to get started:
One of the posts in the community shows how to upload media files (csv).
Using API to upload media files?
I would like to ask if I upload a csv file as an update to a file already in the server, would it overwrite that file? If not then how do I delete media files in the server using API so that I can upload an update?
Thanks
@raffy_m
You need to delete the old file. I have commented the section in the code where it does this.
I have been trying to post a csv file in c#. Here is my code:
private async Task<HttpResponseMessage> UploadMediaToServer(FileInfo file)
{
HttpResponseMessage result = null;
string baseURL = "https://kc.kobotoolbox.org/api/v1/metadata.json";
using (var httpClient = new HttpClient())
{
using (var request = new HttpRequestMessage(new HttpMethod("POST"), baseURL))
{
request.Headers.TryAddWithoutValidation("Authorization", "Token <token>");
var multipartContent = new MultipartFormDataContent();
multipartContent.Headers.Add("xform", _formID);
multipartContent.Headers.Add("data_value", file.Name);
multipartContent.Headers.Add("data_type", "media");
multipartContent.Headers.ContentType = MediaTypeHeaderValue.Parse("text/csv");
multipartContent.Add(new ByteArrayContent(File.ReadAllBytes(file.FullName)), "data_file", file.Name);
request.Content = multipartContent;
result = await httpClient.SendAsync(request);
}
}
return result;
}
I keep on getting a result with status code 415 - unsupported media type.
Any help is greatly appreciated. Thanks
When i add
'data_file_type’: ‘text/csv'
I get a 400 status code or Bad Request.
Do you think it has something to do with how the payload is encoded using ByteArrayContent?
I also tried this:
byte[] bytes = File.ReadAllBytes(file.FullName);
HttpContent fileContent = new ByteArrayContent(bytes);
multipartContent.Add(fileContent);
request.Content = multipartContent;
Thanks
Hi @raffy_m,
My guess is that your issue lies here:
multipartContent.Add(new ByteArrayContent(File.ReadAllBytes(file.FullName)), "data_file", file.Name);
Note the structure of this from the Python code — the content of data_file is within a tuple:
{'data_file': (<filename>, <bites>, <mime>)}
i.e.
{'data_file': (FILENAME, open(fr'{FILE_FOLDER}\{FILENAME}', 'rb').read(), MIME)}
I was able to solve it.
@Josh posted this curl script earlier:
curl https://<kc_url>/api/v1/metadata.json -H “Authorization: Token ”
-F “xform=<xform_id>”
-F “data_value=”
-F “data_type=media”
-F “data_file=@<path/to/file>”
I used this online C# to curl converter and this is the code i got:
using (var httpClient = new HttpClient())
{
using (var request = new HttpRequestMessage(new HttpMethod("POST"), "http://https//<kc_url>/api/v1/metadata.json"))
{
request.Headers.TryAddWithoutValidation("Authorization", "Token <token>");
var multipartContent = new MultipartFormDataContent();
multipartContent.Add(new StringContent(File.ReadAllText("xform_id>")), "xform");
multipartContent.Add(new StringContent(File.ReadAllText("filename>")), "data_value");
multipartContent.Add(new StringContent("media"), "data_type");
multipartContent.Add(new ByteArrayContent(File.ReadAllBytes("<path/to/file>")), "data_file", Path.GetFileName("<path/to/file>"));
request.Content = multipartContent;
var response = await httpClient.SendAsync(request);
}
}
Some lines did not translate well:
multipartContent.Add(new StringContent(File.ReadAllText("xform_id>")), "xform");
multipartContent.Add(new StringContent(File.ReadAllText("filename>")), "data_value");
These should have been
multipartContent.Add(new StringContent("xform_id>"), "xform");
multipartContent.Add(new StringContent("filename>"), "data_value");
This is the working code that I use now:
private async Task<HttpResponseMessage> UploadMediaToServer(FileInfo file)
{
HttpResponseMessage result = null;
string baseURL = "https://kc.kobotoolbox.org/api/v1/metadata.json";
using (var httpClient = new HttpClient())
{
using (var request = new HttpRequestMessage(new HttpMethod("POST"), baseURL))
{
var base64authorization = Convert.ToBase64String(Encoding.ASCII.GetBytes($"{_user}:{_password}"));
request.Headers.TryAddWithoutValidation("Authorization", $"Basic {base64authorization}");
var multipartContent = new MultipartFormDataContent();
multipartContent.Add(new StringContent(_formID), "xform");
multipartContent.Add(new StringContent(file.Name), "data_value");
multipartContent.Add(new StringContent("media"), "data_type");
multipartContent.Add(new StringContent("text/csv"), "data_file_type");
multipartContent.Add(new ByteArrayContent(File.ReadAllBytes(file.FullName)), "data_file", file.Name);
request.Content = multipartContent;
result = await httpClient.SendAsync(request);
}
}
return result;
}
The status code returned is 201, meaning it works perfectly.
Requerying the form metadata, the `text/csv’ attribute of the updated CSV file is not recognized as shown by the JSON text below:
{"url":"https://kc.kobotoolbox.org/api/v1/metadata/1331767?format=json",
"id":1331767,
"xform":<xformID>,
"data_value":"enumerator_select.csv",
"data_type":"media",
"data_file":"https://kobocat-s3.s3.amazonaws.com/<form_username>/form-media/<long random string>",
"data_file_type":"", CSV ATTRIBUTE IS NOT RECOGNIZED
"file_hash":"md5:be30dc27789387659818e428b63d867a"},
This is the json text of a csv file that was not updated. The text/csv value is still there.
{"url":"https://kc.kobotoolbox.org/api/v1/metadata/1299805?format=json", "id":1299805,
"xform":<xformID>,
"data_value":"gear.csv",
"data_type":"media",
"data_file":"https://kobocat-s3.s3.amazonaws.com/<form_username>/form-media/<long random string>",
"data_file_type":"text/csv",
"file_hash":"md5:e89a4c6e61e958a5e7b3a624fc7fa8f1"},
Is the missing attribute something that should be of concern?
Thanks
Hi @raffy_m,
Great to hear that you got it working! I’m not certain if it will cause issues, but perhaps try leaving out this line where you explicitly set it and see what happens?
multipartContent.Add(new StringContent("text/csv"), "data_file_type");
Or instead see if you can add it to the tuple assigned to “data_file” as is the case in the Python code.
So far, I see that the missing text/csv attribute in the metadata after a media file is updated using API has no effect on the files that are downloaded by the mobile device. The e-form using ODK-Collect works as usual.
Additionally, there is no need to re-deploy the form. After updating all the CSV files via API, I downloaded the updated form in ODK-Collect . All the new items in the updated CSVs were visible in the e-form
In the new versions where the media files are uploaded using the KPI interface, adding them programmatically using this approach seems to work, but the files are not seen in the frontend.
Is this a bug? Or is there a KPI API to now upload the files?
