Csv (media) file upload/update via API does not update same files in the Settings->Media section of Kobotoolbox

Hello,

Kind request:
Reposting this issue a second time. The first posting from 2 days ago, was automatically removed by this site’s automated spam filter (Akismet). Not sure what the reason for it could be. It was supposed to be reviewed (by a staff member, the message says) and re-appear, but that has not happened yet. Sincerely hope this does not get auto-removed again. This is a lengthy post to write again as you can see! This is also a serious form management issue for me. Have been managing this manually for some months now, since I did not have the time to write this (long) post. Could really use some help in this matter…

The Problem:

Our forms (on kf.kobotoolbox.org) use csv files. Some of them are master data files that change infrequently. There is however a set of ‘dynamic’ csv files in almost every form, that gets updated via REST API, with every form submission. The form also gets redeployed via the API. This mechanism works very well

However, the API based update of the csv files does not update the same files whose links appear in the Settings->Media section of the kobotoolbox IDE. The version of the csv files in this section is always what was last uploaded manually

Now if a form is modified (its xlsform code that is), or if one of its csv files is to be updated manually (mostly the master data csv files), then the form needs to be redeployed (manually) via the kobotoolbox IDE. In that case, all the csv files that may have gotten modified up till then via API, have to be manually (re)uploaded in the Settings->Media section

Almost every form of ours uses multiple dynamic csv files that get updated with every submission (submission in one form also leads to csv file updates in other forms via the API in most instances, which works very well). Also, many of the ‘master data’ csv files are attached to multiple forms (couple of them apply to all forms). Currently, an update to one these csv files has to be applied manually, which means updating most or all forms csv in their Settings->Media section as described in the previous paragraph. It seems like there are two separate independent locations for csv/media files in the kobotoolbox server?

This is a significant maintenance overhead. Doing this task manually also often leads to ‘human errors’ that eat up additional time. Shouldn’t a csv file update via the API also update that file in the form’s Settings->Media section as well?

Would appreciate any help in this matter. Has anyone else faced this same issue? Is there a solution for this? Can the Kobotoolbox team please look into this problem?

using URL https://kc.kobotoolbox.org/api/v1/forms/ to read form metadata
using URL https://kc.kobotoolbox.org/api/v1/metadata.json for csv file upload
using URL 'KoboToolbox Form Building API for form redeploy

thanks

@aaj, could you share your entire process and the APIs you have been using so that the community could also test and troubleshoot your issue?

Does the REST API not provide an option to upload the file to the settings and to redeploy?
Could you explain, please, why you need such dynamic short-term updates?

@Kal_Lam @wroos

Is there a specific option in the REST API to upload a csv/media file to Settings?

In our case, entry/submission in one form, feeds into another form(s). For instance, an entry made in a skill training youth counselling form is to be made available to another form where a skill training batch is created. This list of trainee candidates is a csv file. A form submission at-times also needs to be fed back into the same form, for a subsequent entry. For instance, moving a skill training batch through multiple stages is done in the same form. So a submission of a training batch update is fed back into the same form by an update to a csv file of that form

Data is being stored in an external (MySQL) database. A form submission is sent to the database via REST services. The updated csv file gets created from the database and is uploaded into Kobotoolbox via REST API. It is a PHP cURL implementation

API code for reading form metadata, to get the list of csv files for a kobotoolbox form (by xform_id);

curl_setopt_array ($ch
					, array(
						CURLOPT_URL => 'https://kc.kobotoolbox.org/api/v1/forms/' . $koboform_id . '.json'
						, CURLOPT_FILE => $fp
						, CURLOPT_HTTPHEADER => array(KOBO_AUTH_TOKEN_STR
													, 'Content-Type: multipart/form-data'
												)
						, CURLOPT_RETURNTRANSFER => true
						, CURLOPT_TIMEOUT => 30
						, CURLOPT_CUSTOMREQUEST => "GET"
						)
					);

$result = curl_exec($ch);

API code to delete a csv file:

curl_setopt_array ($ch
					, array(
						CURLOPT_URL => 'https://kc.kobotoolbox.org/api/v1/metadata/' . $mediafile_id . '.json'
						, CURLOPT_FILE => $fp
						, CURLOPT_HTTPHEADER => array(KOBO_AUTH_TOKEN_STR
													, 'Content-Type: multipart/form-data'
												)
						, CURLOPT_RETURNTRANSFER => true
						, CURLOPT_TIMEOUT => 60
						, CURLOPT_CUSTOMREQUEST => "DELETE"
						)
					);
$result = curl_exec($ch);

API code to upload (an updated copy of) csv file:

curl_setopt_array($ch
					, array(CURLOPT_URL => "https://kc.kobotoolbox.org/api/v1/metadata.json"
							, CURLOPT_FILE => $fp
							, CURLOPT_HTTPHEADER => array(KOBO_AUTH_TOKEN_STR
														, 'Content-Type: multipart/form-data'
													)
							, CURLOPT_RETURNTRANSFER => true
							, CURLOPT_TIMEOUT => 60
							, CURLOPT_POST => true
							, CURLOPT_POSTFIELDS => array('xform' => $koboform_id
															, 'data_value' => pathinfo($file, PATHINFO_BASENAME)
															, 'data_type' => 'media'
															, 'data_file' => curl_file_create($file, 'text/csv', $file)
															, 'from_kpi' => 'true'
													)
							)
$result = curl_exec($ch);

API code to redploy form:

curl_setopt_array ($ch
					, array(
						CURLOPT_URL => $url
						, CURLOPT_FILE => $fp
						, CURLOPT_HTTPHEADER => array(KOBO_AUTH_TOKEN_STR
													, 'Content-Type: multipart/form-data'
												)
						, CURLOPT_RETURNTRANSFER => true
						, CURLOPT_TIMEOUT => 60
						, CURLOPT_CUSTOMREQUEST => "PATCH"
						)
					);
$result = curl_exec($ch);

$ch is the curl session instance
$fp is a text file instance to capture the API call response

This implementation is based on (python, curl) examples available in posts the community forum