Legacy API for file media handling (`/api/v1/metadata`) - bug or intended shut down?

User Support API
1

Hi Kobo Team,

Today I noticed that all our API requests to the legacy endpoint kc.kobotoolobox.org/api/v1/metadata were failing. When I last checked around 7th June, they were working fine with our various web platforms. But now, whenever we try to upload a media file attachment via that endpoint, an error is returned:

HTTP request returned status code 403:
{"detail":"CSRF Failed: Referer checking failed - no Referer."}

I’ve tried this with multiple platforms (we currently have a series of web platforms using these APIs), and get the same error from multiple places. We use the modern v2 KPI for all functions except media attachements - because until the latest release media file attachments could not be managed via the modern KPI. All the requests to the modern KPI still work fine.

So my question is:

Is the legacy /api/v1/metadata endpoint no longer working? Or should it still be active and I’m just encountering a bug?

I know that I’ll eventually need to rewrite our API calls to use the new system for media uploads, but would like to know if I need to do it asap to get our 6+ data platforms up and running again, or if there is an intended period to allow users to migrate over.

Thanks,
Dave

4

Hi @degami,

We did change few things (enforce mimetype validation and project ownership) on this endpoint but none of them should block you to use the legacy api. It should still work.

On my tests, I have been able to upload media files to my project using the legacy interface without any issues.
What kind of authentication process do you use? Basic Auth or Token (FYI: I used the Token authentication for my tests).

1 Like
5

Hi @OlivierL,
Thanks for the info - glad those APIs are still around. I had been using Basic Auth for these requests (I can’t remember exactly why, but back when I was initially writing them I couldn’t get Token auth to work when running from our PHP server). Your note makes me think I should swap out to Token auth and re-test.

I’ll try that and update this thread with the results.

1 Like
6

Ok, I’ve done a few tests. I completed each test with both Basic Auth and Token auth (using the token I got by going to https://kf.kobotoolbox.org/token?format=json

Endpoints that work fine

  1. Access form info via GET request to https://kc.kobotoolbox.org/api/v1/forms?id_string=#####

    • Accessing through Basic Auth and Token Auth both work fine - I get the json response of the form I specify, including a list of all the metadata / media files attached to the form.

  2. Access specific media file info via GET request to https://kc.kobotoolbox.org/api/v1/metadata/###### (the ID of the metadata item).

    • This works fine too - I get a JSON response with the info of that specific media file.

Endpoints that give errors

  1. Add new metadata to the form: POST request to https://kc.kobotoolbox.org/api/v1/metadata with the following data sent as a multi-part form:
"xform": ## form ID ## 
"data_type": "media",
"data_value": "test.csv",
"data_file": ## the actual file  ##
  • This fails to run.
    • With Basic Auth, I get a 403 Forbidden with the response:
{
 "detail": "CSRF Failed: Referer checking failed - no Referer." 
}
  • With Token Auth, I just get a 500 error with the stock error message as html.
500 Error
If this is a significant error that needs to be fixed, please send a support request to support@kobotoolbox.org with information about what triggered this error.
  1. Delete a metadata with a DELETE request to https://kc.kobotoolbox.org/api/v1/metadata/######
  • Exactly the same response as the previous test: when using Basic Auth, I get a 403 forbidden error, and when using Token Auth I get a 500 error.

I’m pretty sure the requests I’m making are correct - at least, they were working before! It may be that what I was doing wasn’t quite right, but got through anyway, and since you’ve updated some of the validation maybe my slightly malformed requests are no longer working! If that’s the case, any suggestions you have would be welcomed.

Hope this information helps - please let me know if there’s anything else you’d like me to try at this end. I know that eventually I’ll have to rewrite these sections using the new KPIs for media file handling, so I should probably do that sooner rather than later!

1 Like
7

Hello,
I’m having the same issue as your #4 using Basic Auth at the following endpoint:

DELETE a record with a DELETE request to:
https://kc.kobotoolbox.org/api/v1/data/[form_ID]/[record_ID]

This returns:
{"detail":"CSRF Failed: Referer checking failed - no Referer."}

This request was working perfectly for years until the last API Update (Release Notes - version 2.021.24)

8

Hi @NTBG,

I’m afraid I didn’t get to the bottom of that error. After a few attempts, I realised it’d probably be quicker and more long-lasting to just switch to the new API endpoints for those requests - which worked rather well.

Might be worth trying - I found it quite simple to switch over, but I guess it totally depends on your setup / scope of app.

In case it helps - For deleting a record with the v2 KPI:

Deletes current submission

DELETE /api/v2/assets/{uid}/data/{id}/

Example

  curl -X DELETE https://[kpi]/api/v2/assets/aSAvYreNzVEkrWg5Gdcvg/data/234/

(taken from https://kf.kobotoolbox.org/api/v2/assets/{form_uid}/data/)

2 Likes
9

Hey @degami
Thank you so much for the update. I’m going to go ahead and migrate to V2!

Thanks again!

2 Likes
10

Hi @NTBG and @degami ,

I am having the exact same issue with the DELETE api requests to the api/v1.
Now, following your conversation here I am now also preparing to migrate to api/v2, but I am running into something unexpected: in my api request I am receiving html back instead of json.

When navigating the api/v2, I receive the expected json back up to here:
https://kf.kobotoolbox.org/api/v2/

But when drilling one layer deeper, even already at
https://kf.kobotoolbox.org/api/v2/assets
I am receiving html back rather than receiving back the json data.

In did include the header Content-Type: application/json,
I am using Postman to navigate the api.

How did you tackle this?

Many thanks in advance!

11

Heya @bert12
Thanks for keeping this topic open.

I’ve been having the exact same issues as you with the V2 API and at this point just gave up and am having staff manually log in to the KoboToolbox website and individually delete data that’s already been imported.

It sure would be nice to have the V1 API fixed so that this just works as it has for years.

Thanks for the response, wish I had better news.

12

Hey @NTBG ,

Have been digging some more and now found that when using token authorization instead of basic authorization with the delete action on endpoints like this:
https://kc.kobotoolbox.org/api/v1/data/[form_ID]/[record_ID]
then it is working well again as before.

Like this:
‘Authorization: Token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx’

Instead of
‘Authorization: Basic xxxxxxxxxxxxxxxxx’

Your token you can find here:
https://kf.kobotoolbox.org/token/?format=json

Hope this helps!

1 Like
13

Thank you @bert12 for sharing this with the community. It should be helpful for a lot of our users.

1 Like
14

@bert12 THANK YOU!!

This trick worked and now the V1 API is working perfectly for me again.

As an aside, this is how I’ve implemented it inside PHP using the CURL extension:

$authorization = "Authorization: Token ". $mytoken;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://kc.kobotoolbox.org/api/v1/data/<formid>/<recordid>');
curl_setopt($ch, CURLOPT_USERPWD,  $mykobouser.":".$mykobopw);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json' , $authorization ));
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "DELETE");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
 $res = curl_exec($ch);

Thanks again for helping me get my application up and running again!

3 Likes