API call sometimes forgets authentication?

I’m trying to access the data from the surveys through some simple VB and run into a strange problem.

When using the very simple code:

Dim httpReq As New MSXML2.ServerXMLHTTP60

Call httpReq.Open(“GET”, “https://kc.kobotoolbox.org/api/v1/data”, False, “my_userid”, “my_password”) <== these are obviously not my real username & password

Debug.Print httpReq.responseText

When I run this line of code,

[{“id”:55229,“id_string”:“aRk84dxcTKZK5wsZegXC9d”,“title”:“JAS - Joint Activity Survey”,“description”:“JAS - Joint Activity Survey”,“url”:“https://kc.kobotoolbox.org/api/v1/data/55229”}]

Which looks correct to me. So I know my data is under 55229.

However when I run the exact same code, but with the number added:

Call httpReq.Open(“GET”, “https://kc.kobotoolbox.org/api/v1/data/55229”, False, “my_userid”, “my_password”)

It returns:

{“detail”:“Not found.”}

However, when I run the same from any webbrowser (and have authenticated during the first request), the second request returns the proper dataset.

If I logout from the browser and run the 2nd request without logging in, I get the “detail not found” error message, so it seems to me related to authentication, but that doesn’t make too much sense as the first request seems to work just fine.

I’ve run this kobotoolbox.org and on humanitarianresponse.info with exactly the same result.

I’ve searched around, but didn’t see anyone having this problem. Any help is appreciated.

I didn’t find a solution for the problem, but I’ve found a work around by using token authentication.

I’m now able to get the dataset.

In case someone runs into the same issue, a simplified version of the VBA code below:

Dim httpReq As New MSXML2.ServerXMLHTTP60

Call httpReq.Open(“GET”, “https://kc.humanitarianresponse.info/api/v1/data/67602?format=xml”, False)

httpReq.setRequestHeader “Authorization”, “Token ”

httpReq.send

Debug.Print httpReq.responseText

···

On Friday, 20 May 2016 16:23:14 UTC+1, Gerard Schut wrote:

I’m trying to access the data from the surveys through some simple VB and run into a strange problem.

When using the very simple code:

Dim httpReq As New MSXML2.ServerXMLHTTP60

Call httpReq.Open(“GET”, “https://kc.kobotoolbox.org/api/v1/data”, False, “my_userid”, “my_password”) <== these are obviously not my real username & password

Debug.Print httpReq.responseText

When I run this line of code,

[{“id”:55229,“id_string”:“aRk84dxcTKZK5wsZegXC9d”,“title”:“JAS - Joint Activity Survey”,“description”:“JAS - Joint Activity Survey”,“url”:“https://kc.kobotoolbox.org/api/v1/data/55229”}]

Which looks correct to me. So I know my data is under 55229.

However when I run the exact same code, but with the number added:

Call httpReq.Open(“GET”, “https://kc.kobotoolbox.org/api/v1/data/55229”, False, “my_userid”, “my_password”)

It returns:

{“detail”:“Not found.”}

However, when I run the same from any webbrowser (and have authenticated during the first request), the second request returns the proper dataset.

If I logout from the browser and run the 2nd request without logging in, I get the “detail not found” error message, so it seems to me related to authentication, but that doesn’t make too much sense as the first request seems to work just fine.

I’ve run this kobotoolbox.org and on humanitarianresponse.info with exactly the same result.

I’ve searched around, but didn’t see anyone having this problem. Any help is appreciated.

Hi Gerard,

I’m glad you found a workaround! I am curious, though, about the trouble you were having with username/password authentication.

When using the API, you do have to send the credentials with each request, whether you’re using username/password or token authentication. By contrast, a web browser perusing the API creates a session the first time you log in, where authentication details are stored in cookies etc., saving you the hassle of logging in again every time you load a different page.

I don’t have a VBA environment handy, but here’s a test using the program “curl” and an account whose username is “apitest” and password is “the password”:

  1. First, querying without a specific ID:

$ curl -u ‘apitest:the password’ https://kc.kobotoolbox.org/api/v1/data

[{“id”:55576,“id_string”:“ahudYAcPM9uivNnhv7N6Be”,“title”:“single question form”,“description”:“single question form”,“url”:“https://kc.kobotoolbox.org/api/v1/data/55576”}]

  1. Next, querying for a specific ID (55576), and still providing the username and password:

$ curl -u ‘apitest:the password’ https://kc.kobotoolbox.org/api/v1/data/55576

[{“_notes”:,“meta/instanceID”:“uuid:4621196a-d851-44c8-9eca-67fd07f088c8”,“end”:“2016-05-23T18:25:44.000-04:00”,“_submission_time”:“2016-05-23T22:25:49”,“_uuid”:“4621196a-d851-44c8-9eca-67fd07f088c8”,“_bamboo_dataset_id”:“”,“_tags”:,“_attachments”:,“start”:“2016-05-23T18:25:35.000-04:00”,_submitted_by,_geolocation],“xform_id_string”:“ahudYAcPM9uivNnhv7N6Be”,“what_s_your_favorite_color”:“chartreuse”,“_status”:“submitted_via_web”,_id,“version”:“1404633”,“formhub/uuid”:“f6ad759c000a46cc93b0745bba155544”}]
You’re right that an unauthenticated request leads to the “Not found” error you received, e.g.

$ curl https://kc.kobotoolbox.org/api/v1/data/55576

{“detail”:“Not found.”}

The maintainer of the software library responsible for that behavior discusses his rationale at DjangoObjectPermissions raises 404 where should raise 403 · Issue #1439 · encode/django-rest-framework · GitHub.

Still, what’s curious to me is that your request:

Call httpReq.Open(“GET”, “https://kc.kobotoolbox.org/api/v1/data/55229”, False, “my_userid”, “my_password”)

…does specify the username and password, and yet it still failed. If that continues to happen, please let us know so that we can investigate further.

Thanks,

John Milner

Developer, KoBoToolbox

···

I didn’t find a solution for the problem, but I’ve found a work around by
using token authentication.

I’m now able to get the dataset.

In case someone runs into the same issue, a simplified version of the VBA
code below:

Dim httpReq As New MSXML2.ServerXMLHTTP60
Call httpReq.Open(“GET”,
https://kc.humanitarianresponse.info/api/v1/data/67602?format=xml”, False)
httpReq.setRequestHeader “Authorization”, “Token ”
httpReq.send

Debug.Print httpReq.responseText

I’m trying to access the data from the surveys through some simple VB and
run into a strange problem.

When using the very simple code:

Dim httpReq As New MSXML2.ServerXMLHTTP60
Call httpReq.Open(“GET”, “https://kc.kobotoolbox.org/api/v1/data”, False,
“my_userid”, “my_password”) <== these are obviously not my real username
& password
Debug.Print httpReq.responseText

When I run this line of code,
[{“id”:55229,“id_string”:“aRk84dxcTKZK5wsZegXC9d”,“title”:“JAS - Joint
Activity Survey”,“description”:“JAS - Joint Activity
Survey”,“url”:“https://kc.kobotoolbox.org/api/v1/data/55229”}]

Which looks correct to me. So I know my data is under 55229.

However when I run the exact same code, but with the number added:

Call httpReq.Open(“GET”, “https://kc.kobotoolbox.org/api/v1/data/55229”,
False, “my_userid”, “my_password”)

It returns:
{“detail”:“Not found.”}

However, when I run the same from any webbrowser (and have authenticated
during the first request), the second request returns the proper dataset.

If I logout from the browser and run the 2nd request without logging in, I
get the “detail not found” error message, so it seems to me related to
authentication, but that doesn’t make too much sense as the first request
seems to work just fine.

I’ve run this kobotoolbox.org and on humanitarianresponse.info with exactly
the same result.

I’ve searched around, but didn’t see anyone having this problem. Any help
is appreciated.


You received this message because you are subscribed to the Google Groups “Kobo Users” group.
To unsubscribe from this group and stop receiving emails from it, send an email to kobo-users+...@googlegroups.com.
To post to this group, send email to kobo-...@googlegroups.com.
Visit this group at https://groups.google.com/group/kobo-users.
For more options, visit https://groups.google.com/d/optout.

On May 20, 2016 at 11:23AM EDT Gerard Schut gera...@gmail.com wrote:

On May 23, 2016 at 10:16AM EDT Gerard Schut gera...@gmail.com wrote:

Any other questions? Visit support.kobotoolbox.org[[93b26426b69bb179b748affe40b335cfef074143-705150624]]

Hi John,

First of all, thank you very much for the reply. It’s much appreciated.

I’ve done some more testing: I’ve installed curl and when I follow the same steps as previously described, it works all as it should. So that’s the good news: I can reproduce what you write and it’s not some proxy in between that is causing the problem I’m experiencing.

I did some more digging and find this article: https://zanstra.home.xs4all.nl/inTec/ServerXMLHTTP.htm

It seems that the MSXML2 object is a bit buggy and it doesn’t send the username & password supplied in the parameters until the server sends back a 401. The only solution seems to be sending the username and password in the function as well as in the header.

I adjusted the code to this:

Dim httpReq As New MSXML2.ServerXMLHTTP60

Call httpReq.Open(“GET”, “https://kc.humanitarianresponse.info/api/v1/data/67602”, False, “username”, “password”)

httpReq.setRequestHeader “Authorization”, "Basic " & Base64Encode(“username:password”)

httpReq.send

Debug.Print httpReq.responseText

and now I get back the response that I was expecting (the same as in curl).

The link you send is exactly where the problem is:

api/v1/data → This one returns a 401 when no username & password supplied

api/v1/data/12345 → This one returns “nothing found” when no username & password supplied

Sorry, but that it not very consistent. I’m happy I got it to work because you pointed me in the right direction, but you have to agree that the function that gets the data set should work similar to it’s “parent”.

I hope I provided enough details that this maybe considered to be changed in a future release.

Kind regards, Gerard

···

On Tuesday, 24 May 2016 04:59:31 UTC+1, KoBoToolbox Support wrote:

Hi Gerard,

I’m glad you found a workaround! I am curious, though, about the trouble you were having with username/password authentication.

When using the API, you do have to send the credentials with each request, whether you’re using username/password or token authentication. By contrast, a web browser perusing the API creates a session the first time you log in, where authentication details are stored in cookies etc., saving you the hassle of logging in again every time you load a different page.

I don’t have a VBA environment handy, but here’s a test using the program “curl” and an account whose username is “apitest” and password is “the password”:

  1. First, querying without a specific ID:

$ curl -u ‘apitest:the password’ https://kc.kobotoolbox.org/api/v1/data

[{“id”:55576,“id_string”:“ahudYAcPM9uivNnhv7N6Be”,“title”:“single question form”,“description”:“single question form”,“url”:“https://kc.kobotoolbox.org/api/v1/data/55576”}]

  1. Next, querying for a specific ID (55576), and still providing the username and password:

$ curl -u ‘apitest:the password’ https://kc.kobotoolbox.org/api/v1/data/55576

[{“_notes”:,“meta/instanceID”:“uuid:4621196a-d851-44c8-9eca-67fd07f088c8”,“end”:“2016-05-23T18:25:44.000-04:00”,“_submission_time”:“2016-05-23T22:25:49”,“_uuid”:“4621196a-d851-44c8-9eca-67fd07f088c8”,“_bamboo_dataset_id”:“”,“_tags”:,“_attachments”:,“start”:“2016-05-23T18:25:35.000-04:00”,_submitted_by,_geolocation],“xform_id_string”:“ahudYAcPM9uivNnhv7N6Be”,“what_s_your_favorite_color”:“chartreuse”,“_status”:“submitted_via_web”,_id,“version”:“1404633”,“formhub/uuid”:“f6ad759c000a46cc93b0745bba155544”}]
You’re right that an unauthenticated request leads to the “Not found” error you received, e.g.

$ curl https://kc.kobotoolbox.org/api/v1/data/55576

{“detail”:“Not found.”}

The maintainer of the software library responsible for that behavior discusses his rationale at https://github.com/tomchristie/django-rest-framework/issues/1439#issuecomment-36012573.

Still, what’s curious to me is that your request:

Call httpReq.Open(“GET”, “https://kc.kobotoolbox.org/api/v1/data/55229”, False, “my_userid”, “my_password”)

…does specify the username and password, and yet it still failed. If that continues to happen, please let us know so that we can investigate further.

Thanks,

John Milner

Developer, KoBoToolbox

On May 23, 2016 at 10:16AM EDT Gerard Schut gera...@gmail.com wrote:

I didn’t find a solution for the problem, but I’ve found a work around by
using token authentication.

I’m now able to get the dataset.

In case someone runs into the same issue, a simplified version of the VBA
code below:

Dim httpReq As New MSXML2.ServerXMLHTTP60
Call httpReq.Open(“GET”,
https://kc.humanitarianresponse.info/api/v1/data/67602?format=xml”, False)
httpReq.setRequestHeader “Authorization”, “Token ”
httpReq.send

Debug.Print httpReq.responseText
On May 20, 2016 at 11:23AM EDT Gerard Schut gera...@gmail.com wrote:

I’m trying to access the data from the surveys through some simple VB and
run into a strange problem.

When using the very simple code:

Dim httpReq As New MSXML2.ServerXMLHTTP60
Call httpReq.Open(“GET”, “https://kc.kobotoolbox.org/api/v1/data”, False,
“my_userid”, “my_password”) <== these are obviously not my real username
& password
Debug.Print httpReq.responseText

When I run this line of code,
[{“id”:55229,“id_string”:“aRk84dxcTKZK5wsZegXC9d”,“title”:“JAS - Joint
Activity Survey”,“description”:“JAS - Joint Activity
Survey”,“url”:“https://kc.kobotoolbox.org/api/v1/data/55229”}]

Which looks correct to me. So I know my data is under 55229.

However when I run the exact same code, but with the number added:

Call httpReq.Open(“GET”, “https://kc.kobotoolbox.org/api/v1/data/55229”,
False, “my_userid”, “my_password”)

It returns:
{“detail”:“Not found.”}

However, when I run the same from any webbrowser (and have authenticated
during the first request), the second request returns the proper dataset.

If I logout from the browser and run the 2nd request without logging in, I
get the “detail not found” error message, so it seems to me related to
authentication, but that doesn’t make too much sense as the first request
seems to work just fine.

I’ve run this kobotoolbox.org and on humanitarianresponse.info with exactly
the same result.

I’ve searched around, but didn’t see anyone having this problem. Any help
is appreciated.


You received this message because you are subscribed to the Google Groups “Kobo Users” group.
To unsubscribe from this group and stop receiving emails from it, send an email to kobo-users+...@googlegroups.com.
To post to this group, send email to kobo...@googlegroups.com.
Visit this group at https://groups.google.com/group/kobo-users.
For more options, visit https://groups.google.com/d/optout.

Any other questions? Visit support.kobotoolbox.org[[93b26426b69bb179b748affe40b335cfef074143-705150624]]