Download via API limited to 100 observations

User Support
,
1

Dear community,

I have been using robotoolbox package in R from @dickoa to download data via API for a long time. Recently the same R scripts will only download 100 observations, regardless which kobotoolbox project I wish to download. Can anyone assist, has there been an API change?

2

What API(s) are you using to do this?

Note, /api/v2/assets/{uid}/data/ is now limited to 1000 records. See KoboToolbox Primary API

3

Hi @Tumaini ,

I hope all is well, and thanks for using robotoolbox. Indeed, the page size was capped at 100 max, I had to change the package default recently. The current robotoolbox should work but it’ll be slightly slower because of the increased number of request.

# install.packages("pak")
pak::pkg_install(“dickoa/robotoolbox”)

I’ll push it to CRAN soon.

I do have a question for @Xiphware and/or someone from the core team: Is there a way to know via a request the default page size and max page size? I understand that I might vary from one server to the next, and I want to be able to give more flexibility instead of having 100/1000 by default across different servers.

4

Hi @Xiphware

I also thought so, but someone using the kf server told me that the limit was set at 100. I wonder I could safely set the default limit to 1000 (maximum allowed).

5

Thanks for the response @dickoa indeed after downloading the latest, I could download all 800 rows as previously, the only difference is the new version renamed my select_multiple variables adding _1 to each, something it never did before, but not an issue.

So is there a limit I will hit at some point, I am currently at 830 observations, are there any surprises awaiting me at some threshold?

Thanks again for the brilliant package.

6

Is there a way using robotoolbox to download data within a fixed interval of time e.g last 14 days?

7

Yes, you can. I added a new ‘query’ parameter you could use to add custom MongoDB query. In your cases, get cutoff date to match your 14 last days

days ← 14
cutoff ← format(Sys.time() - as.difftime(days, units = “days”), “%Y-%m-%dT%H:%M:%S”)
qry ← sprintf(‘{“_submission_time”: {“$gte”: “%s”}}’, cutoff)
df ← kobo_data(uid, query = qry)

In general, all the MongoDB queries supported by KPI should work.

1 Like
8

In theory, no limit. The package will send multiple request if the size of dataset is greater than the page size (default to 100 and max at 1000; robotoolbox use 1000). For a 830 observation, it’s just one pass, but if you end up with 3,250 observations, robotoolbox will send 3 requests and combine them for you. I’ll investiage the “_1” suffix in the select_multiple.

9

Hi @dickoa,

Thank you for the quick update on the robotoolbox package. I just ran into the same issue, and the download is working properly now. The “_1” suffix issue in the select_multiple seemed to be fixed as well, but when I use the to_factor function, I still get the actual codes/names preceded by the underscore sign or just a <empty> mention. Sorry to bring up this issue here (not really related to KobolToolbox at this point).

Thank you!

10

Hi @Elihou

Can you share the output? Particularly the <empty> just to have a sense of the issue. select_multiple are not labelled columns like select_one therefore to_factor/to_character shouldn’t work, but I would love to understand better.
In any case, you do have the option to read the select_multiple labels instead of the values directly through select_multiple_labels = TRUE in kobo_data

11

Presently no. But I’ve opened a ticket to devise a means to expose the paging parameters default and maximum in a manner that is discoverable by clients like your robotoolbox.

So please be patient, and watch this space. :slightly_smiling_face:

1 Like
12

Hi @dickoa ,

Sorry for the confusion. I was talking about select_one questions, not select_mutliple. In the example below, Q001 refers to my region variable, which has 4 categories in the XLSForm. But when I check the attributes, here is what I got:

image
image1301×182 3.75 KB

However, the issue isn’t related to your update. I just tried the CRAN version (1.4.0) and got the same outputs. The kobo_lang function lists “Français” and “French”, although it is not a multi-survey form (standard label, hint, and constraint_message in the XLSForm and no default_language in the settings sheet).

I also tried adding each “Français” and “French” as “lang” arguments to the kobo_data function, but it did not solve the problem. Still wondering where the issue might be, but everything used to work fine on my previous projects.

Thank you!

13

Hi @Elihou,
I’ll need more info to be able to reproduce it, Is that the case for all select_one of this project? Would it be possible to have all the versions of the XLSForms you used or the latest version of the questionnaire ?
You can also try to just use the latest version
kobo_data(…, all_versions = FALSE)

14

Hi @dickoa ,

Thanks so much for your help! I now understand the problem. It was due to having multiple versions of the XLSForm during testing. The initial XLSForm versions included different languages and were made by different team members—some used French, others used Français, and some used English for another form. The mistake was to continue collecting data with the final deployed version instead of deploying a new project with the stable, final version. Although I can’t share the exact work forms, I’d be happy to prepare three or four versions if you’d like to try reproducing the issue. I really appreciate your support and want to thank you for the wonderful package!

1 Like
15

Its OK to discuss Kobo related things on the Forum, including 3rd party Kobo tooling like Robotoolbox. But perhaps just start a new thread if you want to continue this discussion, since its diverged from the original topic.

16

I also working on the API that help us to give me answer auto to the user and get the right result

17

The KoboToolbox API now defaults to 100 observations per request.

To get more data, either:

  • Use the query parameter to filter subsets (e.g., by _submission_time)

  • Implement pagination with limit and skip to loop through all records

It’s not a bug just the API limit.

18

Thank you for the update. Your package is extremely useful for my work!

19

I’m glad you find it useful!
A new version was released on CRAN (1.6.2).
You can install using `install.packages(“robotoolbox”).

21

hello mr.khan,

so u mean to say, we cant use API’s as before ?
I particularly using API for connecting it to Power bi for reports as its easy to manage there. I notices that i was stuck to limited data..

Can u pls guide here. How can i extend for grasping more data in my power bi report ?