Introducing the R package robotoolbox

[dickoa]

Hi @guido167,

Do you mean creating a link to edit via Enketo right ? You can create such link without R ?

Ahmadou

[guido167]

Yes, I mean the links to each submission form (via Enketo) to edit the input. How would I create these?

[guido167]

Hi @dickoa I was wondering if you were able to reproduce this issue?

[dickoa]

Sorry for the delay @guido167, I tried several times on small projects and can’t reproduce it. I’ll love to have more detail and reproduce it. I’ll write to you on dm.
Ahmadou

[guido167]

Hi @dickoa I have sent you a dm to discuss further.

[guido167]

@dickoa unfortunately the above described problem is still persisting.
When I manually download images from the Kobo dashboard (through exporting as zip file) and compare file names with the survey forms these match (uid in file names match with the survey form and photos in the form on the dashboard are the same as the downloaded files). However, when I use the robotoolbox kobo_attachment_download() the file that is downloaded with this file name is completely different. Bottomline: kobo_attachment_download() shuffles media files and does not download the correct media files for the uid’s.

Manually downloading through Kobo export: correct uid/photo combination:

When using kobo_attachment_download(): the same photoname, but a different photo is downloaded!

Unfortunately this would mean I need to change the workflow completely and not rely on robotoolbox… Would you be able to look at this?

[guido167]

An edit of the kobo_attachment_download() and get_attachment_url() functions seemed to have resolved the issue. Attachments are now downloaded and sorted based on uuid to ensure that the correct files are downloaded for each uuid.

get_attachment_url_ <- function(uid) {
  path <- paste0("api/v2/assets/", uid, "/data.json")
  res <- xget(path = path,
              args = list(fields = '["_attachments", "formhub/uuid", "_uuid"]'))
  res <- fparse(res, max_simplify_lvl = "data_frame")
  res <- res$result
  
  attachment_list <- lapply(1:nrow(res), function(i) {
    attachment_df <- res[["_attachments"]][[i]]
    attachment_df[["_uuid"]] <- res[["_uuid"]][i]
    return(attachment_df)
  })
  return(attachment_list)
}

kobo_attachment_download2 <- function(uid, folder) {
  if (!dir.exists(folder)){
    abort(paste(folder, "folder does not exist, create it first!"),
          call = NULL)}
  attachments <- get_attachment_url_(uid = uid)
  bool <- sapply(attachments, is.null)
  path <- character()
  if (any(!bool)) {
    urls <- attachments[!bool] |>
      list_rbind() |>
      mutate(id = .data$instance,
             url = .data$download_url,
             fname = basename(.data$filename),
             fname_id = paste0(.data$id, "_", .data$fname),
             uuid = .data$`_uuid`,
             path = file.path(folder, `_uuid`, .data$fname),
             .keep = "none") |>
      distinct()
    
    headers <- list(Authorization = paste("Token",
                                          Sys.getenv("KOBOTOOLBOX_TOKEN")))
    
    
    for(i in 1:length(unique(urls$uuid))){
      sub <- urls[urls$uuid == unique(urls$uuid)[i],]
      if(nrow(sub)>0){
        if(dir.exists(file.path(folder, unique(urls$uuid)[i])) == F){
          dir.create(file.path(folder, unique(urls$uuid)[i]))
          
          reqs <- lapply(sub$url, function(url) {
            req <- HttpRequest$new(url,
                                   headers = headers)
            req$retry("get",
                      times = 3,
                      retry_only_on = c(500, 502, 503),
                      terminate_on = 404)
          })
          res <- AsyncQueue$new(.list = reqs,
                                bucket_size = Inf,
                                sleep = 0.05)
          
          res$request()
          cond <- res$status_code() >= 300L
          if (any(cond)) {
            msg <- res$content()[cond]
            abort(error_msg(msg[[1]]),
                  call = NULL)
          }
          walk2(res$content(), sub$path, \(x, y) writeBin(x, con = y))
        }
      }
    }
  }
}

[alex_lantitra]

Hi Ahmadou, were you able to fix the issue of the missing submission_uuid in the repeat groups? I am experiencing the same issue in the version that I downloaded from the CRAN repository.
Thanks!
Alex

[agotossou]

I am getting an error using the robotoolbox package in R
library(robotoolbox)

token ← kobo_token(
username = “agotossou”,
password = “xxxxxxxxxxxxxxxxxxx”,
url = “KoboToolbox,
overwrite = TRUE
)

kobo_setup(
url = “KoboToolbox,
token = token
)

Error in .deserialize_json(json = json, query = query, empty_array = empty_array,  : 
  TAPE_ERROR: The JSON document has an improper structure: missing or superfluous commas, braces, missing keys, etc.

[agotossou]

This is the url I used url = “KoboToolbox

[Sulaiman_Anwary]

Hi @dickoa could you please assist in finding the error message, it seems something changed in audit function

```r
robotoolbox:: kobo_audit(x = asset_id, progress = TRUE)

Checking audit data availability [3s]
Error in `select()`: !
Can’t select columns that don’t exist.
Column `instance` doesn’t exist.
Run `rlang::last_trace()` to see where the error occurred.

Downloading audit data [677ms]
Processing audit data

[dickoa]

robotoolbox 1.5-RC1 is here!

First, I want to apologize for the long delay since the last update. I know many of you have been waiting patiently, and I truly appreciate your patience and continued support.

I’m happy to announce that robotoolbox 1.5-RC1 is now available for testing, with a CRAN release planned for early February.

What’s new?

Breaking changes:

• Default pagination limit reduced from 30,000 to 1,000 to comply with the recent KoboToolbox API changes

• Attachment filenames now include the attachment UID for uniqueness ({att_uid}_{filename})

New features:

• Session-level caching for form metadata — significantly improves performance for repeated operations

• Language switching without re-downloading data:

  • kobo_lang_set(data, asset, lang) — instantly switch variable and value labels to a different language

  • kobo_lang_get(data, asset) — detect the current language applied to your dataset

  • Works with both simple data.frame and complex dm objects (nested forms)

Bug fixes:

• Fixed kobo_audit() and kobo_attachment_download() to handle API changes where the instance field was removed from _attachments

• Fixed kobo_attachment_download() overwrite = FALSE check

• Improved error handling for non-JSON API error responses

• Various other fixes (see full changelog below)

Note for private server users: If your server allows higher limits, you can still use kobo_data(asset, page_size = 30000) for better performance on large datasets.

How to install

# Install development version from GitLab
remotes::install_gitlab("dickoa/robotoolbox")
pak::pkg_install("dickoa/robotoolbox")

Feedback welcome!

Please test it on your projects and report any issues on GitLab or GitHub. Your feedback before the CRAN submission will be invaluable.

Thank you all for your patience

[dickoa]

Hi @Sulaiman_Anwary
Sorry for the delay in getting back to you. Can you try the latest version to check if it’s working now?

[ileboo]

@dickoa

kobo_asset_list()

Hi @dickoa I like using robotoolbox, although it has been sometime sinc ei last used it. I have encountered a challenge since I cannot get more than 100 submissions. I understand this is a problem with the KOBO api. Do you have any way around this

[dickoa]

Hi @ileboo

It should be fixed in the dev version. Could you please try the latest version on GitHub?

# install.packages(“pak”)
pak::pkg_install(“dickoa/robotoolbox”)

Check that you have the latest version. As of today, it’s 1.5.0 (do packageVersion("robotoolbox") to check)

[ileboo]

Let me try and report feedback

[ileboo]

@dickoa

pak::pkg_install(“dickoa/robotoolbox”)

@dickoa it has worked. Thanks

[dickoa]

I’m glad it worked!