Contemplating the future of KoBoCAT

#1

Quick summary: The KoBoToolbox developers would like to know how you use KoBoCAT, especially its API and any parts of its graphical user interface. See below for more detail.

Many folks know that KoBoToolbox comprises, among other things, two Django applications: KPI and KoBoCAT. KPI is what you see when you log into https://kf.kobotoolbox.org/ or https://kobo.humanitarianresponse.info/. It’s where you build your forms; view charts, graphs, and tables; export data; share projects; build a question library; etc.

KoBoCAT, on the other hand, is what you see if you click “Projects (legacy).” You can recognize it by its hostnames: kc.kobotoolbox.org and kc.humanitarianresponse.info. You can also see pieces of KoBoCAT inside <iframe>s when running legacy exports, viewing the picture gallery (for now!), and uploading media files (“Existing Form Files”) to forms.

Most importantly, though, KoBoCAT serves as KoBoToolbox’s implementation of the OpenRosa API. KoBoCAT also provides an additional API (/api/v1) that I know many people use to access their data. We, the small team of programmers who work on KoBoToolbox, have worked steadily to have KPI supersede nearly all user-facing parts of KoBoCAT, and we’ve made good progress with this. We have also made some limited and slower progress bringing KoBoCAT’s /api/v1 features into KPI. We’ve only gotten as far as thinking about what to do with the OpenRosa API.

So, what’s the problem? KoBoCAT is old and large, with a dysfunctional unit test suite. Support for Python 2 ends on January 1, 2020, and the version of Django used by KoBoCAT, Django 1.8, has been unsupported since April 1, 2018. Although change is underway, KoBoCAT and KPI are two distinct Django applications that share a single database. When Django needs a major upgrade, dealing with KPI is not too bad, but the shared database and shared tables within prevent upgrading KPI without KoBoCAT coming along in lockstep. A major Django upgrade for KoBoCAT takes a lot of labor. Upgrading KoBoCAT to Python 3 will take surely more.

The time has come to ask: what does KoBoCAT do for you? I know that every time you deploy a project, you use KoBoCAT. I know that every time you retrieve a blank form with Collect or Enketo, you use KoBoCAT, and every time you make a submission, too. These are the essential OpenRosa API functions that we’ll obviously have to replace if we cease using KoBoCAT. I’m more interested in what’s less obvious to us as KoBoToolbox maintainers. What endpoints in /api/v1 do you use? Is there something in the old “Projects (legacy)” interface that’s important to you? We’ll ascertain some of this with log analysis, but that’s not a replacement for hearing directly from people who use the tool.

I look forward to your responses.

1 Like
#3

FWIW… I/we dont use KoBoCAT in our stack, just KPI - specifically your very nice form builder. But we use something else (a custom register) to actually host our forms and receive submissions.

Have you looked at perhaps leveraging ODK Central as a potential replacement to KoBoCAT for a ‘backend’? Its got a nice REST API too! [no affiliation, well, other than ODK TSC… :wink: ]

2 Likes
#4

We run our own on-premise version of Kobo via your Docker containers, and we do not use Kobocat for anything. Intrigued by the progress and status of the API in KPI however, is there a central place I should watch for updates and documentation as to the KPI API?

#5

@kevinholl, we’ll announce good news at https://community.kobotoolbox.org/c/announcements. Thanks for your response, and, of course, thanks to you as well, @Xiphware. ODK Central has definitely caught our interest.

In the short term, we may just need to slim down KoBoCAT to accommodate immediate maintenance needs. I’ve pulled a list of all the views that KC exposes with django-extensions’ handy ./manage.py show_urls command, and there are—ahem—352 of them. I’m going to share that list here as a separate post in a moment.

#6

Creating that big of a Markdown table here was a bit much. Here’s a Google spreadsheet that allows filtering and sorting, and anyone can edit:

#7

@jnm I marked a few items, mostly to maintain Briefcase and media/attachment support in xforms, but those seem obvious must-haves.

Maybe we can mark the ones that are already no longer used/needed to create a shorter list? (e.g. /<username>/forms/<id_string>/sms_submission or /about-us/

1 Like
#8

Hi,
Thanks for starting this discussion, and for the good responses so far.

We are a small consultancy unit, supporting a few teams using products that link to Kobotools in various ways. We use the Kobotools API (/api/v1) extensively, which unfortunately means we’re currently reliant on the legacy Kobocat system for a lot of our work.

We use mainly the /form and /data endpoints, for downloading data submissions, creating / updating / deleting forms, and for sharing forms with our user teams. Mostly core stuff, to be honest, so I think here I’m really just advocating for something you already have on your to-do list! The main thing for us is the ability to fully interact with the forms, media attachments and data submissions programatically.

I’ve marked those key endpoints in the Google Sheet above.

As we use this API in many projects, we have been working on a NodeJS app as a sort of middleware, that lets us extend some of the API in various ways. It’s up on Github here. It’s still a work-in-progress, although we have been using it in a number of small closed-source projects for our partners.

I’ll definitely be watching in the announcements for news related to the Kobo API, so we can update our tools and move all our projects off the legacy and onto the KPI as soon as possible!

Cheers,
~ D.

1 Like
#9

@tinok, sure, feel free to make I just added another column. I’ve really just started to look at the list myself (and added a few “keeps” just as examples for others), but I think the endpoints will generally break down into three categories:

  1. Definitely keep;
  2. Definitely not needed;
  3. More work needed to understand what it does / whether anyone uses it.

@degami, that’s totally understandable, and we won’t remove anything that people are using (especially the core /api/v1/forms and /api/v1/data endpoints) without providing a well-documented alternative and giving lots of notice. At this stage, we’re mainly seeking to reduce the maintenance burden of KoBoCAT’s large code base and minimize its attack surface.

Thanks for contributing to the spreadsheet and for sharing your Node.js project!

1 Like