Contemplating the future of KoBoCAT

Development
,
1

Edit: There’s now a survey that we request you take about your use of KoBoCAT. Click here. For detailed technical discussion, it’s fine to continue replying to this discussion topic.

Quick summary: The KoBoToolbox developers would like to know how you use KoBoCAT, especially its API and any parts of its graphical user interface. See below for more detail.

Many folks know that KoBoToolbox comprises, among other things, two Django applications: KPI and KoBoCAT. KPI is what you see when you log into https://kf.kobotoolbox.org/ or https://kobo.humanitarianresponse.info/. It’s where you build your forms; view charts, graphs, and tables; export data; share projects; build a question library; etc.

KoBoCAT, on the other hand, is what you see if you click “Projects (legacy).” You can recognize it by its hostnames: kc.kobotoolbox.org and kc.humanitarianresponse.info. You can also see pieces of KoBoCAT inside <iframe>s when running legacy exports, viewing the picture gallery (for now!), and uploading media files (“Existing Form Files”) to forms.

Most importantly, though, KoBoCAT serves as KoBoToolbox’s implementation of the OpenRosa API. KoBoCAT also provides an additional API (/api/v1) that I know many people use to access their data. We, the small team of programmers who work on KoBoToolbox, have worked steadily to have KPI supersede nearly all user-facing parts of KoBoCAT, and we’ve made good progress with this. We have also made some limited and slower progress bringing KoBoCAT’s /api/v1 features into KPI. We’ve only gotten as far as thinking about what to do with the OpenRosa API.

So, what’s the problem? KoBoCAT is old and large, with a dysfunctional unit test suite. Support for Python 2 ends on January 1, 2020, and the version of Django used by KoBoCAT, Django 1.8, has been unsupported since April 1, 2018. Although change is underway, KoBoCAT and KPI are two distinct Django applications that share a single database. When Django needs a major upgrade, dealing with KPI is not too bad, but the shared database and shared tables within prevent upgrading KPI without KoBoCAT coming along in lockstep. A major Django upgrade for KoBoCAT takes a lot of labor. Upgrading KoBoCAT to Python 3 will take surely more.

The time has come to ask: what does KoBoCAT do for you? I know that every time you deploy a project, you use KoBoCAT. I know that every time you retrieve a blank form with Collect or Enketo, you use KoBoCAT, and every time you make a submission, too. These are the essential OpenRosa API functions that we’ll obviously have to replace if we cease using KoBoCAT. I’m more interested in what’s less obvious to us as KoBoToolbox maintainers. What endpoints in /api/v1 do you use? Is there something in the old “Projects (legacy)” interface that’s important to you? We’ll ascertain some of this with log analysis, but that’s not a replacement for hearing directly from people who use the tool.

I look forward to your responses.

3 Likes
3

FWIW… I/we dont use KoBoCAT in our stack, just KPI - specifically your very nice form builder. But we use something else (a custom register) to actually host our forms and receive submissions.

Have you looked at perhaps leveraging ODK Central as a potential replacement to KoBoCAT for a ‘backend’? Its got a nice REST API too! [no affiliation, well, other than ODK TSC… :wink: ]

2 Likes
4

We run our own on-premise version of Kobo via your Docker containers, and we do not use Kobocat for anything. Intrigued by the progress and status of the API in KPI however, is there a central place I should watch for updates and documentation as to the KPI API?

5

@kevinholl, we’ll announce good news at https://community.kobotoolbox.org/c/announcements. Thanks for your response, and, of course, thanks to you as well, @Xiphware. ODK Central has definitely caught our interest.

In the short term, we may just need to slim down KoBoCAT to accommodate immediate maintenance needs. I’ve pulled a list of all the views that KC exposes with django-extensions’ handy ./manage.py show_urls command, and there are—ahem—352 of them. I’m going to share that list here as a separate post in a moment.

1 Like
6

Creating that big of a Markdown table here was a bit much. Here’s a Google spreadsheet that allows filtering and sorting, and anyone can edit:
[the sheet is no longer available]

7

@jnm I marked a few items, mostly to maintain Briefcase and media/attachment support in xforms, but those seem obvious must-haves.

Maybe we can mark the ones that are already no longer used/needed to create a shorter list? (e.g. /<username>/forms/<id_string>/sms_submission or /about-us/

1 Like
8

Hi,
Thanks for starting this discussion, and for the good responses so far.

We are a small consultancy unit, supporting a few teams using products that link to Kobotools in various ways. We use the Kobotools API (/api/v1) extensively, which unfortunately means we’re currently reliant on the legacy Kobocat system for a lot of our work.

We use mainly the /form and /data endpoints, for downloading data submissions, creating / updating / deleting forms, and for sharing forms with our user teams. Mostly core stuff, to be honest, so I think here I’m really just advocating for something you already have on your to-do list! The main thing for us is the ability to fully interact with the forms, media attachments and data submissions programatically.

I’ve marked those key endpoints in the Google Sheet above.

As we use this API in many projects, we have been working on a NodeJS app as a sort of middleware, that lets us extend some of the API in various ways. It’s up on Github here. It’s still a work-in-progress, although we have been using it in a number of small closed-source projects for our partners.

I’ll definitely be watching in the announcements for news related to the Kobo API, so we can update our tools and move all our projects off the legacy and onto the KPI as soon as possible!

Cheers,
~ D.

1 Like
9

@tinok, sure, feel free to make I just added another column. I’ve really just started to look at the list myself (and added a few “keeps” just as examples for others), but I think the endpoints will generally break down into three categories:

  1. Definitely keep;
  2. Definitely not needed;
  3. More work needed to understand what it does / whether anyone uses it.

@degami, that’s totally understandable, and we won’t remove anything that people are using (especially the core /api/v1/forms and /api/v1/data endpoints) without providing a well-documented alternative and giving lots of notice. At this stage, we’re mainly seeking to reduce the maintenance burden of KoBoCAT’s large code base and minimize its attack surface.

Thanks for contributing to the spreadsheet and for sharing your Node.js project!

2 Likes
10

Apparently people are using the /api/v1/forms/[form pk]/labels endpoint for tagging:

We’ll have to see if this is widespread and decide whether to maintain support for it.

1 Like
11

Of course, but the most strange thing is that, this problem is only repeated on some forms but not on all the forms we have. Whereas, if for the moment there is only me who meets this challenge, it will be sure that others will face it one day

12

Hi @justinienmariot, would you mind writing a little bit about what you’re doing with tags? I’d like to see if our newer KPI application can accomplish what you need.

1 Like
13

Yes @jnm, We use the kobo API for our uses of kobo data so we use the tags as a unique identification system for each form and we are stuck because we can’t tagging one of our forms because of this error returned by kobo server (still 502 bad gateway).
I beg you to help me if you have the opportunity!!!

14

I am curious about the status of KoBoCat and any progress on migrating to Python 3 and Django 2.2. Having been involved in the migration of a smaller Djanog/Python project, I know there is potential for a lot of tedious work. However, given the EOL status of Python 2 and Django 1.8, this seems like important work.

Is the plan still to upgrade KoboCat instead of moving to an alternative like ODK Central?

I’ve seen this github issue related to these upgrades and it looks like there are working tests, which is great. It also looks like there was an effort to pair down code/api endpoints to reduce the upgrade effort.

What is the current LOE estimate for completing this upgrade? Is there other progress that can be shared?

Thanks!

1 Like
15

Thanks for your inquiry.

this seems like important work.

Indeed: we are prioritizing it.

Is the plan still to upgrade KoboCat instead of moving to an alternative like ODK Central?

For now, yes: we’ll continue maintaining KoBoCAT for the foreseeable future. We’ll keep an eye toward possible integration with ODK Central, or perhaps another means of contributing to an OpenRosa server implementation shared by the larger ODK community.

I’ve seen this github issue related to these upgrades and it looks like there are working tests, which is great. It also looks like there was an effort to pair down code/api endpoints to reduce the upgrade effort.

That’s the right issue to watch :+1:; PRs will be linked to it (as two already are) when portions of the work are ready for review. You’re also welcome to check out the development branch being used for the overall Python 3 / Django 2.2 upgrade effort. There’s been a lull in activity over the past month due to other obligations, but it should pick up again next week, once Allow form media URLs · Issue #2674 · kobotoolbox/kpi · GitHub is complete.

If you run your own instances of KoBo and could help with real-world testing of a trimmed-and-updated KoBoCAT, please let us know :smiley:

2 Likes
17

Hi, as an organization we desperately need the view form option in KoboCAT. We had recently using this as selecting a response, the response could have been opened in a new window where the instance ID could be seen as below:

https://kc.humanitarianresponse.info/XX/forms/agZhwRUdCHnRt4CMwwwCEZ/instance#/185687744

Can this be added to KPI or KoboCAT so that we could continue using this.

18

I’ve opened an issue for this:

1 Like
19

Thanks @jnm. Also to note the below - which is a prior request to add route to view a specific submission in enketo rather than modal. Thanks!