Proposed feature: KoBo P-coding

Suggestion Box New Features
, , ,
1

General goal: Provide the COD P-code for any KoBo record providing geographical coordinates with a geopoint question.

User stories:

  1. A KoBo survey is gathering locations via a geopoint question for a survey of emergency food distributions. The user wants to assign the correct P-code (assigned to polygons in the relevant Administrative Boundaries Common Operational Database or COD-AB) to each KoBo record, to facilitate blending with other humanitarian response data.
  2. Another KoBo survey is also gathering locations the same way, this time for a security assessment. This user wants to assign the correct P-code to each point location mainly to catagorize the responses according to relevant administrative units.
  3. Yet another KoBo survey is used to record field visits by local staff of various NGOs that are members of the Education cluster in a crisis country. The administrator wants to assign the correct P-code, but only at administrative level 1, to each visit record to simplify their reporting to the OCHA Who Does What Where operational database.

All these users want to easily obtain current P-codes but may want to specify a particular administrative level, or request P-codes for all levels, or the lowest available level. They expect the P-codes to be inserted into columns in the tabular KoBo output.

These users currently need to import their KoBo output tables into a Geographic Information System and overlay them with the COD-AB shapefiles to determine the P-codes. Alternatively they provide their data without P-codes, and anybody seeking to combine or analyze their results has to do it. Sometimes more than one person will do the same P-coding task, because it wasn’t integrated early in the workflow. All users want the most up to date and definitive P-code results, without having to know where they come from or whether they have been updated.

What can you contribute to making this feature a reality?

More and more COD-ABs (84 at this writing, prioritizing the most vulnerable countries) are being configured for P-coding with an API (developed by ITOS in support for OCHA) that returns an output like this, for input coordinates latitude 18.281165000, longitude -73.350747400:

[{“admin0Name_ref”:“Null”,“admin0Pcode”:“HT”,“admin0Name_local”:“Ayiti”,“admin1Name_ref”:“Null”,“admin1Pcode”:“HT08”,“admin1Name_local”:“Grandans”,“admin2Name_ref”:“Null”,“admin2Pcode”:“HT0813”,“admin2Name_local”:" “,“admin3Name_ref”:“Null”,“admin3Pcode”:“HT0813-01”,“admin3Name_local”:” ",“admin4Name_ref”:null,“admin4Pcode”:null,“admin4Name_local”:null,“status”:“OK”}]

ITOS lookup API documentation

Can you sketch out how you think this should look/work in practice?

I imagine forms (with a geopoint question for capturing coordinates) containing calculate questions. The calculation field would contain some sort of keyword that did nothing during data collection but triggered the KoBo server to populate the field with the correct P-code. BUT THERE ARE PROBABLY BETTER WAYS.

3

Hi @hayttom_ocha, thank you for this suggestion :slightly_smiling_face: do you know if the use case of this feature would also require offline capability? An unfortunate consequence of pulling data from external sources is that a request would need to be made over an internet connection.

1 Like
4

Hi Josh, No, there would not be any need for acquiring the P-codes while offline. They would only be needed in conjunction with the normal tabular output - so it can be integrated with any and all sorts of other humanitarian information. T.

5

Hi @hayttom_ocha, here’s a proposed workaround for this without making changes to the KoBo codebase to allow for external API requests.

  1. Create a simple form with a geopoint question and extra hidden question:
type name label
geopoint location Where are you?
hidden p_code
  1. Create a Cloudflare Worker that consumes POST data
  2. Copy the Worker URL and use it to set up a REST Service in KoBo
  3. Extend the Worker to parse the submission JSON for location coordinates and use the ITOS P-Code API to get the code
  4. Extend the Worker again to make a PATCH request to /api/v2/assets/<asset_uid>/data/bulk/ to update the p_code question value with the code returned from the API

This may however be more suitable to an AWS Lambda Function, following the same general workflow. We are currently using Lambda and API Gateway with SES to send messages to Zulip when submissions come in to a deployed form.

(Disclaimer: I haven’t had the opportunity to fully test this, but in theory it should work)

1 Like
6

Hi Josh, this looks impressive. It cannot be as user-friendly as I would like but I will try to test it.

1 Like
7

Hi @hayttom_ocha, there is also the easier option of periodically running a script to update submissions with a P-Code — in a similar fashion to the above but without the REST Services and external integration aspect:

  • Have a bunch of submissions come in
  • Pull submissions from JSON API, perhaps filtering by _submission_time so that each run of the script only checks the latest submissions since the last update
  • For each submission:
    • Get the P-Code from to ITOS API
    • Send a PATCH request to /data/bulk with code

You can then run this with a cron job or Lambda Function again so it’s done behind the scenes.

1 Like
8

Thanks once again, @Josh!

So is there no hope of building what I will call an ‘ITOS API caller’ into your systems so that users can get P-codes from KoBo forms without knowing or doing anything special? That’s the type of user we’re hoping to reach. Users who don’t yet know they even want P-codes…

9

Hi @hayttom_ocha,

We are currently working on support for coding/analysis forms that are a meta-survey of sorts which will be integrated with the upcoming NLP/transcription/translation features. The mechanism for automatic machine transcription/translation (requests to an external service), etc. could be extended to allow for this. So to answer your question, there is hope.

1 Like
10

Hi @Josh, OK, I will keep hoping!

2 Likes