Kobo for OpenStreetMap

These are a set of improvements to Kobo which would enable wider use for data collection in OpenStreetMap.

These suggestions come from my technical advising to Map Kibera, who do community mapping in OpenStreetMap for informal settlements in Nairobi, as well as data collection and training across Kenya. This is effective, but there are interesting ways that the methodology of data collection and structure in OpenStreetMap differs from typical surveys: OSM is about physical infrastructure; data collection exercises frequently cover many different types of features; and OSM represents features with tags one or several key/value pairs.

Map Kibera’s current process has form creation and aggregation in Kobo, and data collection in the latest ODK (using some of the great offline map widget features). Note, this is only creating new data in OpenStreetMap, not updating existing data in OSM. In a question’s settings, “Data Column Name” is set to a Tag’s “key”, and the “Value” of the answers set to the Tag’s “value”. Skip logic is used to only present questions relevant to the feature type. To prepare data for upload to OpenStreetMap and work around some current limitations in Kobo, they transform the data using HXL Proxy, and then bring into the JOSM editor for validation and upload to OpenStreetMap.

Ideally, Kobo could output data immediately ready for use in OSM, with the necessary “transformations” defined in the question settings panel in the form editor. Here’s a breakdown of those.

Simple form enhancements

To start, there are a few (seemingly) simple improvements which could help.

  • There are columns that are irrelevant for OSM, both as part of the survey and as metadata (start, username, deviceid). For OSM output, drop the metadata and have a way to filter out questions.
  • Kobo has a length limitation of in data values (14 characters I believe). Some tag’s values are longer than this (ex “amenity=place_of_worship” for a religious facility).
  • “Data Column Name” prohibits the “:” character. “:” is frequently used in Tag keys

Single question, multiple tags

A little more complicated are cases when a single question in a form may correspond to different Tag key, depending on the answer given. This is due to mismatch between what’s the most obvious way to construct a form, and how OSM represents particular features. For example, in Map Kibera’s most recent survey, there is one question about water features (“What kind of water feature is it?”) with answers that range from dam, to drinking fountain, to ponds and irrigation features, which all have different Tag keys. And in some cases (in the case of dams), to represent a feature requires setting OSM tags.

To address this in Kobo, I think would take adding settings for each value. This could be a simple check box adjacent to the value field which if clicked would treat the value as a Tag (or multiple tags). The Tag key/value pairs would be of the form “natural=spring” or for multiple. “water_way=dam;dam:type=earth_dam”.

Multiple questions, single tag key

A related issue is that some OSM Tag keys cover a huge number of different kinds of features, particularly “amenity” which can represent everything from a school to a hospital to a bar. This means that the “amenity” data column name is used multiple times throughout the form, and Kobo automatically assigns a suffix (“amenity_001”).

Solution could be an additional question setting that sets an OSM Tag key for the question, different from the data column name.

Export formats

Export format of GeoJSON and/or OSM XML will allow direct use in JOSM. Since these are not tabular data formats, submissions that have a blank answer can and should be omitted. Also since this might include either point or linear geographic features, need to ensure the output is formatted correctly.

What would also help is the ability to break up the export into multiple files by the answer to a question, or metadata like username. This would enable each data collector to separately upload their edits to OSM.

OSM Mode Toggle

Realize that the additional OSM related settings may crowd the interface and be irrelevant to many Kobo users. One way to manage this would be a survey wide setting that controls whether OSM settings are visible, which would default to off.

Photographs

OSM has tags to associate images with a feature, which should be publicly viewable. Ideally there would be a simple way to publish images to another service for hosting, and record the link of those images in the survey result. This requirement isn’t only OSM related, so think it should be considered separately. It might not be something to implement directly in Kobo, but an external tool that uses Kobo’s API.

Blue Sky

I don’t think these are immediately addressable or most needed, but noting as longer term requirements to consider in the future. These may in fact be more related to ODK ultimately.

Ability to update existing OSM data. This would require having the OSM ID for the feature, and then being able to see the existing tags in order to update them. This is a challenging but in the long run, extremely useful and needed feature.

Ability to configure the basemap in ODK so you can see what’s there now on OSM or in satellite imagery as you map. This is technically possible in ODK but requires building/finding a tile cache, and loading manually on the device. Some way to configure and build and deploy these offline tile packs from Kobo would be very helpful.

Ok that’s all for now! There may be other details I’m not remembering, and I’m sure there’s lots to consider from what I wrote up here. Very excited to dig into these requirements with you all, develop a plan, and figure out ways towards implementation.

-Mikel

Thanks @mikel Shukran

Hi @mikelmaron, thanks for the many great ideas in this post! Some of them could probably be separate feature suggestions.

Regarding this item of matching OSM-expected output with KoBo output:

  • Exporting only the fields you are interested in is possible, though only via the API at the moment.
  • The length of the response name issue is only for using the formbuilder, not when importing from XLS. So setting the response name to amenity=place_of_worship should be no problem if you design your form in XLS.
  • The XML question name can’t include :, but you could use the proper OSM-compatible name in another language (e.g. label::OSM) and then export your data to that language. This could also be a workaround to the previous issue.

Do you have an example on how this work at the moment and how it could be done differently?

I think this could also be handled by adding an additional ‘language’ to your form, which would allow having multiple amenity columns.

GeoJSON is already possible, though only to export points, lines, and polygons created with these question types. The entire dataset could be formatted in a way to allow importing, but we would need much more details to guide the conversion.

3 Likes

If we wanted to design our forms in XLS, we would not use Kobo. We would just do all this directly in ODK. So this is actually a key requirement. We’re in the process of training others to use Kobo and they do not want or need to understand ODK as a whole in terms of creating forms separately. They’re interested in the simplicity of creating the form through Kobo.