We are working on a project to create a tool that will allow Kobo data from a suite of standard forms to be easily mapped to Postgres databases for integration purpose. We want to create a tool that can be re-used by multiple organizations. Below is an outline of what we are working on.
Does anyone have experience in this area that they are willing to share or any other inputs/suggestions?
Vision
Efforts are underway to develop a standard library of Kobo form for conservation (KoboConsForms). Conservation organizations should easily be able to deploy a Kobo Integration Toolkit (KIT) server that:
- Allows organizations to aggregate data from KoboConsForms from multiple accounts into a central Postgres database sources data from both the [Kobo for Researchers and Aid Workers] instance and from local Kobo servers
- Allows multiple KIT’s to be deployed and updated (e.g. separate KITs for different organizations or initiatives)
- Allows deployment of new Kobo forms (including corresponding Postgres table creation script and code to map Kobo forms to database table)
- Allows access to report generated from the Postgres database
- Integrates with R to allow R scripts to be run to analyze data
- Have open APIs to allow data to be fed into other systems
- Support a range of login options (e.g. Google, O365 etc)
Phases
Phase 1
Phase 1 has been completed. This focused on testing the deployment of the Docker based tool [Kobo Aggregator] (https://github.com/wri/django_kobo) (KA) tool from WRI to test it’s ability to deliver a foundation for creating KIT. KA was easy to deploy. Registering accounts to source data from was simple. It supports to Kobo form [BNS] Included in the application is an [NGINX] based web application that allows logged in users to view and analyze their data.
It falls short on the vision in the following priority areas:
- It is not generalized to support additional survey
- It is not simple to add new surveys with corresponding Postgres tables
- It does not have an API for accessing data
Phase 2
The suggested activities for Phase 2 are:
Extend KA to support two additional surveys :
- Offtake:
- Socio Economic Database : WCS Socio Economics Library )
Solve a (to be defined) subset of the [bugs/issues]
Create a proposal on how to generalize the NGINX application to support multiple addition surveys
Support logins to multiple Kobo instances
Create a proposal on how KA could support upload a “package” that defined a Kobo form, a set of Postgres tables to map to and a mechanism or code for mapping them. The proposal should indicate what level of complexity could be supported by the package approach - at a minimum it should support basic forms with no workflow or branching. If possible the proposal should a phased approach to supporting complexity. The proposal should also indicate how a package could be modified or adapted over time e.g. supporting the adding or hiding of questions.
Phase 3
Create KIT delivering at least:
- Allow organizations to aggregate data from KoboConsForms from multiple accounts into a central Postgres database sources data from both the [Kobo for Researchers and Aid Workers] instance and from local Kobo servers
- Allows multiple KIT’s to be deployed and updated (e.g. separate KITs for different organizations or initiatives)
- Allows deployment of new Kobo forms (including corresponding Postgres table creation script and code to map Kobo forms to database table)
- Allows access to report generated from the Postgres database
