Hi, anyone have an idea how to connect Kobo Form with ArcGIS Online with real time updates ?
Hi @ISLEM and welcome to the community. I once did this sometime through the v2 API provided by KoboToolbox.
Below I will share some code snippets of how I managed this.
Keep in mind you would also need to install geopandas in the environment that you will be using
1. Establish connection with ArcGIS Online
from IPython.display import display
from arcgis.gis import GIS
import os
gis = GIS('https://arcgis.com/', '<your-username>', '<your-password>')
gis
Once the connection has been established, then,
2. Create a Pandas dataframe
# import requirements
import requests
import pandas as pd
# Authentication credentials
username = "<your-kobo-username>"
password = "<your-kobo-password>"
# API endpoint and export token
kobo_api_url = "<synchronous-url-to-the-form-data>"
export_token = "<your-kobo-export-token>"
# Create a session with authentication
session = requests.Session()
session.auth = (username, password)
# Construct the export URL
kobo_export_url = f"{kobo_api_url}/data.xlsx?format=xlsx&token={export_token}"
# Fetch data using the authenticated session
response = session.get(kobo_export_url)
# Check if the request was successful
if response.status_code == 200:
# Load data into a pandas DataFrame
kobo_data = pd.read_excel(response.content)
kobo_data.head()
else:
print("Error:", response.status_code)
** 3. Just check if the pandas data frame is now showing**
kobo_data.head()
From this point keep in mind that some of the variables used here relate to the data that I used.
If all is well and we have a dataframe then lets create a GeoDataframe which is required by the AGOL platform
4. Create GeoDataframe and Shapefile
import geopandas as gpd
import zipfile
# Convert datetime columns to string format
# kobo_data['today'] = kobo_data['today'].astype(str)
# kobo_data['Date'] = kobo_data['Date'].astype(str)
kobo_data['Date of Birth'] = kobo_data['Date of Birth'].astype(str)
# Convert the DataFrame to a GeoDataFrame
geometry = gpd.points_from_xy(kobo_data['_Location (Coordinates)_longitude'], kobo_data['_Location (Coordinates)_latitude'])
gdf = gpd.GeoDataFrame(kobo_data, geometry=geometry)
# Set the CRS (Coordinate Reference System) for the GeoDataFrame
crs = '+proj=longlat +datum=WGS84 +no_defs'
gdf.crs = crs
# Path to save the shapefile
shapefile_path = '/arcgis/home/cms/cms_registrations.shp'
# Save the GeoDataFrame as a shapefile
gdf.to_file(shapefile_path, driver='ESRI Shapefile')
# Create a zip file
zip_file = '/arcgis/home/cms/cms_registrations.zip'
with zipfile.ZipFile(zip_file, 'w') as zf:
# Add shapefile files to the zip
for extension in ['.shp', '.shx', '.dbf', '.prj']:
file_path = shapefile_path.replace('.shp', extension)
zf.write(file_path, arcname=file_path.split('/')[-1])
print('Shapefile zipped successfully.')
If this is your first time publishing, somewhere here you need to use the publish() function provided by ArcGIS API
5. Access the already existing feature layer via its URL
# Access the feature-layer through its URL
search_result = gis.content.search("title:CMS - Registrations", item_type = "Feature Layer")
display(search_result)
search_result[0]
Once you have confirmed this is the layer,
6. Let’s overwrite the layer with the new data
from arcgis.features import FeatureLayerCollection
kobo_FL = FeatureLayerCollection.fromitem(search_result[0])
# Overwrite the FL
kobo_FL.manager.overwrite(zip_file)
I hope this will help you get started.
Lastly, you will need to schedule your notebooks in ArcGIS so they can collect the data at intervals that you require it to update.
2 Likes
