Cannot export and download my data

User Support Data Management
, , ,
1

I cannot export my data and download it. I tried CSV and XLS but it always says “Processing” for a very long time, then eventually “failed”. We have 80,000+ records. Could this be the problem? In any case, I really do not want to download all 80,000* records. I am only interested in recent records, say for the past 1-4 weeks maximum. How can I download only these? Also, how can I download records only for a particular country? Ideally, I would like to download by date range and by country to avoid always downloading the entire data set.
This relates to a current Desert Locust emergency in E Africa and Arabia, so I hope someone can help me soon. Thank you.

2

Hi @locust, have you tried using the legacy export options for CSV or XLS?

If you pull data through the API you are able to filter records based on various queries, for example by date and/or country. Please refer to the API documentation at https://{kf_url}/api/v2/assets and https://{kf_url}/api/v2/assets/{asset_uid}/data for more details.

1 Like
3

Hi Josh,
Yes, I tried the legacy but did not work. I don’t use API and instead rely on simply downloading from the platform, which I think should work but does not.
Any suggestions? Thx, Keith

4

Hi @locust, that is concerning. Would you mind please sharing the following with me in a private message so that I can take a look:

  • Server (OCHA/HHI)
  • Username
  • Project name

In the meantime, you can do the following to meet your original need of pulling data based on specific criteria. Let’s assume you have a question called country and you also want to limit the _submission_time property to submissions after a certain date.

In a terminal (or with a REST client), you can do the following:

curl https://KF_URL/api/v2/assets/ASSET_UID/data/?query={"country":"Sudan", "_submission_time":{"$gt":"2021-03-02T18:34:12.498-08:00"}}&format=json \
-H "Authorization: Token TOKEN"

This will filter all records with country equal to “Sudan” with a _submission_time after “2021-03-02T18:34:12.498-08:00”. You can direct the output to a file and consume the data from there.

If you are familiar with Python, you could do something like the following to get it back into an XLS file:

import requests

import pandas as pd

TOKEN = 'your-token'
KF_URL = 'kf.kobotoolbox.org' #or 'kobo.humanitarianresponse.info'
ASSET_UID = 'your-asset-uid'
QUERY = '{"country":"Sudan", "_submission_time":{"$gt":"2021-03-02T18:34:12.498-08:00"}}'
URL = f'https://{KF_URL}/api/v2/assets/{ASSET_UID}/data/?query={QUERY}&format=json'

headers = {"Authorization": f'Token {TOKEN}'}

response = requests.get(URL, headers=headers)
data = response.json()

df = pd.DataFrame(data['results'])
# export to xls
df.to_excel('my-data.xlsx')

Hope that helps.

1 Like
5

Hi Josh,
I sent you private message with the info. I’m pretty limited on the rest of your suggestions below.
Many thanks, Keith

1 Like
7

Hi Josh,

But this doesn’t export data related to repeat group questions as it would be done when exporting from the server.

Regards

Serge

8

Hi @serkial1, you might find the post here helpful:

1 Like