API to get history of submission

mattlangeman · April 23, 2021, 5:21pm

I looking to use the API (/api/v2/assets/) to regularly get the latest data. Ideally, I would also get a history of changes to the data, but I have not figured out a way to get previous (changed) submissions via the API.

I am using /api/v2/assets/{asset_uid}/data/ to get the latest data. I want to get only data that is new or changed since my last pull of data. I first tried to query by _submission_time, but it seems like this is not updated on edit. I found that I can use meta/deprecatedID to get any submissions that have been edited.

/api/v2/assets/{asset_uid}/data/?query={"meta/deprecatedID": {"$exists": true}}

I can store the key:value (submission id : meta/deprecatedID) in order to determine which submissions changed.

I thought I might be able to use the meta/deprecatedID to trace the history of submissions. ?query={"_uuid": "7c4f3d6e-c306-40a5-8579-5ccdc27f2bfb"}

But this does not work for getting deprecated uuids. Once an edit occurs, and a new uuid is created, the API will not pull the previous version.

Is there a way to get the history/past versions of a submission via the API? Maybe some magic query parameter?

Thanks!
Matt

ks_1 · April 7, 2021, 6:07pm

_id parameter doesn’t change on edits, and it’s incremental. _uuid changes upon editing.

Using these 2, you can figure out which are the new records and which are the edited ones.

mattlangeman · April 7, 2021, 7:08pm

Thanks for the reply. I am able to find which records have been edited, but from what I can tell, I cannot get a history of the edits. I was hoping I could use the old _uuid values to get previous versions of edited submissions.

Josh · April 15, 2021, 4:37pm

Hi @mattlangeman, unfortunately this isn’t available through the API ~~and, as far as I know, we don’t store the history of individual submissions, only the form itself. @jnm will be able to correct/elaborate here.~~

ks_1 · April 7, 2021, 7:17pm

Ah… I doubt that’s possible. From what I can understand, every time you edit an instance, the older one is overwritten.

ks_1 · April 7, 2021, 7:25pm

But maybe you can work around it.
If you enforce a workflow where you use the new duplicate function before editing, you’ll have a history of all the edits.
If you have a primary key column in your data, you’ll be able to link them. Or else you can enter the previous _id in your edits to create a kind of linked list.

mattlangeman · April 7, 2021, 7:44pm

Thanks @Josh and @ks_1 .

Having a complete history probably isn’t critical, but would be nice. If I’m pulling in data on a nightly basis, it would only miss changes if there are multiple in a day.

@ks_1 Regarding enforcing a workflow and using the new duplicate function, are you talking about this Create new submission based on existing submission (aka duplicate record) · Issue #2769 · kobotoolbox/kpi · GitHub

So instead of editing, a user would duplicate a submission and then change the duplicate version?

mattlangeman · April 7, 2021, 8:05pm

I decided to dig into the code a bit. It does look like the older submission is overwritten. However, a new InstanceHistory object is created with a copy of the old xml, old uuid, and link to Instance object. Guessing this history is not accessible via the API, but nice to know it is getting stored.

github.com

kobotoolbox/kobocat/blob/78133d519f7b7674636c871e3ba5670cd64a7227/onadata/libs/utils/logger_tools.py#L85-L98


# check if its an edit submission
old_uuid = get_deprecated_uuid_from_xml(xml)
instances = Instance.objects.filter(uuid=old_uuid)

if instances:
    # edits
    check_edit_submission_permissions(submitted_by, xform)
    instance = instances[0]
    InstanceHistory.objects.create(
        xml=instance.xml, xform_instance=instance, uuid=old_uuid)
    instance.xml = xml
    instance._populate_xml_hash()
    instance.uuid = new_uuid
    instance.save()

jnm · April 10, 2021, 11:19pm

Right! Seems like it’d be nice to have in the future. Maybe something for Suggestion Box - KoBoToolbox Community Forum

mattlangeman · April 12, 2021, 12:50pm

@jnm Sounds good. I’ll add a description of this feature over there.

@wroos The suggestion you linked to is a bit different than this one. That is for a history of a form/survey. This is for a history of submissions.

wroos · April 12, 2021, 4:06pm

Sorry @mattlangeman, you are right. I removed the previous post.
I understood that you are just looking for a filter for (new/updated) submissions.

There was another discussion about history (change log) PER submission: https://community.kobotoolbox.org/t/update-an-already-submitted-record-on-a-mobile-client/1745/9.

Maybe setting and using the validation_status or an extra variable might offer a workaround?