I have a form that is getting fairly heavy submission traffic, and that we keep permanently open for submissions all year round.
The data starts to be heavy, and is making analysis / exports really slow, so we’d like to find a way to clean / wipe / archive historical submission data, while keeping the form open with the same id (I know I could just clone the form into a new one, but then it would change the form ID and I would need all collectors to switch over, and redo all my data integrations).
Is there a convenient way to do this, or do I just need to implement my retention / cleaning policy via API?
It might not be a proper way but you can export the data and delete all the previous ones.
May I ask how many submissions do you have in your form? I’m asking this because I’m trying to merge several forms into one form in 2022 and if the data becomes heavy I might not consider doing that.
Hey for anyone coming across this, I indeed ended up solving this by implementing my own retention policy via API.
I’ve used the excellent n8n.io API automation solution (a free, open-source alternative to Zapier), for which I wrote a Kobotoolbox connector, to orchestrate the flow in 2 steps:
1 Workflow will scan or listen to non-validated submissions, both by polling or receiving a real-time webhook. It will then upload the submission content into our BI solution (Elasticsearch) for archiving, and mark the submission as Validated (we use this flag as a “soft delete” to leave us a few days to review it before a real delete).
If you’re interested in using n8n to automate some Kobotoolbox tasks, my connector is not yet available publicly, the PR is pending here (feel free to vote for it to speed up the approval!)
