Archive submission data

User Support Data Management
, ,
1

Hello there! :wave:

I have a form that is getting fairly heavy submission traffic, and that we keep permanently open for submissions all year round.

The data starts to be heavy, and is making analysis / exports really slow, so we’d like to find a way to clean / wipe / archive historical submission data, while keeping the form open with the same id (I know I could just clone the form into a new one, but then it would change the form ID and I would need all collectors to switch over, and redo all my data integrations).

Is there a convenient way to do this, or do I just need to implement my retention / cleaning policy via API?

Many thanks!

2

Hi @yjouanique,

It might not be a proper way but you can export the data and delete all the previous ones.

May I ask how many submissions do you have in your form? I’m asking this because I’m trying to merge several forms into one form in 2022 and if the data becomes heavy I might not consider doing that.

All the best!

3

Our 2 heaviest forms have 100k and 200k submissions, respectively.

But somehow the performance issue is more visible on the 100k form - perhaps because the content is much more complex…

It’s still usable, but not the best experience.

Also, we’re running our own server, so your mileage may vary of course (although we’re not seeing any capacity crunch on the hardware side).

1 Like
4

Thank you so much for the detailed answer @yjouanique, hope you find a solution to your problem.

5

Hey for anyone coming across this, I indeed ended up solving this by implementing my own retention policy via API.

I’ve used the excellent n8n.io API automation solution (a free, open-source alternative to Zapier), for which I wrote a Kobotoolbox connector, to orchestrate the flow in 2 steps:

  • 1 Workflow will scan or listen to non-validated submissions, both by polling or receiving a real-time webhook. It will then upload the submission content into our BI solution (Elasticsearch) for archiving, and mark the submission as Validated (we use this flag as a “soft delete” to leave us a few days to review it before a real delete).

    image
    image1465Ă—400 69.9 KB

  • 1 workflow will poll for submissions older than X days AND Validated

    image
    image968Ă—341 45.3 KB

If you’re interested in using n8n to automate some Kobotoolbox tasks, my connector is not yet available publicly, the PR is pending here (feel free to vote for it to speed up the approval!)

2 Likes
6

Thank you @yjouanique for sharing this with the entire community.