Submission duplication after version 2.023.2

osmanburcu · July 19, 2023, 5:35pm

Hi community,

After releasing of version 2.023.21, me and my partners (more than 20-30 people) started complaining about submission duplications. The duplication rate increased drastically after the release. I am using the humanitarian server. We didn’t see any patterns for duplication, it looks random but there is no way to avoid it. Not only uuid is duplicating, the id is doesn’t duplicating

example form duplication on form: aGy93uHnGfNrKoqAn3GiRW

wroos · July 19, 2023, 9:32pm

What are you using, please: Enketo (Webform) or Collect (Kobo or ODK) ?

Kal_Lam · July 20, 2023, 6:04am

@osmanburcu, are you still seeing the duplication in your submissions? If yes, kindly confirm, and we can investigate it for you.

osmanburcu · July 20, 2023, 6:19am

Hi, we mainly use Enketo, but some field teams use Kobo Collect. It looks both teams are facing a similar issue. Yes Kal_Lam, we are still facing the same issue, it all started after the release and still continue (7/20/2023 09:19 a.m)

Kal_Lam · July 20, 2023, 6:23am

@osmanburcu, would you mind sharing with me the following information through a private message so that I could have a closer look at your case:

Username
Project name
Server

Note: You could share with me a private message as outlined in the post discussed previously:

Kal_Lam · July 20, 2023, 6:51am

@osmanburcu, is this the case you are trying to refer to?

Is this the only case, or are there any others too? If yes, could you also share the UUID so that it would be easy for me to cross-check?

osmanburcu · July 20, 2023, 6:58am

Dear Kal, these two is only example,

I can provide you with more codes; Those are under the same form I shared the information with you. If needed, i can share examples from other forms that are used by different partners.

Kal_Lam · July 20, 2023, 7:03am

Thank you, @osmanburcu! These details are enough for now. I will get back to you if I should need more details.

Kal_Lam · July 20, 2023, 4:30pm

@osmanburcu, this issue seems to be a known issue with KoboToolbox (unrelated to the recent release). You can track the GitHub issue here:

github.com/enketo/enketo-express

Duplicate UUIDs when same draft loaded multiple times

opened 10:44PM - 16 Jun 23 UTC

jnm

bug offline-only needs discussion

**Describe the bug** If someone opens the same form more than once within the s…ame browser session, and then loads the same draft in multiple tabs or windows, the submissions created from those drafts will have the same UUID. **To Reproduce** No special form is needed. I tested with a single `decimal` question. 1. Open an offline-capable form 1. Enter a response in a field and then click outside the field to change the focus 1. This triggers saving an automatic draft (`autosave successful` in the console) 1. Open the same offline-capable form in a new tab 1. When prompted with "Unsaved Record Found", click "Load Record" 1. Go back to the first tab and click submit 1. Go the second tab and click submit * You can make edits to responses here before submitting and also reach the same outcome 1. Result: two separate submissions with the same UUID are sent. This can be seen inside Enketo's IndexedDB `properties`, by looking at the `submitted` array within `<survey ID>:stats` **Expected behavior** The two submissions would have unique UUIDs. **Screenshots** ![image](https://github.com/enketo/enketo-express/assets/2085013/9af17842-e217-4510-9798-c8f399618f48) **Browser and OS (please complete the following information):** - OS: Linux, Windows, Android - Browser: Chrome, Firefox **Additional context** The same is true for named drafts (created with the "Save Draft") button as well as autosaved drafts. I identified this code path for UUID generation: 1. Initial page load in offline-capable mode: 1. https://github.com/enketo/enketo-express/blob/6.1.0/public/js/src/enketo-webform.js#L29-L40 1. https://github.com/enketo/enketo-express/blob/6.1.0/public/js/src/enketo-webform.js#L239-L250 1. Form initialization: 1. https://github.com/enketo/enketo-express/blob/6.1.0/public/js/src/module/controller-webform.js#L105 1. https://github.com/enketo/enketo-core/blob/7.2.3/src/js/form.js#L332 1. Generation of UUID (only if none has been set already): 1. https://github.com/enketo/enketo-core/blob/7.2.3/src/js/preload.js#L170-L179 For loading a (named) draft: 1. https://github.com/enketo/enketo-express/blob/6.1.0/public/js/src/module/controller-webform.js#L808-L814 2. https://github.com/enketo/enketo-express/blob/6.1.0/public/js/src/module/controller-webform.js#L270-L307 3. Then proceeds with "Form initialization" as listed above, which will *not* regenerate the UUID

Until the fix, maybe you could delete the duplicate _uuid manually.

I will also update you once the fix is made.

wroos · July 20, 2023, 6:17pm

Hint: This important issue has been reported often before, see Search results for 'duplicate uuid' - KoboToolbox Community Forum.

As far as I know, the documentation/forum answers for Kobo/ODK says that _uuid and also _id can be regarded as unique keys and users count on this, like we count on unique primary key in a database. (cc: @jnm)

jnm · July 26, 2023, 2:06pm

_id is absolutely unique on Kobo (within a particular server, e.g. _id will never be duplicated on kf.kobotoolbox.org). It is literally the primary key of a relational database table.

_uuid should be unique according to the ODK specifications, but sometimes is not. We’ve prioritized accepting data from clients even if it contains duplicate UUIDs. That is changing, and we will soon reject any submission that duplicates a UUID, even if the rest of the submission content differs. However, we’ll also have to deal with de-duplicating UUIDs among existing submissions—and this is not a trivial process.

Thanks.

happygreenshmoo · November 4, 2023, 4:45pm

Hello @Kal_Lam and @jnm , I am also facing this issue. We’ve deployed a survey and there are currently two forms with different data from the same UUID. Can you please advise on how to fix this? I’ve linked the kobo sychronous excel to PowerBI and so I dont know how to edit the backend to change this (but would prefer to do it there rather than in PBI).

Thank you!