Today (09-nov-2021) onwards kobo shows error when importing

Today onwards, when i try to import existing xls form shows “Value Error:there are duplicates in the name column”. It was working last day. I have column “allow_choice_duplicates” set to “yes” in settings sheet.

Pls help

Welcome to the community, @Lakshmikanthlr! Did you have the duplicate choices on purpose in your xlsform? Have you tried validating your xlsform through this online validator to see if you could fix any syntax issues present?

This post discussed previously should also help you understand how you could get rid of the duplicates from your xlsform:

Hi Kal_Lam,
just now validated the form and it has no errors.

Could you share with the community the error message you are seeing at your end?

1 Like

Hi @Kal_Lam + @Lakshmikanthlr

Interestingly, Today i’ve seen this error for 3 different forms. Exact error message = ValueError: There are duplicates in the name column

All 3 forms are forms that we have successfully deployed to Kobotoolbox in the past, and no variable name changes occured for these forms since.

See attached for an example - this is the exact XLS file we had deployed successfully a few weeks ago, and then failed to deploy today. Note the duplicated “trt_note” on lines 34 and 38.

5_planting.xlsx (152.0 KB)

  • For reference, the deployed form has asset ID: aGfGDkhYQGQei3datNmVV5 (username faw_data_platform)
  • The current (v4) version is our fix for the “duplicates in the name column”, where I just renamed the 2nd trt_note.
  • I cloned the old (v3) version as a new form: aQqkBdMy8xZEmUH2TSaQvY.
  • Downloading the old version (v3) as XLS from Kobo shows that Kobo had done the automatic renaming, so the 2nd note was now called “trt_note_001” and the original name was stored as $given_name.
  • I believe this is how Kobo handles (or handled?) duplicates in the ‘name’ column.

For information, the attached form successfully compiles at the online validator, ODK - XLSForm Online v2.x. The duplicate name in theory shouldn’t matter as one is inside a group (so they have different absolute references in the resulting xml file).

I’m Interested in what the issue is here, as in the past we have often used repeat names inside different groups, especially for notes or other non-critical variables. I’d like to know if this is a deliberate change to how Kobo handles this situation, or if it’s a glitch that’ll get patched soon.

Thanks,
Dave

Hi @degami, could you also let us know the server you are using? Maybe that would be helpful with the troubleshooting.

Sure - it’s the regular (kf.kobotoolbox.org) server. Thanks!

So is it this xlsform 5_planting.xlsx that is having the issue @degami? Have you cross checked with the OCHA server to see if you should see any differences on how it behaves?

@Kal_Lam - yes, that 5_planting.xlsx is one of the forms that I got the error with on kf.kobotoolbox.org. I just tried on a test account on the OCHA server and it uploaded and deployed fine, so it looks like the issue is just with kf.kobotoolbox.org?

1 Like

Great @degami! Will also have a look at my end to see if there is anything different I can spot. BTW, did you make any edits to the xlsform while uploading the same to the OCHA server or could you upload it without any issue?

@Kal_Lam - great - thanks for taking a look. For testing, I uploaded exactly the same xlsform file to the OCHA server as I did to the regular kf.kobotoolbox.org public server.

Hi @degami, @Lakshmikanthlr thank you for reporting this issue. We are busy investigating :+1:

2 Likes

Hi @degami, @Lakshmikanthlr a fix has been deployed to address this. Thank you for your patience.

2 Likes

Hi @degami.

I know the change is buried among many others in our latest release notes, but this change was made deliberately and it will return as soon as we stop including $given_name and __version__ in XLSForm downloads. This work is tracked at Exclude internal stuff from XLSForm downloads · Issue #3539 · kobotoolbox/kpi · GitHub.

You’re right on two counts:

This isn’t a great situation, and our options for improvement were:

  1. Let through forms, unmodified, with duplicate question names;
  2. Prohibit duplicate question names.

We chose option (2) in the interest of safety. As you also observed, it’s not a huge inconvenience to rename questions to achieve uniqueness when designing a survey. I would argue that, in the examples below, the latter is preferable to the former:

<work>
  <address />
  <phoneNumber />
</work>
<home>
  <address_001 />
  <phoneNumber_001 />
</home>
<work>
  <work_address />
  <work_phoneNumber />
</work>
<home>
  <home_address />
  <home_phoneNumber />
</home>

The main rationale, however, is that even with a lot of QA, it’d be very difficult to ensure that a code base as large as ours would handle duplicate names (albeit with unique paths) correctly in all circumstances.

In short, support for duplicate names will end in an upcoming release. We reverted the change for the short term because we didn’t want to inflict unnecessary pain in the download-edit-upload XLSForm workflow that many use. Please take note @Josh, @stephanealoo, @Kal_Lam.

Thank you for your patience and understanding.

3 Likes

Hi @Josh, @jnm ,
Thanks very much for the work, very quick update and detailed explanation. I’ll pass this on to our team.

Your rationale for choosing to reject duplicate names makes a lot of sense, and in the mid-term it shouldn’t be an issue for us to start checking for duplicate names in our forms seperately from using the online xlsform checker.

Also thankyou for reverting this change temporarily - That’s incredibly useful as it will let us wrap up the various projects using existing forms and give us a chance to go through and make updates where needed. (We have some quite complex setups with multiple teams using multiple XLS forms, deployed via the API with regularly updating csv file attachments).

I’ll keep an eye on the github issue you shared, so we’re ready for the change the next time it happens.

3 Likes

Hello @degami,
I think the XLSForm Online validator still creates an error for duplicate (variable) names. Test and correct me if I am wrong, please. So, you don’t need to do something separately, if you check with the online validator.

I think, the discussed change and set back relates to KoBoToolbox (incl. deployment) only.

1 Like