Data is lost when correcting data in a submitted form

BartRoelandt · October 7, 2020, 1:01pm

Hello,
I have a form that uses an external table (csv) as a lookup list. It works very good on the ODK-collect app. But when I open a submitted form in KoboToolbox with the intention of correcting a data error, the form seems not to be able to use the external lookuplist. On top of that, calculated fields based on data coming from the external table, are empty. When I try to correct the data anyway, the original entry is replaced by “code_key”, which is the name of the field from the external table, as is stated in the choices-tab of the xls-form.
Can anyone help?
Bart Roelandt

Kal_Lam · October 7, 2020, 1:19pm

Welcome to the community @BartRoelandt! Could you kindly let the community know the following so that too could test the same at our end.

The question type you are using for the csv file.
Method of data collection (Collect android app or Enketo).

We would also appreciate if you could share with us a dummy of the question you had used (just that particular question) in an xlsform and then the csv response for the same. This would help us save our time to test your issue.

BartRoelandt · October 7, 2020, 1:24pm

Hello Kal Lam,
I can send you the whole form and external csv’s. I have no problem with that.
How can I send something? Is there a mailadress or can I attach files to a message here on the forum?
Bart

Kal_Lam · October 7, 2020, 1:25pm

Maybe if it’s not confidential, you could share the same through this very community post. The entire community could help you with the issue if it’s possible.

BartRoelandt · October 7, 2020, 1:37pm

No, nothing confidential, so no issue there.
I attach following files:

the xlsform: “XLSFORM_help”. I have coloured the rows in orange where the problem occurs
two external csv’s: with both I get the same problem
two jpg’s that are used in the form

To answer former question: I use ODK collect app on an Android mobile device to collect the data on the field.

Thank you for helping out.

OOPS - I get a message “Sorry, new users can not upload attachments.”

Kal_Lam · October 7, 2020, 1:42pm

Please refresh your page and you should be able to upload them. Would even appreciate if you could only upload the xlsform with the specific question that had an issue so that the community need not spend an entire time looking at other questions.

BartRoelandt · October 7, 2020, 1:47pm

So only the XLS-form?
Here it is:

XLSFORM_help.xlsx (84.9 KB)

You do not need the csv’s? The form will not work without them.

Bart R

BartRoelandt · October 7, 2020, 2:59pm

The first calculated data that is lost when opening the form in Kobotoolbox (to correct some other data) is calculated using a pull data expression “pulldata(‘select_level’, ‘code_naam’, ‘code_key’, ${domein})”
This is used to retreive data from the external csv table in a cascading select system

The second data with a problem is retrieved from a second external csv using a search expression. It searches for plantnames in the external table “soorten_hab” using a string entered in ${soortk} and looking for that string in two fields “dutchname_key” and “Scientname_key”.
search(‘soorten_hab’, ‘contains’, ‘dutchname_key, scientname_key’, ${soortk})

It looks as if Kobotoolbox cannot make the form work with the external CSV’s.

Kal_Lam · October 7, 2020, 3:14pm

Sorry, what i meant here was that kindly please only share the specific question through the xlsform and not the entire set of questions. Also please share the csv and other files necessary.

BartRoelandt · October 8, 2020, 8:22am

Hello,
I have shortened the XLS-form so that it only contains the two sets of questions that give the problems.
I have tested this form and it works on ODK Collect v1.28.1 to collect the data and upload those to KoboToolbox.
But when I try to change some of the entries in Kobotoolbox (Tab “Data/table/edit”) it does not work properly and so I cannot change or correct anything.
I send also two export files. The first contains the original entry, the second one is what remains after I tried to change anything in the cacade-select questions and after I tried to change the two first plant-species names.

Thank you for your help
Bart Roelandt

export after correcting testform_-latest_version-labels-_2020-10-08-08-11-31.xlsx|attachment (6.7 KB) export with original values testform_-latest_version-labels-_2020-10-08-08-05-50.xlsx|attachment (6.7 KB) XLSFORM_help2.xlsx (26.5 KB) soorten_hab.csv (1.9 MB) select_level.csv (2.6 MB)

Kal_Lam · October 8, 2020, 8:51am

FYI, i just made a check and could reproduce your issue. It seems like when you have an information filled using the pulldata function in your survey form you don’t see the same while editing it in the KoBoToolbox server.

FYI,

Data is seen in the KoBoToolbox server:

The same data is missing while trying to edit through Enketo:

Will get back to you once we are confirm it it’s an issue related to KoBoToolbox or Enketo.

Kal_Lam · October 8, 2020, 2:53pm

It seems to be an issue with KoBoToolbox! It’s already on the developer’s radar. Thank you once again for bringing this issue to the community!

For tracking you could follow the GitHub issue here:

Kal_Lam · October 29, 2020, 5:44am

Hi @BartRoelandt,

The issue has now been solved with the recent announcement of the new release. FYI:

BartRoelandt · October 29, 2020, 8:28am

Hello Kal Lam,

Thank you for this message! This is good news. I will test it as soon as possible.

Best regards,
Bart Roelandt

BartRoelandt · October 30, 2020, 8:50am

Hello again.

I did test yesterday but unfortunately nothing seems changed.
It takes already 20 to 30 minutes to open the survey I want to edit.

In the fields where a drop-down or list should appear (with the items filtered out of an external csv), nothing appears. (see printscreens)

When I want to delete an item the screen is again frozen for 20 minutes before anything happens.

So the update is not a success up till now.
Can anyone help to see what is the problem here?

Thank you,
Bart Roelandt

Kal_Lam · October 30, 2020, 9:57am

Hi @BartRoelandt,

Could you share with me your username, project name and the _id that is having issue through a private message. Would like to have a closer look at it and see if i am able to find any issue in it.

BartRoelandt · October 30, 2020, 10:24am

Yes, no problem. I will give you a test-username with all permissions on that project.
What is your private email adress - or how do I send a private message?
Bart Roelandt

Kal_Lam · October 30, 2020, 10:30am

Follow the post discussed previously to send a private message:

Kal_Lam · October 30, 2020, 2:34pm

Hi @BartRoelandt,

I observed that you have 2 users where one is the admin and the other is the data_collection account. You have designed and deployed your survey project through the admin user and have shared all the features with the data_collection account. You then collected the data and then tried editing the submissions through the data_collection account.

I tried out the same in my end and it works (with my dummy project). However, i could not open your survey project as the loading was very slow due to a large volume of questions (approximately 368). Would you mind trying the same with a dummy question just to see if it works and not with the correct survey project that has 368 question.

Kindly please let us know what happens. It should work (as it worked for me at my end).

BartRoelandt · October 30, 2020, 3:14pm

Hi Kal_Lam,
I made a new project, called “A test for kobo” in which I only retained the first part of my project that gives problems beeing edited.

You can try it yourself with the login I have sent you before.

But the result is the same (although the form opens quickly now as it is small): I cannnot edit as the dropdown are not filled with the expected items from the csv.