Dynamic Data Attachments throwing a 'repeated value' (indexed-repeat) error, and relevant column not working

[janna]

Hi! @Josh and @Kal_Lam and anyone else who understands how DDA works in the back-end!

I’ve got a conundrum, and I’m wondering if you can help me out

I’m using Dynamic Data Attachments, with a couple of unique elements:

• the parent form has a series of ‘optional’ label IDs entered, across a few different groupings (e.g., could be male/female, powder/non-powder, or atypical - any combination of those could be labelled, for example, given a Label ID).
• The child form then essentially ‘scans’ the Label ID, and then should tell the user what is included within that label.

However, what’s happening right now is that when I open the child form, the whole form is deploying immediately, and my ‘relevant’ column ceases to work. So, for example, I have an error message that should only appear when if the label ID is scanned (i.e., not blank) and does not match the format of a label ID. However, on opening the form, that error message appears right away. I’ve experimented with all kinds of relevants, but nothing changes in the form. So, something is rendering the entire column to not work.

(When uploading the form to ODK Validate, no issues with the form.)

And the second big problem, is that an error message appears saying there’s an indexed-repeat issue. Now, I’ve dealt with this before, and know that this error appears when an ID key is repeated in the lookup column. But what it’s doing is throwing this error when opening the form, and it’s matching the ‘blank’ value (before anything is scanned) to multiple ‘blank’ values in the lookup form.

In testing, this form worked. It’s only after being deployed for the pilot test in the field that this error started to happen.

So that led me to test a couple other things:

1. could I replicate this issue in a similar dummy set of forms, where I’m trying to match values, and have multiple blanks in the lookup column? (result - this does not throw an error; it looks like blanks are allowed in the lookup column)
2. could I simplify the form down to a point where I could pinpoint the error (result - I’ve simplified it down to only about 4 lines, and know where the error is coming from, and still looks like it’s coming from repeated blanks and the relevant being ignored).

Now, if you actually scan a Label ID, the form does work. The DDA is setup correctly and is functioning well. But the problem is that upon opening the form, you just get red error messages and error popups happening BEFORE a label ID has been scanned, which is very offputting for the team - but it also makes me nervous because I can’t pinpoint why it’s happening, what the error is, as I can’t replicate it thus far.

It’s led me to believe that in some way the error may be coming from some way the child form is interacting with the DATA of the parent form, but without being able to replicate it, I don’t know what’s going on. The only thing that works is to delete the rows where the ‘blank’ values are in the lookup column. However, once again, based on my dummy testing, blank values shouldn’t cause an error, so… I’m going around in circles.

Other things I tested:

• deleting all settings, extra tabs, choices, from the form, deleting all extra columns and rows, making sure there’s nothing else hanging out in the form that’s causing this error.
• disconnecting and reconnecting the parent and child form.
• putting the error messages, etc, on a second page (field-list) after the barcode scanning, to give the user the opportunity to scan the label before errors appear, but this still throws the indexed-repeat error when entering the form

Other ideas I’ve thought about overnight to test:

• the dummy form tests a lookup column two groups deep, whereas the real form uses a lookup that’s three groups deep
• length of the names used in the lookup

Or, I’m wondering, because these worked in testing (in May) and don’t work anymore, was there any update on the backend (a release) that would cause an error like this?

What I’m attaching here:
The two dummy forms (that work)
DDA Blank Test 2.xlsx (11.2 KB)
aPohGDPZo9F7CMtndpuUrN.xlsx (11.1 KB)

The child form (that causes the error).
BF_PMB_SWN_PROCESSING_2_v3testeroony8.xlsx (18.2 KB)
(You can see my mental capacity at this stage has run out…I went through naming phases of ‘test, test1, test2’, then moved to ‘testgroup, testgroup1, etc’…now I’ve just landed on ‘testeroony’, lol).

In order to understand the error, I think seeing the backend data might be necessary, so if you need to see anything else, please let me know and I’ll try to record a video or something. But help pointing me in the right direction would be very helpful! If I can figure anything else out, I’ll post here.

This is the image of the lookup column that’s causing errors:

These are 2 pics of the image getting in Collect:

(You can see the row numbers match up to the rows containing blanks)

Thanks for any support you can offer!
Janna

[janna]

Update: I’ve been able to replicate this problem (finally) in my dummy form…I was on the cusp, and was entering last-ditch territory, but after posting this, I was going to do the 3-groups-deep test and the name-length test, and realised these were things that I noticed were ‘different’ between my dummy files, and my real files. So I thought to myself, what else is different between the two files?

So, the only other difference was that in the real files, I was using one column as the lookup/filter, e.g., find out where label ID in the child form equals label ID in the parent form, but then I was returning a different column, essentially a ‘unique ID’ for that whole parent-form submission.

Whereas in my dummy form, I was returning the lookup value itself. For example, lookup ID ‘0006’, and where that equals ‘0006’ in the parent form, then return the value ‘0006’ from the parent form. BUT, to mimic my real form better, I added a unique ID like this: lookup ID ‘0006’, and where that equals ‘0006’ in the parent form, then return the unique value ‘B’ of that whole submission from the parent form (e.g., submission ‘B’ included label IDs 0006 and 0007.

And here’s where the logic breaks down when blanks are present - this is my understanding so far, although my assumption is that my understanding is still a bit incorrect, so if you can help me understand even better, it would be helpful:
if the child form has a blank in the place of the value, then it goes to the parent form data to lookup that blank. It sees 4 blank values in the lookup column. But it’s then asked to return the matching value - and in all 4 cases, the matching value is blank. blank, blank, blank, blank. So there’s no mismatch, and it doesn’t throw an error (my theory).

However, when it’s asked to return a unique row ID that’s separate from the blank value itself, it looks up the blank value in the lookup column, sees 4 blanks, but then sees that the matching row/submission IDs for those blanks are all different: blank=B, blank=C, blank=D, blank=G. And then it throws an error and says ‘indexed-repeat()’ - you’ve got a repeated values problem!!

So here’s my revised support question: I’d been using the ‘relevant’ column to basically say - 'don’t do this DDA calculation until there’s something entered into the label ID lookup value box in the child form! (because I know it throws an error if that value/blank is duplicated). However, that doesn’t seem to work (at least in my real form). I will test this again in the dummy form, to see if I can still replicate the ‘relevant’ error, as that still doesn’t seem to be resolved in my mind.

Any advice on how to get around this issue using relevant, etc, would be helpful.

Note: there’s no error when doing all of the above in Enketo, this is only in Collect.

And yes, to be clear - I am never looking up ‘blank’ values. I’m simply trying to prevent the form from looking up blank values before the enumerator has had a chance to enter the label ID. So this is an error that’s popping up when the form is first opening up, making the whole thing looks clunky and like it’s not working.

My revised files that replicate the error:
aPohGDPZo9F7CMtndpuUrN-testoriginalidlookup.xlsx (11.1 KB)
DDA Blank Test 2-testoriginalidlookup2.xlsx (11.1 KB)

[janna]

One more update, I’ve just updated the child form with skip logic, and I STILL see the error.

SO, this means that relevant / skip logic cannot be used to solve this problem?

I have gone back to my original real file and see if I can get it working once again with relevant, but no. The dummy file updated with relevant still doesn’t solve the problem:

DDA Blank Test 2-testoriginalidlookup2withRelevant.xlsx (11.1 KB)

Why doesn’t skip logic work here? What am I not understanding about this? Help is appreciated!

[janna]

ooookay - the solution. I’ve had a great conversation on here with myself

Use an ‘if’ statement in the calculation that prevents the whole calculation from firing until that label-id has been filled in. Something like this:
if(string-length(${labelid})>0,instance('form1')/root/data[a1/a1_label1 = current()/../labelid]/original_id,'')

Essentially, the form does NOT work in the Collect app when you have the potential of having ‘duplicate blanks’ in the parent form. And the 'relevant’column (skip logic) does NOT work to get around that issue.

So you basically have to use an ‘if’ statement inside the calculation itself to create a forced ‘skip logic’.

I honestly have never seen a form behave in the way that I experienced in this issue…I could tell you about a lot of whacky things the form did that didn’t follow the XLSForm design/coding whatsoever. Like, show all 400 questions of the questionnaire in a single page (totally disregarded ALL skip logicn and ALL field-list appearances, groups, etc). It was sooo weird.

Then I popped in the if statements…and all was right again.

Very whacky and strange. Simple solution in the end, but it was a lot of trial and error to understand the problem before I could get there. Hope my pain solves someone else’s problem some day!!

Once again, the two dummy forms that work:
DDA Blank Test 2-testoriginalidlookup3.xlsx (11.2 KB)
aPohGDPZo9F7CMtndpuUrN-testoriginalidlookup.xlsx (11.1 KB)

[Kal_Lam]

Thank you for sharing the solution with the entire community, @janna!