ODK Briefcase can decrypt some forms but not others

bfaguer · August 14, 2020, 10:18am

Hello everyone,

I have three forms on Kobo Toolbox that I want to download and decrypt using ODK Briefcase.

Form 1 has 8 submissions,
Form 2 has 6 submissions,
Form 3 has 1380 submissions.

I have tried to download and decrypt them using Briefcase 1.15.0 but it can’t even manage to download the forms, saying there is an authentication error.
When I try to download using v1.17.4, it works (with the exact same credentials), but it can’t decrypt.

What I did is write a script using the command-line interface to download the forms using 1.17.4, and decrypt them using 1.15.0.

It works for forms 1 and 2, but not for form 3. The error I get is :
Error: org.opendatakit.briefcase.model.ParsingException: Encrypted file not found

When I look at the instance folders where the data were downloaded, I get for all three forms these files:

submission.xml
submission.xml.enc

The difference is how submission.xml.enc is read for form 3.

Forms 1 and 2 appear like this when opened in a text editor:

Sublime Text 3 tells me it’s UTF-8.

While form 3 is read like this:
5d3f 0f24 3f3f 2679 293f 3f67 cd8d 3f7f c781 435e 3f3f 3f3f 2724 3f43 3f0f 3f3f 543f 3f3a 3f3f 3f26 4f3f 6e6d 6d3f 3f3d 3f3f 2b51 3f65 3e3f 5c3d 3f3d 3f7f 4e3f 4d3f 3f3f 1156 1a37 273f 3f15 023f 3f3f 3f45 573f 3fd7 b324 3f3f 3f58 103f 553b 0d3f 5737 253f 023f 0772 3f69 7b7e 0119 263f 5242 4d3f 3fd5 963f 687d 756c 053f 2d3f 6769 193f 3f3f 3f5e 5859 3f4c 3f4d 13d6 9402 3fce bf24 3f1f 3f0c 3f3f c798
Sublime Text 3 tells me it’s Hexadecimal.

Is there any known reason why it appears encoded in a different way? I guess this is why ODK Briefcase can’t see the encrypted file, but I can’t figure out the reason and the solution of this problem.

Edit: Manually downloading the attachments from the data panel, as a zip acrhive, results in the same submission.xml.enc files that can’t be decrypted.

Edit 2: I have manually sent a new submission on Form 3 with the web interface, and its submission.xml.enc file appears as the ones from form 1 and 2. When I delete all the other submission.xml for the other submissions and only keep the one I just sent, it can be decrypted. If I leave the rest, it tells me it can’t decrypt the files.
The new submission was sent using the Kobo website, while the other 1380 were sent using the Kobo Collect app.
It looks like the website sends properly formatted UTF-8 submission.xml.enc files, while the app sends Hexadecimal-encoded files which can’t be decrypted by Briefcase.

Kal_Lam · August 14, 2020, 3:40pm

Hi @bfaguer,

Would you mind letting the community know the server you are using. Maybe that would unfold the issue you are having. FYI, a lot of community users were able to decrypt their forms using ODK Briefcase v1.15.

Have a great day!

bfaguer · August 14, 2020, 5:40pm

Hi,
I am using https://kc.kobotoolbox.org.
I am unable to download the forms using Briefcase v1.15.0, so I am using v1.17.4 to download, and v1.15.0 to decrypt.

Kal_Lam · August 15, 2020, 7:11am

Hi @bfaguer,

Would you try downloading the forms and then decrypting them with the same Briefcase i.e. v1.15. This should solve your issue. As you say Briefcase i.e. v1.15 does not let you download forms, would you mind trying the same with some other dummy projects to see if it’s only for that particular project or is a case in general.

Have a great day!

bfaguer · August 15, 2020, 8:13am

Thanks for the suggestion, I discovered that Briefcase 1.15.0 can download projects created on my account, but fails for the projects created by my colleague.
The projects I need are created on my colleague’s account by I have the necessary permissions to access them. Proof is I can access them on the web interface and download them on a more recent version of Briefcase.
Is there some settings to change to allow 1.15.0 to download them?

Kal_Lam · August 15, 2020, 3:21pm

Hi @bfaguer,

In this case, would you mind downloading it through your colleagues user account (instead of yours) or request your colleague to download and decrypt.

Have a great day!

bfaguer · August 15, 2020, 3:30pm

All right, I will ask him for his credentials and get back here to update if it worked.

Edit: I have tried with my colleague’s credentials, both using the command-line interface and the GUI and I still can’t download.
The GUI returns these error messages:

Fetching form definition
resolving against briefcase form definitions
Fetching form manifest
preparing to retrieve instance data
processing chunk 1...
Not all submissions retrieved: Error fetching list of submissions: Authentication failure
Failed.

And the detailed logs return this:

2020-08-15 17:34:51,582 [pool-2-thread-1] INFO  o.o.briefcase.util.AggregateUtils - Downloading URL https://kc.kobotoolbox.org/[MY_COLLEAGUE'S_USERNAME]/forms/377866/form.xml into /var/folders/bk/f8510jm57tg4q6v61hvpt4xc0000gn/T/briefcase_2213162032428515256_form.xml
2020-08-15 17:34:51,588 [pool-2-thread-1] INFO  o.o.briefcase.util.AggregateUtils - Parsing URL https://kc.kobotoolbox.org/[MY_COLLEAGUE'S_USERNAME]/forms/377866/form.xml
2020-08-15 17:34:53,237 [pool-2-thread-1] INFO  XFormParser - Reading XML and parsing with kXML2 finished in 5.482 ms
2020-08-15 17:34:53,240 [pool-2-thread-1] INFO  XFormParser - Consolidating text finished in 2.577 ms
2020-08-15 17:34:53,240 [pool-2-thread-1] INFO  XFormParser - Parsing form...
2020-08-15 17:34:53,240 [pool-2-thread-1] INFO  XFormParser - Title: "[FORM_3]"
2020-08-15 17:34:53,273 [pool-2-thread-1] INFO  XFormParser - Creating FormDef from parsed XML finished in 33.009 ms
2020-08-15 17:34:53,274 [pool-2-thread-1] INFO  XFormParser - Reading XML and parsing with kXML2 finished in 0.202 ms
2020-08-15 17:34:53,274 [pool-2-thread-1] INFO  XFormParser - Consolidating text finished in 0.043 ms
2020-08-15 17:34:53,274 [pool-2-thread-1] INFO  XFormParser - Parsing form...
2020-08-15 17:34:53,274 [pool-2-thread-1] INFO  XFormParser - Title: "Encrypted Form"
2020-08-15 17:34:53,275 [pool-2-thread-1] WARN  XFormParser - XForm Parse Warning: Warning: 1 Unrecognized attributes found in Element [bind] and will be ignored: [length] 
    Problem found at nodeset: /html/head/model/bind
    With element <bind nodeset="/data/base64EncryptedKey" type="string" length="2048"/>

2020-08-15 17:34:53,275 [pool-2-thread-1] WARN  XFormParser - XForm Parse Warning: Warning: 1 Unrecognized attributes found in Element [bind] and will be ignored: [length] 
    Problem found at nodeset: /html/head/model/bind
    With element <bind nodeset="/data/base64EncryptedElementSignature" type="string" length="2048"/>

2020-08-15 17:34:53,276 [pool-2-thread-1] INFO  XFormParser - Creating FormDef from parsed XML finished in 1.179 ms
2020-08-15 17:34:53,281 [pool-2-thread-1] INFO  XFormParser - Reading XML and parsing with kXML2 finished in 1.757 ms
2020-08-15 17:34:53,282 [pool-2-thread-1] INFO  XFormParser - Consolidating text finished in 0.597 ms
2020-08-15 17:34:53,282 [pool-2-thread-1] INFO  XFormParser - Parsing form...
2020-08-15 17:34:53,282 [pool-2-thread-1] INFO  XFormParser - Title: "[FORM_3]"
2020-08-15 17:34:53,292 [pool-2-thread-1] INFO  XFormParser - Creating FormDef from parsed XML finished in 9.688 ms
2020-08-15 17:34:53,293 [pool-2-thread-1] INFO  XFormParser - Reading XML and parsing with kXML2 finished in 0.318 ms
2020-08-15 17:34:53,293 [pool-2-thread-1] INFO  XFormParser - Consolidating text finished in 0.046 ms
2020-08-15 17:34:53,293 [pool-2-thread-1] INFO  XFormParser - Parsing form...
2020-08-15 17:34:53,293 [pool-2-thread-1] INFO  XFormParser - Title: "Encrypted Form"
2020-08-15 17:34:53,293 [pool-2-thread-1] WARN  XFormParser - XForm Parse Warning: Warning: 1 Unrecognized attributes found in Element [bind] and will be ignored: [length] 
    Problem found at nodeset: /html/head/model/bind
    With element <bind nodeset="/data/base64EncryptedKey" type="string" length="2048"/>

2020-08-15 17:34:53,293 [pool-2-thread-1] WARN  XFormParser - XForm Parse Warning: Warning: 1 Unrecognized attributes found in Element [bind] and will be ignored: [length] 
    Problem found at nodeset: /html/head/model/bind
    With element <bind nodeset="/data/base64EncryptedElementSignature" type="string" length="2048"/>

bfaguer · August 17, 2020, 8:43am

@Kal_Lam: I don’t know if it’s you or if I did it by mistake, but one of your posts was tagged as my problem’s solution, but the issue remains, so I unselected it.

Did anyone else have the issue regarding the authentication problem using Briefcase 1.15.0?