I’m trying to download an encrypted form from humanitarianresponse.info using ODK briefcase. Other, non-encrypted forms are downloading properly, but when I try to get the encrypted form I get this error message:
Submission not retrieved: org.opendatakit.briefcase.util.ServerFetcher$SubmissionDownloadException: Fetch of a submission failed. Detailed error: INTERNAL SERVER ERROR (500) while accessing: https://kc.humanitarianresponse.info/lmh_rmel/view/downloadSubmission?formId=aYUb6osNQkDVsmqcpVzk5c%5B%40version%3DvyBmy5CEezkKt2b6tnFQes+and+%40uiVersion%3Dnull%5D%2Fdata%5B%40key%3Duuid%3A50833497-f4b9-4f6a-8c22-f131e02065f8%5D
Please verify that the URL, your user credentials and your permissions are all correct.
This survey data is of critical importance to us, any help on this would be really appreciated. Thanks
Hi
When I look at the issue you are having, I can say that this is potentially an issue with your encryption and not the system. This is affecting your retrieval
It’s not a decryption issue. When I manually place the few instances I have in the ODK Briefcase Storage directory, they decrypt (using export) with no problems. It’s specifically an issue with pulling forms from the database, where I get the Internal Server Error (500).
Also not an issue with my credentials, since I’m able to download non-encrypted forms.
In case you are still struggling with this issue:
I had the same. It is bug in ODK Briefcase in versions greater than 1.15. I filed a bug report to the developers. Meanwhile, the only workaround I could find is sticking to version 1.15.0
Additional information (I have just wasted 45 minutes of this, so if it can help someone…):
ODK Briefcase v1.15 is not able to handle a pulldata() call in a calculate field. So if, like me, you have an encrypted form with a pulldata() call in a calculate field, you need to do the following:
Use ODK Briefcase v1.15 to pull the encrypted data from the form
Close it and use a more recent ODK Briefcase (I used v1.16.3 and it worked) to export the decrypted data.
It seems that this issue remains, or has resurfaced with Briefcase 1.18, as raised by @wb_cjc. I’ve also experienced this issue - with failing to load media files.
Can we look into this? Can’t tell you how terrifying it is to collect data and then to not be able to pull it. I’ve also had issues where we wouldn’t be able to have someone from kobotoolbox decrypt it for us, otherwise it would be an IRB violation.
I wasn’t able to do it with v1.15, but tinkered around with different versions and was able to download with v1.11.0, then had to use a later version to actually decrypt.
Personally, I am sticking to ODK Briefcase 1.15, which is the latest version that produces no error when pulling encrypted data produced by KoBo and exporting it.
Later versions work well with encrypted data produced with ODK servers.
Could you tell me if you have attached CSVs in your file, @figuredio ?
Versions 1.17+ tell me there is no ID to be downloaded, and versions 1.16 or less crash at pulling time, the log file telling me my CSV file (the first used in a select_one_from_file in the form) does not have a <label> node whereas it has one. So I am now completely stuck.
I feel there is more and more discrepancies between KoBo and ODK over time and it’s becoming unbearable.
It’s a bit complicated on my side, but just take the bare usual “select_one_from_file.xlsx” and “fruits.csv” that you already have. On my side, version 1.15.0 or lower crash when trying to download its data, whereas it used to work several months ago.
Here are the logs:
2021-10-18 21:54:51,110 [pool-2-thread-4] INFO XFormParser - Reading XML and parsing with kXML2 finished in 1,385 ms
2021-10-18 21:54:51,112 [pool-2-thread-4] INFO XFormParser - Consolidating text finished in 0,561 ms
2021-10-18 21:54:51,113 [pool-2-thread-4] INFO XFormParser - Parsing form...
2021-10-18 21:54:51,113 [pool-2-thread-4] INFO XFormParser - Title: "Select one from file"
2021-10-18 21:54:51,113 [pool-2-thread-4] WARN XFormParser - XForm Parse Warning: Warning: 1 Unrecognized attributes found in Element [model] and will be ignored: [xforms-version]
Problem found at nodeset: /html/head/model
With element <model xforms-version="1.0.0"><instance><aD5kdC78obD69VRs5x8iJr id="aD5kdC78obD69VRs5x8iJr" version="1 (2021-10-18 18:52:09)">...
2021-10-18 21:54:51,125 [pool-2-thread-4] WARN o.j.xform.parse.FormInstanceParser - XForm Parse Warning: Warning: 1 Unrecognized attributes found in Element [instance] and will be ignored: [src]
Problem found at nodeset: /html/head/model[@xforms-version=1.0.0]/instance
With element <instance id="fruits" src="jr://file-csv/fruits.csv"/>
2021-10-18 21:54:51,127 [pool-2-thread-4] WARN o.o.b.model.BriefcaseFormDefinition - bad form definition
org.opendatakit.aggregate.exception.ODKIncompleteSubmissionData: Javarosa failed to construct a FormDef. Is this an XForm definition?
at org.opendatakit.aggregate.parser.BaseFormParserForJavaRosa.<init>(BaseFormParserForJavaRosa.java:463)
at org.opendatakit.briefcase.util.JavaRosaParserWrapper.<init>(JavaRosaParserWrapper.java:13)
at org.opendatakit.briefcase.model.BriefcaseFormDefinition.resolveAgainstBriefcaseDefn(BriefcaseFormDefinition.java:97)
at org.opendatakit.briefcase.model.BriefcaseFormDefinition.resolveAgainstBriefcaseDefn(BriefcaseFormDefinition.java:83)
at org.opendatakit.briefcase.util.ServerFetcher.downloadFormAndSubmissionFiles(ServerFetcher.java:131)
at org.opendatakit.briefcase.util.TransferFromServer.doAction(TransferFromServer.java:71)
at org.opendatakit.briefcase.util.TransferAction$GatherTransferRunnable.run(TransferAction.java:110)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.javarosa.xform.parse.XFormParseException: <label> node for itemset doesn't exist! [instance(fruits)/root/item/label]
at org.javarosa.xform.parse.FormInstanceParser.verifyItemsetBindings(FormInstanceParser.java:344)
at org.javarosa.xform.parse.FormInstanceParser.verifyBindings(FormInstanceParser.java:177)
at org.javarosa.xform.parse.FormInstanceParser.parseInstance(FormInstanceParser.java:84)
at org.javarosa.xform.parse.XFormParser.parseDoc(XFormParser.java:486)
at org.javarosa.xform.parse.XFormParser.parse(XFormParser.java:359)
at org.opendatakit.aggregate.parser.BaseFormParserForJavaRosa.<init>(BaseFormParserForJavaRosa.java:461)
... 9 common frames omitted
Problem is: version 1.16 and above won’t download data that was collected from en encrypted form to kc.humanitarianresponse.info. The submission.xml is doanloaded but all the attachments containing the encrypted actual data fail with error HTTP 403 FORBIDDEN, as already discussed in a few threads.
@freedim, as a quick check would you mind encrypting a normal survey project (without an external file) and then try to decrypt it with v1.15 to see if it works? The thing i am trying to check here is if the briefcase works or refuses to work with an external file.
Good point @Kal_Lam . Indeed, since the log indicates a problem with the <label> element, it makes sense. So I took the “fruits” example but kept the choices in the “choices” sheet instead, and encrypted the form. In that case, version 1.15 is able to download, decrypt and export smoothly. That may drive me to try the select_one_external solution… (not supported by KoBo or Enketo)
