Encryption errors when using Kobo

Hi there,

There is a bug when you try to use an encypted form on Kobo and data has been deleted (at any point) for that form. For example:

  • You submit data to that form, pull it with briefcase, and export it. Everything works fines.
  • You delete one of the previous submission (any).
  • You clear/change the briefcase directory. You pull the data again with briefcase. You try to export again - now you get an error:
    Processing instance: uuid36e43c8c-25ad-448e-a7ae-97557c78da5c

Error decrypting submission uuid36e43c8c-25ad-448e-a7ae-97557c78da5c Cause: org.opendatakit.briefcase.model.FileSystemException: Error decrypting:submission.xml.enc Cause: java.io.IOException: javax.crypto.BadPaddingException: pad block corrupted

However, the very same form is decrypted fine on both ODK Aggregate and ONA and the very same setup & manipulations - therefore it leads me to think this is not related the key manipulation, return chars, etc.

The other peculiar thing to notice is that when you fetch the data with Briefcase, it will also fetch the “deleted” instance from the server. For example, if I have 2 submissions submitted before the first successful pull. Then I delete one. The 2nd pull will still yield 2 submission fetched from Briefcase… Even if you delete form the /form folder in the Briefcase storage it doesn’t work - I’m guessing the manifest doesn’t buy it.

Hi,

Thank you for investigating this. You are correct that this problem is on our end, relating to encrypted submissions on certain kobo deployments.

Are you testing this on your own installation or one of the hosted kobo options (kobo.humanitarianresponse.info or *.kobotoolbox.org)?

We have a fix which we will be testing and deploying this week.

···

On Tuesday, December 13, 2016 at 3:41:47 PM UTC-5, Francis Vachon wrote:

Hi there,

There is a bug when you try to use an encypted form on Kobo and data has been deleted (at any point) for that form. For example:

  • You submit data to that form, pull it with briefcase, and export it. Everything works fines.
  • You delete one of the previous submission (any).
  • You clear/change the briefcase directory. You pull the data again with briefcase. You try to export again - now you get an error:
    Processing instance: uuid36e43c8c-25ad-448e-a7ae-97557c78da5c

Error decrypting submission uuid36e43c8c-25ad-448e-a7ae-97557c78da5c Cause: org.opendatakit.briefcase.model.FileSystemException: Error decrypting:submission.xml.enc Cause: java.io.IOException: javax.crypto.BadPaddingException: pad block corrupted

However, the very same form is decrypted fine on both ODK Aggregate and ONA and the very same setup & manipulations - therefore it leads me to think this is not related the key manipulation, return chars, etc.

The other peculiar thing to notice is that when you fetch the data with Briefcase, it will also fetch the “deleted” instance from the server. For example, if I have 2 submissions submitted before the first successful pull. Then I delete one. The 2nd pull will still yield 2 submission fetched from Briefcase… Even if you delete form the /form folder in the Briefcase storage it doesn’t work - I’m guessing the manifest doesn’t buy it.

Hey Alex,

On humanitarianresponse.info. Will test again later then.

Thanks for the update!

Francis

···

On Tuesday, December 13, 2016 at 4:21:04 PM UTC-5, Alex Dorey wrote:

Hi,

Thank you for investigating this. You are correct that this problem is on our end, relating to encrypted submissions on certain kobo deployments.

Are you testing this on your own installation or one of the hosted kobo options (kobo.humanitarianresponse.info or *.kobotoolbox.org)?

We have a fix which we will be testing and deploying this week.

On Tuesday, December 13, 2016 at 3:41:47 PM UTC-5, Francis Vachon wrote:

Hi there,

There is a bug when you try to use an encypted form on Kobo and data has been deleted (at any point) for that form. For example:

  • You submit data to that form, pull it with briefcase, and export it. Everything works fines.
  • You delete one of the previous submission (any).
  • You clear/change the briefcase directory. You pull the data again with briefcase. You try to export again - now you get an error:
    Processing instance: uuid36e43c8c-25ad-448e-a7ae-97557c78da5c

Error decrypting submission uuid36e43c8c-25ad-448e-a7ae-97557c78da5c Cause: org.opendatakit.briefcase.model.FileSystemException: Error decrypting:submission.xml.enc Cause: java.io.IOException: javax.crypto.BadPaddingException: pad block corrupted

However, the very same form is decrypted fine on both ODK Aggregate and ONA and the very same setup & manipulations - therefore it leads me to think this is not related the key manipulation, return chars, etc.

The other peculiar thing to notice is that when you fetch the data with Briefcase, it will also fetch the “deleted” instance from the server. For example, if I have 2 submissions submitted before the first successful pull. Then I delete one. The 2nd pull will still yield 2 submission fetched from Briefcase… Even if you delete form the /form folder in the Briefcase storage it doesn’t work - I’m guessing the manifest doesn’t buy it.

Has the fix been pushed?

···

On Monday, December 19, 2016 at 8:17:39 PM UTC+1, Francis Vachon wrote:

Hey Alex,

On humanitarianresponse.info. Will test again later then.

Thanks for the update!

Francis

On Tuesday, December 13, 2016 at 4:21:04 PM UTC-5, Alex Dorey wrote:

Hi,

Thank you for investigating this. You are correct that this problem is on our end, relating to encrypted submissions on certain kobo deployments.

Are you testing this on your own installation or one of the hosted kobo options (kobo.humanitarianresponse.info or *.kobotoolbox.org)?

We have a fix which we will be testing and deploying this week.

On Tuesday, December 13, 2016 at 3:41:47 PM UTC-5, Francis Vachon wrote:

Hi there,

There is a bug when you try to use an encypted form on Kobo and data has been deleted (at any point) for that form. For example:

  • You submit data to that form, pull it with briefcase, and export it. Everything works fines.
  • You delete one of the previous submission (any).
  • You clear/change the briefcase directory. You pull the data again with briefcase. You try to export again - now you get an error:
    Processing instance: uuid36e43c8c-25ad-448e-a7ae-97557c78da5c

Error decrypting submission uuid36e43c8c-25ad-448e-a7ae-97557c78da5c Cause: org.opendatakit.briefcase.model.FileSystemException: Error decrypting:submission.xml.enc Cause: java.io.IOException: javax.crypto.BadPaddingException: pad block corrupted

However, the very same form is decrypted fine on both ODK Aggregate and ONA and the very same setup & manipulations - therefore it leads me to think this is not related the key manipulation, return chars, etc.

The other peculiar thing to notice is that when you fetch the data with Briefcase, it will also fetch the “deleted” instance from the server. For example, if I have 2 submissions submitted before the first successful pull. Then I delete one. The 2nd pull will still yield 2 submission fetched from Briefcase… Even if you delete form the /form folder in the Briefcase storage it doesn’t work - I’m guessing the manifest doesn’t buy it.

The fix is deployed to kc.kobotoolbox.org. I will confirm in the morning that it is up on kobo.humanitarianresponse.info

···

On Wed, Jan 11, 2017 at 9:45 PM, hobok...@gmail.com wrote:

Has the fix been pushed?

On Monday, December 19, 2016 at 8:17:39 PM UTC+1, Francis Vachon wrote:

Hey Alex,

On humanitarianresponse.info. Will test again later then.

Thanks for the update!

Francis

On Tuesday, December 13, 2016 at 4:21:04 PM UTC-5, Alex Dorey wrote:

Hi,

Thank you for investigating this. You are correct that this problem is on our end, relating to encrypted submissions on certain kobo deployments.

Are you testing this on your own installation or one of the hosted kobo options (kobo.humanitarianresponse.info or *.kobotoolbox.org)?

We have a fix which we will be testing and deploying this week.

On Tuesday, December 13, 2016 at 3:41:47 PM UTC-5, Francis Vachon wrote:

Hi there,

There is a bug when you try to use an encypted form on Kobo and data has been deleted (at any point) for that form. For example:

  • You submit data to that form, pull it with briefcase, and export it. Everything works fines.
  • You delete one of the previous submission (any).
  • You clear/change the briefcase directory. You pull the data again with briefcase. You try to export again - now you get an error:
    Processing instance: uuid36e43c8c-25ad-448e-a7ae-97557c78da5c

Error decrypting submission uuid36e43c8c-25ad-448e-a7ae-97557c78da5c Cause: org.opendatakit.briefcase.model.FileSystemException: Error decrypting:submission.xml.enc Cause: java.io.IOException: javax.crypto.BadPaddingException: pad block corrupted

However, the very same form is decrypted fine on both ODK Aggregate and ONA and the very same setup & manipulations - therefore it leads me to think this is not related the key manipulation, return chars, etc.

The other peculiar thing to notice is that when you fetch the data with Briefcase, it will also fetch the “deleted” instance from the server. For example, if I have 2 submissions submitted before the first successful pull. Then I delete one. The 2nd pull will still yield 2 submission fetched from Briefcase… Even if you delete form the /form folder in the Briefcase storage it doesn’t work - I’m guessing the manifest doesn’t buy it.

You received this message because you are subscribed to the Google Groups “Kobo Users” group.

To unsubscribe from this group and stop receiving emails from it, send an email to kobo-users+unsubscribe@googlegroups.com.

To post to this group, send email to kobo-...@googlegroups.com.

Visit this group at https://groups.google.com/group/kobo-users.

For more options, visit https://groups.google.com/d/optout.

Alex Dorey
415.886.7537

In general it could be useful to link to issue threads on github, which
branch the fixes are deployed on, and are the fixes available in the latest
docker image. Thanks!