Enketo Express error: certificate has expired

Hi, I have an issue with one self hosted instance of KobotoolBox, when someone try to access to forms the system show the following message:

Captura de pantalla de 2021-10-01 08-24-10

And in log I can’t find which certificate is failing, all Let’s Encrypt certs are working and updated.

Thanks.

We have the similar issue as well here on self hosted KoboToolbox where the certificate still in valid period.

image
It happens since 29th of September 2021.

Same date! could you solve it?

Not yet, but apparently it has anything to do with recent Lets Encrypt Root CA change announcement DST Root CA X3 Expiration (September 2021) - Let's Encrypt

We are also having this problem. What can we do to solve this?

Still unsolved!

I try force renew letsencrypt certificates, but that does work…

I have no more ideas.

Hi @finlay, @bbatobato, @husni.mubarok can you please share your kobo-install, KoBoCAT, KPI and Enketo versions.

I tried Workaround 1 described by this OpenSSL blog post.

The result was the same: “certificate has expired”.

My KoBo instance was deployed in accordance with the instructions posted here on GitHub.

The Enketo container of my docker deployment uses OpenSSL 1.0.2g so it appears that’s where the problem lies when using Let’s Encrypt as a certificate source. That said, Workaround 1 failed so it seems like there’s more to resolving the problem than removing the expired root certificate.

It would be helpful to know more about how Enketo determines certificate validity given the failure of Workaround 1 which OpenSSL identifies as a valid fix for the problem at hand.

-Jake

It’s possible this is related to an outdated version of the Python requests module. We recently encountered a similar case where the temporary fix was to upgrade the module in KPI and restart the uwsgi processes.

cd kobo-install

# check the current version
./run.py -cf exec kpi pip freeze | grep requests 

# upgrade
./run.py -cf exec kpi pip install --upgrade requests 

# check new version
./run.py -cf exec kpi pip freeze | grep requests

# check uWSGI processes
./run.py -cf exec kpi ps fx | grep uwsgi

# restart them
./run.py -cf exec kpi sv restart uwsgi

# check that they're back up
./run.py -cf exec kpi ps fx | grep uwsgi

Thanks for the suggestion to upgrade KPI ‘requests’ package.

Original ‘requests’ version==2.10.0
Upgraded ‘requests’ version==2.26.0

End result: “certificate has expired”.

For good measure Workaround 1 was applied to the following containers: nginx, kobocat, kpi, enketo.

End result: “certificate has expired”.

Not sure where to go from here to resolve what is a Let’s Encrypt root certificate expiration problem that does not respond to conventional wisdom.

-Jake

Looks like we have the same version, my KPI also have 2.10.0 ‘request’ version but updating to 2.26.0 doesn’t fix the error.

For me look more than OpenSSL 1.0.2g / Let’s Encrypt related issue, I like to try idea:

./run.py -cf exec kpi bash

wget https://www.openssl.org/source/old/1.1.0/openssl-1.1.0g.tar.gz
tar xzvf openssl-1.1.0g.tar.gz
cd openssl-1.1.0g
./config
make
make install

ln -s /usr/local/lib64/libssl.so.1.1 /usr/lib64/libssl.so.1.1

openssl version -a

# exit, test if works, if no repeat in kobocat test again, if no repeat in enketo and test again

But now I have no time to do this, I have one production environment with a big collect running on mobiles then I’ll wait two days to try it on test environment, but if someone have time and test environment please try this idea and tell us if that works.

Thanks!

According to what you said, the GH repo you use seems to be outdated. At the first look, the latest commit on their docker.compose file is from August 2017. So I guess their version of KPI, KoBoCAT (and Enketo) are really old and still running under Python 2 (which is mainly the cause of errors with SSL). Python 2 has been deprecated since January 1, 2020. I have to admit I haven’t looked at their docker images though.

I guess you should ask them to update their versions to fix your issues.
Lots of things have changed since 2017 in kobo-docker. I’m afraid that it won’t be an easy upgrade.
If the versions they use are the ones I think about, PostgreSQL, MongoDB data need to be upgraded.

We also use 2 different databases now. We now recommend to use kobo-install to setup kobo-docker. It does handle the migration from one database to two databases (but you need to upgrade to latest version of kobo-docker with shared database first).
So long story short, there is a lot of work to be up-to-date ;-(

2 Likes

Due to deadlines I have not had the opportunity to evaluate an OpenSSL upgrade.

Please see OpenSSl 1.1.0 Changes for more information on what might go wrong with upgrading OpenSSL.

After digging into Ubuntu 16.04 which is the basis for KoBo deployment containers with OpenSSL 1.0.2g, the certificates problem was supposedly resolved by OpenSSL version 1.0.2g-1ubuntu4.20.

According to Amazon Web Services, Ubuntu 16.04 was modified through an OpenSSL package update to handle the Let’s Encrypt certificate expiration. This suggests the problem does not lie in services dependent upon Ubuntu 16.04.

I don’t know enough about how an Enketo form request flows through KoBo to isolate where the form request breaks. I’ve poked around logs here and there but didn’t manage to spot anything obvious. My hunch is that the request breaks at a service that is not Ubuntu 16.04 related.

My KoBo deployment, except for Enketo, worked before and after the Let’s Encrypt change. So I think there’s something in Enketo, probably unrelated to Ubuntu 16.04 and OpenSSL 1.0.2g-1ubuntu4.20, that is broken. What that something is is the question.

-Jake

You are correct. The GitHub repo is old. However, it made use of the latest KoBo version that was found to successfully deploy behind a proxy server with automated security certificate capability.

The docker images themselves are KoBo docker images that were relabeled for deployment stability over time.

Indeed, it was a lot of work developing what is now an outdated GitHub repo. I have not been following KoBo development over the years. If I can manage to deploy and run modern KoBo behind a proxy server I may upgrade. If it looks like the amount of effort to upgrade will equal or exceed my original effort I will put more time into solving the Enketo problem before abandoning the effort.

-Jake

FYI: kobo-install is capable of installing a proxy server in front of kobo-docker containers to automate the installation of SSL certificate with LetsEncrypt.

As I said, to upgrade from your version to latest one won’t be an easy task. You would need to:

  • Upgrade your PostgreSQL and Mongo as mentioned in my previous post.

  • Create new enviroment with kobo-install .

    1. Clone kobo-install from GitHub repository
    git clone https://github.com/kobotoolbox/kobo-install.git
    
    1. Checkout the correct branch:
    git checkout shared-database-obsolete
    
    1. Run the setup. Choose advanced options to configure all your settings. You can try to use your own proxy but ensure to set the correct headers (i.e.: proxy_set_header X-Forwarded-Proto https;). Otherwise use the one provided by kobo-install
    python3 run.py --setup
     ...
     Where do you want to install?
     [/my.new.kobo-docker.folder/]:  # Choose a different location than the one you are currently using
     Do you want to see advanced options?
      1) Yes
      2) No
     [2]: 1
     What kind of installation do you need?
      1) On your workstation
      2) On a server
     [1]: 2
     ...
     Do you want to use HTTPS?
      1) Yes
      2) No
     [2]: 1
     ...
     Auto-install HTTPS certificates with Let's Encrypt?
      1) Yes
      2) No - Use my own reserve-proxy/load-balancer
     [2]: 1
     ...
     Email address for Let's Encrypt: xxx@xxx.com
     ...
    
    1. Kill the process as soon as when you see Launching environment
    2. Copy your PostgreSQL data to /my.new.kobo-docker.folder/.vols/db
    3. Copy your MongoDB data to /my.new.kobo-docker.folder/.vols/mongo
    4. Copy also kobocat_media_uploads and kpi_medias to /my.new.kobo-docker.folder/.vols/
    5. Run kobo-install
    python run.py 
    

    Wait for the app to be ready

    Waiting for environment to be ready. It can take a few minutes.
     ...
    ╔═════════════════════════════════╗
    ║ Ready                           ║
    
    
  • Upgrade to latest version of kobo-docker (as of today 2.021.41a)

    python run.py --stop
    git checkout 2.021.41a
    python run.py --update
    

I cannot guarantee it will work right out-of-the-box since I have not tested upgrading from your version but it gives you a good idea where to go.

Obviously, I strongly recommend to backups your data before doing the migration.

Oops. Did not realize that you were the owner on that repo :wink: