Hi, I have an issue with one self hosted instance of KobotoolBox, when someone try to access to forms the system show the following message:
And in log I can’t find which certificate is failing, all Let’s Encrypt certs are working and updated.
Thanks.
We have the similar issue as well here on self hosted KoboToolbox where the certificate still in valid period.
It happens since 29th of September 2021.
Same date! could you solve it?
Not yet, but apparently it has anything to do with recent Lets Encrypt Root CA change announcement DST Root CA X3 Expiration (September 2021) - Let's Encrypt
We are also having this problem. What can we do to solve this?
Still unsolved!
I try force renew letsencrypt certificates, but that does work…
I have no more ideas.
Hi @finlay, @bbatobato, @husni.mubarok can you please share your kobo-install, KoBoCAT, KPI and Enketo versions.
I tried Workaround 1 described by this OpenSSL blog post.
The result was the same: “certificate has expired”.
My KoBo instance was deployed in accordance with the instructions posted here on GitHub.
The Enketo container of my docker deployment uses OpenSSL 1.0.2g so it appears that’s where the problem lies when using Let’s Encrypt as a certificate source. That said, Workaround 1 failed so it seems like there’s more to resolving the problem than removing the expired root certificate.
It would be helpful to know more about how Enketo determines certificate validity given the failure of Workaround 1 which OpenSSL identifies as a valid fix for the problem at hand.
-Jake
It’s possible this is related to an outdated version of the Python requests module. We recently encountered a similar case where the temporary fix was to upgrade the module in KPI and restart the uwsgi processes.
cd kobo-install
# check the current version
./run.py -cf exec kpi pip freeze | grep requests
# upgrade
./run.py -cf exec kpi pip install --upgrade requests
# check new version
./run.py -cf exec kpi pip freeze | grep requests
# check uWSGI processes
./run.py -cf exec kpi ps fx | grep uwsgi
# restart them
./run.py -cf exec kpi sv restart uwsgi
# check that they're back up
./run.py -cf exec kpi ps fx | grep uwsgi
Thanks for the suggestion to upgrade KPI ‘requests’ package.
Original ‘requests’ version==2.10.0
Upgraded ‘requests’ version==2.26.0
End result: “certificate has expired”.
For good measure Workaround 1 was applied to the following containers: nginx, kobocat, kpi, enketo.
End result: “certificate has expired”.
Not sure where to go from here to resolve what is a Let’s Encrypt root certificate expiration problem that does not respond to conventional wisdom.
-Jake
Looks like we have the same version, my KPI also have 2.10.0 ‘request’ version but updating to 2.26.0 doesn’t fix the error.
For me look more than OpenSSL 1.0.2g / Let’s Encrypt related issue, I like to try idea:
./run.py -cf exec kpi bash
wget https://www.openssl.org/source/old/1.1.0/openssl-1.1.0g.tar.gz
tar xzvf openssl-1.1.0g.tar.gz
cd openssl-1.1.0g
./config
make
make install
ln -s /usr/local/lib64/libssl.so.1.1 /usr/lib64/libssl.so.1.1
openssl version -a
# exit, test if works, if no repeat in kobocat test again, if no repeat in enketo and test again
But now I have no time to do this, I have one production environment with a big collect running on mobiles then I’ll wait two days to try it on test environment, but if someone have time and test environment please try this idea and tell us if that works.
Thanks!
According to what you said, the GH repo you use seems to be outdated. At the first look, the latest commit on their docker.compose file is from August 2017. So I guess their version of KPI, KoBoCAT (and Enketo) are really old and still running under Python 2 (which is mainly the cause of errors with SSL). Python 2 has been deprecated since January 1, 2020. I have to admit I haven’t looked at their docker images though.
I guess you should ask them to update their versions to fix your issues.
Lots of things have changed since 2017 in kobo-docker. I’m afraid that it won’t be an easy upgrade.
If the versions they use are the ones I think about, PostgreSQL, MongoDB data need to be upgraded.
We also use 2 different databases now. We now recommend to use kobo-install to setup kobo-docker. It does handle the migration from one database to two databases (but you need to upgrade to latest version of kobo-docker with shared database first).
So long story short, there is a lot of work to be up-to-date ;-(
Due to deadlines I have not had the opportunity to evaluate an OpenSSL upgrade.
Please see OpenSSl 1.1.0 Changes for more information on what might go wrong with upgrading OpenSSL.
After digging into Ubuntu 16.04 which is the basis for KoBo deployment containers with OpenSSL 1.0.2g, the certificates problem was supposedly resolved by OpenSSL version 1.0.2g-1ubuntu4.20.
According to Amazon Web Services, Ubuntu 16.04 was modified through an OpenSSL package update to handle the Let’s Encrypt certificate expiration. This suggests the problem does not lie in services dependent upon Ubuntu 16.04.
I don’t know enough about how an Enketo form request flows through KoBo to isolate where the form request breaks. I’ve poked around logs here and there but didn’t manage to spot anything obvious. My hunch is that the request breaks at a service that is not Ubuntu 16.04 related.
My KoBo deployment, except for Enketo, worked before and after the Let’s Encrypt change. So I think there’s something in Enketo, probably unrelated to Ubuntu 16.04 and OpenSSL 1.0.2g-1ubuntu4.20, that is broken. What that something is is the question.
-Jake
You are correct. The GitHub repo is old. However, it made use of the latest KoBo version that was found to successfully deploy behind a proxy server with automated security certificate capability.
The docker images themselves are KoBo docker images that were relabeled for deployment stability over time.
Indeed, it was a lot of work developing what is now an outdated GitHub repo. I have not been following KoBo development over the years. If I can manage to deploy and run modern KoBo behind a proxy server I may upgrade. If it looks like the amount of effort to upgrade will equal or exceed my original effort I will put more time into solving the Enketo problem before abandoning the effort.
-Jake
FYI: kobo-install is capable of installing a proxy server in front of kobo-docker containers to automate the installation of SSL certificate with LetsEncrypt.
As I said, to upgrade from your version to latest one won’t be an easy task. You would need to:
Upgrade your PostgreSQL and Mongo as mentioned in my previous post.
Create new enviroment with kobo-install .
git clone https://github.com/kobotoolbox/kobo-install.git
git checkout shared-database-obsolete
proxy_set_header X-Forwarded-Proto https;). Otherwise use the one provided by kobo-installpython3 run.py --setup
...
Where do you want to install?
[/my.new.kobo-docker.folder/]: # Choose a different location than the one you are currently using
Do you want to see advanced options?
1) Yes
2) No
[2]: 1
What kind of installation do you need?
1) On your workstation
2) On a server
[1]: 2
...
Do you want to use HTTPS?
1) Yes
2) No
[2]: 1
...
Auto-install HTTPS certificates with Let's Encrypt?
1) Yes
2) No - Use my own reserve-proxy/load-balancer
[2]: 1
...
Email address for Let's Encrypt: xxx@xxx.com
...
Launching environment
/my.new.kobo-docker.folder/.vols/db
/my.new.kobo-docker.folder/.vols/mongo
kobocat_media_uploads and kpi_medias to /my.new.kobo-docker.folder/.vols/
python run.py
Wait for the app to be ready
Waiting for environment to be ready. It can take a few minutes.
...
╔═════════════════════════════════╗
║ Ready ║
Upgrade to latest version of kobo-docker (as of today 2.021.41a)
python run.py --stop
git checkout 2.021.41a
python run.py --update
I cannot guarantee it will work right out-of-the-box since I have not tested upgrading from your version but it gives you a good idea where to go.
Obviously, I strongly recommend to backups your data before doing the migration.
Oops. Did not realize that you were the owner on that repo 
As this is the only mention of how this Root certificate expiry affects enketo, thought to share what ultimately solved it for me. As per this SO answer, the root certificates are hardcoded in the nodejs source code, so it won’t use these from the OS unless the node process is started with the --use-openssl-ca flag. Quick, dirty, and dangerous fix is to set the NODE_TLS_REJECT_UNAUTHORIZED environment variable to 0
Thanks @punkch for sharing this information, could you be more detailed about how to use it on our case or/and how you solve the issue?
Thanks I really appreciate your help.
Hi @finlay
Quick but dangerous (enketo won’t verify the SSL certificates for any connections) solution:
set environment variable on the enketo-express container NODE_TLS_REJECT_UNAUTHORIZED="0"
the node.js command, that starts enketo-express should include --use-openssl-ca argument, and the docker image to have up to date openssl client.
Thanks @punkch
I try first quick and dangerous doing this:
Adding the NODE_TLS_REJECT_UNAUTHORIZED=0 at the end of /opt/kobo-deployments/envfile.txt and finally works, awesome!!
About the second options I don’t know where or how change the enketo’s nodejs starting script, if you could give me a clue I’ll really appreciate that.
Thanks again.
