Kobo-install http/https/certificate nightmare

Hello,

the past weeks we have gone through http/https/certificate nightmares with kobo-install and we still have not solved our problem.

Last year, we started with kobo-install using https and a Let’s encrypt certificate. Everything worked as expected. But without touching the system, one day, a certificate renewal request from the certbot service was rejected by Let’s encrypt, stating that we had made too many renewal requests in a too short time, so our certificates got blocked.

Then we tried using our own certificate (not self-signed) with kobo-install, so we chose “Use https” YES and “Use Let’s encrypt” No in the kobo-install setup-wizard, but the kobo containers would not startup (no errors, startup did just not finish) and we had no idea where to put our own certificates, so that the docker containers would find them.

Then - and this is our current solution - we told kobo-install to use http internally and use a https-reverse proxy in front. So we chose “Use https” NO and “Use a reverse proxy” YES.
We had to test a lot to geht this scenario working, because now we had a mix of https- and http-requests in our client/server-communication which most current browsers do not allow. But with some additional proxy-configuration we got it working.
BUT, now the problem is what @sheku_munu_kc describes in his/her post (Kobo form could not open): Now, the open button to open the enketo-form is not working and the copy button has disappeared.
This problem is not caused by the proxy, because accessing kobo via http directly (and not via the proxy) also shows the same problem. It seems to be a problem with using http in the kobo-install script.

Does anybody have similar problems? Can anybody help?

We are using the following docker image versions:

kobotoolbox/nginx                          latest 
kobotoolbox/kobocat                        2.019.52-final-shared-database
kobotoolbox/enketo-express-extra-widgets   1.77.0-jnm-grunt-workaround  
kobotoolbox/kpi                            2.019.52-final-shared-database
mongo                                      3.4    
mdillon/postgis                            9.5   
redis                                      3.2

Thanks in advance,
kobokuet

Hi @kobokuet,
Sorry to hear that. We are aware that it seems to have an error with Let’sEncrypt renewal.
Please note that it takes time to debug because Let’sEncrypt blocks after several failed attempts.

If you want to use your own certificates, you must use a reverse proxy/load balancer in front of kobo-docker containers. The certificates must be installed in the reverse proxy.
That’s what we use on production and we don’t have any mix of https and http requests.

The main important thing is to have X-Forwarded-Proto https header on the reverse proxy sent to kobo-docker frontend nginx container.

Please take a look at this response Installing own ssl certificate on Kobo and especially this topic on Git Hub. Let me know if it helps.

2 Likes

Hi @kobokuet
Do you have your installation documented ?
Cif so could you share it with me please?
thanks

@kobokuet,
kobo-docker always use HTTP internally.
If you want to use HTTPS with your own certificates, you have to say YES to use HTTPS , NO to Auto-install HTTPS...

Screen Shot 2020-03-12 at 10.32.37

1 Like

Hi @jparmer, I attach our config as screenshot from the setup wizard. I hope it is still useful, although all names and passwords are removed from the image.

@nolive As I said in my first post, we have a reverse proxy successfully setup on a different server and configured the X-Forwarded-Proto header. This is all working. Our only problem is the absence of the copy button and the open button, which simply does not work. This problem also appears when I access our kobo installation directly and not via the reverse proxy. So, I do not think it is a proxy problem, but a problem with the configuration of “Use https” [NO] in the ekobo-install script.
I linked to @sheku_munu_kc 's ticket, because the same problem is described there. Also, @yala describes this effect in his post Errors with Enketo on a local kobo-install. Our Enketo server is up and running. When I request our enketo url in a browser it answers with a german text saying something like “Enketo is up and running. Please visit your enketo forms from your KoBo Toolbos installation…”

So, what did we do wrong?

Regards,
kobokuet

@kobokuet,
As I said, Use HTTPS must be YES.
This option does not control the way KoBo works internally but whether your domain name has HTTPS endpoint or not.
In your case, I understand that it has. So you must answer YES to Use HTTPS.

In my example 8080 is the port used by reserve proxy to communicate with KoBo NGINX container.

Internet ↔ (80,443 HTTP, HTTPS) ↔ Proxy ↔ (8080 HTTP ONLY) ↔ KoBo Nginx

Hi @nolive,

I tried exactly your settings. Now, the setup script does not finish correctly but is stuck in status “Waiting for environment to be ready. It can take a few minutes.”
However, when I open the kobo url in the browser I can login and the copy button is present and the open button is working.

Can I use this environment safely although the script does not finish correctly? The log from “python run.py --logs” does not show any serious errors.

Can you tell me what does https://kf.yourdomain.tld/service_health/ say?

It says:

OK

Mongo: OK in 0.0061 seconds
Postgres: OK in 0.0042 seconds
Enketo [http://enketourl.internal]: OK in 0.469 seconds
KoBoCAT [http://kobocaturl.internal]: OK in 0.735 seconds

----BEGIN KOBOCAT RESPONSE----
OK

Mongo: OK in 0.0023 seconds
Postgres: OK in 0.255 seconds

---- END KOBOCAT RESPONSE ----

This says that everything is working as expected.
It’s weird that kobo-install is stucked at Waiting for environment to be ready. It can take a few minutes., though
If you run ./run.py --info (with all containers up&running)? What does it say?

1 Like

When executing “python run --logs” I get the message

KoBoToolbox could not start! Please try python run.py --logs to see the logs.

which doesn’t sound very good. The call of https://kf.yourdomain.tld/service_health/ still says “Everything is ok” (as described in the comment above).

In the logs I don’t see anything special related to this situation (only lots of these annoying
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)-messages, but this is off topic and doesn’t seem to be a big problem).

Any recommendations on how to handle this inconsistent state?

i am getting redis connection to rdismail.domain private:6379 failed.

1 Like

@kobokuet, https://kf.yourdomain.tld/service_health/ is the most reliable way to know if the app is up&running. kobo-install scripts use this page to validate whether the app is running.
If kobo-install returns KoBoToolbox could not start... it means it receives something else that a HTTP 200 from the health check page.

It can be several possibilities. To identify the one, would you mind to edit this file kobo-install/helpers/network.py ? Go to line 150 and change this

            response = conn.getresponse()
            return response.status
        except:
            pass

to this

            response = conn.getresponse()
            print(str(response.msg))
            return response.status
        except (httplib.HTTPException, Exception) as e:
            print('ERROR', str(e))

Then try again ./run.py --info. It should give us more info about the issue you are facing.

You can also ping (and dig) kf.yourdomain.tld from the command line of the server you are running the KoBoInstall and see if values are ok.
Then, try telnet on port 443

Successful command should be

telnet kf.yourdomain.tld 443
Trying X.X.X.X...
Connected to kf.yourdomain.tld.
Escape character is '^]'.

Failure should look like something like that

telnet: Unable to connect to remote host: Connection refused

PS: You are right. The nginx warning is not related, it’s a known issue but does not block the app to run.

1 Like

@jparmar, it’s not related, please open another issue (and check if your firewall is not blocking redis ports)

1 Like

@nolive The output of python run.py --logs is:

ERROR [Errno 111] Connection refused

KoBoToolbox could not start! Please try python run.py --logs to see the logs.

Do you have an idea what exactly the connection problem could be?
I can ping all urls from the koboinstall-server. I can telnet to all urls via port 443, but only from remote hosts, because https (443) runs via an external reverse proxy. It is not configured on the koboinstall-server itself.

What else can I check or reconfigure?

Thanks for your patience.

@kobokuet,
Connection is refused from kobo-install server to your reverse proxy.
According to what you say, telnet does not work from the kobo-install but only from external locations.
It means that you may have a firewall that blocks incoming traffic from kobo-install to proxy-server.

You can try this:

curl -I --header "Host: kf.mydomain.tld" http://127.0.0.1:8080

Where 8080 is the port used between reverse proxy and KoBo nginx
It should say something like that

HTTP/1.1 301 Moved Permanently
Server: nginx/1.10.3 (Ubuntu)
Date: Mon, 16 Mar 2020 13:40:44 GMT
Content-Type: text/html
Content-Length: 194
Connection: keep-alive
Location: https://kf.mydomain.tld/
2 Likes

What worked for me was to install kobo as a “workstation” on port 8080 and not “server” but with the public domain I’m using.
Then I setuped my own nginx for ee.kobo.mydomain, kc.kobo.mydomain, kf.kobo.mydomain, kobo.mydomain.
For each domain I setuped ssl a cerbot nginx.
Also, for each, I setuped the proxy like this:
location / {
proxy_pass http://127.0.0.1:8080;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection ‘upgrade’;
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
proxy_set_header X-Forwarded-Proto $scheme;
}
proxy_set_header X-Forwarded-Proto $scheme; is to avoid mixed content in the browser.
Voilà.
It was painful to deal with many long silent errors “Waiting another 600 seconds” before getting there.
But it feels hackish.
I hope it still works with further testing but so far so good.

I find your setup interesting. Please tell me whether you have had any issues so far with it. Many thanks in advance. Like yourself I’ve been stuck at the “Waiting for environment to be ready…” stage.

1 Like