Nginx Reverse Proxy - Infinite Redirect

roemhildtg · May 20, 2020, 8:38pm

Description

I’m running into an issue where https://kf.domain.com/ is redirecting endlessly to the same address.
Related :#54
My architecture is this:

AWS Load balancer (443:https offloading) -> Nginx Reverse proxy (http:80) -> kobo tools docker (http:8080)

I’ve configured Kobo tools with SSL On and this happens. With SSL Off, i run into some other issues - like redirecting to http.

I am passing these relevant headers via nginx:

add_header       X-Served-By $host;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Scheme $scheme;
proxy_set_header X-Forwarded-Proto  $scheme;
proxy_set_header X-Forwarded-For    $remote_addr;
proxy_pass       $forward_scheme://$server:$port;

Here is my config:

[xxx kobo-install]$ python3 run.py --setup
╔═══════════════════════════════════════════════════════════════╗
║ Welcome to KoBoInstall!                                       ║
║                                                               ║
║ You are going to be asked some questions that will            ║
║ determine how to build the configuration of `KoBoToolBox`.    ║
║                                                               ║
║ Some questions already have default values (within brackets). ║
║ Just press `enter` to accept the default value or enter `-`   ║
║ to remove previously entered value.                           ║
║ Otherwise choose between choices or type your answer.         ║
╚═══════════════════════════════════════════════════════════════╝
Where do you want to install?
[/apps/kobo-docker]:
Please confirm path [/apps/kobo-docker]
        1) Yes
        2) No
[1]:
Do you want to see advanced options?
        1) Yes
        2) No
[1]:
What kind of installation do you need?
        1) On your workstation
        2) On a server
[2]:
Please choose which network interface you want to use?
        eth0) xxx
        other) Other
[eth0]:
Do you want to use separate servers for frontend and backend?
        1) Yes
        2) No
[2]:
Public domain name [xxx.com]:
KPI sub domain [kf]:
KoBoCat sub domain [kc]:
Enketo Express sub domain name [ee]:
Do you want to use HTTPS?
        1) Yes
        2) No
[2]: 1
╔════════════════════════════════════════════════════════════════════╗
║ Please note that certificates must be installed on a reverse-proxy ║
║ or a load balancer.                                                ║
║ KoBoInstall can install one, if needed.                            ║
╚════════════════════════════════════════════════════════════════════╝
Auto-install HTTPS certificates with Let's Encrypt?
        1) Yes
        2) No - Use my own reserve-proxy/load-balancer
[2]: 2
Is your reverse-proxy/load-balancer installed on this server?
        1) Yes
        2) No
[1]: 2
Internal port used by reverse proxy?
[8080]:
SMTP server [xxx]:
SMTP port [587]:
SMTP user [xxx]:
SMTP password [BE/xxx]:
Use TLS?
        1) True
        2) False
[2]:
From email address [xxx]:
Super user's username [xxx]:
Super user's password [xxx]:
Docker Compose prefix? (leave empty for default):
Staging mode?
        1) Yes
        2) No
[2]:
KoBoCat PostgreSQL database name?
[xxx]:
KPI PostgreSQL database name?
[xxx]:
PostgreSQL user's username?
[xxx]:
PostgreSQL user's password?
[xxx]:
Do you want to tweak PostgreSQL settings?
        1) Yes
        2) No
[2]:

MongoDB root's username?
[xxx]: MongoDB root's password?
[xxxx]:
MongoDB user's username?
[xxx]:
MongoDB user's password?
[xxx]:
Redis password?
[xxx]:
Do you want to expose backend container ports (`PostgreSQL`, `MongoDB`, `redis`) ?
        1) Yes
        2) No
[1]:
╔═════════════════════════════════════════════════╗
║ WARNING! When exposing backend container ports, ║
║ it's STRONGLY recommended to use a firewall to  ║
║ grant access to frontend containers only.       ║
╚═════════════════════════════════════════════════╝
Do you want to customize service ports?
        1) Yes
        2) No
[2]:
Do you want to use AWS S3 storage?
        1) Yes
        2) No
[2]:
Google Analytics Identifier:
Google API Key:
Do you want to use Sentry?
        1) Yes
        2) No
[2]:
Do you want to tweak uWSGI settings?
        1) Yes
        2) No
[2]:
Do you want to activate backups?
        1) Yes
        2) No
[2]:

Steps to Reproduce

I listed the example I have above. Not sure if this exact scenario is easily reproducible or if its something I can easily fix.

Additional context

stephenoduor · May 21, 2020, 6:09am

@roemhildtg did you configure the domain name with the subdomains?

roemhildtg · May 21, 2020, 1:38pm

Yep I have all three subdomains being routed the same way through that ELB -> NGinx proxy -> Kobo machine.

nolive · June 2, 2020, 1:59am

Hi @roemhildtg,

Can you tell why you are using another NGINX proxy between the ELB and the Kobo Toolbox?
Do you have others app running on the same server? If you are using only KoBo Toolbox, you could get rid of it.

Can you try to force the X-Forwarded-Proto and X-Forwarded-Scheme to https ?
On your NGINX proxy, I would think that $scheme equals http which causes your endless loop within kobo-docker NGINX configuration.

roemhildtg · June 2, 2020, 2:07pm

Hi nolive -

The reason is that we have one elb to perform https offloading in our environment (its easy to manage certs in aws cert manager that way). And to avoid having to pay for multiple ELB load balancers for every app we run. This load balancer just forwards traffic to our nginx proxy where we can easily reverse proxy traffic within our network.

There may have been a better way to set this up but it seems to work for most apps so far and has saved a lot of money in ELB costs.

I’ll give that a shot!

nolive · June 2, 2020, 2:19pm

Indeed. That’s a good reason

Let me know it works.

roemhildtg · June 2, 2020, 2:30pm

Okay - it looks like I did try this. I have this set in the nginx proxy config manager: this sweet app

nolive · June 2, 2020, 7:35pm

I would try with https instead of $forward_scheme

nolive · June 3, 2020, 6:45pm

You can also try to add
proxy_set_header HTTP_X_FORWARDED_PROTO https;

roemhildtg · June 4, 2020, 4:04pm

Okay - I tried both of these and I’m still getting the redirect. I’m going to try looking into the debug mode mentioned in a different message.

Just wanted to double check, this looks correct also:

nolive · June 4, 2020, 4:52pm

You can try to comment out the redirection inside the KoBo NGINX container.

github.com

kobotoolbox/kobo-docker/blob/master/nginx/include.https_redirection.conf.tmpl#L37


}

# `http://` KC URLs are somehow getting into the EE Redis, and then EE
# can't submit due to receiving a 301 on POST. Temporarily bypass
# redirection for submissions
if ($request_uri ~ "/submission") {
    set $redirect_https "NOPE";
}

if ($redirect_https = "OK") {
    return 301 https://$server_name$request_uri;
}

We never tested with a configuration like yours so I don’t know whether you’ll face other issues.

roemhildtg · June 4, 2020, 5:18pm

Yep that definitely fixed it. For kf* anyways

Commenting out this works too:

    #if ($http_x_forwarded_proto != "https") {
    #    set $redirect_https "${redirect_https}K";
    #}

roemhildtg · June 4, 2020, 5:42pm

Have you seen this error before?

ConnectionError at /me/

HTTPSConnectionPool(host=‘kc.wsbeng.com’, port=443): Max retries exceeded with url: /api/v1/user (Caused by NewConnectionError(’<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0d30d12e20>: Failed to establish a new connection: [Errno -2] Name or service not known’))

Update:
Okay looks like the kpi endpoints aren’t working quite right either. This error is being thrown due to this. Looks like
https://kf.wsbeng.com/kobocat/ -> Redirects to https://kc.wsbeng.com

And https://kc.wsbeng.com -> redirects to https://kf.wsbeng.com/accounts/login/?next=/kobocat/#/

nolive · June 5, 2020, 12:57pm

@roemhildtg. Oh it looks like a bug we fixed.
Please pull the latest version of kobo-install and run ./run.py --update. We’ve just released a new version (2.2.1) that fixes a sessions issue on KC. (When the session is invalid, it redirects to KPI).

roemhildtg · June 5, 2020, 4:30pm

Hmm -okay yep that did fix that issue. Now I’m back to solving the /me/ endpoint not working. Maybe it has something to do with me commenting out the stuff in nginx so I am troubleshooting the header issue.

Looks like the app I’m using to manage nginx had an override that was setting:

proxy_set_header X-Forwarded-Proto $scheme;

I’m not entirely sure why EC2 ELB was setting the scheme to http but I changed this to https in the config and kobo tools immediately started working correctly.

If anyone has a suggestion on getting ELB to pass the correct scheme, I’d love to hear it. My setup looks like this in ELB:

ELB HTTPS (443) -> NGINX Proxy (80)

I haven’t been able to get it working with port 443 on nginx.

nolive · June 8, 2020, 1:26pm

FYI,
I know it won’t help you but, as I said, we don’t use an intermediate NGINX between our ELB and kobo-docker NGINX. We leave the HTTPS configuration on the ELB and use the same setup e.g.:
ELB HTTPS (443) -> NGINX Proxy (80)
We don’t have to install (maintain) certificates on the front-end containers. It’s easier for use auto-scale group too.

Kal_Lam · June 16, 2020, 5:10am

2 posts were split to a new topic: OPEN not working while trying to collect data

roemhildtg · June 15, 2020, 8:37pm

Okay - All is good with Kobo. Almost

I have it working with the reverse proxy by manually forcing that property in my reverse proxy intermediary.

The one issue I’m running into - is probably user error, but I’m not seeing anything obvious. I uploaded a form (Excel spreadsheet) and when I click “Open”, it doesn’t do anything. Preview works fine, form editing works fine, just the “Open” button isn’t working:

nolive · June 15, 2020, 9:57pm

It means that Enketo is not up&running, it takes a while the first time.
Please check at https://kf.wsbeng.com/service_health/, you’ll see whether enketo is ok or not.

roemhildtg · June 16, 2020, 12:59pm

My service health looks like this:

OK KPI

Mongo: OK in 0.026 seconds
Postgres: OK in 0.0134 seconds
Enketo [http://ee.wsbeng.internal]: OK in 1.15 seconds
KoBoCAT [http://kc.wsbeng.internal]: OK in 1.52 seconds

----BEGIN KOBOCAT RESPONSE----
OK

Mongo: OK in 0.0165 seconds
Postgres: OK in 0.0111 seconds

---- END KOBOCAT RESPONSE ----

Everything appears okay, no? I still have no action when clicking the open button though. There aren’t any errors in the console, and I haven’t seen any network errors.