Setting up on Kubernetes... almost there!

Kobo On Your Own Server
,
1

Hello there! :wave:

I’ve been trying to set up the full stack on Kubernetes… I’ve been struggling as the way the images work sometimes isn’t entirely in line with the “kubernetes way” - e.g. it’s mounting a lot of files, doing the templating logic inside the containers, containers are sharing volumes (meaning they need to be in the same pod), sharing ports (meaning they shouldn’t be in the same pod!), logs to file instead of stdout…

I have nevertheless made significant progress, and now have all the containers running happily, all the databases seem properly initialized (I’m not seeing any data in Mongo, but the db, collection, and index are there).

BUT…

The statics work, but neither the kpi nor the kobocat containers seem to respond to any http call. I can see that worker threads are being spawned to respond, but after a couple seconds, that worker is terminated (I think as expected?), and the http client receives en empty response.

All logs are clean, even in debug mode, except for warnings about running as root, as well as a Your models have changes that are not yet reflected in a migration, and so won't be applied..

Client call from curl:

curl http://192.168.64.2:31523 -v                                                                                                                          *   Trying 192.168.64.2...
* TCP_NODELAY set
* Connected to 192.168.64.2 (192.168.64.2) port 31523 (#0)
> GET / HTTP/1.1
> Host: 192.168.64.2:31523
> User-Agent: curl/7.64.1
> Accept: */*
>
* Empty reply from server
* Connection #0 to host 192.168.64.2 left intact
curl: (52) Empty reply from server
* Closing connection 0

Logs sample:

uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 54)
spawned uWSGI worker 1 (pid: 104, cores: 1)
*** Stats server enabled on :1818 fd: 15 ***
[2020-12-18 06:17:50,147: INFO/MainProcess] Connected to redis://:**@oaf-kobo-redismain-master:6379/1
[2020-12-18 06:17:50,201: INFO/MainProcess] mingle: searching for neighbors
[2020-12-18 06:17:50,621: INFO/MainProcess] Connected to redis://:**@oaf-kobo-redismain-master:6379/1
[2020-12-18 06:17:50,634: INFO/MainProcess] mingle: searching for neighbors
[2020-12-18 06:17:51,271: INFO/MainProcess] mingle: sync with 2 nodes
[2020-12-18 06:17:51,271: INFO/MainProcess] mingle: sync complete
[2020-12-18 06:17:51,308: INFO/MainProcess] sync_kobocat_xforms_worker@oaf-kobo-kobo-5778c8bf5d-958mt ready.
[2020-12-18 06:17:51,673: INFO/MainProcess] mingle: sync with 2 nodes
[2020-12-18 06:17:51,673: INFO/MainProcess] mingle: sync complete
[2020-12-18 06:17:51,731: INFO/MainProcess] kpi_main_worker@oaf-kobo-kobo-5778c8bf5d-958mt ready.
[2020-12-18 06:17:52,802: INFO/MainProcess] Writing entries...
[2020-12-18 06:18:11,306: INFO/MainProcess] missed heartbeat from sync_kobocat_xforms_worker@oaf-kobo-kobo-5fd5bb84df-hhx4j
[2020-12-18 06:18:11,306: INFO/MainProcess] missed heartbeat from kpi_main_worker@oaf-kobo-kobo-5fd5bb84df-hhx4j
[2020-12-18 06:18:11,687: INFO/MainProcess] missed heartbeat from sync_kobocat_xforms_worker@oaf-kobo-kobo-5fd5bb84df-hhx4j
[2020-12-18 06:18:11,687: INFO/MainProcess] missed heartbeat from kpi_main_worker@oaf-kobo-kobo-5fd5bb84df-hhx4j
[2020-12-18 06:20:53,246: INFO/MainProcess] Writing entries...
[2020-12-18 06:23:53,715: INFO/MainProcess] Writing entries...
[2020-12-18 06:26:54,165: INFO/MainProcess] Writing entries...
[2020-12-18 06:29:54,594: INFO/MainProcess] Writing entries...
spawned uWSGI worker 2 (pid: 108, cores: 1)
worker 1 killed successfully (pid: 104)
uWSGI worker 1 cheaped.

Has anyone successfully managed to run the kobo stack in Kubernetes?
Would anyone have any advice / directions to point me at to investigate? I have zero knowledge of kobo so I’m really just trying things out blindly now…

I’m happy to open-source my helm chart when I have a working version, but I might need help in the last mile here…

Many thanks! :pray:

2

OK, so I misunderstood how the nginx-uwsgi proxying worked, which caused nginx to be unable to contact the application on the kpi and kobocat containers.

This is now fixed, and I now have a working Helm chart, that I think would work on any k8s environment… just need a bit of polishing before I share / publish it…

1 Like
3

Thank you @yjouanique for sharing your experience and workarounds with the community! The community would immensely benefit from the same. :clap:

4

OK, here it goes, a first working Helm chart for KoboToolbox: https://github.com/one-acre-fund/kobo

It’s still a bit rough, but should work well on all cloud providers.

I didn’t provide very detailed instructions so you’ll need to be familiar with Helm and Kubernetes to understand how to run it. For someone familiar this should be very straightforward

1 Like
5

The repo is no longer available. Can you please share it?

6

Hi there, yes, sorry, we made the repo private after we started plugging in some of our own internal CI process…

I’ll need to work on making a public-safe version, maybe even publishing to https://artifacthub.io/ - possibly today!

7

So I’ve moved everything over to GitHub - one-acre-fund/kobo-k8s: A Helm chart for KoboToolbox now!

I have to note that I’m really no expert on KoboToolkit so there might be things I’m not doing really well - I’ve really mostly ported over the logic from kobo-install. The variables could certainly be cleaned up! But it works quite nicely for me.

Feedbacks welcome!

8

Thank you so much!

I am not an expert on kobotoolkit either but this will help tremendously in understanding how all the pieces fit together. kobo-install works but there are a lot of moving parts. I wanted to move the databases to hosted versions for better reliability but has been postponing it due to the time I would need to spend to understand the architecture.

By the way, how are you handling/planning to handle upgrades?