Enketo error - Socket hang up

Kobo On Your Own Server
, ,
1

We have an issue on standalone KoBo server. Loading form produces

image
image822×331 6.4 KB

…and there is a corresponding log entry:

enketo_express_1 | 17:36:34 0|enketo | Unknown Error occurred during attempt to update cache { Error: socket hang up
enketo_express_1 | 17:36:34 0|enketo | at createHangUpError (_http_client.js:331:15)
enketo_express_1 | 17:36:34 0|enketo | at TLSSocket.socketOnEnd (_http_client.js:423:23)
enketo_express_1 | 17:36:34 0|enketo | at emitNone (events.js:111:20)
enketo_express_1 | 17:36:34 0|enketo | at TLSSocket.emit (events.js:208:7)
enketo_express_1 | 17:36:34 0|enketo | at endReadableNT (_stream_readable.js:1064:12)
enketo_express_1 | 17:36:34 0|enketo | at args.(anonymous function) (/usr/lib/node_modules/pm2/node_modules/event-loop-inspector/index.js:138:29)
enketo_express_1 | 17:36:34 0|enketo | at _combinedTickCallback (internal/process/next_tick.js:139:11)
enketo_express_1 | 17:36:34 0|enketo | at process._tickCallback (internal/process/next_tick.js:181:9)
enketo_express_1 | 17:36:34 0|enketo | code: ‘ECONNRESET’ }
m

Any Ideas how to fix this? It is not a new installation, it has been working fine before this started to happen.

Help greatly appreaciated.

Best,

2

Hi @maric.vladimir,

Have you tried submitting your data with KoBoCollect android app? Does it work for you or do you see an error message when submitting it through KoBoCollect android app as well?

Have a great day!

3

HI @Kal_Lam,

It turned out that the spikes of load on Enketo were causing this, nothing more complicated. We have a scenario of extensive API usage from multiple users at the same time.

I presume that separating frontend and backend will help? I am asking this seemingly naive question because the server in trouble has plenty of resources - CPUs in double digits etc, and yet - when the Enketo’s app.js takes 200% each, and there is a number of them, the load spirals out of control. Will separating DBs make any difference?

Thanks,

4

Never seen socket hang up on any production instance, even kf/kc/ee.kobotoolbox.org, which until a couple weeks ago all ran on a single, 4-CPU VM.

You can distribute Enketo (and KoBoCAT and KPI) across multiple front-end machines, or simply have these Node.js and Django applications run on a single, separate machine, while the database servers (Redis, PostgreSQL, MongoDB) run on another.

Unknown Error occurred during attempt to update cache sounds like Enketo’s having a problem connecting to the Redis cache instance. Is it just app.js working hard? Are there no Redis issues, like high CPU consumption by redis-server? Are there as many app.jses as your double-digit CPU cores, and are they exceeding the maximum number of open files or maximum number of connections allowed by Redis? Are your disks fast / is there a lot of iowait?

This is more Linux administration stuff than KoBo stuff :wink: Good luck!

2 Likes
5

It appears - yet to be confirmed - that nofiles was the issue.

Thanks for the great lead!

Best,

3 Likes