Import and export failures on non-humanitarian server

Announcements
, ,
1

Hello, everyone, and thank you for your patience. My colleague @nolive and I have been working (and continue to work) this morning to resolve the issue on kf.kobotoolbox.org that has been preventing most imports and exports from completing successfully. This has also delayed the POSTing of most submissions to external REST Services servers.

A brief explanation: requests made by a browser need to return in a relatively short amount of time, but not all tasks can complete that quickly. Some examples include:

  • Clicking the export button, where the server needs to acknowledge quickly that the export request was received, but the export itself may take up to half an hour for a large data set.
  • Receiving a submission, where the server needs to store the data and acknowledge its receipt immediately, but also needs to send a copy to the appropriate REST Services external servers, which may be slow or require multiple retries.

Our infrastructure currently has one queue for all such tasks, with many worker programs running simultaneously to complete any jobs that appear in that queue. The issue, in this case, was a very large project (over 7 million submissions) with a very slow (or even unresponsive) REST Services external server overwhelming the workers and causing the queue to be filled with over 70,000 tasks. This effectively crowded out everyone else’s tasks, manifesting as the import and export failures that you’ve likely seen if you use kf.kobotoolbox.org.

We are working first to get through the backlog of tasks, which should be done in the next 15-20 minutes (current time is 15:13 UTC on 8 September) [update: this completed on schedule; all service should now be normal]. Following that, we will urgently work to separate the queues so that any future REST Services issue does not impact other tasks like imports and exports. Once that is complete, we will work on a fairness algorithm to make sure one REST Service cannot dominate the queue and prevent all other REST Services from working properly.

Thanks again,

John Milner
Lead Developer

5 Likes
2

Hello,
Sorry, but how can 7 million happen from a (private) user project? Should we better restrict such level of free extensive usage and focus on support for the humanitarian projects, please?

Our own server (http://kf.kobotoolbox.org ) can be used by everyone else, but we ask each user to remain below the limit of 10,000 submissions as well as 5GB file uploads per user per month. We might increase this number in the future, but for the time being we want to avoid that a small number of very heavy users are slowing down the server and user experience for everyone else.
https://community.kobotoolbox.org/t/usage-of-kobo-for-a-busines/445/2
See Help Center article also
https://support.kobotoolbox.org/server.html?highlight=server%20use

Furthermore, in my opinion, users with such exhaustive demands should use their own server instead of stressing common community ressources and KoBo staff.

3

Kudos to you @jnm, @nolive, and all of the team :clap: :clap:

2 Likes
4

@jnm we sincerely apologies for this outage; we were unable to detect the issues immediately as it was the first time this was happening; we have been able to resolve the issue on our servers and promise it wont happen again,

1 Like
5

Hello @leksyde,
Would you mind to also explain to the community, please

  • why are you using the KoBo kc server for such amount of data (normal limit 10 000)?
  • why are you not using your own server directly where you seem to copy almost all submissions by REST service, see https://community.kobotoolbox.org/t/kobo-rest-integration/26066
  • if you will change this situation and comply to the suggested KoBo limits of. “10,000 submissions as well as 5GB file uploads per user per month”?
6

hello @wroos sincere apologies to everyone and the community;
we are currently working on deploying to our own servers soon and would be migrating soon; the project is currently being used to collect vital COVID-19 related data from the field (for which kobo made some allowance for)

Sincere apologies once again.
@jnm sincere apologies we got the email and have responded.

well done @jnm @nolive and all of the team

2 Likes
8

This queue separation is now complete; see Release Notes - version 2.022.24 and related patch releases - #10 by nolive.

1 Like
9

Hi @wroos,

Thanks for being protective of the time and energies of the community and core team. The following point is valid:

The blanket COVID-related quota-exception offering ended as of March 28, 2022, and owners of future projects planning to exceed the limits of kf.kobotoolbox.org do need to contact support@kobotoolbox.org in advance to set up an arrangement.

For what it’s worth, the REST Services queue issue has presented itself (to a lesser extent) on kobo.humanitarianresponse.info as well, and we’d already had queue separation on our list of necessary improvements. The priority of that task got an abrupt boost!

Best regards to all.

3 Likes