Uploading users in bulk in Django

[Josh]

Hi @ks_1, that’s great! I’m not aware of an endpoint that’s exposed to do that, but I’ll leave that open to @jnm to comment. It looks like you can manage this sort of thing just through the existing permissions endpoints as a work-around? You could have a spreadsheet or whatever that links permissions to groups and then assign those to users. Team management is something that’s in the pipeline, so hopefully you can manage this without a work-around soon.

[jnm]

We haven’t done anything with Django’s group features yet.

[Mdkhamru]

Hello @Josh

Can we update bulk users’ passwords for existing users?

[Josh]

Hi @Mdkhamru, you can refer to the Django docs for this:

https://docs.djangoproject.com/en/dev/topics/auth/default/#user-objects

[Mdkhamru]

@Josh thanks

[ks_1]

@Josh

@ks_1, I’ve just discovered that there is in fact a way to create users if you are maintaining your own instance:

Instructions from @jnm:

1. Go to https://kf.your.host/admin/kpi/authorizedapplication/add/ and add a new entry. The name doesn’t matter: it’s just for your own record keeping. The key is used in the next step.
2. Once that’s done, you can POST to https://kf.your.host/authorized_application/users/ using the header Authorization: Token key_you_added and send the following fields:

• username
• password
• first_name
• last_name
• email

1. The user should be created in KoBo immediately, without email confirmation. See here for the code that supports this. The complete PR that added this is User creation and authentication for external apps by jnm · Pull Request #368 · kobotoolbox/kpi · GitHub

We always faced some credential issues when using this method, only now did we realize the problem…
When creating users through this method, the credentials work on KPI, but they don’t work on KC (although the users are created there as well). Users can log into kf.url, but if they try to sign in on ODK Collect using the KC endpoint, the credentials don’t work.
The only way to get it to work is to go to kf.url/admin and reset the passwords for these users.

I think this is a bug.

[Mdkhamru]

ya right even some times after password reset they are still not able to login kobo collect and credentials works only in web

[jnm]

Thanks for reporting this issue. It looks like a workaround is to log in to KPI first with the newly created user’s credentials, for example:

john@world:~$ curl --user special_api_user:not_very_secret http://kf.kobo.local:9000/me/; echo
{"username":"special_api_user","first_name":"","last_name":"","email":"foo@bar.baz","server_time":"2021-11-18T01:41:37Z","date_joined":"2021-11-18T01:40:48Z","projects_url":"http://kc.kobo.local:9000/special_api_user","is_superuser":false,"gravatar":"https://www.gravatar.com/avatar/9050f8ea6abf7524e8ca40107fc8b00c?s=40","is_staff":false,"last_login":null,"extra_details":{"require_auth":false},"git_rev":false}

After that, the KoBoCAT OpenRosa endpoints, which use digest authentication, should work for the new user. I’ve filed a bug; please check the reproduction steps and add comments if your experience differs.

[ks_1]

@jnm
We’ve edited the v2 KPI endpoints to create users and edit permissions (also all other parameters) using superuser credentials… Can I directly submit a pull request?

[jnm]

Thanks for the pull request. Did you intend to expose the usernames, names, email addresses, etc. to anyone who accessed the endpoint, including anonymous users and non-superusers? You also seem to remove an entire method, migrate(), for no apparent reason.

I would request that you please review the checklist in the PR template (none of the boxes are checked). Unit tests are absolutely mandatory for a change like this, and the unit tests should illustrate what you expect the API to do. For example, I would be able to tell if you intentionally wanted to give the general public that level of access to your user list if there were a unit test—and if you didn’t, the unit test would’ve caught the mistake.

Thanks again.

[adreyan]

Does this code only work in kobo?

[olenap]

@raph

At the moment, it seems like the only option is to create each user individually. For a recent project, we had to create 1,700 users; thankfully, we have the technical capacity to add the users directly into the database, but this is not accessible to most, and even for us it is not scalable. As a result, we’d love to see an option to bulk import users from a spreadsheet.

how you add new users

[ks_1]

Thanks for the feedback and sorry I couldn’t respond earlier. Admittedly it was a quickly put-together hack from our end and we didn’t do much testing. We did make some changes later to prevent visibility of all users publicly.
Frankly we’re running quite busy these days, but I’ll be sure to edit the PR with better code in the near future.

[kpk_in]

This is a slightly old post, but I find this fully matches my requirement:
I am new to kobotoolbox and docker. But we have successfully installed and configured kobotoolbox server, which is working fine.
We have a requirement to create 1000s of users for using the forms.
As per the above answer we can access the kpi container to run python scripts.
I couldn’t see any explanation on kpi container anywhere else… Can you provide a little more explanation on this, including how to obtain <kpi> value to be used in the command

docker exec -it <kpi> bash

Edit:
I found that out: I should have looked at the docker documentation!
Running the command

sudo docker ps

will list all the containers, and then using the container-id for the <kpi> value in docker exec -it <kpi> bash will take you to the required shell for running python scripts.

[kpk_in]

I have used the solution by @Josh here above to upload thousands of users to our self-hosted server. Then we face two issues:

1. The user’s email is not ‘verified’ and hence cannot login. (We are using a common email for all the thousands of users since they are unable to go through an email creation process.)

2. When trying to verify the user’s email through the super_admin-tool ([mydomain]/admin/account/emailaddress/), the users are not listed on this page until the user tries to login!

Is there any way to automatically verify the user’s email also when uploading the user-list in csv format?

Or to enable listing of the user in the /admin/account/emailaddress/ page immediately upon creating the user through bulk-upload?

[kpk_in]

Activating bulk uploaded users

I have used the solution by @Josh [here: uploading users in bulk] to upload thousands of users to our self-hosted server using python script and csv-file. Then we observed the following :

1. The user cannot login to any form with permission granted to the users. But we can see through the super_user’s admin-page that the users are listed under the [AUTHENTICATION AND AUTHORIZATION] > [Users] (kf.[mydomain]/admin/auth/user/)
2. If the user tries to login to kf.[mydomain]/account/login/, we see a page Confirm email address. Please click the activation link in the email just sent to you. and a verification email is sent, and only then the user gets listed under [ACCOUTS] > [Email addresses] (/admin/account/emailaddress/)
3. Now the super_user can verify the user’s email through the above admin-interface, or the user can click the link received in his/her email.
4. Again the user has to login to kf.[mydomain]/account/login/ before the form login can succeed (form login is usually at ee.[mydomain]/x/[form_id])
5. Only after completing the above steps, user can access the form for which permission was granted.

So my questions are:

• Is there any way to automatically verify the user’s email also when uploading the user-list in csv format?
• What would be python script for this?
• OR is there any way to enable listing of the user in the /admin/account/emailaddress/ page immediately upon creating the user through bulk-upload?
• But in this case too, the user has to login once again to kf.[mydomain]/account/login before the user can access the form.

[[Right now what I am doing is:

1. Running a script in Selenium-IDE to login all the users one by one at kf.[mydomain]/account/login (the csv-file with username and password is used for this). This generates the message at ‘2’ above.
2. Now the users get listed on the /admin/account/emailaddress/ admin-page of super_user, and all the users are selected and marked as email-address verified
3. Again run a script in Selenium-IDE to login all the users one by one at kf.[mydomain]/account/login. During this login the user actually logs in to his account.
4. The forms can now be accessed.

But this is a time consuming process with the Selenium-script taking about 1 hour for 300 users! (Over 3 hours for 1000 users and about 20 hours for 7000 users!) Hence the request above for a python script or api based solution. ]]