Uploading users in bulk in Django

Suggestion Box
, ,
1

At the moment, it seems like the only option is to create each user individually. For a recent project, we had to create 1,700 users; thankfully, we have the technical capacity to add the users directly into the database, but this is not accessible to most, and even for us it is not scalable. As a result, we’d love to see an option to bulk import users from a spreadsheet.

2

Can you share how you directly added the users in the database?

1 Like
3

@djole is there anything you can share about the process of adding users into the database?

4

Hi @raph, thanks for the suggestion. Is it common for you to create so many users for a project?

If you are using a self-hosted instance of Kobo, you can do this through the Django shell in the KPI container, however I don’t think we have a public API endpoint to achieve this yet.

1 Like
5

This was done by our partners on one project and we assisted them. They used Kobo Api and changed something directly in the PostgreSQL database.
It didn’t go very smoothly, we had problems until we entered all the users and their rights.

6

If you have access to the hosted server:

  • You can get into the kpi container doing:
    docker exec -it <kpi> bash
  • Then enter the Django shell:
    ./manage.py shell_plus
  • Then from inside the ipython shell, you can create a user doing the following:
    User.objects.create(username='<some new user>')
  • Or if you are creating a whole bunch of users:
for user in users:
    User.objects.create(username=user)

(Additionally you can assign permissions to those users as you create them through the shell — perhaps listing usernames and permissions in your spreadsheet, importing that and iterating through as you create and assign, etc. I can send through some sample code if that will be useful)

3 Likes
7

Yes, please, this would be very helpful.

8

Hi @djole,

Lets assume you have some projects already created and are wanting to create users and give them permissions to particular projects. You could create a spreadsheet with the following structure for creating new users (let’s call it users.csv):

username first_name last_name email
johndoe John Doe johndoe@gmail.com
… … … …

And a second sheet with your projects and permissions for each user (let’s call it permissions.csv):

username project_uid permission
johndoe qWeRtyUiOpAsDfGh view_asset
johndoe pOiKjUhdwzkvkue6h view_submissions
… … …

You could then do something like this in the kpi Django shell (if you use pandas, you’ll have to pip install it):

import pandas as pd
# we need to let Django handle the password hashing
from django.contrib.auth.hashers import make_password

users = pd.read_csv('users.csv')
permissions = pd.read_csv('permissions.csv')

# let's create some users with default passwords equal to their username
for i, row in users.iterrows():
    User.objects.create(
		username=row['username'],
		first_name=row['first_name'],
		last_name=row['last_name'],
		email=row['email'],
		password=make_password(row['username'])

# let's assign permissions
for i, row in permissions.iterrows():
    try:
        asset = Asset.objects.get(uid=row['project_uid'])
    except Asset.DoesNotExist:
        print(row['project_uid'])
        continue
    try:
        user = User.objects.get(username=row['username'])
    except User.DoesNotExist:
        print(row['username'])
        continue
    asset.assign_perm(user, row['permission'])

You can find a list of the assignable permissions here and note that some permissions are implied by others, which you can see here.

Hope that helps :slightly_smiling_face:

2 Likes
9

Thanks a lot!

1 Like
10

Hello josh

Can you please guide me Djongo integration with active directory

Please check below post

11

Hi @Mdkhamru I’ll respond on that thread.

1 Like
13

Hello

  1. Could you please let me know where should the users csv should placed
    2)Also if have multiple permission how to enter in permission csv ?
14

Hi @Mdkhamru,

  1. The CSV can be located anywhere as long as you reference the correct path to it when reading from it. In the example above, it was just in the root of the KPI directory.
  2. There is no right or wrong way to do it — my code above is merely a suggested approach to it. If you do want to follow that approach then you can just have multiple lines with the same username and project_uid but different permissions in the permissions.csv file:
username project_uid permission
johndoe aGwegnsvligsgh view_asset
johndoe aGwegnsvligsgh view_submissions
… … …
2 Likes
15

@Josh
Is there a way to add users and assign permissions from an external system using a similar method? I mean other than a clunky set of SSH commands?

16

@ks_1, yes for assigning permissions, no for creating users — you can just integrate the external system with the appropriate API endpoints. I would suggest looking at the network tab when in the KPI UI and see what endpoints are called and then just emulate that behaviour.

For creating users, you could potentially have a script that fills in the HTML sign-up form, but I haven’t tested that.

2 Likes
17

@Josh,
Is there any development plan to implement APIs to add/remove users? If not, I’m planning to get it developed from our end and will create a pull request once done.

1 Like
18

@ks_1, there’s nothing in the pipeline at the moment, but I would suggest we consult with @jnm about the approach to implementing this. It’s best to develop features that can be merged into master to serve the rest of the community rather than creating a fork that you need to maintain.

19

@ks_1, I’ve just discovered that there is in fact a way to create users if you are maintaining your own instance:

Instructions from @jnm:

  1. Go to https://kf.your.host/admin/kpi/authorizedapplication/add/ and add a new entry. The name doesn’t matter: it’s just for your own record keeping. The key is used in the next step.
  2. Once that’s done, you can POST to https://kf.your.host/authorized_application/users/ using the header Authorization: Token key_you_added and send the following fields:
  • username
  • password
  • first_name
  • last_name
  • email
  1. The user should be created in KoBo immediately, without email confirmation. See here for the code that supports this. The complete PR that added this is User creation and authentication for external apps by jnm · Pull Request #368 · kobotoolbox/kpi · GitHub
3 Likes
20

Thanks a lot @Josh! I’ll try this out with our developers.

1 Like
21

It works perfectly. :slight_smile: So adding users in bulk is now a piece of cake. Thanks @Josh and @jnm.

I also have some groups set up for different types of users, and would like to change these from APIs as well. Is there existing functionality for this using Authorized Application?

image
image1498×1031 122 KB

1 Like