Can I push data, which have been pulled from Aggregate?

[imran3may]

Hello Friends,

I am trying to restore data backed up from the ODK Aggregate server using ODK briefcase? I cannot see any straightforward way. Is there any workaround? Our project previously used Aggregate server and planning to move to KoBo as Aggregate is no longer supported by ODK.

Support will be highly appreciated.

Thank you,
Imran

[Kal_Lam]

@imran3may, maybe you could do it as outlined in the post discussed previously using the python code:

Uploading Data from XLS to the server

Hi @brighton_zocs, you can try using this (no longer supported) script: You can also try following these yet-to-be-published, draft tutorials on doing this through the API: Part 1 Part 2

[imran3may]

Thanks, @Kal_Lam; however, it did not really help to resolve my challanges. by I was wondering is there any way to push the files using Briefcase? The steps on the referred post were not very clear to me. Many organizations used ODK previously and are not interested or cannot afford to use ODK as ODK is not convenient anymore. Either they need to go for $169/mo plan for go through an almost mission impossible setup process. Since people are moving towards KoBo Toolbox, KoBo developer team may consider coming up with a solution that may help everyone to backup data from ODK Aggregate server and push that to KoBo toolbox and continue their collection.

Thanks again,
Imran

[franklwambo21]

You might want to download the CSV file associated with the form(s) on the aggregate server either via the GUI or using Windows batch script with the parameters specified at Aggregate using Batch, then explore the options to submit entries to a Kobo toolbox hosted form with reference to the URL submissions via API. The API submitting entries to the Kobo toolbox form would be making reference to the Aggregate-downloaded CSV

[imran3may]

Thanks so much @franklwambo21, I have started to see the light of hope now to move my data from the Aggregate server to the KoBoToolbox. What I have done so far is export all data collected with a single form, which contains 5776 rows as CSV.

What would be the next point?

@franklwambo21

then explore the options to submit entries to a Kobo toolbox hosted form with reference to the URL submissions via API. The API submitting entries to the Kobo toolbox form would be making reference to the Aggregate-downloaded CSV

Now, this is something flying over my head. I need clear guidance to make this happen. I am truly stuck with this and really need to move this set of data and allow our field colleagues to continue collecting data as a continuation of this data on KoBoToolbox.

I more detailed guidance would be much appreciated. Honestly, I am not capable of accomplishing this as described in the next steps. I do not even have an idea, where do I run those bunch of codes!

The next steps may be very simple for some members of this community, I just need a little more clear guidance.

Thanks in advance,
Imran

[tinok]

Hi Im

@imran3may

Thanks, @Kal_Lam; however, it did not really help to resolve my challanges. by I was wondering is there any way to push the files using Briefcase? The steps on the referred post were not very clear to me. Many organizations used ODK previously and are not interested or cannot afford to use ODK as ODK is not convenient anymore. Either they need to go for $169/mo plan for go through an almost mission impossible setup process. Since people are moving towards KoBo Toolbox, KoBo developer team may consider coming up with a solution that may help everyone to backup data from ODK Aggregate server and push that to KoBo toolbox and continue their collection.

Hi @imran3may! The tool above is meant to push data from other sources into Kobo from a CSV/spreadsheet. It requires that there is a matching XLSForm deployed first. Briefcase is a clear alternative since it allows pulling the XML and pushing them to Kobo. This also requires you to first deploy an XLSForm that matches the data structure. If you don’t have a matching XLSForm file you’ll have to create it first. It’s definitely doable. Unfortunately there isn’t yet a kind of a click through import process.

[imran3may]

Hi @tinok,

Thank you so much! I have the copy of the form and deployed it to the KoBoToolbox. I downloaded CSV directly from the Aggregate server; however, cannot find any option on the briefcase to push the data to the KoBoToolbox, which I have downloaded from the Aggregate server as CSV.

Guidance will be appreciated.

Imran

[tinok]

With Briefcase you download instances, stored as XML files on your computer, and you can then push them back to a separate server and account. It may be more involved since the project IDs have to match; I hope someone can support further if needed.

The second proposed solution of pushing data with the small Python script would work from your CSV file, so in that case you wouldn’t need with Briefcase. It was created for this exact use case.

[imran3may]

Hi @tinok,

@tinok

With Briefcase you download instances, stored as XML files on your computer, and you can then push them back to a separate server and account.

This is straight forward if it was another Aggregate or Central server, I could not find any option on the briefcase for adding the KoBoToolbox server.

I believe pushing CSV would ve a better option; however, still not sure where to run that script and how this data will be restored to the KoBo under the same form.

Once again, thank you for your support,
Imran

[imran3may]

Hello,

Help with the next steps will be highly appreciated. I am stuck with my data collection.

Thanks in advance,
Imran

[Kal_Lam]

Hi @imran3may,

You could migrate your KoboToolbox project from one server (from humanitarian to non-humanitarian or vice versa) to the other using ODK Briefcase as follows:

In this post, I will illustrate on how to migrate a survey project from a non-humanitarian server (Kobo server) to a humanitarian server (OCHA server).

At the moment, my non-humanitarian server (Kobo server) has a project with 5 submissions.

I would wish to migrate this project to the humanitarian server (OCHA server) which at the moment has not project at all.

Step 1: Download the XLSForm from the non-humanitarian server (Kobo server):

In this step, you will only need to download the XLSForm. No need to download the dataset for this project.

Step 2: Upload and deploy the XLSForm to the humanitarian server (OCHA server):

In this step, you will need to upload and deploy the XLSForm as a new project to the humanitarian server (OCHA server).

So, your humanitarian server (OCHA server) should now have this survey project as shown in the image below:

Step 3: Download the latest version of ODK Briefcase:

Download the latest version of ODK Briefcase from here.

Step 4: Configure the storage settings of ODK Briefcase:

• Double click the ODK Briefcase that you just downloaded.
• You should be able to see a dialogue box as shown in the image below:
  
• Press OK. You will be directed to ODK Briefcase application as shown in the image below:
  
  
• Configure the Storage Location as per your convenience by pressing Choose. You should now see something similar as shown in the image below (here you will notice that once you configure your Storage Location, ODK Briefcase application activates the Pull, Push and Export tabs):
  
  

Step 5: Configure the Pull settings of ODK Briefcase:

• Select the Pull tab. You should now see a similar screen as shown in the image below:
  
  
• Select Configure. You should see a dialogue box as shown in the image below:
  
• Fill up the URL and login credentials from your non-humanitarian server (Kobo server) as you wish to migrate a project from this server to the humanitarian server (OCHA server).
• You could do it as shown in the image below:
  
• You will need to replace the username superkalyan with your own KoboToolbox username. Please also note that you will need to use this URL format https://kc.kobotoolbox.org/username. The ODK Briefcase will fail to pull the data from your server if you fail to use the URL format advised above.

Step 6: Pulling the project to ODK Briefcase storage:

• Once the Pull configuration settings is done, you should be able to see the following screen as shown in the image below:
  
  
• Now check the project that you wish to pull it to your ODK Briefcase storage. Then select Pull.
  
  
• You should see Success under the Pull Status as shown in the image below:
  
  
• This refers that your project was successfully pulled to your ODK Briefcase storage.

Step 7: Configure the Push settings of ODK Briefcase:

• Select the Push tab. You should now see a similar screen as shown in the image below:
  
  
• Select Configure and fill-up the dialogue box with the URL and login credentials from your humanitarian server (OCHA server) as you wish to migrate everything to this server now (as shown in the image below):
  
• You will need to replace the username superkalyan with your own KoboToolbox username. Please also note that you will need to use this URL format https://kc.humanitarianresponse.info/username. The ODK Briefcase will fail to push the data to your server if you fail to use the URL format advised above.

Step 8: Pushing the project from ODK Briefcase storage to the destination server:

• Once the Push configuration settings is done, you should be able to see the following screen as shown in the image below:
  
  
• Now check the project that you wish to push the project to your other KoboToolbox server. Then select Push.
  
  
• You should see Success under the Push Status as shown in the image below:
  
  
• This refers that your project was successfully pushed to your other KoboToobox server (i.e. to your destination account).

Step 9: Checking the status of the data migration in your destination account:

• Once, the push is successful, go to your account to check if the data has been successfully been migrated there.
• You should still see that the submissions are not present in the KPI as shown in the image below:
  
  
• Go to the legacy UI by selecting the legacy icon as shown in the image below:
  
  
• You should now see that your project along with the data has been migrated there.
  
  
• To further access the data, select the project name. You will then be download or view your data in table as shown in the image below:
  
  

[imran3may]

Dear @Kal_Lam,

Thank you for such clear guidance in importing the data to KoBoToolbox. I managed to complete all the steps. However, challenges are still there since I can see the data is in the Legacy UI and not in the regular interface. How do I import that to the regular interface to continue collecting data on the project I imported?

As I mentioned earlier, I am importing data from ODK Aggregate to continue collecting data on the same project.

Support on this will be highly appreciated.

Imran

[Kal_Lam]

@imran3may, yes, this is still the downside of migrating data to the system. You should be able to see the project in the legacy UI, but you still do not have the option to sync the same with the KPI. However, you should still be able to continue data collection for your project, even in the legacy UI.

[imran3may]

Hi @Kal_Lam,

Thank you so much for your quick response.

@Kal_Lam

However, you should still be able to continue data collection for your project, even in the legacy UI

I can only see the project and the number of submissions there.

This interface only allows downloading the data. This means a legacy interface is just a storage option, I cannot see any options like the regular interface.

I can see only the old forms in the regular interface:

The data I migrated has some error message linked to it:

According to another discussion in this forum looks like there are options to migrate to the regular interface and by following that option I will be able to continue data collection on the same project. I saw that there are some scripts that need to run in that case; however, I do not have any clue where to run the script and what changes need to be changed in that script before executing that.

I know hope there is someone who will be interested to resolve this challenge, I need to resolve this issue, as there are some very important data related to our project and we need to continue collecting data on the same project.

Thanks again @Kal_Lam,
Imran

[imran3may]

Hi @tinok,

As you mentioned, there is an option to push data and which will allow me to continue collecting data for the same project. I would appreciate your support in this.

Thank you,
Imran

[Kal_Lam]

@imran3may, once migrating the data to the legacy UI you should be able to connect your KoboCollect android app with your account (server). You should also be able to get blank forms to your android app. Once you have the blank forms in the app, you should be able to continue collecting data and even submitting the data back to the server.

Yes as you say, syncing the data from the legacy UI to the KPI is a downside at the moment. That is the system will not allow you to sync these migrated project from the legacy UI to the KPI.

cc: @tinok

[enabel_rdc_hkl]

Hi @Kal_Lam thank you for the tutorial i followed the steps but i’m stuck in the pulling step. I can’t see any forms to select. What may be the issue ?

[anais_dctl]

Hi @Kal_Lam,

I am following the procedure mentioned here above, but I cannot find how to do the Step 9.
Indeed, the legacy UI is not available anymore. Thus, I cannot find how to push the submissions, so that they become visible on the KPI (the ODK Briefcase Push has worked - Step 8).
Could you provide me any advice on this, please?

Thanks,