Unable to export data for few surveys error 500

prateek · July 5, 2022, 6:05am

error 500 unable to export data for 3 to 4 surveys only

list of survey names.

Teacher questionnaire endline ,

Teacher questionnaire baseline lesson observation teach endline,

Lesson observation teach baseline,

HT questionnaire endline.

Can you please fix

prateek · July 5, 2022, 6:08am

have tried the XLS Legacy as well none helps I need to download XLS normal

Kal_Lam · July 5, 2022, 6:32am

@prateek, which server are you using? Is this an issue with only a single project or are you having issues with your other projects too?

prateek · July 5, 2022, 7:11am

I am having issues with 3 to 4 projects rest all are fine ( 3 to 4 survveys)

what do you mean by which server name??

Josh · July 5, 2022, 8:35am

Hi @prateek, can you please send me a private message with the following details so that I can investigate:

username
server (humanitarian/non-humanitarian)

prateek · July 8, 2022, 7:21am

@josh
here are the error logs

Waiting for Postgres service.
postgres.newglobe.private:5432 - no response
postgres.newglobe.private:5432 - no response
postgres.newglobe.private:5432 - accepting connections
Postgres service running; ensuring koboform database exists and has PostGIS extensions...
ERROR:  database "koboform" already exists
You are now connected to database "koboform" as user "kobo".
NOTICE:  extension "postgis" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "postgis_topology" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "fuzzystrmatch" already exists, skipping
CREATE EXTENSION
NOTICE:  extension "postgis_tiger_geocoder" already exists, skipping
CREATE EXTENSION
Postgres database koboform ready for use!
Running migrations...
Operations to perform:
  Apply all migrations: admin, auth, authtoken, contenttypes, database, django_celery_beat, django_digest, external_integrations, help, hook, hub, kpi, mfa, oauth2_provider, registration, reversion, sessions, taggit, trench
Running migrations:
  No migrations to apply.
  Your models have changes that are not yet reflected in a migration, and so won't be applied.
  Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
Creating superuser...
User already exists.
Copying static files to nginx volume...
Cleaning up Celery PIDs...
Restore permissions on Celery logs folder
KoBoForm initialization completed.
Running `kpi` container with uWSGI application server.
[uWSGI] getting INI configuration from /srv/src/kpi/uwsgi.ini
ERROR 2022-07-08 05:05:10,384 import_export_task 444 140242820933440 Failed to run exporttask: KeyError('Schools_Master.csv')
Traceback (most recent call last):
  File "/srv/src/kpi/kpi/models/import_export_task.py", line 131, in run
    self._run_task(msgs)
  File "/srv/src/kpi/kpi/models/import_export_task.py", line 824, in _run_task
    super()._run_task(messages)
  File "/srv/src/kpi/kpi/models/import_export_task.py", line 649, in _run_task
    export, submission_stream = self.get_export_object()
  File "/srv/src/kpi/kpi/models/import_export_task.py", line 740, in get_export_object
    pack, submission_stream = build_formpack(
  File "/srv/src/kpi/kobo/apps/reports/report_data.py", line 57, in build_formpack
    pack = FormPack(versions=reversed(schemas), title=asset.name, id_string=asset.uid)
  File "/opt/venv/src/formpack/src/formpack/pack.py", line 55, in _init_
    self.load_all_versions(versions)
  File "/opt/venv/src/formpack/src/formpack/pack.py", line 115, in load_all_versions
    self.load_version(deepcopy(schema))
  File "/opt/venv/src/formpack/src/formpack/pack.py", line 140, in load_version
    form_version = FormVersion(self, schema)
  File "/opt/venv/src/formpack/src/formpack/version.py", line 204, in _init_
    field = FormField.from_json_definition(
  File "/opt/venv/src/formpack/src/formpack/schema/fields.py", line 197, in from_json_definition
    choice = field_choices[choice_id]
KeyError: 'Schools_Master.csv'
ERROR 2022-07-08 05:21:52,381 import_export_task 444 140242820933440 Failed to run exporttask: KeyError('Schools_Master.csv')
Traceback (most recent call last):
  File "/srv/src/kpi/kpi/models/import_export_task.py", line 131, in run
    self._run_task(msgs)
  File "/srv/src/kpi/kpi/models/import_export_task.py", line 824, in _run_task
    super()._run_task(messages)
  File "/srv/src/kpi/kpi/models/import_export_task.py", line 649, in _run_task
    export, submission_stream = self.get_export_object()
  File "/srv/src/kpi/kpi/models/import_export_task.py", line 740, in get_export_object
    pack, submission_stream = build_formpack(
  File "/srv/src/kpi/kobo/apps/reports/report_data.py", line 57, in build_formpack
    pack = FormPack(versions=reversed(schemas), title=asset.name, id_string=asset.uid)
  File "/opt/venv/src/formpack/src/formpack/pack.py", line 55, in _init_
    self.load_all_versions(versions)
  File "/opt/venv/src/formpack/src/formpack/pack.py", line 115, in load_all_versions
    self.load_version(deepcopy(schema))
  File "/opt/venv/src/formpack/src/formpack/pack.py", line 140, in load_version
    form_version = FormVersion(self, schema)
  File "/opt/venv/src/formpack/src/formpack/version.py", line 204, in _init_
    field = FormField.from_json_definition(
  File "/opt/venv/src/formpack/src/formpack/schema/fields.py", line 197, in from_json_definition
    choice = field_choices[choice_id]
KeyError: 'Schools_Master.csv'
ERROR 2022-07-08 05:22:00,951 import_export_task 467 140242820933440 Failed to run exporttask: KeyError('Schools_Master.csv')
Traceback (most recent call last):
  File "/srv/src/kpi/kpi/models/import_export_task.py", line 131, in run
    self._run_task(msgs)
  File "/srv/src/kpi/kpi/models/import_export_task.py", line 824, in _run_task
    super()._run_task(messages)
  File "/srv/src/kpi/kpi/models/import_export_task.py", line 649, in _run_task
    export, submission_stream = self.get_export_object()
  File "/srv/src/kpi/kpi/models/import_export_task.py", line 740, in get_export_object
    pack, submission_stream = build_formpack(
  File "/srv/src/kpi/kobo/apps/reports/report_data.py", line 57, in build_formpack
    pack = FormPack(versions=reversed(schemas), title=asset.name, id_string=asset.uid)
  File "/opt/venv/src/formpack/src/formpack/pack.py", line 55, in _init_
    self.load_all_versions(versions)
  File "/opt/venv/src/formpack/src/formpack/pack.py", line 115, in load_all_versions
    self.load_version(deepcopy(schema))
  File "/opt/venv/src/formpack/src/formpack/pack.py", line 140, in load_version
    form_version = FormVersion(self, schema)
  File "/opt/venv/src/formpack/src/formpack/version.py", line 204, in _init_
    field = FormField.from_json_definition(
  File "/opt/venv/src/formpack/src/formpack/schema/fields.py", line 197, in from_json_definition
    choice = field_choices[choice_id]
KeyError: 'Schools_Master.csv'


ERROR 2022-07-08 05:05:10,384 import_export_task 444 140242820933440 Failed to run exporttask: KeyError('Schools_Master.csv')
Traceback (most recent call last):
  File "/srv/src/kpi/kpi/models/import_export_task.py", line 131, in run
    self._run_task(msgs)
  File "/srv/src/kpi/kpi/models/import_export_task.py", line 824, in _run_task
    super()._run_task(messages)
  File "/srv/src/kpi/kpi/models/import_export_task.py", line 649, in _run_task
    export, submission_stream = self.get_export_object()
  File "/srv/src/kpi/kpi/models/import_export_task.py", line 740, in get_export_object
    pack, submission_stream = build_formpack(
  File "/srv/src/kpi/kobo/apps/reports/report_data.py", line 57, in build_formpack
    pack = FormPack(versions=reversed(schemas), title=asset.name, id_string=asset.uid)
  File "/opt/venv/src/formpack/src/formpack/pack.py", line 55, in init
    self.load_all_versions(versions)
  File "/opt/venv/src/formpack/src/formpack/pack.py", line 115, in load_all_versions
    self.load_version(deepcopy(schema))
  File "/opt/venv/src/formpack/src/formpack/pack.py", line 140, in load_version
    form_version = FormVersion(self, schema)
  File "/opt/venv/src/formpack/src/formpack/version.py", line 204, in init
    field = FormField.from_json_definition(
  File "/opt/venv/src/formpack/src/formpack/schema/fields.py", line 197, in from_json_definition
    choice = field_choices[choice_id]
KeyError: 'Schools_Master.csv'

Josh · July 8, 2022, 7:41am

Hi @prateek, since you’re on a self-hosted instance it’s a bit more difficult to investigate the issue. Can you please send me a private message with the XLSForm file and any additional media such as the Schools_Master.csv file that’s shown in the logs.

Josh · July 8, 2022, 8:03am

I’m able to reproduce the error on my end if I had two versions of a form with v1:

type	name	label
select_one fruits.csv	fruit	Fruits

and v2:

type	name	label
select_one_from_file fruits.csv	fruit	Fruits

The difference being that I changed select_one to select_one_from_file.

If this was the case for you, then you can uncheck the box in the downloads section “Include data from all X versions” so that it only looks at the latest version of the form. Otherwise, since you have your own instance, you can modify the version’s content with the incorrect type. Changing question types across versions can often cause issues.

prateek · July 28, 2022, 7:49am

Hi Josh, Issue still is persisting unable to download the data failed export error

I have done the changes as you informed prior

sharing the csv file

Schools_Master.csv (19.3 KB)