I updated the kpi container to a newer version using a git pull. I was able to run python manage.py collectstatic and some of the images have been updated,
But I am getting various errors when running npm run build.
root@kpi:/srv/src/kpi# npm run build
kpi@0.1.1 build /srv/src/kpi
webpack --config webpack/prod.config.js
/srv/src/kpi/node_modules/webpack-cli/bin/cli.js:93
throw err;
^
Error: Cannot find module ‘webpack-extract-translation-keys-plugin’
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
at Function.Module._load (internal/modules/cjs/loader.js:562:25)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (/srv/src/kpi/node_modules/webpack-cli/node_modules/v8-compile-cache/v8-compile-cache.js:161:20)
at Object. (/srv/src/kpi/webpack/webpack.common.js:2:38)
at Module._compile (/srv/src/kpi/node_modules/webpack-cli/node_modules/v8-compile-cache/v8-compile-cache.js:192:30)
at Object.Module._extensions…js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (/srv/src/kpi/node_modules/webpack-cli/node_modules/v8-compile-cache/v8-compile-cache.js:161:20)
at Object. (/srv/src/kpi/webpack/prod.config.js:5:23)
at Module._compile (/srv/src/kpi/node_modules/webpack-cli/node_modules/v8-compile-cache/v8-compile-cache.js:192:30)
at Object.Module._extensions…js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (/srv/src/kpi/node_modules/webpack-cli/node_modules/v8-compile-cache/v8-compile-cache.js:161:20)
at WEBPACK_OPTIONS (/srv/src/kpi/node_modules/webpack-cli/bin/utils/convert-argv.js:114:13)
at requireConfig (/srv/src/kpi/node_modules/webpack-cli/bin/utils/convert-argv.js:116:6)
at /srv/src/kpi/node_modules/webpack-cli/bin/utils/convert-argv.js:123:17
at Array.forEach ()
at module.exports (/srv/src/kpi/node_modules/webpack-cli/bin/utils/convert-argv.js:121:15)
at yargs.parse (/srv/src/kpi/node_modules/webpack-cli/bin/cli.js:71:45)
at Object.parse (/srv/src/kpi/node_modules/webpack-cli/node_modules/yargs/yargs.js:567:18)
at /srv/src/kpi/node_modules/webpack-cli/bin/cli.js:49:8
at Object. (/srv/src/kpi/node_modules/webpack-cli/bin/cli.js:366:3)
at Module._compile (internal/modules/cjs/loader.js:778:30)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! kpi@0.1.1 build: webpack --config webpack/prod.config.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the kpi@0.1.1 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2021-03-06T21_56_59_193Z-debug.log
And the debug log gives this:
0 info it worked if it ends with ok
1 verbose cli [ ‘/usr/bin/node’, ‘/usr/bin/npm’, ‘run’, ‘build’ ]
2 info using npm@6.14.5
3 info using node@v10.21.0
4 verbose run-script [ ‘prebuild’, ‘build’, ‘postbuild’ ]
5 info lifecycle kpi@0.1.1~prebuild: kpi@0.1.1
6 info lifecycle kpi@0.1.1~build: kpi@0.1.1
7 verbose lifecycle kpi@0.1.1~build: unsafe-perm in lifecycle true
8 verbose lifecycle kpi@0.1.1~build: PATH: /usr/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/srv/src/kpi/node_modules/.bin:/opt/venv/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/srv/src/kpi/node_modules/.bin
9 verbose lifecycle kpi@0.1.1~build: CWD: /srv/src/kpi
10 silly lifecycle kpi@0.1.1~build: Args: [ ‘-c’, ‘webpack --config webpack/prod.config.js’ ]
11 silly lifecycle kpi@0.1.1~build: Returned: code: 1 signal: null
12 info lifecycle kpi@0.1.1~build: Failed to exec build script
13 verbose stack Error: kpi@0.1.1 build: webpack --config webpack/prod.config.js
13 verbose stack Exit status 1
13 verbose stack at EventEmitter. (/usr/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:332:16)
13 verbose stack at EventEmitter.emit (events.js:198:13)
13 verbose stack at ChildProcess. (/usr/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack at ChildProcess.emit (events.js:198:13)
13 verbose stack at maybeClose (internal/child_process.js:982:16)
13 verbose stack at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)
14 verbose pkgid kpi@0.1.1
15 verbose cwd /srv/src/kpi
16 verbose Linux 5.4.0-1037-gcp
17 verbose argv “/usr/bin/node” “/usr/bin/npm” “run” “build”
18 verbose node v10.21.0
19 verbose npm v6.14.5
20 error code ELIFECYCLE
21 error errno 1
22 error kpi@0.1.1 build: webpack --config webpack/prod.config.js
22 error Exit status 1
23 error Failed at the kpi@0.1.1 build script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]
Error: Cannot find module ‘webpack-extract-translation-keys-plugin’ makes it sound like you need to run npm install (and possibly then npm run copy-fonts) before npm run build.
Beware that (almost) everything within a Docker container, including whatever’s inside /srv/src/kpi gets trashed whenever the container is recreated (not restarted, so you may be lulled into thinking that the storage is persistent).
If you’re going to build manually, you should follow what the Dockerfile does:
kobo-install provides a developer mode which we use to build locally, but it’s not designed for use on a public server. However, there’s also a “staging mode” that you can select when you go through kobo-install’s run.py --setup; picking that should allow you to serve serve publicly while also building from source directories for KPI and/or KoBoCAT.
If you’re not using kobo-install, or you’re using a modified version and can’t switch to staging mode, then I’d suggest manually editing the docker-compose front-end YAML file in the kobo-docker directory. Comment out the image: line beneath kpi: (and kobocat:, if needed), and insert a line that starts with build: and ends with the path to your KPI source directory. For example:
Also, if you’re spending any significant amount of time making changes to our source code, make sure you use version control like Git. It won’t necessarily make the process of maintaining a forkeasy, but without it, keeping track of your changes will be a nightmare.
If I may join this thread, I would like to comment on the following:
Thanks for providing the above answers, it just helped me setting up a custom KoBo instance on a testing/public server (using the staging mode) and rebuilding it to observe updates on the source code.
I followed the steps you highlighted on the Dockerfile, but realized that the custom changes I made on KPI were not visible on the browser until I did:
python3 run.py --stop python3 run.py
Note: before restarting KoBo, I tried by clearing the browser cache and using incognito sessions.
Restarting KoBo on the testing server is fine, but it’s probably not a good idea to do so on a production server.
I would then like to ask for the recommended steps when attempting to deploy such custom KoBo instance on a production server. Should I use kobo-install with staging mode during the first-time setup? How could I deploy every new update that is released afterwards without having to restart KoBo?
I had gotten my installation done last year by someone else, and I’m assuming he used kobo-install but as I’m not able to get in touch him, I have no way to make sure.
I am using a github repo each for kpi, kobocat and enketo to make the code changes (which are quite minor - mostly relating to branding and colors).
We have now around 200 GB of collected data in our instance that we need to preserve, in addition to nearly a hundred users and project permissions.
We need to upgrade the code (particularly for kpi and kobocat) to the latest versions to fix some of the bugs we’ve been facing.
The github repos have been updated with the latest code on top of which we have already made the required branding changes.
What I’ve tried so far (unsuccessfully) is what I’ve described in my first post, i.e., trying to do a git pull from within the containers and updating the code.
Let us assume that the installation had been done using kobo-install, what would be the right way to proceed?
Hi @dj_1, we seem to be trying to do similar things.
I’m guessing that run.py --stop then run.py restarts all the docker containers. If that’s the case, I can tell you that it’s perfectly safe to do on a production server, because I’ve done it countless times.
(My KPI and KC keep crashing and giving me 504 errors and I need to keep restarting the containers, which is one of the major reasons that I need to upgrade to a better version soon.)
Thanks a lot @jnm - this did it. Now I have the updated version of kpi running on my servers, and all my data is intact as well. Much appreciated.
Next time I’ll try to do this for the update process:
Then I would need to docker-compose up, right? (Sorry for the noob question, I’m just getting familiar with docker). And this approach would preserve all the captured data?
No problem. Yes, you can start, stop, remove, and recreate the containers without losing any submissions or attached files. Have a look in kobo-docker/.vols and you’ll see what’s kept in persistent storage.
Note that you might need some extra arguments to docker-compose:
One or more -f arguments to specify the compose files used by your instance
Possibly a -p argument to specify the project name, which gets prepended to the container names
When I need to invoke docker-compose for the front-end services (KPI, KoBoCAT, Enketo, NGINX), I use the shortcut kobo-install/run.py -cf <compose command>. For my local installation, that translates into
Thanks @jnm for the detailed explanation. It’s been a great help. I went through the documentation, and I understand quite a bit more now.
So I started doing some kobo upgrade trials on a test server, this is the process I followed:
Git pulled kobo-install from github.com/kobotoolbox/kobo-install
Did a normal server installation (run.py) complete with a new domain and SSL certificates
Tested the domain and confirmed that everything is working
Created a new folder somewhere and git pulled my private kpi repo with the required changes.
Edited the docker-compose.frontend.yml file to make the changes you described earlier (commented the image line for kpi and inserted the build line with the kpi folder I created in step 4. (I didn’t change anything for Kobocat and enketo)
Ran docker-compose -f docker-compose.frontend.yml -f docker-compose.frontend.override.yml -p kobofe up
The process started, but it’s been stuck at this point now for more than 20 minutes.
Ah, you actually need to add a -d argument to docker-compose if you want the containers to run in the background. From docker-compose up --help:
-d, --detach Detached mode: Run containers in the background,
print new container names.
Your screenshot shows normal output from the kobocat and kpi containers when they’re finished starting up and ready to serve requests.
You can continue invoking docker-compose manually if you’d like, but I think you should also be able to use the “staging mode” of kobo-install with your private fork.
Manually go to the assets list endpoint /api/v2/assets/ and see if there’s an error there. It might not show anything useful since Django’s DEBUG is likely set to False. Look at the uWSGI log for KPI to see if there are any errors there. Search this forum for other places where I’ve mentioned uWSGI, and hopefully you’ll find some instructions I’ve left about where to find the log. Unfortunately, I don’t have time to help troubleshoot further right now. Good luck!
hello
I execute the script with -d having put “build: source of kpi folder” in the docker-compose.frontend.yml instead of the image.
but the output I get is this one:
=> ERROR [15/22] RUN rm -rf “/srv/src/kpi/jsapp/fonts” && rm -rf “/srv/src/kpi/jsapp/compiled” && npm run copy-fonts && npm run buil 133.8s
[15/22] RUN rm -rf “/srv/src/kpi/jsapp/fonts” && rm -rf “/srv/src/kpi/jsapp/compiled” && npm run copy-fonts && npm run build: #20 1.382 #20 1.382 > kpi@0.1.1 copy-fonts /srv/src/kpi #20 1.382 > python ./scripts/copy_fonts.py && npm run generate-icons #20 1.382 #20 1.488 Destination folder has been created! #20 1.488 Copying fonts… #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-400-normal.woff #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-100-normal.woff2 #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-500-italic.woff2 #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-300-italic.woff2 #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-500-normal.woff #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-100-italic.woff2 #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-400-normal.woff2 #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-300-italic.woff #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-100-italic.woff #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-900-italic.woff #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-700-normal.woff2 #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-500-italic.woff #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-900-italic.woff2 #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-700-italic.woff #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-400-italic.woff #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-900-normal.woff2 #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-300-normal.woff #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-400-italic.woff2 #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-700-italic.woff2 #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-900-normal.woff #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-700-normal.woff #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-100-normal.woff #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-300-normal.woff2 #20 1.488 ./node_modules/@fontsource/roboto/files/roboto-latin-ext-500-normal.woff2 #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-500-italic.woff #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-variable-wghtOnly-normal.woff2 #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-200-italic.woff #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-500-italic.woff2 #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-300-normal.woff #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-200-normal.woff #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-300-normal.woff2 #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-400-italic.woff #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-600-normal.woff2 #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-100-normal.woff2 #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-300-italic.woff2 #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-700-normal.woff #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-700-normal.woff2 #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-300-italic.woff #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-500-normal.woff #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-variable-wghtOnly-italic.woff2 #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-600-italic.woff2 #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-200-italic.woff2 #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-100-italic.woff2 #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-400-normal.woff2 #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-100-italic.woff #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-600-normal.woff #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-200-normal.woff2 #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-700-italic.woff #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-700-italic.woff2 #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-600-italic.woff #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-500-normal.woff2 #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-100-normal.woff #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-400-italic.woff2 #20 1.488 ./node_modules/@fontsource/roboto-mono/files/roboto-mono-latin-ext-400-normal.woff #20 1.488 DONE #20 1.870 #20 1.870 > kpi@0.1.1 generate-icons /srv/src/kpi #20 1.870 > node ./scripts/generate_icons.js #20 1.870 #20 2.342 *** #20 2.342 Please make sure SVGs are at least 640px in size! Otherwise glyphs will look terrible during svg2ttf conversion. #20 2.342 *** #20 2.345 Reading files… #20 2.348 167 SVGs found. #20 2.348 Generating fonts… #20 3.454 Adding timestamp to files… #20 3.465 Copying k-icons.css to SCSS file… #20 3.466 Generating definition file… #20 3.945 #20 3.945 > kpi@0.1.1 build /srv/src/kpi #20 3.945 > webpack --config webpack/prod.config.js #20 3.945 #20 9.871 Browserslist: caniuse-lite is outdated. Please run: #20 9.871 npx browserslist@latest --update-db #20 28.02 (node:57) DeprecationWarning: loaderUtils.parseQuery() received a non-string value which can be problematic, see Introduce getOptions · Issue #56 · webpack/loader-utils · GitHub #20 28.02 parseQuery() will be replaced with getOptions() in the next major version of loader-utils. #20 133.1 Killed #20 133.3 npm ERR! code ELIFECYCLE #20 133.3 npm ERR! errno 137 #20 133.3 npm ERR! kpi@0.1.1 build: webpack --config webpack/prod.config.js #20 133.3 npm ERR! Exit status 137 #20 133.3 npm ERR! #20 133.3 npm ERR! Failed at the kpi@0.1.1 build script. #20 133.3 npm ERR! This is probably not a problem with npm. There is likely additional logging output above. #20 133.4 #20 133.4 npm ERR! A complete log of this run can be found in: #20 133.4 npm ERR! /root/.npm/_logs/2022-02-21T12_56_06_492Z-debug.log
executor failed running [/bin/sh -c rm -rf “${KPI_SRC_DIR}/jsapp/fonts” && rm -rf “${KPI_SRC_DIR}/jsapp/compiled” && npm run copy-fonts && npm run build]: exit code: 137
ERROR: Service ‘kpi’ failed to build : Build failed
Could you please help me?
However I executed the command docker-compose -f docker-compose.frontend.yml -f docker-compose.frontend.override.yml -p kobofe <compose command> after changing the line image: for mi image uploaded to docker hub. How can I know if this way is working?
Also, if you’re spending any significant amount of time making changes to our source code, make sure you use version control like Git. It won’t necessarily make the process of maintaining a 
