Enketo URL is not permanent

Hi!

I’m hosting KoBo on my own server. I’ve succesfully built a rather large form (~2,4MB) and now I need to share the Enketo form URL with the people responsible of submitting data. The first time the Enketo URL is requested, it takes a while to display the form (about 30 mins). I’ve read that this is “normal” behavior since the HTML generation step seems to be compute intensive (CPU usage is stuck at 100% in a few cores during the 30 mins). Subsequent requests to Enketo are displayed much more quicker, since I suppose they are being served with the HTML cached in the Redis secondary instance. After a few hours, the Enketo URL suddenly stops working throwing a 404 Survey with this ID not found error. Indeed, the Enketo URL to the form changes in the KoBo dashboard and the cache is lost. My two questions are:

Should the Enketo URLs be permanent? If yes, do you know what might be causing the ID obsolescence I’m experiencing? If no, how do you handle providing a permanent URL to the form for people to submit data to it?

Thanks!

Hi @rpalma,

Welcome to the community! Would you mind deploying the same survey project through the KoBoToolbox online server and see if you see the same issue.

Have a great day!

Hi @Kal_Lam,

Thanks for your help. When the form is deployed to the KoBoToolbox online server the expiring Enketo URLs issue doesn’t occur. I think the problem with my own KoboToolbox instance is that for reasons that I currently ignore, the main and secondary Redis instances lose their data. Do you have any pointers regarding things or settings to look for in my own instance in order to solve the URL rotation problem?

Thanks in advance.

1 Like

Hi @rpalma,

Well, in this case would you mind providing the OS and other relevant details you have installed in your device so that i could flag it out to our developers.

Have a great day!

Sure. It’s a Linode instance running Ubuntu 16.04 LTS with the following lshw output:

rodolfo
    description: Computer
    product: Standard PC (Q35 + ICH9, 2009)
    vendor: QEMU
    version: pc-q35-3.1
    width: 64 bits
    capabilities: smbios-2.8 dmi-2.8 vsyscall32
    configuration: boot=normal
  *-core
       description: Motherboard
       physical id: 0
     *-firmware
          description: BIOS
          vendor: SeaBIOS
          physical id: 0
          version: rel-1.12.0-0-ga698c8995f-prebuilt.qemu.org
          date: 04/01/2014
          size: 96KiB
     *-cpu:0
          description: CPU
          product: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
          vendor: Intel Corp.
          physical id: 400
          bus info: cpu@0
          version: pc-q35-3.1
          slot: CPU 0
          size: 2GHz
          capacity: 2GHz
          width: 64 bits
          capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ssbd ibrs ibpb fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat pku ospke
          configuration: cores=1 enabledcores=1 threads=1
     *-cpu:1
          description: CPU
          product: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
          vendor: Intel Corp.
          physical id: 401
          bus info: cpu@1
          version: pc-q35-3.1
          slot: CPU 1
          size: 2GHz
          capacity: 2GHz
          width: 64 bits
          capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ssbd ibrs ibpb fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat pku ospke
          configuration: cores=1 enabledcores=1 threads=1
     *-memory
          description: System Memory
          physical id: 1000
          size: 4GiB
        *-bank
             description: DIMM RAM
             vendor: QEMU
             physical id: 0
             slot: DIMM 0
             size: 4GiB
     *-pci
          description: Host bridge
          product: 82G33/G31/P35/P31 Express DRAM Controller
          vendor: Intel Corporation
          physical id: 100
          bus info: pci@0000:00:00.0
          version: 00
          width: 32 bits
          clock: 33MHz
        *-display
             description: VGA compatible controller
             physical id: 1
             bus info: pci@0000:00:01.0
             version: 02
             width: 32 bits
             clock: 33MHz
             capabilities: vga_controller rom
             configuration: driver=bochs-drm latency=0
             resources: irq:0 memory:fd000000-fdffffff memory:febd0000-febd0fff memory:c0000-dffff
        *-scsi:0
             description: SCSI storage controller
             product: Virtio SCSI
             vendor: Red Hat, Inc
             physical id: 2
             bus info: pci@0000:00:02.0
             version: 00
             width: 64 bits
             clock: 33MHz
             capabilities: scsi msix bus_master cap_list
             configuration: driver=virtio-pci latency=0
             resources: irq:22 ioport:c000(size=64) memory:febd1000-febd1fff memory:fe000000-fe003fff
        *-scsi:1
             description: SCSI storage controller
             product: Virtio SCSI
             vendor: Red Hat, Inc
             physical id: 3
             bus info: pci@0000:00:03.0
             version: 00
             width: 64 bits
             clock: 33MHz
             capabilities: scsi msix bus_master cap_list
             configuration: driver=virtio-pci latency=0
             resources: irq:23 ioport:c040(size=64) memory:febd2000-febd2fff memory:fe004000-fe007fff
        *-network
             description: Ethernet interface
             product: Virtio network device
             vendor: Red Hat, Inc
             physical id: 4
             bus info: pci@0000:00:04.0
             logical name: eth0
             version: 00
             serial: f2:3c:92:73:a8:51
             width: 64 bits
             clock: 33MHz
             capabilities: msix bus_master cap_list rom ethernet physical
             configuration: autonegotiation=off broadcast=yes driver=virtio_net driverversion=1.0.0 ip=45.79.112.121 latency=0 link=yes multicast=yes
             resources: irq:20 ioport:c0c0(size=32) memory:febd3000-febd3fff memory:fe008000-fe00bfff memory:feb80000-febbffff
        *-isa
             description: ISA bridge
             product: 82801IB (ICH9) LPC Interface Controller
             vendor: Intel Corporation
             physical id: 1f
             bus info: pci@0000:00:1f.0
             version: 02
             width: 32 bits
             clock: 33MHz
             capabilities: isa
             configuration: driver=lpc_ich latency=0
             resources: irq:0
        *-storage
             description: SATA controller
             product: 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
             vendor: Intel Corporation
             physical id: 1f.2
             bus info: pci@0000:00:1f.2
             version: 02
             width: 32 bits
             clock: 33MHz
             capabilities: storage msi ahci_1.0 bus_master cap_list
             configuration: driver=ahci latency=0
             resources: irq:32 ioport:c0e0(size=32) memory:febd4000-febd4fff
        *-serial
             description: SMBus
             product: 82801I (ICH9 Family) SMBus Controller
             vendor: Intel Corporation
             physical id: 1f.3
             bus info: pci@0000:00:1f.3
             version: 02
             width: 32 bits
             clock: 33MHz
             configuration: driver=i801_smbus latency=0
             resources: irq:16 ioport:700(size=64)
     *-scsi:0
          physical id: 1
          logical name: scsi0
        *-disk
             description: EXT4 volume
             vendor: Linux
             physical id: 0.0.0
             bus info: scsi@0:0.0.0
             logical name: /dev/sda
             version: 1.0
             serial: b7ea8dc6-3710-46cd-b634-60cea0efd02b
             size: 78GiB
             capabilities: journaled extended_attributes large_files huge_files dir_nlink recover extents ext4 ext2 initialized
             configuration: created=2017-03-07 20:37:08 filesystem=ext4 lastmountpoint=/ logicalsectorsize=512 modified=2019-11-25 19:05:02 mounted=2019-11-25 19:05:02 sectorsize=512 state=clean
     *-scsi:1
          physical id: 2
          logical name: scsi1
        *-disk
             description: Linux swap volume
             physical id: 0.1.2
             bus info: scsi@1:0.1.2
             logical name: /dev/sdb
             version: 1
             serial: 353eb529-a08b-4c6e-86f2-ce9989b058b5
             size: 512MiB
             capacity: 512MiB
             capabilities: swap initialized
             configuration: filesystem=swap logicalsectorsize=512 pagesize=4096 sectorsize=512
  *-network:0 DISABLED
       description: Ethernet interface
       physical id: 1
       logical name: erspan0
       capabilities: ethernet physical
       configuration: broadcast=yes multicast=yes
  *-network:1
       description: Ethernet interface
       physical id: 2
       logical name: veth7689bcc
       serial: b6:64:7d:19:bd:00
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
  *-network:2 DISABLED
       description: Ethernet interface
       physical id: 3
       logical name: gretap0
       capabilities: ethernet physical
       configuration: broadcast=yes multicast=yes
  *-network:3 DISABLED
       description: Ethernet interface
       physical id: 4
       logical name: dummy0
       serial: 9e:34:ba:1f:a5:f9
       capabilities: ethernet physical
       configuration: broadcast=yes driver=dummy driverversion=1.0

Docker and docker-compose versions are 19.03.3 and 1.24.1, respectively.

Thanks!

1 Like

Hi @rpalma,

Thank you for sharing the detailed information! Shall share your issue with our developers!

Have a great day!

Please set up a firewall. It’s likely that someone is accessing your Redis database and doing something untoward.

2 Likes

Indeed, setting up a firewall fixed the issue. Thanks!

2 Likes

Would you kindly share how you achieved this?
We are running kobo on linux droplets

1 Like

Hi,

I didn’t set up the firewall so unfortunately I won’t be able to give you the step-by-step directions. Also, instructions will change depending on your system specs and configuration. However, the problem was what @jnm said, i.e. that there were Redis ports open to the Internet allowing external unauthorized people to mess up with the database. Restricting access in those ports through a firewall fixed the issue.

2 Likes

I have the same problem, if you finally solve this problem please tell me. I’m trying to solve it with iptables rules but for now I have no luck.

After a while I solve this issue with two iptables rules, the process I follow was:

iptables-save > iptables-rules.orig

You can create a copy:

cp iptables-rules.orig iptables-rules.new

Edit the file and add this two rule in filter rules at first after “DOCKER-USER - [0:0]”:

-A FORWARD -i eth0 -p tcp -m tcp --dport 6379 -j DROP
-A FORWARD -i eth0 -p tcp -m tcp --dport 6380 -j DROP

Something like this:

# Generated by iptables-save v1.6.1 on Wed Apr 15 22:58:31 2020
*filter
:INPUT ACCEPT [777488:413876809]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [813511:419978487]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]

-A FORWARD -i eth0 -p tcp -m tcp --dport 6379 -j DROP
-A FORWARD -i eth0 -p tcp -m tcp --dport 6380 -j DROP
...

To load the new rules use iptables-restore command:

iptables-restore < iptables-rules.new

And that’s all no more Enketo URL arbitrary changes, and it’s possible use them rules to secure other open ports like mongo (27017) and/or postgresql (5432).

2 Likes

Thanks Finlay. This resolved the issue

2 Likes