Internal Server Error after Indico installation

Hi there, I’m new at Indico and I wanted to install it in an AWS server I manage so I follow the tutorial from Apache — Indico 2.3.3 documentation

My server is Ubuntu 18.04 LTS and I installed Indico as shown in the tutorial but when I want to access from my browser I got “Internal server error” and I don’t know where the problem is coming. I checked all the logs I and only found this on “/var/log/uwsgi/app/indico.log”

Thu Feb 25 20:11:15 2021 - mem-collector thread started for worker 2
Thu Feb 25 20:11:15 2021 - mem-collector thread started for worker 1
Thu Feb 25 20:11:31 2021 - --- no python application found, check your startup logs for errors ---
Thu Feb 25 20:17:15 2021 - --- no python application found, check your startup logs for errors ---
Thu Feb 25 20:52:56 2021 - --- no python application found, check your startup logs for errors ---
Thu Feb 25 20:53:35 2021 - --- no python application found, check your startup logs for errors ---
Thu Feb 25 20:53:36 2021 - --- no python application found, check your startup logs for errors ---
Thu Feb 25 20:56:08 2021 - --- no python application found, check your startup logs for errors ---
Thu Feb 25 21:04:00 2021 - --- no python application found, check your startup logs for errors ---
Thu Feb 25 21:04:39 2021 - --- no python application found, check your startup logs for errors ---

Also I think I have Postgresql 13

Anyone could help me? I don’t know what to do next. Thank you

Postgres 13 shouldn’t be a problem. Check if you get any errors when running indico shell to see whether something goes wrong when initializing the application.

When I run indico shell It work properly
Indico v2.3.3 is ready for your commands
In [1]:

OK strange. Sounds like something is wrong with your uwsgi config… maybe check if systemctl status uwsgi.service (especially right after starting it) shows anything useful.

I got this response after a restart the service:
● uwsgi.service - LSB: Start/stop uWSGI server instance(s)
Loaded: loaded (/etc/init.d/uwsgi; generated)
Active: active (running) since Fri 2021-02-26 17:54:47 UTC; 5s ago
Docs: man:systemd-sysv-generator(8)
Process: 7580 ExecStop=/etc/init.d/uwsgi stop (code=exited, status=0/SUCCESS)
Process: 7619 ExecStart=/etc/init.d/uwsgi start (code=exited, status=0/SUCCESS)
Tasks: 9 (limit: 1140)
CGroup: /system.slice/uwsgi.service
├─7744 indico uWSGI master
├─7751 indico uWSGI worker 1
├─7752 indico uWSGI worker 2
├─7753 indico uWSGI worker 3
└─7754 indico uWSGI worker 4

Feb 26 17:54:47 ip-172-31-85-132 systemd[1]: Stopped LSB: Start/stop uWSGI server instance(s).
Feb 26 17:54:47 ip-172-31-85-132 systemd[1]: Starting LSB: Start/stop uWSGI server instance(s)...
Feb 26 17:54:47 ip-172-31-85-132 uwsgi[7619]:  * Starting app server(s) uwsgi
Feb 26 17:54:47 ip-172-31-85-132 uwsgi[7619]:    ...done.
Feb 26 17:54:47 ip-172-31-85-132 systemd[1]: Started LSB: Start/stop uWSGI server instance(s).

ok, nothing useful in there :confused: anything else in /var/log/uwsgi/app/indico.log that may be more useful?

This is what that log has

i was hoping for something before that. restart uwsgi and see if you get something new in there

This is the latest log after a restart of the service:

Fri Feb 26 18:12:11 2021 - SIGINT/SIGQUIT received...killing workers...
Fri Feb 26 18:12:12 2021 - worker 1 buried after 1 seconds
Fri Feb 26 18:12:12 2021 - worker 2 buried after 1 seconds
Fri Feb 26 18:12:12 2021 - worker 3 buried after 1 seconds
Fri Feb 26 18:12:12 2021 - worker 4 buried after 1 seconds
Fri Feb 26 18:12:12 2021 - goodbye to uWSGI.
Fri Feb 26 18:12:12 2021 - VACUUM: unix socket /run/uwsgi/app/indico/socket removed.
Fri Feb 26 18:12:12 2021 - VACUUM: unix socket /opt/indico/web/uwsgi-stats.sock (stats) removed.
Fri Feb 26 18:12:12 2021 - *** Starting uWSGI 2.0.15-debian (64bit) on [Fri Feb 26 18:12:12 2021] ***
Fri Feb 26 18:12:12 2021 - compiled with version: 7.3.0 on 28 September 2018 15:41:15
Fri Feb 26 18:12:12 2021 - os: Linux-5.4.0-1038-aws #40~18.04.1-Ubuntu SMP Sat Feb 6 01:56:56 UTC 2021
Fri Feb 26 18:12:12 2021 - nodename: ip-172-31-86-132
Fri Feb 26 18:12:12 2021 - machine: x86_64
Fri Feb 26 18:12:12 2021 - clock source: unix
Fri Feb 26 18:12:12 2021 - pcre jit disabled
Fri Feb 26 18:12:12 2021 - detected number of CPU cores: 1
Fri Feb 26 18:12:12 2021 - current working directory: /
Fri Feb 26 18:12:12 2021 - writing pidfile to /run/uwsgi/app/indico/pid
Fri Feb 26 18:12:12 2021 - detected binary path: /usr/bin/uwsgi-core
Fri Feb 26 18:12:12 2021 - dropping root privileges as early as possible
Fri Feb 26 18:12:12 2021 - setgid() to 33
Fri Feb 26 18:12:12 2021 - setuid() to 999
Fri Feb 26 18:12:12 2021 - your processes number limit is 3802
Fri Feb 26 18:12:12 2021 - your memory page size is 4096 bytes
Fri Feb 26 18:12:12 2021 -  *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejecte$Fri Feb 26 18:12:12 2021 - detected max file descriptor number: 1024
Fri Feb 26 18:12:12 2021 - lock engine: pthread robust mutexes
Fri Feb 26 18:12:12 2021 - thunder lock: disabled (you can enable it with --thunder-lock)
Fri Feb 26 18:12:12 2021 - uwsgi socket 0 bound to UNIX address /run/uwsgi/app/indico/socket fd 3
Fri Feb 26 18:12:12 2021 - uwsgi socket 1 bound to TCP address 127.0.0.1:8008 fd 5
Fri Feb 26 18:12:12 2021 - dropping root privileges after socket binding
Fri Feb 26 18:12:12 2021 - Python version: 3.6.9 (default, Jan 26 2021, 15:33:00)  [GCC 8.4.0]
Fri Feb 26 18:12:12 2021 - Set PythonHome to /opt/indico/.venv
Fri Feb 26 18:12:12 2021 - Python main interpreter initialized at 0x562ac78c9d00
Fri Feb 26 18:12:12 2021 - dropping root privileges after plugin initialization
Fri Feb 26 18:12:12 2021 - python threads support enabled
Fri Feb 26 18:12:12 2021 - your server socket listen backlog is limited to 100 connections
Fri Feb 26 18:12:12 2021 - your mercy for graceful operations on workers is 60 seconds
Fri Feb 26 18:12:12 2021 - mapped 445760 bytes (435 KB) for 4 cores
Fri Feb 26 18:12:12 2021 - *** Operational MODE: preforking ***
    Traceback (most recent call last):
      File "/opt/indico/web/indico.wsgi", line 8, in <module>
        from indico.web.flask.app import make_app
 ModuleNotFoundError: No module named 'indico'
Fri Feb 26 18:12:12 2021 - unable to load app 0 (mountpoint='') (callable not found or import error)
Fri Feb 26 18:12:12 2021 - *** no app loaded. going in full dynamic mode ***
Fri Feb 26 18:12:12 2021 - dropping root privileges after application loading
Fri Feb 26 18:12:12 2021 - spawned uWSGI master process (pid: 1613)
Fri Feb 26 18:12:12 2021 - spawned uWSGI worker 1 (pid: 1620, cores: 1)
Fri Feb 26 18:12:12 2021 - spawned uWSGI worker 2 (pid: 1621, cores: 1)
Fri Feb 26 18:12:12 2021 - spawned uWSGI worker 3 (pid: 1622, cores: 1)
Fri Feb 26 18:12:12 2021 - spawned uWSGI worker 4 (pid: 1623, cores: 1)
Fri Feb 26 18:12:12 2021 - *** Stats server enabled on /opt/indico/web/uwsgi-stats.sock fd: 17 ***
Fri Feb 26 18:12:12 2021 - mem-collector thread started for worker 2
Fri Feb 26 18:12:12 2021 - mem-collector thread started for worker 3
Fri Feb 26 18:12:12 2021 - mem-collector thread started for worker 4
Fri Feb 26 18:12:12 2021 - mem-collector thread started for worker 1

There we go. For some reason it cannot import the indico Python package.

Are you using this config? Is the virtualenv = /opt/indico/.venv line in it?

Yes, the configuration file has the same information than this config.

I don’t know if it matters but the first time I wanted to install indico I created the virtualenv with python3 instead of python 2.7, I couldn’t install so I switched to python 2.7 and installed it correctly.

Maybe you still have some leftovers from the old environment? I’d rm -rf .venv and to delete it for good and then re-do the steps from the setup guide after switching to the indico user.

Okay, I removed .venv and re-do all the process of activating the virtualenv and update my host name (this is because every time I restart the server the IP is different) on /opt/indico/etc/indico.conf, /etc/apache2/sites-available/indico.conf and generated a new SSL key whit the new hostname. Now I have a new error: 503 Service Unavailable and then I look into the logs and got this from /opt/indico/log/apache/error.log/

[Fri Feb 26 21:22:34.571278 2021] [proxy:error] [pid 1747:tid 140338100766464] (111)Connection refused: AH00957: uwsgi: attempt to connect to 127.0.0.1:8008 (127.0.0.1) failed
[Fri Feb 26 21:22:34.572568 2021] [:error] [pid 1747:tid 140338100766464] [client 177.241.53.235:12435] failed to make connection to backend: 127.0.0.1:8008
[Fri Feb 26 21:22:34.919815 2021] [proxy:error] [pid 1746:tid 140337905358592] (111)Connection refused: AH00957: uwsgi: attempt to connect to 127.0.0.1:8008 (127.0.0.1) failed
[Fri Feb 26 21:22:34.919856 2021] [:error] [pid 1746:tid 140337905358592] [client 177.241.53.235:49244] failed to make connection to backend: 127.0.0.1:8008, referer: https://ec2-3-88-26-253.compute-1.amazonaws.com/

That sounds like uwsgi isn’t running or not listening on that port. I just realized that earlier I linked you to the docs for debian+nginx, not the debian+apache docs. You need to use the uwsgi config from the latter.

I double checked the uwsig conf file and in fact I have the configuration for Apache, I tried to open at output rule at the firewall from AWS Console for the port 8008 but didn’t worked.
I stil have the same error, I checked again /var/log/uwsgi/app/indico.log and got this

Sun Feb 28 12:22:01 2021 - thunder lock: disabled (you can enable it with --thunder-lock)
Sun Feb 28 12:22:01 2021 - uwsgi socket 0 bound to UNIX address /run/uwsgi/app/indico/socket fd 3
Sun Feb 28 12:22:01 2021 - uwsgi socket 1 bound to TCP address 127.0.0.1:8008 fd 5
Sun Feb 28 12:22:01 2021 - dropping root privileges after socket binding
Sun Feb 28 12:22:01 2021 - Python version: 3.6.9 (default, Jan 26 2021, 15:33:00)  [GCC 8.4.0]
Sun Feb 28 12:22:01 2021 - Set PythonHome to /opt/indico/.venv
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f76531b45c0 (most recent call first):

I can see two things:

  1. uwsig is running in python 3.6.9, I don’t know it thas causing the issue or not
  2. Pyhton encounters a fatal error with a module called encodings, I installed in the .venv with pip a module called encoding-tools but didn’t worked (I restarted the uwsgi service)

Yes, that’s the problem. You seem to have an uwsgi version that does not match your python version. Seems like you installed uwsgi with the python3 plugin instead of the python2 one…

How do I resolve it? Do I need to reinstall Indico from scratch again?

Are you sure you have installed uwsgi-plugin-python and not uwsgi-plugin-python3?

apt remove uwsgi-plugin-python3
apt install uwsgi-plugin-python

Yes, It seems that I never installed the plugin for python 2.7 and instead of that I was installing for python 3.6. Now that I removed uwsgi-plugin-python3 and installed uwsgi-plugin-python now Indico is running correctly, I can access it without getting an error.

Now I have a small issue with some images not loading at all. Probably is because I don’t have a real SSL certificate and I’m using the “test” way of installing Indico.

Thanks for your support!

As long as you accept the selfsigned cert everything should work. Maybe some problem with a wrong base url?