Getting a 500 Internal Server Error after a reboot
Celery starts with the following log entries:
Aug 06 07:54:18 systemd[1]: Stopped Indico Celery.
Aug 06 07:54:18 systemd[1]: Started Indico Celery.
Aug 06 07:54:21 indico-celery[5722]: /opt/indico/.venv/lib/python2.7/site-packages/indico/core/auth.py:77: UserWarning: There is no default group provider bu
Aug 06 07:54:21 indico-celery[5722]: warn('There is no default group provider but you have providers with group support. ’
Aug 06 07:54:27 indico-celery[5722]: 2020-08-06 07:54:27,007 INFO 0000000000000000 celery.beat beat: Starting…
Aug 06 07:54:27 indico-celery[5722]: 2020-08-06 07:54:27,441 INFO 0000000000000000 celery.worker.consumer.connection Connected to redis://127.0.0.1:637
Aug 06 07:54:27 indico-celery[5722]: 2020-08-06 07:54:27,446 INFO 0000000000000000 celery.worker.consumer.mingle mingle: searching for neighbors
Aug 06 07:54:28 indico-celery[5722]: 2020-08-06 07:54:28,458 INFO 0000000000000000 celery.worker.consumer.mingle mingle: all alone
Aug 06 07:54:28 indico-celery[5722]: 2020-08-06 07:54:28,468 INFO 0000000000000000 celery.apps.worker celery@ready.
Aug 06 07:54:28 indico-celery[5722]: 2020-08-06 07:54:28,591 INFO 0000000000000000 celery.worker.strategy Received task: heartbeat[d44e0a7c-d347-404
Aug 06 07:54:28 indico-celery[5722]: 2020-08-06 07:54:28,671 INFO 0000000000000000 celery.app.trace Task heartbeat[d44e0a7c-d347-4046-add9-d15
Aug 06 07:55:00 indico-celery[5722]: 2020-08-06 07:55:00,036 INFO 0000000000000000 celery.beat Scheduler: Sending due task event_reminder
Aug 06 07:55:00 indico-celery[5722]: 2020-08-06 07:55:00,042 INFO 0000000000000000 celery.worker.strategy Received task: event_reminders[21ce4340-b4
Aug 06 07:55:00 indico-celery[5722]: 2020-08-06 07:55:00,051 INFO 0000000000000000 celery.app.trace Task event_reminders[21ce4340-b413-43d0-9b
but UWSGI is perpetually hanging at “initializing uWSGI” with:
Aug 06 08:18:47 uwsgi[10526]: /opt/indico/.venv/lib/python2.7/site-packages/indico/core/auth.py:77: UserWarning: There is no default group provider but you h
Aug 06 08:18:47 uwsgi[10526]: warn(‘There is no default group provider but you have providers with group support. ’
Aug 06 08:18:51 systemd[1]: Started uWSGI Emperor Service.
Aug 06 08:18:51 uwsgi[10526]: WSGI app 0 (mountpoint=’’) ready in 6 seconds on interpreter 0x1079940 pid: 10526 (default app)
Aug 06 08:18:51 uwsgi[10526]: spawned uWSGI master process (pid: 10526)
Aug 06 08:18:51 systemd[1]: uwsgi.service: main process exited, code=exited, status=1/FAILURE
Aug 06 08:18:51 uwsgi[10526]: spawned uWSGI worker 1 (pid: 10538, cores: 1)
Aug 06 08:18:51 uwsgi[10526]: spawned uWSGI worker 2 (pid: 10540, cores: 1)
Aug 06 08:18:51 uwsgi[10526]: Thu Aug 6 08:18:51 2020 - mem-collector thread started for worker 2
Aug 06 08:18:51 uwsgi[10526]: spawned uWSGI worker 3 (pid: 10542, cores: 1)
Aug 06 08:18:51 uwsgi[10526]: spawned uWSGI worker 4 (pid: 10543, cores: 1)
Aug 06 08:18:51 uwsgi[10526]: bind(): Permission denied [core/socket.c line 230]
Aug 06 08:18:51 uwsgi[10526]: …brutally killing workers…
Aug 06 08:18:51 uwsgi[10526]: unlink(): No such file or directory [core/uwsgi.c line 1673]
Aug 06 08:18:51 uwsgi[10526]: Thu Aug 6 08:18:51 2020 - mem-collector thread started for worker 3
Aug 06 08:18:51 uwsgi[10526]: Thu Aug 6 08:18:51 2020 - uWSGI worker 4 screams: UAAAAAAH my master disconnected: i will kill myself !!!
Aug 06 08:18:51 uwsgi[10526]: Thu Aug 6 08:18:51 2020 - uWSGI worker 3 screams: UAAAAAAH my master disconnected: i will kill myself !!!
Aug 06 08:18:51 uwsgi[10526]: Thu Aug 6 08:18:51 2020 - mem-collector thread started for worker 4
Aug 06 08:18:52 systemd[1]: Unit uwsgi.service entered failed state.
Aug 06 08:18:52 systemd[1]: uwsgi.service failed.
Aug 06 08:18:52 systemd[1]: uwsgi.service holdoff time over, scheduling restart.
Aug 06 08:18:52 systemd[1]: Stopped uWSGI Emperor Service.
Aug 06 08:18:52 systemd[1]: Starting uWSGI Emperor Service…
Aug 06 08:18:52 uwsgi[10548]: [uWSGI] getting INI configuration from /etc/uwsgi.ini
Aug 06 08:18:52 uwsgi[10548]: *** Starting uWSGI 2.0.18 (64bit) on [Thu Aug 6 08:18:52 2020] ***
Aug 06 08:18:52 uwsgi[10548]: compiled with version: 4.8.5 20150623 (Red Hat 4.8.5-39) on 13 May 2020 17:24:14
Aug 06 08:18:52 uwsgi[10548]: os: Linux-3.10.0-1127.18.2.el7.x86_64 #1 SMP Sun Jul 26 15:27:06 UTC 2020
Aug 06 08:18:52 uwsgi[10548]: nodename:
Aug 06 08:18:52 uwsgi[10548]: machine: x86_64
Aug 06 08:18:52 uwsgi[10548]: clock source: unix
Aug 06 08:18:52 uwsgi[10548]: pcre jit disabled
Aug 06 08:18:52 uwsgi[10548]: detected number of CPU cores: 2
Aug 06 08:18:52 uwsgi[10548]: current working directory: /
Aug 06 08:18:52 uwsgi[10548]: writing pidfile to /run/uwsgi/uwsgi.pid
Aug 06 08:18:52 uwsgi[10548]: detected binary path: /usr/sbin/uwsgi
Aug 06 08:18:52 uwsgi[10548]: setgid() to 48
Aug 06 08:18:52 uwsgi[10548]: setuid() to 988
Aug 06 08:18:52 uwsgi[10548]: your processes number limit is 15076
Aug 06 08:18:52 uwsgi[10548]: your memory page size is 4096 bytes
Aug 06 08:18:52 uwsgi[10548]: *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers ***
Aug 06 08:18:52 uwsgi[10548]: detected max file descriptor number: 1024
Aug 06 08:18:52 uwsgi[10548]: lock engine: pthread robust mutexes
Aug 06 08:18:52 uwsgi[10548]: thunder lock: disabled (you can enable it with --thunder-lock)
Aug 06 08:18:52 uwsgi[10548]: uwsgi socket 0 bound to TCP address 127.0.0.1:8008 fd 4
Aug 06 08:18:52 uwsgi[10548]: Python version: 2.7.5 (default, Apr 2 2020, 13:16:51) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
Aug 06 08:18:52 uwsgi[10548]: Set PythonHome to /opt/indico/.venv
Aug 06 08:18:52 uwsgi[10548]: Python main interpreter initialized at 0x1c13940
Aug 06 08:18:52 uwsgi[10548]: python threads support enabled
Aug 06 08:18:52 uwsgi[10548]: your server socket listen backlog is limited to 100 connections
Aug 06 08:18:52 uwsgi[10548]: your mercy for graceful operations on workers is 60 seconds
Aug 06 08:18:52 uwsgi[10548]: mapped 446520 bytes (436 KB) for 4 cores
Aug 06 08:18:52 uwsgi[10548]: *** Operational MODE: preforking ***