Indico never came up after rebooting the OS

[tak0222]

We are using indico 2.3.6 on CentOS 7.9.2009.

After rebooting the OS, the indigo never came up.
There is no httpd process.
There is no new entry in /opt/indico/log/apache/access.log and error.log.
While indico-celery.service seems to be working.

How can I locate the problem and fix it?

[root@indico2 opt]# systemctl status indico-celery.service -l
* indico-celery.service - Indico Celery
   Loaded: loaded (/etc/systemd/system/indico-celery.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-02-08 18:55:13 JST; 1h 53min ago
 Main PID: 2674 (indico)
    Tasks: 10
   CGroup: /system.slice/indico-celery.service
           |-2674 /opt/indico/.venv/bin/python /opt/indico/.venv/bin/indico celery worker -B
           |-2688 /opt/indico/.venv/bin/python /opt/indico/.venv/bin/indico celery worker -B
           |-2689 /opt/indico/.venv/bin/python /opt/indico/.venv/bin/indico celery worker -B
           |-2690 /opt/indico/.venv/bin/python /opt/indico/.venv/bin/indico celery worker -B
           |-2691 /opt/indico/.venv/bin/python /opt/indico/.venv/bin/indico celery worker -B
           |-2692 /opt/indico/.venv/bin/python /opt/indico/.venv/bin/indico celery worker -B
           |-2693 /opt/indico/.venv/bin/python /opt/indico/.venv/bin/indico celery worker -B
           |-2694 /opt/indico/.venv/bin/python /opt/indico/.venv/bin/indico celery worker -B
           |-2695 /opt/indico/.venv/bin/python /opt/indico/.venv/bin/indico celery worker -B
           `-2696 /opt/indico/.venv/bin/python /opt/indico/.venv/bin/indico celery worker -B

Feb 08 20:30:00 indico2.rikn.jp indico-celery[2674]: 2022-02-08 20:30:00,102  INFO     0000000000000000  celery.app.trace          Task heartbeat[a3f3c680-2809-4134-a230-8d29664b1cef] succeeded in 0.0404041960001s: None
Feb 08 20:35:00 indico2.rikn.jp indico-celery[2674]: 2022-02-08 20:35:00,000  INFO     0000000000000000  celery.beat               Scheduler: Sending due task event_reminders (event_reminders)
Feb 08 20:35:00 indico2.rikn.jp indico-celery[2674]: 2022-02-08 20:35:00,005  INFO     0000000000000000  celery.worker.strategy    Received task: event_reminders[fa9cd0da-2069-4372-9b5b-31f1bc6fdb31]
Feb 08 20:35:00 indico2.rikn.jp indico-celery[2674]: 2022-02-08 20:35:00,020  INFO     0000000000000000  celery.app.trace          Task event_reminders[fa9cd0da-2069-4372-9b5b-31f1bc6fdb31] succeeded in 0.0136094060008s: None
Feb 08 20:40:00 indico2.rikn.jp indico-celery[2674]: 2022-02-08 20:40:00,029  INFO     0000000000000000  celery.beat               Scheduler: Sending due task event_reminders (event_reminders)
Feb 08 20:40:00 indico2.rikn.jp indico-celery[2674]: 2022-02-08 20:40:00,033  INFO     0000000000000000  celery.worker.strategy    Received task: event_reminders[35a32cda-96e4-4bc7-979a-07a1d03288ff]
Feb 08 20:40:00 indico2.rikn.jp indico-celery[2674]: 2022-02-08 20:40:00,047  INFO     0000000000000000  celery.app.trace          Task event_reminders[35a32cda-96e4-4bc7-979a-07a1d03288ff] succeeded in 0.0132493830006s: None
Feb 08 20:45:00 indico2.rikn.jp indico-celery[2674]: 2022-02-08 20:45:00,017  INFO     0000000000000000  celery.beat               Scheduler: Sending due task event_reminders (event_reminders)
Feb 08 20:45:00 indico2.rikn.jp indico-celery[2674]: 2022-02-08 20:45:00,021  INFO     0000000000000000  celery.worker.strategy    Received task: event_reminders[bfa68382-054b-42e1-bed9-6153bbe07ed4]
Feb 08 20:45:00 indico2.rikn.jp indico-celery[2674]: 2022-02-08 20:45:00,036  INFO     0000000000000000  celery.app.trace          Task event_reminders[bfa68382-054b-42e1-bed9-6153bbe07ed4] succeeded in 0.0133950119998s: None
[root@indico2 opt]#

[ThiefMaster]

Is it this problem by any chance?

Error 502 after upgrading to CentOS 7.8

I just update centos 7.7 to 7.8 and also indico 2.2.7 to 2.2.8 and i am getting same error. Can you help me? My error is 2020/05/14 19:07:11 [error] 16809#0: *1 upstream prematurely closed connection while reading response header from upstream, client: 95.103.6.204, server: indico.openbrain.sk, request: "GET / HTTP/2.0", upstream: "uwsgi://unix:/opt/indico/w eb/uwsgi.sock:", host: "indico.openbrain.sk"

If not, here are some things to check:

• is uwsgi running
• if yes, anything useful in the output of systemctl status uwsgi.service?

PS: You really need to upgrade to version 3 - indico v2 is no longer supported as its Python version as reached EOL a year ago.

[tak0222]

We did not update Centos Recently.

Is this nomal?

[root@indico2 apache]# systemctl status uwsgi.service
* uwsgi.service - uWSGI Emperor Service
   Loaded: loaded (/usr/lib/systemd/system/uwsgi.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/uwsgi.service.d
           `-old-exec-start.conf
   Active: active (running) since Tue 2022-02-08 18:35:26 JST; 2h 40min ago
 Main PID: 840 (uwsgi)
   Status: "uWSGI is ready"
    Tasks: 9
   CGroup: /system.slice/uwsgi.service
           |- 840 indico uWSGI master
           |-2126 indico uWSGI worker 1
           |-2128 indico uWSGI worker 2
           |-2134 indico uWSGI worker 3
           `-2136 indico uWSGI worker 4

Feb 08 18:35:26 indico2.rikn.jp systemd[1]: Started uWSGI Emperor Service.
Feb 08 18:35:26 indico2.rikn.jp uwsgi[840]: spawned uWSGI worker 1 (pid: 2126, cores: 1)
Feb 08 18:35:26 indico2.rikn.jp uwsgi[840]: Tue Feb  8 18:35:26 2022 - mem-collector thread started for worker 1
Feb 08 18:35:26 indico2.rikn.jp uwsgi[840]: spawned uWSGI worker 2 (pid: 2128, cores: 1)
Feb 08 18:35:26 indico2.rikn.jp uwsgi[840]: Tue Feb  8 18:35:26 2022 - mem-collector thread started for worker 2
Feb 08 18:35:26 indico2.rikn.jp uwsgi[840]: spawned uWSGI worker 3 (pid: 2134, cores: 1)
Feb 08 18:35:26 indico2.rikn.jp uwsgi[840]: Tue Feb  8 18:35:26 2022 - mem-collector thread started for worker 3
Feb 08 18:35:26 indico2.rikn.jp uwsgi[840]: spawned uWSGI worker 4 (pid: 2136, cores: 1)
Feb 08 18:35:26 indico2.rikn.jp uwsgi[840]: *** Stats server enabled on /opt/indico/web/uwsgi-stats.sock fd: 20 ***
Feb 08 18:35:26 indico2.rikn.jp uwsgi[840]: Tue Feb  8 18:35:26 2022 - mem-collector thread started for worker 4
[root@indico2 apache]#

[ThiefMaster]

This looks correct.

And I just realized you mentioned there’s no httpd process. That means your Apache is not running, so maybe you just need to start it? (and enable it in systemd so it autostarts on system startup)

[tak0222]

httpd is enabled by systemctl. But it was not started.

[root@indico2 apache]# systemctl start httpd
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
[root@indico2 apache]# 
[root@indico2 apache]# systemctl status httpd
* httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2022-02-08 21:26:01 JST; 8s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 5163 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
 Main PID: 5163 (code=exited, status=1/FAILURE)

Feb 08 21:26:00 indico2.rikn.jp systemd[1]: Starting The Apache HTTP Server...
Feb 08 21:26:01 indico2.rikn.jp systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Feb 08 21:26:01 indico2.rikn.jp systemd[1]: Failed to start The Apache HTTP Server.
Feb 08 21:26:01 indico2.rikn.jp systemd[1]: Unit httpd.service entered failed state.
Feb 08 21:26:01 indico2.rikn.jp systemd[1]: httpd.service failed.
[root@indico2 apache]# 

[root@indico2 apache]# journalctl -xe
Feb 08 21:20:00 indico2.rikn.jp indico-celery[2674]: 2022-02-08 21:20:00,062  INFO     0000000000000000  celery.worker.strategy    Received task: event_reminders[3d2e00a0-9fc5-4fad-9790-b6267bdd0de0]
Feb 08 21:20:00 indico2.rikn.jp indico-celery[2674]: 2022-02-08 21:20:00,076  INFO     0000000000000000  celery.app.trace          Task event_reminders[3d2e00a0-9fc5-4fad-9790-b6267bdd0de0] succeeded in 0.0129785
Feb 08 21:20:01 indico2.rikn.jp systemd[1]: Started Session 24 of user root.
-- Subject: Unit session-24.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-24.scope has finished starting up.
-- 
-- The start-up result is done.
Feb 08 21:20:01 indico2.rikn.jp CROND[5066]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Feb 08 21:25:00 indico2.rikn.jp indico-celery[2674]: 2022-02-08 21:25:00,023  INFO     0000000000000000  celery.beat               Scheduler: Sending due task event_reminders (event_reminders)
Feb 08 21:25:00 indico2.rikn.jp indico-celery[2674]: 2022-02-08 21:25:00,026  INFO     0000000000000000  celery.worker.strategy    Received task: event_reminders[9766a203-c749-4aa4-9e00-a4e3ef021fcb]
Feb 08 21:25:00 indico2.rikn.jp indico-celery[2674]: 2022-02-08 21:25:00,040  INFO     0000000000000000  celery.app.trace          Task event_reminders[9766a203-c749-4aa4-9e00-a4e3ef021fcb] succeeded in 0.0129559
Feb 08 21:26:00 indico2.rikn.jp polkitd[858]: Registered Authentication Agent for unix-process:5154:1025910 (system bus name :1.118 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/Pol
Feb 08 21:26:00 indico2.rikn.jp systemd[1]: Starting The Apache HTTP Server...
-- Subject: Unit httpd.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit httpd.service has begun starting up.
Feb 08 21:26:01 indico2.rikn.jp systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Feb 08 21:26:01 indico2.rikn.jp systemd[1]: Failed to start The Apache HTTP Server.
-- Subject: Unit httpd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit httpd.service has failed.
-- 
-- The result is failed.
Feb 08 21:26:01 indico2.rikn.jp systemd[1]: Unit httpd.service entered failed state.
Feb 08 21:26:01 indico2.rikn.jp systemd[1]: httpd.service failed.
Feb 08 21:26:01 indico2.rikn.jp polkitd[858]: Unregistered Authentication Agent for unix-process:5154:1025910 (system bus name :1.118, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale C) (disco
[root@indico2 apache]#

[ThiefMaster]

If apache fails to start I’d expect something in the apache error log… Either somewhere in /var/log/ or in /opt/indico/logs/

[tak0222]

Thanks a lot.
I found a problem in the /etc/httpd/conf.d directory (one file was wrong).
After fixing this problem, httpd started and indico started again!
Thank you again!

[ThiefMaster]

Great! Please upgrade to v3.1 as soon as possible nonetheless

https://docs.getindico.io/en/stable/installation/upgrade/#upgrading-from-2-x-to-3-x