An "Internal Server Error" appeared on my indico normally running about 2 years

Administrators
1

Hi guys,

My indico has been running normally for about two years, until the “Internal Server Error” appeared on March 29. The browser displays as follows:

Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error.
More information about this error may be available in the server error log.

Indico version 2.1.8
CentOS Linux release 7.5.1804 (Core)
Apache/2.4.6 (CentOS)
uwsgi-2.0.17.1-2.el7.x86_64
postgresql96-9.6.12-1PGDG.rhel7.x86_64

celery.log outputs of ERRORs:

    [root@indico20190416 log]# tail -n 30000 celery.log | grep 'ERROR'
2021-03-29 10:30:51,413  ERROR    celery.worker.consumer.consumer consumer: Cannot connect to redis:**127_0_0_1:6379/0: Error 111 connecting to 127_0_0_1:6379. Connection refused..
2021-03-30 03:35:42,181  ERROR    celery.worker.consumer.consumer consumer: Cannot connect to ......

celery.log outputs of WARNINGs:

    [root@indico20190416 log]# tail -n 18500 celery.log | grep 'WARNING'
...
2021-03-16 03:59:59,949  WARNING  celery.redirected         Traceback (most recent call last):
2021-03-16 03:59:59,949  WARNING  celery.redirected         File "/opt/indico/.venv/lib/python2.7/site-packages/indico/core/logger.py", line 89, in emit
2021-03-16 03:59:59,949  WARNING  celery.redirected         smtp.sendmail(self.fromaddr, self.toaddrs, msg.as_string())
2021-03-16 03:59:59,949  WARNING  celery.redirected         File "/usr/lib64/python2.7/smtplib.py", line 746, in sendmail
2021-03-16 03:59:59,950  WARNING  celery.redirected         raise SMTPRecipientsRefused(senderrs)
2021-03-16 03:59:59,950  WARNING  celery.redirected         SMTPRecipientsRefused: {u'lee@jtu.edu.cn': (553, '5.7.1 <logger@indico20190416.novalocal>: Sender address rejected: not owned by user lee@jtu.edu.cn')}
2021-03-16 03:59:59,950  WARNING  celery.redirected         Logged from file emails.py, line 61

2021-03-29 10:30:56,419  WARNING  celery.redirected         Traceback (most recent call last):
2021-03-29 10:30:56,419  WARNING  celery.redirected         File "/opt/indico/.venv/lib/python2.7/site-packages/indico/core/logger.py", line 77, in emit
2021-03-29 10:30:56,420  WARNING  celery.redirected         smtp = smtplib.SMTP(self.mailhost, port, timeout=self._timeout)
2021-03-29 10:30:56,421  WARNING  celery.redirected         File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__
2021-03-29 10:30:56,422  WARNING  celery.redirected         (code, msg) = self.connect(host, port)
2021-03-29 10:30:56,422  WARNING  celery.redirected         File "/usr/lib64/python2.7/smtplib.py", line 316, in connect
2021-03-29 10:30:56,423  WARNING  celery.redirected         (code, msg) = self.getreply()
2021-03-29 10:30:56,423  WARNING  celery.redirected         File "/usr/lib64/python2.7/smtplib.py", line 364, in getreply
2021-03-29 10:30:56,423  WARNING  celery.redirected         + str(e))
2021-03-29 10:30:56,423  WARNING  celery.redirected         SMTPServerDisconnected: Connection unexpectedly closed: timed out
2021-03-29 10:30:56,423  WARNING  celery.redirected         Logged from file consumer.py, line 423
2021-03-29 10:31:03,301  WARNING  celery.redirected         /opt/indico/.venv/lib/python2.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <ht*****/initd_org/psycopg/docs/install.html#binary-install-from-pypi>.
......
2

Yesterday (Mar 31), I upgraded my indico to the latest version (2.3.4)

A new warning was logged in celery.log:

    2021-03-31 18:08:30,850  WARNING  celery.worker.consumer.consumer consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 599, in start
    c.loop(*c.loop_args())
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/worker/loops.py", line 83, in asynloop
    next(loop)
  File "/opt/indico/.venv/lib/python2.7/site-packages/kombu/asynchronous/hub.py", line 364, in create_loop
    cb(*cbargs)
  File "/opt/indico/.venv/lib/python2.7/site-packages/kombu/transport/redis.py", line 1083, in on_readable
    self.cycle.on_readable(fileno)
  File "/opt/indico/.venv/lib/python2.7/site-packages/kombu/transport/redis.py", line 354, in on_readable
    chan.handlers[type]()
  File "/opt/indico/.venv/lib/python2.7/site-packages/kombu/transport/redis.py", line 734, in _brpop_read
    **options)
  File "/opt/indico/.venv/lib/python2.7/site-packages/redis/client.py", line 915, in parse_response
    response = connection.read_response()
  File "/opt/indico/.venv/lib/python2.7/site-packages/redis/connection.py", line 739, in read_response
    response = self._parser.read_response()
  File "/opt/indico/.venv/lib/python2.7/site-packages/redis/connection.py", line 470, in read_response
    self.read_from_socket()
  File "/opt/indico/.venv/lib/python2.7/site-packages/redis/connection.py", line 429, in read_from_socket
    raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
ConnectionError: Connection closed by server.

A new warning was logged in indico.log:

    2021-03-31 10:49:54,232  WARNING  0000000000000000  indico.celery             Task heartbeat is locked; not executing it. To manually unlock it, run `indico celery unlock heartbeat`

I turned to .venv/bin and run indico celery unlock heartbeat. The output is:

    [root@indico20190416 bin]# ./indico celery unlock heartbeat
Traceback (most recent call last):
  File "./indico", line 8, in <module>
    sys.exit(cli())
  File "/opt/indico/.venv/lib/python2.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/indico/.venv/lib/python2.7/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/opt/indico/.venv/lib/python2.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/indico/.venv/lib/python2.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/indico/.venv/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/indico/.venv/lib/python2.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/indico/.venv/lib/python2.7/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/indico/.venv/lib/python2.7/site-packages/flask/cli.py", line 425, in decorator
    with __ctx.ensure_object(ScriptInfo).load_app().app_context():
  File "/opt/indico/.venv/lib/python2.7/site-packages/flask/cli.py", line 381, in load_app
    app = call_factory(self, self.create_app)
  File "/opt/indico/.venv/lib/python2.7/site-packages/flask/cli.py", line 117, in call_factory
    return app_factory(script_info)
  File "/opt/indico/.venv/lib/python2.7/site-packages/indico/cli/util.py", line 28, in _create_app
    return make_app()
  File "/opt/indico/.venv/lib/python2.7/site-packages/indico/web/flask/app.py", line 361, in make_app
    app.config['INDICO'] = load_config(only_defaults=testing, override=config_override)
  File "/opt/indico/.venv/lib/python2.7/site-packages/indico/core/config.py", line 193, in load_config
    path = get_config_path()
  File "/opt/indico/.venv/lib/python2.7/site-packages/indico/core/config.py", line 139, in get_config_path
    'move/symlink the config in one of the following locations: {}'.format(', '.join(paths)))
Exception: No indico config found. Point the INDICO_CONFIG env var to your config file or move/symlink the config in one of the following locations: /root/.indico.conf, /etc/indico.conf

When indico was upgrading, some message in red was shown on the screen (which is Bold now):

Collecting pyrsistent>=0.14.0 (from jsonschema==3.2.0->indico)
  Downloading pyrsistent-0.17.2.tar.gz (106kB)
    100% |████████████████████████████████| 112kB 5.8MB/s
**pyrsistent requires Python '>=3.5' but the running Python is 2.7.5**

**ERROR: pip's legacy dependency resolver does not consider dependency conflicts when selecting packages. This behaviour is the source of the following dependency conflicts.**
**botocore 1.20.41 requires urllib3<1.27,>=1.25.4, but you'll have urllib3 1.24.1 which is incompatible.**

Who kindly told me how to solve this problem? Thank you!

3

First of all, you shouldn’t run the indico command as root. While this may not cause any problems in most cases, if running such a command ever creates a new file, that file would now be owned as root, and no longer be writable (maybe not even readable) but the indico user!

It looks like your celery worker is broken because it cannot connect to redis - maybe redis isn’t running?

The error about the python3-only version of pyrsistent most likely happens because your pip is extremely outdated and thus does not know about yanked packages (which is the case for pyrsistent 0.17.2). Use pip install -U pip setuptools to update those tools, and then pip install --force-reinstall indico which should hopefully get rid of any incorrect packages you got due to the outdated pip version.

4

I have checked /opt/indico, its subfolders and files, the owner of them is still indico:apache.

Redis is running:
[indico@indico20190416 ~]$ redis-cli
127.0.0.1:6379> PING
PONG

Before the failure on March 29, my last login was Nov 11, 2020.

5

Do you still have any issues when running the indico commands as the indico user after doing what I suggested in my previous answer?

6

My colleague solved the problem yesterday evening. On November 11 last year, the uwsgi was upgraded, and /usr/lib/systemd/system/uwsgi.service became a new version, causing uwsgi to fail to start. He changed this file and now indico is running.

Thank you very much!

7 
[Unit]
Description=uWSGI Emperor Service
After=syslog.target

[Service]
EnvironmentFile=-/etc/sysconfig/uwsgi
#uwsgi 2.0.18 from EPEL changed ExecStart. It adds --pidfile and
#--stats options
#ExecStart=/usr/sbin/uwsgi --ini /etc/uwsgi.ini --pidfile /run/uwsgi/uwsgi.pid --stats /run/uwsgi/stats.sock
ExecStart=/usr/sbin/uwsgi --ini /etc/uwsgi.ini
ExecReload=/bin/kill -HUP $MAINPID
KillSignal=SIGINT
Restart=always
Type=notify
StandardError=syslog
NotifyAccess=all

[Install]
WantedBy=multi-user.target
8

I wonder if you got hit by this issue… it’s somewhat old, but in case you never did system updates for a while it’s still likely:

9

It is indeed a similar issue.
Thank you for your quickly help!