Indico 3 - Celery task scheduler does not seem to be running after upgrade

Administrators
1

Hi, this morning we upgraded our Indico instance to ver.3.0 but now we have this message in home page:

The Celery task scheduler does not seem to be running. This means that email sending and periodic tasks such as event reminders do not work.

I tried to restart indico-celery.service but command systemctl status indico-celery.service -l returned me:

[root@indicohost2 ~]# systemctl status indico-celery.service -l
● indico-celery.service - Indico Celery
   Loaded: loaded (/etc/systemd/system/indico-celery.service; enabled; vendor preset: disabled)
   Active: active (running) since gio 2021-09-09 10:59:59 CEST; 3h 35min ago
 Main PID: 2450 (indico)
   CGroup: /system.slice/indico-celery.service
           ├─2450 /opt/indico/.venv/bin/python /opt/indico/.venv/bin/indico celery worker -B
           ├─2463 /opt/indico/.venv/bin/python /opt/indico/.venv/bin/indico celery worker -B
           ├─2465 /opt/indico/.venv/bin/python /opt/indico/.venv/bin/indico celery worker -B
           ├─2466 /opt/indico/.venv/bin/python /opt/indico/.venv/bin/indico celery worker -B
           └─2467 /opt/indico/.venv/bin/python /opt/indico/.venv/bin/indico celery worker -B

set 09 11:00:05 indicohost2.frascati.enea.it indico-celery[2450]: Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
set 09 11:00:05 indicohost2.frascati.enea.it indico-celery[2450]: [2021-09-09 11:00:05,960: WARNING/Beat]   File "/opt/indico/.pyenv/versions/3.9.6/lib/python3.9/dbm/__init__.py", line 95, in open
set 09 11:00:05 indicohost2.frascati.enea.it indico-celery[2450]: return mod.open(file, flag, mode)
set 09 11:00:05 indicohost2.frascati.enea.it indico-celery[2450]: [2021-09-09 11:00:05,960: WARNING/Beat]   File "/opt/indico/.pyenv/versions/3.9.6/lib/python3.9/dbm/dumb.py", line 316, in open
set 09 11:00:05 indicohost2.frascati.enea.it indico-celery[2450]: return _Database(file, mode, flag=flag)
set 09 11:00:05 indicohost2.frascati.enea.it indico-celery[2450]: [2021-09-09 11:00:05,960: WARNING/Beat]   File "/opt/indico/.pyenv/versions/3.9.6/lib/python3.9/dbm/dumb.py", line 70, in __init__
set 09 11:00:05 indicohost2.frascati.enea.it indico-celery[2450]: self._create(flag)
set 09 11:00:05 indicohost2.frascati.enea.it indico-celery[2450]: [2021-09-09 11:00:05,960: WARNING/Beat]   File "/opt/indico/.pyenv/versions/3.9.6/lib/python3.9/dbm/dumb.py", line 86, in _create
set 09 11:00:05 indicohost2.frascati.enea.it indico-celery[2450]: with _io.open(self._datfile, 'w', encoding="Latin-1") as f:
set 09 11:00:05 indicohost2.frascati.enea.it indico-celery[2450]: [2021-09-09 11:00:05,960: WARNING/Beat] PermissionError: [Errno 13] Permission denied: 'celerybeat-schedule.dat'

I read other discussions in the forum about Celery service and a command test, indico celery worker -B

This is our output:

 -------------- celery@indicohost2.frascati.enea.it v5.1.2 (sun-harmonics)
--- ***** ----- 
-- ******* ---- Linux-3.10.0-1160.41.1.el7.x86_64-x86_64-with-glibc2.17 2021-09-09 14:46:14
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         task_notify_url:0x7f87b0c0c910
- ** ---------- .> transport:   redis://127.0.0.1:6379/0
- ** ---------- .> results:     redis://127.0.0.1:6379/0
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery

Any idea?

thanks,

Simone

2

Did you accidentally start it as root at some point? Check the ownership of
/opt/indico/tmp/celerybeat-schedule.dat - it should be owned by the indico user.
Note that you can safely delete that file while celery is not running.

3

Hi Adrian, the file is already owned by the indico user. I don’t know if I understand correctly, so I stopped the Celery service, renamed the file and restarted Celery. The messages on the home page have disappeared.

4

Weird… Well, since Celery seems to be working properly again now I guess it’s not really worth looking deeper into it…

5

Hi Adrian, unfortunately we noticed the reminders did not work but the cause was the SELinux disabled and not as described in the Indico install guide. So we renamed again the file (correctly ) celerybeat-schedule.dat and load custom SELinux module as described. I apologize and thanks for your support.

6

Hi, this morning we find in home page same error about Celery and reminder don’t work.

I noticed that the celerybeat-schedule.* files are located in the /opt/indico folder and not in /opt/indico/tmp.

I tried to create a symbolic link but the problem is not solved, only celerybeat-schedule.* files in /opt/indico are modified.

The alert on the home page disappears only if run indico celery worker -B command as indico user, but same command is contained in the indico-celery.service.

I just upgrade to 3.0.2 version.

I have no other ideas.

Could I have made mistakes while importing the db?

Thanks for the support,

Simone

7

I don’t see how the DB affects this… The schedule file should always go to the configured TEMP_DIR. Check if that’s set correctly to /opt/indico/tmp in the config file…

8

already check /opt/indico/tmp in config file … is correct.

I received this email during investigation:

2021-09-10 12:12:17,674  6f89f4e207044587  -       indico.flask - ERROR errors.py:98 -- breadcrumbs.html

Traceback (most recent call last):
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/web/flask/templating.py", line 251, in get_source
    rv = self.fs_loader.get_source(environment, path)
  File "/opt/indico/.venv/lib/python3.9/site-packages/jinja2/loaders.py", line 214, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: core/breadcrumbs.html

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/indico/.venv/lib/python3.9/site-packages/flask/app.py", line 1513, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/indico/.venv/lib/python3.9/site-packages/flask/app.py", line 1499, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/web/flask/util.py", line 79, in wrapper
    return obj().process()
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/web/rh.py", line 281, in process
    res = self._do_process()
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/web/rh.py", line 252, in _do_process
    rv = self._process()
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/modules/categories/controllers/display.py", line 234, in _process
    return WPCategory.render_template('display/category.html', self.category, **params)
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/web/views.py", line 125, in render_template
    return cls(g.rh, *wp_args, **context).display()
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/web/views.py", line 265, in display
    body = self._display(params)
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/web/views.py", line 353, in _display
    return self._apply_decoration(self._get_body(params | self._kwargs))
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/web/views.py", line 348, in _apply_decoration
    breadcrumbs = self._get_breadcrumbs()
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/modules/categories/views.py", line 60, in _get_breadcrumbs
    return render_breadcrumbs(category=self.category)
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/web/breadcrumbs.py", line 47, in render_breadcrumbs
    return render_template('breadcrumbs.html', items=items, management=management)
  File "/opt/indico/.venv/lib/python3.9/site-packages/flask/templating.py", line 148, in render_template
    ctx.app.jinja_env.get_or_select_template(template_name_or_list),
  File "/opt/indico/.venv/lib/python3.9/site-packages/jinja2/environment.py", line 1068, in get_or_select_template
    return self.get_template(template_name_or_list, parent, globals)
  File "/opt/indico/.venv/lib/python3.9/site-packages/jinja2/environment.py", line 997, in get_template
    return self._load_template(name, globals)
  File "/opt/indico/.venv/lib/python3.9/site-packages/jinja2/environment.py", line 958, in _load_template
    template = self.loader.load(self, name, self.make_globals(globals))
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/web/flask/templating.py", line 260, in load
    tpl = super().load(environment, name, globals)
  File "/opt/indico/.venv/lib/python3.9/site-packages/jinja2/loaders.py", line 125, in load
    source, filename, uptodate = self.get_source(environment, name)
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/web/flask/templating.py", line 256, in get_source
    return self._get_fallback(environment, template, path)
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/web/flask/templating.py", line 231, in _get_fallback
    rv = self.fallback_loader.get_source(environment, template)
  File "/opt/indico/.venv/lib/python3.9/site-packages/jinja2/loaders.py", line 542, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: breadcrumbs.html

{'data': {'get': {},
          'headers': {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
                      'Accept-Encoding': 'gzip, deflate, br',
                      'Accept-Language': 'en-us',
                      'Connection': 'keep-alive',
                      'Cookie': 'indico_session=ead655a5-a71b-4c70-9c70-e98dd95fec59; '
                                'experimentation_subject_id=ImEwMjMyMTY1LTkzNDEtNDQwMi05MmY2LTE0MzQ3ZmZkMzljMSI%3D--36d64e66a2bb21a1df76fda48a10c295732dbbcc',
                      'Host': 'agenda.enea.it',
                      'Referer': 'https://agenda.enea.it/category/49/',
                      'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X '
                                    '10_15_7) AppleWebKit/605.1.15 (KHTML, '
                                    'like Gecko) Version/14.1.1 '
                                    'Safari/605.1.15'},
          'json': None,
          'post': {},
          'url': {'category_id': 50}},
 'endpoint': 'categories.display',
 'id': '6f89f4e207044587',
 'ip': '2.41.237.67',
 'method': 'GET',
 'referrer': 'https://agenda.enea.it/category/49/',
 'rh': 'RHDisplayCategory',
 'time': '2021-09-10T12:12:17.749450',
 'url': 'https://agenda.enea.it/category/50/',
 'user': None,
 'user_agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) '
               'AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 '
               'Safari/605.1.15'}

and

2021-09-10 12:12:15,879  563ffc0a96a54b69  -       indico.flask - ERROR errors.py:98 -- [Errno 2] No such file or directory: '/opt/indico/.venv/lib/python3.9/site-packages/indico/web/static/dist/manifest.json'

Traceback (most recent call last):
  File "/opt/indico/.venv/lib/python3.9/site-packages/flask/app.py", line 1513, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/indico/.venv/lib/python3.9/site-packages/flask/app.py", line 1499, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/web/flask/util.py", line 79, in wrapper
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/web/rh.py", line 281, in process
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/web/rh.py", line 252, in _do_process
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/modules/categories/controllers/display.py", line 238, in _process
    return WPCategory.render_template('display/root_category.html', self.category, news=news,
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/web/views.py", line 125, in render_template
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/web/views.py", line 260, in display
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/web/flask/wrappers.py", line 144, in manifest
  File "/opt/indico/.venv/lib/python3.9/site-packages/flask_webpackext/ext.py", line 87, in manifest
    return self.manifest_loader().load(
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/core/webpack.py", line 25, in load
    key = (filepath, os.path.getmtime(filepath))
  File "/opt/indico/.pyenv/versions/3.9.6/lib/python3.9/genericpath.py", line 55, in getmtime
    return os.stat(filename).st_mtime
FileNotFoundError: [Errno 2] No such file or directory: '/opt/indico/.venv/lib/python3.9/site-packages/indico/web/static/dist/manifest.json'

{'data': {'get': {},
          'headers': {'Accept': '*/*',
                      'Host': 'agenda.enea.it',
                      'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; '
                                    'rv:33.0) Gecko/20100101 Firefox/33.0'},
          'json': None,
          'post': {},
          'url': {'category_id': 0}},
 'endpoint': 'categories.display',
 'id': '563ffc0a96a54b69',
 'ip': '192.107.90.105',
 'method': 'GET',
 'referrer': None,
 'rh': 'RHDisplayCategory',
 'time': '2021-09-10T12:12:15.922737',
 'url': 'https://agenda.enea.it/',
 'user': None,
 'user_agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:33.0) Gecko/20100101 '
               'Firefox/33.0'}
9

The manifest.json error usually happens if someone accesses indico during an upgrade when uwsgi is running. So that one is safe to ignore. I think the other one may be because of the same reason.