The Celery task scheduler does not seem to be running. This means that email sending and periodic tasks such as event reminders do not work


#1

I just finished installing * Indico version installed (2.1.7) - i kept getting this red warning “The Celery task scheduler does not seem to be running. This means that email sending and periodic tasks such as event reminders do not work”

I restarted indico-celery.service and it active now but the message is not gone.

● indico-celery.service - Indico Celery
   Loaded: loaded (/etc/systemd/system/indico-celery.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-03-11 22:22:31 UTC; 3s ago
 Main PID: 22498 (indico)
   CGroup: /system.slice/indico-celery.service
           └─22498 /opt/indico/.venv/bin/python /opt/indico/.venv/bin/indico.

#2

Check celery.log if you are affected by this issue:

If you are, there’s a fix in the comments below.


#3

I checked the celery.log and it shows Unrecoverable error: VersionMismatch(u'Redis transport requires redis-py versions 3.2.0 or later. You have 2.10.6',)

See the rest of the logs:

Traceback (most recent call last):
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/bootsteps.py", line 370, in start
    return self.obj.start()
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 316, in start
    blueprint.start(self)
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/worker/consumer/connection.py", line 23, in start
    c.connection = c.connect()
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 403, in connect
    conn = self.connection_for_read(heartbeat=self.amqheartbeat)
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 410, in connection_for
_read
    self.app.connection_for_read(heartbeat=heartbeat))
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 434, in ensure_connect

#4

Yes, please run the comment from my comment on GitHub and restart celery; it should fix the problem.


#5

HI TheifMaster,

Per your comments - I uninstall kombu and install kombu 4.3.0 and then restratered indico-celery.service (started ok). However, the red warning still there and logs shows same version mismatch :

  File "/opt/indico/.venv/lib/python2.7/site-packages/kombu/transport/redis.py", line 956, in _get_client
    'You have {0.__version__}'.format(redis))
VersionMismatch: Redis transport requires redis-py versions 3.2.0 or later. You have 2.10.6

#6

Strange, that sounds like you still have kombu 4.4 since according to their changelog only 4.4 bumped the minimum redis-py version…

Anyway, we’ll make a 2.1.8 release today that fixes this issue properly (by updating redis, and also pinning kombu to 4.4 in case they do more backwards-incompatible changes in the future).


#7

I just released 2.1.8, so pip install -U indico should fix this problem for sure.


#8

Hey TheifMaster,

The upgrade did the trick. Thanks for your help.


#9

Hey Thiefmaster,

I have the same symptoms but I guess a different problem:

2019-03-17 10:07:19,061  INFO     celery.beat               beat: Starting...
2019-03-17 10:07:19,955  INFO     celery.worker.consumer.connection Connected to redis://127.0.0.1:6379/0
2019-03-17 10:07:19,962  INFO     celery.worker.consumer.mingle mingle: searching for neighbors
2019-03-17 10:07:20,976  INFO     celery.worker.consumer.mingle mingle: all alone
2019-03-17 10:07:20,985  INFO     celery.apps.worker        celery@indico.cta-observatory.org ready.
2019-03-17 10:08:59,718  ERROR    celery.beat               Removing corrupted schedule file u'/opt/indico/tmp/celerybeat-schedule': DBFileExistsError(17, 'File exists -- BDB0002 __fop_file_setup:  Retry limit (100) exceeded')
Traceback (most recent call last):
  File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 442, in setup_schedule
    self._store = self._open_schedule()
  File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 432, in _open_schedule
    return self.persistence.open(self.schedule_filename, writeback=True)
  File "/usr/lib/python2.7/shelve.py", line 243, in open
    return DbfilenameShelf(filename, flag, protocol, writeback)
  File "/usr/lib/python2.7/shelve.py", line 227, in __init__
    Shelf.__init__(self, anydbm.open(filename, flag), protocol, writeback)
  File "/usr/lib/python2.7/anydbm.py", line 85, in open
    return mod.open(file, flag, mode)
  File "/usr/lib/python2.7/dbhash.py", line 18, in open
    return bsddb.hashopen(file, flag, mode)
  File "/usr/lib/python2.7/bsddb/__init__.py", line 364, in hashopen
    d.open(file, db.DB_HASH, flags, mode)
DBFileExistsError: (17, 'File exists -- BDB0002 __fop_file_setup:  Retry limit (100) exceeded')
2019-03-17 10:10:41,834  WARNING  celery.redirected         Process Beat:
2019-03-17 10:10:41,834  WARNING  celery.redirected         Traceback (most recent call last):
2019-03-17 10:10:41,834  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/billiard/process.py", line 327, in _bootstrap
2019-03-17 10:10:41,834  WARNING  celery.redirected         self.run()
2019-03-17 10:10:41,835  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 630, in run
2019-03-17 10:10:41,835  WARNING  celery.redirected         self.service.start(embedded_process=True)
2019-03-17 10:10:41,835  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 545, in start
2019-03-17 10:10:41,835  WARNING  celery.redirected         humanize_seconds(self.scheduler.max_interval))
2019-03-17 10:10:41,850  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/kombu/utils/objects.py", line 44, in __get__
2019-03-17 10:10:41,850  WARNING  celery.redirected         value = obj.__dict__[self.__name__] = self.__get(obj)
2019-03-17 10:10:41,850  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 589, in scheduler
2019-03-17 10:10:41,850  WARNING  celery.redirected         return self.get_scheduler()
2019-03-17 10:10:41,850  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 584, in get_scheduler
2019-03-17 10:10:41,851  WARNING  celery.redirected         lazy=lazy,
2019-03-17 10:10:41,851  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 424, in __init__
2019-03-17 10:10:41,851  WARNING  celery.redirected         Scheduler.__init__(self, *args, **kwargs)
2019-03-17 10:10:41,851  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 202, in __init__
2019-03-17 10:10:41,851  WARNING  celery.redirected         self.setup_schedule()
2019-03-17 10:10:41,851  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/indico/core/celery/core.py", line 188, in setup_schedule
2019-03-17 10:10:41,851  WARNING  celery.redirected         super(IndicoPersistentScheduler, self).setup_schedule()
2019-03-17 10:10:41,852  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 450, in setup_schedule
2019-03-17 10:10:41,852  WARNING  celery.redirected         self._store = self._destroy_open_corrupted_schedule(exc)
2019-03-17 10:10:41,852  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 438, in _destroy_open_corrupted_schedule
2019-03-17 10:10:41,852  WARNING  celery.redirected         return self._open_schedule()
2019-03-17 10:10:41,852  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 432, in _open_schedule
2019-03-17 10:10:41,852  WARNING  celery.redirected         return self.persistence.open(self.schedule_filename, writeback=True)
2019-03-17 10:10:41,852  WARNING  celery.redirected         File "/usr/lib/python2.7/shelve.py", line 243, in open
2019-03-17 10:10:41,852  WARNING  celery.redirected         return DbfilenameShelf(filename, flag, protocol, writeback)
2019-03-17 10:10:41,853  WARNING  celery.redirected         File "/usr/lib/python2.7/shelve.py", line 227, in __init__
2019-03-17 10:10:41,853  WARNING  celery.redirected         Shelf.__init__(self, anydbm.open(filename, flag), protocol, writeback)
2019-03-17 10:10:41,853  WARNING  celery.redirected         File "/usr/lib/python2.7/anydbm.py", line 85, in open
2019-03-17 10:10:41,853  WARNING  celery.redirected         return mod.open(file, flag, mode)
2019-03-17 10:10:41,853  WARNING  celery.redirected         File "/usr/lib/python2.7/dbhash.py", line 18, in open
2019-03-17 10:10:41,853  WARNING  celery.redirected         return bsddb.hashopen(file, flag, mode)
2019-03-17 10:10:41,853  WARNING  celery.redirected         File "/usr/lib/python2.7/bsddb/__init__.py", line 364, in hashopen
2019-03-17 10:10:41,853  WARNING  celery.redirected         d.open(file, db.DB_HASH, flags, mode)
2019-03-17 10:10:41,854  WARNING  celery.redirected         DBFileExistsError: (17, 'File exists -- BDB0002 __fop_file_setup:  Retry limit (100) exceeded')
2019-03-17 10:10:56,685  INFO     celery.beat               beat: Shutting down...

The celery Task is starting again and again.


#10

Stop celery using systemctl stop indico-celery (make sure it is fully stopped, ps aux | grep celery should not show anything besides possibly the grep command itself), then delete /opt/indico/tmp/celerybeat-schedule, then start celery again.

Not sure what caused your issue, but since it seems to have a problem with the scheduler’s db, so deleting it should solve it.


#11

Hi ThiefMaster,

Thank you very much. That solved it. Thanks for your very fast help.