The Celery task scheduler does not seem to be running. This means that email sending and periodic tasks such as event reminders do not work

zmohamoud · March 11, 2019, 10:28pm

I just finished installing * Indico version installed (2.1.7) - i kept getting this red warning “The Celery task scheduler does not seem to be running. This means that email sending and periodic tasks such as event reminders do not work”

I restarted indico-celery.service and it active now but the message is not gone.

● indico-celery.service - Indico Celery
   Loaded: loaded (/etc/systemd/system/indico-celery.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-03-11 22:22:31 UTC; 3s ago
 Main PID: 22498 (indico)
   CGroup: /system.slice/indico-celery.service
           └─22498 /opt/indico/.venv/bin/python /opt/indico/.venv/bin/indico.

ThiefMaster · March 11, 2019, 10:31pm

Check celery.log if you are affected by this issue:

If you are, there’s a fix in the comments below.

zmohamoud · March 11, 2019, 10:58pm

I checked the celery.log and it shows Unrecoverable error: VersionMismatch(u'Redis transport requires redis-py versions 3.2.0 or later. You have 2.10.6',)

See the rest of the logs:

Traceback (most recent call last):
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/bootsteps.py", line 370, in start
    return self.obj.start()
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 316, in start
    blueprint.start(self)
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/worker/consumer/connection.py", line 23, in start
    c.connection = c.connect()
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 403, in connect
    conn = self.connection_for_read(heartbeat=self.amqheartbeat)
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 410, in connection_for
_read
    self.app.connection_for_read(heartbeat=heartbeat))
  File "/opt/indico/.venv/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", line 434, in ensure_connect

ThiefMaster · March 11, 2019, 11:00pm

Yes, please run the comment from my comment on GitHub and restart celery; it should fix the problem.

zmohamoud · March 11, 2019, 11:38pm

HI TheifMaster,

Per your comments - I uninstall kombu and install kombu 4.3.0 and then restratered indico-celery.service (started ok). However, the red warning still there and logs shows same version mismatch :

  File "/opt/indico/.venv/lib/python2.7/site-packages/kombu/transport/redis.py", line 956, in _get_client
    'You have {0.__version__}'.format(redis))
VersionMismatch: Redis transport requires redis-py versions 3.2.0 or later. You have 2.10.6

ThiefMaster · March 12, 2019, 12:00pm

Strange, that sounds like you still have kombu 4.4 since according to their changelog only 4.4 bumped the minimum redis-py version…

Anyway, we’ll make a 2.1.8 release today that fixes this issue properly (by updating redis, and also pinning kombu to 4.4 in case they do more backwards-incompatible changes in the future).

ThiefMaster · March 12, 2019, 12:18pm

I just released 2.1.8, so pip install -U indico should fix this problem for sure.

zmohamoud · March 13, 2019, 8:56pm

Hey TheifMaster,

The upgrade did the trick. Thanks for your help.

HDSS · March 17, 2019, 9:25am

Hey Thiefmaster,

I have the same symptoms but I guess a different problem:

2019-03-17 10:07:19,061  INFO     celery.beat               beat: Starting...
2019-03-17 10:07:19,955  INFO     celery.worker.consumer.connection Connected to redis://127.0.0.1:6379/0
2019-03-17 10:07:19,962  INFO     celery.worker.consumer.mingle mingle: searching for neighbors
2019-03-17 10:07:20,976  INFO     celery.worker.consumer.mingle mingle: all alone
2019-03-17 10:07:20,985  INFO     celery.apps.worker        celery@indico.cta-observatory.org ready.
2019-03-17 10:08:59,718  ERROR    celery.beat               Removing corrupted schedule file u'/opt/indico/tmp/celerybeat-schedule': DBFileExistsError(17, 'File exists -- BDB0002 __fop_file_setup:  Retry limit (100) exceeded')
Traceback (most recent call last):
  File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 442, in setup_schedule
    self._store = self._open_schedule()
  File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 432, in _open_schedule
    return self.persistence.open(self.schedule_filename, writeback=True)
  File "/usr/lib/python2.7/shelve.py", line 243, in open
    return DbfilenameShelf(filename, flag, protocol, writeback)
  File "/usr/lib/python2.7/shelve.py", line 227, in __init__
    Shelf.__init__(self, anydbm.open(filename, flag), protocol, writeback)
  File "/usr/lib/python2.7/anydbm.py", line 85, in open
    return mod.open(file, flag, mode)
  File "/usr/lib/python2.7/dbhash.py", line 18, in open
    return bsddb.hashopen(file, flag, mode)
  File "/usr/lib/python2.7/bsddb/__init__.py", line 364, in hashopen
    d.open(file, db.DB_HASH, flags, mode)
DBFileExistsError: (17, 'File exists -- BDB0002 __fop_file_setup:  Retry limit (100) exceeded')
2019-03-17 10:10:41,834  WARNING  celery.redirected         Process Beat:
2019-03-17 10:10:41,834  WARNING  celery.redirected         Traceback (most recent call last):
2019-03-17 10:10:41,834  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/billiard/process.py", line 327, in _bootstrap
2019-03-17 10:10:41,834  WARNING  celery.redirected         self.run()
2019-03-17 10:10:41,835  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 630, in run
2019-03-17 10:10:41,835  WARNING  celery.redirected         self.service.start(embedded_process=True)
2019-03-17 10:10:41,835  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 545, in start
2019-03-17 10:10:41,835  WARNING  celery.redirected         humanize_seconds(self.scheduler.max_interval))
2019-03-17 10:10:41,850  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/kombu/utils/objects.py", line 44, in __get__
2019-03-17 10:10:41,850  WARNING  celery.redirected         value = obj.__dict__[self.__name__] = self.__get(obj)
2019-03-17 10:10:41,850  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 589, in scheduler
2019-03-17 10:10:41,850  WARNING  celery.redirected         return self.get_scheduler()
2019-03-17 10:10:41,850  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 584, in get_scheduler
2019-03-17 10:10:41,851  WARNING  celery.redirected         lazy=lazy,
2019-03-17 10:10:41,851  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 424, in __init__
2019-03-17 10:10:41,851  WARNING  celery.redirected         Scheduler.__init__(self, *args, **kwargs)
2019-03-17 10:10:41,851  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 202, in __init__
2019-03-17 10:10:41,851  WARNING  celery.redirected         self.setup_schedule()
2019-03-17 10:10:41,851  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/indico/core/celery/core.py", line 188, in setup_schedule
2019-03-17 10:10:41,851  WARNING  celery.redirected         super(IndicoPersistentScheduler, self).setup_schedule()
2019-03-17 10:10:41,852  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 450, in setup_schedule
2019-03-17 10:10:41,852  WARNING  celery.redirected         self._store = self._destroy_open_corrupted_schedule(exc)
2019-03-17 10:10:41,852  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 438, in _destroy_open_corrupted_schedule
2019-03-17 10:10:41,852  WARNING  celery.redirected         return self._open_schedule()
2019-03-17 10:10:41,852  WARNING  celery.redirected         File "/opt/indico/.venv/local/lib/python2.7/site-packages/celery/beat.py", line 432, in _open_schedule
2019-03-17 10:10:41,852  WARNING  celery.redirected         return self.persistence.open(self.schedule_filename, writeback=True)
2019-03-17 10:10:41,852  WARNING  celery.redirected         File "/usr/lib/python2.7/shelve.py", line 243, in open
2019-03-17 10:10:41,852  WARNING  celery.redirected         return DbfilenameShelf(filename, flag, protocol, writeback)
2019-03-17 10:10:41,853  WARNING  celery.redirected         File "/usr/lib/python2.7/shelve.py", line 227, in __init__
2019-03-17 10:10:41,853  WARNING  celery.redirected         Shelf.__init__(self, anydbm.open(filename, flag), protocol, writeback)
2019-03-17 10:10:41,853  WARNING  celery.redirected         File "/usr/lib/python2.7/anydbm.py", line 85, in open
2019-03-17 10:10:41,853  WARNING  celery.redirected         return mod.open(file, flag, mode)
2019-03-17 10:10:41,853  WARNING  celery.redirected         File "/usr/lib/python2.7/dbhash.py", line 18, in open
2019-03-17 10:10:41,853  WARNING  celery.redirected         return bsddb.hashopen(file, flag, mode)
2019-03-17 10:10:41,853  WARNING  celery.redirected         File "/usr/lib/python2.7/bsddb/__init__.py", line 364, in hashopen
2019-03-17 10:10:41,853  WARNING  celery.redirected         d.open(file, db.DB_HASH, flags, mode)
2019-03-17 10:10:41,854  WARNING  celery.redirected         DBFileExistsError: (17, 'File exists -- BDB0002 __fop_file_setup:  Retry limit (100) exceeded')
2019-03-17 10:10:56,685  INFO     celery.beat               beat: Shutting down...

The celery Task is starting again and again.

ThiefMaster · March 17, 2019, 9:34am

Stop celery using systemctl stop indico-celery (make sure it is fully stopped, ps aux | grep celery should not show anything besides possibly the grep command itself), then delete /opt/indico/tmp/celerybeat-schedule, then start celery again.

Not sure what caused your issue, but since it seems to have a problem with the scheduler’s db, so deleting it should solve it.

HDSS · March 17, 2019, 9:42am

Hi ThiefMaster,

Thank you very much. That solved it. Thanks for your very fast help.

mmsalehid · August 1, 2019, 8:29am

I have the same problem but I don’t get any errors, I defined a periodic task and I’m using django but i get this and nothing run with celery worker.

celery beat v4.3.0 (rhubarb) is starting.
__    -    ... __   -        _
LocalTime -> 2019-08-01 08:01:13
Configuration ->
    . broker -> redis://localhost:6379//
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> celery.beat.PersistentScheduler
    . db -> celerybeat-schedule
    . logfile -> [stderr]@%INFO
    . maxinterval -> 5.00 minutes (300s)
[2019-08-01 08:01:13,541: INFO/MainProcess] beat: Starting...

ThiefMaster · August 1, 2019, 8:31am

I think you are in the wrong place. This is not a Celery support forum but the forum of the getindico.io project, which happens to use Celery as its task scheduler…

Tebogo_Mokhine · March 8, 2023, 7:57am

Hi @ThiefMaster also getting the same issue with a different log output:

indico-celery.service - Indico Celery
Loaded: loaded (/home/events/etc/indico-celery.service; enabled; vendor prese
Active: active (running) since Wed 2023-03-08 07:39:54 UTC; 16min ago
Main PID: 17639 (indico)
Tasks: 6 (limit: 4915)
CGroup: /system.slice/indico-celery.service
├─17639 /home/events/.venv/bin/python /home/events/.venv/bin/indico c
├─17670 /home/events/.venv/bin/python /home/events/.venv/bin/indico c
├─17671 /home/events/.venv/bin/python /home/events/.venv/bin/indico c
├─17672 /home/events/.venv/bin/python /home/events/.venv/bin/indico c
└─17673 /home/events/.venv/bin/python /home/events/.venv/bin/indico c

Mar 08 07:43:20 saip2020 indico-celery[17639]: 2023-03-08 07:43:20,390 WARNING
Mar 08 07:43:20 saip2020 indico-celery[17639]: 2023-03-08 07:43:20,391 WARNING
Mar 08 07:43:20 saip2020 indico-celery[17639]: 2023-03-08 07:43:20,391 WARNING
Mar 08 07:43:20 saip2020 indico-celery[17639]: 2023-03-08 07:43:20,391 WARNING
Mar 08 07:43:20 saip2020 indico-celery[17639]: 2023-03-08 07:43:20,391 WARNING
Mar 08 07:43:20 saip2020 indico-celery[17639]: 2023-03-08 07:43:20,391 WARNING
Mar 08 07:43:20 saip2020 indico-celery[17639]: 2023-03-08 07:43:20,392 WARNING
Mar 08 07:43:20 saip2020 indico-celery[17639]: 2023-03-08 07:43:20,392 WARNING
Mar 08 07:43:20 saip2020 indico-celery[17639]: 2023-03-08 07:43:20,392 WARNING
Mar 08 07:43:20 saip2020 indico-celery[17639]: 2023-03-08 07:43:20,392 WARNING

Tebogo_Mokhine · March 8, 2023, 8:00am

@ThiefMaster Here is the full log. Apologies there were cut-offs on the last one:

root@saip2020:~# systemctl -l status indico-celery.service
● indico-celery.service - Indico Celery
Loaded: loaded (/home/events/etc/indico-celery.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-03-08 07:39:54 UTC; 19min ago
Main PID: 17639 (indico)
Tasks: 6 (limit: 4915)
CGroup: /system.slice/indico-celery.service
├─17639 /home/events/.venv/bin/python /home/events/.venv/bin/indico celery worker -B
├─17670 /home/events/.venv/bin/python /home/events/.venv/bin/indico celery worker -B
├─17671 /home/events/.venv/bin/python /home/events/.venv/bin/indico celery worker -B
├─17672 /home/events/.venv/bin/python /home/events/.venv/bin/indico celery worker -B
└─17673 /home/events/.venv/bin/python /home/events/.venv/bin/indico celery worker -B

Mar 08 07:43:20 saip2020 indico-celery[17639]: 2023-03-08 07:43:20,390 WARNING 0000000000000000 celery.redirected return DbfilenameShelf(filename,
Mar 08 07:43:20 saip2020 indico-celery[17639]: 2023-03-08 07:43:20,391 WARNING 0000000000000000 celery.redirected File "/usr/lib/python2.7/shelve.
Mar 08 07:43:20 saip2020 indico-celery[17639]: 2023-03-08 07:43:20,391 WARNING 0000000000000000 celery.redirected Shelf.init(self, anydbm.open
Mar 08 07:43:20 saip2020 indico-celery[17639]: 2023-03-08 07:43:20,391 WARNING 0000000000000000 celery.redirected File "/usr/lib/python2.7/anydbm.
Mar 08 07:43:20 saip2020 indico-celery[17639]: 2023-03-08 07:43:20,391 WARNING 0000000000000000 celery.redirected return mod.open(file, flag, mode
Mar 08 07:43:20 saip2020 indico-celery[17639]: 2023-03-08 07:43:20,391 WARNING 0000000000000000 celery.redirected File "/usr/lib/python2.7/dbhash.
Mar 08 07:43:20 saip2020 indico-celery[17639]: 2023-03-08 07:43:20,392 WARNING 0000000000000000 celery.redirected return bsddb.hashopen(file, flag
Mar 08 07:43:20 saip2020 indico-celery[17639]: 2023-03-08 07:43:20,392 WARNING 0000000000000000 celery.redirected File "/usr/lib/python2.7/bsddb/_
Mar 08 07:43:20 saip2020 indico-celery[17639]: 2023-03-08 07:43:20,392 WARNING 0000000000000000 celery.redirected d.open(file, db.DB_HASH, flags,
Mar 08 07:43:20 saip2020 indico-celery[17639]: 2023-03-08 07:43:20,392 WARNING 0000000000000000 celery.redirected DBFileExistsError: (17, 'File ex

bpedersen2 · March 8, 2023, 9:44am

So the same solution should apply:
stop celery, delete the file and restart

see The Celery task scheduler does not seem to be running. This means that email sending and periodic tasks such as event reminders do not work - #10 by ThiefMaster

ThiefMaster · March 8, 2023, 10:09am

I see a python 2.7 mention. This could be because of an outadted celerybeat-schedule file leftover from Indico v2 if you didn’t set up v3 on a new system or somehow ended up copying tmp files as well (in this case deleting it is the way to go).

In case you are still using Indico v2 (on Python 2), you needs to update ASAP since you are using a heavily outdated version with dependencies that are outdated for years as well!