I’ve got an issue where Indico has a very large number of “pending” emails (it is sendings some and marking others as pending). Problem one is that I can’t find where it’s logging the reason that the messages are pending, rather than sent. Problem two is what I’m trying to address in this query: how do I change the length of time celery waits before retrying the messages?
I have tried to set:
SCHEDULED_TASK_OVERRIDE = { ‘send_email’: timedelta(minutes=5) }
My read of the documentation suggests this should work, but I get an error at startup about the format of this variable.
2019-05-11 10:22:05,301 ERROR celery.beat Invalid entry in ScheduledTaskOverride: send_email
Email sending is not “scheduled”, those tasks run immediately (assuming Celery is running). The “scheduled” tasks are the cronjob-like tasks that run on a regular schedule.
There are definitely pending emails.
2019-05-11 09:47:35,794 INFO celery.app.trace Task send_email[f493f6b4-f42a-4d49-a486-728cadf69982] retry: Retry in 3600s
In the logs for the event, these are all marked as pending.
And email sending is noted as one of the scheduler registered tasks:
(.venv) indico@adm:~$ indico celery inspect registered
-> celery@adm.sjc.dns-oarc.net: OK
* category_cleanup
* category_suggestions
* event_reminders
* indico.modules.events.static.tasks.build_static_site
* roombooking_occurrences
* send_email
* static_sites_cleanup
* survey_start_notifications
* temp_cleanup
So if it’s not the event scheduler that manages how long these sit in a queue before being retried, what is it?
I can also see these emails in the scheduler by running “indico celery inspect scheduled”
If you get a retry then sending the email failed for some reason. In this case indico will retry in increasing intervals (30, 60, 120, 300, 600, 1800, 3600, 3600, 7200 seconds) before giving up. Check indico.log if there’s any information on why sending the email failed; it should have something with “Could not send email” in there.
Is there some way I can override that interval, or manually force an immediate retry?
None that I know of. I’d have expected the celery command to have some way to modify the delay of a queued task or run it immediately but I couldn’t find anything.