Scheduled celery email tasks failing

1

Installed version: 3.3.5

SMTP is authenticated. Registration emails get delivered.
Sending emails from the registrants list does not get delivered.

indico celery inspect scheduled

->  celery@domain.ovh: OK
    - empty -

1 node online.

This is an extract of celery log. There are a lot of recurrent celery.exceptions.MaxRetriesExceededError

tail -100 /var/log/indico/celery.log | grep "send_email"
2025-04-05 15:52:46,147  WARNING  celery.redirected           File "/var/www/indico/.venv/lib/python3.12/site-packages/indico/core/emails.py", line 54, in send_email_task
2025-04-05 15:52:46,183  INFO     celery.app.trace          Task send_email[6b37c674-d50f-488b-9c86-84d580a4083d] succeeded in 0.4060682887211442s: None
2025-04-05 15:52:46,198  INFO     celery.app.trace          Task send_email[49c1f25b-affb-473d-af1e-bc696a2b3d4a] succeeded in 0.3723299875855446s: None
2025-04-05 15:52:46,231  WARNING  celery.redirected           File "/var/www/indico/.venv/lib/python3.12/site-packages/indico/core/emails.py", line 41, in send_email_task
    do_send_email(email, log_entry, _from_task=True)
2025-04-05 15:52:46,231  WARNING  celery.redirected           File "/var/www/indico/.venv/lib/python3.12/site-packages/indico/core/emails.py", line 119, in do_send_email
2025-04-05 15:52:46,232  WARNING  celery.redirected           File "/var/www/indico/.venv/lib/python3.12/site-packages/indico/core/emails.py", line 45, in send_email_task
2025-04-05 15:52:46,232  WARNING  celery.redirected         celery.exceptions.MaxRetriesExceededError: Can't retry send_email[42229c40-78d8-4082-a66f-0344dd654b76] args:({'to': {'email@gmail.com'}, 'cc': set(), 'bcc': {'anotheremail@mail.ovh'}, 'from': 'me <anotheremail@mail.ovh>', 'reply_to': set(), 'attachments': [], 'subject': '[INDICO] mail', 'body': Markup("<body>\n        \n\n    \n    <p>anonymized body </body>"), 'html': True}, <EventLogEntry: 7676>) kwargs:{}
2025-04-05 15:52:46,237  WARNING  celery.redirected           File "/var/www/indico/.venv/lib/python3.12/site-packages/indico/core/emails.py", line 54, in send_email_task

indico.conf

# Database
SQLALCHEMY_DATABASE_URI = 'postgresql://__DB_USER__:__DB_PWD__@localhost:5432/__DB_NAME__'

# Change this to something long and random
SECRET_KEY = '__SECRET_KEY__'
BASE_URL = 'https://__DOMAIN____PATH__'
USE_PROXY = True
LOGO_URL="__LOGOURL__"
DEFAULT_TIMEZONE = '__TIMEZONE__'
DEFAULT_LOCALE   = '__LANGUAGE__'

REDIS_CACHE_URL = 'redis://localhost:6379/__REDIS_DB__'
CELERY_BROKER   = 'redis://localhost:6379/__CELERY_DB__'

# Change these emails to something sensible
NO_REPLY_EMAIL = 'noreply@__DOMAIN__'
SUPPORT_EMAIL = '__ADMIN_MAIL__'

ENABLE_ROOMBOOKING = True

LOG_DIR           = '/var/log/__APP__'
TEMP_DIR          = '/tmp'
CACHE_DIR         = '__INSTALL_DIR__/cache'
CUSTOMIZATION_DIR = '__DATA_DIR__/custom'

STORAGE_BACKENDS   = {'default': 'fs:__DATA_DIR__/archive'}
ATTACHMENT_STORAGE = 'default'

# Specify the plugins you need here
PLUGINS = {'payment_manual', 'payment_paypal', 'citadel', 'cloud_captchas', 'livesync', 'owncloud', 'payment_sixpay', 'piwik', 'previewer_code', 'previewer_jupyter', 'prometheus', 'storage_s3', 'ursh', 'vc_zoom'}

indico-celery.service

[Unit]
Description=Indico Celery
After=network.target

[Service]
Type=simple
User=__APP__
Group=__APP__
ExecStart=__INSTALL_DIR__/.venv/bin/indico celery worker -B
Restart=always
SyslogIdentifier=indico-celery
UMask=0027

KillMode=mixed
TimeoutStopSec=300

NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
RestrictNamespaces=yes
RestrictRealtime=yes
DevicePolicy=closed
ProtectClock=yes
ProtectHostname=yes
ProtectProc=invisible
ProtectSystem=full
ProtectControlGroups=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
LockPersonality=yes
SystemCallArchitectures=native
SystemCallFilter=~@clock @debug @module @mount @obsolete @reboot @setuid @swap @cpu-emulation @privileged

# Denying access to capabilities that should not be relevant for webapps
# Doc: https://man7.org/linux/man-pages/man7/capabilities.7.html
CapabilityBoundingSet=~CAP_RAWIO CAP_MKNOD
CapabilityBoundingSet=~CAP_AUDIT_CONTROL CAP_AUDIT_READ CAP_AUDIT_WRITE
CapabilityBoundingSet=~CAP_SYS_BOOT CAP_SYS_TIME CAP_SYS_MODULE CAP_SYS_PACCT
CapabilityBoundingSet=~CAP_LEASE CAP_LINUX_IMMUTABLE CAP_IPC_LOCK
CapabilityBoundingSet=~CAP_BLOCK_SUSPEND CAP_WAKE_ALARM
CapabilityBoundingSet=~CAP_SYS_TTY_CONFIG
CapabilityBoundingSet=~CAP_MAC_ADMIN CAP_MAC_OVERRIDE
CapabilityBoundingSet=~CAP_NET_ADMIN CAP_NET_BROADCAST CAP_NET_RAW
CapabilityBoundingSet=~CAP_SYS_ADMIN CAP_SYS_PTRACE CAP_SYSLOG

[Install]
WantedBy=multi-user.target

indico-uwsgi.service

[Unit]
Description=Indico uWSGI
After=network.target

[Service]
ExecStart=__INSTALL_DIR__/.venv/bin/uwsgi --ini __INSTALL_DIR__/uwsgi-indico.ini
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
WorkingDirectory=__INSTALL_DIR__/
SyslogIdentifier=indico-uwsgi
User=__APP__
Group=www-data
UMask=0027
Type=notify
NotifyAccess=all
KillMode=mixed
KillSignal=SIGQUIT
TimeoutStopSec=300

NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
RestrictNamespaces=yes
RestrictRealtime=yes
DevicePolicy=closed
ProtectClock=yes
ProtectHostname=yes
ProtectProc=invisible
ProtectSystem=full
ProtectControlGroups=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
LockPersonality=yes
SystemCallArchitectures=native
#SystemCallFilter=~@clock @debug @module @mount @obsolete @reboot @setuid @swap @cpu-emulation @privileged

# Denying access to capabilities that should not be relevant for webapps
# Doc: https://man7.org/linux/man-pages/man7/capabilities.7.html
CapabilityBoundingSet=~CAP_RAWIO CAP_MKNOD
CapabilityBoundingSet=~CAP_AUDIT_CONTROL CAP_AUDIT_READ CAP_AUDIT_WRITE
CapabilityBoundingSet=~CAP_SYS_BOOT CAP_SYS_TIME CAP_SYS_MODULE CAP_SYS_PACCT
CapabilityBoundingSet=~CAP_LEASE CAP_LINUX_IMMUTABLE CAP_IPC_LOCK
CapabilityBoundingSet=~CAP_BLOCK_SUSPEND CAP_WAKE_ALARM
CapabilityBoundingSet=~CAP_SYS_TTY_CONFIG
CapabilityBoundingSet=~CAP_MAC_ADMIN CAP_MAC_OVERRIDE
CapabilityBoundingSet=~CAP_NET_ADMIN CAP_NET_BROADCAST CAP_NET_RAW
CapabilityBoundingSet=~CAP_SYS_ADMIN CAP_SYS_PTRACE CAP_SYSLOG

[Install]
WantedBy=multi-user.target

There are no emails in the postfix queue.

2

check indico.log for errors

3

An extract of the indico.log after anonym :

2025-04-05 15:50:54,832  ERROR    0000000000000000  -       indico.emails             Could not send email "subject" (attempt 10/10); giving up [{'email@gmail.com': (553, b'5.7.1 <me@domain.ovh>: Sender address rejected: not owned by user contact'), 'me@domain.ovh': (553, b'5.7.1 <me@domain.ovh>: Sender address rejected: not owned by user contact')}]; stored data in /tmp/failed-email-2025-04-05-93ysria0

Note that ‘me@domain.ovh’ is my email, the email contact of the event is another one (manager@domain.ovh) and the email in indico.conf is contact@domain.ovh (user is contact authenticated with password ans TLS enabled).

When opening the send mail popup, I only see managers of the event or admins of indico.

4

https://docs.getindico.io/en/stable/config/settings/#SMTP_ALLOWED_SENDERS

you’ll have to configure this so the sender from address is correctly rewritten

1 Like
5

oh, I have commented it thinking this will allow sending emails using other emails.

# SMTP_ALLOWED_SENDERS = {'*@domain.com','*@domain.ovh'}

I will uncomment it and come back with feedback
Thank you

6

I have also set SMTP_SENDER_FALLBACK .
So every time I add an admin to indico or to an event, I need to add his email to SMTP_ALLOWED_SENDERS so he can send mails to users ?

7

no, you should keep it limited to the address from which your smtp account can actually send. typically that’s a noreply@yourdomain like address. other sender are then rewritten.

1 Like
8

Is there a way to have “email history” with “status (sent/retry/failed)” in the admin area?

9

No, logs are tied to events, but you could query those EventLogEntry objects via indico shell…