Logger mails on indico 3.1

Administrators
1

On my updated Testsystem (3.1) Indico send emails about python problems every night from logger@SERVERNAME.de to the adminsitration account. It can´t connect to gravator and indico.io not really a big problem for me.

2022-07-18 00:02:07,438  0000000000000000  -       celery.app.trace - ERROR trace.py:265 -- Task update_gravatars[7d25336c-e62d-4374-9f54-dcacb31a62fa] raised unexpected: ConnectionError(MaxRetryError("HTTPSConnectionPool(host='www.gravatar.com', port=443): Max retries exceeded with url:
/avatar/0bb07d93ab945e5fe0391584b09d77c4?d=mp&s=256 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f96348b33d0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"))

Traceback (most recent call last):
  File "/opt/indico/.venv/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/opt/indico/.venv/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/opt/indico/.venv/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/indico/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/opt/indico/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/opt/indico/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
    conn.connect()
  File "/opt/indico/.venv/lib/python3.9/site-packages/urllib3/connection.py", line 358, in connect
    conn = self._new_conn()
  File "/opt/indico/.venv/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f96348b33d0>: Failed to establish a new connection: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/indico/.venv/lib/python3.9/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/opt/indico/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/opt/indico/.venv/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.gravatar.com', port=443): Max retries exceeded with url: /avatar/0bb07d93ab945e5fe0391584b09d77c4?d=mp&s=256 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f96348b33d0>: Failed to establish a new
connection: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/indico/.venv/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/indico/.venv/lib/python3.9/site-packages/sentry_sdk/integrations/celery.py", line 200, in _inner
    reraise(*exc_info)
  File "/opt/indico/.venv/lib/python3.9/site-packages/sentry_sdk/_compat.py", line 54, in reraise
    raise value
  File "/opt/indico/.venv/lib/python3.9/site-packages/sentry_sdk/integrations/celery.py", line 195, in _inner
    return f(*args, **kwargs)
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/core/celery/core.py", line 138, in __call__
    rv = super().__call__(*args, **kwargs)
  File "/opt/indico/.venv/lib/python3.9/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/core/celery/util.py", line 31, in wrapper
    return f(*args, **kwargs)
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/modules/users/tasks.py", line 29, in update_gravatars
    gravatar, lastmod = get_gravatar_for_user(user, identicon=False, lastmod=user.picture_metadata['lastmod'])
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/modules/users/util.py", line 425, in get_gravatar_for_user
    resp = requests.get(gravatar_url, params=params, headers=headers)
  File "/opt/indico/.venv/lib/python3.9/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/opt/indico/.venv/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/indico/.venv/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/indico/.venv/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/opt/indico/.venv/lib/python3.9/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.gravatar.com', port=443): Max retries exceeded with url: /avatar/0bb07d93ab945e5fe0391584b09d77c4?d=mp&s=256 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f96348b33d0>: Failed to establish a new
connection: [Errno 101] Network is unreachable'))

I have set the global http_proxy, HTTP_PROXY, https_proxy and HTTPS_proxy in linux already (we are using a proxy in the dmz).
It would be also okay if i can just disable emails about error ins python completely…

2

You could disable error emails altogether on your test system, and possibly use the logging config (it’s using the standard Python logging dictConfig format in YAML) to silence specific ones.

But it appears you have the Sentry integration enabled? If that’s indeed the case, you should probably completely disable error email by disabling the email handler in logging.yaml - and in sentry you can simply ignore this particular error.

Anyway, the cleanest solution would be to configure your proxy also for the celery worker… I think you need to set the env var(s) in the systemd unit file to do so. That way the requests will most likely simply succeed.

3

i thanks for the tipp i edited the systemd entry with the command “systemctl edit indico-celery” and added the Proxys:

[Service]
Environment="HTTP_PROXY=http://SERVERNAME:PORT"
Environment="HTTPS_PROXY=http://SERVERNAME:PORT"

Tonight no error messages so far.