Root error when upgrading pyenv for 3.3

Administrators
1

Hi! I’m upgrading to 3.3 and running into this error:

(indico) indico@indico:/root$ indico setup upgrade-python --force-version 3.12
updating pyenv
/opt/indico/.pyenv/bin/pyenv: line 73: **cd: /root: Permission denied**

**pyenv update failed - continue anyway?** [y/N]: y
Warning: 3.12 is not within ~=3.9.0 spec
Continue anyway? [y/N]: y
WARNING: You are upgrading from 3.9 to 3.12. This upgrade CANNOT be done in-place.
You will need to `pip install indico` (and any other packages such as plugins)
again after this upgrade! Do not perform this update, unless you are updating to
a new major Indico release which is documented to require this Python version!
Continue? [y/N]: y
Traceback (most recent call last):
  File "/opt/indico/.venv/bin/indico", line 8, in <module>
    sys.exit(cli())
  File "/opt/indico/.venv/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/indico/.venv/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/indico/.venv/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/cli/util.py", line 108, in invoke
    return self._impl.invoke(ctx)
  File "/opt/indico/.venv/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/indico/.venv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/indico/.venv/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/indico/.venv/lib/python3.9/site-packages/indico/cli/setup.py", line 280, in upgrade_python
    pyenv_local_version = Version(subprocess.run(['pyenv', 'version-name'], capture_output=True,
  File "/opt/indico/.venv/lib/python3.9/site-packages/packaging/version.py", line 198, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
packaging.version.InvalidVersion: Invalid version: ''

Where did I go wrong?

2

what does echo $HOME give you? My guess would be that you used su/sudo to switch to the indico user in a way that does not update $HOME. su - indico should do the job for sure

edit: Looks like the line from which the error comes uses $PWD so it’s more likely to be that env var and not $HOME that’s incorrect - but the cause is likely the same

3

Thanks for the tip! After taking a break and re-initiating an ssh, I was able to su - indico, without the root$ prefix, and update to 3.3.

For the record, I got the following:

root@indico:~# $PWD
-bash: /root: Is a directory
root@indico:~# $HOME
-bash: /root: Is a directory

and

root@indico:~# su - indico
(indico) indico@indico:~$ $HOME
-bash: /opt/indico: Is a directory
(indico) indico@indico:~$ $PWD
-bash: /opt/indico: Is a directory

However, I now have an Internal Server Error on display. I followed the specific 3.3 update steps, but it seems uwsgi.service is missing somehow!

(.venv) indico@indico:~$ systemctl status uwsgi.service
Unit uwsgi.service could not be found.
(.venv) indico@indico:~$ su - root
root@indico:~# systemctl status uwsgi.service
Unit uwsgi.service could not be found.
root@indico:~# systemctl daemon-reload
root@indico:~# systemctl restart uwsgi.service
Failed to restart uwsgi.service: Unit uwsgi.service not found.

Running indico shell gives me the following:

(.venv) indico@indico:~$ indico shell
— Logging error —
Traceback (most recent call last):
File “/opt/indico/.pyenv/versions/3.12.2/lib/python3.12/logging/handlers.py”, line 1092, in emit
smtp.send_message(msg)
File “/opt/indico/.pyenv/versions/3.12.2/lib/python3.12/smtplib.py”, line 975, in send_message
return self.sendmail(from_addr, to_addrs, flatmsg, mail_options,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/indico/.pyenv/versions/3.12.2/lib/python3.12/smtplib.py”, line 890, in sendmail
raise SMTPRecipientsRefused(senderrs)
smtplib.SMTPRecipientsRefused: {‘contact@eiss-europa.com’: (450, b’4.1.8 logger@indico.eiss-europa.com.localdomain: Sender address rejected: Domain not found’)}
Call stack:
File “/opt/indico/.venv/bin/indico”, line 8, in
sys.exit(cli())
File “/opt/indico/.venv/lib/python3.12/site-packages/click/core.py”, line 1157, in call
return self.main(*args, **kwargs)
File “/opt/indico/.venv/lib/python3.12/site-packages/click/core.py”, line 1078, in main
rv = self.invoke(ctx)
File “/opt/indico/.venv/lib/python3.12/site-packages/click/core.py”, line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File “/opt/indico/.venv/lib/python3.12/site-packages/click/core.py”, line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File “/opt/indico/.venv/lib/python3.12/site-packages/click/core.py”, line 783, in invoke
return __callback(*args, **kwargs)
File “/opt/indico/.venv/lib/python3.12/site-packages/click/decorators.py”, line 33, in new_func
return f(get_current_context(), *args, **kwargs)
File “/opt/indico/.venv/lib/python3.12/site-packages/flask/cli.py”, line 385, in decorator
app = ctx.ensure_object(ScriptInfo).load_app()
File “/opt/indico/.venv/lib/python3.12/site-packages/flask/cli.py”, line 330, in load_app
app: Flask | None = self.create_app()
File “/opt/indico/.venv/lib/python3.12/site-packages/indico/cli/util.py”, line 26, in _create_app
return make_app()
File “/opt/indico/.venv/lib/python3.12/site-packages/indico/web/flask/app.py”, line 432, in make_app
if not plugin_engine.load_plugins(app):
File “/opt/indico/.venv/lib/python3.12/site-packages/flask_pluginengine/engine.py”, line 43, in load_plugins
plugins = self._import_plugins(state.app)
File “/opt/indico/.venv/lib/python3.12/site-packages/flask_pluginengine/engine.py”, line 63, in _import_plugins
state.logger.error(‘Plugin %s does not exist’, name)
File “/opt/indico/.pyenv/versions/3.12.2/lib/python3.12/logging/init.py”, line 1568, in error
self._log(ERROR, msg, args, **kwargs)
File “/opt/indico/.pyenv/versions/3.12.2/lib/python3.12/logging/init.py”, line 1684, in _log
self.handle(record)
File “/opt/indico/.pyenv/versions/3.12.2/lib/python3.12/logging/init.py”, line 1700, in handle
self.callHandlers(record)
File “/opt/indico/.venv/lib/python3.12/site-packages/sentry_sdk/integrations/logging.py”, line 99, in sentry_patched_callhandlers
return old_callhandlers(self, record)
Message: ‘Plugin %s does not exist’
Arguments: (‘payment_stripe’,)
Traceback (most recent call last):
File “/opt/indico/.venv/bin/indico”, line 8, in
sys.exit(cli())
^^^^^
File “/opt/indico/.venv/lib/python3.12/site-packages/click/core.py”, line 1157, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/indico/.venv/lib/python3.12/site-packages/click/core.py”, line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File “/opt/indico/.venv/lib/python3.12/site-packages/click/core.py”, line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/indico/.venv/lib/python3.12/site-packages/click/core.py”, line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/indico/.venv/lib/python3.12/site-packages/click/core.py”, line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/indico/.venv/lib/python3.12/site-packages/click/decorators.py”, line 33, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/indico/.venv/lib/python3.12/site-packages/flask/cli.py”, line 385, in decorator
app = ctx.ensure_object(ScriptInfo).load_app()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/indico/.venv/lib/python3.12/site-packages/flask/cli.py”, line 330, in load_app
app: Flask | None = self.create_app()
^^^^^^^^^^^^^^^^^
File “/opt/indico/.venv/lib/python3.12/site-packages/indico/cli/util.py”, line 26, in _create_app
return make_app()
^^^^^^^^^^
File “/opt/indico/.venv/lib/python3.12/site-packages/indico/web/flask/app.py”, line 433, in make_app
raise Exception(‘Could not load some plugins: {}’.format(', '.join(plugin_engine.get_failed_plugins(app))))
Exception: Could not load some plugins: payment_stripe

4

You forgot the echo in front of it.

The service is named indico-uwsgi.service since v3.0.

As indicated during the indico setup upgrade-python step:

You will need to pip install indico (and any other packages such as plugins) again after this upgrade!

Since the stripe plugin is not provided by us (via the indico-plugins meta package), you have to reinstall it yourself. In case you installed it from pypi before you can simply run pip install indico-plugin-payment-stripe to reinstall it (I do not know, however, if it’s compatible considering it has not been updated in the last 5 years). Alternatively, disable the plugin in indico.conf for now.

5

My bad! Here it is:

(.venv) indico@indico:~$ echo $home

(.venv) indico@indico:~$ echo $PWD
/opt/indico
(.venv) indico@indico:~$

Ah right, here is what I have:

(.venv) indico@indico:~$ systemctl status indico-uwsgi.service
● indico-uwsgi.service - Indico uWSGI
     Loaded: loaded (/etc/systemd/system/indico-uwsgi.service; enabled; vendor >
     Active: active (running) since Sun 2024-03-31 20:23:27 UTC; 14min ago
   Main PID: 958 (uwsgi)
     Status: "uWSGI is ready"
      Tasks: 9 (limit: 2220)
     Memory: 306.0M
        CPU: 6.900s
     CGroup: /system.slice/indico-uwsgi.service
             ├─ 958 "indico uWSGI master"
             ├─1213 "indico uWSGI worker 1"
             ├─1214 "indico uWSGI worker 2"
             ├─1216 "indico uWSGI worker 3"
             └─1218 "indico uWSGI worker 4"
lines 1-14/14 (END)

I did do that as well, but the plugin most likely isn’t supported for now. I’m not using it currently so it’s not an issue.

6

That status output looks like you successfully restarted it. Simply restart it again (also indico-celery.service) after disabling the broken plugin in your config.

After that everything should be working. Don’t forget to indico db upgrade though if you haven’t done so yet.

7

You were right! I would not have guessed a broken plugin could cause the whole Indico instance to break.

All seems working and up-to-date:

Indico version installed (3.3)
Python version used (3.12.2)
Postgres version used (13.9)
Operating system used (Ubuntu 22.04)
Debug mode (disabled)

Thanks again for your support and reactivity, we really appreciate it :smiley:

1 Like
8

Apologies for re-opening this thread @ThiefMaster , but could this be an issue?

I have just noticed that my instance does not seem to be sending automated emails anymore. Could this be related to localdomain?

The settings in /opt/indico/etc/indico.conf are as follow:

# Email settings
SMTP_SERVER = ('mail.REDACTED.com', 587)
SMTP_USE_TLS = True
SMTP_LOGIN = 'contact@eiss-europa.com'
SMTP_PASSWORD = 'REDACTED'
SUPPORT_EMAIL = 'contact@eiss-europa.com'
PUBLIC_SUPPORT_EMAIL = 'contact@eiss-europa.com'
NO_REPLY_EMAIL = 'contact@eiss-europa.com'
9

Check systemctl status indico-celery.service to make sure the task daemon is running since all emails go through it.

Also, this particular error happens when sending the email about an exception that occurred failed. So check /opt/indico/log/indico.log for the error that happened right before it.

You can also edit /opt/indico/etc/logging.yaml and replace

  email:
    class: indico.core.logger.FormattedSubjectSMTPHandler
    formatter: email
    level: ERROR

with

  email:
    class: indico.core.logger.FormattedSubjectSMTPHandler
    fromaddr: contact@eiss-europa.com
    formatter: email
    level: ERROR

That way those log emails are sent from a valid email address that shouldn’ get rejected by your mail server. There’s a good chance that those error emails were already broken before the update, but now something caused an exception (which is the interesting thing to know about more - see above), and then sending an email about this failed.

10

Celery gives me this:

● indico-celery.service - Indico Celery
     Loaded: loaded (/etc/systemd/system/indico-celery.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2024-03-31 21:45:20 UTC; 1h 14min ago
   Main PID: 957 (indico)
      Tasks: 4 (limit: 2220)
     Memory: 567.5M
        CPU: 1min 96ms
     CGroup: /system.slice/indico-celery.service
             ├─ 957 /opt/indico/.venv/bin/python /opt/indico/.venv/bin/indico celery worker -B
             ├─1207 /opt/indico/.venv/bin/python /opt/indico/.venv/bin/indico celery worker -B
             ├─1208 /opt/indico/.venv/bin/python /opt/indico/.venv/bin/indico celery worker -B
             └─1218 /opt/indico/.venv/bin/python /opt/indico/.venv/bin/indico celery worker -B

Mar 31 22:53:33 indico.eiss-europa.com indico-celery[957]: 2024-03-31 22:53:33,727  INFO     0000000000000000  -       celery.worker.strategy    Task send_email[618fdea8-9c4a-47d9-9fd8-7425d737f027] received
Mar 31 22:53:33 indico.eiss-europa.com indico-celery[1208]: 2024-03-31 22:53:33,771  INFO     0000000000000000  -       celery.app.trace          Task send_email[618fdea8-9c4a-47d9-9fd8-7425d737f027] retry: Retry >
Mar 31 22:53:36 indico.eiss-europa.com indico-celery[957]: 2024-03-31 22:53:36,600  INFO     0000000000000000  -       celery.worker.strategy    Task send_email[6d7039d4-94d3-44fd-bf4d-39b807d65071] received
Mar 31 22:53:36 indico.eiss-europa.com indico-celery[1208]: 2024-03-31 22:53:36,644  INFO     0000000000000000  -       celery.app.trace          Task send_email[6d7039d4-94d3-44fd-bf4d-39b807d65071] retry: Retry >
Mar 31 22:55:00 indico.eiss-europa.com indico-celery[1207]: 2024-03-31 22:55:00,000  INFO     0000000000000000  -       celery.beat               Scheduler: Sending due task indico_vc_zoom.task.refresh_token (indi>
Mar 31 22:55:00 indico.eiss-europa.com indico-celery[957]: 2024-03-31 22:55:00,002  INFO     0000000000000000  -       celery.worker.strategy    Task indico_vc_zoom.task.refresh_token[f93d90e9-8e36-49f6-81aa-ef0ee>
Mar 31 22:55:00 indico.eiss-europa.com indico-celery[1207]: 2024-03-31 22:55:00,005  INFO     0000000000000000  -       celery.beat               Scheduler: Sending due task event_reminders (event_reminders)
Mar 31 22:55:00 indico.eiss-europa.com indico-celery[957]: 2024-03-31 22:55:00,009  INFO     0000000000000000  -       celery.worker.strategy    Task event_reminders[38e60cfd-452b-4708-b673-72e97c716a8c] received
Mar 31 22:55:00 indico.eiss-europa.com indico-celery[1208]: 2024-03-31 22:55:00,018  INFO     0000000000000000  -       celery.app.trace          Task indico_vc_zoom.task.refresh_token[f93d90e9-8e36-49f6-81aa-ef0e>
Mar 31 22:55:00 indico.eiss-europa.com indico-celery[1218]: 2024-03-31 22:55:00,021  INFO     0000000000000000  -       celery.app.trace          Task event_reminders[38e60cfd-452b-4708-b673-72e97c716a8c] succeede>
lines 1-23/23 (END)

Here is the error in the logs:

2024-03-31 22:22:21,009  WARNING  0000000000000000  -       indico.emails             Could not send email "[Indico] Email Server Test" (attempt 1/10); retry in 30s [SMTP.starttls() got an unexpected keyword argument 'keyfile'

I am editing logging.yaml and will report back.

11

Are you setting any of these two options by any chance?

  • SMTP_CERTFILE
  • SMTP_KEYFILE
12

Nevermind, I see the problem. It’s a bug with SMTP_USE_TLS = True which we didn’t spot because our own internal mailserver uses a plain connection on port 25.

Changed in version 3.12: The deprecated keyfile and certfile parameters have been removed.

Probably needs a 3.3.1 release to fix it. I’ll try to reproduce it locally. If I provide you a Python wheel, would you be willing to install it to test if it fixes the bug before I do a proper release with it?

1 Like
13

Sure, happy to help. But I’ll need detailed instructions 'cause I’m really not a pro at this!
I guess that’s the price of early adoption XD

14

Go to this link and download the indico-wheel artifact (it’s a zip file). Unzip it, and upload/copy the indico-3.3.1.dev0+202404010145.532230bd81-py3-none-any.whl file to your server (e.g. to /tmp/).

Then as the indico user on your server run:

  • pip install /tmp/indico-3.3.1.dev0+202404010145.532230bd81-py3-none-any.whl
  • systemctl restart indico-uwsgi.service indico-celery.service

This should fix the email problem.

1 Like
15

I can confirm that this works!
I tested sending a document using the new feature and I got the email right away.

GG!

1 Like
16

Thanks, just released v3.3.1 with the fix. You can pip install -U indico if you want to get the proper release, but besides the version number it is identical to the wheel you installed.

1 Like
17

Noted, thanks! I’ll probably switch back to the non-dev version at the next update.