Indico server went down with database errors

Qudor-Engineer · April 4, 2024, 6:07pm

Hi everybody,

Our indico instance was working normally then we started receiving errors in the email about piwik as shown in the image below:

then other errors about previewer jupyter

then we restarted the instance using: “sudo systemctl restart indico-celery.service indico-uwsgi.service”

then the indico crashed and when open the website nothing showed up except “Internal server error” Then we removed all plugins from indico.conf file then the website shows: “Something went wrong” as in the image below:
a45b4442-f7ef-408b-8fd2-879c2b4b8bbd

and we started receiving errors about some database tables missing. we didn’t do any modification before or after crashing for the code or configuration except removing the plugins fron the configuration.
this is the logs just before the crash happened:
log.zip (34.0 KB)

and this is when I checked the database

:

I hope you can help me to debug and solve the issue.
indico version: 3.1

ThiefMaster · April 4, 2024, 6:26pm

first of all, your indico version is heavily outdated. you need to update ASAP.

anyway, it looks like you updated something (indico or plugins or some libraries) but not everything. pip install -U 'indico-plugins<3.2' 'indico<3.2' should work to fix this without updating to a big new indico release. restart the indico systemd services afterwards.

however, i highly recommend you to use this opportunity to update properly…

Qudor-Engineer · April 4, 2024, 6:40pm

Thank you very much for the fast reply.
I can’t update the indico now because there are many modifications that we made to the codes to meet our requirements.
is there any other suggestions to solve the errors?

ThiefMaster · April 4, 2024, 6:46pm

update the plugins or disable them…

Qudor-Engineer · April 4, 2024, 6:48pm

when I checked the shell commands history I found someone updated the plugins “indico db --all-plugins upgrade”
could be the issue? and if sow how to downgrade the plugins?

ThiefMaster · April 4, 2024, 6:58pm

no, that was just upgrading the db.

please paste the output of these commands:

pip freeze
indico db current -v

Qudor-Engineer · April 4, 2024, 7:18pm

I already disabled all plugins.
the output for "pip freeze:

alembic==1.10.4
amqp==5.1.1
asttokens==2.2.1
async-timeout==4.0.2
attrs==23.1.0
Authlib==1.2.0
Babel==2.12.1
backcall==0.2.0
bcrypt==4.0.1
billiard==4.1.0
bleach==6.0.0
blinker==1.6.2
boto3==1.20.37
botocore==1.23.37
cachelib==0.9.0
captcha==0.5.0
celery==5.3.1
certifi==2023.5.7
cffi==1.15.1
charset-normalizer==3.1.0
click==8.1.3
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.2.0
colorclass==2.2.2
cryptography==41.0.7
decorator==5.1.1
defusedxml==0.7.1
Deprecated==1.2.13
distro==1.8.0
dnspython==2.3.0
email-validator==1.2.1
entrypoints==0.3
executing==1.2.0
feedgen==0.9.0
Flask==2.3.2
Flask-Babel==2.0.0
Flask-Caching==2.0.2
Flask-Limiter==3.3.0
flask-marshmallow==0.15.0
Flask-Migrate==4.0.4
Flask-Multipass==0.4.9
Flask-PluginEngine==0.4.1
Flask-SQLAlchemy==3.0.3
flask-webpackext==1.0.2
Flask-WTF==1.1.1
greenlet==2.0.2
hiredis==2.2.2
html2text==2020.1.16
html5lib==1.1
icalendar==5.0.5
idna==3.4
importlib-metadata==6.6.0
importlib-resources==5.12.0
indico==3.2.9
indico-fonts==1.2
indico-plugin-citadel==3.1
indico-plugin-livesync==3.1
indico-plugin-payment-billplz @ file:///home/ubuntu/payment_billplz/dist/indico_plugin_payment_billplz-1.0-py3-none-any.whl
indico-plugin-payment-manual==3.0
indico-plugin-payment-paypal==3.1
indico-plugin-payment-sixpay==3.1
indico-plugin-payment-stripe @ file:///home/ubuntu/indico-plugin-stripe/dist/indico_plugin_payment_stripe-0.0.1.dev0-py2.py3-none-any.whl
indico-plugin-piwik==3.0
indico-plugin-previewer-code==3.0
indico-plugin-previewer-jupyter==3.0
indico-plugin-storage-s3==3.0
indico-plugin-ursh==3.1
indico-plugin-vc-zoom==3.1
indico-plugins==3.1
ipython==8.13.2
ipython-genutils==0.2.0
iso4217==1.6.20180829
itsdangerous==2.1.2
jedi==0.18.2
Jinja2==3.1.3
jmespath==0.10.0
jsonschema==4.17.3
jupyter-core==4.9.1
kombu==5.3.1
limits==3.4.0
lxml==4.9.2
Mako==1.2.4
Markdown==3.4.3
markdown-it-py==2.2.0
MarkupSafe==2.1.2
marshmallow==3.19.0
marshmallow-dataclass==8.5.14
marshmallow-enum==1.5.1
marshmallow-oneofschema==3.0.1
marshmallow-sqlalchemy==0.29.0
matplotlib-inline==0.1.6
mdurl==0.1.2
mistune==0.8.4
mypy-extensions==1.0.0
nbconvert==5.6.1
nbformat==5.1.3
node-semver==0.8.1
ordered-set==4.1.0
packaging==23.1
pandocfilters==1.5.0
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
pillow==10.2.0
prompt-toolkit==3.0.38
psycopg2==2.9.6
ptyprocess==0.7.0
pure-eval==0.2.2
pycountry==22.3.5
pycparser==2.21
Pygments==2.15.1
PyJWT==2.3.0
pynpm==0.1.2
pyparsing==3.0.6
pypdf==3.12.1
PyPDF2==1.26.0
pypng==0.20220715.0
pyrsistent==0.19.3
python-dateutil==2.8.2
pytz==2023.3
pywebpack==1.2.0
PyYAML==6.0
qrcode==7.4.2
redis==4.5.4
reportlab==3.6.13
requests==2.31.0
rich==13.3.5
s3transfer==0.5.0
sentry-sdk==1.21.1
simplejson==3.19.1
six==1.16.0
speaklater==1.3
SQLAlchemy==1.4.48
stack-data==0.6.2
stripe==8.9.0
terminaltables==3.1.10
testpath==0.5.0
tinycss2==1.1.1
traitlets==5.9.0
translitcodec==0.7.0
typing-inspect==0.8.0
typing_extensions==4.5.0
tzdata==2023.3
ua-parser==0.16.1
urllib3==1.26.17
uWSGI==2.0.20
vine==5.0.0
wcwidth==0.2.6
webargs==8.2.0
webencodings==0.5.1
Werkzeug==2.3.8
wrapt==1.15.0
WTForms==3.0.1
WTForms-dateutil==0.1
WTForms-SQLAlchemy==0.3
XlsxWriter==3.1.0
zipp==3.15.0

the output of " indico db current -v":

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
Current revision(s) for postgresql:///indico:
Rev: b36825c7869e
Parent: dc53d6e8c576
Path: /opt/indico/.venv/lib/python3.9/site-packages/indico/migrations/versions/20210927_1027_b36825c7869e_add_registration_tags.py

    Add registration tags
    
    Revision ID: b36825c7869e
    Revises: dc53d6e8c576
    Create Date: 2021-09-27 10:27:39.335488

Qudor-Engineer · April 4, 2024, 7:20pm

this is some of the latest celery log

Apr 05 03:05:02 ip-172-31-30-213 indico-celery[872561]:                                                              ^
Apr 05 03:05:02 ip-172-31-30-213 indico-celery[872561]: [SQL: SELECT events.reminders.id AS events_reminders_id, events.reminders.event_id AS e>
Apr 05 03:05:02 ip-172-31-30-213 indico-celery[872561]: FROM events.reminders JOIN events.events ON events.events.id = events.reminders.event_id
Apr 05 03:05:02 ip-172-31-30-213 indico-celery[872561]: WHERE NOT events.reminders.is_sent AND NOT events.events.is_deleted AND NOT (EXISTS (SE>
Apr 05 03:05:02 ip-172-31-30-213 indico-celery[872561]: FROM events.labels
Apr 05 03:05:02 ip-172-31-30-213 indico-celery[872561]: WHERE events.labels.id = events.events.label_id AND events.labels.is_event_not_happenin>
Apr 05 03:05:02 ip-172-31-30-213 indico-celery[872561]: [parameters: {'scheduled_dt_1': datetime.datetime(2024, 4, 4, 19, 5, 0, 114726)}]
Apr 05 03:05:02 ip-172-31-30-213 indico-celery[872561]: (Background on this error at: https://sqlalche.me/e/14/f405)
Apr 05 03:10:00 ip-172-31-30-213 indico-celery[872563]: 2024-04-05 03:10:00,069  INFO     0000000000000000  -       celery.beat               S>
Apr 05 03:10:00 ip-172-31-30-213 indico-celery[872548]: 2024-04-05 03:10:00,076  INFO     0000000000000000  -       celery.worker.strategy    T>
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]: 2024-04-05 03:10:00,140  ERROR    0000000000000000  -       celery.app.trace          T>
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]: Traceback (most recent call last):
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:   File "/opt/indico/.venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line >
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:     self.dialect.do_execute(
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:   File "/opt/indico/.venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", li>
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:     cursor.execute(statement, parameters)
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]: psycopg2.errors.UndefinedColumn: column labels.is_event_not_happening does not exist
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]: LINE 5: ...ERE events.labels.id = events.events.label_id AND events.lab...
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:                                                              ^
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]: The above exception was the direct cause of the following exception:
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]: Traceback (most recent call last):
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:   File "/opt/indico/.venv/lib/python3.9/site-packages/celery/app/trace.py", line 477, i>
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:     R = retval = fun(*args, **kwargs)
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:   File "/opt/indico/.venv/lib/python3.9/site-packages/sentry_sdk/integrations/celery.py>
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:     reraise(*exc_info)
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:   File "/opt/indico/.venv/lib/python3.9/site-packages/sentry_sdk/_compat.py", line 60, >
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:     raise value
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:   File "/opt/indico/.venv/lib/python3.9/site-packages/sentry_sdk/integrations/celery.py>
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:     return f(*args, **kwargs)
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:   File "/opt/indico/.venv/lib/python3.9/site-packages/indico/core/celery/core.py", line>
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:     rv = super().__call__(*args, **kwargs)
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:   File "/opt/indico/.venv/lib/python3.9/site-packages/celery/app/trace.py", line 760, i>
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:     return self.run(*args, **kwargs)
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:   File "/opt/indico/.venv/lib/python3.9/site-packages/indico/core/celery/util.py", line>
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:     return f(*args, **kwargs)
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:   File "/opt/indico/.venv/lib/python3.9/site-packages/indico/modules/events/reminders/t>
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:     reminders = (EventReminder.query
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:   File "/opt/indico/.venv/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 27>
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:     return self._iter().all()
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:   File "/opt/indico/.venv/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 29>
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:     result = self.session.execute(
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:   File "/opt/indico/.venv/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line >
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:     result = conn._execute_20(statement, params or {}, execution_options)
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:   File "/opt/indico/.venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line >
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:     return meth(self, args_10style, kwargs_10style, execution_options)
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:   File "/opt/indico/.venv/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line>
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:     return connection._execute_clauseelement(
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:   File "/opt/indico/.venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line >
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:     ret = self._execute_context(
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:   File "/opt/indico/.venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line >
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:     self._handle_dbapi_exception(
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:   File "/opt/indico/.venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line >
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:     util.raise_(
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:   File "/opt/indico/.venv/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line >
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:     raise exception
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:   File "/opt/indico/.venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line >
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:     self.dialect.do_execute(
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:   File "/opt/indico/.venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", li>
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:     cursor.execute(statement, parameters)
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]: sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column labels.is_eve>
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]: LINE 5: ...ERE events.labels.id = events.events.label_id AND events.lab...
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]:                                                              ^
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]: [SQL: SELECT events.reminders.id AS events_reminders_id, events.reminders.event_id AS e>
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]: FROM events.reminders JOIN events.events ON events.events.id = events.reminders.event_id
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]: WHERE NOT events.reminders.is_sent AND NOT events.events.is_deleted AND NOT (EXISTS (SE>
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]: FROM events.labels
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]: WHERE events.labels.id = events.events.label_id AND events.labels.is_event_not_happenin>
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]: [parameters: {'scheduled_dt_1': datetime.datetime(2024, 4, 4, 19, 10, 0, 95190)}]
Apr 05 03:10:02 ip-172-31-30-213 indico-celery[872561]: (Background on this error at: https://sqlalche.me/e/14/f405)

ThiefMaster · April 4, 2024, 7:56pm

indico==3.2.9

So someone updated to the semi-latest Indico version. Run pip install -U indico-plugins and indico db upgrade and indico db --all-plugins upgrade.

Qudor-Engineer · April 4, 2024, 7:59pm

If I run this command to upgrade indico does that mean we will lose the modifications that we made?

ThiefMaster · April 4, 2024, 8:01pm

You already lost those modifications since someone upgraded to 3.2.9 - unless that person created a 3.2.9 build containing the modifications.

You either need to upgrade the plugins to match the Indico version, OR install your old customized version again. I hope those modifications were not made by editing files directly on the server…

ThiefMaster · April 4, 2024, 8:03pm

Just so you are aware of the current state of your installtaion:

Indico v3.2.9
Indico Plugins v3.1.x - not compatible with v3.2.9, need to be updated (OR Indico downgraded to v3.1.x)
Database schema for v3.1.x - you need to run the indico db upgrade commands OR downgrade Indico to v3.1.x again.

Qudor-Engineer · April 4, 2024, 8:16pm

there’s something weird because I checked command history for root and indico users and there’s no upgrade command except “indico db --all-plugins upgrade”.

how to downgrade indico to 3.1?

ThiefMaster · April 4, 2024, 8:28pm

Just downgrading won’t bring back your customizations. You need to install your custom 3.1 python wheel file again. If you can find it or build it again, install it using pip install /path/to/indico-*.whl

Qudor-Engineer · April 4, 2024, 8:46pm

thank you very much for your help and time, I really appreciate it.