Upgrade 3.1.1 to 3.2.1 DB error

I’m having an DB issue upgrading from 3.1.1 to 3.2.1. I’m basically going through the steps from Upgrade — Indico 3.2.1 documentation. Everything looks to work okay (there was an error during the db ugrade about CACHE_BACKEND, but I was able to remove that by changing indico.conf). I’ve moved back to backup a few times and tried a couple different steps (e.g. move to 3.2.0 1st).

But every time, when accessing the website after the upgrade I get a database error page. The following is the error I’m getting from log/indico.log (or the postgres logs). It’s running postgresql 12. The relation ‘indico.affiliations’ not existing appears to be the problem. Looking through the DB pre-update, the only place I can find ‘affiliation’ is in the designer_templates table as a “type” in the json data. Notice it is also singular so it appears unrelated with the error being caused by something new in the upgrade.

I’m relatively new to administering indico, so it may be an obvious error. But any help would be appreciated.

Error below:

name;Traceback (most recent call last):
  File "/var/local/indico/.venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
    self.dialect.do_execute(
  File "/var/local/indico/.venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "indico.affiliations" does not exist
LINE 2: ...1.user_id AND emails_1.is_primary LEFT OUTER JOIN indico.aff...
                                                             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/local/indico/.venv/lib/python3.9/site-packages/flask/app.py", line 1820, in full_dispatch_request
    rv = self.dispatch_request()
  File "/var/local/indico/.venv/lib/python3.9/site-packages/flask/app.py", line 1796, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/var/local/indico/.venv/lib/python3.9/site-packages/indico/modules/events/controllers/entry.py", line 35, in event_or_shorturl
    return RHDisplayEvent().process()
  File "/var/local/indico/.venv/lib/python3.9/site-packages/indico/web/rh.py", line 291, in process
    handle_sqlalchemy_database_error()  # this will re-raise an exception
  File "/var/local/indico/.venv/lib/python3.9/site-packages/indico/web/rh.py", line 281, in process
    res = self._do_process()
  File "/var/local/indico/.venv/lib/python3.9/site-packages/indico/modules/events/controllers/base.py", line 90, in _do_process
    return RHEventBase._do_process(self)
  File "/var/local/indico/.venv/lib/python3.9/site-packages/indico/web/rh.py", line 236, in _do_process
    self._check_access()
  File "/var/local/indico/.venv/lib/python3.9/site-packages/indico/modules/events/controllers/base.py", line 62, in _check_access
    RHProtectedEventBase._check_access(self)
  File "/var/local/indico/.venv/lib/python3.9/site-packages/indico/modules/events/controllers/base.py", line 37, in _check_access
    if not self.event.can_access(session.user):
  File "/var/local/indico/.venv/lib/python3.9/site-packages/indico/util/caching.py", line 54, in memoizer
    cache[key] = f(*args, **kwargs)
  File "/var/local/indico/.venv/lib/python3.9/site-packages/indico/core/db/sqlalchemy/protection.py", line 213, in can_access
    if self.inheriting_have_acl and any(user in entry.principal for entry in iter_acl(self.acl_entries)):
  File "/var/local/indico/.venv/lib/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 482, in __get__
    return self.impl.get(state, dict_)
  File "/var/local/indico/.venv/lib/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 942, in get
    value = self._fire_loader_callables(state, key, passive)
  File "/var/local/indico/.venv/lib/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 978, in _fire_loader_callables
    return self.callable_(state, passive)
  File "/var/local/indico/.venv/lib/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 912, in _load_for_state
    return self._emit_lazyload(
  File "/var/local/indico/.venv/lib/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 1046, in _emit_lazyload
    result = session.execute(
  File "/var/local/indico/.venv/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1714, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/var/local/indico/.venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/var/local/indico/.venv/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/var/local/indico/.venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
    ret = self._execute_context(
  File "/var/local/indico/.venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
    self._handle_dbapi_exception(
  File "/var/local/indico/.venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception
    util.raise_(
  File "/var/local/indico/.venv/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/var/local/indico/.venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
    self.dialect.do_execute(
  File "/var/local/indico/.venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "indico.affiliations" does not exist
LINE 2: ...1.user_id AND emails_1.is_primary LEFT OUTER JOIN indico.aff...
                                                             ^

[SQL: SELECT events.principals.mp_group_provider AS events_principals_mp_group_provider, events.principals.mp_group_name AS events_principals_mp_group_name, events.principals.read_access AS events_principals_read_access, events.principals.full_access AS events_principals_full_access, events.principals.permissions AS events_principals_permissions, events.principals.id AS events_principals_id, events.principals.event_id AS events_principals_event_id, events.principals.type AS events_principals_type, events.principals.user_id AS events_principals_user_id, events.principals.local_group_id AS events_principals_local_group_id, events.principals.email AS events_principals_email, events.principals.ip_network_group_id AS events_principals_ip_network_group_id, events.principals.event_role_id AS events_principals_event_role_id, events.principals.category_role_id AS events_principals_category_role_id, events.principals.registration_form_id AS events_principals_registration_form_id, users_1.title AS users_1_title, emails_1.id AS emails_1_id, emails_1.user_id AS emails_1_user_id, emails_1.email AS emails_1_email, emails_1.is_primary AS emails_1_is_primary, emails_1.is_user_deleted AS emails_1_is_user_deleted, affiliations_1.id AS affiliations_1_id, affiliations_1.name AS affiliations_1_name, affiliations_1.is_deleted AS affiliations_1_is_deleted, affiliations_1.street AS affiliations_1_street, affiliations_1.postcode AS affiliations_1_postcode, affiliations_1.city AS affiliations_1_city, affiliations_1.country_code AS affiliations_1_country_code, affiliations_1.meta AS affiliations_1_meta, users_1.id AS users_1_id, users_1.first_name AS users_1_first_name, users_1.last_name AS users_1_last_name, users_1.affiliation AS users_1_affiliation, users_1.affiliation_id AS users_1_affiliation_id, users_1.phone AS users_1_phone, users_1.address AS users_1_address, users_1.merged_into_id AS users_1_merged_into_id, users_1.is_system AS users_1_is_system, users_1.is_admin AS users_1_is_admin, users_1.is_blocked AS users_1_is_blocked, users_1.is_pending AS users_1_is_pending, users_1.is_deleted AS users_1_is_deleted, users_1.signing_secret AS users_1_signing_secret, users_1.picture_metadata AS users_1_picture_metadata, users_1.picture_source AS users_1_picture_source, groups_1.id AS groups_1_id, groups_1.name AS groups_1_name, ip_networks_1.group_id AS ip_networks_1_group_id, ip_networks_1.network AS ip_networks_1_network, ip_network_groups_1.id AS ip_network_groups_1_id, ip_network_groups_1.name AS ip_network_groups_1_name, ip_network_groups_1.description AS ip_network_groups_1_description, ip_network_groups_1.hidden AS ip_network_groups_1_hidden, ip_network_groups_1.attachment_access_override AS ip_network_groups_1_attachment_access_override, roles_1.id AS roles_1_id, roles_1.event_id AS roles_1_event_id, roles_1.name AS roles_1_name, roles_1.code AS roles_1_code, roles_1.color AS roles_1_color, roles_2.id AS roles_2_id, roles_2.category_id AS roles_2_category_id, roles_2.name AS roles_2_name, roles_2.code AS roles_2_code, roles_2.color AS roles_2_color, forms_1.id AS forms_1_id, forms_1.event_id AS forms_1_event_id, forms_1.title AS forms_1_title, forms_1.is_participation AS forms_1_is_participation, forms_1.introduction AS forms_1_introduction, forms_1.contact_info AS forms_1_contact_info, forms_1.start_dt AS forms_1_start_dt, forms_1.end_dt AS forms_1_end_dt, forms_1.modification_mode AS forms_1_modification_mode, forms_1.modification_end_dt AS forms_1_modification_end_dt, forms_1.is_deleted AS forms_1_is_deleted, forms_1.require_login AS forms_1_require_login, forms_1.require_user AS forms_1_require_user, forms_1.require_captcha AS forms_1_require_captcha, forms_1.registration_limit AS forms_1_registration_limit, forms_1.publish_registrations_public AS forms_1_publish_registrations_public, forms_1.publish_registrations_participants AS forms_1_publish_registrations_participants, forms_1.publish_registrations_duration AS forms_1_publish_registrations_duration, forms_1.publish_registration_count AS forms_1_publish_registration_count, forms_1.publish_checkin_enabled AS forms_1_publish_checkin_enabled, forms_1.moderation_enabled AS forms_1_moderation_enabled, forms_1.base_price AS forms_1_base_price, forms_1.currency AS forms_1_currency, forms_1.notification_sender_address AS forms_1_notification_sender_address, forms_1.message_pending AS forms_1_message_pending, forms_1.message_unpaid AS forms_1_message_unpaid, forms_1.message_complete AS forms_1_message_complete, forms_1.attach_ical AS forms_1_attach_ical, forms_1.manager_notifications_enabled AS forms_1_manager_notifications_enabled, forms_1.manager_notification_recipients AS forms_1_manager_notification_recipients, forms_1.tickets_enabled AS forms_1_tickets_enabled, forms_1.ticket_on_email AS forms_1_ticket_on_email, forms_1.ticket_on_event_page AS forms_1_ticket_on_event_page, forms_1.ticket_on_summary_page AS forms_1_ticket_on_summary_page, forms_1.ticket_template_id AS forms_1_ticket_template_id, forms_1.retention_period AS forms_1_retention_period, forms_1.is_purged AS forms_1_is_purged 
FROM events.principals LEFT OUTER JOIN users.users AS users_1 ON users_1.id = events.principals.user_id LEFT OUTER JOIN users.emails AS emails_1 ON users_1.id = emails_1.user_id AND emails_1.is_primary LEFT OUTER JOIN indico.affiliations AS affiliations_1 ON affiliations_1.id = users_1.affiliation_id LEFT OUTER JOIN users.groups AS groups_1 ON groups_1.id = events.principals.local_group_id LEFT OUTER JOIN indico.ip_network_groups AS ip_network_groups_1 ON ip_network_groups_1.id = events.principals.ip_network_group_id LEFT OUTER JOIN indico.ip_networks AS ip_networks_1 ON ip_network_groups_1.id = ip_networks_1.group_id LEFT OUTER JOIN events.roles AS roles_1 ON roles_1.id = events.principals.event_role_id LEFT OUTER JOIN categories.roles AS roles_2 ON roles_2.id = events.principals.category_role_id LEFT OUTER JOIN event_registration.forms AS forms_1 ON forms_1.id = events.principals.registration_form_id 
WHERE %(param_1)s = events.principals.event_id]
[parameters: {'param_1': 46}]
(Background on this error at: https://sqlalche.me/e/14/f405)

Try running indico db upgrade - that should fix it.

I did, although I did see find someone since posting that had a different DB problem during upgrade that was fixed by running db upgrade a second time. I’ll give that a shot…

And no luck, multilpe ‘indico db ugrades’ doesn’t change the result.

What does indico db current -v give you?

I’m setting up a non-production test version at this point, This the current, v3.1.1 output. I’ll update with the v3.2.1 output I get when the test system is up.

(indico) indico@srv:~$ indico db current -v
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
Current revision(s) for postgresql:///indico:
Rev: b36825c7869e (head)
Parent: dc53d6e8c576
Path: /var/local/indico/.venv/lib/python3.9/site-packages/indico/migrations/versions/20210927_1027_b36825c7869e_add_registration_tags.py

Add registration tags

Revision ID: b36825c7869e
Revises: dc53d6e8c576
Create Date: 2021-09-27 10:27:39.335488

Hmm so the test 3.2.1 fails the same way as the operational, and ‘indico db current -v’ on the test v3.2.1 looks to be the same as the operation v3.1. The test version 3.1 ‘indico db current -v’ was identical as well.


test version 3.1
indico@d10:~$ indico db current -v
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
Current revision(s) for postgresql:///indico:
Rev: b36825c7869e (head)
Parent: dc53d6e8c576
Path: /var/local/indico/.venv/lib/python3.9/site-packages/indico/migrations/versions/20210927_1027_b36825c7869e_add_registration_tags.py

Add registration tags

Revision ID: b36825c7869e
Revises: dc53d6e8c576
Create Date: 2021-09-27 10:27:39.335488

Trying different version changes, Ver.<3.1 and there is a (different) database error (not surprising),. 3.1=<Ver.<3.2 the website works. Ver.>=3.2 and I get the error listed above with indico.affilitions missing. This happens with our without a ‘db upgrade’ between 3.1.1 and 3.2. The output from ‘indico db upgrade --sql’ between 3.1.1 and 3.2 does not have any mention of affiliation in it (singular or pural). It looks like there is code in 3.2 trying to access something that doesn’t exist (for this instance anyway).

Going to try digging into that tomorrow.

You need to check why the migration stops when applying 20211116_1457_8993132179d3_ensure_single_choice_fields_have_item_type.py and later changes.

Make sure the database owner is correctly set (see [1] ) and check the postgres logs for errros.

[1] PostgreSQL Permnission denied "indico db prepare"

I think any db errors would result in a proper exception. The only case I know where the migration silently doesn’t do anything is when someone added a custom revision on top, so the current revision no longer exists and alembic doesn’t know what to apply next…

Can you check the contents of /var/local/indico/.venv/lib/python3.9/site-packages/indico/migrations/versions/ if there are newer revisions than your “current” one? 3.2 added some, so if something went wrong with the installation that could also explain why they aren’t there…

First, thanks everyone for your help! This forum is more responsive than I expeceted and it definitely helped.

Looking at .venv/lib/python3.9/site-packages/indico/migrations/versions/, showed me a lot of revisions that were not getting updated. I tried a number of things, and strangely I found that I could downgrade the DB, but not upgrade it.

I found a similar issue as a general Alembic problem: the database revision ID getting out of sync. In this case, in the indico.alembic_version table which had:

select * from alembic_version ;
version_num

e787389ca868
dc53d6e8c576

e787389ca868 is from 20210129_2232_e787389ca868_add_rejection_reason.py

‘indico db downgrade’ to that version and then upgrading appears to have worked. It leaves that table with only one row with the most current version_num. And the site, running 3.2.1, loads. I have only partially checked so far, but the site seems to work properly and I haven’t found any missing data yet (crossed fingers).

can you paste the list of downgrades it performed? just to see if you lost anything important.

I’m not sure if the downgrade actually did anything, or rather, the DB may not have ever upgraded beyond the version that I downgraded too, the first row version in the alembic table.

I downgraded to 20210129_2232_e787389ca868_add_rejection_reason.py, In theory, that would have included downgrading:

20210211_1613_26985db8ed12_add_attach_ical_to_reminders.py
20210215_1052_f26c201c8254_add_attach_ical_to_registrationform.py
20210219_1428_3782de7970da_rename_oauth_default_scopes.py
20210219_1555_da06d8f50342_separate_authorized_scopes_from_tokens.py
20210222_1754_c36abe1c23c7_make_oauth_pkce_flow_configurable.py
20210222_1914_d354278c6d95_store_tokens_as_hashes.py
20210224_1805_ecc7088914e7_use_cascading_fks_for_oauth.py
20210224_1808_26806768cd3f_remove_flower_oauth_app.py
20210507_1812_d89585afaf2e_add_index_on_user_merged_into_id.py
20210524_1447_cf9e1b4e2f5f_restrict_event_creation_cat_subcats.py
20210527_1314_178d297eae7e_disallow_seconds_in_durations.py
20210607_1548_735dc4e8d2f3_update_merged_users_in_registrations.py
20210615_1612_fda76e047e87_make_sure_contributions_have_titles.py
20210615_1627_79e770865675_add_more_fts_indexes.py
20210622_1655_90384b9b3d22_make_room_division_non_nullable.py
20210630_1104_356b8985ae7c_add_survey_anonymous_submissions_table.py
20210630_1205_1f6738730753_add_personal_access_tokens.py
20210630_1206_1b7e98f581bc_make_token_scopes_not_nullable.py
20210630_1349_5cbb0eb12eb3_add_last_used_ip_and_use_count_to_tokens.py
20210705_1337_420195768776_add_reminder_send_to_speakers_column.py
20210705_1726_cd3fef2095b4_add_is_flat_view_enabled_setting_for_categories.py
20210707_1635_1cec32e42f65_add_reviewed_dt_to_paper_revisions.py
20210802_1514_4b097412a8d9_add_category_logs.py
20210804_1211_9d00917b2fa8_add_event_category_request.py
20210813_1633_dc53d6e8c576_make_event_category_nullable.py
20210927_1027_b36825c7869e_add_registration_tags.py

In practice, I’m not sure if any of the downgrades actually caused a changed in the DB. Then the upgrade was to the current 3.2.1 which would have been adding the above changes and the following:

20211116_1457_8993132179d3_ensure_single_choice_fields_have_item_type.py
20211119_1539_84874a17eaf3_cleanup_registration_fields_is_billable.py
20211121_1403_3dafee32ba7d_remove_is_billable_from_registration_fields.py
20211207_1106_82fb6c6ac6db_add_publish_registrations_with_consent.py
20220225_1335_ef7a8b2e6737_add_access_column_to_menu_entries.py
20220315_1520_5123f24eb41e_add_form_field_retention_period.py
20220318_1614_57696d76f9b0_add_registration_visibility_override.py
20220329_1207_a61ce4bd7549_add_publish_registrations_duration_to_regforms.py
20220406_1431_88eb87ee0d3e_add_registration_retention_period.py
20220407_1306_a707753d16e2_add_registration_is_purged.py
20220421_1316_812aa90a3660_add_favorite_events_table.py
20220513_1333_c39db219f85a_add_predefined_affiliations.py
20220513_1337_1950e5d12ab5_add_predefined_affiliations_to_persons.py
20220531_1439_0c4bb2973536_move_affiliation_into_users_table.py
20220718_1534_33c3ab67d729_add_registration_created_by_manager_.py
20220726_1007_b60f5c45acf7_add_option_to_enable_captcha.py
20220815_1631_b45847c0e62f_add_event_series_title_pattern.py

There was more output during the upgrade (I didn’t save it, unfortunately) than the downgrade. Since the affilitions problem went away, I’d guess the upgrades went okay. At a miniminum, 1+ of the affiliations changes did.