DB migration from 1.2 to 2 fails with empty database

speedytom · October 22, 2020, 8:15am

(.venv) indico@indico2:~$ indico-migrate postgresql:///indico file:///opt/indico-legacy/db/Data.fs --archive-dir /opt/indico-legacy/archive --storage-backend legacy --default-email default@indico.tok.ipp.cas.cz --default-currency EUR --symlink-target ~/archive/legacy_sym
links/ --symlink-backend legacy-symlinks --migrate-broken-events
Trying to open file:///opt/indico-legacy/db/Data.fs...
Done!

I see almost empty database:

postgres=# \c indico
You are now connected to database "indico" as user "postgres".
indico=# \dt
             List of relations
 Schema |      Name       | Type  | Owner  
--------+-----------------+-------+--------
 public | alembic_version | table | indico
(1 row)

 indico=# select * from alembic_version;
 version_num 
-------------
(0 rows)

speedytom · October 22, 2020, 8:18am

I am not able upload migration.log.

speedytom · October 22, 2020, 8:20am

× [global_pre] No valid IPs found
! [users] Skipping 1 - not activated
! [users] Skipping 120 - not activated
! [users] Skipping 146 - not activated
! [users] Skipping 158 - not activated
! [users] Skipping 168 - not activated
! [users] Skipping 169 - not activated
! [users] Skipping 195 - not activated
! [users] Skipping 2 - not activated
! [users] Skipping 24 - not activated
! [users] Skipping 252 - not activated
! [users] Skipping 258 - not activated
! [users] Skipping 271 - not activated
! [users] Skipping 272 - not activated
! [users] Skipping 275 - not activated
! [users] Skipping 276 - not activated
! [users] Skipping 277 - not activated
! [users] Skipping 278 - not activated
! [users] Skipping 279 - not activated
! [users] Skipping 281 - not activated
! [users] Skipping 282 - not activated
! [users] Skipping 283 - not activated
! [users] Skipping 284 - not activated
! [users] Skipping 285 - not activated
! [users] Skipping 286 - not activated
! [users] Skipping 287 - not activated
! [users] Skipping 288 - not activated
! [users] Skipping 289 - not activated
! [users] Skipping 290 - not activated
! [users] Skipping 292 - not activated
! [users] Skipping 302 - not activated
! [users] Skipping 336 - not activated
! [users] Skipping 344 - not activated
! [users] Skipping 350 - not activated
! [users] Skipping 362 - not activated
! [users] Skipping 369 - not activated
! [users] Skipping 370 - not activated
! [users] Skipping 371 - not activated
! [users] Skipping 372 - not activated
! [users] Skipping 373 - not activated
! [users] Skipping 375 - not activated
! [users] Skipping 377 - not activated
! [users] Skipping 378 - not activated
! [users] Skipping 379 - not activated
! [users] Skipping 380 - not activated
! [users] Skipping 381 - not activated
! [users] Skipping 384 - not activated
! [users] Skipping 385 - not activated
! [users] Skipping 391 - not activated
! [users] Skipping 393 - not activated
! [users] Skipping 394 - not activated
! [users] Skipping 395 - not activated
! [users] Skipping 396 - not activated
! [users] Skipping 397 - not activated
! [users] Skipping 398 - not activated
! [users] Skipping 399 - not activated
! [users] Skipping 401 - not activated
! [users] Skipping 402 - not activated
! [users] Skipping 403 - not activated
! [users] Skipping 404 - not activated
! [users] Skipping 405 - not activated
! [users] Skipping 406 - not activated
! [users] Skipping 408 - not activated
! [users] Skipping 409 - not activated
! [users] Skipping 410 - not activated
! [users] Skipping 411 - not activated
! [users] Skipping 412 - not activated
! [users] Skipping 413 - not activated
! [users] Skipping 414 - not activated
! [users] Skipping 415 - not activated
! [users] Skipping 417 - not activated
! [users] Skipping 418 - not activated
! [users] Skipping 419 - not activated
! [users] Skipping 42 - not activated
! [users] Skipping 420 - not activated
! [users] Skipping 422 - not activated
! [users] Skipping 424 - not activated
! [users] Skipping 426 - not activated
! [users] Skipping 427 - not activated
! [users] Skipping 43 - not activated
! [users] Skipping 434 - not activated
! [users] Skipping 77 - not activated
✓ [users] Added new system user: <User(435, None): "Indico System">
×      0  abstract    Legacy contribution type not found: --any--
×     10  abstract    Legacy contribution type not found: --any--
!     18  regform     Email matches another user; associating <Registration(None, registration_form_id=None, XXX@phystech.edu, None, is_deleted=None): "George XXX"> with <User(303, XXX@phystech.edu, is_deleted=True): "George XXX"> instead of <pe
rsistent broken MaKaC.user.Avatar instance '\x00\x00\x00\x00\x00\x18\x99\xcf'>
!      5  regform     Price mismatch: 0 (calculated) != 13000.0 (saved). Setting adjustment of 13000.0
!      5  regform     Price mismatch: 14300.0 (calculated) != 13000.0 (saved). Setting adjustment of -1300.0
×      5  abstract    Legacy contribution type not found: <persistent broken MaKaC.conference.ContributionType instance '\x00\x00\x00\x00\x00\x13q\x7f'>
×      5  abstract    Legacy contribution type not found: <persistent broken MaKaC.conference.ContributionType instance '\x00\x00\x00\x00\x00\x13q~'>
×      5  abstract    Legacy contribution type not found: <persistent broken MaKaC.conference.ContributionType instance '\x00\x00\x00\x00\x00\x13q\x80'>
!      5  abstract    Abstract 88: Author Diagnostics Team already exists (Diagnostics Team [_S_] / Michael XXX [P_s])
!      5  timetable   Primary author "Diagnostics Team" is also co-author

ThiefMaster · October 22, 2020, 8:23am

the data is spread over multiple schemas, which psql doesn’t list by default. if the migration didn’t fail, most likely everything went fine.

speedytom · October 22, 2020, 8:25am

OK. How can I check it???

ThiefMaster · October 22, 2020, 8:25am

select count(*) from events.events; for example

speedytom · October 22, 2020, 8:27am

indico=# select count (*) from events.events;
 count 
-------
    19
(1 row)

OK. It seems reasonable.

speedytom · October 22, 2020, 8:28am

I have another question: How can I delete bot accounts from our system?

ThiefMaster · October 22, 2020, 8:33am

Do you mean accounts created by spam bots? Most likely they were not migrated at all if they had never been activated.

Anyway, once you fully upgraded, the easiest way to check is indico user search -e @ --substring which will list all users. If there are any spammy users, you can try deleting them via indico shell using db.session.delete(User.get(ID)) followed by db.session.commit(). In case deletion fails because they are linked to things that cannot easily be delinked, you can also soft-delete them: User.get(ID).is_deleted = True.

speedytom · October 22, 2020, 8:37am

Thanks for your answer. Please, don’t close this issue. Maybe I will have another question about migration.

speedytom · October 22, 2020, 9:04am

I have question about TEXlive… Should I install manually to /opt/texlive ???

Is it possible install package from UBU 18.04?

ThiefMaster · October 22, 2020, 9:06am

You can try it. Maybe it includes all the packages needed…

speedytom · October 22, 2020, 10:02am

I get this warning:

(.venv) indico@indico2:~$ indico db upgrade
/opt/indico/.venv/local/lib/python2.7/site-packages/indico/core/config.py:176: UserWarning: Ignoring unknown config key ASSETS_DIR
  warnings.warn('Ignoring unknown config key {}'.format(key))
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> f1eee7b4880a, Add event roles table
INFO  [alembic.runtime.migration] Running upgrade f1eee7b4880a -> 640584a3987e, Add event roles to acls
INFO  [alembic.runtime.migration] Running upgrade 640584a3987e -> 790a06790309, Rename roles to permissions
INFO  [alembic.runtime.migration] Running upgrade 790a06790309 -> 1d512a9ebb30, Add default session to tracks
INFO  [alembic.runtime.migration] Running upgrade 1d512a9ebb30 -> 566d5de4e0e5, Add is_private column to contribution types table
INFO  [alembic.runtime.migration] Running upgrade 566d5de4e0e5 -> 2af245be72a6, Add columns to review questions and review ratings tables.
INFO  [alembic.runtime.migration] Running upgrade 2af245be72a6 -> 9c4418d7a6aa, Add session types table
INFO  [alembic.runtime.migration] Running upgrade 9c4418d7a6aa -> 093533d27a96, Add visibility options to contribution fields
INFO  [alembic.runtime.migration] Running upgrade 093533d27a96 -> c820455976ba, Add include_description to reminders
INFO  [alembic.runtime.migration] Running upgrade c820455976ba -> 813ea74ce8dc, Add AttachmentFolder.is_hidden
INFO  [alembic.runtime.migration] Running upgrade 813ea74ce8dc -> 66ecbb1c0ddd, Add natsort function
INFO  [alembic.runtime.migration] Running upgrade 66ecbb1c0ddd -> a13e25814c4c, Add favorite rooms table
INFO  [alembic.runtime.migration] Running upgrade a13e25814c4c -> 93985a8c11ed, Remove room thumbnail
INFO  [alembic.runtime.migration] Running upgrade 93985a8c11ed -> 55893713f6b7, Add room_name_format
INFO  [alembic.runtime.migration] Running upgrade 55893713f6b7 -> f7ab9ee32bdf, Add verbose_name column
INFO  [alembic.runtime.migration] Running upgrade f7ab9ee32bdf -> c0888cb57c58, Remove holidays table
INFO  [alembic.runtime.migration] Running upgrade c0888cb57c58 -> 7a72d63acba9, Update map aspects structure
INFO  [alembic.runtime.migration] Running upgrade 7a72d63acba9 -> c5c21246445a, Rename aspects to map areas
INFO  [alembic.runtime.migration] Running upgrade c5c21246445a -> 0117bd0fa784, Store room coordinates as numbers
INFO  [alembic.runtime.migration] Running upgrade 0117bd0fa784 -> 27c45c384d65, Make equipment types global
INFO  [alembic.runtime.migration] Running upgrade 27c45c384d65 -> ec410be271df, Make room attributes global
INFO  [alembic.runtime.migration] Running upgrade ec410be271df -> 7749141f3e08, Remove room attribute nesting
INFO  [alembic.runtime.migration] Running upgrade 7749141f3e08 -> db32adb8fc4e, Remove equipment nesting
INFO  [alembic.runtime.migration] Running upgrade db32adb8fc4e -> 868c04697dd7, Remove reservation-equipment association
INFO  [alembic.runtime.migration] Running upgrade 868c04697dd7 -> 7e03b2262e9e, Add room features
INFO  [alembic.runtime.migration] Running upgrade 7e03b2262e9e -> 4f98f2f979c7, Remove RoomAttribute.is_required
INFO  [alembic.runtime.migration] Running upgrade 4f98f2f979c7 -> 252c0015c9a0, Add missing blocking principal constraints
INFO  [alembic.runtime.migration] Running upgrade 252c0015c9a0 -> cbe630695800, Add room principals table
INFO  [alembic.runtime.migration] Running upgrade cbe630695800 -> 579a36843848, Use enum for reservation state
INFO  [alembic.runtime.migration] Running upgrade 579a36843848 -> 7ec3949a21c7, Use enum for resv occurrence state
INFO  [alembic.runtime.migration] Running upgrade 7ec3949a21c7 -> ff49d8d05ce7, Disallow empty rejection reasons
INFO  [alembic.runtime.migration] Running upgrade ff49d8d05ce7 -> 416f9c877300, Add reservation link table
INFO  [alembic.runtime.migration] Running upgrade 416f9c877300 -> 7aabedfb5e3a, Make some room cols not nullable
INFO  [alembic.runtime.migration] Running upgrade 7aabedfb5e3a -> 081d4c97060a, Add end notifications
INFO  [alembic.runtime.migration] Running upgrade 081d4c97060a -> 3e4a0c08eae6, Fix truncated constraint names (sqlalchemy 1.3)
INFO  [alembic.runtime.migration] Running upgrade 3e4a0c08eae6 -> fe73a07da0b4, Add Room.notification_emails column
INFO  [alembic.runtime.migration] Running upgrade fe73a07da0b4 -> 8521bce91242, Remove location is_default
INFO  [alembic.runtime.migration] Running upgrade 8521bce91242 -> ad0625914645, Room.is_active -> is_deleted
INFO  [alembic.runtime.migration] Running upgrade ad0625914645 -> a83e77e11e36, Add Location.is_deleted
INFO  [alembic.runtime.migration] Running upgrade a83e77e11e36 -> 7024f7f66e20, Remove assistance/vc columns from rb
INFO  [alembic.runtime.migration] Running upgrade 7024f7f66e20 -> 06a4ec717b84, Switch from JSON to JSONB
INFO  [alembic.runtime.migration] Running upgrade 06a4ec717b84 -> 620b312814f3, Migrate indico-weeks-view theme name
INFO  [alembic.runtime.migration] Running upgrade 620b312814f3 -> eefba82b42c5, Add track_groups table
INFO  [alembic.runtime.migration] Running upgrade eefba82b42c5 -> 1b741c9123f6, Add columns for program codes
INFO  [alembic.runtime.migration] Running upgrade 1b741c9123f6 -> 0f20a5e4018e, Add new abstract state
INFO  [alembic.runtime.migration] Running upgrade 0f20a5e4018e -> f6fba869a27c, Add uuid column to event_abstracts.abstracts
INFO  [alembic.runtime.migration] Running upgrade f6fba869a27c -> a2472148d2c5, Add signature token column to User
INFO  [alembic.runtime.migration] Running upgrade a2472148d2c5 -> bb522e9f9066, Add files table
INFO  [alembic.runtime.migration] Running upgrade bb522e9f9066 -> 39a25a873063, Add editing file_types table
INFO  [alembic.runtime.migration] Running upgrade 39a25a873063 -> 4e459d27adab, Add tables for editing revisions
INFO  [alembic.runtime.migration] Running upgrade 4e459d27adab -> 2496c4adc7e9, Add track principal
INFO  [alembic.runtime.migration] Running upgrade 2496c4adc7e9 -> e01c48de5a5e, Allow more digits in prices
INFO  [alembic.runtime.migration] Running upgrade e01c48de5a5e -> 6a185fdcd4ee, Add filename_template column
INFO  [alembic.runtime.migration] Running upgrade 6a185fdcd4ee -> 4d263fa78830, Add category roles table
INFO  [alembic.runtime.migration] Running upgrade 4d263fa78830 -> b3ce69ab24d9, Add category roles to acls
INFO  [alembic.runtime.migration] Running upgrade b3ce69ab24d9 -> 18a1088f1ea8, Use ondelete=cascade for comments
INFO  [alembic.runtime.migration] Running upgrade 18a1088f1ea8 -> 266d78b1c5db, Add table for event labels
INFO  [alembic.runtime.migration] Running upgrade 266d78b1c5db -> a3295d628e3b, Migrate event labels from settings
INFO  [alembic.runtime.migration] Running upgrade a3295d628e3b -> 56a26a721717, Add slides as EditableType
INFO  [alembic.runtime.migration] Running upgrade 56a26a721717 -> 6444c893a21f, Associate file types with editable types
INFO  [alembic.runtime.migration] Running upgrade 6444c893a21f -> 3c5462aef0b7, Associate review-conditions with editable types
INFO  [alembic.runtime.migration] Running upgrade 3c5462aef0b7 -> 02bf20df06b3, Add table for review conditions
INFO  [alembic.runtime.migration] Running upgrade 02bf20df06b3 -> 933665578547, Migrate review conditions from settings
INFO  [alembic.runtime.migration] Running upgrade 933665578547 -> 7f56252c73ab, Add map_url to events table
INFO  [alembic.runtime.migration] Running upgrade 7f56252c73ab -> 532f0ea25bb1, Add user profile picture
INFO  [alembic.runtime.migration] Running upgrade 532f0ea25bb1 -> c0fc1e46888b, Disallow editing permissions for groups
INFO  [alembic.runtime.migration] Running upgrade c0fc1e46888b -> b6dd0a4ed40d, Add registration form to event acls
INFO  [alembic.runtime.migration] Running upgrade b6dd0a4ed40d -> 05f227f4b938, Add metadata to event logs
INFO  [alembic.runtime.migration] Running upgrade 05f227f4b938 -> 4f0b4dd412b5, Remove email principals from tracks
INFO  [alembic.runtime.migration] Running upgrade 4f0b4dd412b5 -> d89abffb0f63, Add menu_entries.registered_only column.
INFO  [alembic.runtime.migration] Running upgrade d89abffb0f63 -> 12fbf6af2367, Add custom boa file to event
INFO  [alembic.runtime.migration] Running upgrade 12fbf6af2367 -> 497c61b68050, Fix session friendly id index
INFO  [alembic.runtime.migration] Running upgrade 497c61b68050 -> c997dc927fbc, Add default badge to categories
INFO  [alembic.runtime.migration] Running upgrade c997dc927fbc -> f37d509e221c, Add column for profile picture type to User

Should I be scared?

ThiefMaster · October 22, 2020, 10:04am

You can remove this setting from indico.conf; it was removed in 2.2 and is no longer needed

speedytom · October 22, 2020, 10:14am

I have already done it. Thanks for your answer.

speedytom · October 22, 2020, 12:25pm

I need LDAP ahentication like this:

AuthenticatorList    = [('Local', {}),
('LDAP', {
        'uri': 'ldap://url',
        'useTLS': True,
        'peopleDNQuery': ('uid={0}', 'ou=*,dc=compass'),
        'groupDNQuery': ('cn={0}', 'ou=*,dc=compass'),
        'groupStyle': 'SLAPD',
        'accessCredentials': ('', ''),  # anonymous search
        }
)
]

Which python package for it? It will be python-ldap?

ThiefMaster · October 22, 2020, 12:29pm

Yes, see https://docs.getindico.io/en/stable/config/auth/#ldap

PS: When posting code or code-like snippets (e.g. logs / shell output), please wrap them in triple-backtick fences so they are properly formatted in monospace and not as normal text:

```
stuff
here
```