"Circular dependency detected"

Administrators Migration from 1.2 to 2.0
1

Hi,

we are migrating our central INFN Indico from 0.98 to 2.0.3.

Migration from 0.98 to 1.2 seems completed successfully. Now we are trying migrate to 2.0.3 by migration tool ‘indico-migrate’ on a temporary VM with CentOS 7.4.1708. While migrating events we got this error:

(.venv)[indico@dyn151 ~]$ indico-migrate postgresql:///indico file:///opt/indico-db/Data.fs --archive-dir /opt/indico-legacy/archive --storage-backend legacy --default-email xxxx@xxxxx.infn.it --default-currency EUR --symlink-target ~/archive/legacy_symlinks/ --symlink-backend legacy-symlinks --migrate-broken-events --save-restore /tmp/migration.save
Trying to open file:///opt/indico-db/Data.fs...
Done!
Traceback (most recent call last): |/opt/indico/.venv/bin/indico-migrate:11 [<module>]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/cli.py:126 [main]
 |/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:722 [__call__]
 |/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:697 [main]
 |/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:895 [invoke]
 |/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:535 [invoke]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/cli.py:122 [cli]
 debug=debug, **kwargs)
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/migrate.py:106 [migrate]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/importer.py:161 [run]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/util.py:323 [_f]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:183 [migrate]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:211 [migrate_event_data]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:137 [run_step]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/__init__.py:83 [run]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/abstracts.py:116 [migrate]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/scoping.py:157 [do]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py:2192 [flush]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py:2312 [_flush]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py:66 [__exit__]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py:2276 [_flush]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py:381 [execute]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/util/topological.py:36 [sort_as_subsets]
 _gen_edges(edges)

Circular dependency detected. (ProcessState(OneToManyDP(Abstract.duplicate_abstracts), <Abstract at 0xca5f6210>, delete=False), ProcessState(OneToManyDP(Abstract.merged_abstracts), <Abstract at 0xc970a7d0>, delete=False), SaveUpdateState(<Abstract at 0xca5f6210>), SaveUpdateState(<Abstract at 0xc970a7d0>), ProcessState(ManyToOneDP(Abstract.duplicate_of), <Abstract at 0xc970a7d0>, delete=False), ProcessState(ManyToOneDP(Abstract.merged_into), <Abstract at 0xca5f6210>, delete=False))

Any suggestions?

PS: I also have the migration.log but I prefer not attaching it here publicly, because of private data.

Thank you very much!

Best regards,
Dael Maselli.

2

So it looks like an error from the SQLAlchemy library. This error message for sure is not very helpful. Do you know for which object it actually fails?

3

This is one of the harder-to-debug SQLAlchemy errors, especially without having access to some actual data causing this failure…

One thing you could try to see if it helps is adding a db.session.flush() call at the end of this loop (try both inside and outside the loop’s body):

4

Here are the last lines from migration.log before crash:

!  13348  regform     Skipping invalid data (field id mismatch) for obsolete version of "Email"
(registrant 0)
!  13348  regform     Skipping invalid data (field id mismatch) for obsolete version of "Institu
tion" (registrant 0)
!  13348  regform     Skipping invalid data (field id mismatch) for obsolete version of "Country
 of Personal Address" (registrant 0)
!  13348  regform     Skipping invalid data (field id mismatch) for obsolete version of "City of
 Personal Address" (registrant 0)
!  13348  regform     Skipping invalid data (field id mismatch) for obsolete version of "Email"
(registrant 1)
!  13348  regform     Skipping invalid data (field id mismatch) for obsolete version of "Institu
tion" (registrant 1)
!  13348  regform     Skipping invalid data (field id mismatch) for obsolete version of "Country
 of Personal Address" (registrant 1)
!  13348  regform     Skipping invalid data (field id mismatch) for obsolete version of "City of
 Personal Address" (registrant 1)
!  13348  regform     Skipping invalid data (field id mismatch) for obsolete version of "Email"
(registrant 2)
!  13348  regform     Skipping invalid data (field id mismatch) for obsolete version of "Institu
tion" (registrant 2)
!  13348  regform     Skipping invalid data (field id mismatch) for obsolete version of "Country
 of Personal Address" (registrant 2)
!  13348  regform     Skipping invalid data (field id mismatch) for obsolete version of "City of
 Personal Address" (registrant 2)
!  13362  menus       Skipping link "Come raggiungere i LNF" with no URL
!  13362  menus       Skipping link "Accesso alla rete" with no URL
!   1344  timetable   Block boundary (end violated; extending block from 2009-05-11 11:00:00+00:00 to 2009-05-11 13:00:00+00:00)
!   1345  menus       Skipping top-level menu entry program_my_tracks
!  13450  regform     Email matches another user; associating <Registration(None, registration_form_id=None, lizheng_880606@163.com, None, is_deleted=None): "Zheng Li"> with <User(27262, lizheng_880606@163.com, is_deleted=True): "Lee Zheng"> instead of <persistent broken MaKaC.user.Avatar instance '\x00\x00\x00\x00\x0c\xe0j\x1c'>
×  13450  abstract    Legacy contribution type not found: <persistent broken MaKaC.conference.ContributionType instance '\x00\x00\x00\x00\x0b\xf9)\xaf'>
×  13450  abstract    Legacy contribution type not found: <persistent broken MaKaC.conference.ContributionType instance '\x00\x00\x00\x00\x0b\xf9)\xb0'>
!  13450  abstract    No contrib type data, using any [{'_tpl': <persistent broken MaKaC.review.NotificationTemplate instance '\x00\x00\x00\x00\x0c\xfd\xe5\x1b'>, '_id': 0, '__Broken_newargs__': ()}]

===========================================================

I tried with db.session.flush() inside the loop (at the end) and it crash a little before:

Traceback (most recent call last): |/opt/indico/.venv/bin/indico-migrate:11 [<module>]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/cli.py:126 [main]
 |/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:722 [__call__]
 |/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:697 [main]
 |/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:895 [invoke]
 |/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:535 [invoke]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/cli.py:122 [cli]
 debug=debug, **kwargs)
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/migrate.py:106 [migrate]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/importer.py:161 [run]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/util.py:323 [_f]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:183 [migrate]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:211 [migrate_event_data]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:137 [run_step]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/__init__.py:83 [run]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/abstracts.py:115 [migrate]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/abstracts.py:454 [_migrate_abstracts]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/scoping.py:157 [do]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py:2192 [flush]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py:2312 [_flush]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py:66 [__exit__]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py:2276 [_flush]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py:389 [execute]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py:548 [execute]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py:181 [save_obj]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py:835 [_emit_insert_statements]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py:945 [execute]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/sql/elements.py:263 [_execute_on_connection]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py:1053 [_execute_clauseelement]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py:1189 [_execute_context]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py:1402 [_handle_dbapi_exception]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py:203 [raise_from_cause]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py:1182 [_execute_context]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py:470 [do_execute]
 cursor.execute(statement, parameters)

(psycopg2.IntegrityError) new row for relation "abstracts" violates check constraint "ck_abstracts_duplicate_of_id_only_duplicate"
DETAIL:  Failing row contains (2349, 8, 12289, Highlights from heavy-flavour physics measurements with ATLAS de..., 3740, null, 2017-03-07 11:56:27.37528, null, null, 6, , 523, , 2017-03-24 08:58:53.746078, null, null, null, null, f, Abstract: The study of the heavy-flavours dynamics within the St...).
 [SQL: 'INSERT INTO event_abstracts.abstracts (friendly_id, event_id, title, submitter_id, submitted_contrib_type_id, submitted_dt, modified_by_id, modified_dt, state, submission_comment, judge_id, judgment_comment, judgment_dt, accepted_track_id, accepted_contrib_type_id, merged_into_id, duplicate_of_id, is_deleted, description) VALUES (%(friendly_id)s, %(event_id)s, %(title)s, %(submitter_id)s, %(submitted_contrib_type_id)s, %(submitted_dt)s, %(modified_by_id)s, %(modified_dt)s, %(state)s, %(submission_comment)s, %(judge_id)s, %(judgment_comment)s, %(judgment_dt)s, %(accepted_track_id)s, %(accepted_contrib_type_id)s, %(merged_into_id)s, %(duplicate_of_id)s, %(is_deleted)s, %(description)s) RETURNING event_abstracts.abstracts.id'] [parameters: {'modified_dt': None, 'is_deleted': False, 'description': u'Abstract: The study of the heavy-flavours dynamics within the Standard Model (SM) is one of the most intriguing areas in modern Particle Physics. Se ... (333 characters truncated) ... are decays processes involving B-mesons and to CP-violation measurements, which represent real windows towards any New Physics process beyond the SM.', 'title': u'Highlights from heavy-flavour physics measurements with ATLAS detector', 'event_id': 12289, 'accepted_track_id': None, 'merged_into_id': None, 'judgment_comment': u'', 'submitter_id': 3740, 'accepted_contrib_type_id': None, 'state': <AbstractState.duplicate: 6>, 'submission_comment': u'', 'submitted_contrib_type_id': None, 'submitted_dt': datetime.datetime(2017, 3, 7, 11, 56, 27, 375280), 'modified_by_id': None, 'judgment_dt': datetime.datetime(2017, 3, 24, 8, 58, 53, 746078), 'friendly_id': 8, 'duplicate_of_id': None, 'judge_id': 523}]

==========================================================

With the db.session.flush() outside the loop (at the end), the error is:

Traceback (most recent call last): |/opt/indico/.venv/bin/indico-migrate:11 [<module>]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/cli.py:126 [main]
 |/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:722 [__call__]
 |/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:697 [main]
 |/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:895 [invoke]
 |/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:535 [invoke]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/cli.py:122 [cli]
 debug=debug, **kwargs)
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/migrate.py:106 [migrate]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/importer.py:161 [run]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/util.py:323 [_f]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:183 [migrate]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:211 [migrate_event_data]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:137 [run_step]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/__init__.py:83 [run]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/abstracts.py:115 [migrate]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/abstracts.py:454 [_migrate_abstracts]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/scoping.py:157 [do]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py:2192 [flush]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py:2312 [_flush]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py:66 [__exit__]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py:2276 [_flush]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py:384 [execute]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py:481 [execute_aggregate]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py:548 [execute]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py:181 [save_obj]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py:835 [_emit_insert_statements]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py:945 [execute]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/sql/elements.py:263 [_execute_on_connection]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py:1053 [_execute_clauseelement]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py:1189 [_execute_context]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py:1402 [_handle_dbapi_exception]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py:203 [raise_from_cause]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py:1182 [_execute_context]
 |/opt/indico/.venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py:470 [do_execute]
 cursor.execute(statement, parameters)

(psycopg2.IntegrityError) new row for relation "abstract_reviews" violates check constraint "ck_abstract_reviews_prop_abstract_id_only_duplicate_merge"
DETAIL:  Failing row contains (354, 2370, 523, 149, 2017-03-24 08:58:53.747417, null, duplicate of #9, 4, null, null).
 [SQL: 'INSERT INTO event_abstracts.abstract_reviews (abstract_id, user_id, track_id, created_dt, modified_dt, comment, proposed_action, proposed_related_abstract_id, proposed_contribution_type_id) VALUES (%(abstract_id)s, %(user_id)s, %(track_id)s, %(created_dt)s, %(modified_dt)s, %(comment)s, %(proposed_action)s, %(proposed_related_abstract_id)s, %(proposed_contribution_type_id)s) RETURNING event_abstracts.abstract_reviews.id'] [parameters: {'comment': u'duplicate of #9', 'modified_dt': None, 'user_id': 523, 'proposed_contribution_type_id': None, 'abstract_id': 2370, 'track_id': 149, 'created_dt': datetime.datetime(2017, 3, 24, 8, 58, 53, 747417), 'proposed_action': <AbstractAction.mark_as_duplicate: 4>, 'proposed_related_abstract_id': None}]

Last lines of migration.log were:

×  12289  abstract    Legacy contribution type not found: --any--
!  12289  abstract    No contrib type data, using any [{'_tpl': <persistent broken MaKaC.review.
NotificationTemplate instance '\x00\x00\x00\x00\x0b\xddb\xe5'>, '_id': 0, '__Broken_newargs__':
()}]

Thank you!

5

Hello,

Could this be related to the issue that I have posted a few week ago about events with string ids not migrated in version 2.0? My process did stop with an error when it was working on an abstract (the first abstract it tried to process).

Just a thought

Thank you
Penelope

6

doesn’t sound related

7

Could this be an abstract that is marked as a duplicate of itself? That would make sense, since there isn’t an ID before the INSERT, so this would require an extra UPDATE (post_update option mentioned here).

@daelmaselli, you could try adding, right after this line, still within the for loop (but outside the if) the following:

if abstract.duplicate_of == abstract:
    raise Exception('Whoops. Abstract {} in Event {}'.format(abstract.friendly_id, self.event.id))

(untested)
If my theory is correct, you should get the ID of the broken abstract and the corresponding event.

8

I tried to put the two lines but the result is exactly the same as my first post at the same event.

Any other thoughts?

Thank you very much!

9

Hm… maybe you can try the same with abstract.merged_into, just to be sure it’s not that either?

10

I work with Dael at INFN.

I tried with abstract.merged_into but no exception was raised and the result is the same as first post.

Next steps?

Thanks.

Best regards,
Michele Tota

11

OK, new theory. This is the data the script is trying to INSERT when you flush() explicitly:

{
 'comment': u'duplicate of #9',
 'modified_dt': None,
 'user_id': 523,
 'proposed_contribution_type_id': None,
 'abstract_id': 2370,
 'track_id': 149,
 'created_dt': datetime.datetime(2017, 3, 24, 8, 58, 53, 747417),
 'proposed_action': <AbstractAction.mark_as_duplicate: 4>,
 'proposed_related_abstract_id': None
}

As you can see proposed_related_abstract_id is null, which shouldn’t be the case. We have a check that should keep this from happening, but it’s basically a no-op, since an exception is never being raised. This is the patch:

Can you give it a try and let us know?

12

We tried but the migration crashed at the same point.

Any other suggestions?

Thank you very much!

Michele

13

Hi there,
Just to check if you guys have managed to find a solution for this, otherwise let’s have another look at it and try to get you on 2.0 ASAP.

14

Hi Pedro,

we just retried the migration after deleting the event that caused the error. Unfortunately we had another error some events after:

! 13362 menus Skipping link “Accesso alla rete” with no URL
! 1344 timetable Block boundary (end violated; extending block from 2009-05-11 11:00:00+00:00 to 2009-05-11 13:00:00+00:00)
! 1345 menus Skipping top-level menu entry program_my_tracks
! 1346 menus Skipping top-level menu entry program_my_tracks
! 1348 menus Skipping top-level menu entry program_my_tracks
! 13557 shorturl Shorturl http://agenda.infn.it/event/fccp2017 is invalid: url
! 1361 menus Skipping top-level menu entry program_my_tracks

Traceback (most recent call last): |/opt/indico/.venv/bin/indico-migrate:11 []
|/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/cli.py:126 [main]
|/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:722 [call]
|/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:697 [main]
|/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:895 [invoke]
|/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:535 [invoke]
|/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/cli.py:122 [cli]
debug=debug, **kwargs)
|/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/migrate.py:106 [migrate]
|/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/importer.py:161 [run]
|/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/util.py:323 [_f]
|/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:183 [migrate]
|/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:211 [migrate_event_data]
|/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:137 [run_step]
|/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/init.py:83 [run]
|/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/registration.py:177 [migrate]
|/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/registration.py:188 [migrate_regform]
|/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/registration.py:553 [_migrate_registrations]
|/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/registration.py:574 [_migrate_registration]
|/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/registration.py:878 [_migrate_registration_fields]
|/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/registration.py:907 [_migrate_registration_field]
.format(old_item._value))

‘ascii’ codec can’t decode byte 0xc3 in position 1: ordinal not in range(128)
! Saving restore point…
! Restore point saved.

15

The problem probably lies in the fact that old_item._value is a byte string not unicode. Try to edit the line 907 from the file /opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/registration.py.

Instead of

.format(old_item._value))

use

.format(old_item._value.decode('utf-8')))

16

Or you could use the convert_to_unicode function on the old_item._value which should handle that properly as well.

17

Thank you Michal,
the migration goes on (76%). Now we have this error:

×    224  timetable   User 158 doesn't exist
×    224  timetable   User 1239 doesn't exist
×    224  attachments User 1239 doesn't exist


Traceback (most recent call last): |/opt/indico/.venv/bin/indico-migrate:11 [<module>]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/cli.py:126 [main]
 |/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:722 [__call__]
 |/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:697 [main]
 |/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:895 [invoke]
 |/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:535 [invoke]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/cli.py:122 [cli]
 debug=debug, **kwargs)
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/migrate.py:106 [migrate]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/importer.py:161 [run]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/util.py:323 [_f]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:183 [migrate]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:211 [migrate_event_data]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/importer.py:137 [run_step]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/__init__.py:83 [run]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/misc.py:304 [migrate]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/attachments.py:66 [migrate_event_attachments]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/attachments.py:104 [_folder_from_material]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/importer.py:150 [protection_from_ac]
 assert principal is not None
18

@kolodzie, did you produce a patch/PR for this?

19

Thanks for reminding me about it! I haven’t committed it yet.

20

@michele.tota, that looks like broken DB data. You can try this patch:

--- a/indico_migrate/importer.py
+++ b/indico_migrate/importer.py
@@ -145,10 +145,12 @@ class Importer(object):
         elif ac._accessProtection == 1:
             target.protection_mode = ProtectionMode.protected
             acl = getattr(target, acl_attr)
-            for principal in getattr(ac, ac_attr):
-                principal = self.convert_principal(principal)
-                assert principal is not None
-                acl.add(principal)
+            for old_principal in getattr(ac, ac_attr):
+                principal = self.convert_principal(old_principal)
+                if principal is None:
+                    print old_principal, principal
+                else:
+                    acl.add(principal)
         else:
             raise ValueError('Unexpected protection: {}'.format(ac._accessProtection))

… so that you get an idea of the Avatar object(s) that cause(s) this issue.