Issues migrating from Version 1.2 to 2.0

Administrators Migration from 1.2 to 2.0
1

Hello,

At Fermilab we installed the latest indico version v2. The installation went very smoothly. Fantastic job from the CERN team!
BUT we have the following problems:

  1. In two different occasions we imported events from other labs and to maintain their event IDs so that an apache rewrite rule can redirect an old event’s URL the new URL, we prepend the migrated ID with a string.
    The migration process seems to be working (the events are processed) but they are not shown under the category, also if they have abstracts then migration fails (see error log bellow)

  2. we are using mellon for our SSO logins and we are not able to make it work after following all the instruction. We do need some help with how to configure it with flask. We had no problem with indico version 1.2

We would appreciate your help

Thank you
Penelope

---------------------- Migration Error log --------------
i ANLHEP1017  abstract    Contribution field Summary
i ANLHEP1017  abstract    Abstract 10: A New Dimension in Tracking: 4D Silicon Detectors
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
/opt/indico/.venv/bin/indico-migrate in <module>()
      9 if __name__ == '__main__':
     10     sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
---> 11     sys.exit(main())
        global sys.exit = <built-in function exit>
        global main = <function main at 0x62b68c0>

/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/cli.pyc in main()
    124 
    125 def main():
--> 126     return cli()
        global cli = <click.core.Command object at 0x62b4c90>

/opt/indico/.venv/lib/python2.7/site-packages/click/core.pyc in __call__(self=<click.core.Command object>, *args=(), **kwargs={})
    720     def __call__(self, *args, **kwargs):
    721         """Alias for :meth:`main`."""
--> 722         return self.main(*args, **kwargs)
        self.main = <bound method Command.main of <click.core.Command object at 0x62b4c90>>
        args = ()
        kwargs = {}
    723 
    724 

/opt/indico/.venv/lib/python2.7/site-packages/click/core.pyc in main(self=<click.core.Command object>, args=[], prog_name=u'indico-migrate', complete_var=None, standalone_mode=True, **extra={})
    695             try:
    696                 with self.make_context(prog_name, args, **extra) as ctx:
--> 697                     rv = self.invoke(ctx)
        rv = undefined
        self.invoke = <bound method Command.invoke of <click.core.Command object at 0x62b4c90>>
        ctx = <click.core.Context object at 0x62b4d50>
    698                     if not standalone_mode:
    699                         return rv

/opt/indico/.venv/lib/python2.7/site-packages/click/core.pyc in invoke(self=<click.core.Command object>, ctx=<click.core.Context object>)
    893         """
    894         if self.callback is not None:
--> 895             return ctx.invoke(self.callback, **ctx.params)
        ctx.invoke = <bound method Context.invoke of <click.core.Context object at 0x62b4d50>>
        self.callback = <function cli at 0x62b9488>
        ctx.params = {u'verbose': True, u'ldap_provider_name': u'ldap', u'photo_path': None, u'reference_types': (), u'default_currency': u'USD', u'zodb_uri': u'file:///opt/indico-legacy/db/Data.fs', u'symlink_target': u'/opt/indico/archive/legacy_symlinks/', u'system_user_id': None, u'restore_file': None, u'archive_dir': (u'/opt/indico/legacy-archive/',), u'default_group_provider': u'fnal-ldap', u'ignore_local_accounts': False, u'avoid_storage_check': True, u'rb_zodb_uri': None, u'migrate_broken_events': True, u'storage_backend': u'legacy', u'default_email': u'penelope@fnal.gov', u'no_gui': True, u'save_restore': None, u'sqlalchemy_uri': u'postgresql://indico:klf55so@cdpgsdev.fnal.gov:5487/indico', u'symlink_backend': u'legacy-symlinks', u'debug': True, u'dblog': False}
            896 
            897 
        
        /opt/indico/.venv/lib/python2.7/site-packages/click/core.pyc in invoke(*args=(), **kwargs={u'archive_dir': (u'/opt/indico/legacy-archive/',), u'avoid_storage_check': True, u'dblog': False, u'debug': True, u'default_currency': u'USD', u'default_email': u'penelope@fnal.gov', u'default_group_provider': u'fnal-ldap', u'ignore_local_accounts': False, u'ldap_provider_name': u'ldap', u'migrate_broken_events': True, ...})
            533         with augment_usage_errors(self):
            534             with ctx:
        --> 535                 return callback(*args, **kwargs)
                callback = <function cli at 0x62b9488>
                args = ()
                kwargs = {u'verbose': True, u'symlink_backend': u'legacy-symlinks', u'photo_path': None, u'reference_types': (), u'restore_file': None, u'zodb_uri': u'file:///opt/indico-legacy/db/Data.fs', u'symlink_target': u'/opt/indico/archive/legacy_symlinks/', u'avoid_storage_check': True, u'default_currency': u'USD', u'archive_dir': (u'/opt/indico/legacy-archive/',), u'ldap_provider_name': u'ldap', u'ignore_local_accounts': False, u'system_user_id': None, u'rb_zodb_uri': None, u'migrate_broken_events': True, u'storage_backend': u'legacy', u'default_email': u'penelope@fnal.gov', u'default_group_provider': u'fnal-ldap', u'no_gui': True, u'save_restore': None, u'sqlalchemy_uri': u'postgresql://indico:klf55so@cdpgsdev.fnal.gov:5487/indico', u'debug': True, u'dblog': False}
            536 
            537     def forward(*args, **kwargs):
        
        /opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/cli.pyc in cli(sqlalchemy_uri=u'postgresql://indico:klf55so@cdpgsdev.fnal.gov:5487/indico', zodb_uri=u'file:///opt/indico-legacy/db/Data.fs', rb_zodb_uri=None, verbose=True, dblog=False, debug=True, restore_file=None, no_gui=True, **kwargs={u'archive_dir': (u'/opt/indico/legacy-archive/',), u'avoid_storage_check': True, u'default_currency': u'USD', u'default_email': u'penelope@fnal.gov', u'default_group_provider': u'fnal-ldap', u'ignore_local_accounts': False, u'ldap_provider_name': u'ldap', u'migrate_broken_events': True, u'photo_path': None, u'reference_types': (), ...})
            120 
            121     migrate(logger, zodb_root, rb_zodb_uri, sqlalchemy_uri, verbose=verbose, dblog=dblog, restore_file=restore_file,
        --> 122             debug=debug, **kwargs)
                debug = True
                kwargs = {u'save_restore': None, u'ldap_provider_name': u'ldap', u'ignore_local_accounts': False, u'symlink_target': u'/opt/indico/archive/legacy_symlinks/', u'system_user_id': None, u'migrate_broken_events': True, u'symlink_backend': u'legacy-symlinks', u'photo_path': None, u'storage_backend': u'legacy', u'avoid_storage_check': True, u'reference_types': (), u'default_email': u'penelope@fnal.gov', u'archive_dir': (u'/opt/indico/legacy-archive/',), u'default_group_provider': u'fnal-ldap', u'default_currency': u'USD'}
            123 
            124 
        
        /opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/migrate.pyc in migrate(logger=<indico_migrate.logger.StdoutLogger object>, zodb_root={'conferences': <BTrees.OOBTree.OOBTree object a...': <BTrees.OOBTree.OOBTree object at 0x1121a7d0>}, zodb_rb_uri=None, sqlalchemy_uri=u'postgresql://indico:klf55so@cdpgsdev.fnal.gov:5487/indico', verbose=True, dblog=False, restore_file=None, **kwargs={u'archive_dir': (u'/opt/indico/legacy-archive/',), u'avoid_storage_check': True, 'debug': True, u'default_currency': u'USD', u'default_email': u'penelope@fnal.gov', u'ignore_local_accounts': False, u'ldap_provider_name': u'ldap', u'migrate_broken_events': True, u'photo_path': None, u'reference_types': (), ...})
            104                 else:
            105                     step(logger, app, sqlalchemy_uri, zodb_root, verbose, dblog, default_group_provider, tz,
        --> 106                          **kwargs).run()
                kwargs.run = undefined
            107                 MigrationStateManager.register_step(step)
            108             logger.set_success()
        
/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/importer.pyc in run(self=<EventImporter(postgresql://indico:klf55so@cdpgsdev.fnal.gov:5487/indico)>)
    159         self.pre_migrate()
    160         try:
--> 161             self.migrate()
        self.migrate = <bound method EventImporter.migrate of <EventImporter(postgresql://indico:klf55so@cdpgsdev.fnal.gov:5487/indico)>>
    162         finally:
    163             self.post_migrate()

/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/util.pyc in _f(self=<EventImporter(postgresql://indico:klf55so@cdpgsdev.fnal.gov:5487/indico)>, *args=(), **kwargs={})
    321         def _f(self, *args, **kwargs):
    322             self.logger.print_step(description)
--> 323             f(self, *args, **kwargs)
        global f = undefined
        self = <EventImporter(postgresql://indico:klf55so@cdpgsdev.fnal.gov:5487/indico)>
        args = ()
        kwargs = {}
    324         return _f
    325     return _step_description

/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/importer.pyc in migrate(self=<EventImporter(postgresql://indico:klf55so@cdpgsdev.fnal.gov:5487/indico)>)
    181             db.engine.execute(db.text('ALTER TABLE events.{} DISABLE TRIGGER consistent_timetable'.format(table)))
    182         try:
--> 183             self.migrate_event_data()
        self.migrate_event_data = <bound method EventImporter.migrate_event_data of <EventImporter(postgresql://indico:klf55so@cdpgsdev.fnal.gov:5487/indico)>>
    184             db.session.commit()
    185         except:

/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/importer.pyc in migrate_event_data(self=<EventImporter(postgresql://indico:klf55so@cdpgsdev.fnal.gov:5487/indico)>)
    209             for importer in importers:
    210                 with db.session.no_autoflush:
--> 211                     context.run_step(importer)
        context.run_step = <bound method _EventContext.run_step of <indico_migrate.steps.events.importer._EventContext object at 0xc2424850>>
        importer = <EventAbstractImporter(postgresql://indico:klf55so@cdpgsdev.fnal.gov:5487/indico)>
    212 
    213         for importer in importers:

/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/importer.pyc in run_step(self=<indico_migrate.steps.events.importer._EventContext object>, importer=<EventAbstractImporter(postgresql://indico:klf55so@cdpgsdev.fnal.gov:5487/indico)>)
    135     def run_step(self, importer):
    136         importer.bind(self)
--> 137         importer.run()
        importer.run = <bound method EventAbstractImporter.run of <EventAbstractImporter(postgresql://indico:klf55so@cdpgsdev.fnal.gov:5487/indico)>>
    138 
    139     def _fix_naive(self, dt):

/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/__init__.pyc in run(self=<EventAbstractImporter(postgresql://indico:klf55so@cdpgsdev.fnal.gov:5487/indico)>)
     81 
     82     def run(self):
---> 83         self.migrate()
        self.migrate = <bound method EventAbstractImporter.migrate of <EventAbstractImporter(postgresql://indico:klf55so@cdpgsdev.fnal.gov:5487/indico)>>
     84 
     85     @property

/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/abstracts.pyc in migrate(self=<EventAbstractImporter(postgresql://indico:klf55so@cdpgsdev.fnal.gov:5487/indico)>)
    113             self._migrate_contribution_types()
    114             self._migrate_contribution_fields()
--> 115             self._migrate_abstracts()
        self._migrate_abstracts = <bound method EventAbstractImporter._migrate_abstracts of <EventAbstractImporter(postgresql://indico:klf55so@cdpgsdev.fnal.gov:5487/indico)>>
    116         db.session.flush()
    117 

/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/abstracts.pyc in _migrate_abstracts(self=<EventAbstractImporter(postgresql://indico:klf55so@cdpgsdev.fnal.gov:5487/indico)>)
    440     def _migrate_abstracts(self):
    441         for zodb_abstract in self.amgr._abstracts.itervalues():
--> 442             self._migrate_abstract(zodb_abstract)
        self._migrate_abstract = <bound method EventAbstractImporter._migrate_abstract of <EventAbstractImporter(postgresql://indico:klf55so@cdpgsdev.fnal.gov:5487/indico)>>
        zodb_abstract = <persistent broken MaKaC.review.Abstract instance '\x00\x00\x00\x00\x01\xbb$p'>
    443 
    444         if self.event.abstracts:

/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/abstracts.pyc in _migrate_abstract(self=<EventAbstractImporter(postgresql://indico:klf55so@cdpgsdev.fnal.gov:5487/indico)>, old_abstract=<persistent broken MaKaC.review.Abstract instance '\x00\x00\x00\x00\x01\xbb$p'>)
    359         old_contribution = getattr(old_abstract, '_contribution', None)
    360         if old_contribution:
--> 361             assert old_contribution.__class__.__name__ == 'AcceptedContribution'
        old_contribution.__class__.__name__ = 'Contribution'
    362             if old_abstract._currentStatus.__class__.__name__ == 'AbstractStatusAccepted':
    363                 old_contrib_type = old_abstract._currentStatus._contribType

AssertionError: 
> /opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/events/abstracts.py(361)_migrate_abstract()
    359         old_contribution = getattr(old_abstract, '_contribution', None)
    360         if old_contribution:
--> 361             assert old_contribution.__class__.__name__ == 'AcceptedContribution'
    362             if old_abstract._currentStatus.__class__.__name__ == 'AbstractStatusAccepted':
    363                 old_contrib_type = old_abstract._currentStatus._contribType
2

In my initial post on this topic I did not look too deeply into the migration log nor the code (my apologies).

On our site the events are a mixture of several versions and imports from other sites. Imports from remote CDSAgenda and v1.1, migration from 0.9(?) to 0.98 to 1.2 and now to 2.0

At this point I have found the following issues with the migration, looking a the created migration log and the database event entries:

  • Several events that according to the log file were migrated, yet they are not present in the database. One instance: events with ID between 9970 - 9999 are not present in the database but according to the log the file they are migrated. In the migration log there are no errors indicating that something was wrong with their migration. (if needed I can provide the log file lines for these entries)

  • Events that were created as “meeting” type in version 0.98 and then migrated correctly to version 1.2 as meetings, are migrated in v2.0 as “conferences” so the display is not as expected. This is the majority of the events on our site.

  • For some events that do have attachments (contribution material), the material is not migrated and there is no entry in the migration log about its existence.

  • the migration process skipped about 13 hundred events, if I assume that the event processing is sorting the events based on their ids. If this is true then events with IDs > 15772 - ~17120 were not processed at all and the next events processed had IDs starting with ANLHEP.

  • None of the events with string IDs were created in the database although the log file listed these events as created and gives their new integer ID.

  • The category migration had a lot of informational(?) lines like:
    ! [categories] Invalid lecture theme: lecture [667]
    ! [categories] Invalid lecture theme: event [685]
    ! [categories] Invalid meeting theme: cdsagenda [72]

I also, have the following question:
A lot of our events use the “CDS Agenda style” as their default style.
Can the migration process import this style so that it can used in the events that use it:?

Thank you

3

This is very strange. In all the migrations we’ve seen thus far, there haven’t been any missing events.
You mention that your events are a “mixture” of several versions. What does that mean? Was this a linear process from CDSAgenda → 0.9 → 0.98 → 1.2 ?

Events that were created as “meeting” type in version 0.98 and then migrated correctly to version 1.2 as meetings, are migrated in v2.0 as “conferences” so the display is not as expected.

You may want to have a look at this:

The only explanation I can think of is your webfactoryregistry is corrupted. It would be interesting to look at one of such cases and see what was the content of root.webfactoryregistry[event_id] in the old DB.

For some events that do have attachments (contribution material), the material is not migrated and there is no entry in the migration log about its existence.

A possible culprit could be:

Maybe in a past version you had some esoteric material type that we are not accounting for?

A lot of our events use the “CDS Agenda style” as their default style.

This theme has been discontinued and we’re not bundling it with Indico anymore. It’s old, hard to maintain and at least around here, no one was using it.
We didn’t even migrate this theme to Jinja, but if your really want to have it, then you can try giving it a go and create a small plugin that includes it.

4

Thank you for the information.

Some clarifications: for the CDS Agenda import we used as bases the migration code that was included with the 0.9 version at that time. So those events went from 0.9 -> 0.98 -> 1.2 and never made it to 2.0 as the processes never finished. It crashed at the initial error that I reported.

Event type: meeting is converted to conference:
I will try to add some debug code at the areas that you pointed out to see what is going on with the event type.
You mention that it would be interesting to see what the WF information looks like. What information should I try to display from the old DB for the root.webfactoryregistry[event_id]?

Missing events:
Where can I add some code to display what events are going to be processed before the processing starts?

Processed events that are not inserted in the database:
Where shall I add debug code? Or should I use the –dblog in the migration script call? Or perhaps both?

Missing material:
as you indicated, I will add debug code in the …/indico_migrate/attachments.py to check what is going on with the missing material.

As far as the “CDS Agenda style” I will tell the users that it is no longer supported and see what they have to say.

5

Hello,

I found the problem with the abstracts that give an error at the line:

assert old_contribution.__class__.__name__ == 'AcceptedContribution'

It has to do with my event importing script that I did not take into account the Aceepted Abstract condition and did not create the contribution as AcceptedContributrion, so now the

old_contribution.__class__.__name__ == 'Contribution'

Is there a way in version 1.2 to change these contributions’ type to be AcceptedContribution?
I tried to simply set the

contribution.__class__.__name__ = 'AcceptedContribution'

but is does not seem to work.

I would appreciate your help

Thank you
Penelope

6

I would like to ask the following question about indico v2.0:

Do contributions from abstracts have a different class object from regular contributions or the abstract contributions are declared by a flag or some other field?

7

There is no difference except that the Contribution has an abstract_id set if it was created from an abstract.

8

Great!

Then in the migration scripts I can do the following:

assert old_contribution.__class__.__name__ in ('AcceptedContribution', 'Contribution')

This way I do not have to fix the current database.

9

As I am trying to resolve the migration issue that I have with the event type, I found that all the events that were migrated from v0.98 to v1.2 do not have entries in the ‘webfactoryregistry’ so their type is set to ‘conference’. Only the events that I imported have ‘webfactoryresgistry’ entries.

it = zodb_root['webfactoryregistry'].iteritems()
total = len(zodb_root['webfactoryregistry'])

print('Total number of entries in webfactoryregistry: %s' %(total) )
for event_id, wf in it:
    if wf is None:
        # conferences that have been lectures/meetings in the past
        continue

    wf_id = WEBFACTORY_NAME_RE.match(wf.__module__).group(1)
    print('%s --> Is a %s ' %(event_id,wf_id) )

So, the above code (used in the migration to v2) does not return correct information.

When though I run the following script:

catParent = conference.CategoryManager().getRoot()
catList = catParent.getSubCategoryList()
l = []
for cat in catList:
       l[:0] = cat.getAllConferenceList()

for conf in l:
    type = conf.getType()
    id = conf.getId()
    title = conf.title

The event types are correct.

Is there a way in the migration code to get the type is a similar way?

I am still working on the events that were processed but never made it to the PostgreSQL database.

10

The migration from v1.2 to v2.0 was partially successful. The following are the results and comments about the process I followed:

  • I had to add the following code in indico_migrate/cli.py as it was failing at the first utf8 character it encountered:
--- cli.py.orig 2018-06-13 12:06:21.842397392 -0500
+++ cli.py      2018-06-13 12:07:04.289044827 -0500
@@ -18,6 +18,9 @@

 import sys

+reload(sys)
+sys.setdefaultencoding('utf8')
+
 import click
 from IPython.core import ultratb
  • The memory held log file was getting too big and the migration process was killed by the system as it was out of memory (after running for 4 hours). To resolve the issue I added a couple of lines to save the log file every time a message was send to the console:
--- logger.py.orig      2018-06-14 14:44:05.537236201 -0500
+++ logger.py   2018-06-14 16:29:09.094411586 -0500
@@ -52,9 +52,11 @@
         self.buffer.write(b'\n\n' + stack.encode('utf-8') + b'\n')

     def save_to_disk(self):
-        with open('migration.log', 'w') as f:
+        with open('migration.log', 'a') as f:
             self.buffer.seek(0)
             shutil.copyfileobj(self.buffer, f)
+            self.buffer.seek(0)
+            self.buffer.truncate()

     def wait_for_input(self):
         pass
@@ -95,6 +97,7 @@
         if prefix:
             prefix += ' '
         self.buffer.write(strip_cformat(icon + ' ' + prefix + msg \+ suffix + '\n').encode('utf-8'))
+        self.save_to_disk()
 
 
 class StdoutLogger(BaseLogger):

Perhaps you may consider adding the above two fixes to the code.

The following though is my latest error message from the migration process:

 D0_a139  legacyid    -> 35985
- [event] 16934.705397 seconds
 >  Upcoming event settings
- [global_post] 0.020393 seconds
 >  Event series
! [series] Inconsistent series found; merging them [17969]
! [series] Series IDs:    [758, 759, 760, 763, 17970] [17969]
! [series] Reachable IDs: [758, 759, 762, 763, 17969] [17969]
Traceback (most recent call last): |/opt/indico/.venv/bin/indico-migrate:11 [<module>]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/cli.py:129 [main]
 |/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:722 [__call__]
 |/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:697 [main]
 |/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:895 [invoke]
 |/opt/indico/.venv/lib/python2.7/site-packages/click/core.py:535 [invoke]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/cli.py:125 [cli]
 debug=debug, **kwargs)
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/migrate.py:106 [migrate]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/importer.py:161 [run]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/util.py:323 [_f]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/event_series.py:41 [migrate]
 |/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/steps/event_series.py:114 [get_event_series]
 assert len(set(map(frozenset, series))) == 1



*** ERROR
*** There has been an unexpected error during the migration.
*** You may choose to post an error report on gist.github.com which you can then send to the Indico Team for bug debugging purposes.
*** The URL won't be publicly advertised and only data that was shown on the screen will be sent.

Do you wish to submit the error report? [y/N]: y
Traceback (most recent call last):
  File "/opt/indico/.venv/bin/indico-migrate", line 11, in <module>
    sys.exit(main())
  File "/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/cli.py", line 129, in main
    return cli()
  File "/opt/indico/.venv/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/opt/indico/.venv/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/opt/indico/.venv/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/indico/.venv/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/cli.py", line 125, in cli
    debug=debug, **kwargs)
  File "/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/migrate.py", line 127, in migrate
    if not ask_to_paste(logger.buffer):
  File "/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/paste.py", line 53, in ask_to_paste
    return post_gist(buffer.read())
  File "/opt/indico/.venv/lib/python2.7/site-packages/indico_migrate/paste.py", line 69, in post_gist
    gist = gh.create_gist('indico-migrate {} / indico {}'.format(__version__, indico.__version__), files, public=False)
  File "/opt/indico/.venv/lib/python2.7/site-packages/github3/github.py", line 159, in create_gist
    json = self._json(self._post(url, data=new_gist), 201)
  File "/opt/indico/.venv/lib/python2.7/site-packages/github3/models.py", line 100, in _json
    if self._boolean(response, status_code, 404) and response.content:
  File "/opt/indico/.venv/lib/python2.7/site-packages/github3/models.py", line 121, in _boolean
    raise GitHubError(response)
github3.models.GitHubError: 401 Requires authentication
(.venv)[indico@web4603 migrateLogs]$

My questions are:

  • Does indico v.98 (and v1.2) support event series? If so so how are they created/defined to be a series?
    When I check the events that the indico_migration believes is a series of events, I see events that are different and in different categories.
  • If event series is not supported in v1.2, do I need to worry about this step?
    If though it is supported, please help me with this error.

Thank you
Penelope

11

This should not be needed. If you get errors due to UTF8, use convert_to_unicode() on the value to convert it properly instead of messing with the default encoding (there is a reason why Python removes this function by default).

How many events do you have in your DB and how much memory does your machine have?

No, but lecture series, which were indicated by link materials containing the links between the events. We convert these to series during the migration.

If you skip event series, the links between multi-occurrence lectures will not be there, but nothing will break.

12

I had to use sys.setdefaultencoding(‘utf8’) as it was failing in the convert_to_unicode()

My system has 8Gb of memory and 2Gb of swap. The number of events I am transferring is 33,500

Since the event series effects the lecture series, then I do need to a fix the code. Could you please help me in this?

Thank you
Penelope

13

In this case the traceback of the error and ideally also the repr() of the string causing it would be helpful.

The problem with debugging migration issues is that they are extremely hard to debug without access to the actual data. It would be easier if you added some debug code there to see for yourself what’s the problem.
If you need some help, the contents of the series list right before the failure would probably be helpful.

14

Thank you for the prompt response.

I will add debug statements and I will let you know what I find.

I will send the error message for the utf8 as soon as I will be able to run the migration code again.