Need help upgrading from Indico 0.94

Naoya_Ozawa · 2018-06-29 01:56:04 UTC

Hello,

My name is Naoya Ozawa, and I am a member of the network admin at the Center for Nuclear Study, University of Tokyo.

We are currently running Indico 2.1 on CentOS 7 for main purposes, but we also have an old database created with Indico 0.94.

Our plan is to run the main database at indico2.cns.s.u-tokyo.ac.jp and the old database at indico.cns.s.u-tokyo.ac.jp .
For this purpose, we first want to migrate the old database (0.94) to the newest version of Indico (2.1).

Up to now, we have been attempting the upgrade using the method in https://talk.getindico.io/t/need-help-upgrading-from-indico-0-98/156 , but since the Indico GitHub only contains versions down to 0.97, the migration tool cannot run the migration from 0.94 to 0.97.

According to the instructions for 0.97:
https://getindico.io/legacy-docs/wiki/Releases/Indico0.97.html ,
migrations from 0.94 should first go through migrations to 0.96.2, according to the “from older versions” section.

So the steps to be taken will be 0.94->0.96.2->0.97->1.2.2->2.0.3->2.1 .

Unfortunately, the download links for Indico 0.96.2 are broken in the page posted above, so we cannot retrieve the necessary migration tools.

Is there any way to run the migration up to 0.97?

pferreir · 2018-06-29 07:51:08 UTC

Hello,

0.94 is probably some 10 years old. In the meantime things have changed substantially, so even if you get the Indico code you will need to do some “archaeological work” to set up the right dependencies. I’ve made 0.96.2 temporarily available on here.

You will probably want to follow these instructions:

https://getindico.io/legacy-docs/wiki/Admin/InstallationOlder.html

Good luck!

Naoya_Ozawa · 2018-06-29 10:11:36 UTC

Thank you for the link!

We have downloaded the 0.96.2 and got stuck at the installation:
The first step for setting up 0.96.2 (“setup.py upgrade”) requires a MaKaC configuration, but we don’t have a MaKaC installation in the migration environment.

Consulting https://getindico.io/legacy-docs/wiki/Admin/install/from0.html, it seems that a running 0.94 should be existing for the migration.
The thing is, we only have the database (indico.fs) and the archive folder available, so we do not have a running Indico 0.94. The database and archive are mounted on a CentOS 5 Docker container with Python 2.4, along with the Indico 0.96.2 files.

Do you happen to have a copy of 0.94 as well?
Or is there possibly a better workaround?

Naoya_Ozawa · 2018-07-06 10:58:40 UTC

(P.S.)

Currently, we are trying to set up the Indico 0.96.2 without a running 0.94.

When we run the TZMigration.py script, it keeps asking for dependent modules;
for example:

[root@337f6674a81c tools]# python TZMigration.py
Traceback (most recent call last):
  File "TZMigration.py", line 31, in ?
    from MaKaC.common import DBMgr
  File "/usr/lib/python2.4/site-packages/MaKaC/common/__init__.py", line 22, in ?
    from db import DBMgr
  File "/usr/lib/python2.4/site-packages/MaKaC/common/db.py", line 28, in ?
    from ZEO.ClientStorage import ClientStorage
ImportError: No module named ZEO.ClientStorage

According to the instructions, we are aware that ZODB3-3.4.2 is at least required, but for the other dependent modules, we have no clue on which version is required for each package.

Do you happen to have a list of required packages, just like the “requirements.txt” included in the Indico v1.2.2 package?

We are grateful for any assistance.

ThiefMaster · 2018-07-06 11:52:26 UTC

Hi, the latest version on Git is 0.96 and from what I can see there we only required a ZODB3 package… so I’d expect ZEO to be part of that. Maybe the ZODB version you have is too old. Try installing slightly newer ZODB versions until you find one that works (but don’t go too new, they dropped Python 2.4 support at some point for sure)

Naoya_Ozawa · 2018-07-10 09:20:11 UTC

Thank you for the suggestion!

We have installed the ZODB3-3.8.2 and the dependencies (ZConfig, zdaemon, zope.interface, zope.proxy, zope.testing) from PyPI and download.zope.org, and they seem to be installed successfully.

Now retrying the TZMigration.py, as soon as we run it with the “logging filename” parameter and the “category/timezone file” parameter, the process just hangs (more than an hour).

When we force it to stop, we get

Traceback (most recent call last):
  File "TZMigration.py", line 421, in ?
    DBMgr.getInstance().startRequest()
  File "/usr/lib/python2.4/site-packages/MaKaC/common/db.py", line 77, in getInstance
    cls._instance=DBMgr()
  File "/usr/lib/python2.4/site-packages/MaKaC/common/db.py", line 71, in __init__
    self._storage=ClientStorage(cfg.getDBConnectionParams(), username=cfg.getDBUserName(), password=cfg.getDBPassword(), realm=cfg.getDBRealm())
  File "/usr/lib64/python2.4/site-packages/ZODB3-3.8.2-py2.4-linux-x86_64.egg/ZEO/ClientStorage.py", line 347, in __init__
    self._wait(wait_timeout)
  File "/usr/lib64/python2.4/site-packages/ZODB3-3.8.2-py2.4-linux-x86_64.egg/ZEO/ClientStorage.py", line 362, in _wait
    self._rpc_mgr.connect(sync=1)
  File "/usr/lib64/python2.4/site-packages/ZODB3-3.8.2-py2.4-linux-x86_64.egg/ZEO/zrpc/client.py", line 153, in connect
    self.cond.wait(30)
  File "/usr/lib64/python2.4/threading.py", line 222, in wait
    _sleep(delay)
KeyboardInterrupt

It seems to be waiting for something, but what should it be…?

ThiefMaster · 2018-07-10 09:22:50 UTC

That usually means it cannot connect to your ZODB (zeo/zdaemon instance). Unlike decent software that would just abort with an error, ZODB retries pretty much forever until you kill it (which then results in that traceback you posted)

Naoya_Ozawa · 2018-07-10 09:44:59 UTC

Is there a method to connect the freshly installed Indico 0.96.2 to ZODB ?

The Indico installation instructions just assume that the current server is running with Indico 0.94 hooked up with the running ZODB.
But in our case, we only have the (freshly installed) ZODB, so there might be extra steps we have to take.

ThiefMaster · 2018-07-10 09:47:29 UTC

You need to run zdaemon/ZEO so indico can connect to your ZODB. Check the etc/ folder for the example zodb/zdaemon configs, you can probably use those.

IIRC. the command to start zdaemon was zdaemon -C /path/to/zdctl.conf start

Naoya_Ozawa · 2018-07-17 06:35:57 UTC

Using the same method as Indico 0.92, we were able to start the ZODB, and TZMigration.py worked!

Now moving on to the migration from 0.96.2 to 0.97.1, the setup.py install for Indico fails with

Installed /usr/lib/python2.6/site-packages/cds_indico-0.97.1-py2.6.egg
Processing dependencies for cds-indico==0.97.1
Searching for cds-indico-extras>=0.2
Reading http://cdswaredev.cern.ch/indico/wiki/Admin/Installation/IndicoExtras
Download error: [Errno -2] Name or service not known – Some packages may not be found!
Reading http://pypi.python.org/simple/cds-indico-extras/
Couldn’t find index page for ‘cds-indico-extras’ (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading http://pypi.python.org/simple/
No local packages or download links found for cds-indico-extras>=0.2
Best match: None
Traceback (most recent call last):
File “setup.py”, line 402, in
EXTRA_RESOURCES_URL
File “/usr/lib64/python2.6/distutils/core.py”, line 152, in setup
dist.run_commands()
File “/usr/lib64/python2.6/distutils/dist.py”, line 975, in run_commands
self.run_command(cmd)
File “/usr/lib64/python2.6/distutils/dist.py”, line 995, in run_command
cmd_obj.run()
File “/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/install.py”, line 76, in run
File “/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/install.py”, line 104, in do_egg_install
File “/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py”, line 211, in run
File “/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py”, line 427, in easy_install
File “/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py”, line 478, in install_item
File “/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py”, line 519, in process_distribution
File “/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/pkg_resources.py”, line 563, in resolve
File “/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/pkg_resources.py”, line 799, in best_match
File “/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/pkg_resources.py”, line 811, in obtain
File “/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/easy_install.py”, line 434, in easy_install
File “/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/package_index.py”, line 475, in fetch_distribution
AttributeError: ‘NoneType’ object has no attribute ‘clone’

It seems to be looking for the “cds-indico-extras” package, but unable to find it, since the link is already broken.
If you have a copy of it, could you upload it again?
Or can it be ignored (i.e. commented out from the script) for migration?

We are using a CentOS 6.9 Docker container with Python 2.6 now.

ThiefMaster · 2018-07-17 07:24:31 UTC

You can safely skip that package.

Naoya_Ozawa · 2018-07-18 03:47:11 UTC

We skipped the indico-extras package and proceeded.
In the migration step, the command

python2.6 -c “from MaKaC.common.db import DBMgr;
DBMgr.getInstance().startRequest(); from MaKaC.modules.base import ModulesHolder;
del ModulesHolder()._getIdx()[‘upcoming_events’]; DBMgr.getInstance().endRequest()”

failed with a ‘KeyError?: upcoming_events’ error, so we skipped this also (as it is suggested to).
So the only operations in migrating from 0.96.2 -> 0.97 were the two re-indexing scripts.

Now, we are attempting the migration from 0.97 to 1.2.2 based on the forum reference, and got the following output:

This script will migrate your Indico DB to a new version. We recommend that
this operation be executed while the web server is down, in order to avoid
concurrency problems and DB conflicts.


Are you sure you want to execute the migration now? [y/N] y

Executing migration...

Probing DB connection... DONE!

# Adding new plugins and adapting existing ones to new name policies (0.98b2)
  DONE

# Reloading all plugins (0.97)
  DONE

# Fixing AccessController for categories (0.98b)
  DONE

# Adding missing attributes to conference objects and children (0.98b2)
  DONE77 97.111913%] 91 Bi-weekly meeting

# Migrating database tasks from the old format to the new one (0.98b)
  DONE

# Replacing the conference dictionary in the Category objects by a OOTreeSet. (0.98b)
  DONE

# Replacing category date indexes. (0.98b)
  DONE

# Create category date index without visibility. (0.98.1)
  DONE

# Initializing/updating index catalog (0.98b)
  DONE

# Initializing room blocking indexes. (0.98b2)
  DONE

# Initializing room+day => reservation index. (0.98.1)
  DONE

# Migrate the reservation notification system. (0.98-rc2)
  DONE

# Replacing en_US with en_GB. (0.98b2)
  DONE

# Installing new TPLs for meeting/lecture styles (0.98b2)
  DONE

# Modifying Room GUIDs (0.98b2)
  DONE

# Add missing location info to slots of a session that contains location or room (0.98.1)
  DONE77 97.111913%] 91 Bi-weekly meeting

# Creating new "All Requests" index (0.98.3)
  Collaboration plugin not usable - jumping task
  DONE

# Migrating Chat Room index to new structure (0.99)
  IM plugin not usable - jumping task
  DONE

# Removing TimedLinkedEvents (0.99)
  DONE

# Moving from OAI Private Harvesting to a more general IP-based ACL. (1.0)

Migration failed! DB may be in  an inconsistent state:
Traceback (most recent call last):
  File "/indico/bin/migration/migrate.py", line 1105, in main
    dry_run=args.dry_run)
  File "/indico/bin/migration/migrate.py", line 1051, in runMigration
    task(dbi, withRBDB, prevVersion)
  File "/indico/bin/migration/migrate.py", line 619, in ip_based_acl
    ip_set = set(minfo._oaiPrivateHarvesterList)
AttributeError: 'MaKaCInfo' object has no attribute '_oaiPrivateHarvesterList'

We don’t see this error happening on the previous forums.
What is this error?

Naoya_Ozawa · 2018-07-20 09:34:16 UTC

By commenting out the few error-emitting parts in the migration scripts, the DB has now been migrated to Indico 2.0!
Here is a review of the process from 0.97 up to 2.0:

0.97 -> 1.2.2 (On CentOS 6.9 + Python 2.6 (Docker))

The error regarding “_oaiPrivateHarvesterList” (in the previous post) seemed to have no simple solution, so we just decided to ignore it.
The function “ip_based_acl” defined from line 613 of migration.py was modified to exclude “ip_set”, which was causing the error.

This was the only error for this script, and the migration seemed to have succeeded.

1.2.2 -> 2.0 (On CentOS 7 + Python 2.7.5)

The migration script produced the next two errors:

'MaKaCInfo' object has no attribute '_socialAppConfig'

and

u'upcoming_events'

The first one was avoided again by modifying the function “migrate_global_settings” defined from line 84 in /path/to/python/2.7/site-packages/indico_migrate/steps/global_pre_events.py by excluding the “social_settings.set_multi” section.

The second one didn’t even have an error message, but it seems to have occurred from line 46 of /site-packages/indico_migrate/steps/global_post_events.py.
This is inside the definition of a function “migrate_upcoming_event_settings”, which is used in the function “migrate” on line 41 of the same script.
So, the error producing component of this “migrate” function

self.migrate_upcoming_event_settings()

was commented out again.

Following these steps, the migration tools for 0.97 -> 1.2.2 and 1.2.2 -> 2.0 both worked.
The last step will be to upgrade the Indico 2.0 to the latest 2.1.

pferreir · 2018-07-23 13:01:53 UTC

Great news! Now don’t forget to upgrade more often!

Naoya_Ozawa · 2019-03-15 08:58:26 UTC

The final upgrade from 2.0 to 2.1.8 (latest) has been successful by following the standard steps, except for a single retry due to a known issue at the Add AttachmentFolder.is_hidden stage of the upgrade.

Just to sum up, here are the steps we have taken:

with CentOS5 + Python2.4

Install ZODB3 and its dependencies which are the requirements for the Indico 0.96
Start the ZODB running by correctly writing the config files
Run the TZMigration.py for Indico 0.96.2

with CentOS6.9 + Python2.6

Install Indico 0.97.1 and migrate from 0.96.2 to 0.97.1; commenting out the part where it asks for the cds-indico-extras
Install Indico 1.2.2 and migrate from 0.97.1 to 1.2.2, also ignoring error-emitting parts

with CentOS7 + Python2.7.5

Use the distributed migration tool to migrate up to 2.0, also ignoring error-emitting parts
Use the standard method of upgrading to the latest 2.1.8

Thank you for all the support!
(and we will make sure that the upgrade is done more often!)