Migrating from 2.3.3 on CentOS 7 > Ubuntu

Administrators
1

Hi All,

We, like many places, are migrating all of our old CentOS 7 machines in prep for the EoL next month. We have a VM running Indico 2.3.3 on it, I’ve been tasked with migrating to a new Ubuntu machine, as well as upgrading to the new release of Indico.

I saw this post and have copied over everything to an nfs share that’s mounted on the new Ubuntu machine.

I was going to install Indico 2.3.3 and import the data, and once that installation was confirmed working, upgrade up to the latest release. The problem I’m having is that the install guide for 2.3 on the readthedocs site seems to be gone. Is it possible to bring it back up, or is there any guide on how to import data from version 2 to 3? Is it possible or will we have to just recreate the data manually?

2

Don’t bother installing old Indico. Especially w/ Python 2 it’s just a huge pain to do so. So better to install v3.3 straight away.

Only one thing you need to keep in mind: When you’d usually run indico db upgrade, you need to run these two commands due to a bug that will only be fixed in the next release, and that only occurs when going straight from v2 to v3.3: indico db upgrade 735dc4e8d2f3 and then indico db upgrade

In any case, old installation guides can be found on RTD, there’s actually a version picker on the bottom left: Installation guides — Indico 2.3.6-dev documentation But once again, do not waste your time with it. It’ll be painful and you just cause yourself much more work.

3

Good to know, so I just follow the install guide, skipping the database creation with a pg_restore, and then I run the db upgrade commands you mentioned, and I should be good to go? Sounds relatively straightforward.

Also, bizarre that the version picker wasn’t working for me prior but now when I follow your link it works just fine. Wonder if it was just a caching/browser issue on my end…

4

Probably because we renamed some directories (debian → dev, centos → rpm) and it doesn’t know how to map those between old and new versions. Best to go to the main docs page before switchting versions.

5

Any experience in how long the pg_restore of the old database takes? The process seems to just run indefinitely and I don’t seem to be getting any output even with a -v flag. This is the full command I’m running: su - postgres -c "pg_restore -v -U indico -f ./indico_5_23_24_dump.gz"

6

For a typical Indico instance that has maybe a few thousand events max, I would expect a few seconds.

Does pg_restore support (manually-)gzipped dumps? I always use the -Fc format in pg_dump (dunno if it’s compressed or not, I think it is), and restoring that works fine.

And I think you forgot to specify the DB name! Add -d indico to your command. And of course make sure to create the DB and the two Postgres extensions first (since that requires PG superuser privs which you drop with the -U indico)

7

So I created the gz by piping the pg_dump into gzip, may remake it with the -Fc.

Also, based on the other forum post I was thinking I shouldn’t make an indico database prior, so that may be part of it as well.

Should add, pg_restore doesn’t allow you to use the -f and the -d flags together, it inherits the name of whatever database is in your dump file. Should be fine in this case since they’re named the same. (I hope)

8

Create the database and extensions, then pg_restore everything else. You can also use the -O flag to skip ownership data since running it as the indico user will already result in the correct ownership anyway (if you do it without -O then you can run it as the Postgres superuser since the ownership will be changed to indico anyway).

9

Hmm, no luck still. Created the dump on the old machine using this command:

su - postgres -c "pg_dump -Fc indico -f /tmp/indico_5_24_24_dump

And then I tried to restore it with this one, after creating the indico db and adding the extensions as noted in the installation guide.

su - postgres -c "pg_restore -v -U indico -O -f /tmp/indico_5_24_24_dump"

The process still just hangs indefinitely.

10

Postgres is running though?

Maybe see if strace pg_restore ... gives you any useful indication on why it’s hanging. Alternatively, your question may be better suited in the #postgresql IRC channel on Libera.Chat since it’s certainly more related to Postgres than to Indico!

11

Did as you suggested, hopped onto the #postgresql channel. Turns out that unlike every other psql command, pg_restore uses the -f flag to specify file output, not input. Once I dropped the -f command, running this worked like a charm. Leaving it here for posterity.

su - postgres -c "pg_restore -v -O -d indico /tmp/indico_5_24_24_dump"

12

@ThiefMaster Update, after importing the db via pg_restore I tried to run the indico db upgrade 735dc4e8d2f3 command but got the following output:

Traceback (most recent call last):
  File "/opt/indico/.venv/bin/indico", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/indico/cli/util.py", line 108, in invoke
    returnB self._impl.invoke(ctx)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 1685, in invoke
    super().invoke(ctx)
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lBib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/flask/cli.py", line 383, in decorator
    app = ctx.ensure_object(ScriptInfo).load_app()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/flask/cli.py", line 328, in load_app
    app: Flask | None = self.create_app()
                        ^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/indico/cli/util.py", line 26, in _create_app
    return make_app()
           ^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/indico/web/flask/app.Bpy", line 420, in make_app
    multipass.init_app(app)
  File "/opt/indico/.venv/lib/python3.12/site-packages/indico/core/auth.py", line 80, in init_app
    super().init_app(app)
  File "/opt/indico/.venv/lib/python3.12/site-packages/flask_multipass/core.py", line 74, in init_app
    state.identity_providers = ImmutableDict(self._create_providers('IDENTITY', IdentityProvider))
                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/flask_multipass/core.py", line 461, in _create_providers
    cls = resolve_provider_type(base, settings.pop('type'), registry)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.BB12/site-packages/flask_multipass/util.py", line 158, in resolve_provider_type
    cls = entry_point.load()
          ^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/importlib_metadata/__init__.py", line 184, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.pyenv/versions/3.12.3/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _BBfind_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/opt/indico/.venv/lib/python3.12/site-packages/flask_multipass/providers/ldap/__init__.py", line 7, in <module>
    from .providers import LDAPAuthProvider, LDAPGroup, LDAPIdentityProvider, AuthFallbackLDAPIdentityProvider
  File "/opt/indico/.venv/lib/python3.12/site-packages/flask_multipass/providers/ldap/providers.py", line 10, in <module>
    from ldap import INVALID_CREDENTIALS
ModuleNotFoundError: No module named 'ldap'
13

If you use ldap authentication in your config, you need to pip install python-ldap

14

Got it installed, I had it installed on a previous snapshot earlier in this process and forgot to re-install after a snapshot reversion.

Sadly, didn’t clear all the errors, now it seems to be stuck on alembic version:

https://pastebin.com/pseKRkfy

15

That paste is private. Please put code, logs, etc. directly in your post inside markdown code block fences.

16

My bad! Apologies, it’s a doozy of output.

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
Traceback (most recent call last):
  File "/opt/indico/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/opt/indico/.venv/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.InsufficientPrivilege: permission denied for table alembic_version


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/indico/.venv/bin/indico", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/indico/cli/util.py", line 108, in invoke
    return self._impl.invoke(ctx)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/flask/cli.py", line 386, in decorator
    return ctx.invoke(f, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/indico/cli/database.py", line 139, in _call_with_plugins
    func(*args, **kwargs)
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/flask/cli.py", line 386, in decorator
    return ctx.invoke(f, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/flask_migrate/cli.py", line 154, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/opt/indico/.venv/lib/python3.12/site-packages/flask_migrate/__init__.py", line 111, in wrapped
    f(*args, **kwargs)
  File "/opt/indico/.venv/lib/python3.12/site-packages/flask_migrate/__init__.py", line 200, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/opt/indico/.venv/lib/python3.12/site-packages/alembic/command.py", line 403, in upgrade
    script.run_env()
  File "/opt/indico/.venv/lib/python3.12/site-packages/alembic/script/base.py", line 583, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/opt/indico/.venv/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
    module = load_module_py(module_id, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/opt/indico/.venv/lib/python3.12/site-packages/indico/migrations/env.py", line 91, in <module>
    run_migrations_online()
  File "/opt/indico/.venv/lib/python3.12/site-packages/indico/migrations/env.py", line 83, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/opt/indico/.venv/lib/python3.12/site-packages/alembic/runtime/environment.py", line 948, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/opt/indico/.venv/lib/python3.12/site-packages/alembic/runtime/migration.py", line 605, in run_migrations
    heads = self.get_current_heads()
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/alembic/runtime/migration.py", line 542, in get_current_heads
    row[0] for row in self.connection.execute(self._version.select())
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1385, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/indico/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
    self._handle_dbapi_exception(
  File "/opt/indico/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
    util.raise_(
  File "/opt/indico/.venv/lib/python3.12/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/opt/indico/.venv/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/opt/indico/.venv/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.InsufficientPrivilege) permission denied for table alembic_version

[SQL: SELECT public.alembic_version.version_num
FROM public.alembic_version]
(Background on this error at: https://sqlalche.me/e/14/f405)
17

Sounds like some of the tables you restored are not owned by the indico Postgres user. Most likely you did not use -U indico but specified -O…

18

Looks like in all my variations of that command I messed up the flags. I just am running it again as postgres and seems things are running fine now.

The pg_restore command that actually worked: su - postgres -c "pg_restore -v -U indico -O -f /tmp/indico_5_24_24_dump"

Seems that indico_db_upgrade 735dc4e8d2f3 and indico db upgrade worked smoothly after re-restoring the db.

Gonna see if the rest of the process goes smoothly now.

1 Like
19

I managed to make a typo in this post I didn’t see till now. I left the troublesome -f flag in the final pg_restore command.

I also found an alternate version of doing this migration:

  1. On your original server with your indico 2.3 instance, run `su - postgres -c “pg_dump indico | gzip > /tmp/indico_dump.gz” as root.

  2. Copy this .gz file over to your new host.

  3. On the new host, run su - postgres -c "zcat /tmp/indico_dump.gz | psql indico"

  4. Run as the indico user indico db upgrade 735dc4e8d2f3 and then indico db upgrade.

  5. Finally, restart all of your relevant services:

NGINX: sudo systemctl restart nginx.service indico-celery.service indico-uwsgi.service

Apache: sudo systemctl restart apache2.service indico-celery.service indico-uwsgi.service

1 Like