Indico migrations strategy

Administrators
1

Hey folks!

I would like to raise a similar issue to #24 but my goal here is a bit more than just adding a couple startup lines. I’d like to:

  1. discuss & document design decisions
  2. converge if there is a “recommended” way to run migrations in the project
  3. apply the “recommended” way in code if one exists
  4. optionally create some safeguards so the “recommended” way won’t be accidentally broken

This sounds like a lot, but it’s quite possible there exists a single clear answer already and we can just write it down and apply it. It’s also entirely possible that the response is “the Indico project does not wish to recommend any particular approach and the burden of choosing the correct path is always up to the instance admins”.

Prior work

While writing this, I have consulted:

Migration styles

I’ll define the following styles of migrations very broadly. Let’s say we have version 1.0 and we wanna upgrade to version 1.1 that includes DB migrations.

  1. Downtime. Shut down all instances of Indico 1.0 running. Run the migration. Start Indico 1.1.
  2. Zero downtime - after. Start up Indico 1.1 and stop Indico 1.0. Run the migration.
  3. Zero downtime - before. Keep Indico 1.0 running. Run the migration. Start up Indico 1.1 and stop 1.0.

Downtime should work pretty much always, but requires downtime. 0down A(fter) requires that the the new app version can run temporarily on the old DB schema. 0down B(efore) requires that the old Indico app version can run, at least temporarily, on the new DB schema.

I am purposefully avoiding the words “forwards/backwards compatible” as they change direction based on if we’re talking about the app or the schema and can become very confusing.

What is the recommended migration approach?

As far as I have found, no specific migration approach is recommended in the docs. Please correct me if I’ve missed something, or there is a clearly standardized approach in the community that I’m not aware of!

Given the migrations I have seen (e.g this latest one at time of writing), it is possible to run 0down B with no issues. In fact, all the migrations I checked in the last couple years would allow an admin to run 0down B with no issues. In contrast, running 0down A will produce errors as the new version will fail to find columns until the migration is done.

How would we automate each approach?

Assuming that we commit to a preferred approach, how would the admins go about implementing and automating it? I think it’s useful to consider the impact of our choices on them, both in successful and failed migration outcomes.

Breaking

We would use a script or component that is external to the app itself, as it has to scale all running instances down to 0, then perform the upgrade. In distributed systems this is a job often done by the “coordinator” component. In Kubernetes, this is often done by an operator. In its simplest form, this is a shell script or an Ansible playbook. When we run into migration issues, the downtime is extended, but data is not corrupted.

0down A

For this we can either use a script or component as mentioned above, or alternatively, we can add a command to run migrations After startup. These migrations can run either within the app, or within the startup script, or even in a separate component (e.g a Kubernetes Job). When we run into migration issues we may end up with the app responding with errors, and possibly data corruption as it may be attempting to write using a schema that is not fully applied. If we want to avoid errors during normal operation, the new app version would need to be compatible with the old schema, which would mean that we should not be using the new database columns we just created. This approach sounds increasingly problematic and out-of-step with how the Indico project seems (to me) to operate.

0down B

If we add a migration command in the startup script, whether containerized or not, we are locking in the assumption that the schema migrations will always run Before a new version is started. This is not bad, most apps I know work this way [citation needed]! When we run into migration issues the app would typically be unaffected and the schema can be rolled back to its last working state. This would require that our migrations do not break the schema for existing running Indico versions.

There is the following note which hints to this:

Migration should always be possible in both directions, so when writing a migration step make sure to test it and to implement both directions for structure and data even if that means dropping columns or tables. Losing data during a downgrade is acceptable as long as it’s data that didn’t exist before that revision. Please make sure to TEST migration in both directions!

If we were to embrace this approach, I think this note could also communicate: “Data loss during an upgrade is not acceptable. Breaking schema changes should be rare and clearly communicated. Breaking changes to the schema will be accompanied by a major release version bump.”

(It is unclear to me what Indico’s exact versioning policy is, but bumping major versions on breaking schema changes doesn’t require going “full semVer”).

There are two concerns with this approach which I’m just outlining here and can discuss further:

Recap

So, we are down to the “Breaking” and “0down B” options. Regardless of outcome, I think it’d be worth documenting the choice.

Breaking:

  • Always requires downtime
  • Does not require any work or guarantees from the Indico project
  • Admins can create simple scripts to automate their migration and upgrade process

0 downtime - before:

  • Can usually avoid downtime
  • Requires the project commitment to non-breaking migrations by default (kinda exists already)
  • We can embed the approach in the startup scripts

So in the end, my questions are these two:

  1. What process do you, other admins, use for upgrading/migrating?
  2. Do you think we should adopt a recommendation (and if yes which)?
2 Likes
2

Schema changes are pretty much always “breaking” if it’s not just a newly added table - but “breaking” in this context just means that the old version may fail on the new schema (typically when inserting NOT NULL columns, since we usually do not set db-level default value).

In reality, this is usually not a problem - we’re talking about a period of a few seconds here between running `indico db upgrade and processing incoming requests on the new version.