Skipping releases when upgrading Debian systems

[diziet]

Debian does not officially support upgrading from earlier than the previous stable release: you’re not supposed to “skip” releases. Instead, you’re supposed to upgrade to each intervening major release in turn.

However, skipping intervening releases does, in fact, often work quite well. Apparently, this is surprising to many people, even Debian insiders. I was encouraged to write about it some more.

My personal experience

I have three conventionally-managed personal server systems (by which I mean systems which aren’t reprovisioned by some kind of automation). Of these at least two have been skip upgraded at least once:

The one I don’t think I’ve skip-upgraded (at least, not recently) is my house network manager (and now VM host) which I try to keep to a minimum in terms of functionality and which I keep quite up to date. It was crossgraded from i386 (32-bit) to amd64 (64-bit) fairly recently, which is a thing that Debian isn’t sure it supports. The crossgrade was done a hurry and without any planning, prompted by Spectre et al suddenly requiring big changes to Xen. But it went well enough.

My home “does random stuff” server (media server, web cache, printing, DNS, backups etc.), has etckeeper records starting in 2015. I upgraded directly from jessie (Debian 8) to buster (Debian 10). I think it has probably had earlier skip upgrade(s): the oldest file in /etc is from December 1996 and I have been doing occasional skip upgrades as long as I can remember.

And of course there’s chiark, which is one of the oldest Debian installs in existence. I wrote about the most recent upgrade, where I went directly from jessie i386 ELTS (32-bit Debian 8) to bulleye amd64 (64-bit Debian 11). That was a very extreme case which required significant planning and pre-testing, since the package dependencies were in no way sufficient for the proper ordering. But, I don’t normally go to such lengths. Normally, even on chiark, I just edit the sources.list and see what apt proposes to do.

I often skip upgrade chiark because I tend to defer risky-looking upgrades partly in the hope of others fixing the bugs while I wait :-), and partly just because change is disruptive and amortising it is very helpful both to me and my users. I have some records of chiark’s upgrades from my announcements to users. As well as the recent “skip skip up cross grade, direct”, I definitely did a skip upgrade of chiark from squeeze (Debian 6) to jessie (Debian 8). It appears that the previous skip upgrade on chiark was rex (Debian 1.2) to hamm (Debian 2.0).

I don’t think it’s usual for me to choose to do a multi-release upgrade the “officially supported” way, in two (or more) stages, on a server. I have done that on systems with a GUI desktop setup, but even then I usually skip the intermediate reboot(s).

When to skip upgrade (and what precautions to take)

I’m certainly not saying that everyone ought to be doing this routinely. Most users with a Debian install that is older than oldstable probably ought to reinstall it, or do the two-stage upgrade.

Skip upgrading almost always runs into some kind of trouble (albeit, usually trouble that isn’t particularly hard to fix if you know what you’re doing).

However, officially supported non-skip upgrades go wrong too. Doing a two-or-more-releases upgrade via the intermediate releases can expose you to significant bugs in the intermediate releases, which were later fixed. Because Debian’s users and downstreams are cautious, and Debian itself can be slow, it is common for bugs to appear for one release and then be fixed only in the next. Paradoxically, this seems to be especially true with the kind of big and scary changes where you’d naively think the upgrade mechanisms would break if you skipped the release where the change first came in.

I would not recommend a skip upgrade to someone who is not a competent Debian administrator, with good familiarity with Debian package management, including use of dpkg directly to fix things up. You should have a mental toolkit of manual bug workaround techniques. I always should make sure that I have rescue media (and in the case of a remote system, full remote access including ability to boot a different image), although I don’t often need it.

And, when considering a skip upgrade, you should be aware of the major changes that have occurred in Debian.

Skip upgrading is more likely to be a good idea with a complex and highly customised system: a fairly vanilla install is not likely to encounter problems during a two-stage update. (And, a vanilla system can be “upgraded” by reinstalling.)

I haven’t recently skip upgraded a laptop or workstation. I doubt I would attempt it; modern desktop software seems to take a much harder line about breaking things that are officially unsupported, and generally trying to force everyone into the preferred mold.

A request to Debian maintainers

I would like to encourage Debian maintainers to defer removing upgrade compatibility machinery until it is actually getting in the way, or has become hazardous, or many years obsolete.

Examples of the kinds of things which it would be nice to keep, and which do not usually cause much inconvenience to retain, are dependency declarations (particularly, alternatives), and (many) transitional fragments in maintainer scripts.

If you find yourself needing to either delete some compatibility feature, or refactor/reorganise it, I think it is probably best to delete it. If you modify it significantly, the resulting thing (which won’t be tested until someone uses it in anger) is quite likely to have bugs which make it go wrong more badly (or, more confusingly) than the breakage that would happen without it.

Obviously this is all a judgement call.

I’m not saying Debian should formally “support” skip upgrades, to the extent of (further) slowing down important improvements. Nor am I asking for any change to the routine approach to (for example) transitional packages (i.e. the technique for ensuring continuity of installation when a package name changes).

We try to make release upgrades work perfectly; but skip upgrades don’t have to work perfectly to be valuable. Retaining compatibility code can also make it easier to provide official backports, and it probably helps downstreams with different release schedules.

The fact that maintainers do in practice often defer removing compatibility code provides useful flexibility and options to at least some people. So it would be nice if you’d at least not go out of your way to break it.

Building on older releases

I would also like to encourage maintainers to provide source packages in Debian unstable that will still build on older releases, where this isn’t too hard and the resulting binaries might be basically functional.

Speaking personally, it’s not uncommon for me to rebuild packages from unstable and install them on much older releases. This is another thing that is not officially supported, but which often works well.

I’m not saying to contort your build system, or delay progress. You’ll definitely want to depend on a recentish debhelper. But, for example, retaining old build-dependency alternatives is nice. Retaining them doesn’t constitute a promise that it works - it just makes life slightly easier for someone who is going off piste.

If you know you have users on multiple distros or multiple releases, and wish to fully support them, you can go further, of course. Many of my own packages are directly buildable, or even directly installable, on older releases.

Comments

[cjwatson]

One thing I think may help over time here (counterintuitively) is Jelmer's work on having automation that makes MRs to remove (build-)dependency constraints that are no longer needed on oldstable or newer. Because the threshold is set to oldstable, it shouldn't break skip-upgrades. Because the automation exists, I rarely find it worth my time any more to manually remove obsolete version requirements, since I can trust that the automation will do it for me and that's less work.

I was already generally being quite conservative about removing upgrade affordances, but if there are developers who were inclined to be more aggressive but who are similarly lazy about letting automation deal with stuff for them (as I suspect there are), then the net effect ought to be that fewer constraints get removed prematurely.