Ian Jackson

tag2upload in the first month of forky

2025-09-14T15:36:17Z

tl;dr: tag2upload (beta) is going well so far, and is already handling around one in 13 uploads to Debian.

Introduction and some stats
Recent UI/UX improvements
Why we are still in beta
- Retrying on Salsa-side failures
Other notable ongoing work
Common problems
- Reuse of version numbers, and attempts to re-tag
- Discrepancies between git and orig tarballs
Get involved

Introduction and some stats

We announced tag2upload’s open beta in mid-July. That was in the middle of the the freeze for trixie, so usage was fairly light until the forky floodgates opened.

Since then the service has successfully performed 637 uploads, of which 420 were in the last 32 days. That’s an average of about 13 per day. For comparison, during the first half of September up to today there have been 2475 uploads to unstable. That’s about 176/day.

So, tag2upload is already handling around 7.5% of uploads. This is very gratifying for a service which is advertised as still being in beta!

Sean and I are very pleased both with the uptake, and with the way the system has been performing.

Recent UI/UX improvements

During this open beta period we have been hard at work. We have made many improvements to the user experience.

Current git-debpush in forky, or trixie-backports, is much better at detecting various problems ahead of time.

When uploads do fail on the service the emailed error reports are now more informative. For example, anomalies involving orig tarballs, which by definition can’t be detected locally (since one point of tag2upload is not to have tarballs locally) now generally result in failure reports containing a diffstat, and instructions for a local repro.

Why we are still in beta

There are a few outstanding work items that we currently want to complete before we declare the end of the beta.

Retrying on Salsa-side failures

The biggest of these is that the service should be able to retry when Salsa fails. Sadly, Salsa isn’t wholly reliable, and right now if it breaks when the service is trying to handle your tag, your upload can fail.

We think most of these failures could be avoided. Implementing retries is a fairly substantial task, but doesn’t pose any fundamental difficulties. We’re working on this right now.

Other notable ongoing work

We want to support pristine-tar, so that pristine-tar users can do a new upstream release. Andrea Pappacoda is working on that with us. See #1106071. (Note that we would generally recommend against use of pristine-tar within Debian. But we want to support it.)

We have been having conversations with Debusine folks about what integration between tag2upload and Debusine would look like. We’re making some progress there, but a lot is still up in the air.

We are considering how best to provide tag2upload pre-checks as part of Salsa CI. There are several problems detected by the tag2upload service that could be detected by Salsa CI too, but which can’t be detected by git-debpush.

Common problems

We’ve been monitoring the service and until very recently we have investigated every service-side failure, to understand the root causes. This has given us insight into the kinds of things our users want, and the kinds of packaging and git practices that are common. We’ve been able to improve the system’s handling of various anomalies and also improved the documentation.

Right now our failure rate is still rather high, at around 7%. Partly this is because people are trying out the system on packages that haven’t ever seen git tooling with such a level of rigour.

There are two classes of problem that are responsible for the vast majority of the failures that we’re still seeing:

Reuse of version numbers, and attempts to re-tag

tag2upload, like git (and like dgit), hates it when you reuse a version number, or try to pretend that a (perhaps busted) release never happened.

git tags aren’t namespaced, and tend to spread about promiscuously. So replacing a signed git tag, with a different tag of the same name, is a bad idea. More generally, reusing the same version number for a different (signed!) package is poor practice. Likewise, it’s usually a bad idea to remove changelog entries for versions which were actually released, just because they were later deemed improper.

We understand that many Debian contributors have gotten used to this kind of thing. Indeed, tools like dcut encourage it. It does allow you to make things neat-looking, even if you’ve made mistakes - but really it does so by covering up those mistakes!

The bottom line is that tag2upload can’t support such history-rewriting. If you discover a mistake after you’ve signed the tag, please just burn the version number and add a new changelog stanza.

One bonus of tag2upload’s approach is that it will discover if you are accidentally overwriting an NMU, and report that as an error.

Discrepancies between git and orig tarballs

tag2upload promises that the source package that it generates corresponds precisely to the git tree you tag and sign.

Orig tarballs make this complicated. They aren’t present on your laptop when you git-debpush. When you’re not uploading a new upstream version, the tag2upload service reuses existing orig tarballs from the archive. If your git and the archive’s orig don’t agree, the tag2upload service will report an error, rather than upload a package with contents that differ from your git tag.

With the most common Debian workflows, everything is fine:

If you base everything on upstream git, and make your orig tarballs with git archive (or git deborig), your orig tarballs are the same as the git, by construction. We recommend usually ignoring upstream tarballs: most upstreams work in git, and their tarballs can contain weirdness that we don’t want. (At worst, the tarball can contain an attack that isn’t visible in git, as with xz!)

Alternatively, if you use gbp import-orig, the differences (including an attack like Jia Tan’s) are imported into git for you. Then, once again, your git and the orig tarball will correspond.

But there are other workflows where this correspondence may not hold. Those workflows are hazardous, because the thing you’re probably working with locally for your routine development is the git view. Then, when you upload, your work is transplanted onto the orig tarball, which might be quite different - so what you upload isn’t what you’ve been working on!

This situation is detected by tag2upload, precisely because tag2upload checks that it’s keeping its promise: the source package is identical to the git view. (dgit push makes the same promise.)

Get involved

Of course the easiest way to get involved is to start using tag2upload.

We would love to have more contributors. There are some easy tasks to get started with, in bugs we’ve tagged “newcomer” — mostly UX improvements such as detecting certain problems earlier, in git-debpush.

More substantially, we are looking for help with sbuild: we’d like it to be able to work directly from git, rather than needing to build source packages: #868527.

comments

Free Software, internal politics, and governance

2025-05-01T22:03:17Z

There is a thread of opinion in some Free Software communities, that we shouldn’t be doing “politics”, and instead should just focus on technology.

But that’s impossible. This approach is naive, harmful, and, ultimately, self-defeating, even on its own narrow terms.

Today I’m talking about small-p politics
Many people working together always entails politics
If you won’t do politics, politics will do you
- If you don’t see the politics, it’s still happening
Conclusions

Today I’m talking about small-p politics

In this article I’m using “politics” in the very wide sense: us humans managing our disagreements with each other.

I’m not going to talk about culture wars, woke, racism, trans rights, and so on. I am not going to talk about how Free Software has always had explicitly political goals; or how it’s impossible to be neutral because choosing not to take a stand is itself to take a stand.

Those issues are all are important and Free Software definitely must engage with them. Many of the points I make are applicable there too. But those are not my focus today.

Today I’m talking in more general terms about politics, power, and governance.

Many people working together always entails politics

Computers are incredibly complicated nowadays. Making software is a joint enterprise. Even if an individual program has only a single maintainer, it fits into an ecosystem of other software, maintained by countless other developers. Larger projects can have thousands of maintainers and hundreds of thousands of contributors.

Humans don’t always agree about everything. This is natural. Indeed, it’s healthy: to write the best code, we need a wide range of knowledge and experience.

When we can’t come to agreement, we need a way to deal with that: a way that lets us still make progress, but also leaves us able to work together afterwards. A way that feels OK for everyone.

Providing a framework for disagreement is the job of a governance system. The rules say which people make which decisions, who must be consulted, how the decisions are made, and, how, if any, they can be reviewed.

This is all politics.

Consensus is great but always requiring it is harmful

Ideally a discussion will converge to a synthesis that satisfies everyone, or at least a consensus.

When consensus can’t be achieved, we can hope for compromise: something everyone can live with. Compromise is achieved through negotiation.

If every decision requires consensus, then the proponents of any wide-ranging improvement have an almost insurmountable hurdle: those who are favoured by the status quo and find it convenient can always object. So there will never be consensus for change. If there is any objection at all, no matter how ill-founded, the status quo will always win.

This is where governance comes in.

Governance is like backups: we need to practice it

Governance processes are the backstop for when discussions, and then negotiations, fail, and people still don’t see eye to eye.

In a healthy community, everyone needs to know how the governance works and what the rules are. The participants need to accept the system’s legitimacy. Everyone, including the losing side, must be prepared to accept and implement (or, at least not obstruct) whatever the decision is, and hopefully live with it and stay around.

That means we need to practice our governance processes. We can’t just leave them for the day we have a huge and controversial decision to make. If we do that, then when it comes to the crunch we’ll have toxic rows where no-one can agree the rules; where determined people bend the rules to fit their outcome; and where afterwards people feel like the whole thing was horrible and unfair.

So our decisionmaking bodies and roles need to be making decisions, as a matter of routine, and we need to get used to that.

First-line decisionmaking bodies should be making decisions frequently. Last-line appeal mechanisms (large-scale votes, for example) are naturally going to be exercised more rarely, but they must happen, be seen as legitimate, and their outcomes must be implemented in full.

Governance should usually be routine and boring

When governance is working well it’s quite boring.

People offer their input, and are heard. Angles are debated, and concerns are addressed. If agreement still isn’t reached, the committee, or elected leader, makes a decision.

Hopefully everyone thinks the leadership is legitimate, and that it properly considered and heard their arguments, and made the decision for good reasons.

Hopefully the losing side can still get their work done (and make their own computer work the way they want); so while they will be disappointed, they can live with the outcome.

Many human institutions manage this most of the time. It does take some knowledge about principles of governance, and ideally some experience.

Governance means deciding, not just mediating

By making decisions I mean exercising their authority to rule on an actual disagreement: one that wasn’t resolved by debate or negotiation. Governance processes by definition involve deciding, not just mediating. It’s not governance if we’re advising or cajoling: in that case, we’re back to demanding consensus. Governance is necessary precisely when consensus is not achieved.

If the governance systems are to mean anything, they must be able to (over)rule; that means (over)ruling must be normal and accepted.

Otherwise, when the we need to overrule, we’ll find that we can’t, because we lack the collective practice.

To be legitimate (and seen as legitimate) decisions must usually be made based on the merits, not on participants’ status, and not only on process questions.

On the autonomy of the programmer

Many programmers seem to find the very concept of governance, and binding decisionmaking, deeply uncomfortable.

Ultimately, it means sometimes overruling someone’s technical decision. As programmers and maintainers we naturally see how this erodes our autonomy.

But we have all seen projects where the maintainers are unpleasant, obstinate, or destructive. We have all found this frustrating. Software is all interconnected, and one programmer’s bad decisions can cause problems for many of the rest of us. We exasperate, “why won’t they just do the right thing”. This is futile. People have never “just”ed and they’re not going to start “just”ing now. So often the boot is on the other foot.

More broadly, as software developers, we have a responsibility to our users, and a duty to write code that does good rather than ill in the world. We ought to be accountable. (And not just to capitalist bosses!)

Governance mechanisms are the answer.

(No, forking anything but the smallest project is very rarely a practical answer.)

Mitigate the consequences of decisions — retain flexibility

In software, it is often possible to soften the bad social effects of a controversial decision, by retaining flexibility. With a bit of extra work, we can often provide hooks, non-default configuration options, or plugin arrangements.

If we can convert the question from “how will the software always behave” into merely “what should the default be”, we can often save ourselves a lot of drama.

So it is often worth keeping even suboptimal or untidy features or options, if people want to use them and are willing to maintain them.

There is a tradeoff here, of course. But Free Software projects often significantly under-value the social benefits of keeping everyone happy. Wrestling software — even crusty or buggy software — is a lot more fun than having unpleasant arguments.

But don’t do decisionmaking like a corporation

Many programmers’ experience of formal decisionmaking is from their boss at work. But corporations are often a very bad example.

They typically don’t have as much trouble actually making decisions, but the actual decisions are often terrible, and not just because corporations’ goals are often bad.

You get to be a decisionmaker in a corporation by spouting plausible nonsense, sounding confident, buttering up the even-more-vacuous people further up the chain, and sometimes by sabotaging your rivals. Corporate senior managers are hardly ever held accountable — typically the effects of their tenure are only properly felt well after they’ve left to mess up somewhere else.

We should select our leaders more wisely, and base decisions on substance.

If you won’t do politics, politics will do you

As a participant in a project, or a society, you can of course opt out of getting involved in politics.

You can opt out of learning how to do politics generally, and opt out of understanding your project’s governance structures. You can opt out of making judgements about disputed questions, and tell yourself “there’s merit on both sides”.

You can hate politicians indiscriminately, and criticise anyone you see doing politics.

If you do this, then you are abdicating your decisionmaking authority, to those who are the most effective manipulators, or the most committed to getting their way. You’re tacitly supporting the existing power bases. You’re ceding power to the best liars, to those with the least scruples, and to the people who are most motivated by dominance. This is precisely the opposite of what you wanted.

If enough people won’t do politics, and hate anyone who does, your discussion spaces will be reduced to a battleground of only the hardiest and the most toxic.

If you don’t see the politics, it’s still happening

If your governance systems don’t work, then there is no effective redress against bad or even malicious decisions. Your roleholders and subteams are unaccountable power centres.

Power radically distorts every human relationship, and it takes great strength of character for an unaccountable power centre not to eventually become an unaccountable toxic cabal.

So if you have a reasonable sized community, but don’t see your formal governance systems working — people debating things, votes, leadership making explicit decisions — that doesn’t mean everything is fine, and all the decisions are great, and there’s no politics happening.

It just means that most of your community have given up on the official process. It also probably means that some parts of your project have formed toxic and unaccountable cabals. Those who won’t put up with that will leave.

The same is true if the only governance actions that ever happen are massive drama. That means that only the most determined victim of a bad decision, will even consider using such a process.

Conclusions

Respect and support the people who are trying to fix things with politics.
Be informed, and, where appropriate, involved.
If you are in a position of authority, be willing to exercise that authority. Do more than just mediating to try to get consensus.

comments

Rust is indeed woke

2025-03-28T12:59:43Z

Rust, and resistance to it in some parts of the Linux community, has been in my feed recently. One undercurrent seems to be the notion that Rust is woke (and should therefore be rejected as part of culture wars).

I’m going to argue that Rust, the language, is woke. So the opponents are right, in that sense. Of course, as ever, dissing something for being woke is nasty and fascist-adjacent.

Community
Technological values - particularly, compared to C/C++
Notes
Pithy conclusion

Community

The obvious way that Rust may seem woke is that it has the trappings, and many of the attitudes and outcomes, of a modern, nice, FLOSS community. Rust certainly does better than toxic environments like the Linux kernel, or Debian. This is reflected in a higher proportion of contributors from various kinds of minoritised groups. But Rust is not outstanding in this respect. It certainly has its problems. Many other projects do as well or better.

And this is well-trodden ground. I have something more interesting to say:

Technological values - particularly, compared to C/C++

Rust is woke technology that embodies a woke understanding of what it means to be a programming language.

Ostensible values

Let’s start with Rust’s strapline:

A language empowering everyone to build reliable and efficient software.

Surprisingly, this motto is not mere marketing puff. For Rustaceans, it is a key goal which strongly influences day-to-day decisions (big and small).

Empowering everyone is a key aspect of this, which aligns with my own personal values. In the Rust community, we care about empowerment. We are trying to help liberate our users. And we want to empower everyone because everyone is entitled to technological autonomy. (For a programming language, empowering individuals means empowering their communities, of course.)

This is all very airy-fairy, but it has concrete consequences:

Attitude to the programmer’s mistakes

In Rust we consider it a key part of our job to help the programmer avoid mistakes; to limit the consequences of mistakes; and to guide programmers in useful directions.

If you write a bug in your Rust program, Rust doesn’t blame you. Rust asks “how could the compiler have spotted that bug”.

This is in sharp contrast to C (and C++). C nowadays is an insanely hostile programming environment. A C compiler relentlessly scours your program for any place where you may have violated C’s almost incomprehensible rules, so that it can compile your apparently-correct program into a buggy executable. And then the bug is considered your fault.

These aren’t just attitudes implicitly embodied in the software. They are concrete opinions expressed by compiler authors, and also by language proponents. In other words:

Rust sees programmers writing bugs as a systemic problem, which must be addressed by improvements to the environment and the system. The toxic parts of the C and C++ community see bugs as moral failings by individual programmers.

Sound familiar?

The ideology of the hardcore programmer

Programming has long suffered from the myth of the “rockstar”. Silicon Valley techbro culture loves this notion.

In reality, though, modern information systems are far too complicated for a single person. Developing systems is a team sport. Nontechnical, and technical-adjacent, skills are vital: clear but friendly communication; obtaining and incorporating the insights of every member of your team; willingness to be challenged. Community building. Collaboration. Governance.

The hardcore C community embraces the rockstar myth: they imagine that a few super-programmers (or super-reviewers) are able to spot bugs, just by being so brilliant. Of course this doesn’t actually work at all, as we can see from the atrocious bugfest that is the Linux kernel.

These “rockstars” want us to believe that there is a steep hierarchy in programmming; that they are at the top of this hierarchy; and that being nice isn’t important.

Sound familiar?

Memory safety as a power struggle

Much of the modern crisis of software reliability arises from memory-unsafe programming languages, mostly C and C++.

Addressing this is a big job, requiring many changes. This threatens powerful interests; notably, corporations who want to keep shipping junk. (See also, conniptions over the EU Product Liability Directive.)

The harms of this serious problem mostly fall on society at large, but the convenience of carrying on as before benefits existing powerful interests.

Sound familiar?

Memory safety via Rust as a power struggle

Addressing this problem via Rust is a direct threat to the power of established C programmers such as gatekeepers in the Linux kernel. Supplanting C means they will have to learn new things, and jostle for status against better Rustaceans, or be replaced. More broadly, Rust shows that it is practical to write fast, reliable, software, and that this does not need (mythical) “rockstars”.

So established C programmer “experts” are existing vested interests, whose power is undermined by (this approach to) tackling this serious problem.

Sound familiar?

Notes

This is not a RIIR manifesto

I’m not saying we should rewrite all the world’s C in Rust. We should not try to do that.

Rust is often a good choice for new code, or when a rewrite or substantial overhaul is needed anyway. But we’re going to need other techniques to deal with all of our existing C. CHERI is a very promising approach. Sandboxing, emulation and automatic translation are other possibilities. The problem is a big one and we need a toolkit, not a magic bullet.

But as for Linux: it is a scandal that substantial new drivers and subsystems are still being written in C. We could have been using Rust for new code throughout Linux years ago, and avoided very many bugs. Those bugs are doing real harm. This is not OK.

Disclosure

I first learned C from K&R I in 1989. I spent the first three decades of my life as a working programmer writing lots and lots of C. I’ve written C++ too. I used to consider myself an expert C programmer, but nowadays my C is a bit rusty and out of date. Why is my C rusty? Because I found Rust, and immediately liked and adopted it (despite its many faults).

I like Rust because I care that the software I write actually works: I care that my code doesn’t do harm in the world.

On the meaning of “woke”

The original meaning of “woke” is something much more specific, to do with racism. For the avoidance of doubt, I don’t think Rust is particularly antiracist.

I’m using “woke” (like Rust’s opponents are) in the much broader, and now much more prevalent, culture wars sense.

Pithy conclusion

If you’re a senior developer who knows only C/C++, doesn’t want their authority challenged, and doesn’t want to have to learn how to write better software, you should hate Rust.

Also you should be fired.

Edited 2025-03-28 17:10 UTC to fix minor problems and add a new note about the meaning of the word "woke".

comments

derive-deftly 1.0.0 - Rust derive macros, the easy way

2025-02-11T21:14:07Z

derive-deftly 1.0 is released.

derive-deftly is a template-based derive-macro facility for Rust. It has been a great success. Your codebase may benefit from it too!

Rust programmers will appreciate its power, flexibility, and consistency, compared to macro_rules; and its convenience and simplicity, compared to proc macros.

Programmers coming to Rust from scripting languages will appreciate derive-deftly’s convenient automatic code generation, which works as a kind of compile-time introspection.

Rust’s two main macro systems
- macro_rules!
- proc macros
derive-deftly to the rescue
- Example
- Special-purpose derive macros are now worthwhile!
Stability without stagnation

Rust’s two main macro systems

I’m often a fan of metaprogramming, including macros. They can help remove duplication and flab, which are often the enemy of correctness.

Rust has two macro systems. derive-deftly offers much of the power of the more advanced (proc_macros), while beating the simpler one (macro_rules) at its own game for ease of use.

(Side note: Rust has at least three other ways to do metaprogramming: generics; build.rs; and, multiple module inclusion via #[path=]. These are beyond the scope of this blog post.)

`macro_rules!`

macro_rules! aka “pattern macros”, “declarative macros”, or sometimes “macros by example” are the simpler kind of Rust macro.

They involve writing a sort-of-BNF pattern-matcher, and a template which is then expanded with substitutions from the actual input. If your macro wants to accept comma-separated lists, or other simple kinds of input, this is OK. But often we want to emulate a #[derive(...)] macro: e.g., to define code based on a struct, handling each field. Doing that with macro_rules is very awkward:

macro_rules!’s pattern language doesn’t have a cooked way to match a data structure, so you have to hand-write a matcher for Rust syntax, in each macro. Writing such a matcher is very hard in the general case, because macro_rules lacks features for matching important parts of Rust syntax (notably, generics). (If you really need to, there’s a horrible technique as a workaround.)

And, the invocation syntax for the macro is awkward: you must enclose the whole of the struct in my_macro! { }. This makes it hard to apply more than one macro to the same struct, and produces rightward drift.

Enclosing the struct this way means the macro must reproduce its input - so it can have bugs where it mangles the input, perhaps subtly. This also means the reader cannot be sure precisely whether the macro modifies the struct itself. In Rust, the types and data structures are often the key places to go to understand a program, so this is a significant downside.

macro_rules also has various other weird deficiencies too specific to list here.

Overall, compared to (say) the C preprocessor, it’s great, but programmers used to the power of Lisp macros, or (say) metaprogramming in Tcl, will quickly become frustrated.

proc macros

Rust’s second macro system is much more advanced. It is a fully general system for processing and rewriting code. The macro’s implementation is Rust code, which takes the macro’s input as arguments, in the form of Rust tokens, and returns Rust tokens to be inserted into the actual program.

This approach is more similar to Common Lisp’s macros than to most other programming languages’ macros systems. It is extremely powerful, and is used to implement many very widely used and powerful facilities. In particular, proc macros can be applied to data structures with #[derive(...)]. The macro receives the data structure, in the form of Rust tokens, and returns the code for the new implementations, functions etc.

This is used very heavily in the standard library for basic features like #[derive(Debug)] and Clone, and for important libraries like serde and strum.

But, it is a complete pain in the backside to write and maintain a proc_macro.

The Rust types and functions you deal with in your macro are very low level. You must manually handle every possible case, with runtime conditions and pattern-matching. Error handling and recovery is so nontrivial there are macro-writing libraries and even more macros to help. Unlike a Lisp codewalker, a Rust proc macro must deal with Rust’s highly complex syntax. You will probably end up dealing with syn, which is a complete Rust parsing library, separate from the compiler; syn is capable and comprehensive, but a proc macro must still contain a lot of often-intricate code.

There are build/execution environment problems. The proc_macro code can’t live with your application; you have to put the proc macros in a separate cargo package, complicating your build arrangements. The proc macro package environment is weird: you can’t test it separately, without jumping through hoops. Debugging can be awkward. Proper tests can only realistically be done with the help of complex additional tools, and will involve a pinned version of Nightly Rust.

derive-deftly to the rescue

derive-deftly lets you use a write a #[derive(...)] macro, driven by a data structure, without wading into any of that stuff.

Your macro definition is a template in a simple syntax, with predefined $-substitutions for the various parts of the input data structure.

Example

Here’s a real-world example from a personal project:

define_derive_deftly! {
    export UpdateWorkerReport:
    impl $ttype {
        pub fn update_worker_report(&self, wr: &mut WorkerReport) {
            $(
                ${when fmeta(worker_report)}
                wr.$fname = Some(self.$fname.clone()).into();
            )
        }
    }
}

#[derive(Debug, Deftly, Clone)]
...
#[derive_deftly(UiMap, UpdateWorkerReport)]
pub struct JobRow {
    ...
    #[deftly(worker_report)]
    pub status: JobStatus,
    pub processing: NoneIsEmpty<ProcessingInfo>,
    #[deftly(worker_report)]
    pub info: String,
    pub duplicate_of: Option<JobId>,
}

This is a nice example, also, of how using a macro can avoid bugs. Implementing this update by hand without a macro would involve a lot of cut-and-paste. When doing that cut-and-paste it can be very easy to accidentally write bugs where you forget to update some parts of each of the copies:

    pub fn update_worker_report(&self, wr: &mut WorkerReport) {
        wr.status = Some(self.status.clone()).into();
        wr.info = Some(self.status.clone()).into();
    }

Spot the mistake? We copy status to info. Bugs like this are extremely common, and not always found by the type system. derive-deftly can make it much easier to make them impossible.

Special-purpose derive macros are now worthwhile!

Because of the difficult and cumbersome nature of proc macros, very few projects have site-specific, special-purpose #[derive(...)] macros.

The Arti codebase has no bespoke proc macros, across its 240kloc and 86 crates. (We did fork one upstream proc macro package to add a feature we needed.) I have only one bespoke, case-specific, proc macro amongst all of my personal Rust projects; it predates derive-deftly.

Since we have started using derive-deftly in Arti, it has become an important tool in our toolbox. We have 37 bespoke derive macros, done with derive-deftly. Of these, 9 are exported for use by downstream crates. (For comparison there are 176 macro_rules macros.)

In my most recent personal Rust project, I have 22 bespoke derive macros, done with with derive-deftly, and 19 macro_rules macros.

derive-deftly macros are easy and straightforward enough that they can be used as readily as macro_rules macros. Indeed, they are often clearer than a macro_rules macro.

Stability without stagnation

derive-deftly is already highly capable, and can solve many advanced problems.

It is mature software, well tested, with excellent documentation, comprising both comprehensive reference material and the walkthrough-structured user guide.

But declaring it 1.0 doesn’t mean that it won’t improve further.

Our ticket tracker has a laundry list of possible features. We’ll sometimes be cautious about committing to these, so we’ve added a beta feature flag, for opting in to less-stable features, so that we can prototype things without painting ourselves into a corner. And, we intend to further develop the Guide.

comments

derive-deftly is nearing 1.x - call for review/testing

2024-07-03T18:32:06Z

derive-deftly, the template-based derive-macro facility for Rust, has been a great success.

It’s coming up to time to declare a stable 1.x version. If you’d like to try it out, and have final comments / observations, now is the time.

Introduction to derive-deftly
Status
History
Plans - call for review/testing

Introduction to derive-deftly

Have you ever wished that you could that could write a new derive macro without having to mess with procedural macros?

You can!

derive-deftly lets you write a #[derive] macro, using a template syntax which looks a lot like macro_rules!:

use derive_deftly::{define_derive_deftly, Deftly};

define_derive_deftly! {
    ListVariants:

    impl $ttype {
        fn list_variants() -> Vec<&'static str> {
            vec![ $( stringify!( $vname ) , ) ]
        }
    }
}

#[derive(Deftly)]
#[derive_deftly(ListVariants)]
enum Enum {
    UnitVariant,
    StructVariant { a: u8, b: u16 },
    TupleVariant(u8, u16),
}

assert_eq!(
    Enum::list_variants(),
    ["UnitVariant", "StructVariant", "TupleVariant"],
);

Status

derive-deftly has a wide range of features, which can be used to easily write sophisticated and reliable derive macros. We’ve been using it in Arti, the Tor Project’s reimplementation of Tor in Rust, and we’ve found it very useful.

There is comprehensive reference documentation, and more discursive User Guide for a more gentle introduction. Naturally, everything is fully tested.

History

derive-deftly started out as a Tor Hackweek project. It used to be called derive-adhoc. But we renamed it because we found that many of the most interesting use cases were really not very ad-hoc at all.

Over the past months we’ve been ticking off our “1.0 blocker” tickets. We’ve taken the opportunity to improve syntax, terminology, and semantics. We hope we have now made the last breaking changes.

Plans - call for review/testing

In the near future, we plan to declare version 1.0. After 1.x, we intend to make breaking changes very rarely.

So, right now, we’d like last-minute feedback. Are there any wrinkles that need to be sorted out? Please file tickets or MRs on our gitlab. Ideally, anything which might imply breaking changes would be submitted on or before the 13th of August.

In the medium to long term, we have many ideas for how to make derive-deftly even more convenient, and even more powerful. But we are going to proceed cautiously, because we don’t want to introduce bad syntax or bad features, which will require difficult decisions in the future about forward compatibility.

comments

How to use Rust on Debian (and Ubuntu, etc.)

2024-03-21T20:05:46Z

tl;dr: Don’t just apt install rustc cargo. Either do that and make sure to use only Rust libraries from your distro (with the tiresome config runes below); or, just use rustup.

Don’t do the obvious thing; it’s never what you want
- Q. Download and run whatever code from the internet?
Option 1: WTF, no I don’t want curl|bash
Option 2: Biting the curl|bash bullet
- Privilege separation
OMG what a mess

Don’t do the obvious thing; it’s never what you want

Debian ships a Rust compiler, and a large number of Rust libraries.

But if you just do things the obvious “default” way, with apt install rustc cargo, you will end up using Debian’s compiler but upstream libraries, directly and uncurated from crates.io.

This is not what you want. There are about two reasonable things to do, depending on your preferences.

Q. Download and run whatever code from the internet?

The key question is this:

Are you comfortable downloading code, directly from hundreds of upstream Rust package maintainers, and running it ?

That’s what cargo does. It’s one of the main things it’s for. Debian’s cargo behaves, in this respect, just like upstream’s. Let me say that again:

Debian’s cargo promiscuously downloads code from crates.io just like upstream cargo.

So if you use Debian’s cargo in the most obvious way, you are still downloading and running all those random libraries. The only thing you’re avoiding downloading is the Rust compiler itself, which is precisely the part that is most carefully maintained, and of least concern.

Debian’s cargo can even download from crates.io when you’re building official Debian source packages written in Rust: if you run dpkg-buildpackage, the downloading is suppressed; but a plain cargo build will try to obtain and use dependencies from the upstream ecosystem. (“Happily”, if you do this, it’s quite likely to bail out early due to version mismatches, before actually downloading anything.)

Option 1: WTF, no I don’t want `curl|bash`

OK, but then you must limit yourself to libraries available within Debian. Each Debian release provides a curated set. It may or may not be sufficient for your needs. Many capable programs can be written using the packages in Debian.

But any upstream Rust project that you encounter is likely to be a pain to get working, unless their maintainers specifically intend to support this. (This is fairly rare, and the Rust tooling doesn’t make it easy.)

To go with this plan, apt install rustc cargo and put this in your configuration, in $HOME/.cargo/config.toml:

[source.debian-packages]
directory = "/usr/share/cargo/registry"
[source.crates-io]
replace-with = "debian-packages"

This causes cargo to look in /usr/share for dependencies, rather than downloading them from crates.io. You must then install the librust-FOO-dev packages for each of your dependencies, with apt.

This will allow you to write your own program in Rust, and build it using cargo build.

Option 2: Biting the `curl|bash` bullet

If you want to build software that isn’t specifically targeted at Debian’s Rust you will probably need to use packages from crates.io, not from Debian.

If you’re doing to do that, there is little point not using rustup to get the latest compiler. rustup’s install rune is alarming, but cargo will be doing exactly the same kind of thing, only worse (because it trusts many more people) and more hidden.

So in this case: do run the curl|bash install rune.

Hopefully the Rust project you are trying to build have shipped a Cargo.lock; that contains hashes of all the dependencies that they last used and tested. If you run cargo build --locked, cargo will only use those versions, which are hopefully OK.

And you can run cargo audit to see if there are any reported vulnerabilities or problems. But you’ll have to bootstrap this with cargo install --locked cargo-audit; cargo-audit is from the RUSTSEC folks who do care about these kind of things, so hopefully running their code (and their dependencies) is fine. Note the --locked which is needed because cargo’s default behaviour is wrong.

Privilege separation

This approach is rather alarming. For my personal use, I wrote a privsep tool which allows me to run all this upstream Rust code as a separate user.

That tool is nailing-cargo. It’s not particularly well productised, or tested, but it does work for at least one person besides me. You may wish to try it out, or consider alternative arrangements. Bug reports and patches welcome.

OMG what a mess

Indeed. There are large number of technical and social factors at play.

cargo itself is deeply troubling, both in principle, and in detail. I often find myself severely disappointed with its maintainers’ decisions. In mitigation, much of the wider Rust upstream community does takes this kind of thing very seriously, and often makes good choices. RUSTSEC is one of the results.

Debian’s technical arrangements for Rust packaging are quite dysfunctional, too: IMO the scheme is based on fundamentally wrong design principles. But, the Debian Rust packaging team is dynamic, constantly working the update treadmills; and the team is generally welcoming and helpful.

Sadly last time I explored the possibility, the Debian Rust Team didn’t have the appetite for more fundamental changes to the workflow (including, for example, changes to dependency version handling). Significant improvements to upstream cargo’s approach seem unlikely, too; we can only hope that eventually someone might manage to supplant it.

edited 2024-03-21 21:49 to add a cut tag

comments

Don’t use apt-get source; use dgit

2023-12-04T15:10:52Z

tl;dr:

If you are a Debian user who knows git, don’t work with Debian source packages. Don’t use apt source, or dpkg-source. Instead, use dgit and work in git.

Also, don’t use: “VCS” links on official Debian web pages, debcheckout, or Debian’s (semi-)official gitlab, Salsa. These are suitable for Debian experts only; for most people they can be beartraps. Instead, use dgit.

Struggling with Debian source packages?
Just use dgit
Objections

Struggling with Debian source packages?

A friend of mine recently asked for help on IRC. They’re an experienced Debian administrator and user, and were trying to: make a change to a Debian package; build and install and run binary packages from it; and record that change for their future self, and their colleagues. They ended up trying to comprehend quilt.

quilt is an ancient utility for managing sets of source code patches, from well before the era of modern version control. It has many strange behaviours and footguns. Debian’s ancient and obsolete tarballs-and-patches source package format (which I designed the initial version of in 1993) nowadays uses quilt, at least for most packages.

You don’t want to deal with any of this nonsense. You don’t want to learn quilt, and suffer its misbehaviours. You don’t want to learn about Debian source packages and wrestle dpkg-source.

Happily, you don’t need to.

Just use dgit

One of dgit’s main objectives is to minimise the amount of Debian craziness you need to learn. dgit aims to empower you to make changes to the software you’re running, conveniently and with a minimum of fuss.

You can use dgit to get the source code to a Debian package, as a git tree, with dgit clone (and dgit fetch). The git tree can be made into a binary package directly.

The only things you really need to know are:

By default dgit fetches from Debian unstable, the main work-in-progress branch. You may want something like dgit clone PACKAGE bookworm,-security (yes, with a comma).
You probably want to edit debian/changelog to make your packages have a different version number.
To build binaries, run dpkg-buildpackage -uc -b.
Debian package builds are often disastrously messsy: builds might modify source files; and the official debian/rules clean can be inadequate, or crazy. Always commit before building, and use git clean and git reset --hard instead of running clean rules from the package.

Don’t try to make a Debian source package. (Don’t read the dpkg-source manual!) Instead, to preserve and share your work, use the git branch.

dgit pull or dgit fetch can be used to get updates.

There is a more comprehensive tutorial, with example runes, in the dgit-user(7) manpage. (There is of course complete reference documentation, but you don’t need to bother reading it.)

Objections

But I don’t want to learn yet another tool

One of dgit’s main goals is to save people from learning things you don’t need to. It aims to be straightforward, convenient, and (so far as Debian permits) unsurprising.

So: don’t learn dgit. Just run it and it will be fine :-).

Shouldn’t I be using “official” Debian git repos?

Absolutely not.

Unless you are a Debian expert, these can be terrible beartraps. One possible outcome is that you might build an apparently working program but without the security patches. Yikes!

I discussed this in more detail in 2021 in another blog post plugging dgit.

Gosh, is Debian really this bad?

Yes. On behalf of the Debian Project, I apologise.

Debian is a very conservative institution. Change usually comes very slowly. (And when rapid or radical change has been forced through, the results haven’t always been pretty, either technically or socially.)

Sadly this means that sometimes much needed change can take a very long time, if it happens at all. But this tendency also provides the stability and reliability that people have come to rely on Debian for.

I’m a Debian maintainer. You tell me dgit is something totally different!

dgit is, in fact, a general bidirectional gateway between the Debian archive and git.

So yes, dgit is also a tool for Debian uploaders. You should use it to do your uploads, whenever you can. It’s more convenient and more reliable than git-buildpackage and dput runes, and produces better output for users. You too can start to forget how to deal with source packages!

A full treatment of this is beyond the scope of this blog post.

comments

DigiSpark (ATTiny85) - Arduino, C, Rust, build systems

2023-10-22T16:02:09Z

Recently I completed a small project, including an embedded microcontroller. For me, using the popular Arduino IDE, and C, was a mistake. The experience with Rust was better, but still very exciting, and not in a good way.

Here follows the rant.

Introduction
Arduino IDE
Writing C again
Rust on the DigiSpark
RIIR (Rewrite It In Rust)
An offer of help
Conclusions

Introduction

In a recent project (I’ll write about the purpose, and the hardware in another post) I chose to use a DigiSpark board. This is a small board with a USB-A tongue (but not a proper plug), and an ATTiny85 microcontroller, This chip has 8 pins and is quite small really, but it was plenty for my application. By choosing something popular, I hoped for convenient hardware, and an uncomplicated experience.

Convenient hardware, I got.

Arduino IDE

The usual way to program these boards is via an IDE. I thought I’d go with the flow and try that. I knew these were closely related to actual Arduinos and saw that the IDE package arduino was in Debian.

But it turns out that the Debian package’s version doesn’t support the DigiSpark. (AFAICT from the list it offered me, I’m not sure it supports any ATTiny85 board.) Also, disturbingly, its “board manager” seemed to be offering to “install” board support, suggesting it would download “stuff” from the internet and run it. That wouldn’t be acceptable for my main laptop.

I didn’t expect to be doing much programming or debugging, and the project didn’t have significant security requirements: the chip, in my circuit, has only a very narrow ability do anything to the real world, and no network connection of any kind. So I thought it would be tolerable to do the project on my low-security “video laptop”. That’s the machine where I’m prepared to say “yes” to installing random software off the internet.

So I went to the upstream Arduino site and downloaded a tarball containing the Arduino IDE. After unpacking that in /opt it ran and produced a pointy-clicky IDE, as expected. I had already found a 3rd-party tutorial saying I needed to add a magic URL (from the DigiSpark’s vendor) in the preferences. That indeed allowed it to download a whole pile of stuff. Compilers, bootloader clients, god knows what.

However, my tiny test program didn’t make it to the board. Half-buried in a too-small window was an error message about the board’s bootloader (“Micronucleus”) being too new.

The boards I had came pre-flashed with micronucleus 2.2. Which is hardly new, But even so the official Arduino IDE (or maybe the DigiSpark’s board package?) still contains an old version. So now we have all the downsides of curl|bash-ware, but we’re lacking the “it’s up to date” and “it just works” upsides.

Further digging found some random forum posts which suggested simply downloading a newer micronucleus and manually stuffing it into the right place: one overwrites a specific file, in the middle the heaps of stuff that the Arduino IDE’s board support downloader squirrels away in your home directory. (In my case, the home directory of the untrusted shared user on the video laptop,)

So, “whatever”. I did that. And it worked!

Having demo’d my ability to run code on the board, I set about writing my program.

Writing C again

The programming language offered via the Arduino IDE is C.

It’s been a little while since I started a new thing in C. After having spent so much of the last several years writing Rust. C’s primitiveness quickly started to grate, and the program couldn’t easily be as DRY as I wanted (Don’t Repeat Yourself, see Wilson et al, 2012, §4, p.6). But, I carried on; after all, this was going to be quite a small job.

Soon enough I had a program that looked right and compiled.

Before testing it in circuit, I wanted to do some QA. So I wrote a simulator harness that #included my Arduino source file, and provided imitations of the few Arduino library calls my program used. As an side advantage, I could build and run the simulation on my main machine, in my normal development environment (Emacs, make, etc.). The simulator runs confirmed the correct behaviour. (Perhaps there would have been some more faithful simulation tool, but the Arduino IDE didn’t seem to offer it, and I wasn’t inclined to go further down that kind of path.)

So I got the video laptop out, and used the Arduino IDE to flash the program. It didn’t run properly. It hung almost immediately. Some very ad-hoc debugging via led-blinking (like printf debugging, only much worse) convinced me that my problem was as follows:

Arduino C has 16-bit ints. My test harness was on my 64-bit Linux machine. C was autoconverting things (when building for the micrcocontroller). The way the Arduino IDE ran the compiler didn’t pass the warning options necessary to spot narrowing implicit conversions. Those warnings aren’t the default in C in general ~~because C compilers hate us all~~ for compatibility reasons.

I don’t know why those warnings are not the default in the Arduino IDE, but my guess is that they didn’t want to bother poor novice programmers with messages from the compiler explaining how their program is quite possibly wrong. After all, users don’t like error messages so we shouldn’t report errors. And novice programmers are especially fazed by error messages so it’s better to just let them struggle themselves with the arcane mysteries of undefined behaviour in C?

The Arduino IDE does offer a dropdown for “compiler warnings”. The default is None. Setting it to All didn’t produce anything about my integer overflow bugs. And, the output was very hard to find anyway because the “log” window has a constant stream of strange messages from javax.jmdns, with hex DNS packet dumps. WTF.

Other things that were vexing about the Arduino IDE: it has fairly fixed notions (which don’t seem to be documented) about how your files and directories ought to be laid out, and magical machinery for finding things you put “nearby” its “sketch” (as it calls them) and sticking them in its ear, causing lossage. It has a tendency to become confused if you edit files under its feet (e.g. with git checkout). It wasn’t really very suited to a workflow where principal development occurs elsewhere.

And, important settings such as the project’s clock speed, or even the target board, or the compiler warning settings to use weren’t stored in the project directory along with the actual code. I didn’t look too hard, but I presume they must be in a dotfile somewhere. This is madness.

Apparently there is an Arduino CLI too. But I was already quite exasperated, and I didn’t like the idea of going so far off the beaten path, when the whole point of using all this was to stay with popular tooling and share fate with others. (How do these others cope? I have no idea.)

As for the integer overflow bug:

I didn’t seriously consider trying to figure out how to control in detail the C compiler options passed by the Arduino IDE. (Perhaps this is possible, but not really documented?) I did consider trying to run a cross-compiler myself from the command line, with appropriate warning options, but that would have involved providing (or stubbing, again) the Arduino/DigiSpark libraries (and bugs could easily lurk at that interface).

Instead, I thought, “if only I had written the thing in Rust”. But that wasn’t possible, was it? Does Rust even support this board?

Rust on the DigiSpark

I did a cursory web search and found a very useful blog post by Dylan Garrett.

This encouraged me to think it might be a workable strategy. I looked at the instructions there. It seemed like I could run them via the privsep arrangement I use to protect myself when developing using upstream cargo packages from crates.io.

I got surprisingly far surprisingly quickly. It did, rather startlingly, cause my rustup to download a random recent Nightly Rust, but I have six of those already for other Reasons. Very quickly I got the “trinket” LED blink example, referenced by Dylan’s blog post, to compile. Manually copying the file to the video laptop allowed me to run the previously-downloaded micronucleus executable and successfully run the blink example on my board!

I thought a more principled approach to the bootloader client might allow a more convenient workflow. I found the upstream Micronucleus git releases and tags, and had a look over its source code, release dates, etc. It seemed plausible, so I compiled v2.6 from source. That was a success: now I could build and install a Rust program onto my board, from the command line, on my main machine. No more pratting about with the video laptop.

I had got further, more quickly, with Rust, than with the Arduino IDE, and the outcome and workflow was superior.

So, basking in my success, I copied the directory containing the example into my own project, renamed it, and adjusted the path references.

That didn’t work. Now it didn’t build. Even after I copied about .cargo/config.toml and rust-toolchain.toml it didn’t build, producing a variety of exciting messages, depending what precisely I tried. I don’t have detailed logs of my flailing: the instructions say to build it by cd’ing to the subdirectory, and, given that what I was trying to do was to not follow those instructions, it didn’t seem sensible to try to prepare a proper repro so I could file a ticket. I wasn’t optimistic about investigating it more deeply myself: I have some experience of fighting cargo, and it’s not usually fun. Looking at some of the build control files, things seemed quite complicated.

Additionally, not all of the crates are on crates.io. I have no idea why not. So, I would need to supply “local” copies of them anyway. I decided to just git subtree add the avr-hal git tree.

(That seemed better than the approach taken by the avr-hal project’s cargo template, since that template involve a cargo dependency on a foreign git repository. Perhaps it would be possible to turn them into path dependencies, but given that I had evidence of file-location-sensitive behaviour, which I didn’t feel like I wanted to spend time investigating, using that seems like it would possibly have invited more trouble. Also, I don’t like package templates very much. They’re a form of clone-and-hack: you end up stuck with whatever bugs or oddities exist in the version of the template which was current when you started.)

Since I couldn’t get things to build outside avr-hal, I edited the example, within avr-hal, to refer to my (one) program.rs file outside avr-hal, with a #[path] instruction. That’s not pretty, but it worked.

I also had to write a nasty shell script to work around the lack of good support in my nailing-cargo privsep tool for builds where cargo must be invoked in a deep subdirectory, and/or Cargo.lock isn’t where it expects, and/or the target directory containing build products is in a weird place. It also has to filter the output from cargo to adjust the pathnames in the error messages. Otherwise, running both cd A; cargo build and cd B; cargo build from a Makefile produces confusing sets of error messages, some of which contain filenames relative to A and some relative to B, making it impossible for my Emacs to reliably find the right file.

RIIR (Rewrite It In Rust)

Having got my build tooling sorted out I could go back to my actual program.

I translated the main program, and the simulator, from C to Rust, more or less line-by-line. I made the Rust version of the simulator produce the same output format as the C one. That let me check that the two programs had the same (simulated) behaviour. Which they did (after fixing a few glitches in the simulator log formatting).

Emboldened, I flashed the Rust version of my program to the DigiSpark. It worked right away!

RIIR had caused the bug to vanish. Of course, to rewrite the program in Rust, and get it to compile, it was necessary to be careful about the types of all the various integers, so that’s not so surprising. Indeed, it was the point. I was then able to refactor the program to be a bit more natural and DRY, and improve some internal interfaces. Rust’s greater power, compared to C, made those cleanups easier, so making them worthwhile.

However, when doing real-world testing I found a weird problem: my timings were off. Measured, the real program was too fast by a factor of slightly more than 2. A bit of searching (and searching my memory) revealed the cause: I was using a board template for an Adafruit Trinket. The Trinket has a clock speed of 8MHz. But the DigiSpark runs at 16.5MHz. (This is discussed in a ticket against one of the C/C++ libraries supporting the ATTiny85 chip.)

The Arduino IDE had offered me a choice of clock speeds. I have no idea how that dropdown menu took effect; I suspect it was adding prelude code to adjust the clock prescaler. But my attempts to mess with the CPU clock prescaler register by hand at the start of my Rust program didn’t bear fruit.

So instead, I adopted a bodge: since my code has (for code structure reasons, amongst others) only one place where it dealt with the underlying hardware’s notion of time, I simply changed my delay function to adjust the passed-in delay values, compensating for the wrong clock speed.

There was probably a more principled way. For example I could have (re)based my work on either of the two unmerged open MRs which added proper support for the DigiSpark board, rather than abusing the Adafruit Trinket definition. But, having a nearly-working setup, and an explanation for the behaviour, I preferred the narrower fix to reopening any cans of worms.

An offer of help

As will be obvious from this posting, I’m not an expert in dev tools for embedded systems. Far from it. This area seems like quite a deep swamp, and I’m probably not the person to help drain it. (Frankly, much of the improvement work ought to be done, and paid for, by hardware vendors.)

But, as a full Member of the Debian Project, I have considerable gatekeeping authority there. I also have much experience of software packaging, build systems, and release management. If anyone wants to try to improve the situation with embedded tooling in Debian, and is willing to do the actual packaging work. I would be happy to advise, and to review and sponsor your contributions.

An obvious candidate: it seems to me that micronucleus could easily be in Debian. Possibly a DigiSpark board definition could be provided to go with the arduino package.

Unfortunately, IMO Debian’s Rust packaging tooling and workflows are very poor, and the first of my suggestions for improvement wasn’t well received. So if you need help with improving Rust packages in Debian, please talk to the Debian Rust Team yourself.

Conclusions

Embedded programming is still rather a mess and probably always will be.

Embedded build systems can be bizarre. Documentation is scant. You’re often expected to download “board support packages” full of mystery binaries, from the board vendor (or others).

Dev tooling is maddening, especially if aimed at novice programmers. You want version control? Hermetic tracking of your project’s build and install configuration? Actually to be told by the compiler when you write obvious bugs? You’re way off the beaten track.

As ever, Free Software is under-resourced and the maintainers are often busy, or (reasonably) have other things to do with their lives.

All is not lost

Rust can be a significantly better bet than C for embedded software:

The Rust compiler will catch a good proportion of programming errors, and an experienced Rust programmer can arrange (by suitable internal architecture) to catch nearly all of them. When writing for a chip in the middle of some circuit, where debugging involves staring an LED or a multimeter, that’s precisely what you want.

Rust embedded dev tooling was, in this case, considerably better. Still quite chaotic and strange, and less mature, perhaps. But: significantly fewer mystery downloads, and significantly less crazy deviations from the language’s normal build system. Overall, less bad software supply chain integrity.

The ATTiny85 chip, and the DigiSpark board, served my hardware needs very well. (More about the hardware aspects of this project in a future posting.)

comments

DKIM: rotate and publish your keys

2023-08-15T00:15:43Z

If you are an email system administrator, you are probably using DKIM to sign your outgoing emails. You should be rotating the key regularly and automatically, and publishing old private keys. I have just released dkim-rotate 1.0; dkim-rotate is a tool to do this key rotation and publication.

If you are an email user, your email provider ought to be doing this. If this is not done, your emails are “non-repudiable”, meaning that if they are leaked, anyone (eg, journalists, haters) can verify that they are authentic, and prove that to others. This is not desirable (for you).

Non-repudiation of emails is undesirable

This problem was described at some length in Matthew Green’s article Ok Google: please publish your DKIM secret keys.

Avoiding non-repudiation sounds a bit like lying. After all, I’m advising creating a situation where some people can’t verify that something is true, even though it is. So I’m advocating casting doubt. Crucially, though, it’s doubt about facts that ought to be private. When you send an email, that’s between you and the recipient. Normally you don’t intend for anyone, anywhere, who happens to get a copy, to be able to verify that it was really you that sent it.

In practical terms, this verifiability has already been used by journalists to verify stolen emails. Associated Press provide a verification tool.

Advice for all email users

As a user, you probably don’t want your emails to be non-repudiable. (Other people might want to be able to prove you sent some email, but your email system ought to serve your interests, not theirs.)

So, your email provider ought to be rotating their DKIM keys, and publishing their old ones. At a rough guess, your provider probably isn’t :-(.

How to tell by looking at email headers

A quick and dirty way to guess is to have a friend look at the email headers of a message you sent. (It is important that the friend uses a different email provider, since often DKIM signatures are not applied within a single email system.)

If your friend sees a DKIM-Signature header then the message is DKIM signed. If they don’t, then it wasn’t. Most email traversing the public internet is DKIM signed nowadays; so if they don’t see the header probably they’re not looking using the right tools, or they’re actually on the same email system as you.

In messages signed by a system running dkim-rotate, there will also be a header about the key rotation, to notify potential verifiers of the situation. Other systems that avoid non-repudiation-through-DKIM might do something similar. dkim-rotate’s header looks like this:

DKIM-Signature-Warning: NOTE REGARDING DKIM KEY COMPROMISE
 https://www.chiark.greenend.org.uk/dkim-rotate/README.txt
 https://www.chiark.greenend.org.uk/dkim-rotate/ae/aeb689c2066c5b3fee673355309fe1c7.pem

But an email system might do half of the job of dkim-rotate: regularly rotating the key would cause the signatures of old emails to fail to verify, which is a good start. In that case there probably won’t be such a header.

Testing verification of new and old messages

You can also try verifying the signatures. This isn’t entirely straightforward, especially if you don’t have access to low-level mail tooling. Your friend will need to be able to save emails as raw whole headers and body, un-decoded, un-rendered.

If your friend is using a traditional Unix mail program, they should save the message as an mbox file. Otherwise, ProPublica have instructions for attaching and transferring and obtaining the raw email. (Scroll down to “How to Check DKIM and ARC”.)

Checking that recent emails are verifiable

Firstly, have your friend test that they can in fact verify a DKIM signature. This will demonstrate that the next test, where the verification is supposed to fail, is working properly and fails for the right reasons.

Send your friend a test email now, and have them do this on a Linux system:

    # save the message as test-email.mbox
    apt install libmail-dkim-perl # or equivalent on another distro
    dkimproxy-verify <test-email.mbox

You should see output containing something like this:

    originator address: ijackson@chiark.greenend.org.uk
    signature identity: @chiark.greenend.org.uk
    verify result: pass
    ...

If the output ontains verify result: fail (body has been altered) then probably your friend didn’t manage to faithfully save the unalterered raw message.

Checking old emails cannot be verified

When you both have that working, have your friend find an older email of yours, from (say) month ago. Perform the same steps.

Hopefully they will see something like this:

    originator address: ijackson@chiark.greenend.org.uk
    signature identity: @chiark.greenend.org.uk
    verify result: fail (bad RSA signature)

or maybe

    verify result: invalid (public key: not available)

This indicates that this old email can no longer be verified. That’s good: it means that anyone who steals a copy, can’t verify it either. If it’s leaked, the journalist who receives it won’t know it’s genuine and unmodified; they should then be suspicious.

If your friend sees verify result: pass, then they have verified that that old email of yours is genuine. Anyone who had a copy of the mail can do that. This is good for email thieves, but not for you.

For email admins: announcing dkim-rotate 1.0

I have been running dkim-rotate 0.4 on my infrastructure, since last August. and I had entirely forgotten about it: it has run flawlessly for a year. I was reminded of the topic by seeing DKIM in other blog posts. Obviously, it is time to decreee that dkim-rotate is 1.0.

If you’re a mail system administrator, your users are best served if you use something like dkim-rotate. The package is available in Debian stable, and supports Exim out of the box, but other MTAs should be easy to support too, via some simple ad-hoc scripting.

Limitation of this approach

Even with this key rotation approach, emails remain nonrepudiable for a short period after they’re sent - typically, a few days.

Someone who obtains a leaked email very promptly, and shows it to the journalist (for example) right away, can still convince the journalist. This is not great, but at least it doesn’t apply to the vast bulk of your email archive.

There are possible email protocol improvements which might help, but they’re quite out of scope for this article.

Edited 2023-10-01 00:20 +01:00 to fix some grammar

comments

Installing Debian bookworm without systemd

2023-07-19T13:25:46Z

Instructions

Get the official installation image from the usual locations. I got the netinst CD image via BitTorrent.
Boot from the image and go through the installation in the normal way.
1. You may want to select an alternative desktop environment (and unselect GNOME). These steps have been tested with MATE.
2. Stop when you are asked to remove the installation media and reboot.
Press Alt + Right arrow to switch to the text VC. Hit return to activate the console and run the following commands (answering yes as appropriate):

chroot /target bash
apt-get install sysvinit-core elogind ntp dbus-x11
apt-get autoremove
exit

Observe the output from the apt-get install. If your disk arrangements are unusual, that may generate some error messages from update-initramfs.
Go back to the installer VC with Alt + Left arrow. If there were no error messages above, you may tell it to reboot.
If there were error messages (for example, I found that if there was disk encryption, alarming messages were printed), tell the installer to go “Back”. Then ask it to “Install GRUB bootloader” (again). After that has completed, you may reboot.
Enjoy your Debian system without systemd.

Discussion

This is pleasingly straightforward, albeit with an ugly wart. This recipe was not formally developed and tested; it’s just what happened when I tried to actually perform this task.

The official installation guide has similar instructions although they don’t seem to have the initramfs workaround.

`update-initramfs`

The need to go back and have the installer reinstall grub is because if your storage is not very straightforward, the update-initramfs caused by apt-get install apparently doesn’t have all the right context. I haven’t investigated this at all; indeed, I don’t even really know that the initramfs generated in step 3 above was broken, although the messages did suggest to me that important pieces or config might have been omitted. Instead, I simply chose to bet that it might be broken, but that the installer would know what to do. So I used the installer’s “install GRUB bootloader” option, which does regenerate the initramfs. So, I don’t know that step 6 is necessary.

In principle it would be better to do the switch from systemd to sysvinit earlier in the installation process, and under the control of the installer. But by default the installer goes straight from the early setup questions through to the “set the time” or “reboot” questions, without stopping. One could use the expert mode, or modify the command line, or something, but all of those things are, in practice, a lot more typing and/or interaction. And as far as I’m aware the installer doesn’t have an option for avoiding systemd .

The `apt-get install` line

sysvinit-core is the principal part of the sysvinit init system. Asking to install that causes the deinstallation of systemd’s init and ancillary packages.

systemd refuses to allow itself to be deinstalled, if it is already running, so if you boot into the systemd system you can’t then switch init system. This is why the switch is best done at install time. If you’re too late, there are instructions for changing init system post-installation.

elogind is a forked version of some of systemd’s user desktop session functionality. In practice modern desktop environments need this; without it, apt will want to remove things you probably want to keep. Even if you force it, you may find that your desktop environment can’t adjust the audio volume, etc.

ntp is needed because nowadays the default network time client is systemd-timesyncd (which is a bad idea even on systems with systemd as init). We need to specify it because the package dependencies don’t automatically give you any replacement for systemd-timesyncd.

dbus-x11 is a glue component. In theory it ought to be installed automatically. However, there have been problems with the dependencies that meant that (for example) asking for emacs would try to switch the init system. Specifying dbus-x11 explicitly is a workaround for that, which I nowadays adopt out of caution. Perhaps it is no longer needed.

(On existing systems, it may be necessary to manually install orphan-sysvinit-scripts, which exists as a suboptimal technical workaround for the sociopolitical problems of hostile package maintainers and Debian’s governance failures. The recipe above seems to install this package automatically.)

usrmerge

This recipe results in a system which has merged-/usr via symlinks. This configuration is a bad one. Ideally usrmerge-via-symlinks would be avoided.

The un-merged system is declared “not officially supported by Debian” and key packages try very hard to force it on users. However, merged-/usr-via-symlinks is full of bugs (mostly affecting package management) which are far too hard to fix (a project by some folks to try to do so has given up).

I suspect un-merged systems will suffer from fewer bugs in practice. But I don’t know how to persuade d-i to make one.

Installer images

I think there is room in the market for an unofficial installer image which installs without systemd and perhaps without usrmerge. I don’t have the effort for making such a thing myself.

Conclusion

Installing Debian without systemd is fairly straightforward.

Operating Debian without systemd is a pleasure and every time one of my friends has some systemd-induced lossage I get to feel smug.

comments

Never use git submodules

2023-03-02T19:48:20Z

tl;dr

git submodules are always the wrong solution. Yes, even the to the problem they were specifically invented to solve.

What is wrong with git submodules
Better alternatives to git submodules

What is wrong with git submodules

There are two principal sets of reasons why they are terrible:

Fundamentally wrong design. They break the git data model in multiple ways. Critical ways include:
- A git object in your repository is no longer necessarily resolvable/interpetable to meaningful data. (Shallow clones have the same issue but only with respect to history. git submodules do this for the contents of the tree.)
- git submodules violate the usual rule that all URLs, hostnames, and so on, used by git, are provided by the git configuration and the user, rather than appearing in-tree.
- git submodules introduce completely new states your tree can be in, many of them strange or undesirable.
Wrong behaviour in detail. git’s behaviour with submodules is often buggy or bizarre. Some of these problems are implied by the design, but many of them are additional unforced errors. Some of the defects occur even if you don’t git submodule init, so affect all programs and users which interact with your tree.

Just a few examples of lossage with submodules:
- git checkout no longer reliably switches branches
- editing files and trying to commit them no longer reliably works
- locally pulling a new version from main no longer reliably works
- git ls-files can disagree with git log and git cat-file
- URLs from .gitmodules: they can be malicious; they can end up cached in individual trees’ (individual users’) .git/config; etc.
Generally, normal git operations like git checkout and git pull can leave the submodule in a weird state where you have to run one of the git submodule commands to fix it up. Often the easiest way (especially for a non-expert) to get back to a normal state is to throw the whole tree away and re-clone it.

Ultimately, this means that the author of a program which works with git has two options:

Don’t support submodules. Tell users of your program who file bugs involving submodules that they’re not supported.
Do an enormous amount of extra work: At every point you interact with git, experiment to see what bizarre behaviour submodules exhibit, and write code to deal with all the possibilities.

As a result, a substantial subset of git tooling is broken in the presence of submodules. This is especially true of local automation and tooling, which is otherwise an effective way of improving your processes. But, of course this also applies to git itself! Which is one of the causes of the bugs that git itself has when working with submodules.

Better alternatives to git submodules

In my opinion git submodule is never the right answer. Often, git submodule is the worst answer and any of the following would be better.

Use git subtree

git subtree solves many of the same problems as git submodule, but it does not violate the git data model.

Use this when:

You want to track and use, in-tree, a separate project which ought to have its own identity.
The separate project is of reasonable size (compared to your own).

With git subtree, people and programs that do not need to specifically interact with the upstream for the subtree, do not need to know that it even is a subtree. They can make and switch branches, commit, and so on, as they like.

git subtree can automatically separate out changes made in the downstream, for application to (or submission to) the upstream branch.

I have used git subtree and found it capable and convenient, and pleasingly straightforward.

Just have a monorepo

If you are the upstream for all the pieces, it is often more convenient to merge the git trees into a single git tree with a single history.

Use this when:

The maintenance of all the pieces is organisationally and politically cohesive enough that you can share a git history.
The whole monorepo would be of reasonable size.
Any long-running branches you need to make are for release channels, or the similar, not for having separate versions of the internal dependencies for the different pieces in the monorepo.

Use a package management system, and explicit dependencies

Instead of subsuming the dependency’s tree into your own, give the dependency a proper API and reuse it via a package management system. (If necessary, maintain a proper downstream fork of the dependency.)

The package manager might be be:

a distro-style package management system such as apt+dpkg+sbuild (or a proprietary/private dependency-managing build system); or
a language specific package manager (eg cargo).

Use this when:

You are already using, or familiar with, a suitable package manager,
The API provided by the dependency can be reasonably represented in that package manager (even if unstably).

Use the multiple repository tool `mr`

mr(1) is a tool which lets you conveniently manage a possibly large number of trees, usually as sibling directories.

I haven’t used this myself but it looks capable and straightforward. As I understand it, you’d usually use this in combination with the ..-based dependency expectation I describe below.

It seems like it would be good when your project has a fair number of “foreign” dependencies.

Have your build expect to find the dependency in `..`, its parent dir

This is a very lightweight solution. Just have the files in your tree refer to the dependencies with ../dependency-name/. Expect users (and programs) to manually clone and update the right dependency version, alongside your project.

Consider this when:

Your project is at an early stage and you want to get going quickly and worry about this build system stuff later.
The dependency is disabled by default, and almost never neeeded.

Every program or human that wants to run a build that needs the dependency will need to know to clone the dependency, and keep it up to date. This will be a nuisance, and if you’re doing CI it will mean some custom CI scripting. But this is all probably still better than git submodules. At least it will be completely obvious to everyone what’s going on, how to make changes to the dependency, and so on.

Provide an ad-hoc in-tree script to download the dependency

As a last resort, you can embed the URL to find your dependency, and the instructions for downloading it, in your top-level package’s build system. This is clumsy and awkward, but, astonishingly, it is less painful than git submodules.

Use this when:

Most people using/building your software won’t need the dependency at all.
In particular, most people won’t need to edit the dependency.
None of the other options are suitable.

Usually the downstream build runes should git clone the dependency, and the downstream tree should name the precise commitid needed.

Try to avoid this situation. It’s not a good place to be. But:

Yes, really, git submodule is worse than ad-hoc Makefile runes

The ad-hoc shell script route feels very hacky. But it has some important advantages over git submodule. In particular, unlike with git submodule, this approach (like most of the others I suggest) means that:

All tooling that expects to clone your repository, make changes, do builds, track changes, etc., will work correctly.
You are in precise control of when/whether the download occurs: ie, you can arrange to download the dependency precisely when it’s needed.
You are in precise control of your version management and checking of the dependency: your script controls what version of the dependency to use, and whether that should be “pinned” or dynamically updated.

I’m not advocating ad-hoc runes over git submodules because I like ad-hoc runes or think they’re a good idea. It’s just that git submodule is really so very very bad.

comments

derive-adhoc: powerful pattern-based derive macros for Rust

2023-02-03T00:29:28Z

tl;dr

Have you ever wished that you could that could write a new derive macro without having to mess with procedural macros?

Now you can!

derive-adhoc lets you write a #[derive] macro, using a template syntax which looks a lot like macro_rules!.

It’s still 0.x - so unstable, and maybe with sharp edges. We want feedback!

And, the documentation is still very terse. It is doesn’t omit anything, but, it is severely lacking in examples, motivation, and so on. It will suit readers who enjoy dense reference material.

Background - Rust’s two (main) macro systems

Rust has two principal macro systems.

macro_rules! (also known as “macros by example”) is relatively straightforward to use. You have some control over the argument syntax for your macro, and then you can generate output code using a pattern-style template.

But, its power is limited. In particular, although you can specify a pattern to match the arguments to your macro, the pattern matching system has serious limitations (for example, it has a very hard time with Rust’s generic type parameters). Also, you can’t feed existing pieces of your program to a macro without passing them as arguments: so you must write them out twice, or have the macro re-generate its own arguments as a side-effect.

Because of these limitations, code which makes heavy use of macro_rules! macros can be less than clear.

Procedural macros are an extremely powerful and almost fully general code-rewriting macro facility. They work by taking actual rust code (represented as a stream of tokens) as their input, running arbitrary computations, and then generating a new stream of tokens as an output. Procedural macros can be applied (with derive) to Rust’s data structure definitions, to parse them, and autogenerate data-structure-dependent code. They are the basis of many extremely powerful facilities, both in standard Rust, and in popular Rust libraries.

However, procedural macros are hard to write. You must deal with libraries for parsing Rust source code out of tokens. You must generate compile errors manually. You often end up matching on Rust syntax in excruciating detail. Procedural macros run in an inconvient execution context and must live in a separate Rust package. And so on.

“Derive by example” with `derive-adhoc`

derive-adhoc aims to provide much of the power of proc macros, with the convenience of macro_rules!.

You write a template which is expanded for a data structure (for a struct, say). derive-adhoc takes care of parsing the struct, and gives you convenient expansion variables for use in your template.

A simple example - deriving `Clone` without inferred trait bounds

Here is a simple example, taken from clone.rs, in derive-adhoc’s test suite; clone.expanded.rs shows the result of the macro expansion.

This showcases very few of derive-adhoc’s features, but it gives a flavour.

  // Very simple `Clone`
  //
  // Useful because it doesn't infer Clone bounds on generic type
  // parameters, like std's derive of Clone does.  Instead, it
  // unconditionally attempts to implement Clone.
  //
  // Only works on `struct { }` structs.
  //
  // (This does a small subset of what the educe crate's `Clone` does.)
  define_derive_adhoc!{
      MyClone =

      impl<$tgens> Clone for $ttype {
          fn clone(&self) -> Self {
              Self {
                  $(
                      $fname: self.$fname.clone(),
                  )
              }
          }
      }
  }

  // If we were to `#[derive(Clone)]`, DecoratedError<io::Error> wouldn't
  // be Clone, because io::Error isn't, even though the Arc means we can clone.
  #[derive(Adhoc)]
  #[derive_adhoc(MyClone)]
  struct DecoratedError<E> {
      context: String,
      error: Arc<E>,
  }

Replacing an existing bespoke proc macro - a more complex example

Recently, I thought I would try out derive-adhoc in Hippotat, a personal project of mine, which currently uses a project-specific proc macro. This was an enjoyable experience.

I found the new code a huge improvement over the old code. I intend to tidy up this branch and merge it into Hippotat’s mainline, at some suitable point in the release cycles of Hippotat and derive-adhoc.

I won’t copy the whole thing here, but: now we have things like this:

  pub struct InstanceConfig {
    #[adhoc(special="link", skl="SKL::None")]  pub link: LinkName,
  ...

  derive_adhoc!{
    InstanceConfig:

    fn resolve_instance(rctx: &ResolveContext) -> InstanceConfig {
      InstanceConfig {
        $(
          $fname: rctx.
            ${if fmeta(special) {
              ${paste special_ ${fmeta(special)}}
            } else {

Instead of this kind of awfulness:

   } else if attr.path == parse_quote!{ special } {
     let meta = match attr.parse_meta().unwrap() {
       Meta::List(list) => list,
       _ => panic!(),
     };
     let (tmethod, tskl) = meta.nested.iter().collect_tuple().unwrap();
     fn get_path(meta: &NestedMeta) -> TokenStream {
       match meta {
         NestedMeta::Meta(Meta::Path(ref path)) => path.to_token_stream(),
         _ => panic!(),
       }
     }
     method = get_path(tmethod);
     *skl.borrow_mut() = Some(get_path(tskl));
   }

History and acknowledgements

derive-adhoc was my project proposal in last year’s Tor Project Hackweek. Thanks to Tor for giving us the space to do something like this.

Nick Mathewson joined in, and has made important code contributions, also given invaluable opinions and feedback. Thanks very much to Nick - I look forward to working on this more with you. I take responsibility for all bugs, mistakes, and misjudgements of taste.

Future plans

We’re hoping derive-adhoc will become a widely-used library, significantly improving Rust’s expressive power at the same time as improving the clarity of macro-using programs.

We would like to see the wider Rust community experiment with it and give us feedback. Who knows? Maybe it will someday inspire a “Derive By Example” feature in standard Rust.

Two words of warning, though:

The documentation is currently very terse. Many readers will find it far too dense and dry for easy comprehension. The examples are sparse, not well integrated, and not very well explained. We will need your patience - and and your help - as we try to improve it.
The library and its template syntax are still unstable. As more people try derive_adhoc, we expect to find areas where the syntax and behavior need to improve. While still at 0.x, we’ll be keen to make those improvements, without much regard to backward compatibility. So, for now, expect breaking changes.

We hope to release a more-stable and better-documented version 1.x later this lear.

So, please try it out and let us know what you think.

Documentation and references about `derive-adhoc`

edited 2023-02-03 00:34 Z to fix a typo

comments

SGO (and my) VPN and network access tools - in bookworm

2023-01-14T00:41:35Z

Recently, we managed to get secnet and hippotat into Debian. They are on track to go into Debian bookworm. This completes in Debian the set of VPN/networking tools I (and other Greenend) folks have been using for many years.

The Sinister Greenend Organisation’s suite of network access tools consists mainly of:

secnet - VPN.
hippotat - IP-over-HTTP (workaround for bad networks)
userv ipif - user-created network interfaces

secnet

secnet is our very mature VPN system.

Its basic protocol idea is similar to that in Wireguard, but it’s much older. Differences from Wireguard include:

Comes with some (rather clumsy) provisioning tooling, supporting almost any desired virtual network topology. In the SGO we have a complete mesh of fixed sites (servers), and a number of roaming hosts (clients), each of which can have one or more sites as its home.
No special kernel drivers required. Everything is userspace.
An exciting “polypath” mode where packets are sent via multiple underlying networks in parallel, offering increased reliability for roaming hosts.
Portable to non-Linux platforms.
A much older, and less well audited, codebase.
Very flexible configuration arrangements, but things are also under-documented and to an extent under-productised.
Hasn’t been ported to phones/tablets.

secnet was originally written by Stephen Early, starting in 1996 or so. I inherited it some years ago and have been maintaining it since. It’s mostly written in C.

Hippotat

Hippotat is best described by copying the intro from the docs:

Hippotat is a system to allow you to use your normal VPN, ssh, and other applications, even in broken network environments that are only ever tested with “web stuff”.

Packets are parcelled up into HTTP POST requests, resembling form submissions (or JavaScript XMLHttpRequest traffic), and the returned packets arrive via the HTTP response bodies.

It doesn’t rely on TLS tunnelling so can work even if the local network is trying to intercept TLS. I recently rewrote Hippotat in Rust.

`userv ipif`

userv ipif is one of the userv utilities.

It allows safe delegation of network routing to unprivileged users. The delegation is of a specific address range, so different ranges can be delegated to different users, and the authorised user cannot interfere with other traffic.

This is used in the default configuration of hippotat packages, so that an ordinary user can start up the hippotat client as needed.

On chiark userv-ipif is used to delegate networking to users, including administrators of allied VPN realms. So chiark actually runs at least 4 VPN-ish systems in production: secnet, hippotat, Mark Wooding’s Tripe, and still a few links managed by the now-superseded udptunnel system.

`userv`

userv ipif is a userv service. That is, it is a facility which uses userv to bridge a privilege boundary.

userv is perhaps my most under-appreciated program. userv can be used to straightforwardly bridge (local) privilege boundaries on Unix systems.

So for example it can:

Allow a sysadmin to provide a shell script to be called by unprivileged users, but which will run as root. sudo can do this too but it has quite a few gotchas, and you have to be quite careful how you use it - and its security record isn’t great either.
Form the internal boundary in a privilege-separated system service. So, for example, the hippotat client is a program you can run from the command line as a normal user, if the relevant network addresses have been delegated to you. On chiark, CGI programs run as the providing user - not using suexec (which I don’t trust), but via userv.

userv services can be defined by the called user, not only by the system administrator. This allows a user to reconfigure or divert a system-provided default implementation, and even allows users to define and implement ad-hoc services of their own. (Although, the system administrator can override user config.)

Acknowledgements

Thanks for the help I had in this effort.

In particular, thanks to Sean Whitton for encouragement, and the ftpmaster review; and to the Debian Rust Team for their help navigating the complexities of handling Rust packages within the Debian Rust Team workflow.

comments

Rust for the Polyglot Programmer, December 2022 edition

2022-12-20T01:23:21Z

I have reviewed, updated and revised my short book about the Rust programming language, Rust for the Polyglot Programmer.

It now covers some language improvements from the past year (noting which versions of Rust they’re available in), and has been updated for changes in the Rust library ecosystem.

With (further) assistance from Mark Wooding, there is also a new table of recommendations for numerical conversion.

Recap about Rust for the Polyglot Programmer

There are many introductory materials about Rust. This one is rather different. Compared to much other information about Rust, Rust for the Polyglot Programmer is:

Dense: I assume a lot of starting knowledge. Or to look at it another way: I expect my reader to be able to look up and digest non-Rust-specific words or concepts.
Broad: I cover not just the language and tools, but also the library ecosystem, development approach, community ideology, and so on.
Frank: much material about Rust has a tendency to gloss over or minimise the bad parts. I don’t do that. That also frees me to talk about strategies for dealing with the bad parts.
Non-neutral: I’m not afraid to recommend particular libraries, for example. I’m not afraid to extol Rust’s virtues in the areas where it does well.
Terse, and sometimes shallow: I often gloss over what I see as unimportant or fiddly details; instead I provide links to appropriate reference materials.

After reading Rust for the Polyglot Programmer, you won’t know everything you need to know to use Rust for any project, but should know where to find it.

Comments are welcome of course, via the Dreamwidth comments or Salsa issue or MR. (If you’re making a contribution, please indicate your agreement with the Developer Certificate of Origin.)

edited 2022-12-20 01:48 to fix a typo

comments

Rust needs #[throws]

2022-12-16T18:55:09Z

tl;dr:

Ok-wrapping as needed in today’s Rust is a significant distraction, because there are multiple ways to do it. They are all slightly awkward in different ways, so are least-bad in different situations. You must choose a way for every fallible function, and sometimes change a function from one pattern to another.

Rust really needs #[throws] as a first-class language feature. Code using #[throws] is simpler and clearer.

Please try out withoutboats’s fehler. I think you will like it.

A recent personal experience in coding style
What is Ok wrapping? Intro to Rust error handling
A minor inconvenience, or a significant distraction?
- Idioms for Ok-wrapping - a bestiary
What is to be done, then?
Appendix - examples showning code with Ok wrapping is worse than code using #[throws]

A recent personal experience in coding style

Ever since I read withoutboats’s 2020 article about fehler, I have been using it in most of my personal projects.

For Reasons I recently had a go at eliminating the dependency on fehler from Hippotat. So, I made a branch, deleted the dependency and imports, and started on the whack-a-mole with the compiler errors.

After about a half hour of this, I was starting to feel queasy.

After an hour I had decided that basically everything I was doing was making the code worse. And, bizarrely, I kept having to make individual decisons about what idiom to use in each place. I couldn’t face it any more.

After sleeping on the question I decided that Hippotat would be in Debian with fehler, or not at all. Happily the Debian Rust Team generously helped me out, so the answer is that fehler is now in Debian, so it’s fine.

For me this experience, of trying to convert Rust-with-#[throws] to Rust-without-#[throws] brought the Ok wrapping problem into sharp focus.

What is `Ok` wrapping? Intro to Rust error handling

(You can skip this section if you’re already a seasoned Rust programer.)

In Rust, fallibility is represented by functions that return Result<SuccessValue, Error>: this is a generic type, representing either whatever SuccessValue is (in the Ok variant of the data-bearing enum) or some Error (in the Err variant). For example, std::fs::read_to_string, which takes a filename and returns the contents of the named file, returns Result<String, std::io::Error>.

This is a nice and typesafe formulation of, and generalisation of, the traditional C practice, where a function indicates in its return value whether it succeeded, and errors are indicated with an error code.

Result is part of the standard library and there are convenient facilities for checking for errors, extracting successful results, and so on. In particular, Rust has the postfix ? operator, which, when applied to a Result, does one of two things: if the Result was Ok, it yields the inner successful value; if the Result was Err, it returns early from the current function, returning an Err in turn to the caller.

This means you can write things like this:

    let input_data = std::fs::read_to_string(input_file)?;

and the error handling is pretty automatic. You get a compiler warning, or a type error, if you forget the ?, so you can’t accidentally ignore errors.

But, there is a downside. When you are returning a successful outcome from your function, you must convert it into a Result. After all, your fallible function has return type Result<SuccessValue, Error>, which is a different type to SuccessValue. So, for example, inside std::fs::read_to_string, we see this:

        let mut string = String::new();
        file.read_to_string(&mut string)?;
        Ok(string)
    }

string has type String; fs::read_to_string must return Result<String, ..>, so at the end of the function we must return Ok(string). This applies to return statements, too: if you want an early successful return from a fallible function, you must write return Ok(whatever).

This is particularly annoying for functions that don’t actually return a nontrivial value. Normally, when you write a function that doesn’t return a value you don’t write the return type. The compiler interprets this as syntactic sugar for -> (), ie, that the function returns (), the empty tuple, used in Rust as a dummy value in these kind of situations. A block ({ ... }) whose last statement ends in a ; has type (). So, when you fall off the end of a function, the return value is (), without you having to write it. So you simply leave out the stuff in your program about the return value, and your function doesn’t have one (i.e. it returns ()).

But, a function which either fails with an error, or completes successfuly without returning anything, has return type Result<(), Error>. At the end of such a function, you must explicitly provide the success value. After all, if you just fall off the end of a block, it means the block has value (), which is not of type Result<(), Error>. So the fallible function must end with Ok(()), as we see in the example for std::fs::read_to_string.

A minor inconvenience, or a significant distraction?

I think the need for Ok-wrapping on all success paths from fallible functions is generally regarded as just a minor inconvenience. Certainly the experienced Rust programmer gets very used to it. However, while trying to remove fehler’s #[throws] from Hippotat, I noticed something that is evident in codebases using “vanilla” Rust (without fehler) but which goes un-remarked.

There are multiple ways to write the Ok-wrapping, and the different ways are appropriate in different situations.

See the following examples, all taken from a real codebase. (And it’s not just me: I do all of these in different places, - when I don’t have fehler available - but all these examples are from code written by others.)

Idioms for `Ok`-wrapping - a bestiary

Wrap just a returned variable binding

If you have the return value in a variable, you can write Ok(reval) at the end of the function, instead of retval.

    pub fn take_until(&mut self, term: u8) -> Result<&'a [u8]> {
        // several lines of code
        Ok(result)
    }

If the returned value is not already bound to variable, making a function fallible might mean choosing to bind it to a variable.

Wrap a nontrivial return expression

Even if it’s not just a variable, you can wrap the expression which computes the returned value. This is often done if the returned value is a struct literal:

    fn take_from(r: &mut Reader<'_>) -> Result<Self> {
        // several lines of code
        Ok(AuthChallenge { challenge, methods })
    }

Introduce `Ok(())` at the end

For functions returning Result<()>, you can write Ok(()).

This is usual, but not ubiquitous, since sometimes you can omit it.

Wrap the whole body

If you don’t have the return value in a variable, you can wrap the whole body of the function in Ok(…). Whether this is a good idea depends on how big and complex the body is.

    fn from_str(s: &str) -> std::result::Result<Self, Self::Err> {
        Ok(match s {
            "Authority" => RelayFlags::AUTHORITY,
            // many other branches
            _ => RelayFlags::empty(),
        })
    }

Omit the wrap when calling fallible sub-functions

If your function wraps another function call of the same return and error type, you don’t need to write the Ok at all. Instead, you can simply call the function and not apply ?.

You can do this even if your function selects between a number of different sub-functions to call:

    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        if flags::unsafe_logging_enabled() {
            std::fmt::Display::fmt(&self.0, f)
        } else {
            self.0.display_redacted(f)
        }
    }

But this doesn’t work if the returned error type isn’t the same, but needs the autoconversion implied by the ? operator.

Convert a fallible sub-function error with `Ok( ... ?)`

If the final thing a function does is chain to another fallible function, but with a different error type, the error must be converted somehow. This can be done with ?.

     fn try_from(v: i32) -> Result<Self, Error> {
         Ok(Percentage::new(v.try_into()?))
     }

Convert a fallible sub-function error with `.map_err`

Or, rarely, people solve the same problem by converting explicitly with .map_err:

     pub fn create_unbootstrapped(self) -> Result<TorClient<R>> {
         // several lines of code
         TorClient::create_inner(
             // several parameters
         )
         .map_err(ErrorDetail::into)
     }

What is to be done, then?

The fehler library is in excellent taste and has the answer. With fehler:

Whether a function is fallible, and what it’s error type is, is specified in one place. It is not entangled with the main return value type, nor with the success return paths.
So the success paths out of a function are not specially marked with error handling boilerplate. The end of function return value, and the expression after return, are automatically wrapped up in Ok. So the body of a fallible function is just like the body of an infallible one, except for places where error handling is actually involved.
Error returns occur through ? error chaining, and with a new explicit syntax for error return.
We usually talk about the error we are possibly returning, and avoid talking about Result unless we need to.

fehler provides:

An attribute macro #[throws(ErrorType)] to make a function fallible in this way.
A macro throws!(error) for explicitly failing.

This is precisely correct. It is very ergonomic.

Consequences include:

One does not need to decide where to put the Ok-wrapping, since it’s automatic rather than explicitly written out.
Specifically, what idiom to adopt in the body (for example {write!(...)?;} vs {write!(...)} in a formatter) does not depend on whether the error needs converting, how complex the body is, and whether the final expression in the function is itself fallible.
Making an infallible function fallible involves only adding #[throws] to its definition, and ? to its call sites. One does not need to edit the body, or the return type.
Changing the error returned by a function to a suitably compatible different error type does not involve changing the function body.
There is no need for a local Result alias shadowing std::result::Result, which means that when one needs to speak of Result explciitly, the code is clearer.

Limitations of fehler

But, fehler is a Rust procedural macro, so it cannot get everything right. Sadly there are some wrinkles.

You can’t write #[throws] on a closure.
Sometimes you can get quite poor error messages if you have a sufficiently broken function body.
Code inside a macro call isn’t properly visible to fehler so sometimes return statements inside macro calls are untreated. This will lead to a type error, so isn’t a correctness hazard, but it can be nuisance if you like other syntax extensions eg if_chain.
#[must_use] #[throws(Error)] fn obtain() -> Thing; ought to mean that Thing must be used, not the Result<Thing, Error>.

But, Rust-with-#[throws] is so much nicer a language than Rust-with-mandatory-Ok-wrapping, that these are minor inconveniences.

Please can we have `#[throws]` in the Rust language

This ought to be part of the language, not a macro library. In the compiler, it would be possible to get the all the corner cases right. It would make the feature available to everyone, and it would quickly become idiomatic Rust throughout the community.

It is evident from reading writings from the time, particularly those from withoutboats, that there were significant objections to automatic Ok-wrapping. It seems to have become quite political, and some folks burned out on the topic.

Perhaps, now, a couple of years later, we can revisit this area and solve this problem in the language itself ?

“Explicitness”

An argument I have seen made against automatic Ok-wrapping, and, in general, against any kind of useful language affordance, is that it makes things less explicit.

But this argument is fundamentally wrong for Ok-wrapping. Explicitness is not an unalloyed good. We humans have only limited attention. We need to focus that attention where it is actually needed. So explicitness is good in situtions where what is going on is unusual; or would otherwise be hard to read; or is tricky or error-prone. Generally: explicitness is good for things where we need to direct humans’ attention.

But Ok-wrapping is ubiquitous in fallible Rust code. The compiler mechanisms and type systems almost completely defend against mistakes. All but the most novice programmer knows what’s going on, and the very novice programmer doesn’t need to. Rust’s error handling arrangments are designed specifically so that we can avoid worrying about fallibility unless necessary — except for the Ok-wrapping. Explicitness about Ok-wrapping directs our attention away from whatever other things the code is doing: it is a distraction.

So, explicitness about Ok-wrapping is a bad thing.

Appendix - examples showning code with `Ok` wrapping is worse than code using `#[throws]`

Observe these diffs, from my abandoned attempt to remove the fehler dependency from Hippotat.

I have a type alias AE for the usual error type (AE stands for anyhow::Error). In the non-#[throws] code, I end up with a type alias AR<T> for Result<T, AE>, which I think is more opaque — but at least that avoids typing out -> Result< , AE> a thousand times. Some people like to have a local Result alias, but that means that the standard Result has to be referred to as StdResult or std::result::Result.

With `fehler` and `#[throws]`	Vanilla Rust, `Result<>`, mandatory `Ok-wrapping`

Return value clearer, error return less wordy:
`impl Parseable for Secret {`	`impl Parseable for Secret {`
`#[throws(AE)]`
`fn parse(s: Option<&str>) -> Self {`	`fn parse(s: Option<&str>) -> AR<Self> {`
`let s = s.value()?;`	`let s = s.value()?;`
`if s.is_empty() { throw!(anyhow!(“secret value cannot be empty”)) }`	`if s.is_empty() { return Err(anyhow!(“secret value cannot be empty”)) }`
`Secret(s.into())`	`Ok(Secret(s.into()))`
`}`	`}`
`…`	`…`
No need to wrap whole match statement in `Ok( ):`
`#[throws(AE)]`
`pub fn client<T>(&self, key: &’static str, skl: SKL) -> T`	`pub fn client<T>(&self, key: &’static str, skl: SKL) -> AR<T>`
`where T: Parseable + Default {`	`where T: Parseable + Default {`
`match self.end {`	`Ok(match self.end {`
`LinkEnd::Client => self.ordinary(key, skl)?,`	`LinkEnd::Client => self.ordinary(key, skl)?,`
`LinkEnd::Server => default(),`	`LinkEnd::Server => default(),`
`}`	`})`
`…`	`…`
Return value and `Ok(())` entirely replaced by `#[throws]`:
`impl Display for Loc {`	`impl Display for Loc {`
`#[throws(fmt::Error)]`
`fn fmt(&self, f: &mut fmt::Formatter) {`	`fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {`
`write!(f, “{:?}:{}”, &self.file, self.lno)?;`	`write!(f, “{:?}:{}”, &self.file, self.lno)?;`
`if let Some(s) = &self.section {`	`if let Some(s) = &self.section {`
`write!(f, “ ”)?;`	`write!(f, “ ”)?;`
`…`	`…`
`}`	`}`
	`Ok(())`
`}`	`}`
Call to `write!` now looks the same as in more complex case shown above:
`impl Debug for Secret {`	`impl Debug for Secret {`
`#[throws(fmt::Error)]`
`fn fmt(&self, f: &mut fmt::Formatter) {`	`fn fmt(&self, f: &mut fmt::Formatter)-> fmt::Result {`
`write!(f, "Secret(***)")?;`	`write!(f, "Secret(***)")`
`}`	`}`
`}`	`}`
Much tiresome `return Ok()` noise removed:
`impl FromStr for SectionName {`	`impl FromStr for SectionName {`
`type Err = AE;`	`type Err = AE;`
`#[throws(AE)]`
`fn from_str(s: &str) -> Self {`	`fn from_str(s: &str) ->AR< Self> {`
`match s {`	`match s {`
`“COMMON” => return SN::Common,`	`“COMMON” => return Ok(SN::Common),`
`“LIMIT” => return SN::GlobalLimit,`	`“LIMIT” => return Ok(SN::GlobalLimit),`
`_ => { }`	`_ => { }`
`};`	`};`
`if let Ok(n@ ServerName(_)) = s.parse() { return SN::Server(n) }`	`if let Ok(n@ ServerName(_)) = s.parse() { return Ok(SN::Server(n)) }`
`if let Ok(n@ ClientName(_)) = s.parse() { return SN::Client(n) }`	`if let Ok(n@ ClientName(_)) = s.parse() { return Ok(SN::Client(n)) }`
`…`	`…`
`if client == “LIMIT” { return SN::ServerLimit(server) }`	`if client == “LIMIT” { return Ok(SN::ServerLimit(server)) }`
`let client = client.parse().context(“client name in link section name”)?;`	`let client = client.parse().context(“client name in link section name”)?;`
`SN::Link(LinkName { server, client })`	`Ok(SN::Link(LinkName { server, client }))`
`}`	`}`
`}`	`}`

edited 2022-12-18 19:58 UTC to improve, and 2022-12-18 23:28 to fix, formatting

comments

Stop writing Rust linked list libraries!

2022-11-12T15:13:30Z

tl;dr:

Don’t write a Rust linked list library: they are hard to do well, and usually useless.

Use VecDeque, which is great. If you actually need more than VecDeque can do, use one of the handful of libraries that actually offer a significantly more useful API.

If you are writing your own data structure, check if someone has done it already, and consider slotmap or generation_arena, (or maybe Rc/Arc).

Survey of Rust linked list libraries
- Background
- Results
Why are there so many poor Rust linked list libraries ?
- Double-ended queues
- The Cursor concept
Rustic approaches to pointers-to-and-between-nodes data structures
- The alternative for nodey data structures in safe Rust: Rc/Arc
Rust’s package ecosystem demonstrating software’s NIH problem
- The package naming paradox

Survey of Rust linked list libraries

I have updated my Survey of Rust linked list libraries.

Background

In 2019 I was writing plag-mangler, a tool for planar graph layout.

I needed a data structure. Naturally I looked for a library to help. I didn’t find what I needed, so I wrote rc-dlist-deque. However, on the way I noticed an inordinate number of linked list libraries written in Rust. Most all of these had no real reason for existing. Even the one in the Rust standard library is useless.

Results

Now I have redone the survey. The results are depressing. In 2019 there were 5 libraries which, in my opinion, were largely useless. In late 2022 there are now thirteen linked list libraries that ought probably not ever to be used. And, a further eight libraries for which there are strictly superior alternatives. Many of these have the signs of projects whose authors are otherwise competent: proper documentation, extensive APIs, and so on.

There is one new library which is better for some applications than those available in 2019. (I’m referring to generational_token_list, which makes a plausible alternative to dlv-list which I already recommended in 2019.)

Why are there so many poor Rust linked list libraries ?

Linked lists and Rust do not go well together. But (and I’m guessing here) I presume many people are taught in programming school that a linked list is a fundamental data structure; people are often even asked to write one as a teaching exercise. This is a bad idea in Rust. Or maybe they’ve heard that writing linked lists in Rust is hard and want to prove they can do it.

Double-ended queues

One of the main applications for a linked list in a language like C, is a queue, where you put items in at one end, and take them out at the other. The Rust standard library has a data structure for that, VecDeque.

Five of the available libraries:

Have an API which is a subset of that of VecDeque: basically, pushing and popping elements at the front and back.
Have worse performance for most applications than VecDeque,
Are less mature, less available, less well tested, etc., than VecDeque, simply because VecDeque is in the Rust Standard Library.

For these you could, and should, just use VecDeque instead.

The `Cursor` concept

A proper linked list lets you identify and hold onto an element in the middle of the list, and cheaply insert and remove elements there.

Rust’s ownership and borrowing rules make this awkward. One idea that people have many times reinvented and reimplemented, is to have a Cursor type, derived from the list, which is a reference to an element, and permits insertion and removal there.

Eight libraries have implemented this in the obvious way. However, there is a serious API limitation:

To prevent a cursor being invalidated (e.g. by deletion of the entry it points to) you can’t modify the list while the cursor exists. You can only have one cursor (that can be used for modification) at a time.

The practical effect of this is that you cannot retain cursors. You can make and use such a cursor for a particular operation, but you must dispose of it soon. Attempts to do otherwise will see you losing a battle with the borrow checker.

If that’s good enough, then you could just use a VecDeque and use array indices instead of the cursors. It’s true that deleting or adding elements in the middle involves a lot of copying, but your algorithm is O(n) even with the single-cursor list libraries, because it must first walk the cursor to the desired element.

Formally, I believe any algorithm using these exclusive cursors can be rewritten, in an obvious way, to simply iterate and/or copy from the start or end (as one can do with VecDeque) without changing the headline O() performance characteristics.

IMO the savings available from avoiding extra copies etc. are not worth the additional dependency, unsafe code, and so on, especially as there are other ways of helping with that (e.g. boxing the individual elements).

Even if you don’t find that convincing, generational_token_list and dlv_list are strictly superior since they offer a more flexible and convenient API and better performance, and rely on much less unsafe code.

Rustic approaches to pointers-to-and-between-nodes data structures

Most of the time a VecDeque is great. But if you actually want to hold onto (perhaps many) references to the middle of the list, and later modify it through those references, you do need something more. This is a specific case of a general class of problems where the naive approach (use Rust references to the data structure nodes) doesn’t work well.

But there is a good solution:

Keep all the nodes in an array (a Vec<Option<T>> or similar) and use the index in the array as your node reference. This is fast, and quite ergonomic, and neatly solves most of the problems. If you are concerned that bare indices might cause confusion, as newly inserted elements would reuse indices, add a per-index generation count.

These approaches have been neatly packaged up in libraries like slab, slotmap, generational-arena and thunderdome. And they have been nicely applied to linked lists by the authors of generational_token_list. and dlv-list.

The alternative for nodey data structures in safe Rust: `Rc`/`Arc`

Of course, you can just use Rust’s “interior mutability” and reference counting smart pointers, to directly implement the data structure of your choice.

In many applications, a single-threaded data structure is fine, in which case Rc and Cell/RefCell will let you write safe code, with cheap refcount updates and runtime checks inserted to defend against unexpected aliasing, use-after-free, etc.

I took this approach in rc-dlist-deque, because I wanted each node to be able to be on multiple lists.

Rust’s package ecosystem demonstrating software’s NIH problem

The Rust ecosystem is full of NIH libraries of all kinds. In my survey, there are: five good options; seven libraries which are plausible, but just not as good as the alternatives; and fourteen others.

There is a whole rant I could have about how the whole software and computing community is pathologically neophilic. Often we seem to actively resist reusing ideas, let alone code; and are ignorant and dismissive of what has gone before. As a result, we keep solving the same problems, badly - making the same mistakes over and over again. In some subfields, working software, or nearly working software, is frequently replaced with something worse, maybe more than once.

One aspect of this is a massive cultural bias towards rewriting rather than reusing, let alone fixing and using.

Many people can come out of a degree, trained to be a programmer, and have no formal training in selecting and evaluating software; this is even though working effectively with computers requires making good use of everyone else’s work.

If one isn’t taught these skills (when and how to search for prior art, how to choose between dependencies, and so on) one must learn it on the job. The result is usually an ad-hoc and unsystematic approach, often dominated by fashion rather than engineering.

The package naming paradox

The more experienced and competent programmer is aware of all the other options that exist - after all they have evaluated other choices before writing their own library.

So they will call their library something like generational_token_list or vecdeque-stableix.

Whereas the novice straight out of a pre-Rust programming course just thinks what they are doing is the one and only obvious thing (even though it’s a poor idea) and hasn’t even searched for a previous implementation. So they call their package something obvious like “linked list”.

As a result, the most obvious names seem to refer to the least useful libraries.

Edited 2022-11-16 23:55 UTC to update numbers of libraries in various categories following updates to the survey (including updates prompted by feedback received after this post first published).

comments

Skipping releases when upgrading Debian systems

2022-10-10T15:48:27Z

Debian does not officially support upgrading from earlier than the previous stable release: you’re not supposed to “skip” releases. Instead, you’re supposed to upgrade to each intervening major release in turn.

However, skipping intervening releases does, in fact, often work quite well. Apparently, this is surprising to many people, even Debian insiders. I was encouraged to write about it some more.

My personal experience

I have three conventionally-managed personal server systems (by which I mean systems which aren’t reprovisioned by some kind of automation). Of these at least two have been skip upgraded at least once:

The one I don’t think I’ve skip-upgraded (at least, not recently) is my house network manager (and now VM host) which I try to keep to a minimum in terms of functionality and which I keep quite up to date. It was crossgraded from i386 (32-bit) to amd64 (64-bit) fairly recently, which is a thing that Debian isn’t sure it supports. The crossgrade was done a hurry and without any planning, prompted by Spectre et al suddenly requiring big changes to Xen. But it went well enough.

My home “does random stuff” server (media server, web cache, printing, DNS, backups etc.), has etckeeper records starting in 2015. I upgraded directly from jessie (Debian 8) to buster (Debian 10). I think it has probably had earlier skip upgrade(s): the oldest file in /etc is from December 1996 and I have been doing occasional skip upgrades as long as I can remember.

And of course there’s chiark, which is one of the oldest Debian installs in existence. I wrote about the most recent upgrade, where I went directly from jessie i386 ELTS (32-bit Debian 8) to bulleye amd64 (64-bit Debian 11). That was a very extreme case which required significant planning and pre-testing, since the package dependencies were in no way sufficient for the proper ordering. But, I don’t normally go to such lengths. Normally, even on chiark, I just edit the sources.list and see what apt proposes to do.

I often skip upgrade chiark because I tend to defer risky-looking upgrades partly in the hope of others fixing the bugs while I wait :-), and partly just because change is disruptive and amortising it is very helpful both to me and my users. I have some records of chiark’s upgrades from my announcements to users. As well as the recent “skip skip up cross grade, direct”, I definitely did a skip upgrade of chiark from squeeze (Debian 6) to jessie (Debian 8). It appears that the previous skip upgrade on chiark was rex (Debian 1.2) to hamm (Debian 2.0).

I don’t think it’s usual for me to choose to do a multi-release upgrade the “officially supported” way, in two (or more) stages, on a server. I have done that on systems with a GUI desktop setup, but even then I usually skip the intermediate reboot(s).

When to skip upgrade (and what precautions to take)

I’m certainly not saying that everyone ought to be doing this routinely. Most users with a Debian install that is older than oldstable probably ought to reinstall it, or do the two-stage upgrade.

Skip upgrading almost always runs into some kind of trouble (albeit, usually trouble that isn’t particularly hard to fix if you know what you’re doing).

However, officially supported non-skip upgrades go wrong too. Doing a two-or-more-releases upgrade via the intermediate releases can expose you to significant bugs in the intermediate releases, which were later fixed. Because Debian’s users and downstreams are cautious, and Debian itself can be slow, it is common for bugs to appear for one release and then be fixed only in the next. Paradoxically, this seems to be especially true with the kind of big and scary changes where you’d naively think the upgrade mechanisms would break if you skipped the release where the change first came in.

I would not recommend a skip upgrade to someone who is not a competent Debian administrator, with good familiarity with Debian package management, including use of dpkg directly to fix things up. You should have a mental toolkit of manual bug workaround techniques. I always should make sure that I have rescue media (and in the case of a remote system, full remote access including ability to boot a different image), although I don’t often need it.

And, when considering a skip upgrade, you should be aware of the major changes that have occurred in Debian.

Skip upgrading is more likely to be a good idea with a complex and highly customised system: a fairly vanilla install is not likely to encounter problems during a two-stage update. (And, a vanilla system can be “upgraded” by reinstalling.)

I haven’t recently skip upgraded a laptop or workstation. I doubt I would attempt it; modern desktop software seems to take a much harder line about breaking things that are officially unsupported, and generally trying to force everyone into the preferred mold.

A request to Debian maintainers

I would like to encourage Debian maintainers to defer removing upgrade compatibility machinery until it is actually getting in the way, or has become hazardous, or many years obsolete.

Examples of the kinds of things which it would be nice to keep, and which do not usually cause much inconvenience to retain, are dependency declarations (particularly, alternatives), and (many) transitional fragments in maintainer scripts.

If you find yourself needing to either delete some compatibility feature, or refactor/reorganise it, I think it is probably best to delete it. If you modify it significantly, the resulting thing (which won’t be tested until someone uses it in anger) is quite likely to have bugs which make it go wrong more badly (or, more confusingly) than the breakage that would happen without it.

Obviously this is all a judgement call.

I’m not saying Debian should formally “support” skip upgrades, to the extent of (further) slowing down important improvements. Nor am I asking for any change to the routine approach to (for example) transitional packages (i.e. the technique for ensuring continuity of installation when a package name changes).

We try to make release upgrades work perfectly; but skip upgrades don’t have to work perfectly to be valuable. Retaining compatibility code can also make it easier to provide official backports, and it probably helps downstreams with different release schedules.

The fact that maintainers do in practice often defer removing compatibility code provides useful flexibility and options to at least some people. So it would be nice if you’d at least not go out of your way to break it.

Building on older releases

I would also like to encourage maintainers to provide source packages in Debian unstable that will still build on older releases, where this isn’t too hard and the resulting binaries might be basically functional.

Speaking personally, it’s not uncommon for me to rebuild packages from unstable and install them on much older releases. This is another thing that is not officially supported, but which often works well.

I’m not saying to contort your build system, or delay progress. You’ll definitely want to depend on a recentish debhelper. But, for example, retaining old build-dependency alternatives is nice. Retaining them doesn’t constitute a promise that it works - it just makes life slightly easier for someone who is going off piste.

If you know you have users on multiple distros or multiple releases, and wish to fully support them, you can go further, of course. Many of my own packages are directly buildable, or even directly installable, on older releases.

comments

Hippotat (IP over HTTP) - first advertised release

2022-09-28T20:12:48Z

I have released version 1.0.0 of Hippotat, my IP-over-HTTP system. To quote the README:

You’re in a cafe or a hotel, trying to use the provided wifi. But it’s not working. You discover that port 80 and port 443 are open, but the wifi forbids all other traffic.

Never mind, start up your hippotat client. Now you have connectivity. Your VPN and SSH and so on run over Hippotat. The result is not very efficient, but it does work.

Story

In early 2017 I was in a mountaintop cafeteria, hoping to do some work on my laptop. (For Reasons I couldn’t go skiing that day.) I found that local wifi was badly broken: It had a severe port block. I had to use my port 443 SSH server to get anywhere. My usual arrangements punt everything over my VPN, which uses UDP of course, and I had to bodge several things. Using a web browser directly only the wifi worked normally, of course - otherwise the other guests would have complained. This was not the first experience like this I’d had, but this time I had nothing much else to do but fix it.

In a few furious hacking sessions, I wrote Hippotat, a tool for making my traffic look enough like “ordinary web browsing” that it gets through most stupid firewalls. That Python version of Hippotat served me well for many years, despite being rather shonky, extremely inefficient in CPU (and therefore battery) terms and not very productised.

But recently things have started to go wrong. I was using Twisted Python and there was what I think must be some kind of buffer handling bug, which started happening when I upgraded the OS (getting newer versions of Python and the Twisted libraries). The Hippotat code, and the Twisted APIs, were quite convoluted, and I didn’t fancy debugging it.

So last year I rewrote it in Rust. The new Rust client did very well against my existing servers. To my shame, I didn’t get around to releasing it.

However, more recently I upgraded the server hosts my Hippotat daemons run on to recent Debian releases. They started to be affected by the bug too, rendering my Rust client unuseable. I decided I had to deploy the Rust server code.

This involved some packaging work. Having done that, it’s time to release it: Hippotat 1.0.0 is out.

The package build instructions are rather strange

My usual approach to releasing something like this would be to provide a git repository containing a proper Debian source package. I might also build binaries, using sbuild, and I would consider actually uploading to Debian.

However, despite me taking a fairly conservative approach to adding dependencies to Hippotat, still a couple of the (not very unusual) Rust packages that Hippotat depends on are not in Debian. Last year I considered tackling this head-on, but I got derailed by difficulties with Rust packaging in Debian.

Furthermore, the version of the Rust compiler itself in Debian stable is incapable of dealing with recent versions of very many upstream Rust packages, because many packages’ most recent versions now require the 2021 Edition of Rust. Sadly, Rust’s package manager, cargo, has no mechanism for trying to choose dependency versions that are actually compatible with the available compiler; efforts to solve this problem have still not borne the needed fruit.

The result is that, in practice, currently Hippotat has to be built with (a) a reasonably recent Rust toolchain such as found in Debian unstable or obtained from Rust upstream; (b) dependencies obtained from the upstream Rust repository.

At least things aren’t completely terrible: Rustup itself, despite its alarming install rune, has a pretty good story around integrity, release key management and so on. And with the right build rune, cargo will check not just the versions, but the precise content hashes, of the dependencies to be obtained from crates.io, against the information I provide in the Cargo.lock file. So at least when you build it you can be sure that the dependencies you’re getting are the same ones I used myself when I built and tested Hippotat. And there’s only 147 of them (counting indirect dependencies too), so what could possibly go wrong?

Sadly the resulting package build system cannot work with Debian’s best tool for doing clean and controlled builds, sbuild. Under the circumstances, I don’t feel I want to publish any binaries.

comments

prefork-interp - automatic startup time amortisation for all manner of scripts

2022-08-22T23:37:44Z

The problem I had - Mason, so, sadly, FastCGI

Since the update to current Debian stable, the website for YARRG, (a play-aid for Puzzle Pirates which I wrote some years ago), started to occasionally return “Internal Server Error”, apparently due to bug(s) in some FastCGI libraries.

I was using FastCGI because the website is written in Mason, a Perl web framework, and I found that Mason CGI calls were slow. I’m using CGI - yes, trad CGI - via userv-cgi. Running Mason this way would “compile” the template for each HTTP request just when it was rendered, and then throw the compiled version away. The more modern approach of an application server doesn’t scale well to a system which has many web “applications” most of which are very small. The admin overhead of maintaining a daemon, and corresponding webserver config, for each such “service” would be prohibitive, even with some kind of autoprovisioning setup. FastCGI has an interpreter wrapper which seemed like it ought to solve this problem, but it’s quite inconvenient, and often flaky.

I decided I could do better, and set out to eliminate FastCGI from my setup. The result seems to be a success; once I’d done all the hard work of writing prefork-interp, I found the result very straightforward to deploy.

prefork-interp

prefork-interp is a small C program which wraps a script, plus a scripting language library to cooperate with the wrapper program. Together they achieve the following:

Startup cost of the script (loading modules it uses, precompuations, loading and processing of data files, etc.) is paid once, and reused for subsequent invocations of the same script.
Minimal intervention to the script source code:
- one new library to import
- one new call to make from that library, right after the script intialisation is complete
- change to the #! line.
The new “initialisation complete” call turns the program into a little server (a daemon), and then returns once for each actual invocation, each time in a fresh grandchild process.

Features:

Seamless reloading on changes to the script source code (automatic, and configurable).
Concurrency limiting.
Options for distinguishing different configurations of the same script so that they get a server each.
You can run the same script standalone, as a one-off execution, as well as under prefork-interp.
Currently, a script-side library is provided for Perl. I’m pretty sure Python would be fairly straightforward.

Important properties not always satisfied by competing approaches:

Error output (stderr) and exit status from both phases of the script code execution faithfully reproduced to the calling context. Environment, arguments, and stdin/stdout/stderr descriptors, passed through to each invocation.
No polling, other than a long-term idle timeout, so good on laptops (or phones).
Automatic lifetime management of the per-script server, including startup and cleanup. No integration needed with system startup machinery: No explicit management of daemons, init scripts, systemd units, cron jobs, etc.
Useable right away without fuss for CGI programs but also for other kinds of program invocation.
(I believe) reliable handling of unusual states arising from failed invocations or races.

Swans paddling furiously

The implementation is much more complicated than the (apparent) interface.

I won’t go into all the details here (there are some terrifying diagrams in the source code if you really want), but some highlights:

We use an AF_UNIX socket (hopefully in /run/user/UID, but in ~ if not) for rendezvous. We can try to connect without locking, but we must protect the socket with a separate lockfile to avoid two concurrent restart attempts.

We want stderr from the script setup (pre-initialisation) to be delivered to the caller, so the script ought to inherit our stderr and then will need to replace it later. Twice, in fact, because the daemonic server process can’t have a stderr.

When a script is restarted for any reason, any old socket will be removed. We want the old server process to detect that and quit. (If hung about, it would wait for the idle timeout; if this happened a lot - eg, a constantly changing set of services - we might end up running out of pids or something.) Spotting the socket disappearing, without polling, involves use of a library capable of using inotify (or the equivalent elsewhere). Choosing a C library to do this is not so hard, but portable interfaces to this functionality can be hard to find in scripting languages, and also we don’t want every language binding to have to reimplement these checks. So for this purpose there’s a little watcher process, and associated IPC.

When an invoking instance of prefork-interp is killed, we must arrange for the executing service instance to stop reading from its stdin (and, ideally, writing its stdout). Otherwise it’s stealing input from prefork-interp’s successors (maybe the user’s shell)!

Cleanup ought not to depend on positive actions by failing processes, so each element of the system has to detect failures of its peers by means such as EOF on sockets/pipes.

Obtaining prefork-interp

I put this new tool in my chiark-utils package, which is a collection of useful miscellany. It’s available from git.

Currently I make releases by uploading to Debian, where prefork-interp has just hit Debian unstable, in chiark-utils 7.0.0.

Support for other scripting languages

I would love Python to be supported. If any pythonistas reading this think you might like to help out, please get in touch. The specification for the protocol, and what the script library needs to do, is documented in the source code

Future plans for chiark-utils

chiark-utils as a whole is in need of some tidying up of its build system and packaging.

I intend to try to do some reorganisation. Currently I think it would be better to organising the source tree more strictly with a directory for each included facility, rather than grouping “compiled” and “scripts” together.

The Debian binary packages should be reorganised more fully according to their dependencies, so that installing a program will ensure that it works.

I should probably move the official git repo from my own git+gitweb to a forge (so we can have MRs and issues and so on).

And there should be a lot more testing, including Debian autopkgtests.

edited 2022-08-23 10:30 +01:00 to improve the formatting

comments

dkim-rotate - rotation and revocation of DKIM signing keys

2022-08-08T00:20:56Z

Background

Internet email is becoming more reliant on DKIM, a scheme for having mail servers cryptographically sign emails. The Big Email providers have started silently spambinning messages that lack either DKIM signatures, or SPF. DKIM is arguably less broken than SPF, so I wanted to deploy it.

But it has a problem: if done in a naive way, it makes all your emails non-repudiable, forever. This is not really a desirable property - at least, not desirable for you, although it can be nice for someone who (for example) gets hold of leaked messages obtained by hacking mailboxes.

This problem was described at some length in Matthew Green’s article Ok Google: please publish your DKIM secret keys. Following links from that article does get you to a short script to achieve key rotation but it had a number of problems, and wasn’t useable in my context.

dkim-rotate

So I have written my own software for rotating and revoking DKIM keys: dkim-rotate.

I think it is a good solution to this problem, and it ought to be deployable in many contexts (and readily adaptable to those it doesn’t already support).

Here’s the feature list taken from the README:

Leaked emails become unattestable (plausibily deniable) within a few days — soon after the configured maximum email propagation time.
Mail domain DNS configuration can be static, and separated from operational DKIM key rotation. Domain owner delegates DKIM configuration to mailserver administrator, so that dkim-rotate does not need to edit your mail domain’s zone.
When a single mail server handles multiple mail domains, only a single dkim-rotate instance is needed.
Supports situations where multiple mail servers may originate mails for a single mail domain.
DNS zonefile remains small; old keys are published via a webserver, rather than DNS.
Supports runtime (post-deployment) changes to tuning parameters and configuration settings. Careful handling of errors and out-of-course situations.
Convenient outputs: a standard DNS zonefile; easily parseable settings for the MTA; and, a directory of old keys directly publishable by a webserver.

Complications

It seems like it should be a simple problem. Keep N keys, and every day (or whatever), generate and start using a new key, and deliberately leak the oldest private key.

But, things are more complicated than that. Considerably more complicated, as it turns out.

I didn’t want the DKIM key rotation software to have to edit the actual DNS zones for each relevant mail domain. That would tightly entangle the mail server administration with the DNS administration, and there are many contexts (including many of mine) where these roles are separated.

The solution is to use DNS aliases (CNAME). But, now we need a fixed, relatively small, set of CNAME records for each mail domain. That means a fixed, relatively small set of key identifiers (“selectors” in DKIM terminology), which must be used in rotation.

We don’t want the private keys to be published via the DNS because that makes an ever-growing DNS zone, which isn’t great for performance; and, because we want to place barriers in the way of processes which might enumerate the set of keys we use (and the set of keys we have leaked) and keep records of what status each key had when. So we need a separate publication channel - for which a webserver was the obvious answer.

We want the private keys to be readily noticeable and findable by someone who is verifying an alleged leaked email dump, but to be hard to enumerate. (One part of the strategy for this is to leave a note about it, with the prospective private key url, in the email headers.)

The key rotation operations are more complicated than first appears, too. The short summary, above, neglects to consider the fact that DNS updates have a nonzero propagation time: if you change the DNS, not everyone on the Internet will experience the change immediately. So as well as a timeout for how long it might take an email to be delivered (ie, how long the DKIM signature remains valid), there is also a timeout for how long to wait after updating the DNS, before relying on everyone having got the memo. (This same timeout applies both before starting to sign emails with a new key, and before deliberately compromising a key which has been withdrawn and deadvertised.)

Updating the DNS, and the MTA configuration, are fallible operations. So we need to cope with out-of-course situations, where a previous DNS or MTA update failed. In that case, we need to retry the failed update, and not proceed with key rotation. We mustn’t start the timer for the key rotation until the update has been implemented.

The rotation script will usually be run by cron, but it might be run by hand, and when it is run by hand it ought not to “jump the gun” and do anything “too early” (ie, before the relevant timeout has expired). cron jobs don’t always run, and don’t always run at precisely the right time. (And there’s daylight saving time, to consider, too.)

So overall, it’s not sufficient to drive the system via cron and have it proceed by one unit of rotation on each run.

And, hardest of all, I wanted to support post-deployment configuration changes, while continuing to keep the whole the system operational. Otherwise, you have to bake in all the timing parameters right at the beginning and can’t change anything ever. So for example, I wanted to be able to change the email and DNS propagation delays, and even the number of selectors to use, without adversely affecting the delivery of already-sent emails, and without having to shut anything down.

I think I have solved these problems.

The resulting system is one which keeps track of the timing constraints, and the next event which might occur, on a per-key basis. It calculates on each run, which key(s) can be advanced to the next stage of their lifecycle, and performs the appropriate operations. The regular key update schedule is then an emergent property of the config parameters and cron job schedule. (I provide some example config.)

Exim

Integrating dkim-rotate itself with Exim was fairly easy. The lsearch lookup function can be used to fish information out of a suitable data file maintained by dkim-rotate.

But a final awkwardness was getting Exim to make the right DKIM signatures, at the right time.

When making a DKIM signature, one must choose a signing authority domain name: who should we claim to be? (This is the “SDID” in DKIM terms.) A mailserver that handles many different mail domains will be able to make good signatures on behalf of many of them. It seems to me that domain to be the mail domain in the From: header of the email. (The RFC doesn’t seem to be clear on what is expected.) Exim doesn’t seem to have anything builtin to do that.

And, you only want to DKIM-sign emails that are originated locally or from trustworthy sources. You don’t want to DKIM-sign messages that you received from the global Internet, and are sending out again (eg because of an email alias or mailing list). In theory if you verify DKIM on all incoming emails, you could avoid being fooled into signing bad emails, but rejecting all non-DKIM-verified email would be a very strong policy decision. Again, Exim doesn’t seem to have cooked machinery.

The resulting Exim configuration parameters run to 22 lines, and because they’re parameters to an existing config item (the smtp transport) they can’t even easily be deployed as a drop-in file via Debian’s “split config” Exim configuration scheme.

(I don’t know if the file written for Exim’s use by dkim-rotate would be suitable for other MTAs, but this part of dkim-rotate could easily be extended.)

Conclusion

I have today released dkim-rotate 0.4, which is the first public release for general use.

I have it deployed and working, but it’s new so there may well be bugs to work out.

If you would like to try it out, you can get it via git from Debian Salsa. (Debian folks can also find it freshly in Debian unstable.)

comments

chiark’s skip-skip-cross-up-grade

2022-07-19T19:43:18Z

Two weeks ago I upgraded chiark from Debian jessie i386 to bullseye amd64, after nearly 30 years running Debian i386. This went really quite well, in fact!

Background

chiark is my “colo” - a server I run, which lives in a data centre in London. It hosts ~200 users with shell accounts, various websites and mailing lists, moderators for a number of USENET newsgroups, and countless other services. chiark’s internal setup is designed to enable my users to do a maximum number of exciting things with a minimum of intervention from me.

chiark’s OS install dates to 1993, when I installed Debian 0.93R5, the first version of Debian to advertise the ability to be upgraded without reinstalling. I think that makes it one of the oldest Debian installations in existence.

Obviously it’s had several new hardware platforms too. (There was a prior install of Linux on the initial hardware, remnants of which can maybe still be seen in some obscure corners of chiark’s /usr/local.)

chiark’s install is also at the very high end of the installation complexity, and customisation, scale: reinstalling it completely would be an enormous amount of work. And it’s unique.

chiark’s upgrade history

chiark’s last major OS upgrade was to jessie (Debian 8, released in April 2015). That was in 2016. Since then we have been relying on Debian’s excellent security support posture, and the Debian LTS and more recently Freexian’s Debian ELTS projects and some local updates, The use of ELTS - which supports only a subset of packages - was particularly uncomfortable.

Additionally, chiark was installed with 32-bit x86 Linux (Debian i386), since that was what was supported and available at the time. But 32-bit is looking very long in the tooth.

Why do a skip upgrade

So, I wanted to move to the fairly recent stable release - Debian 11 (bullseye), which is just short of a year old. And I wanted to “crossgrade” (as its called) to 64-bit.

In the past, I have found I have had greater success by doing “direct” upgrades, skipping intermediate releases, rather than by following the officially-supported path of going via every intermediate release.

Doing a skip upgrade avoids exposure to any packaging bugs which were present only in intermediate release(s). Debian does usually fix bugs, but Debian has many cautious users, so it is not uncommon for bugs to be found after release, and then not be fixed until the next one.

A skip upgrade avoids the need to try to upgrade to already-obsolete releases (which can involve messing about with multiple snapshots from snapshot.debian.org. It is also significantly faster and simpler, which is important not only because it reduces downtime, but also because it removes opportunities (and reduces the time available) for things to go badly.

One downside is that sometimes maintainers aggressively remove compatibility measures for older releases. (And compatibililty packages are generally removed quite quickly by even cautious maintainers.) That means that the sysadmin who wants to skip-upgrade needs to do more manual fixing of things that haven’t been dealt with automatically. And occasionally one finds compatibility problems that show up only when mixing very old and very new software, that no-one else has seen.

Crossgrading

Crossgrading is fairly complex and hazardous. It is well supported by the low level tools (eg, dpkg) but the higher-level packaging tools (eg, apt) get very badly confused.

Nowadays the system is so complex that downloading things by hand and manually feeding them to dpkg is impractical, other than as a very occasional last resort.

The approach, generally, has been to set the system up to “want to” be the new architecture, run apt in a download-only mode, and do the package installation manually, with some fixing up and retrying, until the system is coherent enough for apt to work.

This is the approach I took. (In current releases, there are tools that will help but they are only in recent releases and I wanted to go direct. I also doubted that they would work properly on chiark, since it’s so unusual.)

Peril and planning

Overall, this was a risky strategy to choose. The package dependencies wouldn’t necessarily express all of the sequencing needed. But it still seemed that if I could come up with a working recipe, I could do it.

I restored most of one of chiark’s backups onto a scratch volume on my laptop. With the LVM snapshot tools and chroots. I was able to develop and test a set of scripts that would perform the upgrade. This was a very effective approach: my super-fast laptop, with local caches of the package repositories, was able to do many “edit, test, debug” cycles.

My recipe made heavy use of snapshot.debian.org, to make sure that it wouldn’t rot between testing and implementation.

When I had a working scheme, I told my users about the planned downtime. I warned everyone it might take even 2 or 3 days. I made sure that my access arrangemnts to the data centre were in place, in case I needed to visit in person. (I have remote serial console and power cycler access.)

Reality - the terrible rescue install

My first task on taking the service down was the check that the emergency rescue installation worked: chiark has an ancient USB stick in the back, which I can boot to from the BIOS. The idea being that many things that go wrong could be repaired from there.

I found that that install was too old to understand chiark’s storage arrangements. mdadm tools gave very strange output. So I needed to upgrade it. After some experiments, I rebooted back into the main install, bringing chiark’s service back online.

I then used the main install of chiark as a kind of meta-rescue-image for the rescue-image. The process of getting the rescue image upgraded (not even to amd64, but just to something not totally ancient) was fraught. Several times I had to rescue it by copying files in from the main install outside. And, the rescue install was on a truly ancient 2G USB stick which was terribly terribly slow, and also very small.

I hadn’t done any significant planning for this subtask, because it was low-risk: there was little way to break the main install. Due to all these adverse factors, sorting out the rescue image took five hours.

If I had known how long it would take, at the beginning, I would have skipped it. 5 hours is more than it would have taken to go to London and fix something in person.

Reality - the actual core upgrade

I was able to start the actual upgrade in the mid-afternoon. I meticulously checked and executed the steps from my plan.

The terrifying scripts which sequenced the critical package updates ran flawlessly. Within an hour or so I had a system which was running bullseye amd64, albeit with many important packages still missing or unconfigured.

So I didn’t need the rescue image after all, nor to go to the datacentre.

Fixing all the things

Then I had to deal with all the inevitable fallout from an upgrade.

Notable incidents:

exim4 has a new tainting system

This is to try to help the sysadmin avoid writing unsafe string interpolations. (“Little Bobby Tables.”) This was done by Exim upstream in a great hurry as part of a security response process.

The new checks meant that the mail configuration did not work at all. I had to turn off the taint check completely. I’m fairly confident that this is correct, because I am hyper-aware of quoting issues and all of my configuration is written to avoid the problems that tainting is supposed to avoid.

One particular annoyance is that the approach taken for sqlite lookups makes it totally impossible to use more than one sqlite database. I think the sqlite quoting operator which one uses to interpolate values produces tainted output? I need to investigate this properly.

LVM now ignores PVs which are directly contained within LVs by default

chiark has LVM-on-RAID-on-LVM. This generally works really well.

However, there was one edge case where I ended up without the intermediate RAID layer. The result is LVM-on-LVM.

But recent versions of the LVM tools do not look at PVs inside LVs, by default. This is to help you avoid corrupting the state of any VMs you have on your system. I didn’t know that at the time, though. All I knew was that LVM was claiming my PV was “unusable”, and wouldn’t explain why.

I was about to start on a thorough reading of the 15,000-word essay that is the commentary in the default /etc/lvm/lvm.conf to try to see if anything was relevant, when I received a helpful tipoff on IRC pointing me to the scan_lvs option.

I need to file a bug asking for the LVM tools to explain why they have declared a PV unuseable.

apache2’s default config no longer read one of my config files

I had to do a merge (of my changes vs the maintainers’ changes) for /etc/apache2/apache2.conf. When doing this merge I failed to notice that the file /etc/apache2/conf.d/httpd.conf was no longer included by default. My merge dropped that line. There were some important things in there, and until I found this the webserver was broken.

`dpkg --skip-same-version` DTWT during a crossgrade

(This is not a “fix all the things” - I found it when developing my upgrade process.)

When doing a crossgrade, one often wants to say to dpkg “install all these things, but don’t reinstall things that have already been done”. That’s what --skip-same-version is for.

However, the logic had not been updated as part of the work to support multiarch, so it was wrong. I prepared a patched version of dpkg, and inserted it in the appropriate point in my prepared crossgrade plan.

The patch is now filed as bug #1014476 against dpkg upstream

Mailman

Mailman is no longer in bullseye. It’s only available in the previous release, buster.

bullseye has Mailman 3 which is a totally different system - requiring basically, a completely new install and configuration. To even preserve existing archive links (a very important requirement) is decidedly nontrivial.

I decided to punt on this whole situation. Currently chiark is running buster’s version of Mailman. I will have to deal with this at some point and I’m not looking forward to it.

Python

Of course that Mailman is Python 2. The Python project’s extremely badly handled transition includes a recommendation to change the meaning of #!/usr/bin/python from Python 2, to Python 3.

But Python 3 is a new language, barely compatible with Python 2 even in the most recent iterations of both, and it is usual to need to coinstall them.

Happily Debian have provided the python-is-python2 package to make things work sensibly, albeit with unpleasant imprecations in the package summary description.

USENET news

Oh my god. INN uses many non-portable data formats, which just depend on your C types. And there are complicated daemons, statically linked libraries which cache on-disk data, and much to go wrong.

I had numerous problems with this, and several outages and malfunctions. I may write about that on a future occasion.

(edited 2022-07-20 11:36 +01:00 and 2022-07-30 12:28+01:00 to fix typos)

comments

Otter (game server) 1.0.0 released

2022-04-02T15:39:52Z

I have just released Otter 1.0.0.

Recap: what is Otter

Otter is my game server for arbitrary board games. Unlike most online game systems. It does not know (nor does it need to know) the rules of the game you are playing. Instead, it lets you and your friends play with common tabletop/boardgame elements such as hands of cards, boards, and so on. So it’s something like a “tabletop simulator” (but it does not have any 3D, or a physics engine, or anything like that).

There are provided game materials and templates for Penultima, Mao, and card games in general.

Otter also supports uploadable game bundles, which allows users to add support for additional games - and this can be done without programming.

For more information, see the online documentation. There are a longer intro and some screenshots in my 2021 introductory blog post about Otter

Releasing 1.0.0

I’m calling this release 1.0.0 because I think I can now say that its quality, reliability and stability is suitable for general use. In particular, Otter now builds on Stable Rust, which makes it a lot easier to install and maintain.

Switching web framework, and async Rust

I switched Otter from the Rocket web framework to Actix. There are things to prefer about both systems, and I still have a soft spot for Rocket. But ultimately I needed a framework which was fully released and supported for use with Stable Rust.

There are few if any Rust web frameworks that are not async. This is rather a shame. Async Rust is a considerably more awkward programming environment than ordinary non-async Rust. I don’t want to digress into a litany of complaints, but suffice it to say that while I really love Rust, my views on async Rust are considerably more mixed.

Future plans

In the near future I plan to add a couple of features to better support some particular games: currency-like resources, and a better UI for dice-like randomness.

In the longer term, Otter’s, installation and account management arrangements are rather unsophisticated and un-webby. There is not currently any publicly available instance for you to try it out without installing it on a machine of your own. There’s not even any provided binaries: you must built Otter yourself. I hope to be able to improve this situation but it involves dealing with cloud CI and containers and so-on, which can all be rather unpleasant.

Users on chiark will find an instance of Otter there.

comments

Rooting an Eos Fairphone 4

2022-02-23T23:44:11Z

Last week I received (finally) my Fairphone 4, supplied with a de-googled operating system, which I had ordered from the E Foundation’s shop in December. (I’m am very hard on hardware and my venerable Fairphone 2 is really on its last legs.)

I expect to have full control over the software on any computing device I own which is as complicated, capable, and therefore, hazardous, as a mobile phone. Unfortunately the Eos image (they prefer to spell it “/e/ os”, srsly!) doesn’t come with a way to get root without taking fairly serious measures including unlocking the bootloader. Unlocking the bootloader wouldn’t be desirable for me but I can’t live without root. So.

I started with these helpful instructions: https://forum.xda-developers.com/t/fairphone-4-root.4376421/

I found the whole process a bit of a trial, and I thought I would write down what I did. But, it’s not straightforward, at least for someone like me who only has a dim understanding of all this Android stuff. Unfortunately, due to the number of missteps and restarts, what I actually did is not really a sensible procedure. So here is a retcon of a process I think will work:

Unlock the bootloader

The E Foundation provide instructions for unlocking the bootloader on a stock FP4, here https://doc.e.foundation/devices/FP4/install and they seem applicable to the “Murena” phone supplied with Eos pre-installed, too.

NB tht unlocking the bootloader wipes the phone. So we do it first.

So:

Power on the phone, with no SIM installed
You get a welcome screen.
Skip all things on startup including wifi
Go to the very end of the settings, tap a gazillion times on the phone’s version until you’re a developer
In the developer settings, allow usb debugging
In the developer settings, allow oem bootloader unlocking
Connect a computer via a USB cable, say yes on phone to USB debugging
adb reboot bootloader
The phone will reboot into a texty kind of screen, the bootloader
fastboot flashing unlock
The phone will reboot, back to the welcome screen
Repeat steps 3-9 (maybe not all are necessary)
fastboot flashing unlock_critical
The phone will reboot, back to the welcome screen

Note that although you are running fastboot, you must run this command with the phone in “bootloader” mode, not “fastboot” (aka “fastbootd”) mode. If you run fastboot flashing unlcok from fastboot you just get a “don’t know what you’re talking about”. I found conflicting instructions on what kind of Vulcan nerve pinches could be used to get into which boot modes, and had poor experiences with those. adb reboot bootloader always worked reliably for me.

Some docs say to run fastboot oem unlock; I used flashing. Maybe this depends on the Android tools version.

Initial privacy prep and OTA update

We want to update the supplied phone OS. The build mine shipped with is too buggy to run Magisk, the application we are going to use to root the phone. (With the pre-installed phone OS, Magisk crashes at the “patch boot image” step.) But I didn’t want to let the phone talk to Google, even for the push notifications registration.

From the welcome screen, skip all things except location, date, time. Notably, do not set up wifi
In settings, “microg” section
1. turn off cloud messaging
2. turn off google safetynet
3. turn off google registration (NB you must do this after the other two, because their sliders become dysfunctional after you turn google registration off)
4. turn off both location modules
In settings, location section, turn off allowed location for browser and magic earth
Now go into settings and enable wifi, giving it your wifi details
Tell the phone to update its operating system. This is a big download.

Install Magisk, the root manager

(As a starting point I used these instructions https://www.xda-developers.com/how-to-install-magisk/ and a lot of random forum posts.)

You will need the official boot.img. Bizarrely there doesn’t seem to be a way to obtain this from the phone. Instead, you must download it. You can find it by starting at https://doc.e.foundation/devices/FP4/install which links to https://images.ecloud.global/stable/FP4/. At the time of writing, the most recent version, whose version number seemed to correspond to the OS update I installed above, was IMG-e-0.21-r-20220123158735-stable-FP4.zip.

Download the giant zipfile to your computer
Unzip it to extract boot.img
Copy the file to your phone’s “storage”. Eg, via adb: with the phone booted into the main operating system, using USB debugging, adb push boot.img /storage/self/primary/Download.
On the phone, open the browser, and enter https://f-droid.org. Click on the link to install f-droid. You will need to enable installing apps from the browser (follow the provided flow to the settings, change the setting, and then use Back, and you can do the install). If you wish, you can download the f-droid apk separately on a computer, and verify it with pgp.
Using f-droid, install Magisk. You will need to enable installing apps from f-droid. (I installed Magisk from f-droid because 1. I was going to trust f-droid anyway 2. it has a shorter URL than Magisk’s.)
Open the Magisk app. Tell Magisk to install (Magisk, not the app). There will be only one option: patch boot file. Tell it to patch the boot.img file from before.
Transfer the magisk_patched-THING.img back to your computer (eg via adb pull).
adb reboot bootloader
fastboot boot magisk_patched-THING.img (again, NB, from bootloader mode, not from fastboot mode)
In Magisk you’ll see it shows as installed. But it’s not really; you’ve just booted from an image with it. Ask to install Magisk with “Direct install”.

After you have done all this, I believe that each time you do an over-the-air OS update, you must, between installing the update and rebooting the phone, ask Magisk to “Install to inactive slot (after OTA)”. Presumably if you forget you must do the fastboot boot dance again.

After all this, I was able to use tsu in Termux. There’s a strange behaviour with the root prompt you get apropos Termux’s request for root; I found that it definitely worked if Termux wasn’t the foreground app…

You have to leave the bootloader unlocked. Howwever, as I understand it, the phone’s encryption will still prevent an attacker from hoovering the data out of your phone. The bootloader lock is to prevent someone tricking you into entering the decryption passkey into a trojaned device.

Other things to change

Probably, after you’re done with this, disable installing apps from the Browser. I will install Signal before doing that, since that’s not in f-droid because of mutual distrust between the f-droid and Signal folks. The permission is called “Install unknown apps”.
Turn off “instant apps” aka “open links in apps even if the app is not installed”. OMG WTF BBQ.
Turn off “wifi scanning even if wifi off”. WTF.
I turned off storage manager auto delete, on the grounds that I didn’t know what the phone might think of as “having been backed up”. I can manage my own space use, thanks very much.

There are probably other things to change. I have not yet transferred my Signal account from my old phone. It is possible that Signal will require me to re-enable the google push notifications, but I hope that having disabled them in microg it will be happy to use its own system, as it does on my old phone.

comments

EUDCC QR codes vs NHS “Travel” barcodes vs TAC Verify

2022-02-04T19:43:37Z

The EU Digital Covid Certificate scheme is a format for (digitally signed) vaccination status certificates. Not only EU countries participate - the UK is now a participant in this scheme.

I am currently on my way to go skiing in the French Alps. So I needed a certificate that would be accepted in France. AFAICT the official way to do this is to get the “international” certificate from the NHS, and take it to a French pharmacy who will convert it into something suitably French. (AIUI the NHS “international” barcode is the same regardless of whether you get it via the NHS website, the NHS app, or a paper letter. NB that there is one barcode per vaccine dose so you have to get the right one - probably that means your booster since there’s a 9 month rule!)

I read on an forum somewhere that you could use the French TousAntiCovid app to convert the barcode. So I thought I would try that. The TousAntiCovid is Free Softare and on F-Droid, so I was happy to install and use it for this.

I also used the French TAC Verify app to check to see what barcodes were accepted. (I found an official document addressed to French professionals recommending this as an option for verifying the status of visitors to one’s establishment.) Unfortunately this involves a googlified phone, but one could use a burner phone or ask a friend who’s bitten that bullet already.

I discovered that, indeed:

My own NHS QR code is not accepted by TAC Verify
My own NHS QR code can be loaded into TousAntiCovid, and added to my “wallet” in the app
If I get TousAntiCovid to display that certificate, it shows a visually different QR code which TAC Verify accepts

This made me curious.

I used a QR code reader to decode both barcodes. The decodings were identical! A long string of guff starting HC1:. AIUI it is an encoded JWT. But there was a difference in the framing: Binary Eye reported that the NHS barcode used error correction level “M” (medium, aka 15%). The TousAntiCovid barcode used level “L” (low, 7%).

I had my QR code software regenerate a QR code at level “M” for the data from the TousAntiCovid code. The result was a QR code which is identical (pixel-wise) to the one from the NHS.

So the only difference is the error correction level. Curiously, both “L” (low, generated by TousAntiCovid, accepted by TAC Verify) and “M” (medium, generated by NHS, rejected by TAC Verify) are lower than the “Q” (25") recommended by what I think is the specification.

This is all very odd. But the upshot is that I think you can convert the NHS “international” barcode into something that should work in France simply by passing it through any QR code software to re-encode it at error correction level “L” (7%). But if you’re happy to use the TousAntiCovid app it’s probably a good way to store them.

I guess I’ll find out when I get to France if the converted NHS barcodes work in real establishments. Thanks to the folks behind sanipasse.fr for publishing some helpful backround info and operating a Free Software backed public verification service.

Footnote

To compare the QR codes pixelwise, I roughly cropped the NHS PDF image using a GUI tool, and then on each of the two images used pnmcrop (to trim the border), pnmscale (to rescale the one-pixel-per-pixel output from Binary Eye) and pnmarith -difference to compare them (producing a pretty squirgly image showing just the pixel edges due to antialiasing).

comments

Debian’s approach to Rust - Dependency handling

2022-01-03T18:35:37Z

tl;dr: Faithfully following upstream semver, in Debian package dependencies, is a bad idea.

Introduction

I have been involved in Debian for a very long time. And I’ve been working with Rust for a few years now. Late last year I had cause to try to work on Rust things within Debian.

When I did, I found it very difficult. The Debian Rust Team were very helpful. However, the workflow and tooling require very large amounts of manual clerical work - work which it is almost impossible to do correctly since the information required does not exist. I had wanted to package a fairly straightforward program I had written in Rust, partly as a learning exercise. But, unfortunately, after I got stuck in, it looked to me like the effort would be wildly greater than I was prepared for, so I gave up.

Since then I’ve been thinking about what I learned about how Rust is packaged in Debian. I think I can see how to fix some of the problems. Although I don’t want to go charging in and try to tell everyone how to do things, I felt I ought at least to write up my ideas. Hence this blog post, which may become the first of a series.

This post is going to be about semver handling. I see problems with other aspects of dependency handling and source code management and traceability as well, and of course if my ideas find favour in principle, there are a lot of details that need to be worked out, including some kind of transition plan.

How Debian packages Rust, and build vs runtime dependencies

Today I will be discussing almost entirely build-dependencies; Rust doesn’t (yet?) support dynamic linking, so built Rust binaries don’t have Rusty dependencies.

However, things are a bit confusing because even the Debian “binary” packages for Rust libraries contain pure source code. So for a Rust library package, “building” the Debian binary package from the Debian source package does not involve running the Rust compiler; it’s just file-copying and format conversion. The library’s Rust dependencies do not need to be installed on the “build” machine for this.

So I’m mostly going to be talking about Depends fields, which are Debian’s way of talking about runtime dependencies, even though they are used only at build-time. The way this works is that some ultimate leaf package (which is supposed to produce actual executable code) Build-Depends on the libraries it needs, and those Depends on their under-libraries, so that everything needed is installed.

What do dependencies mean and what are they for anyway?

In systems where packages declare dependencies on other packages, it generally becomes necessary to support “versioned” dependencies. In all but the most simple systems, this involves an ordering (or similar) on version numbers and a way for a package A to specify that it depends on certain versions of B.

Both Debian and Rust have this. Rust upstream crates have version numbers and can specify their dependencies according to semver. Debian’s dependency system can represent that.

So it was natural for the designers of the scheme for packaging Rust code in Debian to simply translate the Rust version dependencies to Debian ones. However, while the two dependency schemes seem equivalent in the abstract, their concrete real-world semantics are totally different.

These different package management systems have different practices and different meanings for dependencies. (Interestingly, the Python world also has debates about the meaning and proper use of dependency versions.)

The epistemological problem

Consider some package A which is known to depend on B. In general, it is not trivial to know which versions of B will be satisfactory. I.e., whether a new B, with potentially-breaking changes, will actually break A.

Sometimes tooling can be used which calculates this (eg, the Debian shlibdeps system for runtime dependencies) but this is unusual - especially for build-time dependencies. Which versions of B are OK can normally only be discovered by a human consideration of changelogs etc., or by having a computer try particular combinations.

Few ecosystems with dependencies, in the Free Software community at least, make an attempt to precisely calculate the versions of B that are actually required to build some A. So it turns out that there are three cases for a particular combination of A and B: it is believed to work; it is known not to work; and: it is not known whether it will work.

And, I am not aware of any dependency system that has an explicit machine-readable representation for the “unknown” state, so that they can say something like “A is known to depend on B; versions of B before v1 are known to break; version v2 is known to work”. (Sometimes statements like that can be found in human-readable docs.)

That leaves two possibilities for the semantics of a dependency A depends B, version(s) V..W: Precise: A will definitely work if B matches V..W, and Optimistic: We have no reason to think B breaks with any of V..W.

At first sight the latter does not seem useful, since how would the package manager find a working combination? Taking Debian as an example, which uses optimistic version dependencies, the answer is as follows: The primary information about what package versions to use is not only the dependencies, but mostly in which Debian release is being targeted. (Other systems using optimistic version dependencies could use the date of the build, i.e. use only packages that are “current”.)

	Precise	Optimistic
People involved in version management	Package developers, downstream developers/users.	Package developers, downstream developer/users, distribution QA and release managers.
Package developers declare versions V and dependency ranges V..W so that	It definitely works.	A wide range of B can satisfy the declared requirement.
The principal version data used by the package manager	Only dependency versions.	Contextual, eg, Releases - set(s) of packages available.
Version dependencies are for	Selecting working combinations (out of all that ever existed).	Sequencing (ordering) of updates; QA.
Expected use pattern by a downstream	Downstream can combine any declared-good combination.	Use a particular release of the whole system. Mixing-and-matching requires additional QA and remedial work.
Downstreams are protected from breakage by	Pessimistically updating versions and dependencies whenever anything might go wrong.	Whole-release QA.
A substantial deployment will typically contain	Multiple versions of many packages.	A single version of each package, except where there are actual incompatibilities which are too hard to fix.
Package updates are driven by	Top-down: Depending package updates the declared metadata.	Bottom-up: Depended-on package is updated in the repository for the work-in-progress release.

So, while Rust and Debian have systems that look superficially similar, they contain fundamentally different kinds of information. Simply representing the Rust versions directly into Debian doesn’t work.

What is currently done by the Debian Rust Team is to manually patch the dependency specifications, to relax them. This is very labour-intensive, and there is little automation supporting either decisionmaking or actually applying the resulting changes.

What to do

Desired end goal

To update a Rust package in Debian, that many things depend on, one need simply update that package.

Debian’s sophisticated build and CI infrastructure will try building all the reverse-dependencies against the new version. Packages that actually fail against the new dependency are flagged as suffering from release-critical problems.

Debian Rust developers then update those other packages too. If the problems turn out to be too difficult, it is possible to roll back.

If a problem with a depending packages is not resolved in a timely fashion, priority is given to updating core packages, and the depending package falls by the wayside (since it is empirically unmaintainable, given available effort).

There is no routine manual patching of dependency metadata (or of anything else).

Radical proposal

Debian should not precisely follow upstream Rust semver dependency information. Instead, Debian should optimistically try the combinations of packages that we want to have. The resulting breakages will be discovered by automated QA; they will have to be fixed by manual intervention of some kind, but usually, simply updating the depending package will be sufficient.

This no longer ensures (unlike the upstream Rust scheme) that the result is expected to build and work if the dependencies are satisfied. But as discussed, we don’t really need that property in Debian. More important is the new property we gain: that we are able to mix and match versions that we find work in practice, without a great deal of manual effort.

Or to put it another way, in Debian we should do as a Rust upstream maintainer does when they do the regular “update dependencies for new semvers” task: we should update everything, see what breaks, and fix those.

(In theory a Rust upstream package maintainer is supposed to do some additional checks or something. But the practices are not standardised and any checks one does almost never reveal anything untoward, so in practice I think many Rust upstreams just update and see what happens. The Rust upstream community has other mechanisms - often, reactive ones - to deal with any problems. Debian should subscribe to those same information sources, eg RustSec.)

Nobbling cargo

Somehow, when cargo is run to build Rust things against these Debian packages, cargo’s dependency system will have to be overridden so that the version of the package that is actually selected by Debian’s package manager is used by cargo without complaint.

We probably don’t want to change the Rust version numbers of Debian Rust library packages, so this should be done by either presenting cargo with an automatically-massaged Cargo.toml where the dependency version restrictions are relaxed, or by using a modified version of cargo which has special option(s) to relax certain dependencies.

Handling breakage

Rust packages in Debian should already be provided with autopkgtests so that ci.debian.net will detect build breakages. Build breakages will stop the updated dependency from migrating to the work-in-progress release, Debian testing.

To resolve this, and allow forward progress, we will usually upload a new version of the dependency containing an appropriate Breaks, and either file an RC bug against the depending package, or update it. This can be done after the upload of the base package.

Thus, resolution of breakage due to incompatibilities will be done collaboratively within the Debian archive, rather than ad-hoc locally. And it can be done without blocking.

My proposal prioritises the ability to make progress in the core, over stability and in particular over retaining leaf packages. This is not Debian’s usual approach but given the Rust ecosystem’s practical attitudes to API design, versioning, etc., I think the instability will be manageable. In practice fixing leaf packages is not usually really that hard, but it’s still work and the question is what happens if the work doesn’t get done. After all we are always a shortage of effort - and we probably still will be, even if we get rid of the makework clerical work of patching dependency versions everywhere (so that usually no work is needed on depending packages).

Exceptions to the one-version rule

There will have to be some packages that we need to keep multiple versions of. We won’t want to update every depending package manually when this happens. Instead, we’ll probably want to set a version number split: rdepends which want version <X will get the old one.

Details - a sketch

I’m going to sketch out some of the details of a scheme I think would work. But I haven’t thought this through fully. This is still mostly at the handwaving stage. If my ideas find favour, we’ll have to do some detailed review and consider a whole bunch of edge cases I’m glossing over.

The dependency specification consists of two halves: the depending .deb‘s Depends (or, for a leaf package, Build-Depends) and the base .deb’ Version and perhaps Breaks and Provides.

Even though libraries vastly outnumber leaf packages, we still want to avoid updating leaf Debian source packages simply to bump dependencies.

Dependency encoding proposal

Compared to the existing scheme, I suggest we implement the dependency relaxation by changing the depended-on package, rather than the depending one.

So we retain roughly the existing semver translation for Depends fields. But we drop all local patching of dependency versions.

Into every library source package we insert a new Debian-specific metadata file declaring the earliest version that we uploaded. When we translate a library source package to a .deb, the “binary” package build adds Provides for every previous version.

The effect is that when one updates a base package, the usual behaviour is to simply try to use it to satisfy everything that depends on that base package. The Debian CI will report the build or test failures of all the depending packages which the API changes broke.

We will have a choice, then:

Breakage handling - update broken depending packages individually

If there are only a few packages that are broken, for each broken dependency, we add an appropriate Breaks to the base binary package. (The version field in the Breaks should be chosen narrowly, so that it is possible to resolve it without changing the major version of the dependency, eg by making a minor source change.)

When can then do one of the following:

Update the dependency from upstream, to a version which works with the new base. (Assuming there is one.) This should be the usual response.
Fix the dependency source code so that builds and works with the new base package. If this wasn’t just a backport of an upstream change, we should send our fix upstream. (We should prefer to update the whole package, than to backport an API adjustment.)
File an RC bug against the dependency (which will eventually trigger autoremoval), or preemptively ask for the Debian release managers to remove the dependency from the work-in-progress release.

Breakage handling - declare new incompatible API in Debian

If the API changes are widespread and many dependencies are affected, we should represent this by changing the in-Debian-source-package metadata to arrange for fewer Provides lines to be generated - withdrawing the Provides lines for earlier APIs.

Hopefully examination of the upstream changelog will show what the main compat break is, and therefore tell us which Provides we still want to retain.

This is like declaring Breaks for all the rdepends. We should do it if many rdepends are affected.

Then, for each rdependency, we must choose one of the responses in the bullet points above. In practice this will often be a mass bug filing campaign, or large update campaign.

Breakage handling - multiple versions

Sometimes there will be a big API rewrite in some package, and we can’t easily update all of the rdependencies because the upstream ecosystem is fragmented and the work involved in reconciling it all is too substantial.

When this happens we will bite the bullet and include multiple versions of the base package in Debian. The old version will become a new source package with a version number in its name.

This is analogous to how key C/C++ libraries are handled.

Downsides of this scheme

The first obvious downside is that assembling some arbitrary set of Debian Rust library packages, that satisfy the dependencies declared by Debian, is no longer necessarily going to work. The combinations that Debian has tested - Debian releases - will work, though. And at least, any breakage will affect only people building Rust code using Debian-supplied libraries.

Another less obvious problem is that because there is no such thing as Build-Breaks (in a Debian binary package), the per-package update scheme may result in no way to declare that a particular library update breaks the build of a particular leaf package. In other words, old source packages might no longer build when exposed to newer versions of their build-dependencies, taken from a newer Debian release. This is a thing that already happens in Debian, with source packages in other languages, though.

Semver violation

I am proposing that Debian should routinely compile Rust packages against dependencies in violation of the declared semver, and ship the results to Debian’s millions of users.

This sounds quite alarming! But I think it will not in fact lead to shipping bad binaries, for the following reasons:

The Rust community strongly values safety (in a broad sense) in its APIs. An API which is merely capable of insecure (or other seriously bad) use is generally considered to be wrong. For example, such situations are regarded as vulnerabilities by the RustSec project, even if there is no suggestion that any actually-broken caller source code exists, let alone that actually-broken compiled code is likely.

The Rust community also values alerting programmers to problems. Nontrivial semantic changes to APIs are typically accompanied not merely by a semver bump, but also by changes to names or types, precisely to ensure that broken combinations of code do not compile.

Or to look at it another way, in Debian we would simply be doing what many Rust upstream developers routinely do: bump the versions of their dependencies, and throw it at the wall and hope it sticks. We can mitigate the risks the same way a Rust upstream maintainer would: when updating a package we should of course review the upstream changelog for any gotchas. We should look at RustSec and other upstream ecosystem tracking and authorship information.

Difficulties for another day

As I said, I see some other issues with Rust in Debian.

I think the library “feature flag” encoding scheme is unnecessary. I hope to explain this in a future essay.
I found Debian’s approach to handling the source code for its Rust packages quite awkward; and, it has some troubling properties. Again, I hope to write about this later.
I get the impression that updating rustc in Debian is a very difficult process. I haven’t worked on this myself and I don’t feel qualified to have opinions about it. I hope others are thinking about how to make things easier.

Thanks all for your attention!

comments

Ian Jackson

tag2upload in the first month of forky

Free Software, internal politics, and governance

Rust is indeed woke

derive-deftly 1.0.0 - Rust derive macros, the easy way

derive-deftly is nearing 1.x - call for review/testing

How to use Rust on Debian (and Ubuntu, etc.)

Don’t use apt-get source; use dgit

DigiSpark (ATTiny85) - Arduino, C, Rust, build systems

Introduction

Arduino IDE

Writing C again

Rust on the DigiSpark

RIIR (Rewrite It In Rust)

An offer of help

Conclusions

All is not lost

DKIM: rotate and publish your keys

Non-repudiation of emails is undesirable

Advice for all email users

How to tell by looking at email headers

Testing verification of new and old messages

Checking that recent emails are verifiable

Checking old emails cannot be verified

For email admins: announcing dkim-rotate 1.0

Limitation of this approach

Installing Debian bookworm without systemd

Instructions

Discussion

update-initramfs

The apt-get install line

usrmerge

Installer images

Conclusion

Never use git submodules

tl;dr

What is wrong with git submodules

Better alternatives to git submodules

Use git subtree

Just have a monorepo

Use a package management system, and explicit dependencies

Use the multiple repository tool mr

Have your build expect to find the dependency in .., its parent dir

Provide an ad-hoc in-tree script to download the dependency

Yes, really, git submodule is worse than ad-hoc Makefile runes

derive-adhoc: powerful pattern-based derive macros for Rust

tl;dr

Background - Rust’s two (main) macro systems

“Derive by example” with derive-adhoc

A simple example - deriving Clone without inferred trait bounds

Replacing an existing bespoke proc macro - a more complex example

History and acknowledgements

Future plans

Documentation and references about derive-adhoc

SGO (and my) VPN and network access tools - in bookworm

secnet

Hippotat

userv ipif

userv

Acknowledgements

Rust for the Polyglot Programmer, December 2022 edition

Recap about Rust for the Polyglot Programmer

Rust needs #[throws]

tl;dr:

Contents

A recent personal experience in coding style

What is Ok wrapping? Intro to Rust error handling

A minor inconvenience, or a significant distraction?

Idioms for Ok-wrapping - a bestiary

Wrap just a returned variable binding

Wrap a nontrivial return expression

Introduce Ok(()) at the end

Wrap the whole body

Omit the wrap when calling fallible sub-functions

Convert a fallible sub-function error with Ok( ... ?)

Convert a fallible sub-function error with .map_err

What is to be done, then?

Limitations of fehler

Please can we have #[throws] in the Rust language

“Explicitness”

`update-initramfs`

The `apt-get install` line

Use the multiple repository tool `mr`

Have your build expect to find the dependency in `..`, its parent dir

“Derive by example” with `derive-adhoc`

A simple example - deriving `Clone` without inferred trait bounds

Documentation and references about `derive-adhoc`

`userv ipif`

`userv`

What is `Ok` wrapping? Intro to Rust error handling

Idioms for `Ok`-wrapping - a bestiary

Introduce `Ok(())` at the end

Convert a fallible sub-function error with `Ok( ... ?)`

Convert a fallible sub-function error with `.map_err`

Please can we have `#[throws]` in the Rust language

Appendix - examples showning code with `Ok` wrapping is worse than code using `#[throws]`

The `Cursor` concept

The alternative for nodey data structures in safe Rust: `Rc`/`Arc`