Ian Jackson

Debian’s git transition

2025-12-21T23:08:31Z

tl;dr:

There is a Debian git transition plan. It’s going OK so far but we need help, especially with outreach and updating Debian’s documentation.

Goals of the Debian git transition project
- Achievements so far, and current status
Core engineering principle
Distributing the source code as git
- Tracking the relevant git data, when changes are made in the legacy Archive
- Why *.dgit.debian.org is not Salsa
Roadmap
- In progress
- Future Technology
Mindshare and adoption - please help!
- A rant about publishing the source code
- Documentation
Personnel
- Thanks

Goals of the Debian git transition project

Everyone who interacts with Debian source code should be able to do so entirely in git.

That means, more specifically:

All examination and edits to the source should be performed via normal git operations.
Source code should be transferred and exchanged as git data, not tarballs. git should be the canonical form everywhere.
Upstream git histories should be re-published, traceably, as part of formal git releases published by Debian.
No-one should have to learn about Debian Source Packages, which are bizarre, and have been obsoleted by modern version control.

This is very ambitious, but we have come a long way!

Achievements so far, and current status

We have come a very long way. But, there is still much to do - especially, the git transition team needs your help with adoption, developer outreach, and developer documentation overhaul.

We’ve made big strides towards goals 1 and 4. Goal 2 is partially achieved: we currently have dual running. Goal 3 is within our reach but depends on widespread adoption of tag2upload (and/or dgit push).

Downstreams and users can obtain the source code of any Debian package in git form. (dgit clone, 2013). They can then work with this source code completely in git, including building binaries, merging new versions, even automatically (eg Raspbian, 2016), and all without having to deal with source packages at all (eg Wikimedia 2025).

A Debian maintainer can maintain their own package entirely in git. They can obtain upstream source code from git, and do their packaging work in git (git-buildpackage, 2006).

Every Debian maintainer can (and should!) release their package from git reliably and in a standard form (dgit push, 2013; tag2upload, 2025). This is not only more principled, but also more convenient, and with better UX, than pre-dgit tooling like dput.

Indeed a Debian maintainer can now often release their changes to Debian, from git, using only git branches (so no tarballs). Releasing to Debian can be simply pushing a signed tag (tag2upload, 2025).

A Debian maintainer can maintain a stack of changes to upstream source code in git (gbp pq 2009). They can even maintain such a delta series as a rebasing git branch, directly buildable, and use normal git rebase style operations to edit their changes, (git-dpm, 2010; git-debrebase, 2018)

An authorised Debian developer can do a modest update to any package in Debian, even one maintained by someone else, working entirely in git in a standard and convenient way (dgit, 2013).

Debian contributors can share their work-in-progress on git forges and collaborate using merge requests, git based code review, and so on. (Alioth, 2003; Salsa, 2018.)

Core engineering principle

The Debian git transition project is based on one core engineering principle:

Every Debian Source Package can be losslessly converted to and from git.

In order to transition away from Debian Source Packages, we need to gateway between the old dsc approach, and the new git approach.

This gateway obviously needs to be bidirectional: source packages uploaded with legacy tooling like dput need to be imported into a canonical git representation; and of course git branches prepared by developers need to be converted to source packages for the benefit of legacy downstream systems (such as the Debian Archive and apt source).

This bidirectional gateway is implemented in src:dgit, and is allowing us to gradually replace dsc-based parts of the Debian system with git-based ones.

Correspondence between dsc and git

A faithful bidirectional gateway must define an invariant:

The canonical git tree, corresponding to a .dsc, is the tree resulting from dpkg-source -x.

This canonical form is sometimes called the “dgit view”. It’s sometimes not the same as the maintainer’s git branch, because many maintainers are still working with “patches-unapplied” git branches. More on this below.

(For 3.0 (quilt) .dscs, the canonical git tree doesn’t include the quilt .pc directory.)

Patches-applied vs patches-unapplied

The canonical git format is “patches applied”. That is:

If Debian has modified the upstream source code, a normal git clone of the canonical branch gives the modified source tree, ready for reading and building.

Many Debian maintainers keep their packages in a different git branch format, where the changes made by Debian, to the upstream source code, are in actual patch files in a debian/patches/ subdirectory.

Patches-applied has a number of important advantages over patches-unapplied:

It is familiar to, and doesn’t trick, outsiders to Debian. Debian insiders radically underestimate how weird “patches-unapplied” is. Even expert software developers can get very confused or even accidentally build binaries without security patches!
Making changes can be done with just normal git commands, eg git commit. Many Debian insiders working with patches-unapplied are still using quilt(1), a footgun-rich contraption for working with patch files!
When developing, one can make changes to upstream code, and to Debian packaging, together, without ceremony. There is no need to switch back and forth between patch queue and packaging branches (as with gbp pq), no need to “commit” patch files, etc. One can always edit every file and commit it with git commit.

The downside is that, with the (bizarre) 3.0 (quilt) source format, the patch files files in debian/patches/ must somehow be kept up to date. Nowadays though, tools like git-debrebase and git-dpm (and dgit for NMUs) make it very easy to work with patches-applied git branches. git-debrebase can deal very ergonomically even with big patch stacks.

(For smaller packages which usually have no patches, plain git merge with an upstream git branch, and a much simpler dsc format, sidesteps the problem entirely.)

Prioritising Debian’s users (and other outsiders)

We want everyone to be able to share and modify the software that they interact with. That means we should make source code truly accessible, on the user’s terms.

Many of Debian’s processes assume everyone is an insider. It’s okay that there are Debian insiders and that people feel part of something that they worked hard to become involved with. But lack of perspective can lead to software which fails to uphold our values.

Our source code practices — in particular, our determination to share properly (and systematically) — are a key part of what makes Debian worthwhile at all. Like Debian’s installer, we want our source code to be useable by Debian outsiders.

This is why we have chosen to privilege a git branch format which is more familiar to the world at large, even if it’s less popular in Debian.

Consequences, some of which are annoying

The requirement that the conversion be bidirectional, lossless, and context-free can be inconvenient.

For example, we cannot support .gitattributes which modify files during git checkin and checkout. .gitattributes cause the meaning of a git tree to depend on the context, in possibly arbitrary ways, so the conversion from git to source package wouldn’t be stable. And, worse, some source packages might not to be representable in git at all.

Another example: Maintainers often have existing git branches for their packages, generated with pre-dgit tooling which is less careful and less principled than ours. That can result in discrepancies between git and dsc, which need to be resolved before a proper git-based upload can succeed.

That some maintainers use patches-unapplied, and some patches-unapplied, means that there has to be some kind of conversion to a standard git representation. Choosing the less-popular patches-applied format as the canonical form, means that many packages need their git representation converted. It also means that user- and outsider-facing branches from {browse,git}.dgit.d.o and dgit clone are not always compatible with maintainer branches on Salsa. User-contributed changes need cherry-picking rather than merging, or conversion back to the maintainer format. The good news is that dgit can automate much of this, and the manual parts are usually easy git operations.

Distributing the source code as git

Our source code management should be normal, modern, and based on git. That means the Debian Archive is obsolete and needs to be replaced with a set of git repositories.

The replacement repository for source code formally released to Debian is *.dgit.debian.org. This contains all the git objects for every git-based upload since 2013, including the signed tag for each released package version.

The plan is that it will contain a git view of every uploaded Debian package, by centrally importing all legacy uploads into git.

Tracking the relevant git data, when changes are made in the legacy Archive

Currently, many critical source code management tasks are done by changes to the legacy Debian Archive, which works entirely with dsc files (and the associated tarballs etc). The contents of the Archive are therefore still an important source of truth. But, the Archive’s architecture means it cannot sensibly directly contain git data.

To track changes made in the Archive, we added the Dgit: field to the .dsc of a git-based upload (2013). This declares which git commit this package was converted from. and where those git objects can be obtained.

Thus, given a Debian Source Package from a git-based upload, it is possible for the new git tooling to obtain the equivalent git objects. If the user is going to work in git, there is no need for any tarballs to be downloaded: the git data could be obtained from the depository using the git protocol.

The signed tags, available from the git depository, have standardised metdata which gives traceability back to the uploading Debian contributor.

Why *.dgit.debian.org is not Salsa

We need a git depository - a formal, reliable and permanent git repository of source code actually released to Debian.

Git forges like Gitlab can be very convenient. But Gitlab is not sufficiently secure, and too full of bugs, to be the principal and only archive of all our source code. (The “open core” business model of the Gitlab corporation, and the constant-churn development approach, are critical underlying problems.)

Our git depository lacks forge features like Merge Requests. But:

It is dependable, both in terms of reliability and security.
It is append-only: once something is pushed, it is permanently recorded.
Its access control is precisely that of the Debian Archive.
Its ref namespace is standardised and corresponds to Debian releases.
Pushes are authorised by PGP signatures, not ssh keys, so traceable.

The dgit git depository outlasted Alioth and it may well outlast Salsa.

We need both a good forge, and the *.dgit.debian.org formal git depository.

Roadmap

In progress

Right now we are quite focused on tag2upload.

We are working hard on eliminating the remaining issues that we feel need to be addressed before declaring the service out of beta.

Future Technology

Whole-archive dsc importer

Currently, the git depository only has git data for git-based package updates (tag2upload and dgit push). Legacy dput-based uploads are not currently present there. This means that the git-based and legacy uploads must be resolved client-side, by dgit clone.

We will want to start importing legacy uploads to git.

Then downstreams and users will be able to get the source code for any package simply with git clone, even if the maintainer is using legacy upload tools like dput.

Support for git-based uploads to security.debian.org

Security patching is a task which would particularly benefit from better and more formal use of git. git-based approaches to applying and backporting security patches are much more convenient than messing about with actual patch files.

Currently, one can use git to help prepare a security upload, but it often involves starting with a dsc import (which lacks the proper git history) or figuring out a package maintainer’s unstandardised git usage conventions on Salsa.

And it is not possible to properly perform the security release as git.

Internal Debian consumers switch to getting source from git

Buildds, QA work such as lintian checks, and so on, could be simpler if they don’t need to deal with source packages.

And since git is actually the canonical form, we want them to use it directly.

Problems for the distant future

For decades, Debian has been built around source packages. Replacing them is a long and complex process. Certainly source packages are going to continue to be supported for the foreseeable future.

There are no doubt going to be unanticipated problems. There are also foreseeable issues: for example, perhaps there are packages that work very badly when represented in git. We think we can rise to these challenges as they come up.

Mindshare and adoption - please help!

We and our users are very pleased with our technology. It is convenient and highly dependable.

dgit in particular is superb, even if we say so ourselves. As technologists, we have been very focused on building good software, but it seems we have fallen short in the marketing department.

A rant about publishing the source code

git is the preferred form for modification.

Our upstreams are overwhelmingly using git. We are overwhelmingly using git. It is a scandal that for many packages, Debian does not properly, formally and officially publish the git history.

Properly publishing the source code as git means publishing it in a way that means that anyone can automatically and reliably obtain and build the exact source code corresponding to the binaries. The test is: could you use that to build a derivative?

Putting a package in git on Salsa is often a good idea, but it is not sufficient. No standard branch structure git on Salsa is enforced, nor should it be (so it can’t be automatically and reliably obtained), the tree is not in a standard form (so it can’t be automatically built), and is not necessarily identical to the source package. So Vcs-Git fields, and git from Salsa, will never be sufficient to make a derivative.

Debian is not publishing the source code!

The time has come for proper publication of source code by Debian to no longer be a minority sport. Every maintainer of a package whose upstream is using git (which is nearly all packages nowadays) should be basing their work on upstream git, and properly publishing that via tag2upload or dgit.

And it’s not even difficult! The modern git-based tooling provides a far superior upload experience.

A common misunderstanding

dgit push is not an alternative to gbp pq or quilt. Nor is tag2upload. These upload tools complement your existing git workflow. They replace and improve source package building/signing and the subsequent dput. If you are using one of the usual git layouts on salsa, and your package is in good shape, you can adopt tag2upload and/or dgit push right away.

git-debrebase is distinct and does provides an alternative way to manage your git packaging, do your upstream rebases, etc.

Documentation

Debian’s documentation all needs to be updated, including particularly instructions for packaging, to recommend use of git-first workflows. Debian should not be importing git-using upstreams’ “release tarballs” into git. (Debian outsiders who discover this practice are typically horrified.) We should use only upstream git, work only in git, and properly release (and publish) in git form.

We, the git transition team, are experts in the technology, and can provide good suggestions. But we do not have the bandwidth to also engage in the massive campaigns of education and documentation updates that are necessary — especially given that (as with any programme for change) many people will be sceptical or even hostile.

So we would greatly appreciate help with writing and outreach.

Personnel

We consider ourselves the Debian git transition team.

Currently we are:

Ian Jackson. Author and maintainer of dgit and git-debrebase. Co-creator of tag2upload. Original author of dpkg-source, and inventor in 1996 of Debian Source Packages. Alumnus of the Debian Technical Committee.
Sean Whitton. Co-creator of the tag2upload system; author and maintainer of git-debpush. Co-maintainer of dgit. Debian Policy co-Editor. Former Chair of the Debian Technical Committee.

We wear the following hats related to the git transition:

Maintainers of src:dgit
tag2upload Delegates; operators of the tag2upload service.
service operators of the git depository *.dgit.debian.org.

You can contact us:

By email: Ian Jackson ijackson@chiark.greenend.org.uk; Sean Whitton spwhitton@spwhitton.name; git-debpush@packages.d.o.
By filing bugs in the Debian Bug System against src:dgit.
On OFTC IRC, as Diziet and spwhitton.

We do most of our heavy-duty development on Salsa.

Thanks

Particular thanks are due to Joey Hess, who, in the now-famous design session in Vaumarcus in 2013, helped invent dgit. Since then we have had a lot of support: most recently political support to help get tag2upload deployed, but also, over the years, helpful bug reports and kind words from our users, as well as translations and code contributions.

Many other people have contributed more generally to support for working with Debian source code in git. We particularly want to mention Guido Günther (git-buildpackage); and of course Alexander Wirt, Joerg Jaspert, Thomas Goirand and Antonio Terceiro (Salsa administrators); and before them the Alioth administrators.

comments

tag2upload in the first month of forky

2025-09-14T15:36:17Z

tl;dr: tag2upload (beta) is going well so far, and is already handling around one in 13 uploads to Debian.

Introduction and some stats
Recent UI/UX improvements
Why we are still in beta
- Retrying on Salsa-side failures
Other notable ongoing work
Common problems
- Reuse of version numbers, and attempts to re-tag
- Discrepancies between git and orig tarballs
Get involved

Introduction and some stats

We announced tag2upload’s open beta in mid-July. That was in the middle of the the freeze for trixie, so usage was fairly light until the forky floodgates opened.

Since then the service has successfully performed 637 uploads, of which 420 were in the last 32 days. That’s an average of about 13 per day. For comparison, during the first half of September up to today there have been 2475 uploads to unstable. That’s about 176/day.

So, tag2upload is already handling around 7.5% of uploads. This is very gratifying for a service which is advertised as still being in beta!

Sean and I are very pleased both with the uptake, and with the way the system has been performing.

Recent UI/UX improvements

During this open beta period we have been hard at work. We have made many improvements to the user experience.

Current git-debpush in forky, or trixie-backports, is much better at detecting various problems ahead of time.

When uploads do fail on the service the emailed error reports are now more informative. For example, anomalies involving orig tarballs, which by definition can’t be detected locally (since one point of tag2upload is not to have tarballs locally) now generally result in failure reports containing a diffstat, and instructions for a local repro.

Why we are still in beta

There are a few outstanding work items that we currently want to complete before we declare the end of the beta.

Retrying on Salsa-side failures

The biggest of these is that the service should be able to retry when Salsa fails. Sadly, Salsa isn’t wholly reliable, and right now if it breaks when the service is trying to handle your tag, your upload can fail.

We think most of these failures could be avoided. Implementing retries is a fairly substantial task, but doesn’t pose any fundamental difficulties. We’re working on this right now.

Other notable ongoing work

We want to support pristine-tar, so that pristine-tar users can do a new upstream release. Andrea Pappacoda is working on that with us. See #1106071. (Note that we would generally recommend against use of pristine-tar within Debian. But we want to support it.)

We have been having conversations with Debusine folks about what integration between tag2upload and Debusine would look like. We’re making some progress there, but a lot is still up in the air.

We are considering how best to provide tag2upload pre-checks as part of Salsa CI. There are several problems detected by the tag2upload service that could be detected by Salsa CI too, but which can’t be detected by git-debpush.

Common problems

We’ve been monitoring the service and until very recently we have investigated every service-side failure, to understand the root causes. This has given us insight into the kinds of things our users want, and the kinds of packaging and git practices that are common. We’ve been able to improve the system’s handling of various anomalies and also improved the documentation.

Right now our failure rate is still rather high, at around 7%. Partly this is because people are trying out the system on packages that haven’t ever seen git tooling with such a level of rigour.

There are two classes of problem that are responsible for the vast majority of the failures that we’re still seeing:

Reuse of version numbers, and attempts to re-tag

tag2upload, like git (and like dgit), hates it when you reuse a version number, or try to pretend that a (perhaps busted) release never happened.

git tags aren’t namespaced, and tend to spread about promiscuously. So replacing a signed git tag, with a different tag of the same name, is a bad idea. More generally, reusing the same version number for a different (signed!) package is poor practice. Likewise, it’s usually a bad idea to remove changelog entries for versions which were actually released, just because they were later deemed improper.

We understand that many Debian contributors have gotten used to this kind of thing. Indeed, tools like dcut encourage it. It does allow you to make things neat-looking, even if you’ve made mistakes - but really it does so by covering up those mistakes!

The bottom line is that tag2upload can’t support such history-rewriting. If you discover a mistake after you’ve signed the tag, please just burn the version number and add a new changelog stanza.

One bonus of tag2upload’s approach is that it will discover if you are accidentally overwriting an NMU, and report that as an error.

Discrepancies between git and orig tarballs

tag2upload promises that the source package that it generates corresponds precisely to the git tree you tag and sign.

Orig tarballs make this complicated. They aren’t present on your laptop when you git-debpush. When you’re not uploading a new upstream version, the tag2upload service reuses existing orig tarballs from the archive. If your git and the archive’s orig don’t agree, the tag2upload service will report an error, rather than upload a package with contents that differ from your git tag.

With the most common Debian workflows, everything is fine:

If you base everything on upstream git, and make your orig tarballs with git archive (or git deborig), your orig tarballs are the same as the git, by construction. We recommend usually ignoring upstream tarballs: most upstreams work in git, and their tarballs can contain weirdness that we don’t want. (At worst, the tarball can contain an attack that isn’t visible in git, as with xz!)

Alternatively, if you use gbp import-orig, the differences (including an attack like Jia Tan’s) are imported into git for you. Then, once again, your git and the orig tarball will correspond.

But there are other workflows where this correspondence may not hold. Those workflows are hazardous, because the thing you’re probably working with locally for your routine development is the git view. Then, when you upload, your work is transplanted onto the orig tarball, which might be quite different - so what you upload isn’t what you’ve been working on!

This situation is detected by tag2upload, precisely because tag2upload checks that it’s keeping its promise: the source package is identical to the git view. (dgit push makes the same promise.)

Get involved

Of course the easiest way to get involved is to start using tag2upload.

We would love to have more contributors. There are some easy tasks to get started with, in bugs we’ve tagged “newcomer” — mostly UX improvements such as detecting certain problems earlier, in git-debpush.

More substantially, we are looking for help with sbuild: we’d like it to be able to work directly from git, rather than needing to build source packages: #868527.

comments

Never use git submodules

2023-03-02T19:48:20Z

tl;dr

git submodules are always the wrong solution. Yes, even the to the problem they were specifically invented to solve.

What is wrong with git submodules
Better alternatives to git submodules

What is wrong with git submodules

There are two principal sets of reasons why they are terrible:

Fundamentally wrong design. They break the git data model in multiple ways. Critical ways include:
- A git object in your repository is no longer necessarily resolvable/interpetable to meaningful data. (Shallow clones have the same issue but only with respect to history. git submodules do this for the contents of the tree.)
- git submodules violate the usual rule that all URLs, hostnames, and so on, used by git, are provided by the git configuration and the user, rather than appearing in-tree.
- git submodules introduce completely new states your tree can be in, many of them strange or undesirable.
Wrong behaviour in detail. git’s behaviour with submodules is often buggy or bizarre. Some of these problems are implied by the design, but many of them are additional unforced errors. Some of the defects occur even if you don’t git submodule init, so affect all programs and users which interact with your tree.

Just a few examples of lossage with submodules:
- git checkout no longer reliably switches branches
- editing files and trying to commit them no longer reliably works
- locally pulling a new version from main no longer reliably works
- git ls-files can disagree with git log and git cat-file
- URLs from .gitmodules: they can be malicious; they can end up cached in individual trees’ (individual users’) .git/config; etc.
Generally, normal git operations like git checkout and git pull can leave the submodule in a weird state where you have to run one of the git submodule commands to fix it up. Often the easiest way (especially for a non-expert) to get back to a normal state is to throw the whole tree away and re-clone it.

Ultimately, this means that the author of a program which works with git has two options:

Don’t support submodules. Tell users of your program who file bugs involving submodules that they’re not supported.
Do an enormous amount of extra work: At every point you interact with git, experiment to see what bizarre behaviour submodules exhibit, and write code to deal with all the possibilities.

As a result, a substantial subset of git tooling is broken in the presence of submodules. This is especially true of local automation and tooling, which is otherwise an effective way of improving your processes. But, of course this also applies to git itself! Which is one of the causes of the bugs that git itself has when working with submodules.

Better alternatives to git submodules

In my opinion git submodule is never the right answer. Often, git submodule is the worst answer and any of the following would be better.

Use git subtree

git subtree solves many of the same problems as git submodule, but it does not violate the git data model.

Use this when:

You want to track and use, in-tree, a separate project which ought to have its own identity.
The separate project is of reasonable size (compared to your own).

With git subtree, people and programs that do not need to specifically interact with the upstream for the subtree, do not need to know that it even is a subtree. They can make and switch branches, commit, and so on, as they like.

git subtree can automatically separate out changes made in the downstream, for application to (or submission to) the upstream branch.

I have used git subtree and found it capable and convenient, and pleasingly straightforward.

Just have a monorepo

If you are the upstream for all the pieces, it is often more convenient to merge the git trees into a single git tree with a single history.

Use this when:

The maintenance of all the pieces is organisationally and politically cohesive enough that you can share a git history.
The whole monorepo would be of reasonable size.
Any long-running branches you need to make are for release channels, or the similar, not for having separate versions of the internal dependencies for the different pieces in the monorepo.

Use a package management system, and explicit dependencies

Instead of subsuming the dependency’s tree into your own, give the dependency a proper API and reuse it via a package management system. (If necessary, maintain a proper downstream fork of the dependency.)

The package manager might be be:

a distro-style package management system such as apt+dpkg+sbuild (or a proprietary/private dependency-managing build system); or
a language specific package manager (eg cargo).

Use this when:

You are already using, or familiar with, a suitable package manager,
The API provided by the dependency can be reasonably represented in that package manager (even if unstably).

Use the multiple repository tool `mr`

mr(1) is a tool which lets you conveniently manage a possibly large number of trees, usually as sibling directories.

I haven’t used this myself but it looks capable and straightforward. As I understand it, you’d usually use this in combination with the ..-based dependency expectation I describe below.

It seems like it would be good when your project has a fair number of “foreign” dependencies.

Have your build expect to find the dependency in `..`, its parent dir

This is a very lightweight solution. Just have the files in your tree refer to the dependencies with ../dependency-name/. Expect users (and programs) to manually clone and update the right dependency version, alongside your project.

Consider this when:

Your project is at an early stage and you want to get going quickly and worry about this build system stuff later.
The dependency is disabled by default, and almost never neeeded.

Every program or human that wants to run a build that needs the dependency will need to know to clone the dependency, and keep it up to date. This will be a nuisance, and if you’re doing CI it will mean some custom CI scripting. But this is all probably still better than git submodules. At least it will be completely obvious to everyone what’s going on, how to make changes to the dependency, and so on.

Provide an ad-hoc in-tree script to download the dependency

As a last resort, you can embed the URL to find your dependency, and the instructions for downloading it, in your top-level package’s build system. This is clumsy and awkward, but, astonishingly, it is less painful than git submodules.

Use this when:

Most people using/building your software won’t need the dependency at all.
In particular, most people won’t need to edit the dependency.
None of the other options are suitable.

Usually the downstream build runes should git clone the dependency, and the downstream tree should name the precise commitid needed.

Try to avoid this situation. It’s not a good place to be. But:

Yes, really, git submodule is worse than ad-hoc Makefile runes

The ad-hoc shell script route feels very hacky. But it has some important advantages over git submodule. In particular, unlike with git submodule, this approach (like most of the others I suggest) means that:

All tooling that expects to clone your repository, make changes, do builds, track changes, etc., will work correctly.
You are in precise control of when/whether the download occurs: ie, you can arrange to download the dependency precisely when it’s needed.
You are in precise control of your version management and checking of the dependency: your script controls what version of the dependency to use, and whether that should be “pinned” or dynamically updated.

I’m not advocating ad-hoc runes over git submodules because I like ad-hoc runes or think they’re a good idea. It’s just that git submodule is really so very very bad.

comments

git signed commits are a bad idea

2018-01-29T17:18:46Z

Summary

Occasionally I hear that people are using git signed commits. IMO they should probably be doing something else. I am not aware of any use case for which signed git commits are the right answer. All uses should be replaced by signed tags, or signed pushes.

Analysis

The root cause of the problem is that a commit does not say which branch it is destined for. The same commit (tree, message, and ancestry) might be a debugging patch - not intended for deployment, even harmful; or it might be unfinished and buggy; or it might be ready for deployment.

In practical terms, if your system is set up to require every commit to be signed, the expected development model is that developers will configure their git to sign every commit. (The alternative is to batch-rewrite branches later, to add the signatures, which is an obvious subversion of the intention that signed commits were to provide traceability.)

If you make your developers sign every commit as they go, they will sign lots and lots of junk. Test commits, unfinished branches, local experiments, etc. etc. Whatever way your signatures are supposed to be part of your security assurance, is subvertible by anyone who can find copies of these signed junk branches - and git of course encourages promiscuous sharing of branches, including works in progress. (Usually that promiscuious sharing is good.)

Furthermore, developers will have to have their signing key continuously available; so to the extent that that signing key is important for your system's integrity, it is weakened.

What is really required is that person who approves a push to master should makes a signature declaring what they intend. It is at that point that a human tells their computer what the destination of a branch is supposed to be - before that, the branch is an unfinished work-in-progress.

The same is true of other kinds of publication operations where it looks like signed commits might be useful. For example, submission of a pull request for review: you want the signature to say what the person making it intends: ie, that this is a pull request, which branch it is targeted for, etc.

There are two git features that can be used to implement what is really needed:

Signed pushes

git supports attaching a gpg signature to ‘git push’. This would be precisely right for many applications.

The signature covers precisely what it ought to: it describes the set of refs that should be updated, including their previous values. So (if you have a workflow involving reviews) the signature shows whether the signer intended “this should be reviewed as a submission to master” (a request to the reviewer) or “I have reviewed and approved this and it should be pushed to master immediately” (an instruction to the repository). You can tell the difference between “this is intended for master” and “this is intended as a stopgap to fix user issue in ticket #NNNN”.

Unfortunately the git server side it is afflicted by a number of annoyances, and a lack of good docs, which make verification and publication of the push history awkward.

By default the git server in the main git project does not record push signatures anywhere. Obviously you would need to store them. There are some semi[1] standard approaches to this problem, implemented by some other git server projects.

[1] Javascript needed to even see this. Sorry.

Signed tags

An obvious alternative feature is signed tags. A tag has a name; furthermore, it has an associated message. If necessary the message can carry metadata about the author's intent. (Commits can carry such metadata too but it is not desirable to rewrite a commit just to change its intended destination.)

So it is possible to use signed tags to do the job of signed pushes.

In this model, every push to every controlled branch is associated with a signed tag. The signed tag's name identifies the branch it is intended for. The branch update should be accepted if its tag refers to commit which is descended from the current HEAD. Later verification involves observing that the commit objects referred to by the signed tags form the expected DAG.

(The dgit git repository I run for Debian verifies pushes by looking for particular signed tags.)

Signing only the merges to master

There is one caveat to my assertion that signed commits are useless. A signed merge commit can sometimes do the job of a signed push.

In this use model, mainline only advances by merges, and the only signed commits are the merges which bless other branches into mainline. The rest of the merged branch consists of unsigned commits.

If your tools support only signed commits, rather than signed pushes (or don't record push signatures), and can't be made to do useful verification of signed tags, maybe you could get them to expect (only) the mainline merges to be signed.

There would be some subtleties here - for example, because merges can be nontrivial. I wouldn't recommend this approach without thinking harder about it.

comments

Ian Jackson

Debian’s git transition

tag2upload in the first month of forky

Never use git submodules

tl;dr

What is wrong with git submodules

Better alternatives to git submodules

Use git subtree

Just have a monorepo

Use a package management system, and explicit dependencies

Use the multiple repository tool mr

Have your build expect to find the dependency in .., its parent dir

Provide an ad-hoc in-tree script to download the dependency

Yes, really, git submodule is worse than ad-hoc Makefile runes

git signed commits are a bad idea

Summary

Analysis

Signed pushes

Signed tags

Signing only the merges to master

Use the multiple repository tool `mr`

Have your build expect to find the dependency in `..`, its parent dir