Git in the office

I've been using git recently. Not just me, in fact, but lots of open-source projects have started using it. Right now I use it on github, to keep track of my emacs customizations.

Git is a very interesting source control system to use, mainly because it makes it extremely easy to do branches and patching. Linus has a very interesting video on the rationale behind creating git, but to sum up his point in a few sentences: distributed source control, by allowing local modifications and local patches that can be easiliy merged, avoids the problems of latency and namespace pollution you get using centralized source control systems. Also, he says git is the best distributed one out there, because it is very fast. I really can't comment on whether it is the best, I haven't used darcs or any of the other distributed source control systems that have become popular.

Git can be useful for the corporate developer. When you have the ability to do local commits and local branches, development becomes more efficient. You can commit changes at notable points, so if you've spent the last few hours fruitlessly exploring an interesting idea, you can revert back to your last good commit and proceed without having to manually unwind your changes. Or, if you need to make a quick bugfix, you can just branch off of the current head branch, make a fix, then check it in and return to your previous branch. The ability to always stay in one directory tree can be an efficiency gain as well. And you can work offline, doing commits and everything you can do with a centralized source control system. When you get online and you are ready, you just push all those commits to some other git repository.

So, how can software companies use git? Companies have traditionally used centralized source control systems. I've used CVS (ugh), Perforce (not bad), StarTeam (somewhat yucky, but tolerable), and Visual SourceSafe (flaky) at various companies. It's easy to comprehend a centralized system like this; you have your code, it's in one place, and that's all. Git can be like that too, in a way. There could be one central git repository, and each developer would have their own local repository. Developers could do their own commits on their machine, and when they have something worth checking into the main repository, they do a "git push" to push it there. Or, instead someone in charge of the next release specifically pulls relevant changelists from developers machines to create a coherent release. This is basically what open-source projects do. This would work for small companies, who don't need anything more complicated than that.

If small companies could use git like a centralized repository system, large companies can use it in more interesting ways.

Large companies often have distributed development offices where the pain of using a centralized source repository system is fairly large. Also, large companies also have lots of source which they tend to put in one repository. After a certain point, these repositories get slow. Repository slowness plus network lag for remote offices create a real issue for distributed teams. Git can help by allowing each team to have it's own local repository which they use. Changes are aggregated from each developer's git repository to a local team repository. When it's time to do a release, or at regular intervals, each team's repository is merged with the release repository. Similarly, each team has to make sure their repository is in sync with the release repository, because that one has all relevant changes.

There is a disadvantage to this: with distributed repositories, merges don't happen on every checkin. Merges get put off, and when they happen they the conflicts are typically worse. In such situations, it should be the responsibility of the individual teams to fix any merge problems, that happen when either pushing or pulling their changes.

There would be some great advantages. If you run a continuous build, you only see your own build breakages, not others. The release branch can merge only repositories that are passing their tests, and therefore they always get changes that are good. Of course, it should really run its own continuous build. So you need more build machines, but you get more developer productivity, since any developer's local build is not affected by other developer's build breakages.

From what I've been told, Microsoft has a similar tiered source control system for Windows. For any sufficiently complicated project, such a system is in fact a necessity.

Could this work? I've never seen it happen, but git and distributed source control in general is relatively new. The problem is that the benefits are greatest when the company is large, but the decision on what system to use is made when the company is small, where the benefits are less obvious. It may be that some companies will adopt it for certain projects where it makes sense, but not universally.

Why emacs can survive without automated tests

Automated tests are an accepted best practice in software engineering, and it's hard to find a recently-formed, well-run software organization these days that doesn't have some form of automated series of small tests run against their codebase.

The argument for automated tests is relatively simple: without them, there is no easy way to verify that a release of a project works. Manual testing is slow and expensive, and all the work you put into it is lost when the next release comes out. Furthermore, with frequently run automated tests, you can find out very quickly what changes caused which bugs, and so revert or fix those changes.

Given all that, how does emacs survive, given that it has a huge codebase and no tests?

The best thing it has going for it is that it's a desktop app. That means that users have to upgrade from one version to the next, as opposed to web apps in which all users are upgraded when the server version changes. Since users have to upgrade, and are in control of which version they get, user's versions range from very old to a build of yesterday's CVS head. That means that a certain percentage of users are going to be on the bleeding edge. Those users are either contributors on the project, or those that can tolerate increased bugginess.

Another key is that emacs is a tool people use every day. In fact, it's fair to say that everyone who is contributing to emacs is probably using emacs as their development environment. So those contributing are using the CVS head, and are actively using the latest version.

Let's assume that 1% of the emacs-using population is running off of a recent version of CVS head. So, 1% is exposed to the latest code, and since these same people use it all the time, to do a diverse variety of things, they are testing it continuously, while getting their work done at the same time. Not only are they testing it, but since these are power users, they know how to fix most problems they encounter.

For example, imagine two developers, each working off of CVS head. One checks in a change that breaks dired. The second will get the latest changes, build it, run it, work on it, notice the error, know how to fix it, and can submit a fix immediately.

Eventually, at some time period after any particular changes is made, it can be assumed to be safe, because enough emacs users have been running systems with this change in, and have not had a problem.

However, this implies that if emacs ever wants a stable version, it has to stop non-bugfix checkins, otherwise there will always be a change that hasn't been fully vetted yet. And this is what emacs does - for a new release there is a long phase in which no new functionality is put in.
It's worth noting that this phase is particularly long. Emacs release cycles for major versions can last years.

Let's imagine a world in which emacs did have automated tests of reasonable coverage. What would change?

First, the codebase would be more stable overall, so if you are an emacs contributor, you can expect a less buggy environment. This certainly would make contributing more efficient, since you can concentrate on functionality, not fixing bugs.

Secondly, the feature freeze stage could be significantly shortened. If the unit tests were extremely comprehensive, one could imagine that no feature freeze would be necessary.

All in all, then, in exchange for the considerable effort of developing unit tests for emacs, we can expect a codebase that can change must more rapidly and radically. Whether this is a tradeoff that makes sense I can only guess. Emacs is a mature product, not going through many radical changes anymore. But, part of the reason that radical changes are not happening is not because they aren't needed, but because it is frankly just too hard. It would take a very brave developer who would take on making emacs multithreaded, or switching to Scheme as a basis for elisp. That could change with unit tests. The tasks would still be hard, but it would not be so destabilizing anymore. And automated tests wouldn't take much bravery to write, so I think automated tests may be a path to allow the more radical changes in emacs's future.

What these tests would look like is a subject for another post.