Breadcrumb mode for emacs

I found this useful mode via reddit: breadcrumb.  It allows you to define bookmarks with a keypress.  The bookmarks are unnamed, but you can jump between them, either between-buffer or within-buffer.  This should be really useful for working within a changelist.

One thing that may be useful would be loading or saving breadcrumb state, as I switch between working on different projects.  So I may have to add this (it probably isn't that hard).

Saner ansi-term in emacs

I like ansi-term in emacs. It is the closest to a "real" terminal, offering all the goodness that bash, zsh, or whatever your favorite shell can give you. However, you definitely lose a few things in going to it. The following emacs code restores at least the things I find most useful: dabbrev-completion, yanking, and correct colors.

When you use this code, note that dabbrev-completion is C-c /, and yanking is C-c C-y.

Android, Scala, and java.rmi.RemoteException

For fun, I've been experimenting with writing an Android application using scala. Now, being a conscientious developer, I'm following the principles of TDD, at least with the low-level database interface.

This proved a bit hard to do. I won't go into the details of integrating Scala and Android, since Christian Neukirchen did a good job of that with his article Programming for Android with Scala.

After following the advice in that class, you basically have to double it to compile, dex, and trim the test tree as well.

The big issue I had was trying to use ScalaTest with Android testing. As far as I know, no one has tried to do this yet.

First, Android testing is very particular. You can't just run any old class on the phone, instead there is a particular runner called the InstrumentationTestRunner to run tests. So, you need either Android-specific test classes, or JUnit classes. I therefore had to write a converter class:

class ScalaUnitTestRunner extends TestCase {
  def testExerciseList() = {
    new ExerciseListDaoSpec().execute()
  }
}

Simple and needs improvement, but it's the most basic thing that can work. The execute call runs the scalatests in that test class.

The problem was that when running, I would get this error on running:

[exec] android.test.suitebuilder.TestSuiteBuilder$FailedToCreateTests:
[exec] Error in testSuiteConstructionFailed:
[exec] java.lang.RuntimeException: Exception during suite construction
[exec]     at android.test.suitebuilder.TestSuiteBuilder$FailedToCreateTests.testSuiteConstructionFailed(TestSuiteBuilder.java:239)
[exec]     at java.lang.reflect.Method.invokeNative(Native Method)
[exec]     at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:164)
[exec]     at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:151)
[exec]     at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:418)
[exec]     at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1520)
[exec] Caused by: java.lang.TypeNotPresentException: Type java.rmi.RemoteException not present
[exec]     at java.lang.Class.getDeclaredMethods(Native Method)
[exec]     at java.lang.ClassCache.getDeclaredPublicMethods(ClassCache.java
[exec]     at java.lang.ClassCache.getDeclaredMethods(ClassCache.java:179)
[exec]     at java.lang.ClassCache.findAllMethods(ClassCache.java:249)
[exec]     at java.lang.ClassCache.getFullListOfMethods(ClassCache.java:223)
[exec]     at java.lang.ClassCache.getAllPublicMethods(ClassCache.java:204)
[exec]     at java.lang.Class.getMethods(Class.java:1038)
[exec]     at android.test.suitebuilder.TestGrouping.getTestMethods(TestGrouping.java:79)

...

The problem was that for some reason every Scala class contains a special method called $tag. If you used javap to investigate, you'd find the class looks like:

~/src/AshExercises $ javap -classpath testbin/classes com.ash.exercisetests.db.ScalaUnitTestRunner
Compiled from "ScalaUnitTestRunner.scala"
public class com.ash.exercisetests.db.ScalaUnitTestRunner extends junit.framework.TestCase implements scala.ScalaObject{
public com.ash.exercisetests.db.ScalaUnitTestRunner();
public void testExerciseList();
public int $tag()       throws java.rmi.RemoteException;
}

Because Android's tester is using reflection here to find the test methods, it encounters this method with a class that Android doesn't know about. I tried to define a fake java.rmi.RemoteException, but Android is smart and doesn't let you define core java classes.

So, I had to change my tester to this:

class ScalaUnitTestRunner extends TestCase {
  def testExerciseList() = {
    new ExerciseListDaoSpec().execute()
  }

  override def $tag() : Int = {
    try {
      return super.$tag();
    } catch {
      case e: Exception => throw new RuntimeException(e);
    }
  }
}

That fixed that issue.

The next issue it had was that Android was giving me java.lang.VerifyError on running. This turned out to be my fault, but what I had to figure out was that java.lang.VerifyError really meant a ClassCastException. I found this out by looking in the logcat.

E/dalvikvm(  726): Could not find class 'com.ash.exercise.db.ExerciseDatabase', referenced from method com.ash.exercisetests.db.ExerciseListDaoSpec.<init>
W/dalvikvm(  726): VFY: unable to resolve new-instance 13 (Lcom/ash/exercise/db/ExerciseDatabase;) in Lcom/ash/exercisetests/db/ExerciseListDaoSpec;
W/dalvikvm(  726): VFY:  rejecting opcode 0x22 at 0x0019
W/dalvikvm(  726): VFY:  rejected Lcom/ash/exercisetests/db/ExerciseListDaoSpec;.<init> ()V
W/dalvikvm(  726): Verifier rejected class Lcom/ash/exercisetests/db/ExerciseListDaoSpec;
I/TestRunner(  726): failed: testExerciseList(com.ash.exercisetests.db.ScalaUnit

It turned out I was compiling against the non-test src, but wasn't including the non-test src in my proguard step, so that the classes never wound up in my apk.

I hope these tidbits help someone who is exploring Android, Scala, and unit tests. Good luck, and happy coding!

Distractions

Distractions, or no distractions? For your normal coder, most would prefer no distractions. A distraction at the wrong moment could lose a train of thought, wipe out working memory, and set the programmer back by half an hour of time. Not good. And it is common for programmers to feel overwhelmed with distractions.

However, distractions are useful to the programmer as well. Without distractions, a coder would never get important information. They would never hear a conversation about a problem that they have unique insight into. An interesting fact discussed by two colleagues would go unheard and unremembered. Distractions are unplanned and often
random, but that's often why they are beneficial. The benefits of locality via distractions is dealt with in The Social Life of Information by John Seely Brown and Paul Duguid.

A theoretically distraction-free programmer may get a lot done, but they also deprive others of their insight, and they may miss out on random learning opportunities. This is a problem, because the job of the programmer is not only to design, implement features, or fix bugs, but also to exchange information to help others, and to better themselves. These are the valuable kind of distractions programmers need.

Not only work-related distractions are helpful. Taking a break from work entirely is helpful. Standing up and moving around every once and a while is healthy. Doing fun things with coworkers such as playing foosball promotes camaraderie and teamwork.

We need distractions, although clearly too much is a problem. So, the question is how to process distractions and keep our sanity at the same time.

The first thing to realize is that a distraction creates stress because it causes a prioritization conflict. A developer is supposed to be working on x, but now is distracted and doing task y. Well, there are a few possibilities:

x is more important than the distractor y

A fairly normal case, that comes in a variety of flavors. For example, you are programming very productively when you decide to check your mail. This was a mistake. Now you are out of the mindset of x, and are dealing with some mail y that is not ultimately as important as getting your work done. Or, you are programming and get distracted by an unimportant IM message. Or, you are are programming and the loud conversation next to you is constantly pulling your attention away. For all these things, working in a distraction-free environment for at least part of your time is useful to get rid of part of these distractions, and training your mind not to jump around and want to check mail and surf the web is a way to get rid of the other part. Meditation and just plain practice in focusing is important here.

Even though being distraction free for a while is useful, this shouldn't be an all-day activity, even during rush times. Spend some time and open yourself up to distractions for part of your day. You will learn unexpected things, engage in conversation and relaxation, help people, and get to do all the tiny tasks that accumulate throughout the day.

x is less important than the distractor y

For example, you are working on an important refactoring of your code. All of the sudden, your coworker leans over and tells you that your last changelist broke the build. Obviously, you have to drop what you are doing and fix the problem. You lose context and time, but that's unavoidable. Hopefully, this kind of thing is rare. If it isn't, you may want to check that you have the proper processes in place.

If there are still too many urgent interruptions, then it would help to have one person process all the interrupts of the team. For example, for one week Jane Programmer handles all interrupts for the team, and does not attempt to do non-interrupt-based work at all. This is actually psychology comforting, because she doesn't have to worry about her interrupts slowing her down on her main project. She's already decided that the only work she gets done is what are normally distractions. And if she finds herself distraction-free, she can always do little code cleanups and fixes that she always intended to do but never got around to. She's happy and the rest of the team is more productive by not having to worry about distractions. This should more than offset any productivity loss by Jane Programmer's not working on her project.

Hopefully, by following rules like these we can manage to use the opportunities distractions provide, as well as to minimize their impact.

Why I've abandoned Eclipse for Emacs

Maybe this isn't a big surprise. I've always used emacs, even when using Eclipse. I'm used to emacs, and I use it for absolutely everything: shell, mail, task planner and more. For a while, though I did not do most of my Java coding in it. There I mostly used Eclipse, unless I was just doing something quick. After all Eclipse has great tools for refactoring, automatic imports, quick detection and correction of errors, and more. It's a really great product. Not only that, but it's unsurpassed at quick unit test cycles, mostly requiring no complication.

Emacs has none of that.  But it does have a number of things going for it, and because of these things, I've stopped using Eclipse entirely, and have continued to be productive over the last few months using only emacs.  Here's why:

• Editing.  Emacs is a great editor, unquestionably better at editing than Eclipse. For example, I can do comment formatting extremely quickly in emacs (Meta-q). It's possible to do this Eclipse as well, but it's less flexible and requires more interaction. There are many similar minor things that make emacs a better code editor, as far as textual edits go.
• Lack of projects.  Emacs is unconstrained. Editing a file for Project A, I can switch to a file in Project B as quickly as I can switch to another file in Project A. Emacs does not care about projects, which means it can't give you all the goodies Eclipse can, but it also doesn't tie you down in the same way Eclipse does. 
• Speed.  This is probably the most important reasons I've been doing everything in emacs. There are many different kinds of speed, and most of them are important. Emacs, being lightweight in comparison to Eclipse, has greater speed of editing. Eclipse has greater speed in the change/compile/test cycle, which is very important. But the one that seems important to me is that Emacs has no large pauses, where Eclipse does. I switch projects a lot, and every time I do it seems like Eclipse has to build itself for 10 minutes. IntelliJ doesn't have this problem, but that's because it does less up-front work; everything else is slower. When I go work with someone who uses Eclipse, we often spend the first 10 minutes just waiting for Eclipse to build the project. It's very frustrating. If we were using emacs, we would already be on our way.
• Reliability.  Eclipse is a large moving part, and it is one that seems prone to failure. Once you get burned a couple of times by problems that only are in Eclipse, getting that bit of unpredictability out of the system is a time-saver.  Emacs can get that way too, especially if you use something like jdee-mode.  I like to keep it relatively simple in emacs, though. 
• It's emacs. This is somewhat tautological, but the truth is that once emacs is handling several important tasks for you, you get more and more benefit by bringing activity into it. This is why I can read a mail in gnus (using nnimap), store it in my to-do list in org-mode, and then find some relevant piece of code in emacs and link it to the task in org-mode. It all fits together beautifully.  And whatever you need is there for you, somewhere.  Writing a javadoc and want to put a diagram in there?  Emacs can make it easy; use artist-mode.  There's so much stuff to discover here.

For programming in Java, I just use java-mode, but I've written a few things to make my life easier. I use anything-mode to quickly go to any Java file, ffap so I can quickly open up include files, and use M-x compile so I can jump to errors. I've also written a little import management system that I'll post on this blog shortly. So I find that I'm reasonably efficient.

All in all, my startup time is much quicker for emacs, but I lose time on editing. I'd recommend emacs over Eclipse if you tend to work on a variety of tasks and you use emacs now. If you spend most of your day writing Java, Eclipse is probably going to be better suited to that.

Git in the office

I've been using git recently. Not just me, in fact, but lots of open-source projects have started using it. Right now I use it on github, to keep track of my emacs customizations.

Git is a very interesting source control system to use, mainly because it makes it extremely easy to do branches and patching. Linus has a very interesting video on the rationale behind creating git, but to sum up his point in a few sentences: distributed source control, by allowing local modifications and local patches that can be easiliy merged, avoids the problems of latency and namespace pollution you get using centralized source control systems. Also, he says git is the best distributed one out there, because it is very fast. I really can't comment on whether it is the best, I haven't used darcs or any of the other distributed source control systems that have become popular.

Git can be useful for the corporate developer. When you have the ability to do local commits and local branches, development becomes more efficient. You can commit changes at notable points, so if you've spent the last few hours fruitlessly exploring an interesting idea, you can revert back to your last good commit and proceed without having to manually unwind your changes. Or, if you need to make a quick bugfix, you can just branch off of the current head branch, make a fix, then check it in and return to your previous branch. The ability to always stay in one directory tree can be an efficiency gain as well. And you can work offline, doing commits and everything you can do with a centralized source control system. When you get online and you are ready, you just push all those commits to some other git repository.

So, how can software companies use git? Companies have traditionally used centralized source control systems. I've used CVS (ugh), Perforce (not bad), StarTeam (somewhat yucky, but tolerable), and Visual SourceSafe (flaky) at various companies. It's easy to comprehend a centralized system like this; you have your code, it's in one place, and that's all. Git can be like that too, in a way. There could be one central git repository, and each developer would have their own local repository. Developers could do their own commits on their machine, and when they have something worth checking into the main repository, they do a "git push" to push it there. Or, instead someone in charge of the next release specifically pulls relevant changelists from developers machines to create a coherent release. This is basically what open-source projects do. This would work for small companies, who don't need anything more complicated than that.

If small companies could use git like a centralized repository system, large companies can use it in more interesting ways.

Large companies often have distributed development offices where the pain of using a centralized source repository system is fairly large. Also, large companies also have lots of source which they tend to put in one repository. After a certain point, these repositories get slow. Repository slowness plus network lag for remote offices create a real issue for distributed teams. Git can help by allowing each team to have it's own local repository which they use. Changes are aggregated from each developer's git repository to a local team repository. When it's time to do a release, or at regular intervals, each team's repository is merged with the release repository. Similarly, each team has to make sure their repository is in sync with the release repository, because that one has all relevant changes.

There is a disadvantage to this: with distributed repositories, merges don't happen on every checkin. Merges get put off, and when they happen they the conflicts are typically worse. In such situations, it should be the responsibility of the individual teams to fix any merge problems, that happen when either pushing or pulling their changes.

There would be some great advantages. If you run a continuous build, you only see your own build breakages, not others. The release branch can merge only repositories that are passing their tests, and therefore they always get changes that are good. Of course, it should really run its own continuous build. So you need more build machines, but you get more developer productivity, since any developer's local build is not affected by other developer's build breakages.

From what I've been told, Microsoft has a similar tiered source control system for Windows. For any sufficiently complicated project, such a system is in fact a necessity.

Could this work? I've never seen it happen, but git and distributed source control in general is relatively new. The problem is that the benefits are greatest when the company is large, but the decision on what system to use is made when the company is small, where the benefits are less obvious. It may be that some companies will adopt it for certain projects where it makes sense, but not universally.

Why emacs can survive without automated tests

Automated tests are an accepted best practice in software engineering, and it's hard to find a recently-formed, well-run software organization these days that doesn't have some form of automated series of small tests run against their codebase.

The argument for automated tests is relatively simple: without them, there is no easy way to verify that a release of a project works. Manual testing is slow and expensive, and all the work you put into it is lost when the next release comes out. Furthermore, with frequently run automated tests, you can find out very quickly what changes caused which bugs, and so revert or fix those changes.

Given all that, how does emacs survive, given that it has a huge codebase and no tests?

The best thing it has going for it is that it's a desktop app. That means that users have to upgrade from one version to the next, as opposed to web apps in which all users are upgraded when the server version changes. Since users have to upgrade, and are in control of which version they get, user's versions range from very old to a build of yesterday's CVS head. That means that a certain percentage of users are going to be on the bleeding edge. Those users are either contributors on the project, or those that can tolerate increased bugginess.

Another key is that emacs is a tool people use every day. In fact, it's fair to say that everyone who is contributing to emacs is probably using emacs as their development environment. So those contributing are using the CVS head, and are actively using the latest version.

Let's assume that 1% of the emacs-using population is running off of a recent version of CVS head. So, 1% is exposed to the latest code, and since these same people use it all the time, to do a diverse variety of things, they are testing it continuously, while getting their work done at the same time. Not only are they testing it, but since these are power users, they know how to fix most problems they encounter.

For example, imagine two developers, each working off of CVS head. One checks in a change that breaks dired. The second will get the latest changes, build it, run it, work on it, notice the error, know how to fix it, and can submit a fix immediately.

Eventually, at some time period after any particular changes is made, it can be assumed to be safe, because enough emacs users have been running systems with this change in, and have not had a problem.

However, this implies that if emacs ever wants a stable version, it has to stop non-bugfix checkins, otherwise there will always be a change that hasn't been fully vetted yet. And this is what emacs does - for a new release there is a long phase in which no new functionality is put in.
It's worth noting that this phase is particularly long. Emacs release cycles for major versions can last years.

Let's imagine a world in which emacs did have automated tests of reasonable coverage. What would change?

First, the codebase would be more stable overall, so if you are an emacs contributor, you can expect a less buggy environment. This certainly would make contributing more efficient, since you can concentrate on functionality, not fixing bugs.

Secondly, the feature freeze stage could be significantly shortened. If the unit tests were extremely comprehensive, one could imagine that no feature freeze would be necessary.

All in all, then, in exchange for the considerable effort of developing unit tests for emacs, we can expect a codebase that can change must more rapidly and radically. Whether this is a tradeoff that makes sense I can only guess. Emacs is a mature product, not going through many radical changes anymore. But, part of the reason that radical changes are not happening is not because they aren't needed, but because it is frankly just too hard. It would take a very brave developer who would take on making emacs multithreaded, or switching to Scheme as a basis for elisp. That could change with unit tests. The tasks would still be hard, but it would not be so destabilizing anymore. And automated tests wouldn't take much bravery to write, so I think automated tests may be a path to allow the more radical changes in emacs's future.

What these tests would look like is a subject for another post.