Technical Dresese

code-imports

2011-10-08T18:44:00.000-07:00

I finally put up the emacs based code import-management solution I've been using for a while. I wrote a similar thing a few years ago at Google, but I rethought and rewrote this, plus added a bunch of ert tests.I've put it up as code-imports in github. The module allows you to organize imports according to predefined rules, as well as grab other files as imports and add them as imports to a target file.

As opposed to the earlier incarnation, I've extracted as much as the logic as possible into purely functional functions, which makes it easy to test. In my experience, import management is complicated enough that manual testing is just not enough to ensure a stable product. Testing the code thoroughly is one way to solve this problem. I've also simplified the number of concepts in the code, and the number of variable to tweak to the bare minimum.

I'm going to refine this a bit, then announce it more broadly shortly.

A brief demonstration of emacs new lexical bindings

2011-04-09T09:57:00.000-07:00

It wasn't a joke. On April 1st, Stefan Monnier merged the lexical binding branch into the emacs trunk. Unless it needs to be rolled back, emacs 24 should have lexical binding, which is a huge improvement for elisp. If you want to try it out, get the latest emacs from source, and compile it yourself.

To enable it, you need to set lexical-binding to t. This is buffer local, so you need to set it for every buffer that you want to have lexical binding in.

Here's a demonstration in an ielm session (M-x ielm).

ELISP> (setq lexical-binding t)
t
ELISP> (defun make-adder (base) (lambda (num) (+ num base)))
make-adder
ELISP> (setq x 1)
1
ELISP> (setq 1-adder (make-adder x))
(closure
((base . 1)
t)
(num)
(+ num base))
ELISP> (funcall 1-adder 10)
11
ELISP> (funcall 1-adder 100)
101
ELISP> (setq x 100)
100
ELISP> (funcall 1-adder 100)
101

Without lexical binding, the variable x would be a dynamic variable, and so when we set it to 100 later in the session, the 1-adder would change to a 100-adder inadvertently. In fact, if we try, we can't even get to that point. Here's the same session without lexical binding:

ELISP> lexical-binding
nil
ELISP> (defun make-adder (base) (lambda (num) (+ num base)))
make-adder
ELISP> (setq x 1)
1
ELISP> (setq 1-adder (make-adder x))
(lambda
(num)
(+ num base))
ELISP> (funcall 1-adder 10)
*** Eval error *** Symbol's value as variable is void: base

What's going on here? Notice that when we make the 1-adder without lexical binding, we don't get a closure. Instead, we just get a function that references num and base. num is passed into the function, but base is a dynamic variable, so it is interpreted in the emacs global context, as opposed to a context inside a closure. The variable base is not bound to a value in the global context, and when we execute the lambda, which refers to base, emacs looks for the base variable, cannot find it, and throws an error.

This doesn't happen with lexical binding. In that case, we can actually see the closure itself being returned when there is lexical binding:

(closure
((base . 1)
t)
(num)
(+ num base))

The closure here shows a structure in which we can interpret to mean that base is being bound to 1. You can also see num is the argument (since it is not in the same s-expression as the binding of base), and then it proceeds with the simple addition function.

I haven't yet made use of closures in my emacs code. I don't really need it in my initialization files, and if I writes modules that assume it then almost no one can yet use it. So I'll give it time, but it's a huge step in making elisp a better lisp. Read more about lexical bindings Paul Graham's wonderful free book On Lisp.

Lexical bindings are not only a good feature in themselves, they are also is an important step on the way to multithreading. Having nothing but global variables makes threading quite challenging, and lexical bindings in the standard elisp packages can help make multithreading easier.

Data

2011-03-21T20:20:00.001-07:00

Humans are not very good at remembering event frequency statistics. We tend to only remember the unusual, and forget about the common case. There are plenty of other cognitive errors we make, but this seems to me one of the most troublesome.

An example: Every day I take the subway from Forest Hills in Queens to Manhattan. Either the E or F train will arrive first, and I need the E train. I was noticing, though, that it always seemed to be the F train arriving first, making me wait for an extra few minutes.

Why would that be? Could it be a situation like Feynman's girlfriend puzzle, where the E train closely follows the F? I decided to keep some statistics to find out.

Every day, I'd record which train came first. After a few days, I noticed the E had been coming first lately. Surely a fluke.

I kept at it, and before long the trend was clear. The E comes first almost twice as often! I counted 18 E firsts, and only 10 F firsts.

Obviously, I was completely wrong about even the most basic statistics for my morning commute, and that's about something I experience every single workday morning. Think how wrong I can be about everything else!

I take this as a warning about not trusting experiences when making decisions. Get data instead. Even just some data will be better than opinion or memories.

Does everything have to be social?

2011-01-05T09:14:00.001-08:00

All I wanted to do was to download a document. To do that, I had to sign up for Scribd via my Facebook account. Next thing I know, I am getting follower notifications emails from some of my Facebook friends.

I just wanted to download a document. Now I'm connected to all sorts of friends. What do they want from me? Did they even explicitly choose to friend me or was it automatically done by Scribd?

For the time being, I'm just going to ignore it all and hope it goes away.

Macros and meta-macros

2010-11-05T20:48:00.000-07:00

Life is repetition. We do the same things, day after day. Within each day at work, we find ourselves performing the same actions, more or less. The great thing about computers is that tasks performed on computers can be automated. So any time we find that we are repeated ourselves on the computer, we have an opportunity of creating a tool that will allow us to eliminate repetition.

Macros are a great example of this, and one of the defining feature of lisp. Sure, c++ has macros as well, and they share same of the same capabilities as lisp's macros, but in general the power of lisp macros make them better for everyday use. Macros are great tools to reduce repetitive code to simple forms. Code such as:

Seems reasonable enough if you just are opening up one file. But what if you do this all the time? Even the simple boilerplate here starts to add up. That's why common lisp created a macro of an easy way to do this:

What a breath of fresh air that is... there is absolutely nothing wasted here. All repetitive and wasted code has been removed, leaving only the essence of the task. That's the art of the macro. It's one thing to be able to write macros, but to recognize patterns of repetition is a skill worth practicing.

Meta-macros are a term I just made up for this post, which describes repetitive coding tasks we have to perform. To take a simple example, perhaps you are always navigating to the same files. Like macros, it is essential that we realize we are doing something repetitive, and that this task can be removed with automation. In this case, emacs provides a solution: bookmarks. Many times the solution isn't provided by emacs, though, or the patterns are more complicated. Every time I change or add a new function in my .cc file, I have to edit my .h file as well. This is just a natural part of editing C++ code, something so ingrained in us that it's hard to even imagine not going through the trouble of essentially adding function definitions twice. It's a plainly repetitive task, which could certainly be automated with a meta-macro, by which I mean some elisp code.

Sometimes these meta-macro's automation can increase complexity. It works fine 98% of the time, but 2% of the time it screws you up and you stop using the meta-macro in frustration. But macros can have the same trait. Good macros are actually a bit hard to write, and can be very confusing when they fail. Meta-macros can be even worse, due to the complexity of the meta level in which we operate. It takes dedication to solve each new problem you encounter, making your meta-macro more and more robust. But, with enough effort, you will have an elegant solution that makes you more productive, and able to focus on problem solving, not mere coding.

When you do, please share it with the word. We all need the help.

Breadcrumb mode for emacs

2010-10-31T18:52:00.000-07:00

I found this useful mode via reddit: breadcrumb. It allows you to define bookmarks with a keypress. The bookmarks are unnamed, but you can jump between them, either between-buffer or within-buffer. This should be really useful for working within a changelist.

One thing that may be useful would be loading or saving breadcrumb state, as I switch between working on different projects. So I may have to add this (it probably isn't that hard).

Saner ansi-term in emacs

2010-02-23T20:26:00.000-08:00

I like ansi-term in emacs. It is the closest to a "real" terminal, offering all the goodness that bash, zsh, or whatever your favorite shell can give you. However, you definitely lose a few things in going to it. The following emacs code restores at least the things I find most useful: dabbrev-completion, yanking, and correct colors.

When you use this code, note that dabbrev-completion is C-c /, and yanking is C-c C-y.

Android, Scala, and java.rmi.RemoteException

2009-09-25T22:12:00.000-07:00

For fun, I've been experimenting with writing an Android application using scala. Now, being a conscientious developer, I'm following the principles of TDD, at least with the low-level database interface.

This proved a bit hard to do. I won't go into the details of integrating Scala and Android, since Christian Neukirchen did a good job of that with his article Programming for Android with Scala.

After following the advice in that class, you basically have to double it to compile, dex, and trim the test tree as well.

The big issue I had was trying to use ScalaTest with Android testing. As far as I know, no one has tried to do this yet.

First, Android testing is very particular. You can't just run any old class on the phone, instead there is a particular runner called the InstrumentationTestRunner to run tests. So, you need either Android-specific test classes, or JUnit classes. I therefore had to write a converter class:


class ScalaUnitTestRunner extends TestCase {
  def testExerciseList() = {
    new ExerciseListDaoSpec().execute()
  }
}

Simple and needs improvement, but it's the most basic thing that can work. The execute call runs the scalatests in that test class.

The problem was that when running, I would get this error on running:

[exec] android.test.suitebuilder.TestSuiteBuilder$FailedToCreateTests:
[exec] Error in testSuiteConstructionFailed:
[exec] java.lang.RuntimeException: Exception during suite construction
[exec]     at android.test.suitebuilder.TestSuiteBuilder$FailedToCreateTests.testSuiteConstructionFailed(TestSuiteBuilder.java:239)
[exec]     at java.lang.reflect.Method.invokeNative(Native Method)
[exec]     at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:164)
[exec]     at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:151)
[exec]     at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:418)
[exec]     at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1520)
[exec] Caused by: java.lang.TypeNotPresentException: Type java.rmi.RemoteException not present
[exec]     at java.lang.Class.getDeclaredMethods(Native Method)
[exec]     at java.lang.ClassCache.getDeclaredPublicMethods(ClassCache.java
[exec]     at java.lang.ClassCache.getDeclaredMethods(ClassCache.java:179)
[exec]     at java.lang.ClassCache.findAllMethods(ClassCache.java:249)
[exec]     at java.lang.ClassCache.getFullListOfMethods(ClassCache.java:223)
[exec]     at java.lang.ClassCache.getAllPublicMethods(ClassCache.java:204)
[exec]     at java.lang.Class.getMethods(Class.java:1038)
[exec]     at android.test.suitebuilder.TestGrouping.getTestMethods(TestGrouping.java:79)

...

The problem was that for some reason every Scala class contains a special method called $tag. If you used javap to investigate, you'd find the class looks like:

~/src/AshExercises $ javap -classpath testbin/classes com.ash.exercisetests.db.ScalaUnitTestRunner
Compiled from "ScalaUnitTestRunner.scala"
public class com.ash.exercisetests.db.ScalaUnitTestRunner extends junit.framework.TestCase implements scala.ScalaObject{
public com.ash.exercisetests.db.ScalaUnitTestRunner();
public void testExerciseList();
public int $tag()       throws java.rmi.RemoteException;
}

Because Android's tester is using reflection here to find the test methods, it encounters this method with a class that Android doesn't know about. I tried to define a fake java.rmi.RemoteException, but Android is smart and doesn't let you define core java classes.

So, I had to change my tester to this:


class ScalaUnitTestRunner extends TestCase {
  def testExerciseList() = {
    new ExerciseListDaoSpec().execute()
  }

  override def $tag() : Int = {
    try {
      return super.$tag();
    } catch {
      case e: Exception => throw new RuntimeException(e);
    }
  }
}

That fixed that issue.

The next issue it had was that Android was giving me java.lang.VerifyError on running. This turned out to be my fault, but what I had to figure out was that java.lang.VerifyError really meant a ClassCastException. I found this out by looking in the logcat.

E/dalvikvm(  726): Could not find class 'com.ash.exercise.db.ExerciseDatabase', referenced from method com.ash.exercisetests.db.ExerciseListDaoSpec.<init>
W/dalvikvm(  726): VFY: unable to resolve new-instance 13 (Lcom/ash/exercise/db/ExerciseDatabase;) in Lcom/ash/exercisetests/db/ExerciseListDaoSpec;
W/dalvikvm(  726): VFY:  rejecting opcode 0x22 at 0x0019
W/dalvikvm(  726): VFY:  rejected Lcom/ash/exercisetests/db/ExerciseListDaoSpec;.<init> ()V
W/dalvikvm(  726): Verifier rejected class Lcom/ash/exercisetests/db/ExerciseListDaoSpec;
I/TestRunner(  726): failed: testExerciseList(com.ash.exercisetests.db.ScalaUnit

It turned out I was compiling against the non-test src, but wasn't including the non-test src in my proguard step, so that the classes never wound up in my apk.

I hope these tidbits help someone who is exploring Android, Scala, and unit tests. Good luck, and happy coding!

Distractions

2009-06-01T10:28:00.000-07:00

Distractions, or no distractions? For your normal coder, most would prefer no distractions. A distraction at the wrong moment could lose a train of thought, wipe out working memory, and set the programmer back by half an hour of time. Not good. And it is common for programmers to feel overwhelmed with distractions.

However, distractions are useful to the programmer as well. Without distractions, a coder would never get important information. They would never hear a conversation about a problem that they have unique insight into. An interesting fact discussed by two colleagues would go unheard and unremembered. Distractions are unplanned and often
random, but that's often why they are beneficial. The benefits of locality via distractions is dealt with in The Social Life of Information by John Seely Brown and Paul Duguid.

A theoretically distraction-free programmer may get a lot done, but they also deprive others of their insight, and they may miss out on random learning opportunities. This is a problem, because the job of the programmer is not only to design, implement features, or fix bugs, but also to exchange information to help others, and to better themselves. These are the valuable kind of distractions programmers need.

Not only work-related distractions are helpful. Taking a break from work entirely is helpful. Standing up and moving around every once and a while is healthy. Doing fun things with coworkers such as playing foosball promotes camaraderie and teamwork.

We need distractions, although clearly too much is a problem. So, the question is how to process distractions and keep our sanity at the same time.

The first thing to realize is that a distraction creates stress because it causes a prioritization conflict. A developer is supposed to be working on x, but now is distracted and doing task y. Well, there are a few possibilities:

x is more important than the distractor y: A fairly normal case, that comes in a variety of flavors. For example, you are programming very productively when you decide to check your mail. This was a mistake. Now you are out of the mindset of x, and are dealing with some mail y that is not ultimately as important as getting your work done. Or, you are programming and get distracted by an unimportant IM message. Or, you are are programming and the loud conversation next to you is constantly pulling your attention away. For all these things, working in a distraction-free environment for at least part of your time is useful to get rid of part of these distractions, and training your mind not to jump around and want to check mail and surf the web is a way to get rid of the other part. Meditation and just plain practice in focusing is important here.

Even though being distraction free for a while is useful, this shouldn't be an all-day activity, even during rush times. Spend some time and open yourself up to distractions for part of your day. You will learn unexpected things, engage in conversation and relaxation, help people, and get to do all the tiny tasks that accumulate throughout the day.
x is less important than the distractor y: For example, you are working on an important refactoring of your code. All of the sudden, your coworker leans over and tells you that your last changelist broke the build. Obviously, you have to drop what you are doing and fix the problem. You lose context and time, but that's unavoidable. Hopefully, this kind of thing is rare. If it isn't, you may want to check that you have the proper processes in place.

If there are still too many urgent interruptions, then it would help to have one person process all the interrupts of the team. For example, for one week Jane Programmer handles all interrupts for the team, and does not attempt to do non-interrupt-based work at all. This is actually psychology comforting, because she doesn't have to worry about her interrupts slowing her down on her main project. She's already decided that the only work she gets done is what are normally distractions. And if she finds herself distraction-free, she can always do little code cleanups and fixes that she always intended to do but never got around to. She's happy and the rest of the team is more productive by not having to worry about distractions. This should more than offset any productivity loss by Jane Programmer's not working on her project.

Hopefully, by following rules like these we can manage to use the opportunities distractions provide, as well as to minimize their impact.

Why I've abandoned Eclipse for Emacs

2008-11-29T13:07:00.000-08:00

Maybe this isn't a big surprise. I've always used emacs, even when using Eclipse. I'm used to emacs, and I use it for absolutely everything: shell, mail, task planner and more. For a while, though I did not do most of my Java coding in it. There I mostly used Eclipse, unless I was just doing something quick. After all Eclipse has great tools for refactoring, automatic imports, quick detection and correction of errors, and more. It's a really great product. Not only that, but it's unsurpassed at quick unit test cycles, mostly requiring no complication.

Emacs has none of that. But it does have a number of things going for it, and because of these things, I've stopped using Eclipse entirely, and have continued to be productive over the last few months using only emacs. Here's why:

Editing. Emacs is a great editor, unquestionably better at editing than Eclipse. For example, I can do comment formatting extremely quickly in emacs (Meta-q). It's possible to do this Eclipse as well, but it's less flexible and requires more interaction. There are many similar minor things that make emacs a better code editor, as far as textual edits go.
Lack of projects. Emacs is unconstrained. Editing a file for Project A, I can switch to a file in Project B as quickly as I can switch to another file in Project A. Emacs does not care about projects, which means it can't give you all the goodies Eclipse can, but it also doesn't tie you down in the same way Eclipse does.
Speed. This is probably the most important reasons I've been doing everything in emacs. There are many different kinds of speed, and most of them are important. Emacs, being lightweight in comparison to Eclipse, has greater speed of editing. Eclipse has greater speed in the change/compile/test cycle, which is very important. But the one that seems important to me is that Emacs has no large pauses, where Eclipse does. I switch projects a lot, and every time I do it seems like Eclipse has to build itself for 10 minutes. IntelliJ doesn't have this problem, but that's because it does less up-front work; everything else is slower. When I go work with someone who uses Eclipse, we often spend the first 10 minutes just waiting for Eclipse to build the project. It's very frustrating. If we were using emacs, we would already be on our way.
Reliability. Eclipse is a large moving part, and it is one that seems prone to failure. Once you get burned a couple of times by problems that only are in Eclipse, getting that bit of unpredictability out of the system is a time-saver. Emacs can get that way too, especially if you use something like jdee-mode. I like to keep it relatively simple in emacs, though.
It's emacs. This is somewhat tautological, but the truth is that once emacs is handling several important tasks for you, you get more and more benefit by bringing activity into it. This is why I can read a mail in gnus (using nnimap), store it in my to-do list in org-mode, and then find some relevant piece of code in emacs and link it to the task in org-mode. It all fits together beautifully. And whatever you need is there for you, somewhere. Writing a javadoc and want to put a diagram in there? Emacs can make it easy; use artist-mode. There's so much stuff to discover here.

For programming in Java, I just use java-mode, but I've written a few things to make my life easier. I use anything-mode to quickly go to any Java file, ffap so I can quickly open up include files, and use M-x compile so I can jump to errors. I've also written a little import management system that I'll post on this blog shortly. So I find that I'm reasonably efficient.

All in all, my startup time is much quicker for emacs, but I lose time on editing. I'd recommend emacs over Eclipse if you tend to work on a variety of tasks and you use emacs now. If you spend most of your day writing Java, Eclipse is probably going to be better suited to that.

Git in the office

2008-09-27T10:58:00.000-07:00

I've been using git recently. Not just me, in fact, but lots of open-source projects have started using it. Right now I use it on github, to keep track of my emacs customizations.

Git is a very interesting source control system to use, mainly because it makes it extremely easy to do branches and patching. Linus has a very interesting video on the rationale behind creating git, but to sum up his point in a few sentences: distributed source control, by allowing local modifications and local patches that can be easiliy merged, avoids the problems of latency and namespace pollution you get using centralized source control systems. Also, he says git is the best distributed one out there, because it is very fast. I really can't comment on whether it is the best, I haven't used darcs or any of the other distributed source control systems that have become popular.

Git can be useful for the corporate developer. When you have the ability to do local commits and local branches, development becomes more efficient. You can commit changes at notable points, so if you've spent the last few hours fruitlessly exploring an interesting idea, you can revert back to your last good commit and proceed without having to manually unwind your changes. Or, if you need to make a quick bugfix, you can just branch off of the current head branch, make a fix, then check it in and return to your previous branch. The ability to always stay in one directory tree can be an efficiency gain as well. And you can work offline, doing commits and everything you can do with a centralized source control system. When you get online and you are ready, you just push all those commits to some other git repository.

So, how can software companies use git? Companies have traditionally used centralized source control systems. I've used CVS (ugh), Perforce (not bad), StarTeam (somewhat yucky, but tolerable), and Visual SourceSafe (flaky) at various companies. It's easy to comprehend a centralized system like this; you have your code, it's in one place, and that's all. Git can be like that too, in a way. There could be one central git repository, and each developer would have their own local repository. Developers could do their own commits on their machine, and when they have something worth checking into the main repository, they do a "git push" to push it there. Or, instead someone in charge of the next release specifically pulls relevant changelists from developers machines to create a coherent release. This is basically what open-source projects do. This would work for small companies, who don't need anything more complicated than that.

If small companies could use git like a centralized repository system, large companies can use it in more interesting ways.

Large companies often have distributed development offices where the pain of using a centralized source repository system is fairly large. Also, large companies also have lots of source which they tend to put in one repository. After a certain point, these repositories get slow. Repository slowness plus network lag for remote offices create a real issue for distributed teams. Git can help by allowing each team to have it's own local repository which they use. Changes are aggregated from each developer's git repository to a local team repository. When it's time to do a release, or at regular intervals, each team's repository is merged with the release repository. Similarly, each team has to make sure their repository is in sync with the release repository, because that one has all relevant changes.

There is a disadvantage to this: with distributed repositories, merges don't happen on every checkin. Merges get put off, and when they happen they the conflicts are typically worse. In such situations, it should be the responsibility of the individual teams to fix any merge problems, that happen when either pushing or pulling their changes.

There would be some great advantages. If you run a continuous build, you only see your own build breakages, not others. The release branch can merge only repositories that are passing their tests, and therefore they always get changes that are good. Of course, it should really run its own continuous build. So you need more build machines, but you get more developer productivity, since any developer's local build is not affected by other developer's build breakages.

From what I've been told, Microsoft has a similar tiered source control system for Windows. For any sufficiently complicated project, such a system is in fact a necessity.

Could this work? I've never seen it happen, but git and distributed source control in general is relatively new. The problem is that the benefits are greatest when the company is large, but the decision on what system to use is made when the company is small, where the benefits are less obvious. It may be that some companies will adopt it for certain projects where it makes sense, but not universally.

Why emacs can survive without automated tests

2008-09-13T08:51:00.000-07:00

Automated tests are an accepted best practice in software engineering, and it's hard to find a recently-formed, well-run software organization these days that doesn't have some form of automated series of small tests run against their codebase.

The argument for automated tests is relatively simple: without them, there is no easy way to verify that a release of a project works. Manual testing is slow and expensive, and all the work you put into it is lost when the next release comes out. Furthermore, with frequently run automated tests, you can find out very quickly what changes caused which bugs, and so revert or fix those changes.

Given all that, how does emacs survive, given that it has a huge codebase and no tests?

The best thing it has going for it is that it's a desktop app. That means that users have to upgrade from one version to the next, as opposed to web apps in which all users are upgraded when the server version changes. Since users have to upgrade, and are in control of which version they get, user's versions range from very old to a build of yesterday's CVS head. That means that a certain percentage of users are going to be on the bleeding edge. Those users are either contributors on the project, or those that can tolerate increased bugginess.

Another key is that emacs is a tool people use every day. In fact, it's fair to say that everyone who is contributing to emacs is probably using emacs as their development environment. So those contributing are using the CVS head, and are actively using the latest version.

Let's assume that 1% of the emacs-using population is running off of a recent version of CVS head. So, 1% is exposed to the latest code, and since these same people use it all the time, to do a diverse variety of things, they are testing it continuously, while getting their work done at the same time. Not only are they testing it, but since these are power users, they know how to fix most problems they encounter.

For example, imagine two developers, each working off of CVS head. One checks in a change that breaks dired. The second will get the latest changes, build it, run it, work on it, notice the error, know how to fix it, and can submit a fix immediately.

Eventually, at some time period after any particular changes is made, it can be assumed to be safe, because enough emacs users have been running systems with this change in, and have not had a problem.

However, this implies that if emacs ever wants a stable version, it has to stop non-bugfix checkins, otherwise there will always be a change that hasn't been fully vetted yet. And this is what emacs does - for a new release there is a long phase in which no new functionality is put in.
It's worth noting that this phase is particularly long. Emacs release cycles for major versions can last years.

Let's imagine a world in which emacs did have automated tests of reasonable coverage. What would change?

First, the codebase would be more stable overall, so if you are an emacs contributor, you can expect a less buggy environment. This certainly would make contributing more efficient, since you can concentrate on functionality, not fixing bugs.

Secondly, the feature freeze stage could be significantly shortened. If the unit tests were extremely comprehensive, one could imagine that no feature freeze would be necessary.

All in all, then, in exchange for the considerable effort of developing unit tests for emacs, we can expect a codebase that can change must more rapidly and radically. Whether this is a tradeoff that makes sense I can only guess. Emacs is a mature product, not going through many radical changes anymore. But, part of the reason that radical changes are not happening is not because they aren't needed, but because it is frankly just too hard. It would take a very brave developer who would take on making emacs multithreaded, or switching to Scheme as a basis for elisp. That could change with unit tests. The tasks would still be hard, but it would not be so destabilizing anymore. And automated tests wouldn't take much bravery to write, so I think automated tests may be a path to allow the more radical changes in emacs's future.

What these tests would look like is a subject for another post.

Code is the easy part: Preventing data corruptions

2008-08-06T15:10:00.000-07:00

Coding, coding, coding. There's an awful lot of attention paid to coding in the software engineering world. I read redditt's programming category a lot these days, and most software engineering posts are about specifically code. I think this is not because coding is the most important part of being a programmer, but because it is the most interesting and fun part.   Code is easy to change. If you have a bug, you reproduce it and fix it, and then it goes away. Sometimes it's difficult, but even if it is, you fix it, make sure it doesn't happen again by writing test cases (preferably before you fix it), and then you forget about it.

Let's contrast the fixing of bugs in code, with fixing corrupted data. Corrupted data in any reasonable system is probably going to be caused by a software bug. So, you first have to find out why the data is corrupted. Of course, maybe the corruption is old, and the bug has been fixed. Hard to say. You have to investigate. You could try and reproduce it, if you can figure out a likely scenario. Most likely you will fail to reproduce it, and even more likely is that you know you will fail to reproduce it, so you don't try. So instead you look at the code, seeing how it could happen. This is a useful practice. You can sometimes find the cause here. Then you can write a test for it and fix it.   After you fix it, your work isn't done, of course. You have to actually fix the corruptions. This could be a simple as running a SQL query, or as complicated as writing a script to patch things up using a code library to do the work.   Of course, sometimes you never can find out why your data is corrupted, so you just have to fix it, if you can figure out how.

Some amount of corrupted data is an inevitability, and in fact some may come from design decisions. For example, some database systems cannot do two-phased commits, and if you need to hook into one, you may have to accept a certain amount of data corruption due to not having atomicity in your transactions. If the error rate is very low, some corruption may be a fair price to pay for whatever benefits the second system is getting you.   Even so, this is dangerous, and a low error rate today may be a severe error rate tomorrow, leaving you with a lot of angry customers with corrupted data, and a few dejected developers who have to clean it all up.

There's a few best practices you can do to avoid data corruption

Use one transactional system with ACID properties, and use it correctly.
When using SQL, use foreign key references whenever possible.
Before saving data, assert its correctness as a precondition. This includes both the values stored, and the relationship of the data to data that both links to it, and is linked to.
Create a data checker that will run a series of tests on your data. This is basically like unit-tests for data, but you can run it not only after a series of tests, but in your production system too. Run this program regularly, and pay attention to the output. You want to be alerted of any changes in the sanity of your data. Like unit tests, the more invariants you encode into this tool, the better you will be. When changing or adding data, modify the data checker code. After each QA cycle, run the data checker. Any errors should be entered as bugs.
If your data can be repaired, have an automated data repairer. This shouldn't be run regularly, because you don't want to get too complacent about your corruptions. Instead, if you notice that the data checker has picked up some new errors, then you run this, modifying it first if the errors are of a new type.

Doing a good job on all these tips should prevent most data corruption, but not all. Like bugs, even the best preventative measures will not guarantee success.

Having clean data is extremely important. This data is not yours, it is your customer's, and they trust you with it. You need to protect it, and it isn't easy, but preventing data corruptions is always the right thing to do.

In org-mode, abandoning GTD

2008-08-06T14:35:00.000-07:00

I've already mentioned the problem with contexts in the GTD system. In using org-mode, I've come to realize that another important concept of GTD is either flawed or redundant: the next action.

Next actions seem good at first. Every project has a next action, the action that is a short-doable task that will advance the state of the project. Of course, some projects have several actions you can do in parallel, so there are several next actions. As the number of projects you have grow, you get more and more next actions. At some point, the number of next actions becomes too much to keep track of. Some people might say this goes away with proper contexts. However, I've never found a good use for contexts, because in truth at any point I can do any of my next actions. Another strategy for getting rid of excessive projects is to move some to a someday/maybe folder. I think this is reasonable, but sometimes you know that in a particular week you are just going to work on a few different projects, perhaps because of deadlines or other prioritization. What then?

In org-mode, I solved this problem by using the agenda, and scheduling my next actions that I wanted to work on for the current day. I would then see a list of the next actions I had to accomplish that day. If I didn't get them done that day, the next day I'd move them up a day, to the current day. It ended up being a daily-planner-like system a lot like Tom Limoncelli recommends in Time Management for System Administrators. But with next actions.

Last week I had a revelation: scheduled items, for me, were the same as next actions! So I removed all next actions, just having states TODO, WAITING, and DONE. Like next actions, when I finish a scheduled item, the next item in the project becomes scheduled. I like my new system. It combines the flexibility of Limoncelli's day-planner system with the project-planning of GTD.

I spend my day as before, taking tasks from the day view of org-mode, and using a weekly review to schedule or de-schedule items from that view. I also add notes to tasks all the time, which proves to be helpful. I think I've benefitted from next-actions, since they force you to think of actions at level of granularity such that each task is specific, concrete, and immediately actionable. I treat tasks the exact same way now, even without next actions themselves.

For more info about org-mode, check out the talk on org-mode which happened after I invited Carsten Dominik to give a talk at Google about his system.

org-mode

2007-08-31T18:24:00.000-07:00

In my previous post on the problems with GTD, I mentioned a bit about using emacs' org-mode to implement a GTD-like system. I've been using it for a while now, with good success.

I've previously used kGTD, as well as iGTD, but I like org-mode the best. With any other application, you are quite limited in what you can do. You can basically do what they want you to, maybe modulo a few scripts or preferences. With org-mode, the basic functionality is extremely flexible, and since it's all just a bunch of lisp code, you can rewrite and alter it to your heart's content.

org-mode, at it's simplest, presents an outline. You can expand and collapse it. Like other programs, you can put all sorts of stuff in the outline, in a very natural matter. For example:

* Fix bug 1919
** Reproduce the bug
This only breaks for very old users
** Write test case for bug

The stars here represent the indentation level. It's all just text, folks. This is emacs, what else would you expect? But notice that I can just add random text anywhere. It will be collapsed along with the "Reproduce the bug" header. This is very convenient place to put notes when you are working. As other programs do, you can have links. When I write "Fix bug 1919", the "1919" can link to my work's bug tracking system. More impressively, this being emacs, I can link to a specific location in a buffer. Opening it takes me right there, and off I go to do whatever work I need to.

I also use TODO tags, which are states associated with each work item. So, if this is an item that can become, in the GTD sense, a "next action", then it has a tag. There are four tags I use:

TODO: An item which is still to be done, but is not yet actionable.
NEXT: A "next action", which should be done.
WAITING: An action that is waiting on some outside signal to either go back to NEXT or DONE.
DONE: Finished. Hooray!

So, the above outline is more like:

* Fix bug 1919
** NEXT Reproduce the bug
This only breaks for very old users
** TODO Write test case for bug

Here is some customization code to enable this, as well as to enable useful agenda commands:

(setq org-todo-keywords '("TODO" "NEXT" "WAITING" "DONE"))
(setq org-agenda-custom-commands
'(("w" todo "WAITING" nil)
("n" todo "NEXT" nil)))

I like to, with one keystroke, mark an entry done and change the next one to NEXT. I wrote the following lisp code to do it:

(defun ash-org-todo-item-p ()
(save-excursion
(beginning-of-line)
(looking-at "\\*+[ \t]+TODO\\>")))

(defun ash-org-mark-done ()
(interactive)
(save-excursion
;; org-entry-is-done-p has a bug where if you are at the first
;; char of a line it doesn't always work. Let's work around it
;; here.
(end-of-line)
(when (not (org-entry-is-done-p))
(org-todo 'done)
(outline-forward-same-level 1)
(when (ash-org-todo-item-p)
(org-todo "NEXT")))))

(define-key global-map "\C-c\C-x\C-c" 'ash-org-mark-done)

The preceding code gives me a pretty nice, basic GTD system. On top of this I've been adding my own tweaks. One thing I like to do is to keep track of what I'm working on at the current time. This helps me keep focused. Org-mode has a way for keeping track of what you are doing at any one time, which is a clock which you can start or stop on any particular entry. Using this as a basis, I wrote the following nifty function for binding the F9 key to go back to the last current item. I know an item is current if I use the clock-in feature.

(defvar ash-org-current-task-loc nil
"A cons of the buffer & location of the current task")

(defadvice org-clock-in (after ash-org-mark-task activate)
"When the user clocks in, bind F9 to go back to the worked on task."

(setq ash-org-current-task-loc (cons (current-buffer)
(point)))
(define-key global-map [f9] (lambda ()
(interactive)
(switch-to-buffer
(car ash-org-current-task-loc))
(goto-char
(cdr ash-org-current-task-loc)))))

This helps me mark a task as complete, I just hit F9, then C-c C-x C-c, which closes it out (stopping the clock and printing elapsed time), and marks the next item done. Pretty nifty!

This isn't enough, yet. I'm still tweaking the system, trying to perfect it to suit the particular problems I have. I may post more on this in the future.

The problem with GTD

2007-08-04T09:59:00.000-07:00

David Allen's productivity system, GTD (standing for Getting Things Done), continues to be more and more popular. If you are all at interested in being more productive, than likely you've heard of this already. I follow it myself, using emacs and org-mode. Well, actually, when I say I follow it, I currently only follow part of it, because part of it is just not useful.

There are a few key concepts in GTD, but the most often applied are next actions, contexts, and weekly reviews. I think that next actions are an important concept. The general idea is that in your todo list you break down tasks to a level that each are simple and directly actionable with no planning or meta-level thinking required. One of these is your next action, which is the very next thing you have to do to move your project forward. This is a cool thing, because it forces your tasks to be of the correct granularity. Also, most GTD software will show you all your next actions, so that you can see what you have to work on. Once you complete one next action, the next todo item will become a next action. For example, if your project is to fix a bug, then the next action here is to attempt to reproduce the problem locally.

Actually, this concept is slightly simplistic. Sometimes your project can have several next actions. If your project is to launch a new feature, then you might have several next actions, one to write the introduction and high level overview of a design document, the other is to send out a mail to your manager asking who could possibly be allocated to work on this feature. A good GTD system should let you mark items as next actions arbitrarily. org-mode does this, which is one of the reasons I like it.

Another key concept of GTD is the weekly review, which means that you update your tasks and take time to plan new ones. Good advice. I have nothing to add to it.

Finally, GTD has the notion of contexts, which are the bread-and-butter of most GTD apps. Contexts represent some state in which your actions are performed. One obvious place for contexts is for direct communication with someone. You might have several projects that have an action that requires you to talk to your project manager. You can associate those tasks with a context, and now you have a context that you look at to before you go speak to the project manager, to see all the appropriate tasks. Then when you go talk with them, you have a list of things to ask them.

Sounds vaguely useful. And that's the problem. For software engineers and other types of knowledge workers, we just don't have a lot of context. We are pretty well connected. Usually things that can be done at work in front of our workstation can also be done at home, through VPN. I find that most of my actions are to be done in front of my workstation, so the vast majority of my tasks have the context of the workstation. What I find useful is to not use contexts for these tasks, since they all have a default obvious context. I use it only for person-related contexts (such as the context for my project manager in my example). And even then it's only somewhat useful. It's possible that I could start doing work on my subway ride, in which case that is a useful context. The problem would be that things done on the subway is a subset of things I can do at work. Here again the concept of rigid contexts doesn't work. So when I'm at my workstation, every task if fair game. If I'm somewhere specialized, a specific context may come in handy. But, ultimately, for most modern "knowledge-workers", contexts are not something we should be spending a lot of time thinking about.

With contexts, org-mode again comes in handy. It's not a real GTD app, so it doesn't insist on contexts. It just lets you tag items, and those tags are your contexts. So you can put things in multiple contexts, or none. The tags inherit. I think it's really perfect flexible solution for the modern GTD user. Next post I'll detail exactly how I use it and give some customizations.

Generics

2007-07-04T10:30:00.000-07:00

In my last post, I discussed annotations, and their contributions to the Java language. The short version: they are pretty simple, and surprisingly useful. Generics, also introduced with Java 1.5, is a much larger pill to swallow. Not on the useful part, they are undeniably helpful. They do something very important, which is to make run-time errors visible at compile time, while at the same time increasing the self-documentation of code. This is a great thing. However, there is no doubt that they add complexity. To even document all cases of complexity would take a whole blog dedicated to just that subject. I'll give one example:

foo(List<B> list); bar(List<? extends B> list); tor(List<? super B> list);

So, what's the difference between these? Well, suppose B has a subtype C, and a supertype A. So the inheritance hierarchy goes A -> B -> C. You can call foo with a List<B>, but not a List<C>. But you can call bar with a List<C>; as well as List<B>. You can call tor with a List<A> and List<B>. Keep in mind that the contents of the list are not what I am talking about, just the compile-time type of the list.

Generics are a bit more complicated than most people assume. Almost anyone who has spent time with Generics has gotten into several puzzling situations that they can't explain, and where the solution isn't obvious.

The question I wonder about is whether it has done more harm than good. This is not a matter of opinion, I think. If we could count up all the hours of lost productivity in wrestling with generics, and count the hours saved by catching bugs and improving the clarity of the code, would the time saved exceed the time lost? My anecdotal evidence is that it is a net benefit. It has rescued the Java world from the perils of type-casting, and I do remember getting many ClassCastExceptions in the past. Nowadays they are rare, and I don't spend excessive time on the complexities.

Annotations

2007-02-13T20:05:00.000-08:00

This post is a few years too late. Annotations came out in Java 1.5, and we already are on Java 1.6. But I'd like to make a few comments on them, because I think they are a great example of a low-cost but a high-use feature.

Making improvements to a popular language is always difficult. You want each addition in functionality to be well worth it. Each addition should make things possible that were not possible before, and do so in a way that both fits with the spirit of the language, and results in the least possible disruption.

Java annotations are a good example of a good change. I find I use them more and more, and they enable things I now realize I in fact need. Did I need them before annotations were around? It didn't seem so, but now they are here I would find it difficult to live without them.

I should illustrate how useful they are with a made-up example. Let's say you want a simple webserver, one that took the path and used it to call into a dispatcher object. So you would call http://localhost/doSomething?foo=aaa&bar=bbb. This would use Java reflection to call a method Dispatcher.doSomething(String foo, String bar) . But at reflection time you don't know the names of arguments, since those are all compiled away. So, the first challenge is to figure out which argument is foo and which is bar. How can we do this? Before annotations, I would say you cannot do this. You could do something such as populate a String to String map, and pass that in instead of arguments, but this is awkward for the non-reflection callers, and also doesn't enforce important invariants at compile-time. Luckily for us, we do have annotations, and the annotations stick around at runtime (if you want them to). So, we can make an annotation defined as:

import java.lang.annotation.*;

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.PARAMETER)
public @interface ParameterName {
 String value();
}

And we can use it like so:


void doSomething(@ParameterName("foo") String foo,
                @ParameterName("bar") String bar) {
...
}

Now reflection can match HTTP parameters up to the method arguments.

But the fun doesn't stop there. What if you wanted your delegator class to only have some methods accessible from the web? You could try to separate into another class that is only web methods, but that overly constraining on the design. Or, you could maintain a separate list of web-safe methods in a configuration file. That solution seems prone to mistakes - developers will forget to update the configuration file, then wonder why their new method isn't accessible. However, with annotations, you can simply tag methods with an annotation showing that they are web-safe. We can create a runtime method-level annotation called WebExecutable and use it like so:


@WebExecutable
void doSomething(@ParameterName("foo") String foo,
                @ParameterName("bar") String bar) {
...
}

Now, your dispatcher can check for the annotation before it executes. Instant safety! You can imagine further annotations that may classify whether methods needs authentication or not, or other such improvements.

I've really been digging annotations. They make things possible that never were possible before, and don't introduce much complexity or confusion. There is some new syntax to learn, and some non-obvious things (such as "@interface"), but all in all, the part you have to look at the most is very easy and intuitive. I think it's an ideal addition to the language.

Next, I intend to look at generics, which has quite a different profile.

Language growth

2006-10-26T19:34:00.000-07:00

To alter a quote from Annie Hall, programming languages are like sharks. They have to keep moving or they will die. Languages frequently don't succeed at this. Thus a fine language like Common Lisp failed to include threads and networking in its spec, and by the time standards emerge, it is too late; the needs of users have long since passed by.

Java has done a good job of this. They have managed to keep incorporating the needs of users, while trying to stay true to the spirit of the language, and they have not proceeded so fast that they introduce unncessary complications. Some of their additions to the language spec have made things more complicated and confusing, but you always get something out of it.

I'm interested in exploring this topic further in the next few posts. I'd like to explore a few examples of Java language additions, and then examine some potential upcoming changes. What are your opinions about changes in the Java language in recent years?

Introduction

2006-10-19T16:33:00.000-07:00

Not only is this yet another blog, this is yet another in a series of blogs by me. I'm a believer in blogs that suit a specific topic, and therefore this is my blog for technical topics. For a developer, having a technical blog is a way to both refine one's thoughts on particular topics, practice technical writing, and of course advertise oneself as being a knowledgeable dude.

Every blog should know who its audience is. This blog's audience is other developers, mostly those who are searching for topics. Although it would be nice to have people subscribing to my atom feed, I do not expect it.

I program primarily in Java, with some Python, and some dabbling in lisp. Expect posts relevant to those topics, as well as posts about development in general, productivity, IDEs, and similar issues. Do not expect a flurry of posts. I expect to be posting at least once every two weeks, but more if I happen to have a lot to say, or less if I am unusually busy.

Oh, and I actually hope that this will be a good blog, and will actually contribute some knowledge or something of value to the world.