Data

Humans are not very good at remembering event frequency statistics. We tend to only remember the unusual, and forget about the common case.  There are plenty of other cognitive errors we make, but this seems to me one of the most troublesome.

An example: Every day I take the subway from Forest Hills in Queens to Manhattan.  Either the E or F train will arrive first, and I need the E train. I was noticing, though, that it always seemed to be the F train arriving first, making me wait for an extra few minutes.

Why would that be? Could it be a situation like Feynman's girlfriend puzzle, where the E train closely follows the F?  I decided to keep some statistics to find out.

Every day, I'd record which train came first. After a few days, I noticed the E had been coming first lately. Surely a fluke.

I kept at it, and before long the trend was clear. The E comes first almost twice as often! I counted 18 E firsts, and only 10 F firsts.

Obviously, I was completely wrong about even the most basic statistics for my morning commute, and that's about something I experience every single workday morning. Think how wrong I can be about everything else!

I take this as a warning about not trusting experiences when making decisions. Get data instead. Even just some data will be better than opinion or memories.

Does everything have to be social?

All I wanted to do was to download a document. To do that, I had to sign up for Scribd via my Facebook account. Next thing I know, I am getting follower notifications emails from some of my Facebook friends.

I just wanted to download a document. Now I'm connected to all sorts of friends. What do they want from me? Did they even explicitly choose to friend me or was it automatically done by Scribd?

For the time being, I'm just going to ignore it all and hope it goes away.

Macros and meta-macros

Life is repetition.  We do the same things, day after day.  Within each day at work, we find ourselves performing the same actions, more or less.   The great thing about computers is that tasks performed on computers can be automated.  So any time we find that we are repeated ourselves on the computer, we have an opportunity of creating a tool that will allow us to eliminate repetition.

Macros are a great example of this, and one of the defining feature of lisp.  Sure, c++ has macros as well, and they share same of the same capabilities as lisp's macros, but in general the power of lisp macros make them better for everyday use.  Macros are great tools to reduce repetitive code to simple forms.  Code such as:


Seems reasonable enough if you just are opening up one file.  But what if you do this all the time?  Even the simple boilerplate here starts to add up.  That's why common lisp created a macro of an easy way to do this:


What a breath of fresh air that is... there is absolutely nothing wasted here.  All repetitive and wasted code has been removed, leaving only the essence of the task.  That's the art of the macro.  It's one thing to be able to write macros, but to recognize patterns of repetition is a skill worth practicing.

Meta-macros are a term I just made up for this post, which describes repetitive coding tasks we have to perform.  To take a simple example, perhaps you are always navigating to the same files.  Like macros, it is essential that we realize we are doing something repetitive, and that this task can be removed with automation.  In this case, emacs provides a solution: bookmarks.  Many times the solution isn't provided by emacs, though, or the patterns are more complicated.   Every time I change or add a new function in my .cc file, I have to edit my .h file as well.  This is just a natural part of editing C++ code, something so ingrained in us that it's hard to even imagine not going through the trouble of essentially adding function definitions twice.  It's a plainly repetitive task, which could certainly be automated with a meta-macro, by which I mean some elisp code.

Sometimes these meta-macro's automation can increase complexity.  It works fine 98% of the time, but 2% of the time it screws you up and you stop using the meta-macro in frustration.  But macros can have the same trait.  Good macros are actually a bit hard to write, and can be very confusing when they fail.  Meta-macros can be even worse, due to the complexity of the meta level in which we operate.  It takes dedication to solve each new problem you encounter, making your meta-macro more and more robust.  But, with enough effort, you will have an elegant solution that makes you more productive, and able to focus on problem solving, not mere coding.

When you do, please share it with the word.  We all need the help.

Breadcrumb mode for emacs

I found this useful mode via reddit: breadcrumb.  It allows you to define bookmarks with a keypress.  The bookmarks are unnamed, but you can jump between them, either between-buffer or within-buffer.  This should be really useful for working within a changelist.

One thing that may be useful would be loading or saving breadcrumb state, as I switch between working on different projects.  So I may have to add this (it probably isn't that hard).

Saner ansi-term in emacs

I like ansi-term in emacs. It is the closest to a "real" terminal, offering all the goodness that bash, zsh, or whatever your favorite shell can give you. However, you definitely lose a few things in going to it. The following emacs code restores at least the things I find most useful: dabbrev-completion, yanking, and correct colors.

When you use this code, note that dabbrev-completion is C-c /, and yanking is C-c C-y.

Android, Scala, and java.rmi.RemoteException

For fun, I've been experimenting with writing an Android application using scala. Now, being a conscientious developer, I'm following the principles of TDD, at least with the low-level database interface.

This proved a bit hard to do. I won't go into the details of integrating Scala and Android, since Christian Neukirchen did a good job of that with his article Programming for Android with Scala.

After following the advice in that class, you basically have to double it to compile, dex, and trim the test tree as well.

The big issue I had was trying to use ScalaTest with Android testing. As far as I know, no one has tried to do this yet.

First, Android testing is very particular. You can't just run any old class on the phone, instead there is a particular runner called the InstrumentationTestRunner to run tests. So, you need either Android-specific test classes, or JUnit classes. I therefore had to write a converter class:

class ScalaUnitTestRunner extends TestCase {
  def testExerciseList() = {
    new ExerciseListDaoSpec().execute()
  }
}

Simple and needs improvement, but it's the most basic thing that can work. The execute call runs the scalatests in that test class.

The problem was that when running, I would get this error on running:

[exec] android.test.suitebuilder.TestSuiteBuilder$FailedToCreateTests:
[exec] Error in testSuiteConstructionFailed:
[exec] java.lang.RuntimeException: Exception during suite construction
[exec]     at android.test.suitebuilder.TestSuiteBuilder$FailedToCreateTests.testSuiteConstructionFailed(TestSuiteBuilder.java:239)
[exec]     at java.lang.reflect.Method.invokeNative(Native Method)
[exec]     at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:164)
[exec]     at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:151)
[exec]     at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:418)
[exec]     at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1520)
[exec] Caused by: java.lang.TypeNotPresentException: Type java.rmi.RemoteException not present
[exec]     at java.lang.Class.getDeclaredMethods(Native Method)
[exec]     at java.lang.ClassCache.getDeclaredPublicMethods(ClassCache.java
[exec]     at java.lang.ClassCache.getDeclaredMethods(ClassCache.java:179)
[exec]     at java.lang.ClassCache.findAllMethods(ClassCache.java:249)
[exec]     at java.lang.ClassCache.getFullListOfMethods(ClassCache.java:223)
[exec]     at java.lang.ClassCache.getAllPublicMethods(ClassCache.java:204)
[exec]     at java.lang.Class.getMethods(Class.java:1038)
[exec]     at android.test.suitebuilder.TestGrouping.getTestMethods(TestGrouping.java:79)

...

The problem was that for some reason every Scala class contains a special method called $tag. If you used javap to investigate, you'd find the class looks like:

~/src/AshExercises $ javap -classpath testbin/classes com.ash.exercisetests.db.ScalaUnitTestRunner
Compiled from "ScalaUnitTestRunner.scala"
public class com.ash.exercisetests.db.ScalaUnitTestRunner extends junit.framework.TestCase implements scala.ScalaObject{
public com.ash.exercisetests.db.ScalaUnitTestRunner();
public void testExerciseList();
public int $tag()       throws java.rmi.RemoteException;
}

Because Android's tester is using reflection here to find the test methods, it encounters this method with a class that Android doesn't know about. I tried to define a fake java.rmi.RemoteException, but Android is smart and doesn't let you define core java classes.

So, I had to change my tester to this:

class ScalaUnitTestRunner extends TestCase {
  def testExerciseList() = {
    new ExerciseListDaoSpec().execute()
  }

  override def $tag() : Int = {
    try {
      return super.$tag();
    } catch {
      case e: Exception => throw new RuntimeException(e);
    }
  }
}

That fixed that issue.

The next issue it had was that Android was giving me java.lang.VerifyError on running. This turned out to be my fault, but what I had to figure out was that java.lang.VerifyError really meant a ClassCastException. I found this out by looking in the logcat.

E/dalvikvm(  726): Could not find class 'com.ash.exercise.db.ExerciseDatabase', referenced from method com.ash.exercisetests.db.ExerciseListDaoSpec.<init>
W/dalvikvm(  726): VFY: unable to resolve new-instance 13 (Lcom/ash/exercise/db/ExerciseDatabase;) in Lcom/ash/exercisetests/db/ExerciseListDaoSpec;
W/dalvikvm(  726): VFY:  rejecting opcode 0x22 at 0x0019
W/dalvikvm(  726): VFY:  rejected Lcom/ash/exercisetests/db/ExerciseListDaoSpec;.<init> ()V
W/dalvikvm(  726): Verifier rejected class Lcom/ash/exercisetests/db/ExerciseListDaoSpec;
I/TestRunner(  726): failed: testExerciseList(com.ash.exercisetests.db.ScalaUnit

It turned out I was compiling against the non-test src, but wasn't including the non-test src in my proguard step, so that the classes never wound up in my apk.

I hope these tidbits help someone who is exploring Android, Scala, and unit tests. Good luck, and happy coding!

Distractions

Distractions, or no distractions? For your normal coder, most would prefer no distractions. A distraction at the wrong moment could lose a train of thought, wipe out working memory, and set the programmer back by half an hour of time. Not good. And it is common for programmers to feel overwhelmed with distractions.

However, distractions are useful to the programmer as well. Without distractions, a coder would never get important information. They would never hear a conversation about a problem that they have unique insight into. An interesting fact discussed by two colleagues would go unheard and unremembered. Distractions are unplanned and often
random, but that's often why they are beneficial. The benefits of locality via distractions is dealt with in The Social Life of Information by John Seely Brown and Paul Duguid.

A theoretically distraction-free programmer may get a lot done, but they also deprive others of their insight, and they may miss out on random learning opportunities. This is a problem, because the job of the programmer is not only to design, implement features, or fix bugs, but also to exchange information to help others, and to better themselves. These are the valuable kind of distractions programmers need.

Not only work-related distractions are helpful. Taking a break from work entirely is helpful. Standing up and moving around every once and a while is healthy. Doing fun things with coworkers such as playing foosball promotes camaraderie and teamwork.

We need distractions, although clearly too much is a problem. So, the question is how to process distractions and keep our sanity at the same time.

The first thing to realize is that a distraction creates stress because it causes a prioritization conflict. A developer is supposed to be working on x, but now is distracted and doing task y. Well, there are a few possibilities:

x is more important than the distractor y

A fairly normal case, that comes in a variety of flavors. For example, you are programming very productively when you decide to check your mail. This was a mistake. Now you are out of the mindset of x, and are dealing with some mail y that is not ultimately as important as getting your work done. Or, you are programming and get distracted by an unimportant IM message. Or, you are are programming and the loud conversation next to you is constantly pulling your attention away. For all these things, working in a distraction-free environment for at least part of your time is useful to get rid of part of these distractions, and training your mind not to jump around and want to check mail and surf the web is a way to get rid of the other part. Meditation and just plain practice in focusing is important here.

Even though being distraction free for a while is useful, this shouldn't be an all-day activity, even during rush times. Spend some time and open yourself up to distractions for part of your day. You will learn unexpected things, engage in conversation and relaxation, help people, and get to do all the tiny tasks that accumulate throughout the day.

x is less important than the distractor y

For example, you are working on an important refactoring of your code. All of the sudden, your coworker leans over and tells you that your last changelist broke the build. Obviously, you have to drop what you are doing and fix the problem. You lose context and time, but that's unavoidable. Hopefully, this kind of thing is rare. If it isn't, you may want to check that you have the proper processes in place.

If there are still too many urgent interruptions, then it would help to have one person process all the interrupts of the team. For example, for one week Jane Programmer handles all interrupts for the team, and does not attempt to do non-interrupt-based work at all. This is actually psychology comforting, because she doesn't have to worry about her interrupts slowing her down on her main project. She's already decided that the only work she gets done is what are normally distractions. And if she finds herself distraction-free, she can always do little code cleanups and fixes that she always intended to do but never got around to. She's happy and the rest of the team is more productive by not having to worry about distractions. This should more than offset any productivity loss by Jane Programmer's not working on her project.

Hopefully, by following rules like these we can manage to use the opportunities distractions provide, as well as to minimize their impact.