Don't Think - Just Write Tests, Right ?

I gave a brief talk on Continuous Integration at the local Java User Group a few weeks ago and I happened to mention that although I value automated regression testing very much I do not think it gives you any real guarantees that your software is error free.

I think this provoked some in audience a bit - I got questions along the line of "why do you hate unit testing so much?" and I realized that I unwittingly had stepped on some toes there. Sorry guys, I hope we are still friends.

Yes, I am a sceptic but for the record: I don't hate unit testing and I certainly do not think that people who do unit testing are idiots.

What I do have a problem with are when people set up their test suite, complete with red light for errors, green light for no errors - and then conclude the following:

• Red light: there is a bug in the application code
• Green light: there are no bugs in the application code

I think this assumption is incredibly naive and will try to explain why.

Your test system consists of two components: The application code and the test code itself (junit test cases, mocking etc.). Software bugs can be present in either component. This should be a known fact to anyone who have been working seriously with automated testing.

Consequently, red light means you have either

• A bug in your application code
• OR you have a bug in your test code!

I admit that this is very useful information and it also gives you an idea about where the problem lies - it is either in the code being tested or in the test itself.

Green light on the other hand is much more tricky. Really, all it tells you is that

1. You have no bugs anywhere...
2. OR .. you have a bug in your application code but the test that was supposed to cover it is faulty as well!
3. OR .. you have a bug in your application code that is not covered by a test

When we see the green light we want to believe that we are dealing with case #1 - no bugs. But we really have no way of knowing this for sure.

Case #2 is fairly common in large test systems and is generally detected by someone else down the line - either QA or the end user. How do know that your test code is error free anyway? Are you writing unit tests to test your unit tests? And if so, are you also testing the test code that is testing the original test code... ?

This brings us to case #3 which has to do with test coverage. Yes, you can minimize the case 3 bugs by writing more tests. There is a catch, however: All the new test code you write could potentially have bugs, thus introducing more of the case #2 scenarios.

If you have ten lines of test code for every line of application code it means that there is a ten times higher risk of bugs in your test code. There is some powerful mathematics working against us in our attempt to achieve a full, error-free test coverage.

Integration testing is a way to test more lines of application code with less lines of test code but the same principle applies: As your test suite grows, so does the likelihood of having bugs in the test suite.

All this doesn't mean that you should stop doing automated testing. What is does mean is that you should look at your test results with a certain amount of scepticism. A failed test means you have a problem - either with your application code or your tests. No failed tests just means that there are potential undetected problems out there. And the way to solve these problems may not be to just blindly write more test code. Don't fire your QA people just yet.

Robust EJB3 Sequence Generation Using JPA Locking

or Recovering from OptimisticLockException on JBoss 4.0.5

In a recent project we needed to generate unique ids in sequence from multiple threads.

To persist the current value of the sequence we used an entity bean

@Entity
public class SequenceEntity {

    ...

    @Id
    @GeneratedValue(strategy= GenerationType.AUTO)
    public long getId() {
        return id;
    }

    ...

    public Long getSequence() {
        return sequence;
    }

    ...

    @Version
    public Integer getVersion() {
        return version;
    }
    ...
}

and the code for incrementing and returning the next sequence value was essentially something like

    ...
    sequenceEntity = entityManager.find(SequenceEntity.class, sequenceId);
    entityManager.lock(sequenceEntity, LockModeType.WRITE);
    sequenceEntity.setSequence(sequenceEntity.getSequence() + 1);
    ...

Note that for JPA locking to work, the entity must have a field with a @Version annotation.

The call to entityManager.lock() blocks other threads from accessing the row while it is being updated. However, because we have to call find() first, there is a risk that another thread has already found and locked the row before we can call lock, resulting in an OptimisticLockException. The probability of this happening is not great but still significant, even for relatively modest loads.

The solution is of course to catch the OptimisticLockException and try again. However, to make this work the locking and incrementing should take place in its own transaction.

    ...
    do {
        try {
            nextSeq = sequenceEngine.getNextSequenceInNewTransaction(sequenceId);
        } catch (EJBException e) {
            if (e.getCausedByException() instanceof OptimisticLockException) {
                tries++;
            } else {
                throw e;
            }
        }
    } while (nextSeq < 0 && tries < maxTries);
    ...

The getNextSequenceInNewTransaction() method has a @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW) declaration. For this declaration to take effect, the method should be placed in another session bean (or, if one prefers, in the same bean, but called through the interface). So basically we end up with two stateless session beans, looking something like:

public class SequenceFacadeBean implements SequenceFacade {

    @EJB
    private SequenceEngine sequenceEngine;

    public long getNextSequence(long sequenceId) {
        int tries = 0;
        final int maxTries = 10;
        long nextSeq = -1;
        do {
            try {
                nextSeq = sequenceEngine.getNextSequenceInNewTransaction(sequenceId);
            } catch (EJBException e) {
                if (e.getCausedByException() instanceof OptimisticLockException) {
                    tries++;
                } else {
                    throw e;
                }
            }
        } while (nextSeq < 0 && tries < maxTries);

        if (nextSeq < 0) {
            final String msg = "Exceeded maxTries=" + maxTries + " attempts";
            throw new EJBException(msg);
        }

        return nextSeq;
    }
}

public class SequenceEngineBean implements SequenceEngine {

    @PersistenceContext(unitName = "sequenceUnit")
    private EntityManager entityManager;

    @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
    public long getNextSequenceInNewTransaction(long sequenceId) {
        SequenceEntity sequenceEntity;
        try {
            sequenceEntity = entityManager.find(SequenceEntity.class, sequenceId);
            entityManager.lock(sequenceEntity, LockModeType.WRITE);
        } catch (NoResultException e) {
            sequenceEntity = new SequenceEntity();
            sequenceEntity.setSequence(0L);
            entityManager.persist(sequenceEntity);
        }
        sequenceEntity.setSequence(sequenceEntity.getSequence() + 1);
        return sequenceEntity.getSequence();
    }
}

Not pretty, but the alternative would be to handle transactions manually and forgo declarative transaction management.

SFTP in Java with JSch Using Private Key Authentication

JSch is an excellent library for ssh in Java. One bad thing is that there is no real documentation - or rather, the source itself is the documentation. And the good news is of course that the source code is available along with a nice collection of examples.

I recently had to use SFTP from a Java application and it did take some source-diving to figure out exactly how it works (specifically to get the parameters for JSch.addIdentity right). It turned out to be rather straight-forward:

    ...

    private void execute() throws JSchException, SftpException {
        JSch jSch = new JSch();
        final byte[] prvkey = readMyPrivateKeyFromFile(); // Private key must be byte array
        final byte[] emptyPassPhrase = new byte[0]; // Empty passphrase for now, get real passphrase from MyUserInfo

        jSch.addIdentity(
            "myusername",    // String userName
            prvkey,          // byte[] privateKey 
            null,            // byte[] publicKey
            emptyPassPhrase  // byte[] passPhrase
        );
        Session session = jSch.getSession("myusername", "hostname.com", 22);
        UserInfo ui = new MyUserInfo(); // MyUserInfo implements UserInfo
        session.setUserInfo(ui);
        session.connect();
        Channel channel = session.openChannel("sftp");
        ChannelSftp sftp = (ChannelSftp) channel;
        sftp.connect();

        final Vector files = sftp.ls(".");
        for (Object obj : files) {
            // Do stuff with files
        }
        sftp.disconnect();
        session.disconnect();
    }

    ...

The MyUserInfo class must implement the UserInfo interface. There are a number of good examples of this in the jsch distribution.

Lean and Metaphors

A criticism of Lean Software Development is hard to find. While I think it offers a number of useful ideas and tools, I am somewhat sceptic towards the widespread use of metaphors from manufacturing found in Lean Software Development: An Agile Toolkit.. and elsewhere.

Metaphors are powerful in the way that they allow us to transfer insights from one area of experience to another. However, for these insights to be valid it is required that the metaphor has the same characteristics and dynamics as the system it is supposed to model. Or to put it more bluntly: Lean will solve all your problems to the extent that your software development organisation is like a manufacturing plant...

It is not all bad, though. The assembly line model may fail to capture all aspects of software development - but it does bring issues such as queues and waste into focus and provide a framework for discussing and adressing them. So even if Lean is not the final answer to everything it remains a useful tool for certain applications.

However, if software development is a field in its own right why do we then always have to use metaphors and analogues to describe it? It is clear that there is much to be learned from viewing software development as .. well, software development.

.NET Class Generation - Delegating Object Properties

In a recent assignment I had the need for a dynamically generated class that - among other things - could be generated with a field of arbitrary type and delegate the properties of said field. In other words something like

public class MyWrapperClass : MyParentClass
{
private ObjectType _WrappedObject;

public ObjectType WrappedObject
{
    set { _WrappedObject = value; }
}

public string Field1
{
    get { return _WrappedObject.Field1; }
}

public int Field2
{
    get { return _WrappedObject.Field2; }
}

// ... other stuff
}

where ObjectType can be any type and MyParentClass is a base class.

Fortunately, .NET reflection and class generation is pretty straight-forward. Generating the class itself requires an assembly, module etc.:

AppDomain appDomain = Thread.GetDomain();
AssemblyName assemblyName = new AssemblyName();
assemblyName.Name = "MyAssembly";
AssemblyBuilder assemblyBuilder
= appDomain.DefineDynamicAssembly(assemblyName, AssemblyBuilderAccess.Run);
ModuleBuilder moduleBuilder = assemblyBuilder.DefineDynamicModule("MyModule");
TypeBuilder typeBuilder
= moduleBuilder.DefineType("MyWrapperClass", TypeAttributes.Public);
typeBuilder.SetParent(typeof(MyParentClass));

The field holding our wrapped object must be declared to be the same type as the object to be wrapped

private ObjectType _ObjectToBeWrapped

This can be done with the following code:

Type objectType = wrappedObject.GetType();
FieldBuilder wrappedObjectField
= typeBuilder.DefineField("_WrappedObject", objectType, FieldAttributes.Private);

We will also need a setter for the object

public ObjectType WrappedObject { set { _WrappedObject = value; } }

And this requires some IL code generation:

// Create property ObjectToBeWrapped
PropertyBuilder wrappedObjPropertyBuilder
  = typeBuilder.DefineProperty("WrappedObject", PropertyAttributes.None,
  objectType, null);

// Create property setter
MethodBuilder wrappedObjSetBuilder =
  typeBuilder.DefineMethod("set_WrappedObject", MethodAttributes.Public,
  null, new Type[] { objectType });
ILGenerator wrappedObjSetILG = wrappedObjSetBuilder.GetILGenerator();
wrappedObjSetILG.Emit(OpCodes.Ldarg_0);
wrappedObjSetILG.Emit(OpCodes.Ldarg_1);
wrappedObjSetILG.Emit(OpCodes.Stfld, wrappedObjectField);
wrappedObjSetILG.Emit(OpCodes.Ret);
wrappedObjPropertyBuilder.SetSetMethod(wrappedObjSetBuilder);

Note that the "set { value = .. }" notation actually corresponds to a method "set_MyProperty(object value) {...", much like Java-style getters & setters.

Now for the fun bit: Create a getter for each property in the wrapped object. We iterate through the properties and generating a getter method works much like the setter above.

foreach (PropertyInfo propertyInfo in objectType.GetProperties())
{
  string name = propertyInfo.Name;
  if (IgnoreProperty(name))
  {
     continue;
  }
  PropertyBuilder propertyBuilder =
     typeBuilder.DefineProperty(name, PropertyAttributes.None,
     propertyInfo.PropertyType, null);

  MethodBuilder getBuilder =
     typeBuilder.DefineMethod("get_" + name, MethodAttributes.Public,
     propertyInfo.PropertyType, Type.EmptyTypes);

  ILGenerator getILG = getBuilder.GetILGenerator();
  getILG.Emit(OpCodes.Ldarg_0);
  getILG.Emit(OpCodes.Ldfld, wrappedObjectField);
  getILG.Emit(OpCodes.Call, propertyInfo.GetGetMethod());
  getILG.Emit(OpCodes.Ret);

  propertyBuilder.SetGetMethod(getBuilder);
}

JavaFX in IntelliJ IDEA

No, at this point in time there is no JavaFX plugin for IDEA. It is fairly simple to get it up and running, though.

1. First you want to get openjfx (subversion or .zip).

2. Create a Java project in IDEA and put the files from trunk/lib (Filters.jar, javafxrt.jar & swing-layout.jar) in the classpath (e.g. add as a project library).

3. Create a .fx file


4. Create a run configuration

Basically you want to run net.java.javafx.FXShell with your .fx file as an argument.

4. Make sure that your .fx file is in the classpath
..Otherwise it won't work

As for syntax highlighting this can be set up by making a file type definition for *.fx files:



So all in all pretty ugly. A real JavaFX plugin is still on the IDEA wishlist.

And yes, there is already a plugin for Eclipse. There is also a fairly comprehensive tutorial using NetBeans on openjfx.dev.java.net