MutationTesting

Mutation testing tools: Jester, Jumble & Test Police

Squirrel, Wolf, Eric, Ulrich, Erik, Mieke, Robert, Marteijn, Mark, David, Ivan, Kirsten

Jester is a mutation test tool that change the code. If the tests still pass, then it means that tests are missing. Problems: it is hard to integrate it with build scripts, and with multiple tests. Promise of a new release! It is too slow to run on an entire project.

Jumble: similar, but changes bytecode, not source.

• Problems: difficult to use, not clear how well maintained. Also, Class.getPackage() doesn't return the package name you expect.

some metrics:

• 5 seconds per compilation
• 20 mutations per class, each requiring compilation
• 1200 classes
• 120000 seconds to run = 33.3 hours

Ajester: similar to Jester, but with bytecode. The one interesting thing is that it is running on the classes that already have a good code coverage.

Martijn: it'd be nice if you could detect tests that always find out the same changes in the code, as tey would be redundant.

Ivan: an "MD5 test" is when a test is almost a rehash of the code, and it makes it harder to change the code.

David: it'd be nice if we could combine Jumble and Emma, and get reports that tell which lines are exercised by which tests.

muJava (uJava? cannot get the mu Greek letter on my keyboard): a nasty competitor to Jester! does mutation of classes

Iven: there is probably a tool integrated with Maven. Must find it again. Must also look at Hansel (& Gretel)

Tuomas: I'd like to look into infinitest, and see if that could be used to show which lines are exercised by which tests: http://code.google.com/p/infinitest/ also, I'm interested in seeing how mutation testing can properly identify which abstract superclasses should be mutated. (I wrote Test Police)

Links to tools

• Jumble: [1]
• Jester: [2]
• Test Police: [3]
• MuJava: [4]
• More links on [5]