Showing unused TestNG tests

We use TestNG for our system tests. One of the slight problems is that if a new test is written, but not added to the testng.xml file, it won’t be run. I like to monitor our testing progress, so I decided it would be nice to be able to see what tests exist in the code, but aren’t currently configured to run. There are a few different ways you can achieve this, depending on your test configuration, and how accurate you need the report to be. We store all of our system tests in a specific directory, so it is easy to access the classes. However, not all of the classes in the directory have to be tests, some of them could be helper files. For this reason, I used the directory to get an initial list of classes to work from, but then used reflection to check what classes actually have the TestNG “Test” annotation. The code was:

 // first get a list of all the system test classes
 List<File&gt; systemTestFiles
   = FileListing.getFileListing(new File(systemTestSourceDirectory));
 // not all of these will be tests. Some of them will be helper classes,
 // so iterate over the corresponding class files and check which ones
 // have test methods in.
 for (File testFile : systemTestFiles) {
   // need to ignore the testng.xml file
   if (testFile.getName().endsWith(".java")) {
     // get the package name. The format is:
     // package
     // one or more spaces s
     // package name, which is non-whitespace S - put brackets round this
     // to capture it
     // semi-colon
     Set<GrepMatch&gt; matches = Grep.search("packages+(S+);", testFile);
     // there really should be only one match
     final String packageName = matches.iterator().next().group1;
     // now get the class name. It is the name of the file, up to the .java
     final int endOfName = testFile.getName().indexOf(".");
     final String fullyQualifiedClassName = packageName + "."
       + testFile.getName().substring(0, endOfName);
     // load this class and check if it has test methods
     Class c = Class.forName(fullyQualifiedClassName);
     for (Method m : c.getMethods()) {
       if (m.isAnnotationPresent(Test.class)) {
         testClasses.add(fullyQualifiedClassName);
       }
     }
 }

I then used another grep to get the test classes in the testng.xml file:

Set<String&gt; classesInTestNGXML =
  Grep.searchAndReturnCapturingGroup("<classs+name="(S+)"",
                   new File(testNGXMLFileName));

After this it is easy to compare the two lists and print out the names of the tests that aren’t being run. The grep code I used was an enhanced version of the example given by Sun: http://download.oracle.com/javase/1.4.2/docs/guide/nio/example/Grep.java and the file listing code is based on: http://www.javapractices.com/topic/TopicAction.do?Id=68