Automating Selenium testing with TestNG, Ant and CruiseControl

If you are using Selenium for web testing, most likely you’ll want to make your tests as automated as possible so that they can be run automatically on a regular basis. You can do this by using a few additional technologies:

1. Integrate with a Java test framework

For any serious testing, your tests will need to be written in Java, rather than the basic Selenium html format. You can save the tests in Java from the Selenium IDE by going to Options -> Format. You have a choice of either TestNG or JUnit format. Personally I prefer TestNG as it is a more powerful framework. I’m not going to give a tutorial on how to using TestNG as there is plenty of information on the web for this, but the key features are:

  • Easy to define what tests you want to run and their parameters by using a testng.xml file.
  • Parameters can be automatically passed to tests using the @Parameters annotation.
  • For data driven tests, data provider methods can be used to invoke the tests multiple times with different sets of input and expected output.
  • Assertion mechanism for the checks in your tests.
  • HTML reporting.

2. Write an Ant script to compile and run your tests

The main things your Ant script needs to do:

  • Compile your tests.
  • Start the Selenium RC server (assuming you are using Selenium 1).
  • Run the tests.
  • Stop the Selenium server.

Here is a sample of what your script should look like:

?Download build.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
<project name="Selenium tests">
  <!-- The tests should only depend on three libraries: Selenium, TestNG and the MySQL Java Connector. -->
  <property name="ant.dir" location="/ant-1.7.0" />
  <echo message="Ant directory: ${ant.dir}" />
 
  <property name="lib.ant.dir" location="${ant.dir}/lib" />
  <echo message="Ant lib directory: ${lib.ant.dir}" />
 
  <!-- define TestNG location -->
  <path id="lib.testng">
    <fileset dir="${lib.ant.dir}" includes="testng-5.11-jdk15.jar"/>
  </path>
  <taskdef resource="testngtasks" classpathref="lib.testng" />
 
  <!-- define Java connector location -->
  <property name="lib.mysql.connector" location="/mysql_connector" />
 
  <!-- use the Antelope if task rather than the standard Ant one -->
  <taskdef name="iftest" classname="ise.antelope.tasks.IfTask"  
    classpath="${ant.dir}/antelope/AntelopeApp_3.1.9.jar" />
 
  <!-- define source location -->
  <property name="src.selenium-tests.dir" location="../src/selenium" />
 
  <!-- Compile tests -->
  <target name="compile-selenium-tests">
    <!-- Create the classes directory if it doesn't exist -->
    <iftest>
      <bool>
        <available file="classes" type="dir" />
      </bool>
      <echo message="Classes directory already exists" />
      <else>
        <echo message="Classes directory does not exist, creating..." />
        <mkdir dir="classes" />  
      </else>
    </iftest>
    <!-- CruiseControl seems to have problems picking up a Windows environment -->
    <!-- variable to say where the javac compiler is, -->
    <!-- so if we set fork="yes" then we can specify the location with -->
    <!-- the executable parameter. -->
    <javac srcdir="${src.selenium-tests.dir}" destdir="classes" fork="yes" 
	  executable="C:\usr\java\jdk1.6.0_06\bin\javac.exe" >
      <classpath>  
        <fileset dir="${lib.ant.dir}" includes="selenium-server-standalone-2.0b3.jar"/>
        <fileset dir="${lib.ant.dir}" includes="testng-5.11-jdk15.jar"/>
        <fileset dir="${lib.mysql.connector}" includes="mysql-connector-java-5.0.8-bin.jar" />
      </classpath>
    </javac>
  </target>
 
  <!-- Targets to start and stop the Selenium RC server. Taken from the Selenium documentation -->
  <!-- http://wiki.openqa.org/display/SRC/Selenium-RC+and+Continuous+Integration -->  
  <target name="start-selenium-server">
    <echo message="Starting Selenium RC server (runs as a separate process)" />
    <!-- To make it easier to see if the Selenium RC server is running, we copy the -->
    <!-- normal java.exe to create a different executable called -->
    <!-- javaSeleniumRCServer.exe. That way, if you list the processes Windows is running -->
    <!-- you can see if the Selenium RC server is running, rather than just seeing a -->
    <!-- process called "java". -->
    <java jar="${lib.ant.dir}/selenium-server-standalone-2.0b3.jar" fork="true" 
	  spawn="true" jvm="javaSeleniumRCServer">
      <!-- Enable logging for the Selenium server -->
      <arg line="-log selenium.log -browserSideLog"/>
    </java>
  </target>
 
  <target name="stop-selenium-server">
    <echo message="Stopping Selenium RC server" />
      <get taskname="selenium-shutdown" 
	    src="http://localhost:4444/selenium-server/driver/?cmd=shutDownSeleniumServer"
        dest="result.txt" ignoreerrors="true" />
      <echo taskname="selenium-shutdown" message="DGF Errors during shutdown are expected" />
  </target>
 
  <!-- Run tests. -->
  <target name="run-selenium-tests" depends="compile-selenium-tests>
 
      <property name="build.classes.dir" location="classes" />
 
      <!-- Start the Selenium RC server -->
      <antcall target="start-selenium-server" />
 
      <echo message="Running Selenium Tests." />
      <!-- Put the output in the same place as our other tests -->
      <!-- We don't want to immediately fail the build if the tests fail, as we -->
      <!-- need to stop the Selenium server, so we set the property testsFailed -->
      <!-- and check that AFTER we have stopped the Selenium RC server. -->
      <testng outputdir="test-output" haltonfailure="false" 
	    haltonskipped="false" failureProperty="testsFailed">
        <!-- Compiled classes are in the build/classes directory -->
        <classpath>
           <pathelement location="${build.classes.dir}" />
          <fileset dir="${lib.ant.dir}" includes="selenium-server-standalone-2.0b3.jar"/>
          <fileset dir="${lib.ant.dir}" includes="testng-5.11-jdk15.jar"/>
          <fileset dir="${lib.mysql.connector}" includes="mysql-connector-java-5.0.8-bin.jar" />
        </classpath>
        <!-- List of tests to be run is in the testng.xml file in the src/system_tests/selenium directory -->
        <xmlfileset dir="${src.selenium-tests.dir}" includes="testng.xml"/>
      </testng>
 
      <!-- Stop the Selenium RC server -->
      <antcall target="stop-selenium-server" />
 
      <!-- CruiseControl needs the test results in JUnit format, it can't read TestNG output. -->
      <!-- However, TestNG comes with an Ant command called junitreport that will conver the output. -->
      <mkdir dir="${base_dir}/unit_tests"/>
      <junitreport todir="${base_dir}/unit_tests">
        <fileset dir="${base_dir}/build/test-output">
          <include name="*/*.xml"/>
          <exclude name="*/testng-failed.xml"/>        
        </fileset>        
        <report format="noframes"  todir="${base_dir}/unit_tests"/>
      </junitreport>
 
      <!-- Check the flag set by TestNG and fail the build if we had test failures. -->
      <!-- Unfortunately using this mechanism it looks like we can't check for -->
	  <!-- skips, only failures. -->
      <iftest name="testsFailed" exists="true">
        <fail message="Test failures."/>
      </iftest>
  </target>
</project>

3. Check your tests into a source control system

Although you can get away without using source control, for any serious testing it’s a must. It will make it easier to keep track of changes to the tests and maintain multiple versions of them if you need to. The most commonly used free source control systems are Subversion and Git, although my example below uses Perforce.

4. Set up CruiseControl to run the tests

Although I’d recommend using the CruiseControl Java GUI (CC-Config) for configuring CruiseControl, under the covers it is just making changes to the config.xml file, and to explain the settings it is easiest just to give an example of that file:

?Download config.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<cruisecontrol>
  <dashboard />
  <project name="Selenium tests" requiremodification="false">
    <modificationset>
      <p4 View="//Selenium_dp/..." Port="9.20.123.456:1666" 
	    Client="Selenium_workspace" User="auto" Passwd="password" />
    </modificationset>
    <schedule Interval="86400">
      <ant Target="run-selenium-tests" 
	    AntHome="C:\Program Files\CruiseControl\apache-ant-1.7.0" 
	    BuildFile="C:\Selenium_tests\build.xml" UseDebug="true" />
    </schedule>
    <log />
    <listeners>
      <currentbuildstatuslistener 
	    File="C:\Program Files\CruiseControl\logs\status.html" />
    </listeners>
    <bootstrappers>
      <p4bootstrapper Port="9.20.123.456:1666" View="//Selenium_dp/..." 
	    User="auto" Passwd="password" Client="Selenium_workspace" />
    </bootstrappers>
  </project>
</cruisecontrol>

I’ll explain the various parts of the configuration:

  • project – requiremodification is set to false. If set to true, CruiseControl will only run the build if changes have been made to the code. For building a product, this makes sense, but not for running tests. The product you are testing could have changed, so you still want to rerun the tests even if they haven’t changed themselves.
  • modificationset – tells CruiseControl how to check if any of your tests have been updated in your source control system. This example shows the syntax for connecting to Perforce.
  • schedule – tells CruiseControl how to run your build, and how often. The time interval is measured in seconds, so an interval of 86400 seconds means the tests will be run once a day.
  • listeners – defines how to see the current progress of the build. I’ve just set this to the default log location.
  • bootstrappers – bootstrapping is what occurs before every build. In this configuration, I’ve set the bootstrapper to get all of the test files from source control. For large projects, this wouldn’t make sense, as the bootstrapper will get all of the files regardless of whether any of them have actually been updated. In that situation, you would usually write your Ant script so that it contains targets that can extract your files from source control. The bootstrapper would just get the Ant script, and only if a modification was detected in the files would a build run and the modified files be extracted from source control. However, for a small or moderately sized set of test files, the current configuration is fine.

Useful links

TestNG: http://testng.org/doc/index.html
Apache Ant manual: http://ant.apache.org/manual/index.html
CruiseControl config.xml: http://cruisecontrol.sourceforge.net/main/configxml.html
My blog post on Selenium and XPath: Tutorial: Writing XPath selectors for Selenium tests

This entry was posted in Ant, Selenium, Testing and tagged , . Bookmark the permalink.

2 comments on “Automating Selenium testing with TestNG, Ant and CruiseControl

  1. Zack L on said:

    Great write-up. This is very close to what I’m trying to do. I have a couple questions.

    I know you said CruiseControl will only take JUnit style reports and that testNG has the junitreport task. However, due to the structure of my project, the JUnit reports from that list all my tests 2 times. There is another JUnit report that is generated by default without the Ant Task in testNG6.1, but this report is the XML style, ready to pull up in Eclipse.

    So my question is, can CruiseControl look at the JUnit Eclipse Style XML (ex. ‘TESTS-TestSuites.xml’) or does it have to be ‘junit-noframes.html’?

    Also, I didn’t see where you tell CruiseControl where the Junit Report is. I’m very new to CruiseControl, so forgive my noobiness.

    • hedleyproctor on said:

      Sorry, I didn’t explain how you tell CruiseControl about the test results. You do it by merging the test results into the log with something like:

      <log>
      <merge dir=”your_path/test-results”/>
      </log>

      Once you’ve merged the test results into the log file, they should appear on the test results tab of the CruiseControl admin page and build status page. I don’t think CruiseControl will understand the raw XML but I can try it on my local CruiseControl instance and tell you want it does.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

HTML tags are not allowed.