Nelz's Blog

Today, I set up a Maven module to run SeleniumRC (Java) tests via TestNG. It wasn’t a straightforward or easy task, so I thought I’d share some of what I found with you.

We opt to have all of our modules stay in lock-step version-wise, so we keep a pretty large structure in our SVN trunk. We include/exclude modules for compilation by using Maven profiles, so we don’t have to build the whole structure if we are only changing one small component. Our top-level pom.xml file looks similar to this:

<project>
  <groupId>...
  <artifactId>...
  <version>...
  <packaging>pom</packaging>
  <modules>
    <module>baseModule</module>
  </modules>
  <profiles>
    <profile>
      <id>all</id>
      <modules>
        <module>otherModule</module>
        ...
        <module>web-test</module>
      </modules>
    </profile>
    <profile>
      <id>web-test</id>
      <modules>
        <module>web-test</module>
      </modules>
    </profile>
  </profiles>
</project>

As you can see, running “mvn clean install” at this top level would only end up building the baseModule. To get all the modules, we run “mvn clean install -Pall” which builds baseModule along with otherModule and web-test and anything else as represented by the ellipse. The web-test module is the one we will be interested in.

For our heavy-weight tests (non-unit tests, like functional or web testing), we like to have the module compile, but not run unless we specifically trigger the tests. To achieve this, we create a pom.xml (for the web-test module) that looks something like this:

<project>
  <parent>...
  <artifactId>web-test</artifactId>
  <packaging>pom</packaging>
  <build>
    <testSourceDirectory>src/it/java</testSourceDirectory>
    <testResources>
      <testResource>
        <directory>src/it/resources</directory>
      </testResource>
    </testResources>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>2.0.2</version>
        <configuration>
          <source>1.5</source>
          <target>1.5</target>
        </configuration>
        <executions>
          <execution>
            <goals>
              <goal>testCompile</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.4.2</version>
        <configuration>
          <testSourceDirectory>src/it/java</testSourceDirectory>
          <excludedGroups>web-test</excludedGroups>
        </configuration>
        <executions>
          <execution>
            <phase>integration-test</phase>
            <goals>
              <goal>test</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

This pom has several features:

• It sets up src/it/java as the location holding the source code for the tests
• It sets the surefire plugin (which runs your tests) to only run at the integration-test part of the Maven lifecycle
• It says to ignore any tests in the TestNG “web-test” group
• Note the packaging says this module is of the type “pom”. I found that if this is set to “jar”, the tests run twice for some unknown reason

Now, if you had tests in src/it/java (that were all annotated as “web-test”) and you ran “mvn clean install”, you’d see that zero tests ran. This is what we want! Now, we have to get the tests running if we pass a specific profile in. To achieve this, I added the following profile definition to the web-test pom.xml.

<profiles>
    <profile>
      <id>run-web-test</id>
      <build>
        <plugins>
          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.4.2</version>
            <configuration>
              <suiteXmlFiles>
                <suiteXmlFile>src/it/resources/testng.xml</suiteXmlFile>
              </suiteXmlFiles>
              <!--<groups>web-test</groups>-->
              <!--<forkMode>always</forkMode>-->
            </configuration>
            <executions>
              <execution>
                <phase>integration-test</phase>
                <goals>
                  <goal>test</goal>
                </goals>
              </execution>
            </executions>
          </plugin>
        </plugins>
      </build>
    </profile>
  </profiles>

This profile is invoked when you run “mvn clean install -Prun-web-test”. There are several things I’d like to point out here:

• At runtime, this overrides the previously-declared surefire plugin configuration, but only when invoking the profile.
• Originally, I was mostly getting stuff to work without the <suiteXmlFiles /> section, but with the <groups /> and <forkMode /> sections… But it was running every single test class as it’s own TestSuite.
• To fix the multiple TestSuite (and @Before*, @After* problems that I’ll discuss later), I ended creating my own testng.xml file. Having this file and using the <suiteXmlFiles /> element made things a whole lot easier for configuring my TestNG executions than trying to configure the tests through the clumsy interference of the surefire plugin.
• As much as I like Maven, I have to say that the lack of documentation and general obfuscation that happens when using their plugins can really drive a person BATTY!

Now, I need to start integrating the stuff I need for using SeleniumRC. I found some decent help on how to start up and shut down the selenium-server, however I disagree with using a post-integration-test phase to shut down the server, because if the tests fail in the integration-test, Maven won’t progress to the next phase and you’ll be left with a spuriously running selenium-server. The client API’s do have access to the command that the was being run to shut the server down, so it is just a matter of having some TestNG @AfterSuite or @AfterGroup method issue the Selenium.shutDownSeleniumServer() command.

This brings me to some of the TestNG wonkiness I saw. I decided to use a Singleton Data Provider (named  SeleniumFactory) to deliver the Selenium objects to the individual tests. (Selenium objects are time-expensive to create, so I wanted to only create 1 for the duration of the entire test suite.) I wanted to initialize the SeleniumFactory object in a @BeforeGroup method, deliver the Selenium object as a Data Provider, and then tear the SeleniumFactory down in a @AfterGroup method. (At first, I was fighting Maven’s handling of each test as its own test suite which was giving me sequence issues, which is why I ended up defining my own testng.xml file, as I stated above.) Let me start with some code to show you what I mean:

public class SequenceEndsTest {
    @BeforeGroups(groups = {"web-test"})
    public void startupSelenium() {
        System.out.println("-- > Called startup.");
    }
    @AfterGroups(groups = {"web-test"}, alwaysRun = true)
    public void cleanupSelenium() {
        System.out.println("-- > Called cleanup.");
    }
}
public class SeleniumFactory {
    @DataProvider(name = "selenium")
    public static Object[][] getSelenium() {
        System.out.println("-- > Called data provider.");
        return new Object[][] { new Object[] {null}};
    }
}
public class SequenceTest {
    @Test(groups={"web-test"},
            dataProvider = "selenium", dataProviderClass = SeleniumFactory.class)
    public void testSomething(final Selenium selenium) {
        System.out.println("-- > Called test.");
    }
}

This is what I expected to see after running this code:

-- > Called startup.
-- > Called data provider.
-- > Called test.
-- > Called cleanup.

But this is what I *actually* saw:

-- > Called data provider.
-- > Called startup.
-- > Called test.
-- > Called cleanup.

I would have assumed that an @BeforeGroup method got called before a data provider method, but I guess my assumption is wrong, eh? After I figured out this order-of-method-calls business, I was able to adapt my SeleniumFactory accordingly.

… And that is my story. To summarize: each of these tools could be better documented, but at the end of the day I was able to get a Maven module up and running SeleniumRC tests via TestNG. Now, on to the test-writing!