Working with JUnit 4++

This section explores the advanced features of the JUnit 4 framework and includes the following topics: parameterized test, Hamcrest matchers and assertThat, assumption, theory, timeout, categories, rules, test suites, and tests order.

Ignoring a test

Suppose a failing test blocks you to check-in a mission critical code, and you come to know that the owner of the code is on a vacation. What do you do? You try to fix the test or just comment out or delete the test to proceed with your check-in (committing files to a source control such as SVN), or you wait until the test is fixed.

Sometimes we comment out tests because the feature is not developed. JUnit came up with a solution for this. Instead of commenting a test, we can just ignore it by annotating the test method with @Ignore. Commenting out a test or code is bad as it does nothing but increases the code size and reduces its readability. Also, when you comment out a test, then the test report doesn't tell you anything about the commented-out test; however, if you ignore a test, then the test report will tell you that something needs to be fixed as some tests are ignored. So, you can keep track of the ignored test.

Use @Ignore("Reason: why do you want to ignore?"). Giving a proper description explains the intention behind ignoring the test. The following is an example of, where a test method is ignored because the holiday calculation is not working:

@Test
@Ignore("John's holiday stuff failing")
public void when_today_is_holiday_then_stop_alarm() {
}

The following is a screenshot from Eclipse:

You can place the @Ignore annotation on a test class, effectively ignoring all the contained tests.

Executing tests in order

JUnit was designed to allow execution in a random order, but typically they are executed in a linear fashion and the order is not guaranteed. The JUnit runner depends on reflection to execute the tests. Usually, the test execution order doesn't vary from run to run; actually, the randomness is environment-specific and varies from JVM to JVM. So, it's better that you never assume they'll be executed in the same order and depend on other tests, but sometimes we need to depend on the order.

For example, when you want to write slow tests to insert a row into a database, then first update the row and finally delete the row. Here, unless the insert function is executed, delete or update functions cannot run.

JUnit 4.11 provides us with an @FixMethodOrder annotation to specify the execution order. It takes enum MethodSorters.

To change the execution order, annotate your test class using @FixMethodOrder and specify one of the following available enum MethodSorters constant:

• MethodSorters.JVM: This leaves the test methods in the order returned by the JVM. This order may vary from run to run.
• MethodSorters.NAME_ASCENDING: This sorts the test methods by the method name in the lexicographic order.
• MethodSorters.DEFAULT: This is the default value that doesn't guarantee the execution order.

We will write a few tests to verify this behavior.

Add a TestExecutionOrder test and create tests, as shown in the following code snippet:

public class TestExecutionOrder {
  @Test   public void edit() throws Exception {
    System.out.println("edit executed");
  }
  @Test   public void create() throws Exception {
    System.out.println("create executed");
  }
  @Test   public void remove() throws Exception {
    System.out.println("remove executed");
  }  
}

Run the tests. The execution order may vary, but if we annotate the class with @FixMethodOrder(MethodSorters.NAME_ASCENDING), the tests will be executed in the ascending order as follows:

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class TestExecutionOrder { … }

The following Eclipse screenshot displays the test execution in the ascending order:

Learning assumptions

In multisite projects, sporadically, a date or time zone tests fail in a local CI server but run fine in other servers in a different time zone. We can choose to not run those automatic tests in our local server.

Sometimes our tests fail due to a bug in a third-party code or external software, but we know that after some specific build or version, the bug will be fixed. Should we comment out the code and wait until the build is available?

In many projects, Jenkins (for test automation) and SONAR (for code-quality metrics) run in a server. It has been observed that due to low resources, the automatic tests run forever when SONAR is processing and the tests run simultaneously.

JUnit has the answer to all these issues. It recommends using an org.junit.Assume class.

Like Assert, Assume offers many static methods, such as assumeTrue(condition), assumeFalse(condition), assumeNotNull(condition), and assumeThat(condition). Before executing a test, we can check our assumption using the assumeXXX methods. If our assumption fails, then the assumeXXX methods throw AssumptionViolatedException, and the JUnit runner ignores the tests with failing assumptions.

So, basically, if our assumption is not true, the tests are just ignored. We can assume that the tests are run in the EST time zone; if the tests are run somewhere else, they will be ignored automatically. Similarly, we can assume that the third-party code version is higher than the build/version 123; if the build version is lower, the tests will be ignored.

Let's write the code to validate our assumption about Assume.

Here, we will try to solve the SONAR server issue. We will assume that SONAR is not running. If SONAR runs during the test execution, the assumption will fail and the tests will be ignored.

Create an Assumption test class. The following is the body of the class:

public class Assumption {
   
  boolean isSonarRunning = false;
  @Test 
  public void very_critical_test() throws Exception {
    isSonarRunning = true;
    Assume.assumeFalse(isSonarRunning);
    assertTrue(true);
  }
  
}

Here, for simplicity, we added a isSonarRunning variable to replicate a SONAR server facade. In the actual code, we can call an API to get the value. We will set the variable to false. Then, in the test, we will reset the value to true. This means SONAR is running. So, our assumption that SONAR is not running is false; hence, the test will be ignored.

The following screenshot shows that the test is ignored. We didn't annotate the test using @Ignore:

When we change the value of the isSonarRunning variable to false, as given in the following code snippet, the test will be executed:

public void very_critical_test() throws Exception {
    isSonarRunning = false;
    Assume.assumeFalse(isSonarRunning);
    assertTrue(true);
}

Continuous integration tools such as Jenkins can run multiple tools such as Sonar to acquire code-quality metrics. It's always a good practice to have a build pipeline where the code quality is only checked after the tests pass. This prevents the CPU-intensive tasks from occurring at the same time.

Assumption is also used in the @Before methods, but be careful not to overuse it. Assumption is good for use with TDD where one writes pretests ahead of time.

Exploring the test suite

To run multiple test cases, JUnit 4 provides Suite.class and the @Suite.SuiteClasses annotation. This annotation takes an array (comma separated) of test classes.

Create a TestSuite class and annotate the class with @RunWith(Suite.class). This annotation will force Eclipse to use the suite runner.

Next, annotate the class with @Suite.SuiteClasses({ AssertTest.class, TestExecutionOrder.class, Assumption.class }) and pass comma-separated test class names.

The following is the code snippet:

import org.junit.runner.RunWith;
import org.junit.runners.Suite;

@RunWith(Suite.class)
@Suite.SuiteClasses({ AssertTest.class, TestExecutionOrder.class,Assumption.class })
public class TestSuite {

}

During execution, the suite will execute all the tests. The following is a screenshot of the suite run. Check whether it runs seven tests out of the three test fixtures: AssertTest, TestExecutionOrder, and Assumption.

A test suite is created for group-related tests such as a group of data access, API usage tests, or a group of input validation logic tests.

Asserting with assertThat

Joe Walnes created the assertThat(Object actual, Matcher matcher) method. General consensus is that assertThat is readable and more useful than assertEquals. The syntax of the assertThat method is as follows:

  public static void assertThat(Object actual, Matcher matcher

Here, Object is the actual value received and Matcher is an implementation of the org.hamcrest.Matcher interface. This interface comes from a separate library called hamcrest.jar.

A matcher enables a partial or an exact match for an expectation, whereas assertEquals uses an exact match. Matcher provides utility methods such as is, either, or, not , and hasItem. The Matcher methods use the builder pattern so that we can combine one or more matchers to build a composite matcher chain. Just like StringBuilder, it builds a string in multiple steps.

The following are a few examples of matchers and assertThat:

• assertThat(calculatedTax, is(not(thirtyPercent)) );
• assertThat(phdStudentList, hasItem(DrJohn) );
• assertThat(manchesterUnitedClub, both( is(EPL_Champion)).and(is(UEFA_Champions_League_Champion)) );

The preceding examples are more English than a JUnit test code. So, anyone can understand the intent of the code and test, and a matcher improves readability.

Hamcrest provides a utility matcher class called org.hamcrest.CoreMatchers.

A few utility methods of CoreMatchers are allOf, anyOf, both, either, describedAs, everyItem, is, isA, anything, hasItem, hasItems, equalTo, any, instanceOf, not, nullValue, notNullValue, sameInstance, theInstance ,startsWith, endsWith, and containsString. All these methods return a matcher.

We worked with assertEquals; so, let's start with equalTo. The equalTo method is equivalent to assertEquals.

Comparing matchers – equalTo, is, and not

Create a AssertThatTest.java JUnit test and static import org.hamcrest.CoreMatchers.*; as follows:

import static org.hamcrest.CoreMatchers.not;
import static org.hamcrest.CoreMatchers.is;
import static org.junit.Assert.assertThat;

import org.junit.Test;

public class AssertThatTest {

  @Test
  public void verify_Matcher() throws Exception {
    int age = 30;
    assertThat(age, equalTo(30));
    assertThat(age, is(30));
    
    assertThat(age, not(equalTo(33)));
    assertThat(age, is(not(33)));
  }
}

Set the age variable to 30 and then likewise for assertEquals and call equalTo, which here is Matcher. The equalTo method takes a value. If the Matcher value doesn't match the actual value, then assertThat throws an AssertionError exception.

Set the age variable value to 29 and rerun the test. The following error will occur:

The is(a) attribute takes a value and returns a Boolean and behaves similar to equalTo(a). The is(a) attribute is the same as is(equalTo(a)).

The not attribute takes a value or a matcher. In the preceding code, we used assertThat(age, is(not(33)));. This expression is nothing but age is not 33 and is more readable than the assert methods.

Working with compound value matchers – either, both, anyOf, allOf, and not

In this section, we will use either, both, anyOf, allOf, and not. Add the following test to the AssertThatTest.java file:

@Test
  public void verify_multiple_values() throws Exception {
        
    double marks = 100.00;
    assertThat(marks, either(is(100.00)).or(is(90.9)));
    
    assertThat(marks, both(not(99.99)).and(not(60.00)));
    
    assertThat(marks, anyOf(is(100.00),is(1.00),is(55.00),is(88.00),is(67.8)));
    
    assertThat(marks, not(anyOf(is(0.00),is(200.00))));
    
    assertThat(marks, not(allOf(is(1.00),is(100.00), is(30.00))));
  }

In the preceding example, a marks double variable is initialized with a value of 100.00. This variable value is asserted with an either matcher.

Basically, using either, we can compare two values against an actual or calculated value. If any of them match, then the assertion is passed. If none of them match, then AssertionError is thrown.

The either(Matcher) method takes a matcher and returns a CombinableEitherMatcher class. The CombinableEitherMatcher class has a or(Matcher other) method so that either and or can be combined.

The or(Matcher other) method is translated to return (new CombinableMatcher(first)).or(other); and finally to new CombinableMatcher(new AnyOf(templatedListWith(other)));.

Using both, we can compare two values against an actual or calculated value. If any of them don't match, then the AssertionError exception is thrown. If both of them match, then the assertion is passed.

A numeric value such as a math score cannot be equal to both 60 and 80. However, we can negate the expression. If the math score is 80, then using the both matcher we can write the expression as assertThat (mathScore , both (not(60)). and(not (90))).

The anyOf matcher is more like either with multiple values. Using anyOf, we can compare multiple values against an actual or calculated value. If any of them match, then the assertion is passed. If none of them match, then the AssertionError exception is thrown.

The allOf matcher is more like both with multiple values. Using allOf, we can compare multiple values against an actual or calculated value. If any of them don't match, then the AssertionError exception is thrown. Similar to both, we can use allOf along with not to check whether a value does or doesn't belong to a set.

In the preceding example, using allOf and not, we checked whether the marks attribute is not 1, 100, or 30.

Working with collection matchers – hasItem and hasItems

In the previous section, we asserted a value against multiple values. In this section, we will assert a collection of values against a value or numerous values.

Consider the following example. A salary list is populated with three values: 50.00, 200.00, and 500.00. Use hasItem to check whether a value exists in a collection, and use hasItems to check whether multiple values exist in a collection, as shown in the following code:

   @Test
  public void verify_collection_values() throws Exception {
        
    List<Double> salary =Arrays.asList(50.0, 200.0, 500.0);
    
    assertThat(salary, hasItem(50.00));
    assertThat(salary, hasItems(50.00, 200.00));
        assertThat(salary, not(hasItem(1.00)));
  }

The hasItem matcher has two versions: one takes a value and the other takes a matcher. So, we can check a value in a collection using hasItem, or check whether a value doesn't exist in a collection using not and hasItem. The hasItems matcher operates on a set of values.

Exploring string matchers – startsWith, endsWith, and containsString

In this section, we will explore the string matchers. CoreMatchers has three built-in string matcher methods. In the following example, a String variable name is assigned a value and then we assert that the name starts with a specific value, contains a value, and ends with a value:

@Test
  public void verify_Strings() throws Exception {
    String name = "John Jr Dale";
    assertThat(name, startsWith("John"));
    assertThat(name, endsWith("Dale"));
    assertThat(name, containsString("Jr"));
  }

The startsWith matcher operates on string only. It checks whether the string starts with the given string. The endsWith matcher checks whether the string ends with the given string. The containsString matcher checks whether the string contains another string.

Sometimes, a method calls to return a JSON response. Using containsString, a specific value can be asserted.

Note

Note that startsWith, endsWith, and containsStrings are not the only string matchers. Other built-in matchers such as both, either, anyOf, and so on, can be applied to a String object.

Exploring built-in matchers

JUnitMatchers has built-in matcher methods, but all of these methods are deprecated. Use Hamcrest matchers instead of using JUnitMatchers.

Building a custom matcher

We can build our own matchers to use in assertThat. How about building a matcher that will compare two values and return true only if the actual object is less than or equal to the expected value?

Call it a lessThanOrEqual matcher. It should be allowed to use with any object that can be compared so that we can use an Integer or Double or String type or any custom class that implements the Comparable interface.

For example, assertThat(100, lessThanOrEqual(200)) should pass, but assertThat(100, lessThanOrEqual(50)) should fail and assertThat("john123", lessThanOrEqual("john123")) should pass, but assertThat("john123", lessThanOrEqual("john12")) should fail.

Follow the ensuing steps to build the lessThanOrEqual matcher:

1. Create a LessThanOrEqual class under the com.packtpub.junit.recap package.
2. To build a custom matcher, a class should implement the Matcher interface. However, Hamcrest recommends extending org.hamcrest.BaseMatcher rather than implementing the Matcher interface. So, we will extend BaseMatcher. The BaseMatcher class is an abstract class, and it doesn't implement describeTo(Description description) and matches(Object t).

   The public boolean matches(Object obj) method is invoked by assertThat. If this method returns false, then an AssertionError exception is thrown.

   The public void describeTo(Description description) method is invoked when matches(Object obj) returns false. This method builds the description of an expectation.

   The following code snippet explains how assertThat works:

     if(!matcher.matches(actual)){
            Description description = new StringDescription();
            description.appendText(reason).appendText("\nExpected: ).appendDescriptionOf(matcher).appendText("\n   but: ");
              
            matcher.describeMismatch(actual, description);
            throw new AssertionError(description.toString());
     }

   Note that when matcher.matches() returns false, the description is built from the actual value and the matcher. The appendDescriptionOf() method calls the describeTo() method of the matcher to build the error message.

   Finally, matcher.describeMismatch(actual, description) appends the string but: was <<actual>>.

3. The lessThanOrEqual class needs to compare two objects, so the Matcher class should be operated on the Comparable objects. Create a generic class that operates with any type that implements the Comparable interface, as follows:

   public class LessThanOrEqual<T extends Comparable<T>> extends BaseMatcher<Comparable<T>> {
   
   }

4. Now we need to implement the describeTo and matches methods. The assertThat method will pass the actual value to the matcher's matches(Object o) method, and lessThanOrEqual will accept a value to compare with the actual. So, in the matches method, we need two comparable objects: one passed as a parameter and the other passed to a matcher object. The expected value is passed during the matcher object instantiation as follows:

   assertThat (actual, matcher(expectedValue)).

   We will store the expectedValue during the Matcher object creation and use it in the matches() method to compare the expectedValue with the actual as follows:

   public class LessThanOrEqual<T extends Comparable<T>> extends BaseMatcher<Comparable<T>> {
     private final Comparable<T> expectedValue;
     
     public LessThanOrEqual(T expectedValue) {
      this.expectedValue = expectedValue;
     }
   
   
     @Override
     public void describeTo(Description description) {
       description.appendText(" less than or equal(<=) "+expectedValue);
     }
   
   
     @Override
     public boolean matches(Object t) {
       int compareTo = expectedValue.compareTo((T)t);
       return compareTo > -1;
     }
   }

   The preceding LessThanOrEqual class should return true only if expectedValue.compareTo(actual) >= 0 and then the describeTo() method appends the string "less than or equals (<=) "+ expectedValue text to the description, so that if the assertion fails, then the "less than or equals (<=) "+ expectedValue message will be shown.

5. The assertThat method takes a matcher but new LessThanOrEqual(expectedValue) doesn't look good. We will create a static method in the LessThanOrEqual class to create a new object of LessThanOrEqual. Call this method from the assertThat method as follows:

     @Factory
     public static<T extends Comparable<T>>  Matcher<T>        
       lessThanOrEqual(T t) {
       return new LessThanOrEqual(t);
       }

   The @Factory annotation isn't necessary but needed for a Hamcrest tool. When we create many custom matchers, then it becomes annoying to import them all individually. Hamcrest ships with a org.hamcrest.generator.config.XmlConfigurator command-line tool that picks up predicates annotated with the @Factory annotation and collects them in a Matcher class for easy importing.

6. Static import the LessThanOrEqual class and add a test to AssertThatTest.java to validate the custom matcher, as shown in the following code:

      @Test
     public void lessthanOrEquals_custom_matcher() throws Exception
     {
       int actualGoalScored = 2;
       assertThat(actualGoalScored, lessThanOrEqual(4));
       assertThat(actualGoalScored, lessThanOrEqual(2));
       
       double originalPI = 3.14;
       assertThat(originalPI, lessThanOrEqual(9.00));
   
       String authorName = "Sujoy";
       assertThat(authorName, lessThanOrEqual("Zachary"));
     }

   This test should pass.

7. How about testing the code with a greater value? In Java, Integer.MAX_VALUE holds the maximum integer value and Integer.MIN_VALUE holds the minimum integer value. If we expect that the maximum value will be greater than or equal to the minimum value, then the assertion should fail. Consider the following code snippet:

       int maxInt = Integer.MAX_VALUE;
       assertThat(maxInt, lessThanOrEqual(Integer.MIN_VALUE));

   This will throw the following error:

   

Creating parameterized tests

Parameterized tests are used for multiple iterations over a single input to stress the object in test. The primary reason is to reduce the amount of test code.

In TDD, the code is written to satisfy a failing test. The production code logic is built from a set of test cases and different input values. For example, if we need to build a class that will return the factorial of a number, then we will pass different sets of data and verify that our implementation passes the validation.

We know that the factorial of 0 is 1, the factorial of 1 is 1, the factorial of 2 is 2, the factorial of 3 is 6, the factorial of 4 is 24, and so on.

So, if we write tests such as factorial_of_1_is_1 and factorial_of_4_is_24, then the test class will be polluted very easily. How many methods will we write?

We can create two arrays: one with the expected values and the other with the original numbers. Then, we can loop through the arrays and assert the result. We don't have to do this because the JUnit 4 framework provides us with a similar solution. It gives us a Parameterized runner.

We read about the @RunWith annotation in the preceding section. Parameterized is a special type of runner and can be used with the @RunWith annotation.

Parameterized comes with two flavors: constructor and method.

Working with parameterized constructors

Perform the following steps to build a parameterized test with a constructor:

1. Create a source folder src and add a Factorial.java class under src/ com.packtpub.junit.recap.
2. Implement the factorial algorithm. Add the following code to the Factorial.java class:

   package com.packtpub.junit.recap;
   
   public class Factorial {
   
     public long factorial(long number) {
       if(number == 0) {
         return 1;
       }
       
       return number*factorial(number-1);
     }
   }

3. Add a ParameterizedFactorialTest.java test under test/ com.packtpub.junit.recap and annotate the class with @RunWith(Parameterized.class) as follows:

   import org.junit.runner.RunWith;
   import org.junit.runners.Parameterized;
   
   @RunWith(Parameterized.class)
   public class ParameterizedFactorialTest {
   
   }

4. Add a method to create a dataset for factorial algorithm. The method should return Collection of the Object[] method. We need a collection of two dimensional arrays to hold the numbers and factorial values. To define the data parameters, annotate the method with @Parameters.

   The following code snippet defines a @parameters method factorialData():

     @Parameters
     public static Collection<Object[]> factorialData() {
       return Arrays.asList(new Object[][] {
         
         { 0, 1 }, { 1, 1 }, { 2, 2 }, { 3, 6 }, { 4, 24 }, { 5, 120 },{ 6, 720 }  
       });
     }

   Check whether the arrays hold the number and the expected factorial result (0's factorial is 1, 5's factorial is 120, and so on).

5. The Parameterized runner needs a constructor to pass the collection of data. For each row in the collection, the 0th array element will be passed as the 1st constructor argument, the next index will be passed as 2nd argument, and so on, as follows:

       private int number;
       private int expectedResult;
   
       public ParameterizedFactorialTest(int input, int expected) {
           number= input;
           expectedResult= expected;
       }

   In the test class, we added two members to hold the number and the expected factorial value. In the constructor, set these values. The Parameterized runner will loop through the data collection (annotated with a @Parameters annotation) and pass the values to the constructor.

   For example, it will pass 0 as input and 1 as expected, then 1 as input and 1 as expected, and so on.

6. Now, we need to add a test method to assert the number and the factorial as follows:

      @Test
     public void factorial() throws Exception {
       Factorial fact = new Factorial();
       assertEquals(fact.factorial(number),expectedResult);
     }

   We created a Factorial object and passed the number to get the actual result and then asserted the actual value with expectedResult. Here, the runner will create seven instances of the test class and execute the test method.

   The following screenshot shows the result of the test run taken from Eclipse:

   

   Note that the seven tests run and the tests names are [0] factorial[0], [1] factorial[1], and so on till [6].

   Note

   If the dataset returns an empty collection, the test doesn't fail; actually, nothing happens.

   If the number of parameters in the object array and the constructor argument don't match, then a java.lang.IllegalArgumentException: wrong number of arguments exception is thrown. For example, { 0, 1, 3 } will throw an exception as 3 arguments are passed, but constructor can accept only 2.

   If the constructor is not defined but the data set contains a value, then the java.lang.IllegalArgumentException: wrong number of arguments exception is thrown.

Working with parameterized methods

We learned about the parameterized constructor; now we will run the parameterized test excluding the constructor. Follow the ensuing steps to run the test using the @Parameter annotation:

1. Add a ParameterizeParamFactorialTest.java test class.
2. Copy the content from the constructor test and delete the constructor. Change the class members to public, as follows:

   @RunWith(Parameterized.class)
   public class ParameterizeParamFactorialTest {
   
     @Parameters
     public static Collection<Object[]> factorialData() {
       return Arrays.asList(new Object[][] {
         
         { 0, 1 }, { 1, 1 }, { 2, 2 }, { 3, 6 }, { 4, 24 }, { 5, 120 },{ 6, 720 }  
       });
     }
     
     public int number;
     public int expectedResult;
   
     
     @Test
     public void factorial() throws Exception {
       Factorial fact = new Factorial();
     assertEquals(fact.factorial(number),expectedResult);
     }
   }

3. If we run the test, it will fail as the reflection process won't find the matching constructor. JUnit provides an annotation to loop through the dataset and set the values to the class members. @Parameter(value=index) takes a value. The value is the array index of the data collection object array. Make sure that the number and expectedResult variables are public; otherwise, the security exception will be thrown. Annotate them with the following parameters:

       @Parameter(value=0)
       public int number;
       @Parameter(value=1)
       public int expectedResult;

   Here, for each row in the data collection, the number variable will hold the 0th index of the array and the expectedResult variable will hold the 1st index.

4. Run the test; seven tests will be executed.

Giving a name

In the constructor example, we found that the test names are assigned with indexes such as [0], [1], and so on. So, if a test fails, then it is not easy to identify the data. To identify an individual test case in a parameterized test, a name is required. The @Parameters annotation allows placeholders that are replaced at runtime, and we can use them. The following are the placeholders:

• {index}: This represents the current parameter index
• {0}, {1},…: This represents the first, second, and so on, parameter values

The following code snippet annotates the dataset with the name placeholders:

  @Parameters(name = "{index}: factorial({0})={1}")
    public static Collection<Object[]> factorialData() {
      return Arrays.asList(new Object[][] {
      
        { 0, 1 }, { 1, 1 }, { 2, 2 }, { 3, 6 }, { 4, 24 }, { 5, 120 },{ 6, 720 }  
      });
    }

Eclipse has a bug that chops off the name.

Working with timeouts

JUnit tests are automated to get quick feedback after a change in the code. If a test runs for a long time, it violates the quick feedback principle. JUnit provides a timeout value (in milliseconds) in the @Test annotation to make sure that if a test runs longer than the specified value, the test fails.

The following is an example of a timeout:

  @Test(timeout=10)
  public void forEver() throws Exception {
    Thread.sleep(100000);
  }

Here, the test will fail automatically after 10 milliseconds. The following is an Eclipse screenshot that shows the error:

Exploring JUnit theories

A theory is a kind of a JUnit test but different from the typical example-based JUnit tests, where we assert a specific data set and expect a specific outcome. JUnit theories are an alternative to JUnit's parameterized tests. A JUnit theory encapsulates the tester's understanding of an object's universal behavior. This means whatever a theory asserts is expected to be true for all data sets. Theories are useful for finding bugs in boundary-value cases.

Parameterized tests allow us to write flexible data-driven tests and separate data from the test methods. Theories are similar to parameterized tests—both allow us to specify the test data outside of the test case.

Parameterized tests are good but they have the following drawbacks:

• Parameters are declared as member variables. They pollute the test class and unnecessarily make the system complex.
• Parameters need to be passed to the single constructor or variables need to be annotated, simply making the class incomprehensible.
• Test data cannot be externalized.

Theory comes up with many annotations and a runner class. Let's examine the important annotations and classes in theory, as follows:

• @Theory: Like @Test, this annotation identifies a theory test to run. The @Test annotation doesn't work with a theory runner.
• @DataPoint: This annotation identifies a single set of test data (similar to @Parameters), that is, either a static variable or a method.
• @DataPoints: This annotation identifies multiple sets of test data, generally an array.
• @ParametersSuppliedBy: This annotation provides the parameters to the test cases.
• Theories: This annotation is a JUnit runner for the theory-based test cases and extends org.junit.runners.BlockJUnit4ClassRunner.
• ParameterSupplier: This is an abstract class that gives us the handle on the parameters that we can supply to the test case.

We will start with a simple theory and then explore more. Perform the following steps:

1. Create a MyTheoryTest.java class and annotate the class with @RunWith(Theories.class). To run a theory, this special runner is required. Consider the following code:

   @RunWith(Theories.class)
   public class MyTheoryTest {
   
   }

2. Now run the test. It will fail with the java.lang.Exception: No runnable methods error because no theory is defined yet. Like the @Test annotation, we will define a method and annotate it with @Theory as follows:

   @RunWith(Theories.class)
   public class MyTheoryTest {
     
     @Theory
     public void sanity() {
       System.out.println("Sanity check");
     }
   }

   Run the theory, and it will be executed with no error. So, our theory setup is ready.

3. Define a public static String with a name variable and annotate this variable with @DataPoint. Now execute the test, nothing special happens. If a theory method (annotated with @Theory) takes an argument and a variable annotated with @DataPoint matches the type, then the variable is passed to the theory during execution. So, change the sanity method and add a String argument to pass @DataPoint to the sanity() method, as follows:

   @RunWith(Theories.class)
   public class MyTheoryTest {
     @DataPoint public static String name ="Jack";
     
     @Theory
     public void sanity(String aName) {
       System.out.println("Sanity check "+aName);
     }
   }

   Now run the theory. It will pass the @DataPoint name to the sanity(String aName) method during execution and the name will be printed to the console.

4. Now, add another static @DataPoint, call it mike, and rename the name variable to jack, as follows:

   @RunWith(Theories.class)
   public class MyTheoryTest {
     @DataPoint public static String jack ="Jack";
     @DataPoint public static String mike ="Mike";
     
     @Theory
     public void sanity(String aName) {
       System.out.println("Sanity check "+aName);
     }
   }

   During theory execution, both the @DataPoint variables will be passed to the sanity(String aName) method. The output will be as follows:

   
5. Now, slightly modify the sanity() method—rename the aName argument to firstName and add a second String argument, lastName. So now the sanity method takes the String arguments, fistName and lastName. Print these variables using the following code:

   @RunWith(Theories.class)
   public class MyTheoryTest {
     @DataPoint public static String jack ="Jack";
     @DataPoint public static String mike ="Mike";
     
     @Theory
     public void sanity(String firstName, String lastName) {
       System.out.println("Sanity check "+firstName+", "+lastName);
     }
   }

   When executed, the output will be as follows:

   

   So, 2 x 2 = 4 combinations are used. When the multiple @DataPoint annotations are defined in a test, the theories apply to all possible well-typed combinations of data points for the test arguments.

6. So far we have only examined single-dimension variables. The @DataPoints annotation is used to provide a set of data. Add a static char array to hold the character variables and add a Theory method to accept two characters. It will execute the theory with 9 (3 ^ 2) possible combinations as follows:

     @DataPoints  public static char[] chars = 
             new char[] {'A', 'B', 'C'};
     @Theory
     public void build(char c, char d) {
       System.out.println(c+" "+d);
     }

   The following is the output:

   

Externalizing data using @ParametersSuppliedBy and ParameterSupplier

So far, we have covered how to set up test data using @DataPoint and @DataPoints. Now, we will use external classes to supply data in our tests using @ParametersSuppliedBy and ParameterSupplier. To do this, perform the following steps:

1. Create an Adder.java class. This class will have two overloaded add() methods to add numbers and strings. We will unit test the methods using theory.

   The following is the Adder class:

     public class Adder {
     
       public Object add(Number a, Number b) {
         return a.doubleValue()+b.doubleValue();
       }
     
       public Object add(String a, String b) {
         return a+b;
       }
     }

2. Create an ExternalTheoryTest.java theory as follows:

   @RunWith(Theories.class)
   public class ExternalTheoryTest {
   
   }

3. We will not use @DataPoints to create data. Instead, we will create a separate class to supply numbers to validate the add operation. JUnit provides a ParameterSupplier class for this purpose. ParameterSupplier is an abstract class, and it forces you to define a method as follows:

   public abstract List<PotentialAssignment> getValueSources(ParameterSignature parametersignature);

   PotentialAssignment is an abstract class that JUnit theories use to provide test data to test methods in a consistent manner. It has a static forValue method that you can use to get an instance of PotentialAssignment.

   Create a NumberSupplier class to supply different types of numbers: float, int, double, long, and so on. Extend the ParameterSupplier class as follows:

     import org.junit.experimental.theories.ParameterSignature;
     import org.junit.experimental.theories.ParameterSupplier;
     import org.junit.experimental.theories.PotentialAssignment;
   
     public  class NumberSupplier extends ParameterSupplier {
       @Override
         public List<PotentialAssignment>       
         getValueSources(ParameterSignature sig) {
           List<PotentialAssignment> list = new ArrayList<PotentialAssignment>();
           list.add(PotentialAssignment.forValue("long", 2L));
           list.add(PotentialAssignment.forValue("float", 5.00f));
           list.add(PotentialAssignment.forValue("double", 89d));
           return list;
     }
   
   };

   Check whether the overridden method creates a list of PotentialAssignment values of different numbers.

4. Now, modify the theory to add two numbers. Add a theory method as follows:

   import org.junit.experimental.theories.ParametersSuppliedBy;
   import org.junit.experimental.theories.Theories;
   import org.junit.experimental.theories.Theory;
   import org.junit.runner.RunWith;
   
   @RunWith(Theories.class)
   public class ExternalTheoryTest {
   
     @Theory
     public void adds_numbers(
     @ParametersSuppliedBy(NumberSupplier.class) Number num1,
     @ParametersSuppliedBy(NumberSupplier.class) Number num2) 
     {
       System.out.println(num1 + " and " + num2);
     }
   
   }

   Check the adds_numbers method; two Number arguments num1 and num2 are annotated with @ParametersSuppliedBy(NumberSupplier.class).

   When this theory is executed, the NumberSupplier class will pass a list.

5. Execute the theory; it will print the following result:
   
6. Now, we can check our Adder functionality. Modify the theory to assert the result.

   Create an instance of the Adder class and call the add method by passing num1 and num2. Add the two numbers and assert the value with the results of Adder.

   The assertEquals(double, double) method is deprecated as the double value calculation results in an unpredictable result. So, the assert class adds another version of assertEquals for doubles; it takes three arguments: actual, expected, and a delta. If the difference between the actual and the expected value is greater than or equal to delta, then the assertion passes as follows:

   @RunWith(Theories.class)
   public class ExternalTheoryTest {
   
     @Theory
     public void adds_numbers(
     @ParametersSuppliedBy(NumberSupplier.class) Number num1,
     @ParametersSuppliedBy(NumberSupplier.class) Number num2) {
       Adder anAdder = new Adder();
       double expectedSum = num1.doubleValue()+num2.doubleValue();
       double actualResult = (Double)anAdder.add(num1, num2);
       assertEquals(actualResult, expectedSum, 0.01);
     }
   
   }

   The Adder class has an add method for String. Create a StringSupplier class to supply String values to our theory and modify the theory class to verify the add (String, String) method behavior. You can assert the Strings as follows:

   • String expected = str1+str2;
   • assertEquals(expected, actual);

   Here, str1 and str2 are the two method arguments of the theory.

Dealing with JUnit rules

Rules allow very flexible addition or redefinition of the behavior of each test method in a test class. Rules are like Aspect Oriented Programming (AOP); we can do useful things before and/or after the actual test execution. You can find more information about AOP at http://en.wikipedia.org/wiki/Aspect-oriented_programming.

We can use the inbuilt rules or define our custom rule.

In this section, we will look at the inbuilt rules and create our custom Verifier and WatchMan rule.

Playing with the timeout rule

The timeout rule applies the same timeout to all the test methods in a class. Earlier, we used the timeout in the @Test annotation as follows:

@Test(timeout=10)

The following is the syntax of the timeout rule:

import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.Timeout;

public class TimeoutTest {
    
    @Rule
    public Timeout globalTimeout =  new Timeout(20);
    
    @Test
    public void testInfiniteLoop1() throws InterruptedException{
      Thread.sleep(30);
    }
    
    @Test
    public void testInfiniteLoop2() throws InterruptedException{
      Thread.sleep(30);
    }
    
}

When we run this test, it times out after 20 milliseconds. Note that the timeout is applied globally to all methods.

Working with the ExpectedException rule

The ExpectedException rule is an important rule for handling exceptions. It allows you to assert the expected exception type and the exception message, for example, your code may throw a generic exception (such as IllegalStateException) for all failure conditions, but you can assert the generic exception message to verify the exact cause.

Earlier, we used @Test(expected=Exception class) to test the error conditions.

The ExpectedException rule allows in-test specification of expected exception types and messages.

The following code snippet explains how an exception rule can be used to verify the exception class and the exception message:

public class ExpectedExceptionRuleTest {

   @Rule
    public ExpectedException thrown= ExpectedException.none();

    @Test
    public void throwsNothing() {

    }

    @Test
    public void throwsNullPointerException() {
      thrown.expect(NullPointerException.class);
      throw new NullPointerException();
    }

    @Test
    public void throwsIllegalStateExceptionWithMessage() {
      thrown.expect(IllegalStateException.class);
      thrown.expectMessage("Is this a legal state?");
     
      throw new IllegalStateException("Is this a legal state?");
    }
}

The expect object sets the expected exception class and expectMessage sets the expected message in the exception. If the message or exception class doesn't match the rule's expectation, the test fails. The ExpectedException object thrown is reset on each test.

Unfolding the TemporaryFolder rule

The TemporaryFolder rule allows the creation of files and folders that are guaranteed to be deleted when the test method finishes (whether it passes or fails). Consider the following code:

@Rule
  public TemporaryFolder folder = new TemporaryFolder();

  @Test
  public void testUsingTempFolder() throws IOException {
    File createdFile = folder.newFile("myfile.txt");
    File createdFolder = folder.newFolder("mysubfolder");
    
  }

Exploring the ErrorCollector rule

The ErrorCollector rule allows the execution of a test to continue after the first problem is found (for example, to collect all the incorrect rows in a table and report them all at once) as follows:

import org.junit.rules.ErrorCollector;
import static org.hamcrest.CoreMatchers.equalTo;

public class ErrorCollectorTest {

   @Rule
   public ErrorCollector collector = new ErrorCollector();
    
   @Test
   public void fails_after_execution() {
   collector.checkThat("a", equalTo("b"));
   collector.checkThat(1, equalTo(2));
   collector.checkThat("ae", equalTo("g"));
   }
}

In this example, none of the verification passes but the test still finishes its execution, and at the end, notifies all errors.

The following is the log—the arrows indicate the errors—and also note that only one test method is being executed but Eclipse indicates three failures:

Working with the Verifier rule

Verifier is a base class of ErrorCollector, which can otherwise turn passing tests into failing tests if a verification check fails. The following example demonstrates the Verifier rule:

public class VerifierRuleTest {
  private String errorMsg = null;

  @Rule
  public TestRule rule = new Verifier() {
    protected void verify() {
      assertNull("ErrorMsg should be null after each test execution",errorMsg);
    }
  };
  
  
  @Test
  public void testName() throws Exception {
    errorMsg = "Giving a value";
  }
}

Verifier's verify method is executed after each test execution. If the verify method defines any assertions, and that assertion fails, then the test is marked as failed.

In the preceding example, the test should not fail as the test method doesn't perform any comparison; however, it still fails. It fails because the Verifier rule checks that after every test execution, the errorMsg string should be set as null, but the test method sets the value to Giving a value; hence, the verification fails.

Learning the TestWatcher rule

TestWatcher (and the deprecated TestWatchman) are base classes for rules that take note of the testing action, without modifying it. Consider the following code:

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class TestWatcherTest {

  private static String dog = "";

    @Rule
    public TestWatcher watchman = new TestWatcher() {
      @Override
      public Statement apply(Statement base, Description description) {
        return super.apply(base, description);
      }

      @Override
      protected void succeeded(Description description) {
        dog += description.getDisplayName() + " " + "success!\n";
      }

      @Override
      protected void failed(Throwable e, Description description) {
        dog += description.getDisplayName() + " " + e.getClass().getSimpleName() + "\n";
      }

      @Override
      protected void starting(Description description) {
        super.starting(description);
      }

      @Override
      protected void finished(Description description) {
        super.finished(description);
      }
    };

    @Test
    public void red_test() {
      fail();
    }

    @Test
    public void green() {
    }
  
    @AfterClass
    public static void afterClass() {
      System.out.println(dog);
    }
}

We created a TestWatcher class to listen to every test execution, collected the failure, and success instances, and at the end, printed the result in the afterClass() method.

The following is the error shown on the console:

green(com.packtpub.junit.recap.rule.TestWatcherTest) success!
red_test(com.packtpub.junit.recap.rule.TestWatcherTest) AssertionError

Working with the TestName rule

The TestName rule makes the current test name available inside test methods. The TestName rule can be used in conjunction with the TestWatcher rule to make a unit testing framework compile a unit testing report.

The following test snippet shows that the test name is asserted inside the test:

public class TestNameRuleTest {

  @Rule
    public TestName name = new TestName();
    
    @Test
    public void testA() {
      assertEquals("testA", name.getMethodName());
    }
    
    @Test
    public void testB() {
      assertEquals("testB", name.getMethodName());
    }
}

The following section uses the TestName rule to get the method name before test execution.

Handling external resources

Sometimes JUnit tests need to communicate with external resources such as files or databases or server sockets. Dealing with external resources is always messy because you need to set up state and tear it down later. The ExternalResource rule provides a mechanism that makes resource handling a bit more convenient.

Previously, when you had to create files in a test case or work with server sockets, you had to set up a temporary directory, or open a socket in a @Before method and later delete the file or close the server in an @After method. But now, JUnit provides a simple AOP-like mechanism called the ExternalResource rule that makes this setup and cleanup work the responsibility of the resource.

The following example demonstrates the ExternalResource capabilities. The Resource class represents an external resource and prints the output in the console:

class Resource{
  public void open() {
    System.out.println("Opened");
  }
  
  public void close() {
    System.out.println("Closed");
  }
  
  public double get() {
    return Math.random();
  }
}

The following test class creates ExternalResource and handles the resource lifecycle:

public class ExternalResourceTest {
  Resource resource;
  public @Rule TestName name = new TestName();

  public @Rule ExternalResource rule = new ExternalResource() {
    @Override protected void before() throws Throwable {
      resource = new Resource();
      resource.open();
      System.out.println(name.getMethodName());
    }
    
    @Override protected void after()  {
      resource.close();
      System.out.println("\n");
    }
  };
  
  @Test
  public void someTest() throws Exception {
    System.out.println(resource.get());
  }
  
  @Test
  public void someTest2() throws Exception {
    System.out.println(resource.get());
  }
}

The anonymous ExternalResource class overrides the before and after methods of the ExternalResource class. In the before method, it starts the resource and prints the test method name using the TestName rule. In the after method, it just closes the resource.

The following is the test run output:

Opened
someTest2
0.5872875884671511
Closed

Opened
someTest
0.395586457988541
Closed

Note that the resource is opened before test execution and closed after the test. The test name is printed using the TestName rule.

Exploring JUnit categories

The Categories runner runs only the classes and methods that are annotated with either the category given with the @IncludeCategory annotation or a subtype of that category. Either classes or interfaces can be used as categories. Subtyping works, so if you use @IncludeCategory(SuperClass.class), a test marked @Category({SubClass.class}) will be run.

We can exclude categories by using the @ExcludeCategory annotation.

We can define two interfaces using the following code:

public interface SmartTests { /* category marker */ }
public interface CrazyTests { /* category marker */ }

public class SomeTest {
  @Test
  public void a() {
    fail();
  }

  @Category(CrazyTests.class)
  @Test
  public void b() {
  }
}

@Category({CrazyTests.class, SmartTests.class})
public class OtherTest {
  @Test
  public void c() {

  }
}

@RunWith(Categories.class)
@IncludeCategory(CrazyTests.class)
@SuiteClasses( { SomeTest.class, OtherTest.class }) // Note that Categories is a kind of Suite
public class CrazyTestSuite {
  // Will run SomeTest.b and OtherTest.c, but not SomeTest.a
}

@RunWith(Categories.class)
@IncludeCategory(CrazyTests.class)
@ExcludeCategory(SmartTests.class)
@SuiteClasses( { SomeTest.class, OtherTest.class }) 
public class CrazyTestSuite {
  // Will run SomeTest.b, but not SomeTest.a or OtherTest.c
}