Sometimes, the cases fail, not because of application bugs but because of unexpected events like a browser issue, network lag, etc. Now, we might need to rerun the failed cases to see whether or not those unanticipated events still exist. TestNG provides us with some ways to do just that.
- We can rerun failed cases using the testng-failed.xml file
- Or, we can rerun the failed cases using the RetryAnalyzer
Let’s look at each of the ways one by one.
Rerun failed test cases using the testng-failed.xml file
Let’s first try to run a test class. Our test class will have three test cases.
package Test;
import org.testng.Assert;
import org.testng.annotations.Test;
public class CodekruTest {
@Test
public void test1() {
System.out.println("Executing test1 in CodekruTest class");
Assert.assertTrue(true);
}
@Test
public void test2() {
System.out.println("Executing test2 in CodekruTest class");
Assert.assertTrue(false); // failing this test case
}
@Test
public void test3() {
System.out.println("Executing test3 in CodekruTest class");
Assert.assertTrue(false); // failing this test case as well
}
}
Below is the XML file to run the above test class
<suite name="codekru">
<test name="codekruTest">
<classes>
<class name="Test.CodekruTest" />
</classes>
</test>
</suite>
Output after running the above XML file –
Executing test1 in CodekruTest class
Executing test2 in CodekruTest class
Executing test3 in CodekruTest class
===============================================
codekru
Total tests run: 3, Passes: 1, Failures: 2, Skips: 0Now, two cases failed ( test2 and test3 ), and a testng-failed.xml file must have been created in the testng-output, which can be used to run the failed test cases.
The content of the testng-failed.xml file contains the necessary information to run the failed cases once again.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Failed suite [Failed suite [codekru]]" guice-stage="DEVELOPMENT">
<test thread-count="5" name="codekruTest(failed)(failed)">
<classes>
<class name="Test.CodekruTest">
<methods>
<include name="test3"/>
<include name="test2"/>
</methods>
</class> <!-- Test.CodekruTest -->
</classes>
</test> <!-- codekruTest(failed)(failed) -->
</suite> <!-- Failed suite [Failed suite [codekru]] -->
Output after running the testng-failed.xml file –
[RemoteTestNG] detected TestNG version 7.4.0
Executing test3 in CodekruTest class
Executing test2 in CodekruTest class
===============================================
Failed suite [Failed suite [codekru]]
Total tests run: 2, Passes: 0, Failures: 2, Skips: 0Here, we can see that only the failed cases got executed.
What if the failed cases are dependent on other passed test cases?
Let’s include one more test case (test4) in our CodekruTest class, and test2 and test3 will be dependent on test4 using the dependsOnMethods attribute.
package Test;
import org.testng.Assert;
import org.testng.annotations.Test;
public class CodekruTest {
@Test
public void test1() {
System.out.println("Executing test1 in CodekruTest class");
Assert.assertTrue(true);
}
@Test(dependsOnMethods = "test4")
public void test2() {
System.out.println("Executing test2 in CodekruTest class");
Assert.assertTrue(false); // failing this test case
}
@Test(dependsOnMethods = "test4")
public void test3() {
System.out.println("Executing test3 in CodekruTest class");
Assert.assertTrue(false); // failing this test case as well
}
@Test
public void test4() {
System.out.println("Executing test4 in CodekruTest class");
Assert.assertTrue(true);
}
}
Let’s run our CodekruTest class again
Executing test1 in CodekruTest class
Executing test4 in CodekruTest class
Executing test2 in CodekruTest class
Executing test3 in CodekruTest class
PASSED: test4
PASSED: test1
FAILED: test3
FAILED: test2
===============================================
Default test
Tests run: 4, Failures: 2, Skips: 0
===============================================Here test4 case is passed, whereas test2 and test3 still failed. Now. Let’s see the contents of the testng-failed.xml file.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Failed suite [Default suite]" guice-stage="DEVELOPMENT">
<test thread-count="5" name="Default test(failed)">
<classes>
<class name="Test.CodekruTest">
<methods>
<include name="test4"/>
<include name="test3"/>
<include name="test2"/>
</methods>
</class> <!-- Test.CodekruTest -->
</classes>
</test> <!-- Default test(failed) -->
</suite> <!-- Failed suite [Default suite] -->
We can see that test4 is also included in the testng-failed.xml file because test2 and test3 depend on test4. So, now three test cases will run after executing the testng-failed.xml file.
Executing test4 in CodekruTest class
Executing test3 in CodekruTest class
Executing test2 in CodekruTest class
===============================================
Failed suite [Default suite]
Total tests run: 3, Passes: 1, Failures: 2, Skips: 0
===============================================Rerun failed cases using RetryAnalyzer
Running the failed cases through testng-failed.xml is a good way to rerun the test cases, but your failed cases would be executed after TestNG executed the whole test suite. TestNG provides us with a feature that can rerun our case as soon as it fails, and we can achieve this by using the retryAnalyzer attribute.
Some advantages of using the retryAnalyzer attribute
- The failed case would be executed as soon as it fails without waiting for TestNG to execute the whole suite.
- We can retry the failed as many times as we want.
So, how can we use the retry analyzer in our code?
- Make a class that implements the IRetryAnalyzer interface.
- Bind the class that implemented the IRetryAnalyzer interface to the @Test annotation. E.g. @Test(retryAnalyzer = Retry.class)
Let’s make our Retry class that will implement the IRetryAnalyzer interface. IRetryAnalyzer has one retry() method that our Retry class will override.
package Test;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
public class Retry implements IRetryAnalyzer {
int counter = 0;
int retryLimit = 2; // maximum number of times, the case can be retried
public boolean retry(ITestResult result) {
if (counter < retryLimit) {
counter++;
return true; // retry the case if retryLimit is not exhausted
}
return false; // Do not retry anymore, as retryLimit is exhausted
}
}
- Here, the counter variable stores the number of times a case has been executed.
- retryLimit variable tells the maximum number of retries for a single test case.
It will retry the case until a case passes or the retries have been exhausted. Now, we will use it in our test class ( CodekruTest ).
package Test;
import org.testng.Assert;
import org.testng.annotations.Test;
public class CodekruTest {
@Test
public void test1() {
System.out.println("Executing test1 in CodekruTest class");
Assert.assertTrue(true);
}
@Test(retryAnalyzer = Retry.class)
public void test2() {
System.out.println("Executing test2 in CodekruTest class");
Assert.assertTrue(false); // failing this test case
}
@Test()
public void test3() {
System.out.println("Executing test3 in CodekruTest class");
Assert.assertTrue(false); // failing this test case as well
}
}
Now, the test2() case would be retried 2 more times after failing. So, let’s now try to run our CodekruTest class using the below XML file.
<suite name="codekru">
<test name="codekruTest">
<classes>
<class name="Test.CodekruTest" />
</classes>
</test>
</suite>
Output –
Executing test1 in CodekruTest class
Executing test2 in CodekruTest class
Executing test2 in CodekruTest class
Executing test2 in CodekruTest class
Executing test3 in CodekruTest class
===============================================
codekru
Total tests run: 5, Passes: 1, Failures: 2, Skips: 0, Retries: 2We can see that test2() has been retried 2 more times.
How to put retry on all cases of a class?
Sometimes, it takes time to put retryAnalyzer on every case, especially if the class has a lot of test cases. To save us from this, TestNG provides us with the functionality of class-level annotations. We can put the @Test((retryAnalyzer = Retry.class) at the class level only once, and it would be applied to all cases within the class.
package Test;
import org.testng.Assert;
import org.testng.annotations.Test;
@Test(retryAnalyzer = Retry.class)
public class CodekruTest {
public void test1() {
System.out.println("Executing test1 in CodekruTest class");
Assert.assertTrue(true);
}
public void test2() {
System.out.println("Executing test2 in CodekruTest class");
Assert.assertTrue(false); // failing this test case
}
public void test3() {
System.out.println("Executing test3 in CodekruTest class");
Assert.assertTrue(false); // failing this test case as well
}
}
Now, all of the test cases would be retried if they failed. Let’s try it by executing the above class.
Executing test1 in CodekruTest class
Executing test2 in CodekruTest class
Executing test2 in CodekruTest class
Executing test2 in CodekruTest class
Executing test3 in CodekruTest class
Executing test3 in CodekruTest class
Executing test3 in CodekruTest classtest2() and test3() executed 2 more times.
How to put retry on all cases of a suite?
Let’s now talk about how we can put retry on all cases running in the suite?
We can achieve that by using the IAnnotationTransformer interface, where we can implement the transform method. We will take a class ( say AnnotationTransformer ) to implement the IAnnotationTransformer interface.
This interface is used to add the annotations at runtime programmatically, and its transform method executes on every test case run. So, we will rewrite the transform method and add the retryAnalyzer functionality in our test case at runtime.
package Test;
import org.testng.IAnnotationTransformer;
import org.testng.annotations.ITestAnnotation;
import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
public class AnnotationTransformer implements IAnnotationTransformer {
public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) {
annotation.setRetryAnalyzer(Retry.class); // setting the retryAnalyzer
}
}
Now, how to use AnnotationTransformer class so it can add retryAnalyzer functionality to our test cases. We have to add this class as a listener in our testng.xml file, and then all cases run by the XML file will have the retryAnalyzer property added to them.
Now, our CodekruTest class will look like
package Test;
import org.testng.Assert;
import org.testng.annotations.Test;
public class CodekruTest {
@Test
public void test1() {
System.out.println("Executing test1 in CodekruTest class");
Assert.assertTrue(true);
}
@Test
public void test2() {
System.out.println("Executing test2 in CodekruTest class");
Assert.assertTrue(false); // failing this test case
}
@Test
public void test3() {
System.out.println("Executing test3 in CodekruTest class");
Assert.assertTrue(false); // failing this test case as well
}
}
And our XML file would be
<suite name="codekru">
<listeners>
<listener class-name="Test.AnnotationTransformer" />
</listeners>
<test name="codekruTest">
<classes>
<class name="Test.CodekruTest" />
</classes>
</test>
</suite>
Output after running the above XML file –
Executing test1 in CodekruTest class
Executing test2 in CodekruTest class
Executing test2 in CodekruTest class
Executing test2 in CodekruTest class
Executing test3 in CodekruTest class
Executing test3 in CodekruTest class
Executing test3 in CodekruTest class
===============================================
codekru
Total tests run: 7, Passes: 1, Failures: 2, Skips: 0, Retries: 4
===============================================We can see that the failed cases were automatically retried.
One last important thing
You might have noticed by now that whenever we are retrying the cases, our total cases count increases, and so does the number of skipped cases. Below is the summary of our last execution
Note: The newer versions of TestNG have the Retries field added to the summary.
Executing test1 in CodekruTest class
Executing test2 in CodekruTest class
Executing test2 in CodekruTest class
Executing test2 in CodekruTest class
Executing test3 in CodekruTest class
Executing test3 in CodekruTest class
Executing test3 in CodekruTest class
===============================================
codekru
Total tests run: 7, Passes: 1, Failures: 2, Skips: 0, Retries: 4
===============================================
We can see the skipped cases or retries count is 4, and the total cases count is 7. But sometimes, we do not want retries to be counted in overall cases and would only want to display the number of cases that have been written in our test class.
So, how not to count the retries in overall cases?
Here, we will have to use one more interface, ITestListener, and one class ( say TestListenerClass) to implement the interface.
package Test;
import org.testng.ITestListener;
import org.testng.ITestResult;
public class TestListenerClass implements ITestListener {
public void onTestSuccess(ITestResult result) {
if (result.getMethod().getRetryAnalyzer(result) != null) {
int numberOfSkippedCases = result.getTestContext().getSkippedTests().getResults(result.getMethod()).size();
for (int i = 0; i < numberOfSkippedCases; i++) {
result.getTestContext().getSkippedTests().removeResult(result.getMethod());
}
}
}
public void onTestFailure(ITestResult result) {
if (result.getMethod().getRetryAnalyzer(result) != null) {
int numberOfSkippedCases = result.getTestContext().getSkippedTests().getResults(result.getMethod()).size();
for (int i = 0; i < numberOfSkippedCases; i++) {
result.getTestContext().getSkippedTests().removeResult(result.getMethod());
}
}
}
}
So, what are we doing here?
- onTestFailure() method executes every time a case fails, and onTestSuccess() method executes if a test case passes.
- So, whenever a test case is passed or failed, we will check whether the test case has enabled retryAnalyzer or not by using result.getMethod().getRetryAnalyzer(result) != null condition.
- And if the test case has set the retryAnalyzer attribute, we will count the number of skipped cases. Because each time we retry a case, it is marked as skipped.
- And once we get the count of the skipped case. We will run a for loop to remove the skipped cases from the result.
Now, we must add TestListenerClass as a listener in our XML file. So, our updated XML file would be
<suite name="codekru">
<listeners>
<listener class-name="Test.AnnotationTransformer" />
<listener class-name="Test.TestListenerClass" />
</listeners>
<test name="codekruTest">
<classes>
<class name="Test.CodekruTest" />
</classes>
</test>
</suite>
Output after running the above XML file, which again executes the cases of our CodekruTest class –
Executing test1 in CodekruTest class
Executing test2 in CodekruTest class
Executing test2 in CodekruTest class
Executing test2 in CodekruTest class
Executing test3 in CodekruTest class
Executing test3 in CodekruTest class
Executing test3 in CodekruTest class
===============================================
codekru
Total tests run: 3, Passes: 1, Failures: 2, Skips: 0
===============================================Here we can see that the count shown is 3 only and not 7.
Though the TestNG summary would still look like
We hope that you liked the article. This is about retrying and rerunning the cases. If you have any doubts or concerns, please feel free to write us in the comments or mail us at admin@codekru.com.