How to retry the failed test cases in TestNG?

TestNG

In the world of software testing, ensuring the stability and reliability of applications is paramount. However, test cases don’t always pass on the first attempt. Various factors, such as network glitches, temporary server downtimes, or even issues with test environment configurations, can lead to test failures. While these failures might not always indicate a defect in the application, they can disrupt the testing process and lead to unnecessary delays. This is where the concept of retrying failed test cases becomes a game-changer in automation testing.

In this article, we’ll explore how to effectively implement a retry mechanism for failed test cases. But before that, let’s see why we would need to retry the test cases.

Why Do We Need to Retry Failed Test Cases?

Automated test cases are designed to simulate real-world scenarios and verify the functionality of an application. However, the testing ecosystem often faces transient issues that can cause tests to fail sporadically. Here are a few reasons why retrying failed test cases is essential:

  1. Transient Issues: Tests may fail due to temporary issues like network instability, server overload, or environment setup delays. Retrying can help bypass such issues without manual intervention.
  2. Flaky Tests: Some tests are inherently flaky due to external dependencies or timing issues. Retrying allows these tests to re-execute, reducing false negatives in test reports.
  3. Efficient Debugging: When a test fails consistently even after retries, it highlights a genuine issue that needs immediate attention, making debugging more focused and effective.
  4. Time-Saving: Automating the retry process saves significant time by eliminating the need for manual re-execution of failed tests.
How can we retry failed Test Cases?

TestNG provides us with various ways to retry failed test cases, and we have mentioned a couple of them. We will go in-depth for each of them so that you understand when to use what.

Let’s look at each of the ways one by one.

Retry failed test cases using the testng-failed.xml file

Let’s first try to run a test class. Our test class will have three test cases.

package Test;

import org.testng.Assert;
import org.testng.annotations.Test;

public class CodekruTest {

	@Test
	public void test1() {
		System.out.println("Executing test1 in CodekruTest class");
		Assert.assertTrue(true);
	}

	@Test
	public void test2() {
		System.out.println("Executing test2 in CodekruTest class");
		Assert.assertTrue(false); // failing this test case
	}

	@Test
	public void test3() {
		System.out.println("Executing test3 in CodekruTest class");
		Assert.assertTrue(false); // failing this test case as well
	}

}

Below is the XML file to run the above test class

<suite name="codekru">
	<test name="codekruTest">
		<classes>
			<class name="Test.CodekruTest" />
		</classes>
	</test>
</suite>

Output after running the above XML file –

run as testng suite
Executing test1 in CodekruTest class
Executing test2 in CodekruTest class
Executing test3 in CodekruTest class

===============================================
codekru
Total tests run: 3, Passes: 1, Failures: 2, Skips: 0

Now, two cases failed (test2 and test3), and a testng-failed.xml file has been created in the testng-output folder, which we will use to rerun the failed cases.

testmg-failed.xml package structure

The content of the testng-failed.xml file contains the necessary information to run the failed cases once again.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Failed suite [Failed suite [codekru]]" guice-stage="DEVELOPMENT">
  <test thread-count="5" name="codekruTest(failed)(failed)">
    <classes>
      <class name="Test.CodekruTest">
        <methods>
          <include name="test3"/>
          <include name="test2"/>
        </methods>
      </class> <!-- Test.CodekruTest -->
    </classes>
  </test> <!-- codekruTest(failed)(failed) -->
</suite> <!-- Failed suite [Failed suite [codekru]] -->

Output after running the testng-failed.xml file –

[RemoteTestNG] detected TestNG version 7.4.0
Executing test3 in CodekruTest class
Executing test2 in CodekruTest class

===============================================
Failed suite [Failed suite [codekru]]
Total tests run: 2, Passes: 0, Failures: 2, Skips: 0

Here, we can see that only the failed cases were executed.

What if the failed cases are dependent on other passed test cases?

Let’s include one more test case (test4) in our CodekruTest class, and test2 and test3 will depend on test4 using the dependsOnMethods attribute.

package Test;

import org.testng.Assert;
import org.testng.annotations.Test;

public class CodekruTest {

	@Test
	public void test1() {
		System.out.println("Executing test1 in CodekruTest class");
		Assert.assertTrue(true);
	}

	@Test(dependsOnMethods = "test4")
	public void test2() {
		System.out.println("Executing test2 in CodekruTest class");
		Assert.assertTrue(false); // failing this test case
	}

	@Test(dependsOnMethods = "test4")
	public void test3() {
		System.out.println("Executing test3 in CodekruTest class");
		Assert.assertTrue(false); // failing this test case as well
	}

	@Test
	public void test4() {
		System.out.println("Executing test4 in CodekruTest class");
		Assert.assertTrue(true);
	}
}

Let’s run our CodekruTest class again.

Executing test1 in CodekruTest class
Executing test4 in CodekruTest class
Executing test2 in CodekruTest class
Executing test3 in CodekruTest class
PASSED: test4
PASSED: test1
FAILED: test3
FAILED: test2
===============================================
    Default test
    Tests run: 4, Failures: 2, Skips: 0
===============================================

Here, the test4 case passes, whereas tests 2 and 3 still fail. Now, let’s examine the contents of the testng-failed.xml file.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Failed suite [Default suite]" guice-stage="DEVELOPMENT">
  <test thread-count="5" name="Default test(failed)">
    <classes>
      <class name="Test.CodekruTest">
        <methods>
          <include name="test4"/>
          <include name="test3"/>
          <include name="test2"/>
        </methods>
      </class> <!-- Test.CodekruTest -->
    </classes>
  </test> <!-- Default test(failed) -->
</suite> <!-- Failed suite [Default suite] -->

We can see that test4 is also included in the testng-failed.xml file along with test2 and test3. This happened because the test2 and test3 were dependent upon the test4, and we would need to run the test4 before running the test2 or test3.

Output after running the testng-failed.xml file

Executing test4 in CodekruTest class
Executing test3 in CodekruTest class
Executing test2 in CodekruTest class

===============================================
Failed suite [Default suite]
Total tests run: 3, Passes: 1, Failures: 2, Skips: 0
===============================================
Retry failed cases using RetryAnalyzer

Running the failed cases through testng-failed.xml is an effective way to rerun the test cases, but your failed cases will be executed after TestNG executes the entire test suite. If you want to rerun the test cases as soon as they fail, we can use the retryAnalyzer attribute with @Test annotation.

Some advantages of using the retryAnalyzer attribute
  • The failed case would be executed as soon as it fails, without waiting for TestNG to execute the whole suite.
  • We can retry the failed as many times as we want.
So, how can we use the retry analyzer in our code?
  • Make a class that implements the IRetryAnalyzer interface.
  • Use that class’s name as the value of the retryAnalyzer attribute. E.g., @Test(retryAnalyzer = Retry.class)

Let’s make our Retry class that will implement the IRetryAnalyzer interface. IRetryAnalyzer has one retry() method which we will implement.

package Test;

import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public class Retry implements IRetryAnalyzer {

	int counter = 0;
	int retryLimit = 2; // maximum number of times, the case can be retried

	public boolean retry(ITestResult result) {
		if (counter < retryLimit) {
			counter++;
			return true; // retry the case if retryLimit is not exhausted
		}
		return false; // Do not retry anymore, as retryLimit is exhausted
	}

}
  • Here, the counter variable stores the number of times a case has been executed.
  • retryLimit variable tells the maximum number of retries for a single test case.

It will retry the case until a case passes or the retries have been exhausted. Now, we will use it in our test class ( CodekruTest ).

package Test;

import org.testng.Assert;
import org.testng.annotations.Test;

public class CodekruTest {

	@Test
	public void test1() {
		System.out.println("Executing test1 in CodekruTest class");
		Assert.assertTrue(true);
	}

	@Test(retryAnalyzer = Retry.class)
	public void test2() {
		System.out.println("Executing test2 in CodekruTest class");
		Assert.assertTrue(false); // failing this test case
	}

	@Test()
	public void test3() {
		System.out.println("Executing test3 in CodekruTest class");
		Assert.assertTrue(false); // failing this test case as well
	}
}

Now, the test2() case would be retried twice after failing. So, let’s now try to run our CodekruTest class using the below XML file.

<suite name="codekru">
	<test name="codekruTest">
		<classes>
			<class name="Test.CodekruTest" />
		</classes>
	</test>
</suite>

Output –

Executing test1 in CodekruTest class
Executing test2 in CodekruTest class
Executing test2 in CodekruTest class
Executing test2 in CodekruTest class
Executing test3 in CodekruTest class

===============================================
codekru
Total tests run: 5, Passes: 1, Failures: 2, Skips: 0, Retries: 2

We can see that the test2 method was executed two more times, as we have kept the retry count as 2.

How to put retry on all cases of a class?

There might be a scenario where you need to place a retry analyzer on all test cases of a class, and the test class contains hundreds of test cases. It could become quite cumbersome to place the retryAnalyzer on each and every test case. In such a scenario, we can place @Test((retryAnalyzer = Retry.class) at the class level, and the retries will automatically be applied to all cases of that class.

package Test;

import org.testng.Assert;
import org.testng.annotations.Test;

@Test(retryAnalyzer = Retry.class)
public class CodekruTest {

	public void test1() {
		System.out.println("Executing test1 in CodekruTest class");
		Assert.assertTrue(true);
	}

	public void test2() {
		System.out.println("Executing test2 in CodekruTest class");
		Assert.assertTrue(false); // failing this test case
	}

	public void test3() {
		System.out.println("Executing test3 in CodekruTest class");
		Assert.assertTrue(false); // failing this test case as well
	}
}

Now, all of the test cases would be retried if they failed. Let’s try it by executing the above class.

Executing test1 in CodekruTest class
Executing test2 in CodekruTest class
Executing test2 in CodekruTest class
Executing test2 in CodekruTest class
Executing test3 in CodekruTest class
Executing test3 in CodekruTest class
Executing test3 in CodekruTest class
How to put retry on all cases of a suite?

Now, you might say that we can also have a large number of classes, and again, putting the retryAnalyzer attribute on each class and its cases would be tiresome. Don’t worry, we have one easy way.

We will implement the IAnnotationTransformer interface class to put a retry analyzer on every test case of a suite.

This interface is used to manipulate the annotations at runtime programmatically, and its transform method executes on every test run. So, we will rewrite the transform method and add the retryAnalyzer functionality.

package Test;

import org.testng.IAnnotationTransformer;
import org.testng.annotations.ITestAnnotation;
import java.lang.reflect.Constructor;
import java.lang.reflect.Method;

public class AnnotationTransformer implements IAnnotationTransformer {

	public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) {
		annotation.setRetryAnalyzer(Retry.class); // setting the retryAnalyzer
	}

}

Now, we need to add the AnnotationTransformer class as a listener in testng.xml file, and then retries will be automatically added to all cases executed by the testng.xml file.

Now, our CodekruTest class will look like

package Test;

import org.testng.Assert;
import org.testng.annotations.Test;

public class CodekruTest {

	@Test
	public void test1() {
		System.out.println("Executing test1 in CodekruTest class");
		Assert.assertTrue(true);
	}

	@Test
	public void test2() {
		System.out.println("Executing test2 in CodekruTest class");
		Assert.assertTrue(false); // failing this test case
	}

	@Test
	public void test3() {
		System.out.println("Executing test3 in CodekruTest class");
		Assert.assertTrue(false); // failing this test case as well
	}
}

And our XML file would be

<suite name="codekru">
	<listeners>
		<listener class-name="Test.AnnotationTransformer" />
	</listeners>
	<test name="codekruTest">
		<classes>
			<class name="Test.CodekruTest" />
		</classes>
	</test>
</suite>

Output after running the above XML file –

Executing test1 in CodekruTest class
Executing test2 in CodekruTest class
Executing test2 in CodekruTest class
Executing test2 in CodekruTest class
Executing test3 in CodekruTest class
Executing test3 in CodekruTest class
Executing test3 in CodekruTest class

===============================================
codekru
Total tests run: 7, Passes: 1, Failures: 2, Skips: 0, Retries: 4
===============================================

We can see that the failed cases were automatically retried.

One last important thing

You might have noticed by now that the number of total tests increases whenever we retry the cases, depending upon the number of retries.

Note: The newer versions of TestNG have the Retries field added to the summary.

Executing test1 in CodekruTest class
Executing test2 in CodekruTest class
Executing test2 in CodekruTest class
Executing test2 in CodekruTest class
Executing test3 in CodekruTest class
Executing test3 in CodekruTest class
Executing test3 in CodekruTest class

===============================================
codekru
Total tests run: 7, Passes: 1, Failures: 2, Skips: 0, Retries: 4
===============================================
test case summary after retry

But we only executed 3 cases, so we may only want to show the total tests as 3 only and not 7. Otherwise, the number of total tests will not be consistent and depend upon the retries during the execution.

So, how to exclude the retries count from the total tests count?

Here, we will have to use one more interface, ITestListener, and implement its onTestSuccess() and onTestFailure() methods.

package Test;

import org.testng.ITestListener;
import org.testng.ITestResult;

public class TestListenerClass implements ITestListener {

	public void onTestSuccess(ITestResult result) {

		if (result.getMethod().getRetryAnalyzer(result) != null) {
			int numberOfSkippedCases = result.getTestContext().getSkippedTests().getResults(result.getMethod()).size();

			for (int i = 0; i < numberOfSkippedCases; i++) {
				result.getTestContext().getSkippedTests().removeResult(result.getMethod());
			}
		}

	}
	
	public void onTestFailure(ITestResult result) {

		if (result.getMethod().getRetryAnalyzer(result) != null) {
			int numberOfSkippedCases = result.getTestContext().getSkippedTests().getResults(result.getMethod()).size();

			for (int i = 0; i < numberOfSkippedCases; i++) {
				result.getTestContext().getSkippedTests().removeResult(result.getMethod());
			}
		}

	}

}

So, what are we doing here?

  • onTestFailure() method executes every time a case fails, and onTestSuccess() method executes if a test case passes.
  • Whenever a test case passes or fails, we’ll use the condition result.getMethod().getRetryAnalyzer(result) != null to check if the test case has the retryAnalyzer enabled.
  • If the test case has retryAnalyzer enabled, we will count the number of skipped cases. Because each time we retry a case, it is marked as skipped ( latest TestNG versions shows them as retries)
  • Once we get the number of times a test case was skipped (which means the test case was retried that many times), then we will run a loop to remove the skipped cases from the result.

Now, add the TestListenerClass as a listener in the testng.xml file.

<suite name="codekru">
	<listeners>
		<listener class-name="Test.AnnotationTransformer" />
		<listener class-name="Test.TestListenerClass" />
	</listeners>
	<test name="codekruTest">
		<classes>
			<class name="Test.CodekruTest" />
		</classes>
	</test>
</suite>

Output after running the above XML file, which again executes the cases of our CodekruTest class –

Executing test1 in CodekruTest class
Executing test2 in CodekruTest class
Executing test2 in CodekruTest class
Executing test2 in CodekruTest class
Executing test3 in CodekruTest class
Executing test3 in CodekruTest class
Executing test3 in CodekruTest class

===============================================
codekru
Total tests run: 3, Passes: 1, Failures: 2, Skips: 0
===============================================

Here, we can see that the count shown is 3 only and not 7.

We hope that you liked the article. This is about retrying and rerunning the cases. If you have any doubts or concerns, please feel free to write to us in the comments or email us at admin@codekru.com.

Liked the article? Share this on

Leave a Comment

Your email address will not be published. Required fields are marked *