TestNG Tutorials 68 : Rerun Failed Test Method Using IRetryAnalyzer Interface – Implement Using retryAnalyzer attribute in the @Test annotation

Hello Folks,

TestNG provides a built in mechanism to re-run failed test case for N times (Ideally) instantly using IRetryAnalyzer interface. It means that if a test method fails, we can rerun it for N times or until it is passed during run. Let’s see this concept in more details:

IRetryAnalyzer Interface:

IRetryAnalyzer is an interface in org.testng package. It has only method named retry. By overriding this method in your class, you can control the number of attempts to rerun a failed test case.  You can refer code of IRetryAnalyzer below:

package org.testng;

/**
 * Interface to implement to be able to have a chance to retry a failed test.
 *
 * @author tocman@gmail.com (Jeremie Lenfant-Engelmann)
 *
 */
public interface IRetryAnalyzer {

  /**
   * Returns true if the test method has to be retried, false otherwise.
   *
   * @param result The result of the test method that just ran.
   * @return true if the test method has to be retried, false otherwise.
   */
  public boolean retry(ITestResult result);
}

When a @Test annotated method has value for “retryAnalyzer” attribute, reTry method will be called only if Test method is failed. “retry” method will not be executed at all if Test method is passed or skipped.

How to implement above mechanism?

Since IRetryAnalyzer is an interface, we must have to implement it in our class to provide definition to “retry” method. You can implement this interface in any normal class or TestNG class. If you keep a separate implemented class, it will be good for best readability and maintain. So, let’s create a class and implement  IRetryAnalyzer interface.

package RetryAnalyzer;

import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public class RetrySetup implements IRetryAnalyzer{
	
	// Variable to keep counter of retry
	private int retryCount = 0;
	
	// Variable to define max retry count
	private static final int maxRetryCount = 3;

	@Override
	public boolean retry(ITestResult result) {
		// We need to implement a cap, otherwise execution might go indefinite loop. 
		if(retryCount < maxRetryCount)
		{
			retryCount++;
			System.out.println("Retrying Test method : "+result.getName() + " for " + retryCount +" times. ");
			return true;
		}
		return false;
	}

}

Now, you need to add an attribute called “retryAnalyzer” to @Test annotated method with value as Implemented class name. If we do not add this attribute, remember “retry” will not be called automatically. It provides you flexibility of trying to rerun those test method which fails sometimes because of some random issues (not functional) only.

package RetryAnalyzer;

import org.testng.Assert;
import org.testng.annotations.Test;

public class RetryTest {

	// Need to mention implemented class name of IRetryAnalyzer
	@Test(retryAnalyzer= RetrySetup.class)
	public void method1()
	{
		System.out.println("Failing method purposefully.");
		Assert.fail("Purposefully failing");	
	}

	
}

Execution flow:

  1. “method1” will be run and it will fail as we are failing it wantedly.
  2. Then control will go to retry method of “RetrySetup class as @Test annotation has “retryAnalyzer attribute.
  3. Since retryCount < maxRetryCount (0 < 3),  retry will return true and retryCount counter will be increased by one. “method1” will be executed again. Earlier run will be marked as skipped.
  4. Same flow will be repeated till ( retryCount < maxRetryCount) becomes false and finally “method1” will be marked as failed.

Output:

[java]
[RemoteTestNG] detected TestNG version 6.14.3
Failing method purposefully.
Retrying Test method : method1 for 1 times.
Failing method purposefully.
Retrying Test method : method1 for 2 times.
Failing method purposefully.
Retrying Test method : method1 for 3 times.
Failing method purposefully.
FAILED: method1
java.lang.AssertionError: Purposefully failing
at org.testng.Assert.fail(Assert.java:96)
at RetryAnalyzer.RetryTest.method1(RetryTest.java:13)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:583)
at org.testng.internal.Invoker.retryFailed(Invoker.java:839)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1010)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.privateRun(TestRunner.java:648)
at org.testng.TestRunner.run(TestRunner.java:505)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
at org.testng.SuiteRunner.run(SuiteRunner.java:364)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
at org.testng.TestNG.runSuites(TestNG.java:1049)
at org.testng.TestNG.run(TestNG.java:1017)
at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:114)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:251)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:77)

SKIPPED: method1
java.lang.AssertionError: Purposefully failing
at org.testng.Assert.fail(Assert.java:96)
at RetryAnalyzer.RetryTest.method1(RetryTest.java:13)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:583)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:989)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.privateRun(TestRunner.java:648)
at org.testng.TestRunner.run(TestRunner.java:505)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
at org.testng.SuiteRunner.run(SuiteRunner.java:364)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
at org.testng.TestNG.runSuites(TestNG.java:1049)
at org.testng.TestNG.run(TestNG.java:1017)
at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:114)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:251)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:77)

SKIPPED: method1
java.lang.AssertionError: Purposefully failing
at org.testng.Assert.fail(Assert.java:96)
at RetryAnalyzer.RetryTest.method1(RetryTest.java:13)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:583)
at org.testng.internal.Invoker.retryFailed(Invoker.java:839)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1010)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.privateRun(TestRunner.java:648)
at org.testng.TestRunner.run(TestRunner.java:505)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
at org.testng.SuiteRunner.run(SuiteRunner.java:364)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
at org.testng.TestNG.runSuites(TestNG.java:1049)
at org.testng.TestNG.run(TestNG.java:1017)
at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:114)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:251)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:77)

SKIPPED: method1
java.lang.AssertionError: Purposefully failing
at org.testng.Assert.fail(Assert.java:96)
at RetryAnalyzer.RetryTest.method1(RetryTest.java:13)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:583)
at org.testng.internal.Invoker.retryFailed(Invoker.java:839)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1010)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.privateRun(TestRunner.java:648)
at org.testng.TestRunner.run(TestRunner.java:505)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
at org.testng.SuiteRunner.run(SuiteRunner.java:364)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
at org.testng.TestNG.runSuites(TestNG.java:1049)
at org.testng.TestNG.run(TestNG.java:1017)
at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:114)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:251)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:77)

===============================================
Default test
Tests run: 4, Failures: 1, Skips: 3
===============================================

===============================================
Default suite
Total tests run: 4, Failures: 1, Skips: 3
===============================================

[/java]

You need to understand output carefully. Points are as below:

1. @Test method will be marked as failure once retry count reaches maximum allowed count.
2. Each try will be counted as test run. You can see above Total tests run is 4.
3. “retry” method will not be called if @Test annotated method passed before max retry count. For example: If you want to retry a method 5 times on failure, and method passed in 3rd attempt, remaining attempts will not be considered.
4. All attempt will be marked as Skipped except last attempt in result. You can see Skips count as 3 which is actually a retry count.

Consider a scenario where @Test method passed in retry:

package RetryAnalyzer;

import org.testng.Assert;
import org.testng.annotations.Test;

public class RetryTest {
	
	private static int counter= 0;

	// Need to mention implemented class name of IRetryAnalyzer
	@Test(retryAnalyzer= RetrySetup.class)
	public void method1()
	{
		counter++;
		if(counter <= 2)
		{
			System.out.println("Failing method purposefully.");
			Assert.fail("Purposefully failing");
		}
			
		System.out.println("Test passed");
	}

	
}

 

Output:

[java]
[RemoteTestNG] detected TestNG version 6.14.3
Failing method purposefully.
Retring Test method : method1 for 1 times.
Failing method purposefully.
Retring Test method : method1 for 2 times.
Test passed
PASSED: method1
SKIPPED: method1
java.lang.AssertionError: Purposefully failing
at org.testng.Assert.fail(Assert.java:96)
at RetryAnalyzer.RetryTest.method1(RetryTest.java:18)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:583)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:989)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.privateRun(TestRunner.java:648)
at org.testng.TestRunner.run(TestRunner.java:505)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
at org.testng.SuiteRunner.run(SuiteRunner.java:364)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
at org.testng.TestNG.runSuites(TestNG.java:1049)
at org.testng.TestNG.run(TestNG.java:1017)
at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:114)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:251)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:77)

SKIPPED: method1
java.lang.AssertionError: Purposefully failing
at org.testng.Assert.fail(Assert.java:96)
at RetryAnalyzer.RetryTest.method1(RetryTest.java:18)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:583)
at org.testng.internal.Invoker.retryFailed(Invoker.java:839)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1010)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.privateRun(TestRunner.java:648)
at org.testng.TestRunner.run(TestRunner.java:505)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
at org.testng.SuiteRunner.run(SuiteRunner.java:364)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
at org.testng.TestNG.runSuites(TestNG.java:1049)
at org.testng.TestNG.run(TestNG.java:1017)
at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:114)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:251)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:77)

===============================================
Default test
Tests run: 3, Failures: 0, Skips: 2
===============================================

===============================================
Default suite
Total tests run: 3, Failures: 0, Skips: 2
===============================================

[/java]

Only last run will be marked as final result. ALl earlier run will be marked as skipped.

If you have any doubt, feel free to comment below.
If you like my posts, please like, comment, share and subscribe.
#ThanksForReading
#HappySelenium

5 thoughts on “TestNG Tutorials 68 : Rerun Failed Test Method Using IRetryAnalyzer Interface – Implement Using retryAnalyzer attribute in the @Test annotation

  1. Only last run will be marked as final result. ALl earlier run will be marked as skipped.——–Why earlier test cases marked as skipped

  2. How can we implement this in BDD framework, I have created the RetrySetup class, then where will provide the @Test(retryAnalyzer= RetrySetup.class) and we are using the hooks class of cucumber

Leave a Reply

Your email address will not be published. Required fields are marked *