Retry Failed Tests in TestNG using the IReTryAnalyzer Interface

IRetryAnalyzer Interface is used to rerun the failed tests. Every time when the tests failed in a suite, TestNG creates a file called testng-failed.xml in the target-output. This XML file contains the detailed information about the test that gets failed and allowed to rerun only these methods without having to run the entire suite which contains all the tests. This XML file contains all the failed test information without any skip it will rerun the failed tests when the IRetryAnalyzer interface is implemented in the Framework.

Retry Analyzer is specified at class level and suite level:

For class level:

Te execute the retry need to specify in each class in order to run.

@Test(retryAnalyzer=IRetryAnalyzerClassFile.class)

For Suite level:

implementation can be mentioned in the IAnnotationTransform interface method by calling the IRetryAnalyzerClassFile.

<listeners>

<listener class-name="com.utilities.ITestListenerFile" />

</listeners>

Implementation Steps to follow:

Step 1:

Create an class file which extends BaseClass, implements ITestListener and IAnnotationTransformer as shown in picture below:

• Import the IAnnotationTransformer from org.testng.IAnnotationTransformer.

Step2:

Right click on IAnnotationTransformer go to source then click to override/Implement Methods and use the methods what you need for the test execution.

Step 3:

Select the method as shown in picture below.

Step 4:

Delete the line 18 to write the customized script for the failed testcases that needs to be rerun.

Check the below script: Write the code as per the required custom reports.

package com.utilities;

import java.lang.reflect.Constructor;

import java.lang.reflect.Method;

import org.testng.IAnnotationTransformer;

import org.testng.ITestContext;

import org.testng.ITestListener;

import org.testng.ITestResult;

import org.testng.Reporter;

import org.testng.annotations.ITestAnnotation;

import com.testBase.BaseClass;

public class ITestListernerFile extends BaseClass implements ITestListener, IAnnotationTransformer {

public static String getMethodName(ITestResult result) {

return result.getMethod().getConstructorOrMethod().getName();

}

public String getMethodNameFromIAnnotationMethod(Method testMethod) {

return testMethod.getName();

}

// IAnnotationTransformer Interface Implemented Method

@Override

public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) {

annotation.setRetryAnalyzer(IRetryAnalyzerClassFile.class);

System.out.println("transform method " + getMethodNameFromIAnnotationMethod(testMethod));

// LoggerLoad.info("transform method " +

// getMethodNameFromIAnnotationMethod(testMethod));

Reporter.log("transform method " + getMethodNameFromIAnnotationMethod(testMethod));

}

// ITestListener Interface implemented Methods

@Override

public void onTestStart(ITestResult result) {

LoggerLoad.info(getMethodName(result) + " Test is starting.");

System.out.println(getMethodName(result) + " Test is starting.");

Reporter.log(getMethodName(result) + " Test is starting.");

}

@Override

public void onTestSuccess(ITestResult result) {

LoggerLoad.info(getMethodName(result) + " Test is succeed.");

System.out.println(getMethodName(result) + " Test is starting.");

Reporter.log(getMethodName(result) + " Test is succeed.");

}

@Override

public void onTestFailure(ITestResult result) {

LoggerLoad.info(getMethodName(result) + " Test is failed.");

Reporter.log("onTestFailure" + result.getName());

}

@Override

public void onTestSkipped(ITestResult result) {

LoggerLoad.info(getMethodName(result) + " Test is Skipped.");

Reporter.log("onTestSkipped" + result.getName());

}

@Override

public void onTestFailedButWithinSuccessPercentage(ITestResult result) {

LoggerLoad.info(getMethodName(result) + " Test Failed within Success Percentage.");

Reporter.log("onTestFailedButWithinSuccessPercentage" + result.getName());

}

@Override

public void onTestFailedWithTimeout(ITestResult result) {

LoggerLoad.info(getMethodName(result) + " Test Failed with Timeout.");

Reporter.log("onTestFailedWithTimeout" + result.getName());

}

@Override

public void onStart(ITestContext context) {

LoggerLoad.info(" onStart method " + context.getName());

Reporter.log("onStart method " + context.getName());

}

@Override

public void onFinish(ITestContext context) {

LoggerLoad.info("onFinish method " + context.getName());

Reporter.log("onFinish method " + context.getName());

}

}

***************************************************************************************************************************

Lets see the script file for rerun the failed test case:

Create a class name which implements the IRetryAnalyzer interface and have the specified retrycount for the failed testcases. Below is the example.

package com.utilities;

import org.testng.IRetryAnalyzer;

import org.testng.ITestResult;

public class IRetryAnalyzerClassFile implements IRetryAnalyzer {

private int count = 0;

private int retryCount = 3;

public String getResultStatusName(int status) {

String resultName = null;

if (status == 1)

resultName = "SUCCESS";

if (status == 2)

resultName = "FAILURE";

if (status == 3)

resultName = "SKIP";

return resultName;

}

@Override

public boolean retry(ITestResult testResult) {

System.out.println("Retry method starts From IRetryAnalyzerClassFile");

if (!testResult.isSuccess()) {

if (count < retryCount) {

System.out.println("Retrying the test " + testResult.getName() + " method with status " + "'"

+ getResultStatusName(testResult.getStatus()) + "'" + " for the " + (count + 1) + " time(s).");

LoggerLoad.info("Retrying the test " + testResult.getName() + " method with status " + "'"

+ getResultStatusName(testResult.getStatus()) + "'" + " for the " + (count + 1) + " time(s).");

testResult.setStatus(testResult.SKIP);

System.out.println("************************** IRetryAnalyzerClassFile**************************** ");

count++;

return true;

} else {

System.out.println("************************** IRetryAnalyzerClassFile**************************** ");

testResult.setStatus(testResult.FAILURE);

}

} else {

System.out.println("************************** IRetryAnalyzerClassFile**************************** ");

testResult.setStatus(testResult.SUCCESS);

}

return false;

}

}

Implementing the RetryAnalyzer at Class Level : Annotate the RetryAnalyzer on each class in the test script as shown in picture below for executing the test at only class level.

Implementing the ITestListener at the Suite Level :  The ITestListener class file is specified with package name as shown in picture below to implement listeners in the TestNG .XML file.

Note: If we mention at

Note: If you do not want to run retryAnalyzer at class level, remove this line "@Test(retryAnalyzer=IRetryAnalyzerClassFile.class)" in class before you run in suite level.

At suite level, this line "IRetryAnalyzerClassFile.class" is mentioned inside the transform method using IAnnotationTransfrom Interface, as shown in below picture.

Reports:

You can use the Reporter Log, Extent Report, Allure Report inside the override method to see the results.

Screenshot for testng-failed.xml file:

You can see the failed.XML under the test-ouput as show in picture.

Console Output:

To see the message in the console output.

Emailable-Report HTML:

After executing the TestNG.XML file, refresh the maven project, right click, then go to emailable-report.html under the test-ouput.

Using the Reporter Log :

To see the message in the Emailable report, the reporter logs is used and below is the example.

Conclusion:'

In TestNG , Retry Analyzer is used to improve the reliability of your test suite. It allows the failed tests with multiple chances to pass. By following the steps from this article and customizing the retry test as per the test needs, you can ensure that automation frameworks is more stable when the tests failures happened, it gives the reliable test results.