_edited_edited.png)
In the world of software development, delivering high-quality products efficiently is paramount. Test automation plays a crucial role in achieving this goal by enabling continuous testing, quick feedback, and early detection of issues. One of the essential aspects of test automation is generating and distributing test reports. In this blog, we’ll explore how to configure test automation reports emailing via Jenkins admin console, with a focus on using Cucumber BDD Java for generating various reports.
These reports provide comprehensive information about the tests, including:
Test Execution Results: Which tests passed, failed, or were skipped.
Execution Time: How long each test took to run.
Environment Details: Information about the test environment, such as OS, browser, and application version.
Error Logs and Screenshots: Details about any failures, including error messages and screenshots.
Test Coverage: Metrics indicating which parts of the application were tested.
The primary purpose of test automation reports is to provide clear and concise feedback on the health and stability of the software application being tested. They help in:
Identifying Issues: Quickly pinpointing bugs and errors in the application.
Tracking Progress: Monitoring the progress of testing efforts over time.
Informing Stakeholders: Providing necessary information to developers, testers, and managers to make informed decisions.
Maintaining Quality: Ensuring the software meets the required quality standards before release.
Benefits of Test Automation Reports
Increased Visibility: Offers a transparent view of the testing process and results to all stakeholders.
Faster Feedback: Provides immediate feedback on the quality of the application, allowing for quicker remediation.
Consistency: Ensures tests are executed and reported consistently, reducing the chance of human error.
Detailed Insights: Provides in-depth analysis of test failures, helping to diagnose and fix issues efficiently.
Historical Data: Maintains a history of test executions and outcomes, useful for identifying trends and recurring issues.
Overview of few Test Automation Reporting Tools
Cucumber Test Report
Description: Cucumber is a tool for running automated tests written in plain language. It generates detailed reports based on the execution of scenarios defined in Gherkin syntax.
Key Features:
Human-Readable Format: Reports are easy to understand as they are based on the Gherkin language, which is close to natural language.
Scenario Breakdown: Provides detailed information on each scenario, including steps, execution time, and outcomes.
Integration: Works seamlessly with BDD (Behavior-Driven Development) frameworks and supports integration with various CI/CD tools.
Best Use Cases:
Behavior-Driven Development (BDD): Best suited for projects practicing BDD where test scenarios are written in Gherkin.
Stakeholder Communication: Useful for communicating test results to non-technical stakeholders due to its human-readable format.
Allure Report
Description: Allure Report is a flexible and comprehensive reporting framework designed to provide clear and concise test reports. It integrates with various test frameworks and provides a rich visualization of the test execution results.
Key Features:
Interactive Reports: Offers detailed and interactive reports with drill-down capabilities.
Customization: Allows customization of report layouts and templates.
Attachments: Supports attaching screenshots, logs, and other artifacts to test results.
Integration: Compatible with various test frameworks such as JUnit, TestNG, Cucumber, and more.
Best Use Cases:
Complex Test Suites: Ideal for projects with complex test suites needing detailed and interactive reports.
Enhanced Visualization: When there's a need for rich visualization of test results, including detailed statistics and trends.
Extent Report
Description: Extent Reports is a reporting library that provides detailed and visually appealing HTML reports for test executions. It supports multiple testing frameworks and offers advanced customization options.
Key Features:
Customizable Reports: Highly customizable with themes, charts, and widgets.
Real-Time Reporting: Can generate reports in real-time during test execution.
Interactive Features: Supports interactive elements like filters, search, and drill-downs.
Integration: Compatible with frameworks like TestNG, JUnit, and more.
Best Use Cases:
Detailed Custom Reports: Ideal for projects requiring highly customizable and visually appealing reports.
Real-Time Feedback: Suitable for environments where real-time feedback during test execution is crucial.
JUnit Automation Reports
Description: JUnit is a popular testing framework for Java applications, and its built-in reporting features provide basic but essential details about test execution.
Key Features:
Simple Reports: Generates XML reports that include information about test cases, execution time, and outcomes.
Integration: Easily integrates with CI/CD tools like Jenkins, which can further process and visualize the JUnit reports.
Standardization: Widely used and supported by many tools and IDEs.
Best Use Cases:
Java Projects: Best suited for projects using JUnit for testing Java applications.
Basic Reporting Needs: Suitable for teams needing straightforward, no-frills test reports that can be integrated with CI tools for further processing.
Configuring Emailable reports with Jenkins Build
Install Necessary Plugins
Navigate to Manage Jenkins:
Open the Jenkins dashboard.
Click on "Manage Jenkins" in the left-hand sidebar.
Manage Plugins:
Click on "Manage Plugins".
Go to the "Available" tab.
Search for and install the following plugins:
Email Extension Plugin
Allure Jenkins Plugin
Cucumber Reports Plugin
JUnit Plugin
Extent Reports Plugin
Configure Jenkins Job for Test Execution and Reporting
For a Freestyle Job:
Source Code Management:
Configure the source code repository (e.g., Git).
Build Steps:
Add build steps to compile and run tests. For example:
Execute shell: mvn clean install
Execute shell: mvn test
Post-Build Actions:
Allure Reports:
Add post-build action "Allure Report".
Specify the results directory (e.g., target/allure-results).
Cucumber Reports:
o Add post-build action "Publish Cucumber results".
o Specify the results JSON directory (e.g., target/cucumber-reports/*.json).
JUnit Reports:
o Add post-build action "Publish JUnit test result report".
o Specify the results XML directory (e.g., target/surefire-reports/*.xml).
Extent Spark Reports:
o Ensure the report is generated as part of the build script and stored in the workspace (e.g., target/extent-reports/).
o Add post-build action "Archive the artifacts" and specify the pattern (e.g., target/extent-reports/*.html).
Configure Email Notifications
Navigate to Manage Jenkins > Configure System:
Scroll down to the "Extended E-mail Notification" section.
Configure the SMTP server, default user email suffix, and other email settings. Below steps will guide using a gmail to email the test reports
Below is an option to configure a Gmail account to use in Java code to send an email. Please refer the detailed steps and instructions from GMAIL documentation for latest information and recommended secure best practices to send an email using Gmail account.
Use an App Password (Recommended for Accounts with 2-Step Verification)
Go to your Google Account.
Click on Security in the left sidebar.
Under Signing in to Google, click on App passwords.
You might need to sign in again.
Select Mail as the app and Other as the device, then give it a name (e.g., Java Mail).
Click on Generate.
Copy the 16-character app password displayed. This password will be used in place of password or credential field
Email Notifications in Job:
In the Jenkins job configuration, add a post-build action "Editable Email Notification".
Configure the email details:
Project Recipient List: Enter recipient email addresses.
Default Subject: "Build ${PROJECT_NAME} - ${BUILD_NUMBER} - ${BUILD_STATUS}"
Default Content: Include the body of the email and attach relevant reports.
Attachments: Specify patterns to attach reports (e.g., target/**/*.html).
Example Email Notification Configuration:
Project Recipient List: your-email@example.com
Default Subject: Build ${PROJECT_NAME} - ${BUILD_NUMBER} - ${BUILD_STATUS}
By following above steps, the test reports will be emailed to configured recipients
Advantages of Integrating Test Automation Reports with Jenkins
1. Continuous Integration and Continuous Deployment (CI/CD):
Automated Testing Pipeline: Integrating test reports with Jenkins enables automated execution of tests as part of the CI/CD pipeline. Tests can be run automatically after code changes are pushed, ensuring new changes do not break existing functionality.
Consistent Execution: Tests are run in a consistent environment, reducing the "it works on my machine" problem.
2. Real-time Feedback:
Immediate Notifications: Jenkins can be configured to send notifications via email, Slack, or other communication tools when tests fail, providing instant feedback to developers.
Dashboard and Trends: Jenkins provides dashboards that display test results and trends over time, allowing teams to monitor the health of their application continuously.
3. Scalability:
Parallel Execution: Jenkins can distribute and run tests in parallel across multiple nodes, speeding up the testing process.
Resource Management: Efficiently manages resources by allocating tests to different Jenkins agents based on availability and capacity.
4. Enhanced Reporting:
Custom Reports: Jenkins can integrate with various reporting tools (like Allure, JUnit, TestNG) to generate detailed and customizable test reports.
Historical Data and Trends: Keeps track of test results over multiple builds, providing insights into long-term trends and stability of the application.
5. Integration with Other Tools:
Defect Tracking: Jenkins can be integrated with defect tracking systems like Jira to automatically create tickets for failed tests.
Version Control: Integration with version control systems like Git ensures that test results are tied to specific code changes, helping in traceability and debugging.
Conclusion:
By following these steps, Jenkins can be configured to generate comprehensive test automation reports and send them via email. This setup ensures that the team receives timely and detailed feedback on the quality of the software, facilitating continuous improvement and faster delivery cycles. With Jenkins' robust capabilities and extensive plugin ecosystem, Jenkins job or pipeline can be customized to meet the specific needs of the project and organization.
.png)
Comments