Introduction

The Real Time Reporting Portal allows you to monitor the test suite executions that are in progress or completed. It updates the results of a test suite execution in real-time after each step in the test case iteration is executed.

Also, QAS updates the local copy of the report after each task iteration is completed. If a test case does not have tasks, then it updates after each test case iteration execution is completed.

The Real Time Reporting Portal opens automatically when executing a test suite in the following ways:

Through a Qualitia Client: It directly displays the Suite of Suites Screen.
Through an Offline Suite: It displays the login screen if the offline suite used is present on your machine and configured with the Qualitia Server. Ensure that you have access to the associated project to track the test suite execution progress.

Real Time Reporting shows the reports of test cases executed through a TFS/Azure DevOps pipeline only after the pipeline execution is completed.

Real Time Reporting Portal does not support test cases executed through HP ALM.

Accessing Real Time Reporting Portal

To access Real Time Reporting Portal, you need to contact Qualitia Admin who installed Qualitia Server to get the Reports and Dashboards URL.

Alternatively, you can access the real time reports from the Reports tab of Qualitia Client.

Pre-Requisites

Before you start test suite execution either through Qualitia Client or Offline Package, you must ensure the following:

Enable the Real Time Reporting checkbox in the mapped execution profile or Systems Execution Profile Settings.
The Real Time Reporting Server has enough drive space to displays test suite results. For more information, refer to Server Space Notification.
For an offline suite execution, your computer is connected to Qualitia Server. Also, you need to specify the Qualitia Server URL in the Settings tab of Qualitia Offline Configuration Manager.

Additional Configurations for Log Files

You can customize the logs generated with suite execution results in the mapped execution profile or Systems Execution Profile Settings as shown in the following table:

Settings	Description
Enable Real-Time Reporting	Displays the test execution results in the Real Time Reporting Portal. Also, if Web Dashboards feature is enabled, you can view these records in the Dashboard portal.
Maximum Size Per Log File	Lets you specify the maximum size for every log file of a test suite execution. The default value is 1MB.
Application Step Log Size	Lets you specify the maximum size for logs of a step. The default value is 1MB and the maximum value you can define is 2MB.
Application Log Level	Generates Info Logs, Error Logs, or All Logs for the test suite executions. You can select None, if you do not want any.

Suite of Suites Screen

The Suite of Suites screen shows the list of the test suite executions associated with the projects to which you have access. You can choose to view the test suite executions that are in progress, completed, aborted or all of them. It also updates the in progress test suite executions in terms of percentage.

For every test suite, it displays the following details:

Test Suite Environment Details
- Shows the test suite name, machine name, IP address, and project name.
- Qualitia denotes the test suite execution is through a Qualitia Client and Offline denotes the offline suite execution. If the offline suite is executed through a Continuous Integration (CI) tool, then the CI tool name is specified beside the suite execution result.
- Mouseover on the Environment Icon to view the operating system and browsers used for test case execution.
The total execution duration
Test Suite Execution Details
- The total number of test cases, with details of the number of test cases that passed, failed, have defects, and are not executed. It also shows a progress chart with the completion percentage for the executions in the process.

Realtime Reporting Portal displays the execution results as per the local time of your computer.
If the execution progress for a suite execution shows a red dot, it indicates that the suite execution results are not updated from the last 5 minutes. You need to check whether the test suite execution is in progress or the connection to Qualitia server is lost.
Real Time Reporting portal marks the suite execution results as Aborted when they are not updated for more than 1 hour. For more information, refer to Aborted Test Suite Executions.

Server Space Notifications

As the number of suite executions increase, the available drive space where Real Time reporting assets are stored starts reducing. The Real-time reporting portal shows a notification when the available drive space is:

10% to 30%: A warning message in amber appears at the top of the Portal page, which indicates the amount of drive space available along with the percentage. You must monitor the drive space and ensure that it does not reduce to less than 10%.
Less than 10%: A warning message in red appears at the top, which indicates that Real Time Reporting cannot display information on new suite executions. Also, you cannot upload or save a report.

In both cases, you need to contact the Qualitia Admin to purge reports. For detailed instructions on how to purge reports, refer to Purging Reports.

Filtering a Test Suite List

You can filter the list of test suites based on associated projects, test suite name, CI tool, users, and date range. You can also search for a project, test suite, and user.

To filter the test suite list:

Select a date range or click the Filter button in the top-right corner.
Select the appropriate options from the list of filters.

Uploading a Test Suite Execution Summary Report

Whenever a test suite is executed either through Qualitia Client or Offline Package, the test suite execution summary report is locally saved on the computer used for execution.

You can upload a locally saved test suite execution report to the Real Time Reporting when:

During the execution, the results are not pushed to the Real Time Reporting Portal due to network failure or the services of Qualitia Server restarted
An offline suite execution was not connected with Qualitia Server
A test suite execution was not configured to display its results on the Real Time Reporting portal
A test suite executed through parallel execution using Power Shell Utility

To upload multiple suite execution summary reports, refer to Uploading Multiple Suite Execution Reports to Real Time Reporting Portal.

Note: If you are uploading a locally saved test suite execution report that was executed using any Qualitia version lesser than v8.0.0, then you need to upload the report from the Old Reports screen of a Qualitia Client.

To upload a test suite execution summary report:

Create a ZIP file of the report folder that you want to upload.
Open the Suite of Suites screen.
Click the Upload button placed at the top-right corner.
Drag the ZIP file of the report or browse to upload the ZIP file of the locally saved test suite execution report.

After uploading the test report, its execution details are available in the Web Dashboards from the next day.

Test Suite Execution Summary Report

You can access a Test Suite Execution Summary Report by clicking a test suite execution on the Suites of Suites screen.

The Test Suite Execution Summary Report contains the execution status for each test case and its iterations. You can view the execution duration for each test case along with start and end time.

You can view the total number of test cases, with details of the number of test cases that passed, failed, have defects, and are not executed.

The following details of the test suite are displayed.

Report Element	Description
Status Icon	Indicates the number of test case iterations that have failed and have defects along with the total number of test case iterations. Also, a refresh icon denotes the relevant test case iteration execution that is in progress. If the current project is integrated with a defect management system, then for defects logged automatically or manually, it displays a bug icon. Upon hovering on the bug icon, all the associated defect IDs along with their links are displayed. Also, it shows whether the defect is logged automatically.
Test Case Element	Indicates the test case name or a test case iteration number.
Module	Indicates the associated module name.
Epic	Indicates the associated epic name.
Story	Indicates the associated story name.
Scenario	Indicates the associated Test Scenario name.
Manual TC Id	Indicates the associated Manual TC Id.

The execution time for the suites executed on desktop applications does not count the time spent on launching UFT.

The Module, Epic, and Story columns appear only if the associated project of the test suite is integrated with a Qualitia Design Studio project.

If the test suite execution is in progress, a blue-sticky status bar appears at the bottom of the screen, which shows the following details of the test case:
- The test case name
- The iteration number in progress and the total number of iterations
- A timer shows the amount of time lapsed since execution

Filtering the Test Suite Execution Summary Report

You can filter the list of test cases based on the associated status, test case, modules, epics, stories, scenario, and Manual TC Ids.

To filter the Test Suite Execution Summary Report:

Click the Filter button.
Select the appropriate filters.

Alternatively, you can filter the report by clicking a status in the top-right corner.

Exporting the Test Suite Execution Summary Report

You can export the Test Suite Execution Summary Report as an excel file.

To export the Test Case Execution Summary Report:

Click three dots, and select Export to Excel.

Sharing the Test Suite Execution Summary Report

You can share the link of this report with the applied filters to another Qualitia user who has access to the associated project.

To share the Test Suite Execution Summary Report:

Click three dots, and select Share Link.
Click Copy to copy the link.

Wrapping up the Test Suite Execution Summary Report

A cell in the report does not display more than 100 characters. Hence, you need to wrap the report to view all the data of a cell that contains more than 100 characters,

To wrap the report, click the Wrap Text button ( ).

Customizing the Test Suite Execution Summary Report

To customize the Test Suite Execution Summary Report:

Click three dots, and select Table Config to show or hide the columns.

Saving the Test Suite Execution Summary Report

You can save the report as an offline HTML file on your computer.

To save the Test Suite Execution Summary Report:

Click three dots, and select Save As Report.
The report is saved in the Download folder, which is specified in the browser's settings.

Logging a Defect Manually in the Integrated Defect Management System

You can log defects manually for a test case iteration whose execution status is Pass, Fail, or Defect. You need to configure the project with a Defect management System: Jira or TFS/Azure DevOps. For more information, refer to Integrating a Qualitia Project with Defects Management.

Test Case Iteration Detailed Report

You can access a Test Case Iteration Detailed Report by opening a test case node, under the Hierarchy column, and clicking the appropriate test case iteration name.

The Test Case Iteration Detailed Report shows in-depth step-level details for a test case iteration.

The topmost bar shows the following details of the test case iteration:

The execution duration along with the start and end time
The number of steps that have failed or have defects along with the total number of steps
The number of tasks that have failed or have defects along with the total number of tasks

You can expand this section to view the following details of the Test Suite execution

The execution duration along with the start and end time.
The total number of test cases, with details of the number of test cases that passed, failed, have defects, and are not executed. Also, a pie-chart shows the test case count per status for the test suite.

The following details of the test case iteration are displayed.

Report Element	Description
No	Indicates the step number.
Status Icon	Indicates the execution status of the step. If the current project is integrated with a defect management system, then for defects logged automatically or manually, it displays a bug icon. You can click the defect id links to view the defects in the associated defect management system. Also, you can know the defects that are auto-created.
Test Case Element	Indicates one of the following test case elements: Iteration number Task name Condition type Step name, which indicates the action performed. If a step contains results for an API response or generic actions such as CompareTabularResults, you can view those results in the associated info logs.
Data	Indicates the data associated with the action.
Object	Indicates the object name.
Execution Time	Indicates the execution time as per the associated hierarchy type.

Also, if the test suite execution is in progress, a blue-sticky status bar appears at the bottom of the screen, which shows the details of the test case execution.

Filtering the Test Case Iteration Detailed Report

You can filter the report based on execution statuses.

To filter the Test Case Iteration Detailed Report:

Click the Filter button.
Select the appropriate options from the list of filters.

Wrapping up the Test Case Iteration Detailed Report

A cell in the report does not display more than 100 characters. Hence, you need to wrap the report to view all the data of a cell that contains more than 100 characters,

To wrap the report, click the Wrap Text button ( ).

Navigating through the Steps and Tasks

To navigate to a specific line, enter the line number in the Go to Line text box and press Enter.

You can navigate through the failed steps and tasks in the report using the navigation controls shown in the following figure:

Note: Click the Failure Summary button to know failure categories for the test case iteration. For more information, refer to Viewing Failure Summary Report.

Viewing Screenshot and Logs for a Step

You can view the screenshots and logs of the application under test for each step execution.

To view screenshot and logs for a step:

Select an appropriate step and click the Screenshot and Log icon. The screenshot and the log appears.
You can enlarge a screenshot and navigate across the screenshots of the next or previous steps.
The Info Log section shows the following details:
1. Step Itinerary : Indicates the task, test case, scenario, and suite name.
2. Message: Describes how the step was executed. If the step has failed, it shows the reason of failure.
  If the step contains the results for an API response or generic actions, it contains a link that opens the results.
  You can view the results in the following formats:
  - json
  - xls
  - js (used for opening XML files)
3. Additional Messages: Indicates the name of the action, its execution Start and End time, and associated message received from the engine. If it contains the additional message, click the + symbol to see the additional messages.
4. Execution Status: Indicates the execution status of the step as an integer.
  1. 0 denotes the step has passed.
  2. 1 denotes the step has failed.
  3. 2 denotes the step has a defect.
The Error Log section shows if any exception occurred in execution.
The Configuration section shows the configured settings used for the execution.

Aborted Test Suite Executions

You can access the Aborted test suite executions from the Aborted tab.

A test suite execution is aborted:

If the user terminates the test suite execution forcefully
If the Qualitia Client or offline execution crashes
If the Qualitia Server services restart
If the Qualitia Server loses its connection with the Qualitia Client or Offline Package due to network failure

However, for the last two scenarios, the test suite execution continues, but its results are not pushed to Qualitia Server. In these scenarios, after the completion of the test suite execution, you can import the locally saved results from the Suite of Suites screen.

In case of network failure scenarios, if the connectivity resumes before the test suite execution completes, then its results are displayed. As per the execution status, the test suite execution is displayed on the Completed or Aborted tab.

Viewing an Aborted Test Suite Execution Summary Report

If you drill-down an Aborted test suite execution, both Suite Execution Summary Report and Test Case Iteration detailed report screens display the results as per the last update that was received.

Also, both screen shows the Aborted icon in the top-right corner. Upon hovering on the icon, it shows why the test suite execution was aborted for a known reason such as user forcefully terminating the execution or it shows a generic message.

Viewing a Failure Summary

Qualitia has categorized the failures that occur in the executions to help you troubleshoot the issues in test suite executions. Qualitia sorts the failure category as per the number of corresponding failures.

You can select an execution failure category and drill-down to the failed steps.

Failure categorization are not available for desktop projects.

The following table shows the pre-defined failure categories:

Failure Category	Description
Action Failure	Indicates that the defined action for the step did not execute correctly.
Action Is Not Supported	Indicates that the specified action in the step is not defined in Qualitia or the specified action cannot be used in the selected browser or environment.
Cannot Select From Dropdown	Indicates that the step did not execute as the specified drop-down list does not exist or it does not contain the option that is defined as a parameter.
Class File Is Missing In The Project	Indicates that the custom action did not execute because its source file is missing.
Failed To Store Data	Indicates that the action related to storing data failed.
File Does Not Exist	Indicates that either the specified file is not found in the specified location or the file extension is incorrect, or the step failed to create a file in the specified location.
Incorrect Data Format	Indicates that the parameters contain data in an invalid format, such as invalid date format, invalid file format, or invalid data encryption format.
Data Is Not Supported	Indicates that the parameter contains data that the object or the action does not support.
Invalid Expression	Indicates that the conditional expression syntax or the expression provided as input parameter for an action, is invalid.
Invalid Input Parameters	Indicates that the test data provided for the action is invalid.
Object Does Not Exist	Indicates that the object is not found or disabled on the selected browser or environment.
Testdata Provided Empty Or NULL	Indicates that the test data is not provided for the action.
Timeout Encountered	Indicates that the associated object is not found within the specified Wait Time.
Uniqueness Violation	Indicates that the unique values are not provided for this category. For example, if a value is already used in Qualitia and you try to use the same value in custom action it is reported as a uniqueness violation.
TestCase_Cannot_Execute	Indicates that the whole test case was aborted due to reasons such as incorrect application URL, incorrect environment selected, or invalid Javascript expression.
Expected Result Not Met	Indicates the result of a step is a defect when actual results and expected results are different.
Lack Of Permissions	Indicates that you do not have access to the environment or on application under test feature you are testing.

You can add custom failure categories to custom actions as per your requirement, refer to Adding Custom Categories in Custom Actions for Web and Mobile Projects.

To view a failure summary for a suite execution:

From the Suite of Suites screen, click a test suite execution for which you want to view the failure summary.
In the Test Suite Execution Summary Report, click the Failure Summary button.
A list of associated failure categories is displayed along with the number of failed steps.
Do the following in the order listed:
1. Click a failure category associated with the test suite.
  The failure category expands to show the list of associated test case iterations where failure has occurred.
2. Click a test case iteration.
  A list of applicable failure categories in that test case iteration is displayed.
3. Click a failure category associated with the selected test case iteration.
  The failure category expands to show the list of associated failed steps of the selected test case iteration.
  If the step is part of a task, then the task name and task iteration are also shown.
4. Click a failed step.
  The associated test case iteration detailed report opens and it highlights the selected step.
5. You can view the screenshot and info log for the failed step.

On This Page:

Browser not supported