Execute an Azure DevOps build or release pipeline by mapping it with an offline suite. Create an offline suite that includes the test cases relevant to the pipeline.

Pre-requisites:

Ensure that your computer:

• Contains the Azure DevOps self-hosted agent.
• Satisfies all the pre-requisites for offline suite executions. For more details, refer to Prerequisites for Offline Suite Execution. 
  
  Qualitia also recommends that you must know the fundamentals of offline suite executions except for the CLI commands. For more information, refer to Offline Package.

Implementing the Components in your Computer

Enabling your computer to execute an offline suite through Azure DevOps includes the following process:

Configuring the Offline Package File

To configure the Offline Package file:

1. Open the extracted Qualitia Offline folder.
2. Open the config folder.
3. Open the startupSettings file in a test editor, such as Notepad.
4. Set value of CI_Tool to Azure DevOps.

Mapping an Azure DevOps Pipeline with an Offline Suite

After creating a build or a release pipeline, map it with the corresponding offline suite. In the Azure DevOps pipeline, create two tasks that are used to:

• Invoke the Offline Package and the mapped offline suite as per their specified locations.
• Get the total number of test cases executed and the number of test cases executed for each execution status.

To map an Azure DevOps pipeline with an offline suite:

1. Open the pipeline in Azure DevOps.
2. Add two Command Line Prompt Utilities with names of your choice.
   Azure DevOps adds these two as tasks.
   To know how to add pipeline tasks, refer https://docs.microsoft.com/en-us/azure/devops/?view=azure-devops
3. Open the first task to enter the following information, in Script:
   1. Enter the location of the Java path enclosed with inverted commas.
   2. Add -jar separated by space next to Java path
   3. Add Qualitia offline suite jar path enclosed with inverted commas next to -jar.
      For example, "C:\Program Files\Java\jre1.8.0_261\bin\java.exe" -jar "D:\offliesuites\Qualitia Offline\qualitiaoffline-8.3.1.jar"
   4. Optionally, you can override the execution profile or the browser type:
      • To override the mapped execution profile, enter the location of the new execution profile next to the offline suite jar path.
        For example, enter "D:\offliesuites\Qualitia Offline\Edge.json" separated by space after the Qualitia offline suite jar path.
      • To override the browser type, enter BrowserName=GoogleChrome chromedriverpath=D:\chromedriver_win32\chromedriver.exe.
4. Expand Advanced, enter the location of offline suite enclosed with inverted commas, in Working Directory.
   For example, "D:\offliesuites\PaymentModule\Qualitia Offline".
5. Open the second task to enter the following information, in Script:
   @echo PassedCount = %PASSEDCOUNT%
   @echo FailedCount = %FAILEDCOUNT%
   @echo DefectCount = %DEFECTCOUNT%
   @echo NotExecutionCount = %NECOUNT%
   @echo TotalCount = %TOTALTCCOUNT%
   Note: You can specify the names of tasks, such as Qualitia Offline Suite Name and Suite Name Settings to identify the pipeline with the offline suite.

You can now execute the Azure DevOps pipeline.

Execution Guidelines and Best Practices

Qualitia recommends considering the following best practices to optimize the Azure DevOps pipeline executions that are mapped with offline suites: 

• If you want to push suite execution results to the Real Time Reporting portal, ensure that you specify the Qualitia Server URL and enable the Real Time Reporting checkbox in the mapped execution profile.
• Before executing a pipeline, ensure that the mapped execution profile of the corresponding offline suite is configured as per the requirements. For detailed instructions, refer to Configuring the Settings for an Offline Suite.
• You can execute a pipeline on different browsers, environments and other execution settings. Create different execution profiles in the QAS client and export them into the Offline Suite folder of your computer. For detailed information on execution profile, refer to Working with Execution Profiles.
• Whenever relevant test cases are updated or newly added relevant in the corresponding test suite, perform the following steps:
  • Create the offline suite again.
  • Save the offline suite in another location of your computer.
  • Open the Azure DevOps pipeline to replace the mapped offline suite.
• You can edit the test data for an offline suite to cover more test scenarios. For detailed instructions on editing test data for an offline suite, refer to Editing the Test Data for an Offline Suite.

Viewing the Suite Execution Reports

If you have configured the Qulaitia Server URL in the mapped execution profile, you can track the Azure DevOps offline execution in the Real Time Reporting Portal.

The suite execution result is displayed as Offline | Azure DevOps besides to the suite execution cart.

However, if you did not connect with Qualitia Server or the connection with Qualitia Server was lost during the pipeline execution, you can upload the locally saved suite execution report in the following ways:

• Uploading a Test Suite Execution Summary Report
• Uploading Multiple Suite Execution Reports to Real-Time Reporting Portal

Known Issues or Limitations

• You cannot have parallel executions of test cases.
• Does not send the suite execution results to Azure DevOps.