Configuring Azure DevOps for Test Executions

In order to be consistently test and build your code, Azure DevOps pipelines combine continuous integration (CI) and continuous delivery (CD) mechanism. You can execute Qualitia automated test cases as a part of your build process with few customizations in the Azure DevOps CI-CD pipeline.

Adding the Get Test Assembly Extension in Azure DevOps for Integration

Qualitia provisions to execute automated test cases from Qualitia mapped to Azure DevOps, as a part of your build and release pipelines. In order to do so, you first need to add Qualitia-Get Test Assembly extensions in Azure DevOps instance.

You can add these extensions in Azure DevOps by following the steps mentioned herein.

Adding and installing these components is a one-time activity and needs administrative privileges on Azure DevOps server. Once they are added, you may use it in configuring different multiple build and release pipelines accordingly. The installation can be performed by Azure DevOps Admins only. Normal users do not have the privileges to do the same.

Before initiating Azure DevOps integration with Qualitia, you must share Azure DevOps organization details with Qualitia. Qualitia will then share the extensions which will help you to integrate Azure DevOps projects to Qualitia projects.

After Qualitia shares these extensions, you can add and install these extensions to use them in the build and release pipeline. In case these extensions are not shared with your organization, you may contact Qualitia Support.

Once these tasks are shared by Qualitia in Azure DevOps instance, you may add, install, and use them in build and release pipelines.

To add the Get Test Assembly extensions into Azure DevOps instance:

Sign in to your Azure DevOps instance.
From the top right corner, navigate to Marketplace > Manage Extensions.
You will find Qualitia-Get Test Assembly extension shared by Qualitia.
Click the Qualitia-Get Test Assembly and install it.
Navigate back to Extensions > Manage Extensions.
Your Azure DevOps instance is now ready for Qualitia integration.

Configure Qualitia-Get Test Assembly Task

You must first configure your CI-CD pipeline accordingly to run test cases mapped to Azure DevOps test cases as a build process. Configuring CI-CD includes adding two custom tasks into the CI-CD pipeline and configure it for Qualitia automated test case execution. The section below details out steps to add and configure Qualitia-Get Test Assembly Task.

Adding Task in CI-CD Pipeline

Log on to your Azure DevOps instance.
Add new CI-CD pipeline or edit the existing pipeline.
To create new pipeline, select required repository or use the Select classic view option.
For more information on creating and configuring CI-CD pipelines, refer to the Azure DevOps documentation.
Click the + icon to add new task.
Search and select the Qualitia-Get Test Assembly and add it.
Set this task to execute prior to VS Test Task.
Your task is now ready for use in the CI-CD pipeline.

Configure Qualitia-Get Test Assembly Task

After you add it the custom task into the CI-CD pipeline, you must configure it adding some details.

Control Options
Display Name	Enter the desired display name for this task.
Qualitia Server IP	Enter the hostname or IP of the machine where Qualitia server is installed.
Qualitia Server Port	By default, this will use port 8887. However, you may change the port during Qualitia Server installation.
Path to save Qualitia Test Project	This is the local path on the DevOps agent machine where the Qualitia test assemblies are downloaded. These test assemblies contain unit test methods created by Qualitia during test case linking process to invoke the Qualitia remote agent from the automated unit test method. By default, this field comes pre-populated with $(System.DefaultWorkingDeirectory). This location need not be changed if the Qualitia tests are to be executed in release pipelines as well. It is recommended not to edit/update content from this field. For more information on updating value set under this variable, refer to Azure DevOps documentation.
Enabled	This shows the current state of this customized task. This option gives you control over this customized task. By default, this option is selected. Clear this option to disable a task. It is useful when you want to temporarily take task out of the process for testing or for specific deployments.
Continue on error	Select this option if you want to continue Qualitia test case execution even after getting any error in the pipeline. By default, this option is not enabled.
Timeout	Set the timeout for the task in minutes. Default is 0 which refers to infinite timeout. Value set here overrides the value set for the parent task job.
Run this task	Here you can set the condition to run this task. You can use one of the pre-designed options in the list or you can create new customized condition. You may refer to Azure DevOps documentation for more information on creating custom conditions as expressions.
Attach Zip to Test Results	Select this option to view test results consolidated as a ZIP file in Azure DevOps.
Qualitia Get Assembly Version Details	For Qualitia version 7.3.5 and above, select 2.

Configuring a Build Pipeline

Executing Qualitia test cases mapped to Azure DevOps tests in the build pipeline requires you to configure the build pipeline accordingly. Configuring a build pipeline includes adding a Visual Studio Test Task and configuring it for the build pipeline.

You must add and configure a visual studio test task to get all the required information from Qualitia and execute Qualitia test cases as a build pipeline.

To configure a Build Pipeline:

Launch Azure DevOps instance.
Edit existing build pipeline or create new build pipeline.
Add a Visual Studio Test Platform Installer.
Add a Get Assembly Task.
For more information, refer to Configure Qualitia-Get Test Assembly Task.
Add an Agent Job.
(Optional) To execute test cases in parallel, click Execution> Parallelism, click the Multi-Agent button, and enter the number of Azure DevOps agents that you want to use, in Number of agents.

Qualitia recommends not to configure more than 5 Azure DevOps agents on a system for parallel execution.
Ensure that the number of Azure DevOps agents must be same and running on the system where you want to execute.
Add new Visual Studio Test Task and fill up the details as mentioned in the table below for the Test Selection section:

Test Selection
Display Name	Enter the desired display name.
Select tests using	Select Test Plan option to execute tests from the test plan which have Qualitia automated test cases attached to it. Qualitia supports this option only.
Test Plan	Select a test plan containing test suites with Qualitia automated test cases.
Test Suite	Select test suites containing Qualitia automated test cases. You can select more than one suites here.
Test Configuration	Select test configuration.
Search folder	This is the directory system will search for the test assemblies. This field comes pre-populated and should be same as the Path to save Qualitia Test Project settings set under Qualitia-Get Test Assembly task. For more information on updating value set under this variable, refer to Azure DevOps documentation.
Test mix contains UI tests	This option enables to run UI tests. This option has no significance with reference to Qualitia automated test cases and is deselected by default. For more information on configuring agent to run in interactive mode, refer to the Azure DevOps documentation.

For executing Qualitia automated test cases as a build pipeline, it is mandatory to fill up the information for these fields as mentioned in the table above. You may fill up the information in other sections like Execution options, Reporting options, and Control options, and so forth based on the project and build pipelines requirements.

Configuring a Release Pipeline

Executing the Qualitia test cases mapped to Azure DevOps tests in the release pipeline requires you to configure the release pipeline accordingly. You must add and configure a visual studio test task to get all the required information from Qualitia and execute Qualitia test cases as a release pipeline.

To configure a Release Pipeline:

Launch Azure DevOps instance.
Edit existing release pipeline or create new release pipeline.
Create an Agent Job.
(Optional) To execute test cases in parallel, click Execution> Parallelism, click the Multi-Agent button, and enter the number of Azure DevOps agents that you want to use, in Number of agents.

Qualitia recommends not to configure more than 5 Azure DevOps agents on a system for parallel execution.
Ensure that the number of Azure DevOps agents must be same and running on the system where you want to execute .
Add a Visual Studio Test Platform Installer.
Add a Get Assembly Task.
For more information, refer to Configure Qualitia-Get Test Assembly Task.
Add a new Visual Studio Test Task and fill up the details as mentioned in the table below for the Test Selection section:

Test Selection
Display Name	Enter the desired display name.
Select tests using	Select the method using which you want to select Qualitia automated test cases for execution: Test plan: Use this option to execute tests from the test plans which have Qualitia automated test cases attached to it. (recommended) Test run: Use this option when you are setting up an environment to run tests from test plans.
Test Plan	Select a test plan containing test suites with Qualitia automated test cases
Test Suite	Select test suites containing Qualitia automated test cases. You can select more than one suites here.
Test Configuration	Select test configuration.
Search folder	This is the directory system will search for the test assemblies. This field comes pre-populated. For more information on updating value set under this variable, refer to Azure DevOps documentation.
Test mix contains UI tests	This option enables to run UI tests. This option has no significance with reference to Qualitia automated test cases and is deselected by default. For more information on configuring agent to run in interactive mode, refer to the Azure DevOps documentation.

For executing Qualitia automated test cases as a release pipeline, it is mandatory to fill up the information for these fields as mentioned in the table above. You may fill up the information in other sections like Execution options, Reporting options, and Control options, and so forth based on the project and release pipelines requirements.

Configuring the On Demand Test Runs

Using Azure DevOps, you can execute Qualitia automated tests in three different methods. Configuring the build and release pipelines help you to run Qualitia automated test cases as a part of build and release pipelines.

Additionally, you can run Qualitia automated test cases on demand from the test run section. For executing Qualitia automated tests on-demand, you must first configure a visual studio task under release pipeline accordingly under the test run section.

To Configure the On Demand Test Runs:

Log on to your Azure DevOps instance.
From the top menu items, select Build and Release.
Under the Releases section, navigate to Tasks.
Add a Visual Studio Test Platform Installer.
Add a Get Assembly Task.
For more information, refer to Configure Qualitia-Get Test Assembly Task.
(Optional) To execute test cases in parallel, click Execution> Parallelism, click the Multi-Agent button, and enter the number of Azure DevOps agents that you want to use, in Number of agents.

Qualitia recommends not to configure more than 5 Azure DevOps agents on a system for parallel execution.
Ensure that the number of Azure DevOps agents must be same and running on the system where you want to execute.
Add a new Visual Studio task and configure it as mentioned below:
1. Enter the desired Display name.
2. Under the Select test using section, select Test Run.
3. Leave the Test Run and Search folder options with their default values.

Updating Test Execution Properties Run-time

Qualitia gives you the provision to override the basic set of configurations provided under Qualitia Remote Agent. This gives you the flexibility to change the configurations settings for executing test cases at run-time based on the release/build requirements without updating configuration settings under Qualitia Remote Agent every time before you run the build or release pipelines.

In order to do so, you must configure the Visual Studio Test task under release and build pipelines accordingly.

Important Note

By default, Azure DevOps considers settings provided in the .runsettings file while test case execution. It is important to have this file in your source code of the application under test so that Azure DevOps can access and use settings provided in this file while executing Qualitia automated test cases.

Copy the content provided in the box below, add it in an empty file.
Save this file in the source code directory with .runsettings extension.

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<TestRunParameters>
    <Parameter name="QualitiaRuntimeSettings" value="" />
    <Parameter name="QualitiaEnvironmentSettings" value="" />
</TestRunParameters>
</RunSettings>

The same file can also be used to update the environment variables runtime which were created and used while developing test cases in Qualitia automation studio.

Add the desired values in key=value pairs against QualitiaEnvironmentSettings in the .runtimesettings file in the following format and you are good to go.

Configuring Visual Studio Test Task

You must configure Execution options in the visual studio test task under build and release pipelines to be able to update execution properties run-time.

To configure Execution options under Visual Studio Test task:

Launch Azure DevOps instance.
Edit existing or create new build/release pipeline.
Click the Visual Studio Test task and its details will appear in the right pane.
In the Execution Options section, enter the detail as explained in the table below:

Execution Options

Settings File

Select the .runsettings file you added in the source code directory. You can select this file using the browse button.

Override test run parameters

Enter the parameters you want to override from the configuration settings provided under Qualitia remote agent. Parameters should be given in the key=value pairs enclosed in the double inverted comma in the following format. Multiple parameters should be given in separated by a semicolon (;). You can give as many parameters as you want using this method.

-QualitiaRuntimeSettings "BrowserName=googlechrome;chromedriverpath=C:\Program Files (x86)\Common Files\Qualitia\WD\QualitiaIntegrationRemoteAgent\WebDriver\Selenium\chromedriver.exe"
-QualitiaEnvironmentSettings "AppUrl=http://www.dev.google.com;username=devuser01;"

Alternatively, you can perform through Azure DevOps by performing the following steps:

Define a Pipeline variable as URL along with its value. If this variable already exists, then it is overridden.
In the Pipelines, open the VsTest - testAssemblies, in the Override Test run parameters, enter the following details:
1. Name as QualitiaEnvironmentSettings
2. Value as OpenURL=$(URL), in which OpenURL is the Qualitia Environment variable and URL is the Azure DevOps variable.

You can leave all the other fields to their default options or you may configure it based on the release or build pipeline requirements.