In order to be consistently test and build your code, Azure DevOps pipelines combine continuous integration (CI) and continuous delivery (CD) mechanism. You can execute Qualitia automated test cases as a part of your build process with few customizations in the Azure DevOps CI-CD pipeline.
Adding the Get Test Assembly Extension in Azure DevOps for Integration
Qualitia provisions to execute automated test cases from Qualitia mapped to Azure DevOps, as a part of your build and release pipelines. In order to do so, you first need to add Qualitia-Get Test Assembly extensions in Azure DevOps instance.
You can add these extensions in Azure DevOps by following the steps mentioned herein.
Before initiating Azure DevOps integration with Qualitia, you must share Azure DevOps organization details with Qualitia. Qualitia will then share the extensions which will help you to integrate Azure DevOps projects to Qualitia projects.
After Qualitia shares these extensions, you can add and install these extensions to use them in the build and release pipeline. In case these extensions are not shared with your organization, you may contact Qualitia Support.
Once these tasks are shared by Qualitia in Azure DevOps instance, you may add, install, and use them in build and release pipelines.
To add the Get Test Assembly extensions into Azure DevOps instance:
- Sign in to your Azure DevOps instance.
- From the top right corner, navigate to Marketplace > Manage Extensions.
You will find Qualitia-Get Test Assembly extension shared by Qualitia. - Click the Qualitia-Get Test Assembly and install it.
- Navigate back to Extensions > Manage Extensions.
Your Azure DevOps instance is now ready for Qualitia integration.
Configure Qualitia-Get Test Assembly Task
You must first configure your CI-CD pipeline accordingly to run test cases mapped to Azure DevOps test cases as a build process. Configuring CI-CD includes adding two custom tasks into the CI-CD pipeline and configure it for Qualitia automated test case execution. The section below details out steps to add and configure Qualitia-Get Test Assembly Task.
Adding Task in CI-CD Pipeline
- Log on to your Azure DevOps instance.
- Add new CI-CD pipeline or edit the existing pipeline.
To create new pipeline, select required repository or use the Select classic view option.
For more information on creating and configuring CI-CD pipelines, refer to the Azure DevOps documentation. - Click the + icon to add new task.
- Search and select the Qualitia-Get Test Assembly and add it.
- Set this task to execute prior to VS Test Task.
Your task is now ready for use in the CI-CD pipeline.
Configure Qualitia-Get Test Assembly Task
After you add it the custom task into the CI-CD pipeline, you must configure it adding some details.
| Display Name | Enter the desired display name for this task. |
|---|---|
| Qualitia Server IP | Enter the hostname or IP of the machine where Qualitia server is installed. |
| Qualitia Server Port | By default, this will use port 8887. |
| Path to save Qualitia Test Project | This is the local path on the DevOps agent machine where the Qualitia test assemblies are downloaded. These test assemblies contain unit test methods created by Qualitia during test case linking process to invoke the Qualitia remote agent from the automated unit test method. By default, this field comes pre-populated with $(System.DefaultWorkingDeirectory). This location need not be changed if the Qualitia tests are to be executed in release pipelines as well. It is recommended not to edit/update content from this field. For more information on updating value set under this variable, refer to Azure DevOps documentation. |
| Control Options | |
| Enabled | This shows the current state of this customized task. This option gives you control over this customized task. By default, this option is selected. Clear this option to disable a task. It is useful when you want to temporarily take task out of the process for testing or for specific deployments. |
| Continue on error | Select this option if you want to continue Qualitia test case execution even after getting any error in the pipeline. By default, this option is not enabled. |
| Timeout | Set the timeout for the task in minutes. Default is 0 which refers to infinite timeout. Value set here overrides the value set for the parent task job. |
| Run this task | Here you can set the condition to run this task. You can use one of the pre-designed options in the list or you can create new customized condition. You may refer to Azure DevOps documentation for more information on creating custom conditions as expressions. |
| Attach Zip to Test Results | Select this option to view test results consolidated as a ZIP file in Azure DevOps. |
| Qualitia Get Assembly Version Details | For Qualitia version 7.3.5 and above, select 2. |
Configuring a Build Pipeline
Executing Qualitia test cases mapped to Azure DevOps tests in the build pipeline requires you to configure the build pipeline accordingly. Configuring a build pipeline includes adding a Visual Studio Test Task and configuring it for the build pipeline.
You must add and configure a visual studio test task to get all the required information from Qualitia and execute Qualitia test cases as a build pipeline.
To configure a Build Pipeline:
- Launch Azure DevOps instance.
- Edit existing build pipeline or create new build pipeline.
- Add a Visual Studio Test Platform Installer.
- Add a Get Assembly Task.
For more information, refer to Configure Qualitia-Get Test Assembly Task. - Add an Agent Job.
(Optional) To execute test cases in parallel, click Execution> Parallelism, click the Multi-Agent button, and enter the number of Azure DevOps agents that you want to use, in Number of agents.
Qualitia recommends not to configure more than 5 Azure DevOps agents on a system for parallel execution.
Ensure that the number of Azure DevOps agents must be same and running on the system where you want to execute.
- Add new Visual Studio Test Task and fill up the details as mentioned in the table below for the Test Selection section:
| Display Name | Enter the desired display name. |
|---|---|
| Test Selection | |
| Select tests using | Select Test Plan option to execute tests from the test plan which have Qualitia automated test cases attached to it. Qualitia supports this option only. |
| Test Plan | Select a test plan containing test suites with Qualitia automated test cases. |
| Test Suite | Select test suites containing Qualitia automated test cases. You can select more than one suites here. |
| Test Configuration | Select test configuration. |
| Search folder | This is the directory system will search for the test assemblies. This field comes pre-populated and should be same as the Path to save Qualitia Test Project settings set under Qualitia-Get Test Assembly task. For more information on updating value set under this variable, refer to Azure DevOps documentation. |
| Test mix contains UI tests | This option enables to run UI tests. This option has no significance with reference to Qualitia automated test cases and is deselected by default. For more information on configuring agent to run in interactive mode, refer to the Azure DevOps documentation. |
For executing Qualitia automated test cases as a build pipeline, it is mandatory to fill up the information for these fields as mentioned in the table above. You may fill up the information in other sections like Execution options, Reporting options, and Control options, and so forth based on the project and build pipelines requirements.
Configuring a Release Pipeline
Executing the Qualitia test cases mapped to Azure DevOps tests in the release pipeline requires you to configure the release pipeline accordingly. You must add and configure a visual studio test task to get all the required information from Qualitia and execute Qualitia test cases as a release pipeline.
To configure a Release Pipeline:
- Launch Azure DevOps instance.
- Edit existing release pipeline or create new release pipeline.
- Create an Agent Job.
(Optional) To execute test cases in parallel, click Execution> Parallelism, click the Multi-Agent button, and enter the number of Azure DevOps agents that you want to use, in Number of agents.
Qualitia recommends not to configure more than 5 Azure DevOps agents on a system for parallel execution.
Ensure that the number of Azure DevOps agents must be same and running on the system where you want to execute .
- Add a Visual Studio Test Platform Installer.
- Add a Get Assembly Task.
For more information, refer to Configure Qualitia-Get Test Assembly Task. - Add a new Visual Studio Test Task and fill up the details as mentioned in the table below for the Test Selection section:
| Display Name | Enter the desired display name. |
|---|---|
| Test Selection | |
| Select tests using | Select the method using which you want to select Qualitia automated test cases for execution:
|
| Test Plan | Select a test plan containing test suites with Qualitia automated test cases |
| Test Suite | Select test suites containing Qualitia automated test cases. You can select more than one suites here. |
| Test Configuration | Select test configuration. |
| Search folder | This is the directory system will search for the test assemblies. This field comes pre-populated. For more information on updating value set under this variable, refer to Azure DevOps documentation. |
| Test mix contains UI tests | This option enables to run UI tests. This option has no significance with reference to Qualitia automated test cases and is deselected by default. For more information on configuring agent to run in interactive mode, refer to the Azure DevOps documentation. |
For executing Qualitia automated test cases as a release pipeline, it is mandatory to fill up the information for these fields as mentioned in the table above. You may fill up the information in other sections like Execution options, Reporting options, and Control options, and so forth based on the project and release pipelines requirements.
Configuring the On Demand Test Runs
Using Azure DevOps, you can execute Qualitia automated tests in three different methods. Configuring the build and release pipelines help you to run Qualitia automated test cases as a part of build and release pipelines.
Additionally, you can run Qualitia automated test cases on demand from the test run section. For executing Qualitia automated tests on-demand, you must first configure a visual studio task under release pipeline accordingly under the test run section.
To Configure the On Demand Test Runs:
- Log on to your Azure DevOps instance.
- From the top menu items, select Build and Release.
- Under the Releases section, navigate to Tasks.
- Add a Visual Studio Test Platform Installer.
- Add a Get Assembly Task.
For more information, refer to Configure Qualitia-Get Test Assembly Task. (Optional) To execute test cases in parallel, click Execution> Parallelism, click the Multi-Agent button, and enter the number of Azure DevOps agents that you want to use, in Number of agents.
Qualitia recommends not to configure more than 5 Azure DevOps agents on a system for parallel execution.
Ensure that the number of Azure DevOps agents must be same and running on the system where you want to execute.
- Add a new Visual Studio task and configure it as mentioned below:
- Enter the desired Display name.
- Under the Select test using section, select Test Run.
- Leave the Test Run and Search folder options with their default values.
Updating Test Execution Properties Run-time
Qualitia gives you the provision to override the basic set of configurations provided under Qualitia Remote Agent. This gives you the flexibility to change the configurations settings for executing test cases at run-time based on the release/build requirements without updating configuration settings under Qualitia Remote Agent every time before you run the build or release pipelines.
In order to do so, you must configure the Visual Studio Test task under release and build pipelines accordingly.
Important Note
By default, Azure DevOps considers settings provided in the .runsettings file while test case execution. It is important to have this file in your source code of the application under test so that Azure DevOps can access and use settings provided in this file while executing Qualitia automated test cases.
- Copy the content provided in the box below, add it in an empty file.
- Save this file in the source code directory with .runsettings extension.
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<TestRunParameters>
<Parameter name="QualitiaRuntimeSettings" value="" />
<Parameter name="QualitiaEnvironmentSettings" value="" />
</TestRunParameters>
</RunSettings>
The same file can also be used to update the environment variables runtime which were created and used while developing test cases in Qualitia automation studio.
Add the desired values in key=value pairs against QualitiaEnvironmentSettings in the .runtimesettings file in the following format and you are good to go.
Configuring Visual Studio Test Task
You must configure Execution options in the visual studio test task under build and release pipelines to be able to update execution properties run-time.
To configure Execution options under Visual Studio Test task:
- Launch Azure DevOps instance.
- Edit existing or create new build/release pipeline.
- Click the Visual Studio Test task and its details will appear in the right pane.
- In the Execution Options section, enter the detail as explained in the table below:
| Execution Options | |
|---|---|
| Settings File | Select the .runsettings file you added in the source code directory. You can select this file using the browse button. |
| Override test run parameters | Enter the parameters you want to override from the configuration settings provided under Qualitia remote agent. Parameters should be given in the key=value pairs enclosed in the double inverted comma in the following format. Multiple parameters should be given in separated by a semicolon (;). You can give as many parameters as you want using this method. -QualitiaRuntimeSettings "BrowserName=googlechrome;chromedriverpath=C:\Program Files (x86)\Common Files\Qualitia\WD\QualitiaIntegrationRemoteAgent\WebDriver\Selenium\chromedriver.exe" -QualitiaEnvironmentSettings "AppUrl=http://www.dev.google.com;username=devuser01;" Alternatively, you can perform through Azure DevOps by performing the following steps:
|
You can leave all the other fields to their default options or you may configure it based on the release or build pipeline requirements.
