In order to be consistently test and build your code, Azure DevOps pipelines combine continuous integration (CI) and continuous delivery (CD) mechanism. You can execute Qualitia automated test cases as a part of your build process with few customizations in the Azure DevOps CI-CD pipeline.
Adding the Get Test Assembly Extension in Azure DevOps for Integration
Qualitia provisions to execute automated test cases from Qualitia mapped to Azure DevOps, as a part of your build and release pipelines. In order to do so, you first need to add Qualitia-Get Test Assembly extensions in Azure DevOps instance.
...
- Sign in to your Azure DevOps instance.
- From the top right corner, navigate to Marketplace > Manage Extensions.
You will find Qualitia-Get Test Assembly extension shared by Qualitia. - Click the Qualitia-Get Test Assembly and install it.
- Navigate back to Extensions > Manage Extensions.
Your Azure DevOps instance is now ready for Qualitia integration.
Configure Qualitia-Get Test Assembly Task
| Table of Contents |
|---|
You must first configure your CI-CD pipeline accordingly to run test cases mapped to Azure DevOps test cases as a build process. Configuring CI-CD includes adding two custom tasks into the CI-CD pipeline and configure it for Qualitia automated test case execution. The section below details out steps to add and configure Qualitia-Get Test Assembly Task.
Adding Task in CI-CD Pipeline
- Log on to your Azure DevOps instance.
- Add new CI-CD pipeline or edit the existing pipeline.
To create new pipeline, select required repository or use the Select classic view option.
For more information on creating and configuring CI-CD pipelines, refer to the Azure DevOps documentation. - Click the + icon to add new task.
- Search and select the Qualitia-Get Test Assembly and add it.
- Set this task to execute prior to VS Test Task.
Your task is now ready for use in the CI-CD pipeline.
Configure Qualitia-Get Test Assembly Task
After you add it the custom task into the CI-CD pipeline, you must configure it adding some details.
| Display Name | Enter the desired display name for this task. |
|---|---|
| Qualitia Server IP | Enter the hostname or IP of the machine where Qualitia server is installed. |
| Qualitia Server Port | By default, this will use port 8887. |
| Path to save Qualitia Test Project | This is the local path on the DevOps agent machine where the Qualitia test assemblies are downloaded. These test assemblies contain unit test methods created by Qualitia during test case linking process to invoke the Qualitia remote agent from the automated unit test method. By default, this field comes pre-populated with $(System.DefaultWorkingDeirectory). This location need not be changed if the Qualitia tests are to be executed in release pipelines as well. It is recommended not to edit/update content from this field. For more information on updating value set under this variable, refer to Azure DevOps documentation. |
| Control Options | |
| Enabled | This shows the current state of this customized task. This option gives you control over this customized task. By default, this option is selected. Clear this option to disable a task. It is useful when you want to temporarily take task out of the process for testing or for specific deployments. |
| Continue on error | Select this option if you want to continue Qualitia test case execution even after getting any error in the pipeline. By default, this option is not enabled. |
| Timeout | Set the timeout for the task in minutes. Default is 0 which refers to infinite timeout. Value set here overrides the value set for the parent task job. |
| Run this task | Here you can set the condition to run this task. You can use one of the pre-designed options in the list or you can create new customized condition. You may refer to Azure DevOps documentation for more information on creating custom conditions as expressions. |
| Attach Zip to Test Results | Select this option to view test results consolidated as a ZIP file in Azure DevOps. |
| Qualitia Get Assembly Version Details | For Qualitia version 7.3.5 and above, select 2. |
Configuring a Build Pipeline
Executing Qualitia test cases mapped to Azure DevOps tests in the build pipeline requires you to configure the build pipeline accordingly. Configuring a build pipeline includes adding a Visual Studio Test Task and configuring it for the build pipeline.
...
| Note |
|---|
For executing Qualitia automated test cases as a build pipeline, it is mandatory to fill up the information for these fields as mentioned in the table above. You may fill up the information in other sections like Execution options, Reporting options, and Control options, and so forth based on the project and build pipelines requirements. |
Configuring a Release Pipeline
Executing the Qualitia test cases mapped to Azure DevOps tests in the release pipeline requires you to configure the release pipeline accordingly. You must add and configure a visual studio test task to get all the required information from Qualitia and execute Qualitia test cases as a release pipeline.
...
| Note |
|---|
For executing Qualitia automated test cases as a release pipeline, it is mandatory to fill up the information for these fields as mentioned in the table above. You may fill up the information in other sections like Execution options, Reporting options, and Control options, and so forth based on the project and release pipelines requirements. |
Configuring the On Demand Test Runs
Using Azure DevOps, you can execute Qualitia automated tests in three different methods. Configuring the build and release pipelines help you to run Qualitia automated test cases as a part of build and release pipelines.
...
- Log on to your Azure DevOps instance.
- From the top menu items, select Build and Release.
- Under the Releases section, navigate to Tasks.
- Add a Visual Studio Test Platform Installer.
- Add a Get Assembly Task.
For more information, refer to Configure Qualitia-Get Test Assembly Task. (Optional) To execute test cases in parallel, click Execution> Parallelism, click the Multi-Agent button, and enter the number of Azure DevOps agents that you want to use, in Number of agents.
Info Qualitia recommends not to configure more than 5 Azure DevOps agents on a system for parallel execution.
Ensure that the number of Azure DevOps agents must be same and running on the system where you want to execute.
- Add a new Visual Studio task and configure it as mentioned below:
- Enter the desired Display name.
- Under the Select test using section, select Test Run.
- Leave the Test Run and Search folder options with their default values.
Updating Test Execution Properties Run-time
Qualitia gives you the provision to override the basic set of configurations provided under Qualitia Remote Agent. This gives you the flexibility to change the configurations settings for executing test cases at run-time based on the release/build requirements without updating configuration settings under Qualitia Remote Agent every time before you run the build or release pipelines.
...
Add the desired values in key=value pairs against QualitiaEnvironmentSettings in the .runtimesettings file in the following format and you are good to go.
Configuring Visual Studio Test Task
You must configure Execution options in the visual studio test task under build and release pipelines to be able to update execution properties run-time.
...