Azure DevOps Integration

Visual Studio, a product from Microsoft, helps product teams with tools and technologies in all the stages of application development. Qualitia Automation Studio (QAS) helps replace the manual testing procedures in the application development processes by tightly integrating with Azure DevOps. This makes the application development process robust and simple.

Qualitia's integration to Azure DevOps lets organizations leverage their existing investments as part of an integrated platform for automation testing development and allows managing automated test cases prepared by Qualitia inside Azure DevOps.

Key Benefits of QAS and Azure DevOps Integration:

Seamless Synchronization: Effortlessly synchronize updated and new test cases, even when a pipeline involves several test cases.
Parallel Test Execution: Supports parallel execution of test cases, reducing pipeline execution time.
Result Visibility: View pipeline execution results from both Azure DevOps server and within QAS.

Features:

With Azure DevOps advanced integration, you can:

Map QAS Project to Azure DevOps Project.
Map automated QAS test case to Azure DevOps test case.
Execute QAS test cases locally in Azure DevOps (using self-hosted agents).
Run QAS test cases as part of Azure DevOps build pipelines

To integrate QAS with Azure DevOps, you need to first set up the environment.

Azure DevOps Integration

Let’s set up the environment, to get started with the Azure DevOps Integration. This section depicts information related to all the prerequisites and procedures to get prepared for QAS's integration to Azure DevOps.

This section describes how Azure DevOps interacts with QAS components to execute QAS test cases successfully.

Qualitia Components:

Qualitia Server
Qualitia Client

Azure DevOps Components:

Azure DevOps as the Cloud Version
Azure DevOps Self Hosted Agent

To integrate QAS with Azure DevOps, follow these three key steps:

Association Between Qualitia Automation Studio and Azure DevOps

This step involves initiating the association from within Qualitia Automation Studio (QAS), mapping projects and test cases between QAS and Azure DevOps.

Key Activities:

Map QAS Project to Azure DevOps Project: Establish a link between your project in QAS and the corresponding project in Azure DevOps.
Map Automated Test Cases: Link QAS automated test cases with Azure DevOps manual test cases, ensuring that test cases are synchronized across both platforms.
Associate Test Suites: Map Qualitia Suites to Azure DevOps Test Suites to group relevant test cases for streamlined execution.

Execution Setup in Azure DevOps (One-Time Setup)

This one-time setup ensures that QAS test cases can be executed directly from Azure DevOps pipelines.

Key Activities:

Setup Build/Release Pipelines: Create and configure build or release pipelines in Azure DevOps to execute the associated QAS test cases.
Task Extension for Custom Tasks: Install the Qualitia Task Extension, which allows custom Qualitia tasks to be included in the pipeline installation. (Task Extension is used for Qualitia Custom tasks for pipeline installation)
Configure Self-Hosted Agents: Set up a self-hosted Azure DevOps agent for executing the QAS test cases locally.
Agent Pool Setup: Create and configure an agent pool in Azure DevOps to manage agents responsible for running QAS executions.

Result Publishing and Execution

This step ensures that the results of the executed test cases are published and available both in Azure DevOps and within QAS.

Key Activities:

Publish Results in Azure DevOps: After each pipeline execution, results of the QAS test cases are published to Azure DevOps test cases and test runs.
Result Sync with QAS: Test case results are synchronized with Qualitia Automation Studio to provide a holistic view of the automation results within QAS.

The following table explains how different components of Azure DevOps and QAS interact with each other.

Component	Roles Played in Different Stages
Azure DevOps	Association Responds to QAS server using standard out of the box APIs offered by Azure DevOps. Execution Configuration of build/release pipelines. Agent pool configurations. Initiate pipeline execution. Result Publish Host results of pipeline execution.
Azure DevOps Agent	Association None Execution Executes the jobs in the pipeline along with the contained tasks. Select Test Plan and Test Suite that contain the automated test cases and suites within the Azure DevOps projects. Result Publish Execution of the Publish Result task of QAS to publish results to QAS Server and Azure DevOps Test Runs. Uses QAS Server APIs to publish result.
Qualitia Server	Association Interacts with the Azure DevOps server to update automation status of Azure DevOps test cases. Result Publish Receives test case results directly from QAS - Client with the help of QAS APIs.
Qualitia Client	Association Allows the authorized users to view Azure DevOps projects and associated projects and test cases. Execution Publishes the test cases to QAS project repository. The published test cases are used in the Azure DevOps pipeline execution. Result Publish Shows the status of test cases executed through Azure DevOps pipeline in Real Time Reporting Portal.

Prerequisites

Prerequisites for Azure DevOps

Azure DevOps Version

Azure DevOps (Cloud Version)
Azure DevOps Server 2019

Qualitia Plugins

Share Azure DevOps URL to Qualitia team to get the Get Assembly task plugin.

Roles and Privileges

Azure DevOps Administrator must install the plugins from Shared Extensions.
Qualitia does not recommend this option for PAT generation as it changes the access levels for all the users in the group.

Note: Please connect with your Azure DevOps Admin team to discuss the PAT generation and access level management.

Qualitia integration with Azure DevOps expects that the user has “Update Work Items” access that is used to generate the PAT (Personal Access Token).

The PAT with required access can be procured in one of the following ways:

Generate the PAT with a user that is part of the Project Administrators group.
Qualitia recommends this option for generating a PAT.
Allow Bypass Rules on Work Items Updates
Perform the following steps:
- In Azure DevOps project, ensure that the user that is used to generate PAT is member Contributors group / Project Team members group.
- Click Project Settings > Permissions.
- Select the user group that has the user to whom you want to provide access (Contributors/Project Team members).
- From the Bypass rules on work items updates list, select Allow.
  Refer to the following screenshot.

Have project access to create the following:
- A new pipeline (Build/Release)
- An Agent Pool

Prerequisites for Qualitia

Roles and Privileges	You must have administrative privileges to access Qualitia Project that you want to integrate with Azure DevOps tests.
Network Access	Ensure that the Qualitia Server URL is reachable from the Qualitia Client, Azure DevOps Server, and Azure DevOps agent.

Configuring Azure DevOps for Test Executions

To execute automated test cases from Qualitia mapped to Azure DevOps as part of your build and release pipelines, you need to add the Qualitia - Test Execution extensions to your Azure DevOps instance. Follow these steps:

Share Azure DevOps Organization Details: Before integrating Azure DevOps with Qualitia, provide your Azure DevOps organization details to Qualitia. They will then share the necessary extensions for integration with your Azure DevOps projects.
Add and Install Extensions: Once Qualitia shares the extensions, you can add and install them in your Azure DevOps instance.
Installation Steps:
- Sign into Azure DevOps instance.
- Click the top-right corner to navigate to Marketplace > Manage Extensions.
- Locate the Qualitia - Test Execution extension shared by Qualitia.
- Click the Qualitia - Test Execution and install it.
Verify Installation: Return to Extensions > Manage Extensions to confirm that the installation was successful.

Your Azure DevOps instance is now configured for Qualitia integration and ready to use in build and release pipelines.

Configure Qualitia - Test Execution Task

Add Task in CI-CD Pipeline

Log into Azure DevOps instance.
Add new CI-CD pipeline or edit the existing pipeline.
To create new pipeline, select required repository or use the Select classic view option.
Click the + icon to add new task.
Search and select Qualitia - Test Execution and add it.

After you add the custom task into the CI-CD pipeline, you must configure it by adding the following details:

Display Name	Enter the desired display name for this task.
Qualitia Option
Enter Qualitia Server URL	Enter the hostname or IP of the machine where QAS server is installed and enter server port number, by default it will use port 8887. However, you may change the port during QAS - Server installation. Eg. http://127.0.0.1:8887
Enter Qualitia Project Name	Name of QAS project that is integrated with Azure DevOps.
Execution Parameter	Specify QAS execution parameter such as non-default browser to use will execution and override environment variable. By default, None is selected.
Test Selection
Test Plan	Select a test plan containing test suite with QAS automated test cases.
Test Suite	Select test suite containing QAS automated test cases. You can select more than one suite here.
Execution Option
Distribution mode	Batch test across agents: Distribute test case across agent in define number of tests per batch. Replicate all tests across all agents: Execute all test cases on all agent configured.
Control Option
Enabled	This shows the current state of the customized task, and gives you control over the customized task. By default, this option is selected. Clear this option to disable a task. It is useful when you want to temporarily take task out of the process for testing or for specific deployments.
Continue on error	Select this option if you want to continue QAS test case execution even after getting errors in the pipeline. By default, this option is not enabled.
Timeout	Set the timeout for the task in minutes. Default is 0, which refers to infinite timeout. Value set here overrides the value set for the parent task job.
Run this task	Here you can set the condition to run a task. You can use one of the pre-designed options in the list or you can create new customized condition. You may refer to Azure DevOps documentation for more information on creating custom conditions as expressions.

You need to enable Allow scripts to access the OAuth token from Agent job.

Connecting Qualitia Artifacts to Azure DevOps

After you set up the environment in Azure DevOps for automating tests using Qualitia, it is now time to map your Qualitia artifacts to Azure DevOps where its starts with mapping a Qualitia project to Azure DevOps project.

Qualitia Test Case Lifecycle for Azure DevOps Integration

During the integration process with Azure DevOps, the Qualitia test case progresses through various stages, impacting the status of the associated Azure DevOps manual test case. Below is a description of each stage and its implications:

The following diagram shows the Integration Lifecycle status for a test case.

The following sections explain the different stages of a Qualitia test case and the impact of the same on associated Azure DevOps manual test case.

Test Case Creation

When a new test case is created in Qualitia under an Azure DevOps-integrated project, it remains unassociated with any Azure DevOps manual test case. The status of the Qualitia test case is marked as Not Linked.

Test Case Linking with Azure DevOps Test Case

The association between Qualitia test case and Azure DevOps test case is established on linking a test case with Azure DevOps test case. Refer to the following link for more details on how to link a test case with Azure DevOps manual test case.

Linking Qualitia Test cases to Azure DevOps Tests
Qualitia Test Case Status Change:
Not Linked → Linked/Not Published
Azure DevOps Test Case Automation Status Change:
Not Automated → Planned

Publishing Test Case

Once the Automation Engineer completes the test case in Qualitia and is ready for execution in Azure DevOps, the test case must be published.

Qualitia Test Case Status Change:
Linked/Not Published → Linked/Published
Azure DevOps Test Case Automation Status Change:
Planned → Automated

Editing a Test Case after Publishing

If changes are made to the Qualitia test case after publishing, these changes do not reflect in Azure DevOps. The published version is used for test execution.

Qualitia Test Case Status Change:
Linked/Published → Linked/Changes Not Published
Azure DevOps Test Case Automation Status:
Automated → Automated

Test Case Unlinking with Azure DevOps Test Case

The link between the Qualitia test case and the Azure DevOps test case can be removed at any time by unlinking the test case.

Qualitia Test Case Status Change:
Linked/Published or Changes Not Published → Not Linked
Azure DevOps Test Case Automation Status Change:
Automated → Not Automated

Linking a QAS Project to an Azure DevOps Project

To link a Qualitia project to an Azure DevOps project:

Log into Qualitia as a project administrator.
From the Expand Menu, click Project Management.
In the left pane, click a project that you want to link.
In the main pane, next to Integrations (ADO), click on link icon.
In the Azure DevOps tab, enter the following details of Azure DevOps.
1. Server URL: Enter the Azure DevOps URL.
2. PAT Token: Enter Personal Access Token (PAT) for authentication.
3. Connect: Click to connect to Azure DevOps server.
4. Azure DevOps Project: Select an Azure DevOps project from the drop-down list. User can select multiple projects.
5. Click Integrate.

Linking a Qualitia Test Case to an Azure DevOps Test

You can link a test case either from the Test Case Editor screen or from Ready for Execution screen.

To link a test case from the Test Case Editor screen:

From the Expand Menu, click Develop.
From the left pane, select a scenario, and open the test case.
Click the Link button

To link a test case from the Ready for Execution screen:

From the Expand Menu, click Execute.
Click the Ready for Execution tab.
A list of test cases is displayed in Test Case tab

Update the following details The below screen will open in both the cases:

🔧 Got a problem? Let’s fix!

Mobile test case execution is failing on ADO: To execute mobile test case successfully on ADO, you need to provide all the mobile capabilities manually.
Here is the list of required capabilities in ADO Pipeline.

Capability Name	Values (As per execution needs)
MobilePlatform	Android
MobileBrowser	Chrome
MobileExecutionEnvironment	Local
AppiumServerURL	http://127.0.0.1:4723/
androidAppiumServerURL	http://127.0.0.1:4723/
android.platformName	android
android.browserName	chrome
android.appium:udid	emulator-5558
android.appium:deviceName	Pixel_3a_XL_API_34
android.appium:automationName	UiAutomator2
appiumOption	Custom/Automatic
androidAppiumOption	Custom/Automatic

Follow the below steps to add capabilities:

Go to ADO.
Edit the pipeline.
Click three dots next to Enter environment variables.
Add capabilities & their values from the above table.
Click OK.
Save the pipeline.