Versions Compared

Old Version 5

changes.mady.by.user Shruti Agrawal

Saved on

compared with

New Version Current

changes.mady.by.user Shruti Agrawal

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Visual Studio, a Microsoft product from Microsoft, helps product teams with , provides tools and technologies in that support all the stages of application development.  Qualitia Automation Qualitia Automation Studio (QAS) helps replace the enhances this process by replacing manual testing procedures in the application development processes by tightly with automated procedures, seamlessly integrating with Azure DevOps. This makes integration simplifies and strengthens the application development process robust and simple. Qualitia's integration to

With QAS integrated into Azure DevOps lets , organizations leverage can maximize their existing investments as part of an integrated by using a unified platform for automation testing development and allows managing , while managing and maintaining automated test cases prepared by Qualitia inside directly within Azure DevOps.


Key Benefits of QAS and Azure DevOps Integration:

...

To integrate QAS with Azure DevOps, you need to first set up the environment.

Prerequisites

Prerequisites for Azure DevOps

...

Let’s set up the environment, to get started with the Azure DevOps Integration. This section depicts information related to all the prerequisites and procedures to get prepared for QAS's integration to Azure DevOps.

This section describes how Azure DevOps interacts with QAS components to execute QAS test cases successfully.

Qualitia Components:

  • Qualitia Server

  • Qualitia Client

Azure DevOps Components:

  • Azure DevOps as the Cloud Version

  • Azure DevOps Self Hosted Agent

To integrate QAS with Azure DevOps, follow these three key steps:

Association Between Qualitia Automation Studio and Azure DevOps

This step involves initiating the association from within Qualitia Automation Studio (QAS), mapping projects and test cases between QAS and Azure DevOps.

Key Activities:

  • Map QAS Project to Azure DevOps Project: Establish a link between your project in QAS and the corresponding project in Azure DevOps.

  • Map Automated Test Cases: Link QAS automated test cases with Azure DevOps manual test cases, ensuring that test cases are synchronized across both platforms.

  • Associate Test Suites: Map Qualitia Suites to Azure DevOps Test Suites to group relevant test cases for streamlined execution.

Execution Setup in Azure DevOps (One-Time Setup)

This one-time setup ensures that QAS test cases can be executed directly from Azure DevOps pipelines.

Key Activities:

  • Setup Build/Release Pipelines: Create and configure build or release pipelines in Azure DevOps to execute the associated QAS test cases.

  • Task Extension for Custom Tasks: Install the Qualitia Task Extension, which allows custom Qualitia tasks to be included in the pipeline installation. (Task Extension is used for Qualitia Custom tasks for pipeline installation)

  • Configure Self-Hosted Agents: Set up a self-hosted Azure DevOps agent for executing the QAS test cases locally.

  • Agent Pool Setup: Create and configure an agent pool in Azure DevOps to manage agents responsible for running QAS executions.

Result Publishing and Execution

This step ensures that the results of the executed test cases are published and available both in Azure DevOps and within QAS.

Key Activities:

  • Publish Results in Azure DevOps: After each pipeline execution, results of the QAS test cases are published to Azure DevOps test cases and test runs.

  • Result Sync with QAS: Test case results are synchronized with Qualitia Automation Studio to provide a holistic view of the automation results within QAS.

The following table explains how different components of Azure DevOps and QAS interact with each other.

Component

Roles Played in Different Stages

Azure DevOps

Association

  • Responds to QAS server using standard out of the box APIs offered by Azure DevOps.

Execution

  • Configuration of build/release pipelines.

  • Agent pool configurations.

  • Initiate pipeline execution.

Result Publish

  • Host results of pipeline execution.

Azure DevOps Agent

Association

None

Execution

  • Executes the jobs in the pipeline along with the contained tasks.

  • Select Test Plan and Test Suite that contain the automated test cases and suites within the Azure DevOps projects.

Result Publish

  • Execution of the Publish Result task of QAS to publish results to QAS Server and Azure DevOps Test Runs.

  • Uses QAS Server APIs to publish result.

Qualitia Server

Association

  • Interacts with the Azure DevOps server to update automation status of Azure DevOps test cases.

Result Publish

  • Receives test case results directly from QAS - Client with the help of QAS APIs.

Qualitia Client

Association

  • Allows the authorized users to view Azure DevOps projects and associated projects and test cases.

Execution

  • Publishes the test cases to QAS project repository. The published test cases are used in the Azure DevOps pipeline execution.

Result Publish

  • Shows the status of test cases executed through Azure DevOps pipeline in Real Time Reporting Portal.

Prerequisites

Prerequisites for Azure DevOps

...

Azure DevOps Version

...

  • Azure DevOps (Cloud Version)

  • Azure DevOps Server 2019

...

 Qualitia Plugins

...

Share Azure DevOps URL to Qualitia team to get the Get Assembly task plugin.

...

 Roles and Privileges

  • Azure DevOps Administrator must install the plugins from Shared Extensions.

  • Qualitia does not recommend this option for PAT generation as it changes the access levels for all the users in the group.

Note: Please connect with your Azure DevOps Admin team to discuss the PAT generation and access level management.

Qualitia integration with Azure DevOps expects that the user has “Update Work Items” access that is used to generate the PAT (Personal Access Token).

The PAT with required access can be procured in one of the following ways:

  • Generate the PAT with a user that is part of the Project Administrators group.
    Qualitia recommends this option for generating a PAT.

  • Allow Bypass Rules on Work Items Updates
    Perform the following steps:

    • In Azure DevOps project, ensure that the user that is used to generate PAT is member Contributors group / Project Team members group.

    • Click Project Settings > Permissions.

    • Select the user group that has the user to whom you want to provide access (Contributors/Project Team members).

    • From the Bypass rules on work items updates list, select Allow.
      Refer to the following screenshot.

  • Have project access to create the following:

    • A new pipeline (Build/Release)

    • An Agent Pool

Prerequisites for Qualitia

Roles and Privileges

  • You must have administrative privileges to access Qualitia Project that you want to integrate with Azure DevOps tests.

 Network Access

  • Ensure that the Qualitia Server URL is reachable from the Qualitia Client, Azure DevOps Server, and Azure DevOps agent.

Configuring Azure DevOps for Test Executions

...

Azure DevOps Version

  • Azure DevOps (Cloud Version)

  • Azure DevOps Server 2019

 Qualitia Plugins

Share Azure DevOps URL to Qualitia team to get the Get Assembly task plugin.

 Roles and Privileges

  • Azure DevOps Administrator must install the plugins from Shared Extensions.

  • Qualitia does not recommend this option for PAT generation as it changes the access levels for all the users in the group.

Note: Please connect with your Azure DevOps Admin team to discuss the PAT generation and access level management.

Qualitia integration with Azure DevOps expects that the user has “Update Work Items” access that is used to generate the PAT (Personal Access Token).

The PAT with required access can be procured in one of the following ways:

  • Generate the PAT with a user that is part of the Project Administrators group.
    Qualitia recommends this option for generating a PAT.

  • Allow Bypass Rules on Work Items Updates
    Perform the following steps:

    • In Azure DevOps project, ensure that the user that is used to generate PAT is member Contributors group / Project Team members group.

    • Click Project Settings > Permissions.

    • Select the user group that has the user to whom you want to provide access (Contributors/Project Team members).

    • From the Bypass rules on work items updates list, select Allow.
      Refer to the following screenshot.

  • Have project access to create the following:

    • A new pipeline (Build/Release)

    • An Agent Pool

Prerequisites for Qualitia

Roles and Privileges

  • You must have administrative privileges to access Qualitia Project that you want to integrate with Azure DevOps tests.

 Network Access

  • Ensure that the Qualitia Server URL is reachable from the Qualitia Client, Azure DevOps Server, and Azure DevOps agent.

Azure DevOps Integration

Let’s set up the environment, to get started with the Azure DevOps Integration. This section depicts information related to all the prerequisites and procedures to get prepared for QAS's integration to Azure DevOps.

How Azure DevOps interacts with QAS components to execute QAS test cases successfully.

Qualitia Components:

  • Qualitia Server

  • Qualitia Client

Azure DevOps Components:

  • Azure DevOps as the Cloud Version

  • Azure DevOps Self Hosted Agent

To integrate QAS with Azure DevOps, follow these three key steps:

Association Between Qualitia Automation Studio and Azure DevOps

This step involves initiating the association from within Qualitia Automation Studio (QAS), mapping projects and test cases between QAS and Azure DevOps.

Key Activities:

  • Map QAS Project to Azure DevOps Project: Establish a link between your project in QAS and the corresponding project in Azure DevOps.

  • Map Automated Test Cases: Link QAS automated test cases with Azure DevOps manual test cases, ensuring that test cases are synchronized across both platforms.

  • Associate Test Suites: Map Qualitia Suites to Azure DevOps Test Suites to group relevant test cases for streamlined execution.

Execution Setup in Azure DevOps (One-Time Setup)

This one-time setup ensures that QAS test cases can be executed directly from Azure DevOps pipelines.

Key Activities:

  • Setup Build/Release Pipelines: Create and configure build or release pipelines in Azure DevOps to execute the associated QAS test cases.

  • Task Extension for Custom Tasks: Install the Qualitia Task Extension, which allows custom Qualitia tasks to be included in the pipeline installation. Task Extension is used for Qualitia Custom tasks for pipeline installation.

  • Configure Self-Hosted Agents: Set up a self-hosted Azure DevOps agent for executing the QAS test cases locally.

  • Agent Pool Setup: Create and configure an agent pool in Azure DevOps to manage agents responsible for running QAS executions.

Result Publishing and Execution

This step ensures that the results of the executed test cases are published and available both in Azure DevOps and within QAS.

Key Activities:

  • Publish Results in Azure DevOps: After each pipeline execution, results of the QAS test cases are published to Azure DevOps test cases and test runs.

  • Result Sync with QAS: Test case results are synchronized with Qualitia Automation Studio to provide a holistic view of the automation results within QAS.

The following table explains how different components of Azure DevOps and QAS interact with each other.

Component

Roles Played in Different Stages

Azure DevOps

Association

  • Responds to QAS server using standard out of the box APIs offered by Azure DevOps.

Execution

  • Configures of build/release pipelines.

  • Agent pool configurations.

  • Initiates pipeline execution.

Result Publish

  • Hosts results of pipeline execution.

Azure DevOps Agent

Association

None

Execution

  • Executes the jobs in the pipeline along with the contained tasks.

  • Select Test Plan and Test Suite that contain the automated test cases and suites within the Azure DevOps projects.

Result Publish

  • Execution of the Publish Result task of QAS to publish results to QAS Server and Azure DevOps Test Runs.

  • Uses QAS Server APIs to publish result.

Qualitia Server

Association

  • Interacts with the Azure DevOps server to update automation status of Azure DevOps test cases.

Result Publish

  • Receives test case results directly from QAS - Client with the help of QAS APIs.

Qualitia Client

Association

  • Allows the authorized users to view Azure DevOps projects and associated projects and test cases.

Execution

  • Publishes the test cases to QAS project repository. The published test cases are used in the Azure DevOps pipeline execution.

Result Publish

  • Shows the status of test cases executed through Azure DevOps pipeline in Real Time Reporting Portal.

Configuring Azure DevOps for Test Executions

To execute automated test cases from QAS mapped to Azure DevOps as part of your build and release pipelines, you need to add the Qualitia QAS - Test Execution extensions to your Azure DevOps instance.

Follow these steps to add the QAS - Test Execution extensions to your Azure DevOps instance:

  1. Share Azure DevOps Organization Details: Before integrating Azure DevOps with Qualitiaintegration, provide your Azure DevOps organization details to Qualitia. They Qualitia will then share the necessary extensions for integration with your Azure DevOps projects.

  2. Add and Install Extensions: Once Qualitia shares the extensions, you can add proceed with adding and install installing them in your Azure DevOps instance.

  3. Installation Steps:

    • Sign into Azure DevOps instance.

    • Click the top-right corner to navigate to Marketplace > Manage Extensions.

    • Locate the Qualitia - Test Execution extension shared by Qualitia.

    • Click the Qualitia - Test Execution and install it.

  4. Verify Installation: Return to Extensions > Manage Extensions to confirm that ensure the installation extension was successfulinstalled successfully.

Your Azure DevOps instance is now configured for Qualitia QAS integration and ready to for use in build and release pipelines.

 

Configure Qualitia - Test Execution Task

...

Follow the steps to configure the Qualitia - Test Execution task in your pipeline within Azure DevOps:

Add Task in Pipeline

  1. Log into your Azure DevOps instance.

  2. Add new CI-CD pipeline or edit the an existing pipeline.

    • To create new pipeline, select required repository or use the Select classic view option.

  3. Click the + icon to add new task.

  4. Search and select Qualitia QAS - Test Execution and add it.

After you add adding the custom task into to the CI-CD pipeline, you must configure it by adding the following details:

Display Name

Enter the desired display name for this task.

Qualitia Option

Enter Qualitia Server URL

Enter the hostname or IP of the machine where QAS server is installed and enter , along with the server port number, by default it will use port 8887.
However, you may change the port . The default port is 8887, but you can change it during QAS - Server installation.

Eg. For example, http://127.0.0.1:8887

 

Enter Qualitia Project Name

Name Provide the name of the QAS project that is integrated with Azure DevOps.

Execution Parameter

Specify any additional QAS execution parameter parameters, such as selecting a non-default browser to use will while execution and override or overriding environment variablevariables. By default, this is set to None is selected.

Test Selection

Test Plan

Select a test plan containing that contains test suite with QAS automated test cases.

Test Suite

Select test suite(s) containing QAS automated test cases. You can select more than one suite heremultiple test suites.

Execution Option

Distribution mode

Batch test across agents:

Distribute test case cases across agent in define agents in batches, with a defined number of tests per batch.

Replicate all tests across all agents:

Execute all test cases on all every configured agent configured. 

Control Option

Enabled

This shows Shows the current state of the customized task, and gives you control over the customized taskcustom task. By default, this option is selected. Clear this option to disable a task. It is useful when you want to temporarily take task out of the process the task temporarily (useful for testing or for specific deployments). 

Continue on error

Select this option if you want to continue QAS test case execution even after getting errors in the pipeline.

By default, this option is not enabledError

If selected, the pipeline will continue executing QAS test cases even after encountering errors. This option is disabled by default.

Timeout

Set the task timeout for the task in minutes. Default The default is 0, which refers to infinite timeout. Value set here This value overrides the value timeout set for the parent task job.

Run this task

Here you can set the condition to run a Set conditions for running this task. You can use one of choose from the pre-designed defined options in the list or you can create new customized condition.You may refer to Azure DevOps documentation for more information create custom conditions.

For more details on creating custom conditions as expressions, refer to Azure DevOps documentation.

...

You need to enable Allow scripts to access the OAuth token from Agent job.

...

Connecting

...

QAS Artifacts to Azure DevOps

After you Once the environment is set up the environment in Azure DevOps for automating tests test automation using QualitiaQAS, it the next step is now time to map your Qualitia QAS artifacts to Azure DevOps where its starts . This begins with mapping a Qualitia QAS project to its corresponding Azure DevOps project.

...

This connection ensures that automated test cases created and managed in QAS can be seamlessly integrated and executed within Azure DevOps pipelines, leveraging the tools and features of both platforms.

Qualitia Test Case Lifecycle for Azure DevOps Integration

During the integration process of QAS with Azure DevOps, the Qualitia QAS test case progresses through various stages, impacting affecting the status of the associated Azure DevOps manual test case. Below is a description of each stage and its implicationsan overview of these stages and their impact:

The following diagram shows the Integration Lifecycle status for a test case.

...

...

Test Case Lifecycle Stages

1. Test Case Creation

  • When a new test case is created in

...

  • QAS under a project integrated with Azure DevOps

...

  • , it

...

  • is not associated with any Azure DevOps manual

...

  • test case

...

  • .

  • QAS Test Case

...

  • Status: Not Linked

2. Test Case Linking with Azure DevOps Test Case

  • The association between

...

  • a QAS test case and an Azure DevOps test case is established

...

Publishing Test Case

Once the Automation Engineer completes the test case in Qualitia and is ready for execution in Azure DevOps, the test case must be published.

  • Qualitia Test Case Status Change:
    Linked/Not Published → Linked/Published

  • Azure DevOps Test Case Automation Status Change:
    Planned → Automated

Editing a Test Case after Publishing

If changes are made to the Qualitia test case after publishing, these changes do not reflect in Azure DevOps. The published version is used for test execution.

  • Qualitia Test Case Status Change:
    Linked/Published → Linked/Changes Not Published

  • Azure DevOps Test Case Automation Status:
    Automated → Automated

Test Case Unlinking with Azure DevOps Test Case

The link between the Qualitia test case and the Azure DevOps test case can be removed at any time by unlinking the test case.

  • Qualitia Test Case Status Change:
    Linked/Published or Changes Not Published → Not Linked

  • Azure DevOps Test Case Automation Status Change:
    Automated → Not Automated

Linking a QAS Project to an Azure DevOps Project

To link a Qualitia project to an Azure DevOps project:

  1. Log into Qualitia as a project administrator.

  2. From the Expand Menu, click Project Management.

    image-20241008-102501.pngImage Removed

  3. In the left pane, click a project that you want to link.

  4. In the main pane, next to Integrations (ADO), click on link icon.

  5. In the Azure DevOps tab, enter the following details of Azure DevOps.

    1. Server URL: Enter the Azure DevOps URL.

    2. PAT Token: Enter Personal Access Token (PAT) for authentication.

    3. Connect: Click to connect to Azure DevOps server.

    4. Azure DevOps Project: Select an Azure DevOps project from the drop-down list. User can select multiple projects.

    5. Click Integrate.     

 

Linking a Qualitia Test Case to an Azure DevOps Test

You can link a test case either from the Test Case Editor screen or from Ready for Execution screen.

To link a test case from the Test Case Editor screen:

  1. From the Expand Menu, click Develop.

  2. From the left pane, select a scenario, and open the test case.

  3. Click the Link button

 To link a test case from the Ready for Execution screen:

  1. From the Expand Menu, click Execute.

  2. Click the Ready for Execution tab.

  3. A list of test cases is displayed in Test Case tab

Update the following details The below screen will open in both the cases:

...

  • by linking them.

  • Status Changes:

    • QAS: Not Linked → Linked/Not Published

    • Azure DevOps Automation: Not Automated → Planned

3. Publishing Test Case

  • After the Automation Engineer completes writing the test case in QAS, it must be published for execution in Azure DevOps.

  • Status Changes:

    • QAS: Linked/Not Published → Linked/Published

    • Azure DevOps Automation: Planned → Automated

4. Editing a Test Case After Publishing

  • Changes made to a QAS test case after publishing do not reflect in Azure DevOps. The previously published version is used for execution.

  • Status Changes:

    • QAS: Linked/Published → Linked/Changes Not Published

    • Azure DevOps Automation: Automated → Automated

5. Unlinking Test Case

  • The link between a QAS test case and an Azure DevOps test case can be removed anytime.

  • Status Changes:

    • QAS: Linked/Published or Linked/Changes Not Published → Not Linked

    • Azure DevOps Automation: Automated → Not Automated

Linking a QAS Project to an Azure DevOps Project

To link a QAS project to an Azure DevOps project:

  1. Log into QAS - Client as a project administrator.

  2. Access Project Management from the left navigation.

    image-20241008-102501.pngImage Added

  3. In the left pane, select a project that you want to link.

  4. In the main pane, next to Integrations (ADO), click the link icon.

  5. In the Azure DevOps tab, enter:

    1. Server URL: The Azure DevOps URL.

    2. PAT Token: Personal Access Token (PAT) for authentication.

    3. Click Connect to link the Azure DevOps server.

    4. Select the Azure DevOps Project from the drop-down list. You can select multiple projects.

    5. Click Integrate to complete the connection.     

Linking a QAS Test Case to an Azure DevOps Test

You can link a Qualitia test case to an Azure DevOps test from two screens: Test Case Editor or Ready for Execution.

To link a test case from the Test Case Editor screen:

  1. Log into QAS - Client.

  2. Access Develop from left navigation.

  3. From the left pane, select a scenario, and open the test case.

  4. Click Link.

 To link a test case from the Ready for Execution screen:

  1. Log into QAS - Client.

  2. Access Execute from left navigation.

  3. Click the Ready for Execution tab.

  4. A list of test cases will be displayed. Select the desired test case and click Link.

Configure Test Case Linking:

  • In both methods, the same screen appears where you need to fill in:

    • ADO Project: Select the Azure DevOps project.

    • Select ADO Test Plan: Select the associated Azure DevOps test plan.

    • Select ADO Test Suite: Select the Azure DevOps test suite.

    • Optionally, use the Search box to find the Azure DevOps test by name or ID.

    • Select the test case to link and click Link.

    • The test case is now linked, and the Automation Status in Azure DevOps will be updated to Automated.

...

Viewing Status in Azure DevOps 

Once publish any test case from Qualitia, the status of the test in Azure DevOps is marked as Automated.

...

To view the status of the test case in Azure DevOps:

  1. In Azure DevOps, select the test suite.

  2. In the Automation Status column, the status will be updated to Automated.

    image-20241010-103943.pngImage Added

  3. Open this test case and in the steps section, you will find the status as Automated.
    This means this test case is successfully linked to QAS automated test case.

  4. Open the test case to verify that the steps section also reflects the Automated status.

This confirms that the test case is successfully linked to QAS and is ready to be executed as part of the build or release pipeline in Azure DevOps.

...

Linking a QAS Suite to an Azure DevOps Suite

You can link a suite from Ready for Execution screen.

To link a suite from the Ready for Execution screen:

  1. Access Execute from the left navigation.

  2. Click the Ready for Execution tab.

  3. A list of suites is displayed in Suites tab

    image-20241010-104353.pngImage Added

  4. Select the suite and click link icon (image-20241014-041107.pngImage Added) under column Linked ADO Suites to open Link with ADO Suites window, as shown in the screenshot.

    image-20241010-104415.pngImage Added

Working with QAS Test cases In Azure DevOps

Once the environment is set up in Azure DevOps and QAS test cases are linked, you can start executing these test cases as part of your build or release pipelines. Additionally, execution reports of QAS automated test cases can be accessed directly from Azure DevOps.

Executing QAS Automated Tests 

Execute QAS Automated Tests Using Build Pipelines

  1. Sign in to your Azure DevOps instance.

  2. From the top menu, click Build and Release.

  3. Under the Builds section, click the edit icon (...) and select Queue new build....

  4. A new build will be added in a queue, which will eventually initiate execution of QAS automated tests.

Execute QAS Automated Tests Using Release Pipelines

  1. Sign in to your Azure DevOps instance.

  2. From the top menu, click Build and Release.

  3. Go to the Releases section, click +Release > New release.

  4. In the Create new release section, choose the desired environment.

  5. Artifacts are pre-selected by default.

  6. Add Release description, if required.

  7. Click Create to start a new release and execute QAS automated tests.

Accessing QAS Test Execution Reports in Azure DevOps

After executing QAS automated tests, QAS test execution reports are attached to the respective test in Azure DevOps. You can access these reports from the Test run section.

Each test run in Azure DevOps is assigned a unique Run ID for easy identification.

Steps to Access Reports:

  1. Sign in to your Azure DevOps instance.

  2. Navigate to Test > Runs from the top menu.

  3. Double-click the Run ID to view more information about that instance.

  4. Select Test results.
    A list of tests appears, which were executed in the selected run instance.

  5. Double-click the test to view more detailed information about it. 

Accessing Execution Results from QAS

You can also access execution reports from within Qualitia Automation Studio (QSA) or through the Real-Time Reporting Portal after executing tests via build or release pipelines. For more information, refer to Real Time Reporting Portal.
While displaying reports for QAS automated tests, Pipeline Name is displayed in the Suite Name column, as shown in the following figure:

...

🔧 Got a problem? Let’s fix!

...