Our latest posts

Quality Assurance engineers

Tested projects

Tools & Technologies

We build modern Web Applications, based on Angular, React and Vue javascript frameworks

Benefit from our experience and multiple completed QA projects. QA teams will be tailored to your needs and expectations.

Experience

Procedures, guidelines, and documentation based on the best industry practices and standards. Established and optimised QA processes from day one.

Industry standards

Automated QA delivery processes integrated with your DevOps. Agile approach from planning to execution and final reporting.

Automation and Agility

We're ready to expand QA operations on demand. Both in terms of team size and the range of QA services.

QA operations on demand

ISTQB-certified Dedicated Quality Assurance teams

Quality Assurance as a

QA teams tailored to your needs and expectations, using the best industry practices and standards and equipped with all the necessary tools, licences and devices.

A full range of testing services

Appium

Cypress

Cucumber

Webdriver.io

Playwright

Our top pick and favorite technologies used in the latest projects

Top platforms and frameworks

Quality Assurance as a Service

Eugene Furseyeu

## Why test metrics?

- **Visibility.** Test automation metrics can help to solve QA visibility problems. Metrics can provide a high-level overview of the testing process by aggregating key test automation data such as test cases quantity, test execution time, automation error patterns.
- **A Way to improve.** Test automation metrics can help **to improve QA processes**. Metrics can enable an organization to filter out ineffectiveness and improve over time. If you cannot measure it, you’ll not be able to improve it.
- **Decision making.** Test automation metrics can help decision makers to get a better understanding of the product quality and what should be done next.

## Test Automation Coverage.

This metric illustrates the percentage of tests executed automatically compared to those done manually.

**Metric Value:** It helps to track whether a QA team is meeting automated test coverage goals, or not. In the Agile methodology test automation coverage should increase sprint to sprint.

## Test Automation Progress.

This metric shows the ratio of automated test cases to the automatable test cases over the time.

**Metric Value:** It helps to understand the progress and pace of test automation compared to the plans and discover test automation problems on the early stage if deviations are detected. It also shows how well QA engineers respond to the objective of automated software testing and if such objectives were right.

## Automatable Test Cases.

Understanding the percentage of test cases automatable compared to the total test cases number is very helpful in formulating an appropriate testing strategy and keeping a balance between automated and manual testing.

**Metric Value:** This metric can help to determine the priorities in the QA processes and focus on the most important test cases first.

## Automation Script Effectiveness.

An evaluation of defects found via automated testing compared to the total number of acceptable defects is an _Automation Script Effectiveness_.

**Metric Value:** It allows us to see the problems with test automation quality. Having a low percentage of Automation Script Effectiveness indicates that there might be  problems with automation suite quality.

## Automated Test Pass Rate.

It measures how many automated tests have passed vs. how many have failed.

**Metric Value:** It helps to understand the **stability** of the automation suite as well as its **effectiveness**. Having a low failure rate indicates that the script logic is correct, while a low pass rate requires an effort validating failures. Also it raises a question if those failures occurred because of the flaky test cases or they discovered issues with the software. However, it’s important to understand that the metric cannot show the quality of those tests passed.

## Build Stability.

When test automation suites are a part of software building pipelines in a CI/CD this metric can demonstrate the efficacy of the tests by calculating the ratio of failed builds to passed ones.

**Metric Value:** This metric gives an idea whether the tests implemented are sufficient to ensure a stable build process.

### Conclusions

Test automation has become an integrated part of the agile testing processes as well as CI/CD pipelines so that many companies found it beneficial to invest time and effort to set up proper test  automation processes. 

_Test automation metrics_ help QA teams to evaluate  testing strategy promptly and make appropriate adjustments, resulting in better quality software.

In qaVisor, we focus on high-quality QA services by having experienced QA engineers in the team and providing the appropriate metrics to ensure quality assurance processes develop in the right direction. 

thumbnail_illustration1.jpg

illustration1.jpg

thumbnail_laptop1.jpg

laptop1.jpg

Key Test Automation Metrics in Agile Testing

Klaudijus Lupeika

Nowadays, it is quite important to make sure that the managerial level of the company has a full understanding of what kind of testing the QA teams are performing. For a person without any technical background it might get frustrating looking at the complex code. To understand what kind of testing has been done, some help from software engineers is definitely needed. To fill in this gap, we have a BDD (Behavior Driven Development) testing framework. BDD framework simplifies the complex and robust test suites into human readable text. One of the most popular tools in BDD is Cucumber. 

## What is Cucumber?

Cucumber is an open-source software testing tool. In simple terms, Cucumber enables you to write test cases that most people can understand even without any technical background. The secret sauce is the simple, yet powerful syntax - Gherkin. The main focus of Gherkin language is to define the "behavior" rather than the "technical tests". Let's have a look at the example below:

```gherkin
Feature: Web search

  Scenario: Search for rental cars

    Given the Google.com is open
        When I search for rental cars
        Then Search result for rental cars is visible
```

Firstly, we create a file with the name of _webSearch.feature_. The feature file in cucumber holds the test scenarios and the sequence of the test steps (defined as _Given_, _When_, _Then_ steps). The scenario name represents the current test suite case of what we are trying to test.

## What is Gherkin?

Gherkin is a set of grammar rules that makes plain text structured enough for Cucumber to understand. The scenario above is written in Gherkin. Gherkin documents are stored in **.feature** text files and are typically versioned in source control alongside the software.

Gherkin serves multiple purposes:
- Unambiguous executable specification
- Automated testing using Cucumber
- Document how the system _actually_ behaves

As you can see, it is pretty self explanatory and it is clear what is happening in this scenario. Firstly, we have a _Given_ step, where we provide the test some background on what is given to start with, Next, we have a _When_ step, which is being executed after the _Given_. Lastly, we have _Then_ step, where appropriate logic will be executed at this step. It is worth to mention that, there can be multiple _Given-When-Then_ steps within a scenario, which can be repeated using _And_ or _*_ symbol.

Each keyword has their purpose. As a person with no technical background, you can understand it well as plain text.
- Given: What we have
- When: What we do
- Then: What's the expected result

```gherkin
Feature: Web search

  Scenario: Search for rental cars

    Given the Chrome browser is open
    * The Google.com is open

    When I search for rental cars
    And when I select image search

    Then Search result for rental cars is visible
    *  Search result contains pictures
```

While this approach can be really helpful, one might say that this method of writing test cases can become redundant. To avoid it, Cucumber allows us to run _parameterized_ tests, which can be run in _parallel_ mode (multiple test scenarios at once). This way, we can speed up the test runs and perform different tasks with reusable steps. Next picture represents a parameterized test scenario, where we can pass our own parameters in to test logic:

```gherkin
Feature: Web search
  
  Scenario Outline: Search for items
  
    Given the Chrome browser is open
    * the Google.com is open

    When I search for <item>
    And when I select image search

    Then Search result of <item> is visible
    * Search result contains pictures
  
  Examples:
  |item              |
  |cars              |
  |fruit             |
  |office supplies   |
  |laptops           |
  |groceries         |
```

Firstly, we have to let Cucumber know that this scenario should run on a loop using our _Examples_ given below. Then it replaces the strings with passed parameters.

Cucumber is widely used throughout the QA practice and is one of the leading BDD frameworks these days. It can be combined with various other frameworks to support the logic of the tests. The picture below shows Cucumber's ability to be combined with most modern frameworks.

Since we already have _.feature_ files which consist of the behavior that should be executed, we need a tool to be able to automate this process and actually perform the activities needed to fulfill the test suites. As seen earlier, there can be many different combinations with Cucumber, but for this demonstration purpose, we will use a modern and rapidly growing _Playwright_ framework.

## What is Playwright?

While Cucumber feature files possess the behavior of the tests, we need a tool to be able to perform a logic based on what we want to do during the test. Here comes the _Playwright_.

Playwright is an open-source test automation library initially developed by Microsoft contributors. It supports programming languages such as Java, Python, C#, and NodeJS. Playwright comes with Apache 2.0 License and is most popular with NodeJS with which results in tests being able to be written in Javascript or Typescript.

I have chosen Playwright, not just because it might seem a comfortable tool, but with the features it comes with:
- Easy installation and configuration
- Multi-Browser Support
- Multi-Language Support
- Parallel Browser Testing
- Support for Multiple Tab/Browser Window
- Built-in Reporters
- CI/CD Integration Support

Ever since I started to work with Playwright, few of the features made automation much easier. Firstly, the Playwright allows you to use multiple _Contexts_ within the testing suite, which results in a better way of handling each browser's cache and session data. It allows testing the software with particular data related to the session. In addition, it allows you to work around the popups and it has integration with most modern browser engines: Chromium, WebKit and Firefox.

If you are still hesitating in using Playwright, let me give you a couple more examples, why it might be a great fit for your project. Playwright supports cross-platform, which allows you to test on Windows, Linux, and macOS, locally or on CI, even headless or headed! It can be really handy to test modern JavaScript framework components, since Playwright selectors pierce shadow DOM and allow entering frames seamlessly.

![Browsers](https://nextjsstrapi.s3.eu-central-1.amazonaws.com/browsers_84599bdb89.png)

Also, Playwright has a tool called _Codegen_, which can be really useful to visually perform the actions needed, which will be returned to you in any selected language. This tool will allow you to generate tests quickly.

![Playwright](https://nextjsstrapi.s3.eu-central-1.amazonaws.com/playwright_469d17485d.png)

Since now we know what are the benefits of Cucumber and Playwright, let's put it in to use and see how it works in action!

## Cucumber and Playwright integration

In order for Cucumber and Playwright to work together, we need to add NodeJS to our project. It will give Playwright the ability to execute JS which will be defined as our test logic. After installing NodeJS, Cucumber and Playwright in our project, it is time to try and run our previously created feature file.

```plaintext
1) Scenario: Search for items # features\future.feature:18
    ✔ Before # cucumber.conf.js:38
    ?  Given the Chrome browser is open
        Undefined. Implement with the following snippet:

            Given(‘the Chrome browser is open', async function () {
              // Write code here that turns the phrase above into concrete actions
              Return 'pending';
            });
    ?  * the Google.com is open
        Undefined. Implement with the following snippet:

            Given('the Google.com is open', async function () {
              // Write code here that turns the phrase above into concrete actions
              Return 'pending';
            });
```

As you can see above, Cucumber test runner indicates to us that we have feature files written, however, we do not have logic to be executed. In simple words, we need to create a _steps.js_ file, where we will store each step logic and using Playwright, execute the code, so let's go ahead and create these steps.

```javascript
Given('the Chrome browser is open', async function() {
        Browser = await chromium.launch({
            Headless: false,
        });
        context = await browser.newContext();
        page = await context.newPage();
    });
```

The picture above shows how the steps should be implemented. Inside curly braces we include our code to be executed in the step. For demo purposes, I have taken the very first given step. Here we define our browser, context and page, which can carry different cache and session data. The other steps should contain the logic that we want to execute within the steps, which can be seen on the next image.

```javascript
Given('the Chrome browser is open', async function() {
        Browser = await chromium.launch({
            Headless: false,
        });
        context = await browser.newContext();
        page = await context.newPage();
});
Given('the Google.com is open', async function() {
    await page.goto('https://www.google.com');
});

When('I search for {string}', async function(string) {
// Fill search input [aria-label="Search"]
    await page.locator('[aria-label="Search"]').fill(string);
    await page.locator('input[type="submit"]').click();
});

When('I select image search', async function() {
    // Click images search
    await page.locator('a:text-is("Images")');
});

Then ('Search result contains pictures', async function() {
expect (await page.locator('.search-result')).toBeVisible();
});

Then ('Search result contains pictures', async function() {
    expect (await page.locator('.search-result.img')).toBeVisible();
});
```

As you can see, every step now contains logic and the test scenario is ready to be run. Now, to explain the flow of the logic: the Cucumber test runner will execute the feature file. Feature file explain Cucumber, the order of the steps they should be executed. Then It looks for the steps and executes the logic in each step. At the very end, we get the result of how things went in our test scenario.

```plaintext
.....
1 scenario (1 passed)
3 steps (3 passed)
0m33.611s (executing steps: 0m32.015s)
```

## Advanced tips and tricks

One must mention that a wide variety of customization is available using Cucumber and Playwright. Probably one of the most useful features in Cucumber test runner is _profiles_. Using profiles, we can set a particular data according to our test needs. This can be very useful while testing different environments of the software.

Also, it is worth mentioning that Playwright lets us create separate browser contexts, which can hold particular data for testing purposes. Session is being active until the same context is active and to create a new browser context is just a simple one-liner. In addition, Cucumber lets us run tests in parallel mode. This means that multiple tests can be run at the same time, which greatly reduces the duration of testing. 

The last feature of Playwright I wanted to mention is ability to execute particular code before or after each, or all tests. This way, we can simply tell Playwright to create a new browser context every time we run a new test. This will assure our session is limited to every test scenario.

To conclude, I must say that Cucumber and Playwright combination is lightweight, fast and has many customization options to suit your project's requirements. It is easy to learn if you have some basic JavaScript background. Using this combination of Cucumber and Playwright will assure that the software is working as expected and flaws will be discovered immediately.

thumbnail_illustration4.jpg

illustration4.jpg

thumbnail_code.jpg

code.jpg

E2E testing: Cucumber and Playwright

According to its headline Playwright enables **reliable end-to-end testing for modern web apps**. The Playwright library provides cross-browser automation through a single API. 

[Playwright](https://playwright.dev/docs/intro) is a [Node.js](https://nodejs.org/) based library to automate [Chromium](https://www.chromium.org/Home), [Firefox](https://www.mozilla.org/firefox), and [WebKit](https://webkit.org/) based browsers with a single API. Playwright is built to enable cross-browser web automation that is reliable, and fast. It supports programming languages such as Java, Python, C#, and NodeJS. Playwright comes with Apache 2.0 License and is most popular with NodeJS with Javascript/Typescript.

Let’s review Playwright's main features and how QA engineers benefit from them.

## Cross-browser, Cross-platform and ready for Mobile Web.

Playwright supports all modern rendering engines including Chromium, WebKit, and Firefox. So, that means QA engineers can test all major browsers using tools provided by Playwright out of the box. Also tests run on Windows, Linux, and macOS, locally or on CI, headless or headed. At the same time Playwright provides native mobile emulation of Google Chrome for Android and Mobile Safari. Using such emulation it’s possible to test mobile web both on QA engineers local machines and in the Cloud.

## Designed to run for multiple processes, work with dynamic elements and Shadow DOM.

The downside of the previous generation of test runners was that they usually run within a single process and therefore were not able to run web content belonging to different origins in different processes. Playwright is aligned with the modern browsers architecture and runs tests out-of-process. This makes Playwright free of the typical in-process test runner limitations. As a result it is possible to create test scenarios that span multiple **tabs**, multiple **origins** and multiple **users**. Such scenarios with different contexts for different users can run all in one test.

Playwright uses real browser input pipeline indistinguishable from the real user such as hover elements, interact with dynamic controls and produce trusted events. 

Playwright comes with robust auto wait and it waits for elements to be actionable prior to performing actions. It also has a rich set of introspection events. The combination of the two **eliminates the need for hardcoded timeouts** which decrease flaky tests and speed up tests execution considerably. Playwright assertions are created specifically for the dynamic web. Checks are automatically retried until the necessary conditions are met. Playwright selectors pierce shadow DOM and allow entering frames seamlessly.

## Full isolation in the browser context.

Playwright creates a browser context for each test. Browser context is equivalent to a brand new browser profile. This delivers full test isolation with zero overhead cause creating a new browser context takes only a few milliseconds.

The other important benefit of the isolation is that the authentication state of the context can be saved and reused in all the tests. This allows it to bypass repetitive log-in operations in each test, yet delivering full isolation of independent tests.

## Set of powerful tools.

Playwright comes with a set of powerful tools which allow QA engineers to create, modify and debug automated tests much faster. First of all, it's **Codegen** which allows generating tests by recording the user's actions. With **Playwright inspector** it’s possible to inspect a page, generate selectors, step through the test execution, see click points, explore execution logs. **Trace Viewer** allows capturing all the information to investigate the test failure. Playwright trace contains test execution screencast, live DOM snapshots, action explorer, test source, and many more.

**VS Code extension** allows to run a test or a group of tests with a single click. While tests are running, the execution line is highlighted, once the line has completed, step time is rendered as an editor decoration.

### Conclusions

As a conclusion we can say that at qaVisor Playwright is used as a powerful tool, which helped our QA engineers to solve real-life problems for our clients:

With Playwright it’s much easier to automate various user login scenarios for eg. login with Google, Facebook, Apple ID, username and password. It became possible due to browser context isolation and saving authentication state between tests.

Playwright made our automated tests much faster.  It leads to the cost reduction for our clients both for testing with emulators and device farms. More tests can be executed within the same time interval compared to the previous generation tools.

Playwright is a great end-to-end automation solution for the web apps created with javascript frameworks such as React, Vue JS and Angular due to its support of dynamic elements and Shadow DOM.

Since Playwright is based on Node.js it fully integrates in the Javascript ecosystem and it does not require complex or non-trivial configuration and can be easily be used by junior QA engineers.

Playwright works well with most of the CI tools and also has great integration capabilities with 3rd party tools such as test management systems, device farms, reporting tools and a variety of test runners.

Finally Playwright is supported by Microsoft and used for testing its large projects such as Visual Studio Code and Edge browser. So it’s trusted and widely used by large companies and therefore qaVisor recommends Playwright as a tool for web application testing.
