Tag: test-automation

Introduction:

In this tutorial, we are going to explore public website: https://practicesoftwaretesting.com

More examples of automation testing friendly websites you can find in the repo throughly curated by Butch Mayhew.

In Playwright, structuring a test suite involves organizing your test cases within descriptive blocks (test.describe) and utilizing setup and teardown functions (test.beforeEach and test.afterEach) to ensure consistent test environments. Here’s a brief description of each component and an example:

1. test.describe block provides a high-level description of the test suite, allowing you to group related test cases together. It helps in organizing tests based on functionality or feature sets.
2. Inside test.describe, individual test cases are defined using the test block. Each test block represents a specific scenario or behavior that you want to verify.
3. test.beforeEach block is used to define setup actions that need to be executed before each test case within the test.describe block. It ensures that the test environment is in a consistent state before each test runs.
4. test.afterEach block is utilized for defining teardown actions that need to be executed after each test case within the test.describe block. It helps in cleaning up the test environment and ensuring that resources are properly released.

Here’s an example demonstrating the structure of a test suite in Playwright:

import { chromium, Browser, Page } from 'playwright';

// Define the test suite
test.describe('Login functionality', () => {
  let browser: Browser;
  let page: Page;

  // Setup before each test case
  test.beforeEach(async () => {
    browser = await chromium.launch();
    page = await browser.newPage();
    await page.goto('https://example.com/login');
  });

  // Teardown after each test case
  test.afterEach(async () => {
    await browser.close();
  });

  // Test case 1: Verify successful login
  test('Successful login', async () => {
    // Test logic for successful login
  });

  // Test case 2: Verify error message on invalid credentials
  test('Error message on invalid credentials', async () => {
    // Test logic for error message on invalid credentials
  });
});

DOM Terminology

Before we start writing test cases, it will be useful to brush up our memory on DOM Terminology

1. HTML tags are simple instructions that tell a web browser how to format text. You can use tags to format italics, line breaks, objects, bullet points, and more. Examples: <input>, <div>, <p>
2. Elements in HTML have attributes; these are additional values that configure the elements or adjust their behavior in various ways to meet the criteria the users want. Sometimes these attributes can have a value and sometimes doesn’t. Refer to Developer Mozilla Website for more information.”Class” and “id” are the most used attributes in HTML. (image: show class attribute, class value)
3. Value in between angle braces is a plain text
4. HTML tags usually come in pairs of Opening and Closing Tags.

Locator Syntax Rules

Locate Element by tag name:

page.locator('img');

Locate by id:

page.locator('.img-fluid');

Locate by class value:

page.locator('.img-fluid');

Locate by attribute:

page.locator('[data-test="nav-home"]');

Combine several selectors:

page.locator('img.img-fluid');

Locate by full class value:

page.locator('[class=collapse d-md-block col-md-3 mb-3]');

Locate by partial text match:

page.locator(':text("Combination")');

Locate by exact text match:

page.locator(':text-is("Combination Pliers")');

XPATH:

As for XPath: it is not recommended approach to locate elements according to Playwright Best Practices:

Source: https://playwright.dev/docs/other-locators#xpath-locator

User-facing Locators.

There are other ways to locate elements by using built-in APIs Playwright provides.

There is one best practice we have to keep in mind: automated tests must focus on verifying that the application code functions as intended for end users, while avoiding reliance on implementation specifics that are not typically visible, accessible, or known to users. Users will only see or interact with the rendered output on the page; therefore, tests should primarily interact with this same rendered output. Playwright documentation: https://playwright.dev/docs/best-practices#test-user-visible-behavior.

There are recommended built-in locators:

1. page.getByRole() to locate by explicit and implicit accessibility attributes.
2. page.getByText() to locate by text content.
3. page.getByLabel() to locate a form control by associated label’s text.
4. page.getByPlaceholder() to locate an input by placeholder.
5. page.getByAltText() to locate an element, usually image, by its text alternative.
6. page.getByTitle() to locate an element by its title attribute.
7. page.getByTestId() to locate an element based on its data-testid attribute (other attributes can be configured).

Let’s check out the example:

test('User facing locators', async({page}) => {
  await page.getByPlaceholder('Search').click();
  await page.getByPlaceholder('Search').fill("Hand Tools");
  await page.getByRole('button', {name: "Search"}).click();
  await expect (page.getByRole('heading', {name: "Searched for: Hand Tools"})).toBeVisible();
})

where we would like to explore search functional test:

1. click on the Search Placeholder

await page.getByPlaceholder('Search').click();

2. enter “Hand Tools” text to search for available items.

await page.getByPlaceholder('Search').fill("Hand Tools");

3. locate Search button and click it to confirm.

4. Then we have to verify if no items have been found by asserting text on this page:

await expect (page.getByRole('heading', {name: "Searched for: Hand Tools"})).toBeVisible();

5. Run this test case and make sure test is passing.

Assertions

Playwright incorporates test assertions utilizing the expect function. To perform an assertion, utilize expect(value) and select a matcher that best represents the expectation. Various generic matchers such as toEqual, toContain, and toBeTruthy are available to assert various conditions.

General Assertions

// Using toEqual matcher
test('Adding numbers', async () => {
  const result = 10 + 5;
  expect(result).toEqual(15);
});

Assert that the title of the product is “Combination Pliers”.

const element = page.locator('.col-md-9 .container').first().locator('.card-title');
const text = element.textContent();
expect(text).toEqual('Combination Pliers');

Locator Assertions

Playwright provides asynchronous matchers, ensuring they wait until the expected condition is fulfilled. For instance, in the following scenario:

const element = page.locator('.col-md-9 .container').first().locator('.card-title');
await expect(element).toHaveText('Combination Pliers');

!Note: do not forget to use await when asserting locators

Playwright continuously checks the element with the test id of “status” until it contains the text “Combination Pliers”. This process involves repeated fetching and verification of the element until either the condition is satisfied or the timeout limit is reached. You have the option to either specify a custom timeout or configure it globally using the testConfig.expect value in the test configuration.

By default, the timeout duration for assertions is set to 5 seconds.

There are two types assertion though: Auto-Retrying Assertions and Non-Retrying Assertions.

Auto-Retrying assertions provided below will automatically retry until they pass successfully or until the assertion timeout is exceeded. It’s important to note that these retrying assertions operate asynchronously, necessitating the use of the await keyword before them.

Non-Retrying assertions enable testing various conditions but do not automatically retry.

It’s advisable to prioritize auto-retrying assertions whenever feasible.

Soft Assertions

As a default behavior, when an assertion fails, it terminates the test execution. However, Playwright offers support for soft assertions. In soft assertions, failure doesn’t immediately stop the test execution; instead, it marks the test as failed while allowing further execution.

For example, if we take the previous example and put .soft it assertion, in case assertion fails, it will not lead to termination of test execution.

const element = page.locator('.col-md-9 .container').first().locator('.card-title');
await expect.soft(element).toHaveText('Combination Pliers');

Conclusion.

In conclusion, we’ve explored the aspects of writing test cases using Playwright. We delved into the standard structure of a test case, incorporating essential elements such as hooks and grouping for efficient test management. Additionally, we examined various strategies for locating elements within web pages. Lastly, we discussed the importance of assertions in verifying expected behaviors, covering different assertion techniques to ensure robust and reliable testing. Examples of code, you can see in repository.

This is the second part of a series on Playwright using Typescript and today we are going to talk about challenges in UI Test Framework and explore how leveraging Playwright Best Practices can help us overcome them.

End-to-end test cases have unique challenges due to their complex nature, as they involve testing the entire application user flow from start to finish. These tests often require coordination between different systems and components, making them non-sensitive to environmental inconsistencies and complex dependencies.

What are other challenges we might encounter while working with UI Test Frameworks?

1. Test cases can be slow to execute, as they often involve the entire application stack, including backend, frontend, database.
2. End-to-End tests can be fragile, as they vulnerable to breaking whenever there is a change in DOM, even if the functionality stays the same.
3. UI Tests consume more resources compared to other types of testing, requiring robust infrastructure to run efficiently.
4. This type of test cases suffering from flakiness. Oh, yes, did I say flakiness? It could be a very annoying problem.

Flaky tests pose a risk to the integrity of the testing process and the product. I would refer to great resource where The Domino Effect of Flaky Tests described.

Main idea: while a single test with a flaky failure rate of 0.05% may seem insignificant, the challenge becomes apparent when dealing with numerous tests. An insightful article highlights this issue by demonstrating that a test suite of 100 tests, each with a 0.05% flaky failure rate, yields an overall success rate of 95.12%. However, in larger-scale applications with thousands of tests, this success rate diminishes significantly. For instance, with 1,000 flaky tests, the success rate drops to a concerning 60.64%. And seems, this problem is real and we have to handle it otherwise it will be “expensive” and annoying for test execution for a large-scale applications.

Remember: Most of the time, flakiness is not the outcome of a bad test framework. Instead, it is the result of how you design the test framework and whether you follow its best practices.

By following best practices and designing your tests carefully, you can prevent many flaky tests from appearing in the first place. That’s why before diving right into the implementation, let’s take a look at best practices for Playwright framework.

1. Locate Elements on the page:

• 👉 Use locators! Playwright provides a whole set of built-in locators. It comes with auto waiting and retry-ability. Auto waiting means that Playwright performs a range of actionability checks on the elements, such as ensuring the element is visible and enabled before it performs the click.

await page.getByLabel('User Name').fill('John');

await page.getByLabel('Password').fill('secret-password');

await page.getByRole('button', { name: 'Sign in' }).click();

await expect(page.getByText('Welcome, John!')).toBeVisible();

• 👉 Prefer user-facing attributes over XPath or CSS selectors when selecting elements. The DOM structure of a web page can easily change, which can lead to failing tests if your locators depend on specific CSS classes or XPath expressions. Instead, use locators that are resilient to changes in the DOM, such as those based on role or text.
• 🚫 Example of locator which could lead to flakiness in the future: page.locator('button.buttonIcon.episode-actions-later');
• ✅ Example of robust locator, which is resilient to DOM change: page.getByRole('button', { name: 'submit' });

• 👉 Make use of built-in codegen tool. Playwright has a test generator, which can generate locators and code for you. By leveraging this tool, you might get the most optimised locator. There is more information on codegen tool and capability to generate locators using VS Code Extension in the introductory article I wrote before.
• 👉 Playwright has an amazing feature of auto-waiting. You can leverage this feature in web-first assertions. In this case, Playwright will wait until the expected condition is met. Consider this example: await expect(page.getByTestId('status')).toHaveText('Submitted'); . Playwright will be re-testing the element with the test id of status until the fetched element has the "Submitted" text. It will re-fetch the element and check it over and over, until the condition is met or until the timeout is reached. By default, the timeout for assertions is set to 5 seconds.
• 🤖 The following assertions will retry until the assertion passes, or the assertion timeout is reached. Note that retrying assertions are async, so you must await them: https://playwright.dev/docs/test-assertions#auto-retrying-assertions
• 🤖 Though you have to be careful, since not every assertion has auto-wait feature, please find them in the link by following this link: https://playwright.dev/docs/test-assertions#non-retrying-assertions.
• ✅ Prefer auto-retrying assertions whenever possible.

​2. Design test cases thoughtfully:

• 👉 Make tests isolated. Each test should be completely isolated, not rely on other tests. This approach improves maintainability, allows parallel execution and make debugging easier.
• To avoid repetition, you might consider using before and after hooks. More ways of achieving isolation in Playwright, you can find by following this link: https://playwright.dev/docs/browser-contexts
• Examples:
• 🚫 Not Isolated test case which assumes that the first test case should always pass and it will be a precondition for the next one (in this case, in the first test case user is logging in, and then test case has been reused in the next one. What if the first test case has been failed?

test('Login', async () => {
  // Login
  await login(username, password);

  // Verify Logged In
  await verifyLoggedIn();
});

test('Create Post', async () => {
  // Assuming already logged in for this test
  // Create Post
  await createPost(title, content);

  // Verify Post Created
  await verifyPost(title, content);
});

• ✅ In order to make test cases isolated, before and after hooks come handy to set up preconditions for the second test case.

describe('Test Login', () => {

  // Login
  await login(username, password);

  // Verify Logged In
  await verifyLoggedIn();

});

describe('Post Management', () => {

  beforeEach(async () => {
    await login(username, password);
  });

  test('Create Post', async () => {
    // Create Post
    await createPost(title, content);

    // Verify Post Created
    await verifyPost(title, content);
  });

   // more test cases could be added
});

• 👉 Keep test cases small and avoid million assertions in one test case. Make sure, that one test case has one reason for test failure. You will thank yourself later for that.
• 👉 Make sure you handle data correctly in the test case. Ensure that each test case is independent and does not rely on the state of previous tests. Initialize or reset the test data as needed before each test to prevent data dependency issues. When testing functionalities that interact with external services or APIs, consider using mock data or stubs to simulate responses.

How to combat flaky tests?

• 👉 Use debugging capabilities of Playwright tool. Run test cases with the flag --debug. This will run tests one by one, and open the inspector and a browser window for each test. it will display a debug inspector and give you insights on what the browser actually did in every step. 
• 👉 Playwright supports verbose logging with the DEBUG environment variable: DEBUG=pw:api npx playwright test. In one of my articles, I also explain how to enable this mode from VSCode Extension.
• 👉 Playwright provides a tracing feature that allows you to capture a detailed log of all the actions and events taking place within the browser. With tracing enabled, you can closely monitor network requests, page loads, and code execution. This feature is helpful for debugging and performance optimization.
• To record a trace during development mode set the --trace flag to on when running your tests: npx playwright test --trace on
• You can then open the HTML report and click on the trace icon to open the trace: npx playwright show-report.
• 👉 You might want to slow down test execution by test.slow() to see more details. Slow test will be given triple the default timeout.
• Example:

import { test, expect } from '@playwright/test';

test('slow test', async ({ page }) => {
  test.slow();
  // ...
});

Conclusion

In conclusion, as you start working with new test automation tool, it’s vital to dive into best practices and familiarize yourself with the tool’s capabilities. Remember, flakiness isn’t solely the fault of the test tool itself; more often than not, it comes from how you utilize and implement it.

Summing up best practices for Playwright:

1. Utilize Locators and prioritize user-facing attributes.
2. Ensure test isolation.
3. Leverage built-in code generation functionalities.
4. Make debugging your ally