Part2: Have your test cases been suffering from ‘Flakiness’?

This is the second part of a series on Playwright using Typescript and today we are going to talk about challenges in UI Test Framework and explore how leveraging Playwright Best Practices can help us overcome them.

End-to-end test cases have unique challenges due to their complex nature, as they involve testing the entire application user flow from start to finish. These tests often require coordination between different systems and components, making them non-sensitive to environmental inconsistencies and complex dependencies.

What are other challenges we might encounter while working with UI Test Frameworks?

1. Test cases can be slow to execute, as they often involve the entire application stack, including backend, frontend, database.
2. End-to-End tests can be fragile, as they vulnerable to breaking whenever there is a change in DOM, even if the functionality stays the same.
3. UI Tests consume more resources compared to other types of testing, requiring robust infrastructure to run efficiently.
4. This type of test cases suffering from flakiness. Oh, yes, did I say flakiness? It could be a very annoying problem.

Flaky tests pose a risk to the integrity of the testing process and the product. I would refer to great resource where The Domino Effect of Flaky Tests described.

Main idea: while a single test with a flaky failure rate of 0.05% may seem insignificant, the challenge becomes apparent when dealing with numerous tests. An insightful article highlights this issue by demonstrating that a test suite of 100 tests, each with a 0.05% flaky failure rate, yields an overall success rate of 95.12%. However, in larger-scale applications with thousands of tests, this success rate diminishes significantly. For instance, with 1,000 flaky tests, the success rate drops to a concerning 60.64%. And seems, this problem is real and we have to handle it otherwise it will be “expensive” and annoying for test execution for a large-scale applications.

Remember: Most of the time, flakiness is not the outcome of a bad test framework. Instead, it is the result of how you design the test framework and whether you follow its best practices.

By following best practices and designing your tests carefully, you can prevent many flaky tests from appearing in the first place. That’s why before diving right into the implementation, let’s take a look at best practices for Playwright framework.

1. Locate Elements on the page:

• 👉 Use locators! Playwright provides a whole set of built-in locators. It comes with auto waiting and retry-ability. Auto waiting means that Playwright performs a range of actionability checks on the elements, such as ensuring the element is visible and enabled before it performs the click.

await page.getByLabel('User Name').fill('John');

await page.getByLabel('Password').fill('secret-password');

await page.getByRole('button', { name: 'Sign in' }).click();

await expect(page.getByText('Welcome, John!')).toBeVisible();

• 👉 Prefer user-facing attributes over XPath or CSS selectors when selecting elements. The DOM structure of a web page can easily change, which can lead to failing tests if your locators depend on specific CSS classes or XPath expressions. Instead, use locators that are resilient to changes in the DOM, such as those based on role or text.
• 🚫 Example of locator which could lead to flakiness in the future: page.locator('button.buttonIcon.episode-actions-later');
• ✅ Example of robust locator, which is resilient to DOM change: page.getByRole('button', { name: 'submit' });

• 👉 Make use of built-in codegen tool. Playwright has a test generator, which can generate locators and code for you. By leveraging this tool, you might get the most optimised locator. There is more information on codegen tool and capability to generate locators using VS Code Extension in the introductory article I wrote before.
• 👉 Playwright has an amazing feature of auto-waiting. You can leverage this feature in web-first assertions. In this case, Playwright will wait until the expected condition is met. Consider this example: await expect(page.getByTestId('status')).toHaveText('Submitted'); . Playwright will be re-testing the element with the test id of status until the fetched element has the "Submitted" text. It will re-fetch the element and check it over and over, until the condition is met or until the timeout is reached. By default, the timeout for assertions is set to 5 seconds.
• 🤖 The following assertions will retry until the assertion passes, or the assertion timeout is reached. Note that retrying assertions are async, so you must await them: https://playwright.dev/docs/test-assertions#auto-retrying-assertions
• 🤖 Though you have to be careful, since not every assertion has auto-wait feature, please find them in the link by following this link: https://playwright.dev/docs/test-assertions#non-retrying-assertions.
• ✅ Prefer auto-retrying assertions whenever possible.

​2. Design test cases thoughtfully:

• 👉 Make tests isolated. Each test should be completely isolated, not rely on other tests. This approach improves maintainability, allows parallel execution and make debugging easier.
• To avoid repetition, you might consider using before and after hooks. More ways of achieving isolation in Playwright, you can find by following this link: https://playwright.dev/docs/browser-contexts
• Examples:
• 🚫 Not Isolated test case which assumes that the first test case should always pass and it will be a precondition for the next one (in this case, in the first test case user is logging in, and then test case has been reused in the next one. What if the first test case has been failed?

test('Login', async () => {
  // Login
  await login(username, password);

  // Verify Logged In
  await verifyLoggedIn();
});

test('Create Post', async () => {
  // Assuming already logged in for this test
  // Create Post
  await createPost(title, content);

  // Verify Post Created
  await verifyPost(title, content);
});

• ✅ In order to make test cases isolated, before and after hooks come handy to set up preconditions for the second test case.

describe('Test Login', () => {

  // Login
  await login(username, password);

  // Verify Logged In
  await verifyLoggedIn();

});

describe('Post Management', () => {

  beforeEach(async () => {
    await login(username, password);
  });

  test('Create Post', async () => {
    // Create Post
    await createPost(title, content);

    // Verify Post Created
    await verifyPost(title, content);
  });

   // more test cases could be added
});

• 👉 Keep test cases small and avoid million assertions in one test case. Make sure, that one test case has one reason for test failure. You will thank yourself later for that.
• 👉 Make sure you handle data correctly in the test case. Ensure that each test case is independent and does not rely on the state of previous tests. Initialize or reset the test data as needed before each test to prevent data dependency issues. When testing functionalities that interact with external services or APIs, consider using mock data or stubs to simulate responses.

How to combat flaky tests?

• 👉 Use debugging capabilities of Playwright tool. Run test cases with the flag --debug. This will run tests one by one, and open the inspector and a browser window for each test. it will display a debug inspector and give you insights on what the browser actually did in every step. 
• 👉 Playwright supports verbose logging with the DEBUG environment variable: DEBUG=pw:api npx playwright test. In one of my articles, I also explain how to enable this mode from VSCode Extension.
• 👉 Playwright provides a tracing feature that allows you to capture a detailed log of all the actions and events taking place within the browser. With tracing enabled, you can closely monitor network requests, page loads, and code execution. This feature is helpful for debugging and performance optimization.
• To record a trace during development mode set the --trace flag to on when running your tests: npx playwright test --trace on
• You can then open the HTML report and click on the trace icon to open the trace: npx playwright show-report.
• 👉 You might want to slow down test execution by test.slow() to see more details. Slow test will be given triple the default timeout.
• Example:

import { test, expect } from '@playwright/test';

test('slow test', async ({ page }) => {
  test.slow();
  // ...
});

Conclusion

In conclusion, as you start working with new test automation tool, it’s vital to dive into best practices and familiarize yourself with the tool’s capabilities. Remember, flakiness isn’t solely the fault of the test tool itself; more often than not, it comes from how you utilize and implement it.

Summing up best practices for Playwright:

1. Utilize Locators and prioritize user-facing attributes.
2. Ensure test isolation.
3. Leverage built-in code generation functionalities.
4. Make debugging your ally